From 2ef3a1d8135a1bece5574e8e21e4f9abfa35f644 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 5 Jul 2026 23:05:27 +0800 Subject: [PATCH 001/334] =?UTF-8?q?[doc](catalog)=20P6=20iceberg=20done=20?= =?UTF-8?q?+=20merged=20(#64688);=20sync=20handoff/progress=20=E2=86=92=20?= =?UTF-8?q?P7=20hive?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P6 iceberg migration + tests are complete and squash-merged into upstream branch-catalog-spi as #64688 (8b391c7459d). Overwrite the rolling HANDOFF and sync all related progress trackers to reflect "P6 DONE → next = P7 hive/HMS". - HANDOFF.md: full rewrite to a P7-kickoff handoff (plan pointers to master-plan §3.8 + connectors/hive.md, sub-phases P7.1-P7.5, R-002 ACID gate, reusable ops/commit rules). - PROGRESS.md: §一 P6→100%/merged + P7→active + global ~64%; §二 kanban iceberg→100%/merged; §三 P6 block→P7 block; §四 new 2026-07-05 entry; §六 D-073/DV-049 counts; §七 session status. - connectors/iceberg.md: status rows → ✅ merged #64688; drop DV-038 flip-blocker label; progress-log entry. - connectors/hive.md: status → active P7 target; progress-log entry. - tasks/P6-iceberg-migration.md + P6.6-flip-blockers-tasklist.md: status banners → ✅ COMPLETE / merged #64688. Note: #64688 removed the native iceberg subsystem but 23 fe-core datasource/iceberg/ HMS-iceberg support classes remain (iceberg-on-HMS still routes through fe-core); they are deleted with P7 hive (阶段四, decision D5/Q3=B). Recorded in all touched docs. Docs only, no product code. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 480 ++---------------- plan-doc/PROGRESS.md | 30 +- plan-doc/connectors/hive.md | 9 +- plan-doc/connectors/iceberg.md | 12 +- plan-doc/tasks/P6-iceberg-migration.md | 4 +- .../P6.6-iceberg-flip-blockers-tasklist.md | 2 +- 6 files changed, 83 insertions(+), 454 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 9125c3e121454f..87d8cf1e9663cf 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -1,461 +1,79 @@ # 🤝 Session Handoff > 滚动文档：每次 session 结束**覆盖式更新**（不累积历史；历史见 `git log plan-doc/HANDOFF.md`）。协作规范：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md) -> **范围** = catalog-spi **主线** handoff。metastore/storage 抽取子线**已 CLOSED**（2026-06-22 合入 #64446/#64653/#64655）——`metastore-storage-refactor/` 仅历史留存、勿读；metastore-spi 现状直接读代码 `fe/fe-connector/fe-connector-metastore-spi/`。 +> **范围** = catalog-spi **主线** handoff。metastore/storage 抽取子线**已 CLOSED**（合入 #64446/#64653/#64655）——`metastore-storage-refactor/` 仅历史留存、勿读；metastore-spi 现状直接读代码 `fe/fe-connector/fe-connector-metastore-spi/`。 --- -# 🚨 架构裁定升级（2026-07-05 续² 末，用户拍板）= **fe-core 不再持有任何属性解析/组装；已迁移连接器（paimon+iceberg）完全走"插件解析 → BE thrift → 回传 fe-core"** +# 🎯 当前状态（2026-07-05）= **P6 iceberg 迁移 + 测试全部完成，已 squash-合入 `branch-catalog-spi`（#64688 `8b391c7459d`）；下一阶段 = P7 hive/HMS** -> **触发**：本轮做属性/鉴权迁移到连接器时，vended 判定撞到"完整删除 vs 铁律 vs 风险"三者不可兼得；查 Trino（连接器独占存储/凭据解析，引擎从不建静态凭据表、也不必知 vended）后，用户升级为更根本的架构原则： -> - **fe-core 不应再持有任何属性解析能力**，全部由插件完成。 -> - **storage 属性解析** → **fe-filesystem 模块**（插件侧）；**meta 属性解析** → **fe-connector**（插件侧）。 -> - **数据流**：插件组装 → 传给 BE 的 thrift → 再回传给 fe-core（fe-core 只接收成品、不自己解析）。 -> - **范围**：未迁移连接器暂留残留代码；但 **paimon 与 iceberg 必须完全按新架构走**（=paimon 也纳入范围，不再是 iceberg-only）。 +> **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： +> P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 > -> **影响（重估原 P5 7 刀计划）**：① 原"属性簇删除 + rewire"是这个大原则的**子集/前奏**，需按新架构重写设计；② **CUT 4（在 fe-core `CatalogProperty` 加 warehouse→fs.defaultFS helper）方向反了**（那仍是 fe-core 解析 storage）——应搬到插件/fe-filesystem 侧，待新设计定夺后可能 revert/改向；③ CUT 1（连接器自建 HMS 鉴权器）方向对（meta→连接器）；④ CUT 2（SHOW CREATE 脱敏在 fe-core）多半保留（展示层），但敏感键理想应由插件供。 -> -> **✅ 完整新设计已出 = `plan-doc/tasks/designs/plugin-owns-property-parsing-arch-design.md`**（依据侦察工作流 `wf_61c70f0d-bce`，替代/升级 iceberg-metastore-auth-connector-rewire-design.md）。**关键实证（大幅利好）：目标架构大部分已就位** —— `fe-filesystem-api/spi` 已是插件可用共享模块（parent-first、排除打包，无须抽取）；连接器读/扫描路径**已** fe-filesystem-native；fe-core `StorageProperties.createAll` 只是**冗余的第二份解析**。故本迁移 = **退掉 fe-core 第二份解析**（10 步 S1-S10），非新建模块，比预想小。**vended 难题也顺带解掉**：fe-core 对所有插件 catalog **无条件停建静态存储表**（插件 100% 拥有 static+vended），不判别、不 gate、守铁律。 -> **✅ 中央决策已裁定（用户 2026-07-05）= A 家族：共享宿主 classpath + 连接器发起 bind**（fe-filesystem 解析器留宿主 parent-first、不打包；连接器直接调 fe-filesystem-api 发起 bind，不经 fe-core context；fe-core 零解析。B 每插件打包不采纳）。 -> **✅✅ S1–S10 全部落地（2026-07-05 续⁵，见下方最新一轮）= 已迁连接器（iceberg+paimon）属性解析全归插件、fe-core 零解析目标达成。** S8 用「连接器发起派生」变体落地：新增中立 SPI `Connector.deriveStorageProperties`，iceberg 连接器自持 warehouse→fs.defaultFS（仅 hadoop flavor），fe-core 插件路径经 `CatalogProperty.setPluginDerivedStorageDefaultsSupplier` 折叠、不再调 getMetastoreProperties；S7 删 fe-core Iceberg/Paimon 簇（-4914 行，含 recon 新发现的 5 个 fe-core iceberg 连通性探测器 + 协调器 iceberg 分支）。⚠️ legacy 静态 map 方法（`CatalogProperty` getStoragePropertiesMap/getBackendStorageProperties/getHadoopProperties/getMetastoreProperties + `StorageProperties.createAll` + 共享基类 HMSBaseProperties/AWSGlueMetaStoreBaseProperties/AliyunDLFBaseProperties/AbstractMetastorePropertiesFactory）已按 S10 铁律**保活**，供 Hive/Hudi/LakeSoul/HMS-iceberg 继续用，直到它们全离 SPI（阶段四）。 -> **⏭ 下个 session 起步 = 回翻闸主线**（连通性 F2/F3/F15/F16 → ENG-3 flip-gated e2e 全跑 → 用户二签翻闸最后原子提交）；属性簇迁移子线已 CLOSED。 -> -> **已 commit 未 push（前序轮，[DEC-FLIP-1] 铁律）**：CUT 1 `cf8dda9f058` / CUT 2 `eb9201dc0a6` / CUT 4 `0de34db83fb`。**CUT 4 已由 S8 `3e7a687e6e2` 接管——`deriveHdfsDefaultFsFromWarehouse` helper 已从 fe-core 删除、逻辑搬入 iceberg 连接器。** - ---- - -# 🎯 最新（2026-07-10 续²）= **P6.6-C6 #63068 OIDC 会话凭证迁移：Tasks 0–7 全部 DONE + 三层 commit stack + clean-room 复审通过（未 push）** - -> ⚠️ 本段最新，**取代下方所有 🎯 历史**。**三层 commit stack（未 push，[DEC-FLIP-1] 铁律）**：SPI `55c991d0e87`（fe-connector-api：中立 DTO + 能力位）→ FE `930e4778064`（fe-core：能力位门控注入 + 缓存旁路 + 2 测）→ 连接器 `a07216ef102`（iceberg：per-user 路由 + 适配器 + 属性校验 + 4 测）。设计权威 = `plan-doc/P6.6-C6-oidc-session-migration-design.md` 第 12 节 + `...-research-notes.md`。 - -## ✅ P6.6-C6 实现完成（起步先读本块 + 设计文档；行号可能过时、信控制流不信注释） - -**Tasks 0–7 全部 DONE。验证：连接器 full test（禁 cache）GREEN（966 用例 0 fail）；fe-core 注入 + 旁路门 test GREEN；三模块 checkstyle 0。下方 Task 1–5 描述仍准（历史细节）。** - -- **Task 1 SPI（fe-connector-api）**：新增中立 DTO `ConnectorDelegatedCredential`（Type{ACCESS_TOKEN,ID_TOKEN,JWT,SAML}+token+OptionalLong expiresAtMillis+isExpired；toString 脱敏）；`ConnectorSession` 加 `getDelegatedCredential():Optional`（default empty）+ `getSessionId():String`（default=getQueryId 兜底）；`ConnectorCapability` 加 `SUPPORTS_USER_SESSION`。 -- **Task 2 FE 注入（fe-core）**：`ConnectorSessionImpl` 加 sessionId+delegatedCredential 字段+构造参+getSessionId/getDelegatedCredential 覆写；`ConnectorSessionBuilder.from(ctx)` 存 ctx，加 `withUserSessionCapability(bool)`/`withSessionId`/`withDelegatedCredential`，`build()` 仅当 capable 时从 `ctx.getSessionContext()` 拉凭证+sessionId 映射进中立 DTO（`toConnectorCredential` 按 enum name 桥接，不匹配即 fail-loud）；`PluginDrivenExternalCatalog.buildConnectorSession()` **from(ctx) 分支**加 `.withUserSessionCapability(supportsUserSession())`（create() 后台分支不带凭证=O3 后台跳过雏形），加私有 `supportsUserSession()`。 -- **Task 3 缓存旁路（fe-core）**：**大发现——表级旁路机制在 rebase 后【保留】**（`ExternalCatalog.shouldBypassTableNameCache` 默认 false + `ExternalDatabase.loadTableNamePairs/getTableNullableWithoutCache/matchesLocalTableName` 全在，接入 isTableExist/getTableNamesWithLock/getTableNullable），此前无人 override→休眠。本轮只加 `PluginDrivenExternalCatalog.shouldBypassTableNameCache` override（能力位+凭证在场）。**O2 库级旁路是真新增**（旧在已删 IcebergRestExternalCatalog）：`ExternalCatalog` 新增 `shouldBypassDbNameCache` 默认 false + `getFilteredDatabaseNames(boolean updateDbNameLookup)` 重载（bypass 传 false 不动共享 lowerCaseToDatabaseName）+ getDbNames/getDbNullable 加 bypass 分支 + `getDbNullableWithoutCache`（buildDbForInit checkExists=false 防 getDbNames 递归）+ `matchesLocalDbName`；`PluginDrivenExternalCatalog.shouldBypassDbNameCache` override 同门控。 -- **Task 4 连接器属性**：`IcebergConnectorProperties` 加 REST_SESSION/SESSION_NONE/SESSION_USER/REST_DELEGATED_TOKEN_MODE/DELEGATED_TOKEN_MODE_*/REST_SESSION_TIMEOUT 常量；`IcebergRestMetaStoreProperties`（**验证在此跑**，经 IcebergConnectorProvider.validateProperties→bindForType(rest)→validate()）加 @ConnectorProperty session+delegatedTokenMode 字段 + `validateUserSession()`（session/token-mode 枚举 + session=user⇒oauth2）+ **放宽** oauth2-requires-token（session=user 不要求静态 cred，加 `!isUserSession()`）；本模块不依赖 fe-connector-iceberg 故内联本地常量；`IcebergConnector.getCapabilities()` 条件加 SUPPORTS_USER_SESSION（新 `isUserSessionEnabled()`=session=user && rest）。 -- **Task 5 连接器会话目录+适配器**：新增 `IcebergDelegatedCredentialUtils`（中立 Type→OAuth2Properties key）；新增 `IcebergSessionCatalogAdapter`（持 Catalog+Optional+DelegatedTokenMode；catalog/delegatedCatalog/viewCatalog/delegatedViewCatalog(ConnectorSession)+static toIcebergSessionContext；**fail-closed** delegated* 无凭证抛 DorisConnectorException；含 nested enum DelegatedTokenMode.fromString）；`IcebergConnector` 加 restSessionCatalog+sessionCatalogAdapter 字段 + `buildRestSessionCatalogDefault`（session=user REST 走 `new RESTSessionCatalog()`+configureHadoopConf+initialize+asCatalog(empty)，删 catalog-impl 键，session-timeout→AUTH_SESSION_TIMEOUT_MS）+ `newCatalogBackedOps(session)` 重载（session=user 经 adapter.delegatedCatalog/delegatedViewCatalog 拿 per-user）+ getMetadata 改用它 + close() 关 restSessionCatalog。 - - **关键坑（已解，验过 iceberg 1.6/1.10 javap）**：`RESTSessionCatalog.asCatalog(ctx)` 返回 `AsCatalog implements Catalog,SupportsNamespaces` 但 **NOT ViewCatalog**（视图要单独 `asViewCatalog(ctx)`）。故给 `CatalogBackedIcebergCatalogOps` 加**可选显式 viewCatalog 参**（新 6 参构造；旧 5 参 delegate viewCatalog=null；`resolveViewCatalog()`=显式优先否则 catalog instanceof ViewCatalog；isViewCatalogEnabled+5 处 `((ViewCatalog)catalog)`→resolveViewCatalog()）。session=none 行为逐字不变。 - - **为何元数据路由这么省**：`PluginDrivenExternalCatalog` **每 op 都新建** `connector.getMetadata(session)` 且同 session 传给后续方法（20+ 站点核实），故只在 `IcebergConnector.newCatalogBackedOps(session)` 一处 resolve per-user catalog 即覆盖所有元数据 op，**不动 1765 行 IcebergConnectorMetadata、不动 ops seam**。 - -### ✅ Task 6 DONE（扫描/写/过程 session 路由，commit `a07216ef102`） -- 3 provider 各加 `Function catalogOpsResolver`；connector 传 `this::newCatalogBackedOps`；**旧构造器 = 常量 `s->catalogOps`**（保测试/普通目录 byte 不变）。`resolveTable`/`resolveSysTable`/`runInAuthScope`/`planInAuthScope` → `resolver.apply(session).loadTable(...)`，**先解析 ops 出 auth 作用域**（fail-closed DorisConnectorException 原样抛、不被 executeAuthenticated 的 catch 包裹）。**写入 COMMIT 本就 per-user**（`IcebergConnectorTransaction` 经 `beginTransaction(session)` 继承 session-aware metadata ops），无需改。**测试适配**：`IcebergConnectorTest.catalogOpsOf` 反射 `catalogOps`→`catalogOpsResolver`（`apply(null)`，非 user-session 忽略 session）；`IcebergProcedureOpsTest` 裸 `null` 参加 `(IcebergCatalogOps)` cast 消歧两构造器（bare null 同时匹配 ops/Function 构造器）。 - -### ✅ Task 7 DONE（测试 + clean-room 复审 + commit） -- **新增/改测**：`IcebergProviderSessionRoutingTest`(4：每 provider resolver 用 call session + fail-closed) / `IcebergSessionCatalogAdapterTest`(7：凭证 key 每模式 access_token=verbatim bearer、token_exchange=typed key + fail-closed + fromString) / `IcebergConnectorValidatePropertiesTest`+5(session=user⇒oauth2 / 放宽静态 token / 非 user 仍需 token / 非法 session/token-mode) / `ConnectorSessionImplTest`+3(capable→注入 / 非 capable→跳过 / capable 无 cred) / `PluginDrivenExternalCatalogSessionBypassTest`(3：旁路判定门=能力位&凭证在场) / **`ExternalDatabaseSessionContextTest`(1，commit `e83f2b77bc4`：#63068 缓存泄漏**数据流**重迁——mockStatic(SessionContext.current) 驱动每令牌身份 + TestableCatalog 重写 listDatabaseNames 每令牌返回各自库并记录令牌；两用户结果不相交 + 每次读都 live 重列(含重复令牌)=证无跨用户泄漏+无共享缓存;#63068 的"bootstrap 无令牌读"在本分支 fail-close，故改用"每读 live 记录令牌"作等价可观测)。 -- **Task 0 保留基座已核实 GREEN**（本会话 `mvn test -pl fe-core -am`）：DelegatedCredentialTest 3 / ConnectProcessorDelegatedCredentialTest 5 / FEOpExecutorDelegatedCredentialTest 1 / MysqlAuthPacketCredentialExtractorTest 4 = 13/13。通用 auth 抓凭证+SessionContext+thrift 1004-1007 转发全在。 -- **clean-room 对抗复审（自审 + 独立 agent 交叉核对；subagent 首轮环境故障 0 tool-use、重派次轮 25 tool-use 成功且独立复现结论）**： - - ① **路由完整性 = 零缺口**：每个 query-time 表/视图/命名空间 load 都经 per-request session；shared `asCatalog(empty)` 仅被 **0-caller 的 graceful `catalog(session)/viewCatalog(session)`** 和 `!isUserSessionEnabled()` 门控的 no-arg `newCatalogBackedOps()` 用；`new IcebergConnectorTransaction` **唯一站点**=`IcebergConnectorMetadata.beginTransaction`(session-aware)；`testConnection` 是 CREATE-CATALOG-time（非 query 路径）；method-ref `this::newCatalogBackedOps` 正确解析到 1-参 session-aware 重载。 - - ② **缓存泄漏 = 无**：旁路走 live `loadTableNamePairs(ctx,false)`/`getFilteredDatabaseNames(false)`，`updateLookup=false` 时**不动**共享 `lowerCaseTo{Table,Database}Name`/`metaCache`（ExternalDatabase:186/192/201/230、ExternalCatalog:607/628）；库级旁路仍 re-append `information_schema`+`mysql`（ExternalCatalog:599-602）；无凭证→`SessionContext.empty()`/`current()`→连接器 fail-closed 抛→共享缓存**永不被 per-user 数据污染**。 - -### ⚠️ 遗留 / follow-up（非阻塞） -- **O3 后台日志降噪 = 未做（登记独立小 follow-up）**：用户裁定"保留无令牌即拒硬 parity（**DONE**）+ 后台无令牌日志降噪"。**降噪经查非小改动**（当初 option 框定失准）——噪音出在后台统计守护 `StatisticsAutoCollector`(:109 通用 catch + :187) 对 session=user 表 `getRowCount`→`getMetadata` fail-closed 抛的 WARN，且拒绝异常穿 `PluginDrivenExternalTable.fetchRowCount`→行数缓存边界（要精确只静音这一种、不掩盖真失败、不误伤手动 ANALYZE，须 marker 异常 + 遍历 cause 链 + 动后台守护 catch）。故**不塞进本安全提交**，作独立 follow-up（识别拒绝异常→后台 catch 降 DEBUG）。触发＝仅"session=user 且对其开自动统计"边缘配置。 -- **复审 2 小 nit（非泄漏非阻塞）**：① `ExternalDatabase.matchesLocalTableName` 纯大小写敏感 exact 模式用 `equals`，而缓存路走小写 map（近似大小写不敏感）——罕见 case-sensitive 外表下有分歧、旁路更严格，非泄漏；② `isTableExist/getTableNullable` 非旁路分支用 `SessionContext.current()` 填缓存 vs 旁路判定用参 `sessionContext`——实践同源、仅理论不一致。③ 适配器 graceful `catalog(session)/viewCatalog(session)` **零调用方**（连接器只用 fail-closed 变体；保留对称、非 bug，可后续清）。 -- **flip-gated e2e 全未跑**（本地无 REST 会话集群 Polaris/Lakekeeper）：per-user 真绑定/token_exchange/跨 FE sessionId 稳定性待翻闸后 e2e。 - ---- - - -- **tier3（#65094 读路径列名大小写保留）DONE — commit `ea8b1c1e0ff` (2026-07-10)**：iceberg+paimon 外表**顶层**列名保留远端大小写（纯 FE、**BE 不动**；字节匹配不变式 slot名==TSchema顶层名 两侧一起从小写翻成保留大小写；嵌套 struct 子列保持不变）。both-connector `mvn package -pl :fe-connector-iceberg,:fe-connector-paimon -am`(禁 cache) **GREEN，0 checkstyle，0 fail**。含审计漏项 **IcebergPartitionUtils:627**（listPartitions 值map key，是 :164 的孪生；漏了会让混合大小写 iceberg 分区列在 MTMV/裁剪丢分区值）+ 回归测试。 - -- **#63068 OIDC 会话凭证迁移 — 设计文档完成，待实现（用户 2026-07-10 拍板全采纳推荐）**： - - 文档：**`plan-doc/P6.6-C6-oidc-session-migration-design.md`**（第 12 节 = 7 步 TODO）+ **`...-research-notes.md`**（Trino 源码 + #63068 `e545f1ad08a` as-built + 当前 SPI，全 source-verified via 2 subagents）。 - - **关键发现**：通用 session 基座（MySQL鉴权捕获 / `SessionContext` / `DelegatedCredential` / thrift 1004-1007 跨FE转发 / `ConnectProcessor` / `FEOpExecutor`）本分支**全保留且可用** → `ConnectContext.get().getSessionContext().getDelegatedCredential()` 已在查询线程可取。迁移只剩 iceberg 消费侧（3 删类 `IcebergUserSessionCatalog`/`SessionCatalogAdapter`/`DelegatedCredentialUtils` + 路由 + `iceberg.rest.session` 配置 + 缓存旁路），**从 fe-core 重定向到 `fe-connector-iceberg` SPI**。 - - **决策（已定，全采纳推荐）**：**O1**=加 `ConnectorCapability.SUPPORTS_USER_SESSION`（能力位门控 FE 注入；最小权限）；**O2**=库级缓存旁路放通用 `PluginDrivenExternalCatalog`、能力位+凭证在场门控；**O3**=后台/异步任务(preload/auto-analyze/refresh)在 `session=user` 下**跳过**，交互查询**fail-closed**（无凭证即抛，同 #63068）；**O4**=SPI `getSessionId()` **复用转发的 `SessionContext.sessionId`**（稳定 AuthSession key），不用 queryId；**subject-JWT=不做**（按 #63068 verbatim 传 token：access_token / token_exchange 两模式；Trino 式 `sub=` 签名列 follow-up）。 - - **架构对齐**：凭证走**中立 SPI DTO** `ConnectorDelegatedCredential`（连接器不 import fe-core），**一次性注入在 `PluginDrivenExternalCatalog.buildConnectorSession()`**（Trino 式 `session.getIdentity().getExtraCredentials()`，非 #63068 逐方法 SessionContext 穿参）。iceberg 用**单个共享** `RESTSessionCatalog` + per-request `SessionContext`（勿建 per-user catalog）。 - - **硬约束**：纯 FE、不改 BE；`session=user` 的 fail-closed 与缓存旁路是**安全语义**（防跨用户缓存泄漏，参 Trino CVE-2026-34214）需保 parity；凭证勿 editlog/持久化/SHOW/profile/information_schema。 - ---- - -# 🎯 最新一轮（2026-07-05 续⁵）= **属性簇迁移收官 S7–S10：连接器自持存储派生 + 删除 fe-core Iceberg/Paimon 元存储簇 = 2 代码 commit（未 push）** - -> **本轮范围** = 承接 S6，一次把 **S7、S8、S9、S10** 全部处理掉（用户本 session 明确要求）。先跑 6-agent 侦察工作流（`wf_225ef82c-53c`：删除安全闭包 / 插件 getMetastoreProperties 闭包 / S8 派生 parity / S9-S10-auth 死码 + 2 路对抗核验），**对抗核验揪出设计一句话漏掉的 compile 阻塞**（`datasource/connectivity/` 子系统硬引用 iceberg 簇），据此扩大 S7 删除面。需用户拍板处（派生逻辑放哪）已用中文讲清背景+例子后拿到裁定 = **搬进连接器**。 -> -> **✅ S8 `3e7a687e6e2`（连接器自持 warehouse→fs.defaultFS，插件路径退掉 fe-core 元存储解析）**：新增中立 SPI `Connector.deriveStorageProperties(rawProps)`（默认空）；`IcebergConnector` 实现 hadoop-flavor `warehouse=hdfs://ns/path`→`fs.defaultFS=hdfs://ns`（**仅 hadoop 类型派生**，逐字节镜像原 `IcebergFileSystemMetaStoreProperties.getDerivedStorageProperties`，铁律：连接器 inline hdfs 常量、无 fe-core import）；`PluginDrivenExternalCatalog.initLocalObjectsImpl` 经 `CatalogProperty.setPluginDerivedStorageDefaultsSupplier` 把连接器派生结果懒折叠进存储表，**FE 绑定（raw supplier）与 BE 扫描（typed supplier）两路同源**；`CatalogProperty` 的 `initStorageProperties`/`getEffectiveRawStorageProperties` 改走 `mergeDerivedStorageDefaults()`→`resolveDerivedStorageDefaults()`：插件 catalog 用 supplier（不调 getMetastoreProperties），legacy Hive/Hudi 仍走 msp。删 fe-core 中立 `deriveHdfsDefaultFsFromWarehouse` helper（搬入连接器）。**为何这是 S7 前置**：un-register 后任何对插件 catalog 的 getMetastoreProperties 都会抛「Unsupported metastore type」，本刀切断插件存储路径对它的**唯一**依赖（recon 实证：typed+raw 两 supplier 是仅有的插件消费点，连通性协调器被 PluginDriven.checkWhenCreating override 绕开）。验收=fe-core 10 测 + iceberg 7 测 0 失败、三模块 checkstyle 0、连接器无 fe-core import。 -> -> **✅ S7 `3c69bfa8265`（删 fe-core Iceberg/Paimon MetastoreProperties 簇 + un-register，−4914 行）**：`MetastoreProperties` 摘 register(Type.ICEBERG/PAIMON)（枚举值留、工厂不注册→fail-loud）+ 删死方法 `initExecutionAuthenticator`；删 **16 簇类**（9 iceberg + 7 paimon）+ **2 property/common 凭据类**（IcebergAwsAssumeRole/ClientCredentials，仅簇内消费）+ **5 fe-core iceberg 连通性探测器** + `CatalogConnectivityTestCoordinator` 摘 4 iceberg 分支/import（Hive 逐字保留）；删 14 簇单测、迁 `CatalogPropertyEffectiveRawStoragePropsTest` 到 supplier 路径。验收=fe-core main+test BUILD SUCCESS（删 37 文件零悬挂引用=闭包完整）+ checkstyle 0 + 28 存储/脱敏/派生用例 + 7 存活 Hive 元存储用例 0 失败。 -> -> **✅ S9（决策落地，无代码）= BE-thrift 方向保持现状（INDIRECT 默认）**：实证插件 iceberg/paimon 的 BE thrift **已全在连接器侧**建（`IcebergScanRange/IcebergScanPlanProvider.populate*`、`PaimonScanRange/PaimonScanPlanProvider.populate*`；fe-core 无 paimon 包；fe-core `IcebergScanNode.setIcebergParams` 只服务 legacy HMS-iceberg）。HDFS/kerberos 保持 INDIRECT（唯一 typed build = `HdfsResource.generateHdfsParam` 单转换器留 fe-core），S3/对象存储已 `params.setProperties(map)`。「插件→BE thrift」原则由 by-reference populate* 接缝结构性满足，无须动码。 -> -> **✅ S10（铁律验证，无代码）= legacy 残留全保活**：`CatalogProperty` getStoragePropertiesMap/getBackendStorageProperties/getHadoopProperties/getMetastoreProperties（:260/:308/:333/:284）+ `StorageProperties.createAll`（:137，main 3 处消费）+ 共享基类（HMSBaseProperties/AWSGlueMetaStoreBaseProperties/AliyunDLFBaseProperties/AbstractMetastorePropertiesFactory）实测均在，供 Hive/Hudi/LakeSoul/HMS-iceberg 直读解析出 BE thrift（清单见 recon `wf_225ef82c-53c` s10KeepAliveConsumers）。**只 repoint 了插件路，未删任何共享方法。** -> -> **⚠️ flip-gated 未验（本地无集群）**：iceberg hadoop-catalog 只带 warehouse 的 HDFS-HA 绑定 e2e（S8 派生逐字节镜像，单测证 FE+BE 折叠一致 + 逻辑，但真绑定 flip 后才能 e2e）；重部署类加载冒烟。**全部未 push（[DEC-FLIP-1] 铁律）。** 侦察结论持久化在 `wf_225ef82c-53c` journal + 设计文档 §4。 - ---- - -# 🎯 上一轮（2026-07-05 续⁴）= **属性簇迁移 S3–S6 一口气落地（写入凭据改绑 / vended 机制退休 / URI 接缝保留决策 / Kerberos 鉴权全归连接器）= 4 代码 commit + 1 doc** - -> **本轮范围** = 承接 S2，按设计 §4 刀序把 **S3、S4、S5、S6** 全部处理掉（用户本 session 明确要求）。逐刀独立 commit + build + test + checkstyle + import-gate；每刀凭据/行为差异都做了核对，需用户拍板处已用中文讲清背景+例子后拿到裁定。**全部未 push（[DEC-FLIP-1] 铁律）。** -> -> **✅ S3 `72680f03995`（iceberg 写入路径 BE 静态凭据改绑 fe-filesystem）**：`IcebergWritePlanProvider.buildHadoopConfig` 由 `context.getBackendStorageProperties()`（fe-core CredentialUtils 解析）改为遍历 `context.getStorageProperties()` 取 `toBackendProperties().toMap()`——与扫描路径同源。**parity 核对**（专派 agent 逐 backend）：带凭据的对象存储/HDFS 写认证键（AWS_*/use_path_style/超时）逐字节一致、HDFS 派生默认值一致；唯一非逐字节差异是老路经 createAll 默认注入的 HDFS 兜底多带两个 BE 对象存储写不读的惰性键（ipc.client.fallback-to-simple-auth-allowed、hdfs.security.authentication=simple），新路不带——**扫描早已在用新形且线上运行=存在性证明够用**。验收=iceberg 939 测 0 失败 + 写测 41 mutation KILLED + checkstyle 0 + import 净。 -> -> **✅ S4 `f2976900852`（vended 凭据机制彻底退休，用户裁定「彻底退休」，−1083 行）**：删 3 类（VendedCredentialsFactory/AbstractVendedCredentialsProvider/IcebergVendedCredentialsProvider + 3 测）；删 `CatalogProperty.shouldBuildStaticStorage` gate，`initStorageProperties`+`getEffectiveRawStorageProperties` 无条件走 `mergeDerivedStorageDefaults`；`IcebergScanNode`(legacy HMS) 直读 `getStoragePropertiesMap()`（HMS-typed 恒 null，逐字等价）；删已死的 `MetastoreProperties.isVendedCredentialsEnabled()`(+PaimonRest 覆写)。**行为变化（用户已接受，flip-gated）**：vended 目录若同时配静态 key，语义由「vended 取代静态」变「vended 叠加静态」（同名仍 vended 优先、最终认证不变，仅额外仅静态键也发 BE）；纯 REST 无变化。验收=fe-core 113 测 0 失败 + checkstyle 0 + import 净。 -> -> **✅ S5（决策落地无代码，用户裁定「保留现状」）= 保留 fe-core 薄 URI 规范化/文件类型接缝**：`normalizeStorageUri`/`getBackendFileType` 已引擎中立（String 进出、无引擎名分支），仅复用已解析存储做 `LocationPath` 规范化。fe-filesystem 无等价 API（`LocationPath`/`TFileType` 均 fe-core-only、fe-filesystem 刻意无 Thrift），迁移代价大且对合并读删有正确性风险，无硬性要求故不做（低优先，将来硬性要求再评估）。 -> -> **✅ S6 前置 `498d3230916` + 落地 `12599958bce`（Kerberos 鉴权全归连接器，用户裁定「本 session 完整做掉、接受 flip-gated 风险」）**：**前置**=paimon 连接器补 HMS 元存储 Kerberos 鉴权器（抽 `PaimonConnector.buildPluginAuthenticator` 静态、镜像 iceberg CUT1、新增 HMS-Kerberos-简单存储分支 + 5 单测），补齐 recon 发现的「paimon 缺 HMS 鉴权器」blocker。**落地**=删 `PluginDrivenExternalCatalog.initPreExecutionAuthenticator` 重写→继承基类 no-op（插件 catalog fe-core handle 非空但不做 doAs，真 doAs 在连接器）；删无调用方的 `getOrderedStoragePropertiesList` getter + 内联字段。验收=fe-core 375 测 + paimon 5 测 0 失败 + 双 checkstyle 0 + import 净。**⚠️ flip-gated**：端到端 Kerberos（对真 KDC doAs）本地无集群不可验，单测只证「鉴权器已构建 + 句柄非空 + 插入不崩」。 -> -> **本轮 recon（4 路后台 agent 侦察，结论已固化进设计 §4）**：S3 parity 逐 backend 核对、S4 vended 控制流全图（gate 双路 + IcebergScanNode 空转 + 无 legacy 回归）、S5 消费者图 + fe-filesystem 无等价、S6 doAs 全图（iceberg 冗余 / paimon HMS blocker）。**下一步 = S7**（退 paimon/iceberg MetastoreProperties 簇 + S8 CUT4 搬回连接器）。 - ---- - -# 🎯 上一轮（2026-07-05 续²）= **属性/鉴权迁移到连接器启动：recon 纠正计划前提 + 用户裁定完整删除 + 设计文档 + CUT 1 done** - -> **本轮范围** = 承接 P4 DONE，启动"属性/鉴权迁移到连接器 + fe-core iceberg 属性簇删除"（removal-execution-plan **§P5**，死码删除收官最大一块）。**做了 recon → 暴露 Rule 7 前提错误 → 用户拍板 → 写设计 → 落 CUT 1**。 -> -> **⚠️⚠️ Rule 7 关键更正（recon `wf_73999dcf-412` 实证，纠正 §P5 / removal-plan v2 / 旧 HANDOFF 的核心前提）**： -> - 旧说"给连接器 metastore provider 补 authenticator 构建（**镜像 paimon 已有做法**）"——**paimon 无此样板**。实证：paimon 的 HMS 鉴权器**也在 fe-core 建**（`PaimonHMSMetaStoreProperties:60`，与 iceberg `IcebergHMSMetaStoreProperties` 同型），paimon 连接器**不**自建 HMS 鉴权器（只在 storage kerberos 时自建，与 iceberg 连接器同）；fe-core paimon 属性簇**完好在用**（非死码）。→ 真删 fe-core iceberg 簇须做 paimon 都没做的新东西（连接器自建 HMS 鉴权器）= CUT 1，**非"照抄 paimon"**。 -> - **方向仍对（pro-Trino）**：Trino 连接器自持元数据鉴权 + 存储凭据解析，引擎核心只给中立 session/identity + doAs 执行面；本仓 metastore-spi 边界正是照此设计。P5 删掉 fe-core 凭据层最后一处引擎专属耦合（iceberg 是 `VendedCredentialsFactory` 唯一残留 case）。 -> - **by-design 保留（P5 后仍在 fe-core，非泄漏）**：`executeAuthenticated` doAs 面、存储凭据 NORMALIZATION 到 BE 规范 AWS_*/hadoop 键、fe-filesystem provider registry（paimon 同赖）。**勿把 normalization 挪进连接器。** -> -> **✅ 用户裁定（2026-07-05）= 完整删除、一次到位**（接受鉴权刀 flip-gated 本地无法验证；已用中文讲清背景/风险/三选项后选定）。 -> -> **✅ 权威设计 = `plan-doc/tasks/designs/iceberg-metastore-auth-connector-rewire-design.md`**（含纠正后前提 + 7 刀序 + 已裁定子决策 D1-D4 + 风险 + 验收口径）。**下个 session/续跑先读它。** 7 刀：**✅CUT 1（`cf8dda9f058`）** → **✅CUT 2（`eb9201dc0a6`，SHOW CREATE 脱敏改绑）** → CUT 3 vended gate 走 raw prop（⚠️见下，待用户拍板）→ **✅CUT 4（`0de34db83fb`，warehouse→fs.defaultFS 重新安置）** → CUT 5 REWIRE 停造 fe-core iceberg 簇（flip-gated 全 flavor e2e）→ CUT 6 删死连通性探测 → CUT 7 删属性簇。 -> -> **✅ CUT 1 `cf8dda9f058`（连接器自建 HMS-metastore Kerberos 鉴权器，PREP，最高风险/flip-gated）**：抽包内 static `IcebergConnector.buildPluginAuthenticator(props, storageHadoopConfig)`；storage-kerberos 分支逐字不变；新增 HMS 分支——flavor==hms 且 `HmsMetaStoreProperties.kerberos()` present-with-creds 时用 HMS client principal/keytab 建 `KerberosAuthenticationConfig`（逐字镜像 fe-core `HMSBaseProperties.initHadoopAuthenticator:176-180`）。补 `IcebergConnectorPluginAuthenticatorTest` 5 例。**验收**：fe-connector-iceberg BUILD SUCCESS + checkstyle 0 + 5/5 绿 + mutation 击杀。**⚠️ flip-gated 未跑**：Kerberized-HMS-on-simple-storage e2e（本地无集群，登记 ENG-3）；CUT 1 使该窄场景从 fe-core-delegate doAs 立即改走 plugin-UGI doAs（同 principal/keytab、且 plugin-UGI 才是 plugin FileSystem 正确副本 → 更正确，但须 e2e 证）。 -> -> **✅ CUT 2 `eb9201dc0a6`（SHOW CREATE 脱敏改绑，PREP，纯本地可验）**：`DatasourcePrintableMap.SENSITIVE_KEY` 删 `getSensitiveKeys(IcebergRestProperties.class)` 反射 + import → 显式 add 4 键（byte-identical）。**recon 深挖纠错**：与 `S3Properties` 重叠不均——`iceberg.rest.secret-access-key` 冗余于 S3 敏感 secret-key，但 `iceberg.rest.session-token` 是 S3 **非 sensitive** 字段别名 → 只由本处保护，省略即静默泄漏 → 四键全显式。**验收**：fe-core BUILD SUCCESS + checkstyle 0 + `DatasourcePrintableMapTest` 18/18 + mutation 击杀（oauth2.token/session-token 丢即转红）。 -> -> **✅ CUT 4 `0de34db83fb`（warehouse→fs.defaultFS 桥重新安置，PREP，纯本地可验）**：`CatalogProperty` 加包内 static `deriveHdfsDefaultFsFromWarehouse`（中立 warehouse 键 + hdfs scheme，守铁律），`initStorageProperties` 统一 derived 源（msp!=null 用 msp、msp==null 用中立 helper）。**recon 后弃"通用化对所有 engine 生效"**（会误改 paimon fs + core-site.xml fs.defaultFS 冲突场景）→ 用 msp==null 分支天然只覆盖 iceberg，paimon（有 msp）不受影响、行为保持。补 `CatalogPropertyDeriveHdfsWarehouseTest` 5 例；BUILD SUCCESS + checkstyle 0 + 5/5 + mutation 2/2 击杀。 -> -> **⏭ 续跑起步任务 = CUT 3（vended gate，但须先找用户拍板一个设计难点）→ 然后 CUT 5 REWIRE（翻掉 iceberg msp）→ CUT 6/7 删除**。**⚠️ CUT 3 设计难点（CUT 4 的 msp==null 中立 helper 模式不能直接套用）**：paimon 靠 `msp.isVendedCredentialsEnabled()` 判 vended **是因保留 msp**；iceberg CUT 5 后 msp==null → 须让 vended 判定在 msp==null 下工作，但 vended 要读 iceberg 专属键 `iceberg.rest.vended-credentials-enabled`（引擎名串=铁律，不同于 CUT 4 的中立 warehouse 键）。**动 CUT 3/CUT 5 前按 `ask-user-explain-in-chinese-first` 找用户拍板**（候选见设计文档 §3 CUT 3）。 -> -> **⚠️ 全部未 push**（[DEC-FLIP-1] 铁律）。recon 结论持久化在 `wf_73999dcf-412` journal + 已固化进设计文档。 - ---- - -# 🎯 上一轮（2026-07-05）= **iceberg 4 死实体类删除全完成（第 4、5、6 刀 done）= P4 DONE** - -> **本轮范围** = 承接上一轮（cut 1-3 done），完成删原生 iceberg 4 死实体类（`IcebergExternalTable`/`IcebergExternalDatabase`/`IcebergSysExternalTable`/`IcebergExternalCatalog` base）的**第 4、5 刀前置 + 用户 sign-off 后第 6 刀原子删**。**至此整条 P4（删 4 实体类簇）= DONE**。**下一 = P5 属性/鉴权迁移到连接器（用户 2026-07-05 已裁定为下个 session 起步任务）**，权威刀序见执行计划 §P5 + 文末 🚀 段。 -> -> **✅ 本轮完成 2 刀（均独立 commit + BUILD SUCCESS + checkstyle 0 + 测试全绿，未 push）**： -> - **第 4 刀 `ac0cef5b9a4`（修 buildDbForInit 回放）**：`ExternalCatalog.buildDbForInit case ICEBERG` 由构造 `IcebergExternalDatabase` 改为 `PluginDrivenExternalDatabase`（对齐翻闸后运行时类型 + GSON remap；JDBC/TRINO/PLUGIN 三同胞 case 早已同签名构造），删去现已无用的 IcebergExternalDatabase import。**保留 case 标签**（删则 fall-through `default→return null` 在旧 InitCatalogLog(Type.ICEBERG) 回放崩 db init）；Type.ICEBERG 枚举保留供老镜像反序列化。验收：IcebergGsonCompatReplayTest 3/3。 -> - **第 5 刀 `96020c70e99`（迁测试脱死实体到活类型）**：跑 10-agent 对抗核验分类工作流（每 verdict 逐一驳斥核验、防悄悄丢活覆盖 Rule 9/12），把仍测**活逻辑**、当前经死实体类作载体的单测改挂活类型：**6 mock 改挂**（DbsProcDirTest/UserAuthenticationTest/StatementContextTest/StatisticsUtilTest/IcebergMetadataOpTest/IcebergUtilsTest → HMS*/PluginDriven*/mock(ExternalCatalog)）+ **IcebergSysTableResolverTest trim** 到活断言（保留测 SUPPORTED_SYS_TABLES 排除 position_deletes、删死路径 position_deletes 报错用例）+ **迁 IcebergUtils 分区助手两测**（testGetPartitionRange/testSortRange，fe-core 唯一覆盖 getPartitionRange/sortPartitionMap/mergeOverlapPartitions）到新文件 `IcebergUtilsPartitionRangeTest`。**顺带修好一处此前空跑的测试**（UserAuthenticationTest：旧 iceberg mock 不满足生产已收敛的 `instanceof PluginDrivenSysExternalTable`→换活类型后 getSourceTable 委派 + checkTblPriv 断言真正生效）。验收：9 测试类 69 测 0 失败。 -> -> **⚠️ Rule 7 更正（cut 5 实证，纠正旧 HANDOFF/执行计划）**：旧 HANDOFF 说 `IcebergExternalTableBranchAndTagTest` 应"改挂 IcebergMetadataOps 活路径 + port getPartitionRange 覆盖"是**双重误判**——① 该测**零** getPartitionRange 引用（getPartitionRange 覆盖实在 `IcebergExternalTableTest`，已迁）；② 其测的 fe-core `IcebergMetadataOps` branch/tag 车道翻闸后**孤儿**（native 走连接器 `IcebergCatalogOps` 独立重实现、HMS 走 `HiveMetadataOps` 抛错，仅死的 `IcebergExternalCatalog:123` 接 dispatching metadataOps），连接器 `CatalogBackedIcebergCatalogOpsDdlTest`/`IcebergConnectorMetadataDdlTest` 已等价覆盖 → 该测随 cut 6 删、**不迁移**、零活覆盖损失。 -> -> **✅ cut-6 前置实证 done（grep 全仓 code-vs-comment 分类）= cut 6 确认为干净原子删**：删 4 类后**零 ALIVE 代码引用会断编译**——main-src 仅剩 `GsonUtils.java` 3 处**字符串标签**（`registerCompatibleSubtype(PluginDriven*.class, "IcebergExternal*")` 老镜像升级 remap，字符串非类引用，删后照编）+ 各 ALIVE 文件过时**注释**（cosmetic）。`IcebergUtils`/`IcebergMetadataOps`/`source/`/cache/ **零**死类引用（HANDOFF 担心的"以死类为参/字段类型的活方法"早被 cut 1 搬常量 + cut 3 删 showCreateView 重载清掉）→**无须改任何 ALIVE 签名**。test-src 真实代码引用仅剩 3 文件随删。 -> -> **✅ 第 6 刀 `1ca3617a51a`（原子删 7 文件，-1822 行，用户 2026-07-05 sign-off 后执行）**： -> - 4 实体类 `datasource/iceberg/{IcebergExternalTable,IcebergExternalDatabase,IcebergSysExternalTable,IcebergExternalCatalog}.java` -> - 测试夹具 `test/.../iceberg/TestIcebergExternalCatalog.java` -> - 2 纯死路径测试 `test/.../iceberg/{IcebergExternalTableTest,IcebergExternalTableBranchAndTagTest}.java` -> - **留**：`GsonUtils` 3 字符串标签（老镜像升级 remap）、`IcebergGsonCompatReplayTest`（纯字符串标签，证升级路径）。 -> - **验收（Rule 12 实测）**：fe-core main+test BUILD SUCCESS；12 smoke 测试类 89 测 0 失败（含 **IcebergGsonCompatReplayTest 3/3** = 证删老类后老 iceberg 镜像仍正确反序列化为 PluginDriven，升级兼容成立 + IcebergUtilsPartitionRangeTest 2/0 = 迁移覆盖存活）；checkstyle 0；连接器零 import 死类（连接器对旧类名引用全为 parity 注释）。 -> -> **📌 遗留（登记，非本轮 blocker）**： -> - **cosmetic**：ALIVE 文件（MaterializeProbeVisitor/LogicalFileScan/ShowCreateTableCommand/… + 连接器 parity 注释）里仍有提及已删旧类名的**过时注释**，不影响编译，留后续清理。 -> - **ENG-1**：cut 3 从 `MaterializeProbeVisitor.SUPPORT_RELATION_TYPES` 删 `IcebergExternalTable.class` 后未加 `PluginDrivenExternalTable.class`（潜在 MTMV 物化孪生缺口）+ `RefreshManager` 同型 → 值得 ENG-1 核（删死码不改运行时行为）。 -> -> **⏭ 下个 session 起步任务（用户 2026-07-05 已裁定）= P5 = 属性/鉴权迁移到连接器 + 属性簇删除**（权威 = 执行计划 **§P5**）。**删死码工作的最后一大块、也是最重的——功能增项 + 改接线，非纯删除**。起步要点（起步先读 §P5 + 对照真实代码 recon，行号可能过时、信控制流不信注释）： -> - **为何属性簇还活**：翻闸后 plugin iceberg catalog 仍经 fe-core 属性簇建鉴权/凭据——`PluginDrivenExternalCatalog.initPreExecutionAuthenticator → catalogProperty.getMetastoreProperties()(Type.ICEBERG → IcebergPropertiesFactory)` 建 kerberos 鉴权器；`CatalogProperty.initStorageProperties → VendedCredentialsFactory case ICEBERG + msp.getDerivedStorageProperties()` 建凭据/derived-storage。 -> - **为何可自足删（不受 HMS-iceberg 阻塞）**：HMS-iceberg 走 `type=hms → HivePropertiesFactory`，**从不解析 Type.ICEBERG** → 属性簇只被 plugin 路径钉 → rewire 掉 plugin 路径即可删。 -> - **rewire 是真活且非平凡**：连接器 `fe-connector-metastore-iceberg` 承接类已存在（IcebergHms/Glue/Dlf/Rest/Jdbc/NoOp MetaStoreProperties+Provider），**但它们尚不自建 authenticator（现依赖 fe-core context authenticator）** → **须先给连接器 metastore provider 补 authenticator 构建（镜像 paimon 已有做法），再把 PluginDrivenExternalCatalog 的鉴权/凭据/derived-storage 接线改走连接器 metastore-spi**，rewire 完才能删属性簇（清单见 §P5：`property/metastore/{AbstractIcebergProperties,IcebergPropertiesFactory,Iceberg*MetaStoreProperties,IcebergRestProperties}` + `property/common/IcebergAws*Properties` + `MetastoreProperties` enum/register + `VendedCredentialsFactory case ICEBERG` + 连通性 testers F2/F3/F15/F16 死臂）。 -> - **正交备选（非本次）**：翻闸主线（连通性 F2/F3/F15/F16 → ENG-3 flip-gated e2e → 二签）——P5 与翻闸正交，用户已定先做 P5。 -> -> **⚠️ 全部未 push**（[DEC-FLIP-1] 铁律）。cut-5 分类/执行工作流结论持久化在 `wf_4e9d5818-e50`（classify）/`wf_6186b19e-815`（apply）journal + 已固化进执行计划 §P4。 - ---- - -# 🎯 上一轮（2026-07-04 续）= **fe-core iceberg 死码删除 P3（catalog flavor 簇 + factory，2 独立 commit）** - -> **做了什么**（承接「删死码优先」裁定，按执行计划 `designs/fe-core-iceberg-removal-execution-plan.md` §P3 刀序；先 5-agent 逐 test-file 核验 fixture 处置 + 末尾 3-lens 对抗核验删除安全）：一刀删掉翻闸后不再实例化的 iceberg catalog flavor 全簇（原生 iceberg catalog 现建成 `PluginDrivenExternalCatalog`）。 -> - **前置 `c051ff2c3d2`**：削 `IcebergMetadataOps` 两处死 `instanceof IcebergRestExternalCatalog` 臂（`listNestedNamespaces` REST 嵌套 namespace 递归臂 + `isViewCatalogEnabled` REST view-enabled 臂）+ 净化无用 import。**死判定实证**：`IcebergMetadataOps` 仅两处活构造边——`IcebergExternalCatalog:127`（翻闸后 base 不实例化=死）与 `HMSExternalCatalog:246`（活；`dorisCatalog` 恒 `HMSExternalCatalog`，从不是 Rest flavor）→两臂恒 false，删臂对活 HMS 路径行为逐字不变。REST 嵌套/view-enabled 的**活路径已由连接器 `IcebergCatalogOps` 承接并测**（`CatalogBackedIcebergCatalogOpsTest` 注释明写镜像 legacy gate）→无能力孪生缺口。 -> - **删除 `a91e6b0a641`**（−344 行）：删 `IcebergExternalCatalogFactory`（零 caller，`CatalogFactory` 内置 case 已在 GSON 切换删、仅留注释）+ **7 flavor**（`Iceberg{Rest,HMS,Glue,Hadoop,Jdbc,S3Tables,DLF}ExternalCatalog`）。GSON 旧类名标签走 `registerCompatibleSubtype` 字符串 remap（非类引用）。base `IcebergExternalCatalog` **留**（阶段四；HMS-iceberg 仍活）。测试 fixture 迁移：新增 test-only 具体子类 `TestIcebergExternalCatalog`（镜像已删 flavor 空构造器）供需真实 `catalogProperty` 的用例（`IcebergUtilsTest`×3、`IcebergExternalTableBranchAndTagTest` spy）；仅占位/mock 的 3 用例改 `Mockito.mock(IcebergExternalCatalog.class)`；`IcebergUnityCatalogRestCatalogTest` 删 @Disabled 的 `testCreateRestCatalog`（测翻闸后已死的原生 REST getAllDbs 端到端路径）。 -> **验收（Rule 12 实测）**：全仓零活引用（仅剩历史/parity 注释）；fe-core BUILD SUCCESS + checkstyle 0 + import-gate 净；8 测试类全绿（IcebergUtilsTest 16 / StatisticsUtilTest 9 / IcebergMetadataOpTest 7 / ExternalMetaCacheRouteResolverTest 7 / DbsProcDirTest 6 / IcebergExternalTableBranchAndTagTest 3 / **IcebergGsonCompatReplayTest 3=证删 flavor 后 GSON 旧类名标签 remap 完整、老集群升级不受影响** / IcebergMetadataOpsValidationTest 13）。 -> **实证纠正执行计划（Rule 7）**：计划原写「6 flavor」实为 **7**（含 Rest）；`IcebergDLFExternalCatalog` 不引用 P2 已删 DLF 子树，不受 P2→P3 顺序阻塞（已回填执行计划 §P3）。 -> **⚠️ 全部未 push**（[DEC-FLIP-1] 铁律：删死码/翻闸做最后原子提交再 push）。 -> **⏭ 下一 = P4 实体类 + base catalog**（见下文 🚀 段 + 执行计划 §P4）：删 `IcebergExternalTable`/`IcebergExternalDatabase`/`IcebergSysExternalTable`/`IcebergExternalCatalog`(base)。**前置重**：① 搬 `IcebergExternalCatalog` 常量到 IcebergUtils/新常量类（被活文件读）；② 修 `ExternalCatalog.buildDbForInit case ICEBERG` 回放为 `PluginDrivenExternalDatabase`；③ 削 ~15 处翻闸后恒 false 的 `instanceof IcebergExternalTable/IcebergSysExternalTable` 死臂（含 F4 `redirectSysTableToSource` helper）。**⚠️ P3 迁移保命但 P4-doomed 的测试**（`IcebergExternalTableBranchAndTagTest` 随 IcebergExternalTable 死）：到时若仍要验 IcebergMetadataOps branch/tag 语义，须改挂 HMS/table 活路径而非丢失。 - ---- - -# 🎯 上一轮（2026-07-04）= **ENG-1 除连通性外剩余 5 条缺口批量修完（F4/F13·F9/F10/F12·F11·F6/F7·F14；直接动码不 recon + 末尾统一对抗 review；5 独立 commit）** - -> **做了什么**（用户 2026-07-04 裁定协议：照审计结论直接动码、不逐条 recon/写单独 design、末尾统一 review）：ENG-1 审计 16 条确认缺口里，除已修 F1、已接受 F8、留后续的连通性 F2/F3/F15/F16 外的 **5 条 low** 一次性修完。信源=报告 §三 + 任务清单 §5b（逐条已回填 DONE + 修法 + commit）+ 完成记录 `designs/ENG1-batch2-remaining-gaps-summary.md`。 -> **5 条 commit**：`cd7618ef53e`(F4/F13 SHOW CREATE sys 表 redirectSysTableToSource) · `c8b39f871e3`(F9/F10/F12 iceberg getTableComment) · `50e4a6bcb5d`(F6/F7 EXPLAIN nested columns 通用重发) · `50ad635d9b0`(F14 AWS 非 DEFAULT PROVIDER_CHAIN carrier，新 `AwsCredentialsProviderModes`) · `bc5c39157aa`(F11 元数据预热改 `SUPPORTS_METADATA_PRELOAD` 能力位)。**铁律全守**（无 fe-core if(iceberg)/instanceof/引擎名新 seam；连接器禁 import fe-core；F11/F14 走中立能力位/连接器自包含 twin）。 -> **验收（Rule 12 实测）**：8 测试类全绿 + 广义 fe-core 回归 19 类 0 fail；**mutation 7/7 KILLED**（每缺口各对应测试转红）；checkstyle 0（api/fe-core/iceberg/jdbc）；import-gate 净。**统一对抗 review**（5 维度×驳斥，`.claude/wf-eng1-batch2-review.js`）抓出 **F14 真 bug**（`S3_MODE_KEYS` 漏 `iceberg.rest.credentials_provider_type` 别名 + providerFor 4 模式漏测）已修+re-verify 绿；1 nit（F4 死臂无测）驳回。 -> **两个有意偏离（Rule 7 记录，非缺陷）**：**F10** 保留共享 twin 单引号转义（消费者单引号包裹→合法 SQL；legacy `SqlUtils.escapeQuota` 双引号含 `'` 会坏 SQL；无引号 comment 字节相同→选正确性+外科不动共享码）；**F6/F7** iceberg field-id 编号注解 `col(3).sub(5)` 不复刻（cosmetic、FU-h10-deadcode、SlotDescriptor 不能越界给连接器；BE 仍收编号形，查询无影响）。**F14 STS base 凭证**仍走默认链（assume-role 已孪生，非 F14 焦点）。 -> **e2e flip-gated 全未跑**（本轮无集群）：F14 credential、F6/F7 EXPLAIN、F9 SHOW CREATE/information_schema、F4 SHOW CREATE、F11 preload 均翻闸后 e2e → 登记 **ENG-3**。 ---- - -# 🚀 下个 session 的任务（用户 2026-07-05 裁定）= **P5 属性/鉴权迁移到连接器 metastore-spi**（死码删除 P0–P4 均已 DONE，见顶部 🎯 段；本段属性/鉴权 sub-task 是删死码工作的收官大块） - -> **用户裁定**：翻闸主线（连通性 F2/F3/F15/F16 → ENG-3 e2e → 二签，见文末）**往后放**；先做**删除代码**——正交于翻闸，可独立推进。权威设计 = `plan-doc/fe-core-iceberg-removal-plan.md` v2（§阶段一死码 + §4 属性簇 + §6c 鉴权 + §8 用户已裁定 Q1+Q2=A/Q3=B）。**两件事**： -> 1. **删除所有已可删的死码**（阶段一余量）； -> 2. **属性/鉴权迁移**——fe-core iceberg 属性簇改走连接器已有的承接模块。 -> -> **⚠️ 起步先读 recon 结论（2026-07-04 本轮实证，纠正 v2 部分过时判断）**： -> - **去 SDK 化七刀已删掉一大批阶段一死码**：`rewrite/`、`action/`、四个 DML 执行器（Delete/Merge/Insert/Rewrite Executor）、`LogicalIcebergTableSink`、`PhysicalIcebergTableSink`、`IcebergTransaction`、`IcebergTransactionManager`、`IcebergApiSource`、`IcebergDmlCommandUtils`、helper 的两个 RewritableDeletePlan* 等**均已 GONE**（见 `af7e244c3fe`/`64b03892b20`/`bf326c04741`/`4e7220d81c7`）。**残留死执行器仅 `LogicalIcebergMergeSink` 一个还在**（LogicalIcebergTableSink 已删但 MergeSink 漏删=待清）。 -> - **逐文件三态分类 + 可执行刀序 = `plan-doc/tasks/designs/fe-core-iceberg-removal-execution-plan.md`**（本轮 5-agent 分类工作流 `wf_7f1358fa-35d` 逐调用方核验产出，20 DELETE_NOW / 18 NEEDS_PREP / 23 ALIVE_HMS）。**下个 session 起步先读它的刀序（P0-P5 + 阶段四）再动手。** 刀序：**✅P0=broker/+helper（`b52703dc1b5`）→ ✅P1=fileio/（`6a169f1dd98`）→ ✅P2=DLF 子树+HiveCompatibleCatalog（`b29e9ffcbde`）→ ✅P3=catalog flavor 簇+factory（前置臂 `c051ff2c3d2` + 删除 `a91e6b0a641`）已删** → **✅P4 实体类 + base catalog（6 刀全完成，搬常量 `e6024ea632d` + 清 catalog-type 臂 `816585ef2ab` + 清 table-type 臂 `4b6381b6964` + 修 buildDbForInit 回放 `ac0cef5b9a4` + 迁测试 `96020c70e99` + 原子删 4 类 `1ca3617a51a`）** → **P5 属性/鉴权迁移到连接器 ◀ 下一（用户 2026-07-05 裁定起步）**。**ALIVE_HMS 23 文件禁删**（`IcebergUtils`/`IcebergMetadataOps`/`source/`/cache/ 等，HMS-iceberg 经 `PhysicalPlanTranslator:825` DlaType.ICEBERG 活）——挂 hive 迁 SPI（阶段四）。 -> - **属性/鉴权（sub-task 2 = P5，最大一块）**：连接器 `fe-connector-metastore-iceberg` 承接类已存在（IcebergHms/Glue/Dlf/Rest/Jdbc/NoOp MetaStoreProperties+Provider），**但 rewire 是真活且非平凡**：核验确认翻闸后 plugin iceberg 仍经 `PluginDrivenExternalCatalog.initPreExecutionAuthenticator:147 → getMetastoreProperties()(Type.ICEBERG)` 建 kerberos 鉴权器 + `CatalogProperty.initStorageProperties:181 → VendedCredentialsFactory case ICEBERG + getDerivedStorageProperties`。**关键：HMS-iceberg 走 type=hms 从不吃 Type.ICEBERG → 属性簇只被 plugin 路径钉 → rewire 掉 plugin 路径即可删（自足，不受 HMS-iceberg 阻塞）**。但**连接器 metastore provider 尚不自建 authenticator（现依赖 fe-core context authenticator）→ 须先给它补 authenticator 构建（镜像 paimon），再改接线**——这是个功能增项+rewire，非纯删除，详见执行计划 §P5。 -> -> **执行纪律（复用本仓铁律）**：删除/动码前必 grep 全调用方 + 区分 DEAD vs STILL-CONSUMED（信控制流不信注释）；每刀独立 commit + build + test + checkstyle 0 + import-gate 净；**NEEDS_PREP 文件先做前置（搬常量 / 修 buildDbForInit 回放 / 削死臂 / 改 p2 测试）再删**；用户已裁定 Q1+Q2=A（fileio/ 并入删、接受 io-impl 极端配置失效、同步改 p2 回归）、Q3=B（HMS-iceberg 随 hive 迁移，不建重定向接缝）。**iceberg-core 依赖 + HMS-iceberg 活代码是阶段四、挂 hive 迁移，短期删不掉。** ---- - -# 🎯 翻闸主线（往后放，但仍需完成才能翻闸二签）= 连通性 4 条 → ENG-3 e2e → 二签 - -> ENG-1 除连通性外已全清（F1 + 本批 5 条 low）。剩 **连通性 F2/F3/F15/F16（medium，opt-in `test_connection`，默认 false）**：hms/glue-flavor iceberg 的 `test_connection` 元存储探测静默 no-op（`IcebergConnector.testConnection` `TYPE_REST`-only）→ 修法=连接器侧为 hms/glue flavor 补 HMS thrift/Glue GetDatabases 探测（信源报告 §三 F2/F3）。之后 **ENG-3 flip-gated e2e 全跑 → 用户二签 → 翻闸最后原子提交（含 GSON 迁移 `e68eb5c00c9`）**。**全部翻闸/review/删死码工作均未 push**（[DEC-FLIP-1]：翻闸做成最后原子提交再 push）。 ---- - -# 🎯 上一轮（2026-07-05）= **行级 DML 去 SDK 化七步全部完成（设计 Status=DONE，7 个独立 commit）** - -> **七步 commit 全谱**：`af7e244c3fe`(1/7 rewrite/action 死车道) + `64b03892b20`(2/7 DML 死臂闭包) + `bf326c04741`(3/7 INSERT 死车道并入) + `4e7220d81c7`(4/7 四小类搬中立包) + `255bcaf52a2`(5/7 checkstyle 门禁) + `e5972dfc8a2`(6a/7 rewrite re-derive 补 doAs) + `890b8698e6f`(6b/7 DML 预执行窗口补回滚) + 本文档轮 commit（7/7）。前三刀 -11,000+ 行；设计=`plan-doc/tasks/designs/iceberg-rowlevel-dml-desdk-design.md`（Status=DONE，完成记录在其头部）；removal-plan §6b 已补落地记录。 -> **本轮（4-7 步）要点**： -> - **4/7 搬包**：IcebergMergeOperation→`nereids.trees.plans.commands.merge.MergeOperation`（改名）、IcebergNereidsUtils 存活半→`commands.RowLevelDmlRowIdUtils`（改名）、IcebergRowId/IcebergMetadataColumn 保名进 `commands`；实际涟漪比设计多 3 个测试文件 + IcebergExternalTable 补 import；IcebergHiddenColumnTest/IcebergMetadataColumnTest 随类搬包时被 nereids 测试包 ImportControl（禁 JUnit4）强制 **JUnit4→5 机械转换**（断言逐条不变）。 -> - **5/7 门禁**：nereids（含 fe-sql-parser）/planner 两 subpackage 禁 `org.apache.iceberg`；**mutation 击杀验证**（注入 SDK import → checkstyle FAILURE → 回滚双绿）。**豁免清单 = 19 处 datasource.iceberg import/14 文件 + 3 测试**（明细在 commit message）：legacy 豁免臂 + 翻闸后死 instanceof 臂 + **活 IcebergUtils 引用**（isIcebergRowLineageColumn v3 行谱系 ×3 + 常量读 ×2——设计"活 import 归零"的说法过于乐观，此面登记待后续中立化，非本刀回归）。 -> - **6a/7**（连接器侧）：registerRewriteSourceFiles 的 pinned-snapshot planFiles re-derive 包 `context.executeAuthenticated`（镜像 commit():438）；新增 2 接线 UT（authCount 递增 + failAuth 证 seam 在 authenticator 内）+ mutation 击杀。 -> - **6b/7**（fe-core）：begin→finalize 窗口抽 `beginTransactionAndFinalizeSink` 包 catch(Throwable)→onFail（镜像 InsertIntoTableCommand:372-388）；`BaseExternalTableInsertExecutor.onFail` protected→public（调用方在父包）；新 RowLevelDmlCommandTest 3 case + mutation 击杀。 -> **整刀验收（Rule 12 口径）**：nereids/planner 零 `org.apache.iceberg` import 且 checkstyle 上锁；13 个 gate 套件 137 测 0 失败（PhysicalPlanTranslatorIcebergRowLevelDml/PhysicalIcebergMergeSink/AdmissionGate/PluginDrivenTableSink/PluginDrivenExternalTable/WriteConstraintExtractor/两 Converter/IcebergDeletePlan/GsonCompatReplay/RowLevelDmlCommand/RowLevelDmlRowIdUtils/IcebergRowLevelDmlTransform）；**docker e2e（dml 4 套件 + action/ 8 套件）flip-gated 未跑**（死码删除理论零行为差；6a kerberos e2e 也 flip-gated——rewrite 车道翻闸后才活）。 -> **⏭ 下一 session（按用户 07-01 既定顺序回翻闸主线）**：**ENG-1 能力孪生审计**（任务清单 §5 + review 报告 §七；见下文 🎯 段）。备选并行项：① 有集群时补跑 flip-gated 回归（meta-cache 两回归 + 类加载冒烟 + `iceberg_branch_complex_queries`/`partition_operations`）；② 盯 PR #64689 CI 的 `test_iceberg_hadoop_catalog_kerberos`（第四刀实证）；③ removal-plan 阶段一余量（fileio/、broker/ 等孤岛死码刀，正交可独立做）。 - ---- +> **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`，工作树干净）。P6 期间所有"未 push（[DEC-FLIP-1] 铁律）"的翻闸/删死码/属性迁移工作**已全部随 #64688 一次性合入**——**该铁律现已解除**，P7 走常规「连接器内实现 → 翻闸 → 删 legacy → squash-合入」流程（复用 P5/P6 样板）。 -# 🎯 上一轮（2026-07-04 深夜）= **行级 DML 去 SDK 化设计完成并四项裁定（APPROVED）：实证改判为死车道删除，设计 = `plan-doc/tasks/designs/iceberg-rowlevel-dml-desdk-design.md`** +## #64688 做了什么（685 文件，+79738/−23744） -> **做了什么**：11-agent 研究工作流（4 路清点：fe-core DML 车道 82+ 引用点 / 连接器 SPI 面 / Trino master merge 模型（@280b81bbc4e）/ BE 契约 → 完备性批评家 → 6 路补盲：删除闭包双证明、HMS-DLA 共享面、测试面全图、鉴权包装逐 crossing、失败/KILL 生命周期、e2e pin 面）。产出设计文档并经用户四项裁定为 **APPROVED**。 -> **核心改判（已同步进移除计划 §6b/§8-Q6）**：原定四项中,操作码与行标识**已中立**（IcebergMergeOperation 纯常量、rowid 列经 getSyntheticWriteColumns 声明+中立注入,与 Trino 逐点同构,无需新 SPI）;表达式下沉与暂存句柄化的**目标是翻闸后死码**（转换函数仅剩死调用方,连接器自带 IcebergPredicateConverter 三模式;SDK 暂存两端皆死,中立替身 rewriteSourceFilePaths 已上线）→ 改判**死车道删除**。死码双证明：实例源 3 处全不可达 + GSON 读侧重映射/写侧只写 PluginDriven（保留死码零回滚价值）。 -> **用户四项裁定**：①接受改判为删除;②原生 INSERT 死车道并入同刀（IcebergTransaction 被两组死执行器共同钉住）;③4 个 SDK-free 小类现在搬 nereids 中立包（MergeOperation 改中立名,IcebergRowId/IcebergMetadataColumn 保名搬包）;④两个顺带活问题随本轮独立 commit 修：**rewrite 提交前 registerRewriteSourceFiles 的 planFiles 在 kerberos 裸奔**（镜像 commit():438 包 executeAuthenticated）+ **RowLevelDmlCommand.run 预执行窗口无回滚**（镜像 InsertIntoTableCommand:372-388 catch→onFail）。 -> **动码安全边界（设计 §4-§6 已固化,此处仅提醒）**：IcebergScanNode 是 **HMS-DLA 活类禁整删**（仅成员级手术）;测试面 12 直删+3 case 手术+**2 个 KEEP 测试须先搬出待删目录**（IcebergDeletePlanTest + isRowIdInjectionTarget 半）;错误消息/结果形状/隐藏列名 pin 面全在连接器侧,删 fe-core 死副本不影响;勿动 StatementContext 的 rewriteSourceFilePaths/rewriteSharedTransaction（活替身）。 -> **⏭ 该轮的"按设计 §8 TODO 动码"前三步已于 2026-07-04/05 完成（见顶部最新一轮），剩 4-7 步。**研究工作流归档在 session scratchpad `ice-dml/*.json`（易失）,关键结论已全部固化进设计文档。 +把原生 iceberg（元数据/scan/write/procedures/sys-tables/行级 DML）整体迁到自包含 `fe-connector-iceberg` + 翻闸（iceberg 入 `CatalogFactory.SPI_READY_TYPES`）+ 删 fe-core 原生 iceberg 子系统： +- **删除**：`IcebergExternalCatalog`/`IcebergExternalTable`/`IcebergTransaction`/`IcebergNereidsUtils`、7 catalog flavor + factory、`broker/`·`dlf/`·`fileio/`·`action/`·`rewrite/`·`helper/`、四个 DML 执行器、planner 三 sink、`IcebergTransactionManager` + fe-core Iceberg/Paimon 元存储属性簇（S7，−4914 行）等。 +- **翻闸 + GSON 迁移**：旧 8 catalog 变体 + db + table 标签 `registerCompatibleSubtype`→`PluginDriven*`（保升级老集群反序列化）。 +- **属性/鉴权全归插件**（S1–S10，用户 2026-07-05 架构裁定）：fe-core 不再解析任何属性；已迁连接器（iceberg+paimon）走「插件解析 → BE thrift → 回传 fe-core」。详见 memory `catalog-spi-no-property-parsing-in-fecore`。 ---- - -# 🎯 上一轮（2026-07-04 夜）= **fe-core iceberg 移除全量分析完成（只分析未动码）：`plan-doc/fe-core-iceberg-removal-plan.md` v2 实证重写，用户已裁定 Q1-Q5** +## ⚠️ 关键遗留（P7 必须接手）= fe-core `datasource/iceberg/` 还剩 **23 个 HMS-iceberg 支撑类** -> **做了什么**：39-agent 工作流（7 路清点 → 每个"可删"结论双镜头对抗反驳 + 12 条高风险死臂孪生抽查（12/12 COVERED）+ 8 存活集群移除路径设计（含 Trino 参照）+ 完备性复扫）。**v1 草稿记载失实已确认**（broker/fileio 并未删除）→ 计划文档全量重写为 v2。 -> **关键更正/新发现**：① 属性簇翻闸后仍活（`PluginDrivenExternalCatalog:147` initPreExecutionAuthenticator → MetastoreProperties Type.ICEBERG，每个插件 iceberg 目录都跑）；② `IcebergAws*Properties` 非死码（`IcebergRestProperties.addGlueRestCatalogProperties:345-361` 在 initNormalizeAndCheckProps 链上活，REST signing-name=glue|s3tables）；③ fileio/ 有配置注入反射活路（HMS-iceberg `io-impl` 透传 + p2 测试在用）；④ `ExternalCatalog.buildDbForInit:972` case ICEBERG 回放边缘（翻闸前 InitDatabaseLog 回放会构造原生 db）——删实体类前须改；⑤ IcebergExternalCatalog 常量被 IcebergUtils:1876/IcebergMetadataOps/IcebergScanNode:197 活读，删 flavor 前须搬常量；⑥ 原生 rewrite/ + action/ 是死孪生（插件走中立 ConnectorRewriteDriver）可删；⑦ v1 漏了整条目录外 sink/executor 死车道（LogicalIcebergTableSink/MergeSink、planner 三 sink、四 executor、IcebergTransactionManager+factory 方法）。 -> **依赖裁决**：iceberg-core 暂留（HMS-iceberg+DML 合成+vendored DeleteFileIndex 钉）；iceberg-aws/s3tables/s3-tables-catalog-for-iceberg 阶段二外科剥离后可摘；avro/s3-transfer-manager 保留（hudi/hadoop-aws 非 iceberg 消费者）。 -> **用户已裁定（2026-07-04，详见计划文档 §8）**：Q1+Q2=A（接受 io-impl 极端配置失效：fileio/ 并入阶段一删、iceberg-aws 阶段二照常摘）；Q3=B（HMS-iceberg 随 hive 整体迁移，不建重定向接缝）；Q4=A（行级 DML 去 SDK 化现在做，设计先行）；Q5=继续只分析。 -> **⏭ 该轮遗留的"下一步 = 行级 DML 去 SDK 化详细设计"已于 2026-07-04 深夜完成并改判（见顶部最新一轮）**。分析工作流归档（82 臂全清单等）在 session scratchpad `ice/*.json`（易失），关键结论已固化进计划文档。 +`#64688` 删的是**原生 iceberg 子系统**；但 **iceberg-on-HMS**（`type=hms` 下 `DlaType.ICEBERG` 的表）仍走 fe-core，因此以下 23 文件**故意保活**（decision D5 / Q3=B：HMS-iceberg 随 hive 一起迁）： +`IcebergUtils`、`IcebergMetadataOps`、`source/IcebergScanNode`(+`IcebergHMSSource`/`IcebergSource`/`IcebergSplit`/…)、`cache/`、`IcebergMvccSnapshot`、`IcebergSchema/Snapshot/Partition*CacheValue`、`profile/IcebergMetricsReporter`、`DorisTypeToIcebergType`、`IcebergCatalogConstants`。 +→ **P7 hive 迁移把 HMS-iceberg 挪到连接器路径后，这 23 文件才能删（阶段四）**。同理 fe-core `datasource/hudi/` 与 `datasource/hive/` 也在 P7 范围。 --- -# 🎯 上一轮（2026-07-04 续）= **第四刀已落地并推送：iceberg `d5541bbb384` + paimon parity `58dff8c2790`（已推 origin，待 CI 实证 kerberos case 转绿）** - -> **第四刀实现**（对应下文分析）： -> - **iceberg `d5541bbb384`**：数据路径 = 对象级 FileIO seam（`wrapTableForScan`：resolveTable 后把 loaded 表 ops 包成 -> `IcebergAuthenticatedTableOperations`+`IcebergAuthenticatedFileIO`，factory 时刻 doAs 捕获 secured FS → 任意线程 -> newStream 复用，worker 池/streaming lazy 消费全覆盖）；sys 路径 = 线程级 `executeAuthenticated` 包 -> `planSystemTableScan` 全程**单一 scope**（`resolveSysTable` 内层 wrap 拆除；刻意不用对象级包装——FileScanTask 要 -> Java 序列化给 BE JNI，wrapper 带 authenticator 不可序列化）。新 UT `IcebergScanPlanProviderKerberosScanIoTest` -> 5 用例；两处修复点 mutation 逐一击杀；全模块 923 测 0 失败、checkstyle 0。 -> - **paimon `58dff8c2790`**（parity，无 kerberos e2e 闸）：`planSplits` helper 包 `scan.plan().splits()`（sys 表同咽喉点）； -> 真表接线 UT + mutation 击杀；全模块 329 测 0 失败。 -> - **3-lens 对抗审查（bundled SDK 字节码级）= 0 confirmed**：唯一 high（$partitions 经 worker 池逃逸 doAs）被驳回—— -> iceberg 1.10.1 `ParallelIterable` 任务推进只在消费线程，`newInputFile` factory 全在规划线程 doAs 内 eager 捕获 -> kerberos DFSClient，worker 只用已捕获 FS 开流（与写路径 CI 实证机制一致）。 -> - **[FU-kerberos-scan-residual]（low/info，未修）**：① paimon `resolveScanTable` 的 `table.copy(scanOptions)` -> time-travel 快照/schema 读落在两 wrap 之间；② paimon `buildSchemaEvolutionParam` 的 SchemaManager 读在 scope 外 -> （两者靠 paimon 非 UGI 键控 FS cache 的 wrapped-load 先行温热缓解；反面=冷首触会污染 cache）；③ iceberg -> kerberos×REST-vended 互斥是配置性非结构性（畸形同配会丢 vended 凭据）；④ kerberos 下 MetricsReporter 退化为 -> default Logging（观测性）。 -> - **推送状态**：`25cd9d9f242..58dff8c2790` 已推 `origin/catalog-spi-10-iceberg`（PR #64689）。**⚠️ 下一步 = 盯 CI**： -> ① `test_iceberg_hadoop_catalog_kerberos` 转绿（第四刀唯一 e2e 实证）；② paimon 批量 case 应随 CI 环境恢复回绿 -> （若再挂，先对失败签名：rpc 超时/not alive = 环境延续，非代码）。 - ---- +# 🚀 下个 session 任务 = **P7 hive (+HMS) 迁移（最复杂、最后做的连接器）** -# 🎯 上一轮（2026-07-04）= **CI build 985672 失败分析：三刀实证生效；第四刀根因定位（已修，见上）** +> **权威计划**：[`00-connector-migration-master-plan.md` §3.8](./00-connector-migration-master-plan.md)（P7.1–P7.5 子阶段 + 风险）+ [`connectors/hive.md`](./connectors/hive.md)（子阶段/SPI 缺口/特殊性）。**单连接器 13 步 playbook = master plan §4。** +> **模块已就绪**：`fe-connector-hive` + `fe-connector-hms`（后者是 hive/hudi/iceberg-HMS/paimon-HMS 的**共享元存储库**，P3/P5/P6 已在用、稳定）。 -> **① kerberos case（真代码问题，本 PR 相关）**：INSERT 已过（三刀 `8d352049394` **CI 实证生效**）；失败点前移到 INSERT 后 -> `select count(1)` 的 **plan 期 manifest-list 读**——`PluginDrivenScanNode.getSplits → onPluginClassLoader(只 pin TCCL、无 doAs) -> → IcebergScanPlanProvider.planScan:299 → planScanInternal:498 → planCountPushdown:712 → SnapshotScan.planFiles -> → ManifestLists.read(snap-*.avro) → DFSClient.open → SASL "Client cannot authenticate via:[TOKEN, KERBEROS]"` -> （985672 归档 fe.log 01:22:54）。scan 路径从未包 executeAuthenticated——之前该 case 死在 INSERT、读路径从未被走到， -> 是**新暴露**非新引入。sys-table 路径（同文件 :1572/:1594 context.executeAuthenticated）已是既有范式； -> **master parity**：legacy `IcebergScanNode:477` `preExecutionAuthenticator.execute(() -> doGetSplits)`， -> streaming 臂 :510/:524 亦包。**修法（第四刀）**：`planScan` 两 overload + `streamingSplitEstimate` + `streamSplits` -> 包 `context.executeAuthenticated`；⚠️ `streamSplits` 返回的 `ConnectorSplitSource` 是 **lazy** 的——engine pump 线程 -> 消费 planFiles iterator，须在 split source 的逐批拉取处也包 doAs（镜像 master :510/:524），只包 streamSplits 入口不够。 -> UT 走 TcclPinningConnectorContextTest 同款接线断言（recording fake context）。paimon `PaimonScanPlanProvider.planScan` -> 同型缺口（无 kerberos e2e 闸）= parity 候选，随四刀一起或单独 commit。 -> -> **② paimon 8 case + hive_text_write 大面积失败 = CI 环境宿主资源饥饿，非本 PR（勿逐 case 修）**： -> BE 内嵌 JVM GC Real≫User+Sys（Young GC 2s→30s 递增；GC(276) Real=218s/Sys=48s = 重度内存 reclaim/CPU 争抢）、 -> BE JVM 连 KDC 都 SocketTimeout、brpc 8062 从 01:53 瘫（send fragments rpc timeout）、FE 02:03:26 心跳判 dead -> （BE 02:01:44 还在报 tablet=进程活着服务瘫）、02:26 流水线优雅重启 BE（be.out LSAN 泄漏报告=exit() 实证非 crash； -> dmesg 无 OOM-kill）。**跨 PR 同签名实锤**：64923/65175/64891/64924/64854/65031 六个别人的 PR 在**各自不同 agent** -> 上同窗口（07-03 晚 ~ 07-04 凌晨）挂同批 case（hive_text_write_insert + test_paimon_catalog/jdbc_catalog/... + -> mv external_table + load_p0 连带）；本 PR 984925（07-03 10:57）paimon 全绿。**与 `2a5a6aff2d3`（paimon parity doAs）、 -> `c2d9631511c`（paimon reader-type）无关**（纯时间巧合；后者的 FORMAT_ORC file-scanner-v2 warning 只是 v1 回落噪音， -> legacy 同行为）。处置：等环境恢复 retrigger / 向流水线维护者（onemorechance）反馈证据。 -> -> 分析证据/日志：985672 归档 tarball（GC log、be.out、fe.log、regression log 时间线）+ TeamCity REST 跨 build 对比。 +## P7 为什么是复杂度顶点（master plan §3.8） ---- +- **HMS 是共同元数据后端**：hive、hudi、iceberg-HMS-flavor、paimon-HMS-flavor 共用。`fe-connector-hms` 须持续稳定。 +- **21 个 metastore event 类 + `MetastoreEventsProcessor`** → 用 P0 已建的 `ConnectorMetaInvalidator` 承接，event pipeline 整体搬进 `fe-connector-hms`（决策 D4）。 +- **`HMSTransaction`（1866 行）+ `HiveTransactionMgr`** —— ACID 写路径，**最难**，须重写。**R-002（ACID 数据不一致，High）= 项目最大风险**；gate = 专门的 ACID 集成测试套件。 +- **`HMSExternalTable`（1293 行）** —— hive/hudi/iceberg 三种 `dlaType` 分流；由 DLA 模型吸收（决策 D5：`ConnectorTableSchema.tableFormatType` 区分）。 +- **31 处反向 `instanceof HMSExternalCatalog/HMSExternalTable`** 散落热路径（`nereids/glue/translator`、`tablefunction/MetadataGenerator`、`AnalyzeTableCommand`、`ShowPartitionsCommand`）。 +- **子阶段（见 connectors/hive.md §子阶段）**：P7.1 HiveMetadataOps 全功能搬迁（2 周）→ P7.2 event pipeline 搬 + `ConnectorMetaInvalidator`（1.5 周）→ P7.3 HMSTransaction+HiveTransactionMgr ACID 写路径（2 周）→ P7.4 DLA 分流改造（0.5 周）→ P7.5 删 fe-core `datasource/hive/` + 31 instanceof（0.5 周）。 +- **连带清理**：P7 收尾要一并删掉上面那 23 个 HMS-iceberg 支撑类 + fe-core `datasource/hudi/`；**hudi 批 E（live cutover）并入 P7**（PROGRESS §一 P3 note）。 +- **Kerberos UGI**：`fe-kerberos`（P3b #64655 已收口）+ `ConnectorContext.executeAuthenticated` 已支持，P7 逐条审查复用（R-004）。 -# 🎯 上一轮（2026-07-03 深夜）= **kerberos INSERT 三刀收口：temp() 补包装 `8d352049394`（CI 已实证 INSERT 过）** +## P7 起步要点（下个 session 开场） -> **背景**：`test_iceberg_hadoop_catalog_kerberos` INSERT 连挂多轮。一刀 DDL doAs（`a46e420b871`，已生效——CI 里 DDL 全过）； -> 二刀 FileIO 包装（`ba7d04fc8d8`）**上车后 CI（build 985573）原样再挂**——本轮根因分析（字节码级）： -> `BaseTransaction.TransactionTableOperations.io()` 从不读传入 ops 的 io()，而是 `tempOps.io()` -> （`tempOps = ops.temp(current)`，每次中间 commit 重建）；而 `IcebergAuthenticatedTableOperations.temp()` -> 原样转发 delegate.temp() → hadoop catalog 下 `HadoopTableOperations$1.io()` 直通裸 HadoopFileIO → -> worker 池 manifest 写从未穿过 doAs（985452 归档 fe.log 实证：`iceberg-worker-pool-8` 上 -> `DistributedFileSystem.create` SASL 拒）→ 二刀是 no-op。 -> **本轮修复** `8d352049394`：`temp()` 改为 `new IcebergAuthenticatedTableOperations(delegate.temp(m), io)`。 -> TDD 先红后绿（新 UT `IcebergAuthenticatedTableOperationsTest` 用 `Transactions.newTransaction(...).table().io()` -> 纯公开 API 复现绕过机制）；全套 918 测 0 失败、checkstyle 0。 -> **⚠️ 待办**：推分支后**盯 External Regression 的该 case 转绿**（本地无 kerberos docker，e2e 只能 CI 实证）。 -> **模式教训（第 2 次栽同型坑）**：iceberg 内部有自己的路由/线程池（第一次 worker-pool TCCL，这次 temp() 路由）—— -> "边界包一层"式修复必须沿 iceberg 内部调用链（bytecode/源码）走到真正的消费点验证，别只验证包装层自身行为。 +1. **先读**：master plan §3.8 + `connectors/hive.md` + `git show master:` 对照 fe-core `datasource/hive/` 真实代码 recon（**HANDOFF/计划的行号、依赖名、缺口清单可能过时；信控制流不信注释**）。 +2. **建 P7 阶段拆分 spec** `tasks/P7-hive-migration.md`（phase-level plan，镜像 `tasks/P6-iceberg-migration.md`）+ 起步 code-grounded recon（AGENT-PLAYBOOK §7）。 +3. **决策已定（master plan §5，勿重议）**：D4=event pipeline 放 `fe-connector-hms`；D5=DLA 用 `tableFormatType` 区分；写路径 SPI / procedure SPI / DLA 模型复用 P6 已建成果。 +4. **纪律**：每阶段独立 commit + build + test + checkstyle 0 + import-gate 净；**每一轮完成后即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。ACID 写路径必须有独立集成测试作 gate（R-002）。 --- -# 🎯 下一个 session 的任务 = **回主线（iceberg 翻闸：ENG-1 能力孪生审计）或补跑 meta-cache 翻闸回归** - -> **本轮（2026-07-01）已完成 = 三个手写连接器缓存全部上共享缓存框架（独立复制策略收官）**。 -> 设计+完成记录：**`plan-doc/tasks/designs/metacache-connector-port-design.md`**（Status: DONE）+ -> 任务清单 `plan-doc/tasks/metacache-connector-port-tasklist.md`。 -> -> **做了什么**：把 `IcebergLatestSnapshotCache` / `IcebergManifestCache` / `PaimonLatestSnapshotCache` 三个手写 -> `ConcurrentHashMap` 缓存改成 `connector.cache.MetaCacheEntry` 的**薄适配器**（对外方法签名/值类型不变→调用方零改动， -> 手写 CHM 机制退役）。5 个独立 commit： -> - `24e4c830aeb` 框架副本 Caffeine `3.2.3→2.9.3`（**关键侦察修正**：独立复制下框架随各插件**子加载器**打包、链接 -> **各插件自带**的 Caffeine；iceberg 带 2.9.3、paimon **原本无 Caffeine**→按 2.9.3 编译 + 给 paimon pom 补 2.9.3 依赖）； -> - `0be2679a7ac` iceberg 最新快照适配器；`bc27505eace` iceberg manifest 适配器；`47c4bcc6fd9` paimon 适配器(+Caffeine 依赖)； -> - `808c0cb0f0c` 对抗审查发现的过时"共享单一 Class 身份"注释更正（纯文档）。 -> - **适配器统一 flags**：`contextualOnly=true`、`manualMissLoadEnabled=true`（loader 在 Caffeine 锁外单飞，且使 -> `ttl<=0` 禁用路径为**确定性 bypass**，不靠 `maximumSize(0)` 异步淘汰）、`autoRefresh=false`、 -> `executor=ForkJoinPool.commonPool()`。**易错点已守**：`CacheSpec` 里 `-1`=永不过期，故适配器把连接器契约的 -> `ttl<=0`→翻译成 `ttl==0`(禁用)，专门加 `-1` 单测。 -> - **验证**：iceberg+paimon **整模块测试套全绿**（0 fail）；三模块 checkstyle 0；import gate 我的文件净； -> paimon 插件 zip 实测只含唯一可用 `caffeine-2.9.3.jar`（无版本冲突）。**3 视角对抗审查**（行为/框架 API/打包类 -> 加载）+ 对抗核验：仅 1 条确认（纯文档、已修），其余全驳回。 -> -> **⚠️ 仍 flip-gated 未跑（本轮无集群）**：`test_iceberg_table_meta_cache` / `test_paimon_table_meta_cache` 回归 + -> **重新部署后的类加载冒烟检查**（唯一能端到端证明"插件内 `MetaCacheEntry` 正确链接插件那份 Caffeine 子加载器"的 -> 手段；单测只证逻辑）。**下个有集群的 session 起步补跑**。 -> -> **⚠️ 大框架统一尚未收官**：本轮只迁完 iceberg/paimon 连接器；fe-core 老框架 `datasource.metacache` **仍原封不动** -> 服务 hive/hudi/doris/iceberg-on-HMS，**等所有连接器都迁完才能删 fe-core 那份**（远期）。 -> -> **下一步（用户定优先级）** = ①（有集群时）补跑上面两个 meta-cache 翻闸回归 + 类加载冒烟；或 ②**回 iceberg 翻闸主线： -> ENG-1 能力孪生审计**（见下文 🎯 段）。meta-cache 迁移与翻闸主线正交,可独立推进。 -> -> **⚠️ HMS import-gate 命中 = 误报,非违规（用户 2026-07-01 确认,非本任务）**：`fe-connector-hms/.../HiveMetaStoreClient.java` import `datasource.hive.HiveVersionUtil`（补丁版 HMS client）解析到的是 **fe-connector-hms 内 vendored 的同名自包含副本**（非 fe-core,该模块零 fe-core 依赖）→ **未破规则**；`check-connector-imports` 只是按包前缀 grep 误伤。**勿改连接器代码/重新暴露**。仅是 cache-clean reactor 构建/CI 的门禁噪音:`-Dexec.skip=true` 跳过 gate exec（`-pl ` 不带 -am 对叶子连接器不行——撞 `${revision}`）。详见 memory `catalog-spi-hms-hiveversionutil-gate-false-positive`。 - ---- +# 📦 分支 / Commit 须知 -# 🎯 （之后）下一个任务 = **flip-gated e2e 重跑（确认两 branch fix 绿）→ ENG-1 能力孪生审计** - -> **本 session（2026-07-01）已完成并各自独立 commit 两个产品 bug**（设计 + 完成记录见 `plan-doc/tasks/designs/iceberg-branch-mvcc-and-static-partition-overwrite-fixes.md` 末尾 Status）： -> - **① complex_queries = 通用 MVCC 快照塌缩 → `de1af7a594e`**：`StatementContext.snapshots` 改按 (ctl,db,table,**版本**) 键化（`MvccTableInfo` 加 version；`loadSnapshots` versionKeyOf 键化；新增版本感知 `getSnapshot(TableIf,ts,sp)` + 版本盲智能回退 default→lone→empty；`MvccUtil` 重载；`PluginDrivenScanNode.pinMvccSnapshot` 改版本感知）。UT 5/5 + mutation 2/2 KILLED + checkstyle。**共享核心，已过 clean-room 3-agent 对抗审（key 稳定/无读者回归/未能 break，无 blocker）**。 -> - **② partition_operations = 写路径丢静态分区字面量 → `98e00a14c37`**：新增中立能力位 `ConnectorCapability.SINK_MATERIALIZE_STATIC_PARTITION_VALUES`（iceberg 声明、MaxCompute 不声明）；`PluginDrivenExternalTable.materializeStaticPartitionValues()`；`BindSink.bindConnectorTableSink` full-schema 分支门控投影静态分区字面量（逐行镜像 legacy `bindIcebergTableSink:783-795`）。IcebergConnectorTest 断言 + mutation KILLED + checkstyle。镜像臂 = 焦点验证（非多 agent）。 -> - **⚠️ e2e flip-gated 未跑**（本 session 无 live 集群/iceberg-docker）：**下个 session 起步 redeploy 后重跑 `iceberg_branch_complex_queries` + `iceberg_branch_partition_operations` 确认绿**（`tag_retention` 仍是 spark 容器环境，非代码）。 -> - **follow-up 已登记**（设计文档末）：[FU-mvcc-mixed-schema]（同语句同表 schema 分歧→版本盲 base schema 取 main，pre-existing 单 schema 限制；SPI 分区裁剪恒列 latest）、[FU-connector-staticpart-validate]（通用 sink 缺 legacy 静态分区校验，应落连接器侧 fail-loud）。 -> -> **⚠️ 仍未 commit 的前序工作（勿丢、勿与新工作混提交）**：工作区 `IcebergConnector.java` 仍含**前序 session 的 worker 池 TCCL 修复**（`pinIcebergWorkerPoolToPluginClassLoader`+barrier，已 redeploy 实证）+ 新文件 `IcebergConnectorWorkerPoolPinTest.java`/`TcclPinningConnectorContext.java`+其测试 + `iceberg_branch_tag_edge_cases.groovy` 文案对齐。本 session commit 两 fix 时用 `git apply --cached` 单 hunk 隔离 `IcebergConnector.java`、**未触碰这些前序改动（仍 uncommitted）**。worker 池经验在 memory `catalog-spi-plugin-tccl-classloader-gotcha`（第三 locus）。 -> -> **⚠️ 本 session（2026-07-01 后半）另做了一个 meta-cache 属性校验修复（未 commit，待用户裁量）**：`test_iceberg_table_meta_cache` 失败——SPI 切换丢了 `ttl-second=-2` 校验（`CacheSpec.checkLongProperty`），新连接器只保留 best-effort 解析。修复=把 `CacheSpec` 表达模式落一份到 **fe-connector-api**（`org.apache.doris.connector.api.cache.CacheSpec`，校验改抛 `IllegalArgumentException`→fe-core `checkProperties` 原样包成 DdlException），iceberg（6 knob）+ paimon（3 knob）的 `validateProperties` 接回校验（字节对齐 legacy `IcebergExternalCatalog`/已删 `PaimonExternalCatalog` 的 checkProperties）；Phase 2 把 iceberg manifest 的手写 `isCacheEnabled`/`propLong`/`getLong` 改用共享 CacheSpec。设计+完成记录 = `plan-doc/tasks/designs/metacache-connector-cachespec-design.md`。**已 unit 全绿（api CacheSpecTest 9/9、Iceberg/Paimon ValidatePropertiesTest 11/15、iceberg 全模块 892/0-fail），checkstyle 净，import gate 我的文件净；docker 两个 meta_cache 回归未跑（无集群）**。**⚠️ staging 坑**：`IcebergScanPlanProvider.java` 有前序未提交 hunk（~L991/L1003，非本任务），我的 Phase 2 改动同文件——独立提交需 `git apply --cached` 单 hunk 隔离。paimon 单测 `deadTableCacheKeyIsAcceptedNotRejected`（断言 dead knob 不校验）已按恢复指令翻成 `rejectsMalformedMetaCacheKnob`。 -> -> **⚠️ 上述校验修复之后，用户把范围扩大为「整套 metacache 框架统一」**（设计文档 `plan-doc/tasks/designs/metacache-framework-unification-design.md`）：三个连接器手抄 cache（`IcebergManifestCache`/`IcebergLatestSnapshotCache`/`PaimonLatestSnapshotCache`）都是 fe-core 框架 entry 的移植；native iceberg/paimon 的 fe-core `IcebergExternalMetaCache`（含 manifestEntry）**已死**（只 HMS-iceberg 还活），paimon 连 fe-core cache 都没有。**用户已定：Option A**（框架搬到新模块 `fe-connector-cache`，连接器自持 cache）+ 新建 `fe-connector-cache` 模块。关键：`org.apache.doris.connector.*` 是 **parent-first**（`ConnectorPluginManager:64`），框架搬那儿=单 app-loader 身份；`MetaCacheEntry` 对外 API 无 Caffeine 类型 → split-brain 可规避（安全红线：`CacheFactory`/Caffeine 类型不得越界给连接器）。**P1 已完成 skeleton**：建 `fe-connector-cache`（pom caffeine **provided 3.2.3**、注册进 aggregator、fe-core 加依赖、package-info），`-pl fe-connector/fe-connector-cache install` 直接构建 **SUCCESS**。**P1 剩余搬类步骤见设计文档 §8**（CacheSpec 三份合一→其余 leaf→CacheFactory+MetaCacheEntry 改 Config 为 ctor 注入→8 个 plugin-zip 加排除）。 -> **⚠️⚠️ 新暴露的预存在 gate blocker（非本任务引入，但挡住 P1 reactor 构建 + 会挂 CI）**：加新模块使 aggregator 的 build-cache 失效→`check-connector-imports` gate 重跑并 **FAIL**（Phase1/2 只是命中了缓存的 pass）。唯一违规是 commit `4acb5f91e1a` 的 `fe-connector-hms/.../HiveMetaStoreClient.java:21-22` import `datasource.hive.HiveVersionUtil`（补丁版 HMS client）。**需决定**：给该补丁 client 加 gate allowlist，或把 HiveVersionUtil 换个连接器可见的暴露方式。临时绕过：`-pl ` 不带 `-am` 单模块构建。 -> -> **之后 = ENG-1 能力孪生审计**（全部 Medium M-1..M-11 已 ✅），详见下文： - -# 🎯 （之后）= **ENG-1 能力孪生审计（全部 Medium M-* 已 ✅；翻闸 BLOCKED，先修后翻）** - -> **进度**：P0（B-1/B-2）+ 全部关键 P1（H-1..H-10）+ **全部 Medium（M-1..M-11）已全 ✅**——本轮收尾 **M-9 `0d8c5669f9b`**（dropDb 改用 REMOTE 名，镜像 dropTable）/ **M-11 `177f84a7ac9`**（FORCE 删恢复容忍远端已删 namespace，方案 B 含 HMS loadNamespaceLocation 步）/ **M-8 决定=接受偏离不改码**（保留省略空 LOCATION 的 cleaner 输出，用户 2026-06-30 裁定）。逐条状态/commit 见**任务清单 §1–§3** + `git log`（HANDOFF 不再累积「修完成」条目）。 -> -> **⏭ 之后（两个 branch fix 完成后）= ENG-1 能力孪生审计**（全部 Medium 已 ✅）： -> - **入口**：任务清单 **§5 ENG-1** + review 报告 **§七**（残留旧逻辑 / 能力门控）。**全部 Medium M-1..M-11 ☑**（本轮收尾 M-9 `0d8c5669f9b` / M-11 `177f84a7ac9` / M-8 决定=接受偏离不改码；M-10+H-11 ☑ 已并入 B-2 `ba80cfb0439`）。 -> - **ENG-1 = 全量审计 legacy iceberg `instanceof Iceberg*` 臂的能力孪生覆盖**：翻闸后运行时类型 `PluginDriven*`，所有 `instanceof IcebergExternalTable/Catalog/Sys` 求值 false，正确性逐点依赖人工写的「能力孪生臂」；**H-10（嵌套裁剪）是已实证一次漏写=静默回归**。需逐个 legacy iceberg 臂核对是否有等价 PluginDriven 臂/能力门控——**防「逐点静默回归」的唯一保证**。 -> - **处理顺序**：**iceberg branch_tag 两个 fix（complex_queries + partition_operations）◀ 下一（用户 07-01 指定）** → ENG-1 → P3(L-BATCH) → ENG-3 flip-gated e2e 全跑 → 用户二签翻闸。（⚠️ 任务清单 §8 顺序已过时，以此为准。） -> - **每条走 step-by-step-fix**（recon→design→impl→test→clean-room→**独立 commit**→回填任务清单）。**⚠️ 认领前先 recon+`git show master:` 重裁，HANDOFF/review 行号/不变式可能过时（信控制流不信注释）**；冲突项回代码重裁（Rule 7）。 -> - **⚠️ M-3 引入新中立 SPI（`ConnectorSplitSource` + `streamingSplitEstimate`/`streamSplits`）= 流式 split 通道**：将来 Hive/Hudi 迁插件路径可复用（file-count 流式是它们共用老套路）。**v3 iceberg 暂闸出流式**（commit-bridge delete stash 写规划点读，流式懒填太晚→复活已删行）；放开 v3 需先设计 plan-time stash barrier（登记 follow-up）。 - -> **⚠️ 为何 BLOCKED（2026-06-28）**：一轮 clean-room 对抗 review 推翻了「翻闸代码基本完成」的旧结论——发现 **2 blocker + 11 high + 11 medium + 25 low + 18 info**，blocker/high 密集覆盖写入、MTMV、统计、time-travel、缓存一致性等核心路径。**翻闸代码侧写完了但不正确**：P0+关键 P1 现已逐条修完，但仍需关 Medium、跑 ENG-1 审计与 flip-gated e2e 才能二签翻闸。 - -> **📋 任务跟踪入口（下个 session 必先读）**： -> 1. **`plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md`** ← **master checkbox 任务清单**，逐条 ID 对齐 review 报告（B-1/B-2/H-1..H-11/M-1..M-11/L-BATCH/ENG-1..4）。**每条任务的状态、位置、修法、验收、依赖、⚠️RECONCILE 标记都在这里。逐步处理 = 按此表逐条 ☐→◐→☑。** -> 2. **`plan-doc/reviews/P6.6-iceberg-cleanroom-adversarial-review-2026-06-28.md`** ← 完整证据源（每条发现的 file:line、vs master 差异、真回归 vs 内生缺陷、验证者保留意见；**Medium 见 §四**）。 - ---- - -# 🔑 翻闸现状 = **代码侧写完、P0+关键 P1 已修；翻闸 BLOCKED（待 Medium + ENG-1 + e2e + 二签）** - -- **路由翻闸已在分支**（`18e1b297d7e`）：`SPI_READY_TYPES` 含 `"iceberg"`，建/重放 iceberg catalog 走 `PluginDrivenExternalCatalog`；连接器 ServiceLoader 注册 + plugin-zip 打包齐备。**⚠️ 这意味着 review 所有"this path is live"成立，in-code 的 "dormant / not yet in SPI_READY_TYPES" 注释普遍已过时（false claims）——动码时勿信注释，信控制流。** -- **GSON 兼容迁移已在分支**（`e68eb5c00c9`）：旧 8 catalog 变体 + db + table 标签 `registerCompatibleSubtype`→PluginDriven（table→Mvcc 变体）+ 删 CatalogFactory legacy case。保升级老集群（全新/docker 零影响）。**review §六确认完整且写安全（正面）。** -- **未 push、未二签**：路由翻闸 + GSON 迁移**必须一起 push**（[DEC-FLIP-1] 铁律），但**当前不应 push**——先修完 review 发现。 - -## ⛔ 翻闸 gate（全绿才能二签翻闸最后原子提交） -1. **P0 全清** ✅：B-1（云存储写 fs.s3a.* vs AWS_*）+ B-2（MTMV listPartitions 缺）。 -2. **关键 P1 关** ✅：H-1..H-10（破坏主力部署的回归）逐条修完，详见任务清单 §2。 -3. **ENG-1**：能力孪生全量审计 **✅ 已完成**（`fadf844f44c`，16 条确认缺口）；**F1 已修**（`6e14fecc21b`）；**本批 5 条 low 已修完**（`cd7618ef53e`/`c8b39f871e3`/`50e4a6bcb5d`/`50ad635d9b0`/`bc5c39157aa`，统一对抗 review 已过，见顶部 🎯 段 + 任务清单 §5b）；**仅剩连通性 F2/F3/F15/F16（medium，opt-in）= 下个 session 首任务**。 -4. **ENG-3**：flip-gated e2e 全套实跑（DV/V3/MTMV/time-travel branch/vended 写/Kerberized HDFS/rewrite）。 -5. **用户二签**。 -> Medium `M-*`（P2，「翻闸窗口或紧随其后」）= 用户重排的**下一步工作**（见顶部 ⏭），非严格 flip-gate；逐条见任务清单 §3。 - ---- - -# ⚖️ 关键决策（沿用，用户已签） - -## [DEC-FLIP-1] 持久化 GSON 迁移 = 方向 A（已落地 `e68eb5c00c9`） -> **⚠️ 推送顺序铁律不变**：路由翻闸（`18e1b297d7e`）与 GSON 迁移（`e68eb5c00c9`）**必须一起 push/上线**。单 push 路由翻闸而漏 GSON 迁移到会被升级的老集群 → 老 iceberg 镜像反序列化崩。**但当前两者都不应 push——先修完 review 发现，翻闸做成最后一个原子提交（路由+GSON 已在前序 commit，最后补齐 fix + e2e + 二签）。** - -## [视图范围] = parity only（B0/B1/B2/B3 全 DONE） -查询 B1 / DROP+删库级联 B2 / SHOW CREATE B3 / 中立地基 B0 全完。CREATE/RENAME VIEW 出范围（fail-loud）。翻闸后视图 schema 回归 = H-8（已修，见任务清单 §2）；视图面残留 low（L-17/L-18/L-19/L-20 文案/缓存）随 P3。 - -## [REVIEW 纪律] clean-room，不注入先验（本轮已执行） -本轮 review 刻意不注入开发先验（忽略 plan-doc/注释/commit message）。**后果：部分发现与历史记忆冲突**（最突出=M-10 SHOW PARTITIONS：本轮判真回归 vs 旧记忆判"误报死码翻闸反改善"，已裁定 M-10 正确并随 B-2 修）。**认领冲突项时回代码 + `git show master:` 重裁，不盲信任一方（Rule 7）。** - ---- - -# ⚠️⚠️ 用户铁律：**fe-core 不得新增 `if(iceberg)` / `instanceof Iceberg*` / `import IcebergUtils` / 引擎名字符串判别（新 seam）** -iceberg 逻辑落 `fe-connector` 经中立 SPI / ConnectorCapability。**legacy 豁免类**保留 iceberg 引用合法（C4 dead 子树 + commit-bridge 旧清单 + `PhysicalIcebergTableSink`/`bindIcebergTableSink` + `StatementContext` 旧 iceberg-typed stash + `IcebergExternalCatalog` + `ShowCreateDatabaseCommand`/`Env.getDdlStmt` legacy iceberg 臂 + `BindRelation case ICEBERG_EXTERNAL_TABLE` + `ShowCreateTableCommand` legacy ICEBERG 视图臂 + `InsertUtils` 既有 `UnboundIcebergTableSink` 分支）。 -> **修 Medium 发现时尤其注意**：若需新增门控走 `ConnectorCapability` / 中立 SPI 而非 instanceof / 引擎名（例：M-7 DLF 守护；M-4 字段编号链路可参照已完成的 H-10）。 - ---- - -# 🟡 已登记 follow-up（部分已并入任务清单） -- **[FU-forcedrop-nosuchns]** = 任务清单 **M-11**（pre-existing）：**namespace 级已修** `177f84a7ac9`（FORCE 删恢复 catch NoSuchNamespaceException，含 HMS loadNamespaceLocation 步=方案 B）；**per-table 级残留 partial**——连接器 `dropTable` seam 缺 master 的 `tableExist`+ifExists 守护，per-table NoSuchTableException 仍不容忍（但 master 亦不经 catch 容忍它→出范围）。 -- **[FU-flip-e2e]** = 任务清单 **ENG-3**（真翻闸端到端未跑）。 -- **[FU-rewrite-output-sizing]（R6/R8）** 中立 driver 未线程 target-file-size + 自适应并行度（与已完成 H-9 同文件族）。 -- **[FU-paimon-topn-dict]（low，M-4 clean-room 两 reader 独立发现，非 M-4 回归，出范围）** = 迁移后 paimon `PaimonScanPlanProvider.buildSchemaEvolutionParam` 的 `-1` 当前 schema 条目按**裁剪列**建（legacy paimon 恒全列），与 iceberg M-4 同型潜在 Top-N 懒物化缺口；但 paimon 另发**每 committed schema-id 的全列 history 条目**（iceberg 只发单 `-1`），其 topn 安全性（若有）或赖于此 → **需独立验证**（确认 paimon BE 按 row-id 回表补取是否经那些全列 history 条目解析、是否真有 wrong-rows）。若确认有缺口，可复用 M-4 的 `applyTopnLazyMaterialization` SPI（paimon 覆写即可）。 -- **[FU-h10-deadcode]（LOW，cosmetic/非正确性）** 两条翻闸后死码，留 ENG-1/cleanup：① `PlanNode.mergeIcebergAccessPathsWithId`（`instanceof IcebergScanNode` EXPLAIN 访问路径合并臂，翻闸后只显示 `name` 非 `name(id)`，BE 仍收编号形路径）；② `LogicalFileScan.supportPruneNestedColumn` 的 legacy `IcebergExternalTable||IcebergSysExternalTable→return true` 臂（与 L2 现已不一致、仅反翻闸成隐患）。 -- **[FU-view-gson-roundtrip] / [FU-view-exception-arms] / [FU-getsqldialect-deadcode] / [FU-showcreatedb-render-ut] / [FU-createtablelike-plugin]**（低）见 git log 历史 + 任务清单 L-BATCH。 -- 其余（nested-nullability / where-literal-coercion / broker-write〔=M-5〕/ doris-version-prop〔=L-13〕等）多已被 review 重新发现并归入任务清单。 - ---- +- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**（复用 P5-T29 #64653 / P6 #64688 范式）。 +- **公开 tracking issue = apache/doris#65185**（catalog-SPI 迁移 umbrella，大步骤勾选清单 + 连接器表）；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 +- **⚠️ path-whitelist `git add`，严禁 `git add -A`**（scrub `regression-test/conf/regression-conf.groovy` 明文 key + `*.bak` + scratch `.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...` + `plan-doc/reviews/P5-paimon-rereview3-*`；`.claude/` 是 memory、非仓内）。 +- commit message：见 `git log` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 任务清单 + 设计文档单独 commit。 # ⚙️ 操作须知（复用） -- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**fe-core 只依赖 `fe-connector-api`** → `:fe-core -am` 不拖 paimon。**fe-connector-paimon 单独 build 必须 `package`**（HiveConf 来自 optional shade，`test-compile` 假错）。**iceberg/api** 正常 `-am test`。 -- **⚠️ checkstyle 别加 `-am`**：`-am` 把 `fe-common`（2381 既存 error）拖进假红 → `mvn -pl : checkstyle:check`（不带 -am）。 -- **⚠️ bash 工具默认 timeout 120s**：fe-core build 超时 → 调 `timeout` ~590000ms 或后台跑（全模块 ~2min）。 -- **⚠️ maven 经管道 `$?` 是管道尾的** → 用 `${PIPESTATUS[0]}` 或 grep `BUILD SUCCESS`；`-q` 抑制 console → 读 surefire **XML** 的 `tests=`/`failures=`。 -- **⚠️ stale .class 假红坑**：mutation 后 `os.utime`；**commit 前最终验证务必 fresh recompile**。 -- **⚠️ fe-connector-iceberg 全模块测试套有预存在 flaky 污染（M-2 期间实证，非任一 fix 引入）**：跑全 849 测时偶发 3 个 field-id/能力测试红（`IcebergConnectorTest.declaresNestedColumnPruneCapability`、`IcebergTypeMappingReadTest.nestedFieldIdsCarriedForBeFieldIdScan`、`IcebergConnectorMetadataTest.getTableSchemaParsesColumnsFromLoadedTable`——field-id 读 -1 / 能力读 false），**取决于 surefire 类执行顺序**（顺序相关共享静态态污染）；三类**单独跑全绿**、stash 改动后**clean tree 同样偶发**——即非确定性、非改动引入。另 `fe-connector-metastore-iceberg` 的 `IcebergMetaStoreProvidersDispatchTest` 亦预存在 flaky（clean tree 也红，且它是 iceberg 的 -am 上游→其红会 skip 整个 iceberg 测试致 XML stale；隔离验证可加 `-Dtest='!IcebergMetaStoreProvidersDispatchTest'`）。**判 iceberg fix 是否破测勿信单次全量红**：① 单独跑相关类；② stash 后对比 clean tree。建议归 ENG（测试隔离修复），非单点 fix 范围。 -- **⚠️ 后台 task 通知的 "exit code" 是末尾 echo/df 的、非 maven 的**（M-2 又踩一次）：读 LOG 里 `MAVEN_EXIT=`/`BUILD` 行或 surefire XML，别信通知的 exit 0。 -- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`。**连接器测试无 Mockito**（真 InMemoryCatalog/Recording fakes）；**fe-core 用 Mockito**（`CALLS_REAL_METHODS` + `Deencapsulation.setField` + stub `getConnector`/`getMetadata`/`buildConnectorSession`）。**⚠️ Mockito `anyString()` 不匹配 null**。 -- **mutation-check（Rule 9/12）**：范式 scratchpad `mutate_*.py`（单行 exact-string 锚点 count==1 守；KILLED=maven rc!=0）。**⚠️ Python 3.6**：`subprocess.run(stdout=PIPE,stderr=STDOUT,universal_newlines=True)`（无 `capture_output`）。**⚠️ review（读源）与 mutation（改源）务必串行**。 +- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**fe-core 只依赖 `fe-connector-api`**。**fe-connector-hive/hms 单独 build 注意 optional-shade 依赖**（如 HiveConf；参照 paimon `package` 而非 `test-compile` 的坑）。 +- **⚠️ checkstyle 别加 `-am`**：`-am` 把 `fe-common`（大量既存 error）拖进假红 → `mvn -pl : checkstyle:check`（不带 -am）。 +- **⚠️ bash 工具默认 timeout 120s**：fe-core build 超时 → 调 `timeout` ~590000ms 或后台跑（全模块 ~2min）。**后台 task 通知的 "exit code" 是末尾 echo/df 的、非 maven 的**——读 LOG 里 `BUILD SUCCESS`/`MAVEN_EXIT=` 行或 surefire XML（`tests=`/`failures=`），别信通知 exit。 +- **⚠️ maven 经管道 `$?` 是管道尾的** → 用 `${PIPESTATUS[0]}` 或 grep `BUILD SUCCESS`。**mutation/review 后 commit 前务必 fresh recompile**（stale `.class` 假红坑）。 +- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`。**HMS `HiveVersionUtil` import 命中 = 误报非违规**（`fe-connector-hms` 内 vendored 同名副本，非 fe-core）——勿改，详见 memory `catalog-spi-hms-hiveversionutil-gate-false-positive`。 +- **连接器测试无 Mockito**（真 InMemoryCatalog/Recording fakes）；**fe-core 用 Mockito**（`CALLS_REAL_METHODS` + `Deencapsulation.setField`；`anyString()` 不匹配 null）。详见 memory `catalog-spi-fe-core-test-infra`。 - **cwd 会被 harness 重置** → 一律绝对路径。 -- **⚠️ 环境**：`/mnt/disk1` 紧（2.0T，96% used）。**下个 session 起步先 `df -h /mnt/disk1`**；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 - -# ⚠️ Commit 须知（任何 `git add` 前必读） -- **path-whitelist `git add`，严禁 `git add -A`**（scrub `regression-test/conf/regression-conf.groovy` 明文 key + `*.bak` + scratch `.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`·仓根游离 `fe/IcebergScanPlanProvider.java`·`plan-doc/reviews/P5-paimon-rereview3-*`)。 -- commit message：见 `git log` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。PR base = `branch-catalog-spi`，squash。 -- **每条 fix = 独立 commit**（沿用 P4-T06e-FIX-* 范式）；HANDOFF + 任务清单 + 设计文档单独 commit（memory 在 `.claude/`、非仓内）。 - -# 📦 阶段状态 -- **工作分支 = `catalog-spi-10-iceberg`**（off `branch-catalog-spi` @ `e5959e1b53d`，PR base = `branch-catalog-spi`，squash）。 -- **进度**：P6.1–P6.5 ✅ / P6.6 C1–C3 ✅ / C4 R1–R7 ✅ / C5 DDL/ALTER B1–B5 ✅ / flip-readiness 只读退化 ✅ / 视图 B0–B3 ✅ / 路由翻闸 `18e1b297d7e` ✅ / GSON 迁移 `e68eb5c00c9` ✅ → **⛔ 现卡在 clean-room review 发现修复**：**P0（B-1/B-2）+ 关键 P1（H-1..H-10）全 ✅**（逐条 commit 见任务清单 §1–§2 + `git log`）→ **全部 Medium（M-1..M-11）✅**（收尾 M-9 `0d8c5669f9b` / M-11 `177f84a7ac9` / M-8 决定接受偏离不改码） → **iceberg branch_tag 两个 fix（complex_queries MVCC + partition_operations 静态分区 overwrite）◀ 下一（用户 07-01 指定，见顶部 🎯 + 设计文档）** → ENG-1 能力孪生审计 → P3(L-BATCH) → ENG-3 flip-gated e2e → 二签翻闸。 -- **⚠️ 推送状态**：P6.4 T01–T06+arg-move 已推 `origin`；**其后全部未 push**（含路由翻闸 + GSON 迁移 + 视图 + C4/C5 + 全部 review fix）。**先修 review 发现，勿 push 半成品翻闸。** 留用户裁量。 -- **⚠️ 分支 2026-06-28 被 rebase**：commit 哈希全重写，本文档/旧 commit message 旧哈希以 `git log` 为准。 +- **⚠️ 环境**：`/mnt/disk1` 紧（2.0T，~96% used）。**起步先 `df -h /mnt/disk1`**；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 -# 🧠 给下一个 agent 的 meta -- **逐步处理 = 按任务清单逐条**：每条 Medium (M-*) 走 step-by-step-fix（recon→design 文档 `designs/P6.6-FIX-M--design.md`→impl→test+mutation→clean-room review→独立 commit→回填任务清单状态）。 -- **删除/parity/动码前必 grep 全调用方 + 区分 DEAD vs STILL-CONSUMED**；**HANDOFF/review/设计的依赖名/行号/不变式可能过时** —— 动码前先 recon（grep+实证）再信文档。**翻闸已生效 → in-code "dormant" 注释普遍过时，信控制流不信注释。** -- **⚠️ 冲突优先暴露（Rule 7）**：review 与历史记忆冲突项（M-10 等）回代码重裁，不盲信任一方。`git show master:` 是 legacy 原逻辑的权威来源（工作区 `datasource/iceberg/**` 是迁移后残壳，不可信）。 -- **clean-room 对抗 review 偏好**：moderate+ 改动 = 多 reader 对抗 + critic（review 读源与 mutation 改源不可并发）。verbatim 镜像臂则焦点验证即可。 -- **flip-gated 诚实**：真 post-flip 写/MTMV/time-travel e2e 翻闸后才能跑——**每条 fix 验收的 e2e 项标注 flip-gated 未跑，勿谎称已验**（Rule 12）。 -- **上下文超 30% 即交接**。 +# 🧠 起步必读 -## 📖 起步必读 -0. **顶部 🚀 段（下个 session 任务）** + **`plan-doc/reviews/P6.6-ENG1-capability-twin-audit-2026-07-04.md`** §三（本批修法信源，逐条 master 行为/缺口/failureScenario/建议修法方向）+ 任务清单 **§5b**（16 条缺口状态表）。**用户裁定：直接照审计结论动码、不再 recon，末尾统一 review。** -1. **`plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md`**（master 任务清单）+ **`plan-doc/reviews/P6.6-iceberg-cleanroom-adversarial-review-2026-06-28.md`**（证据源，Medium 见 §四）。 -2. memory（仅列现存相关项）：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`memory-keep-only-general-or-requested`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-plugindriven-no-source-specific-code`、`catalog-spi-connector-session-tz-gotcha`、`catalog-spi-be-java-ext-shared-classpath`、`catalog-spi-h9-rewrite-where-rewrite-mode-done`。 -3. `plan-doc/tasks/designs/P6.6-C5-flip-readiness.md`（C 类 docker 清单 + 翻闸开关/持久化全景）。 +1. **本文档顶部 🎯/🚀 段** + master plan [§3.8](./00-connector-migration-master-plan.md)（P7 战略）+ [`connectors/hive.md`](./connectors/hive.md)（P7 子阶段/SPI 缺口/特殊性）+ master plan §4（13 步 playbook）。 +2. **样板**：`tasks/P5-paimon-migration.md`（full-adopter + 翻闸 + 删 legacy 全流程）；`tasks/P6-iceberg-migration.md`（阶段拆分 spec 范式，P7 spec 照此建）。 +3. **铁律**：fe-core 不得新增 `if(hive)` / `instanceof HMSExternal*` / 引擎名字符串判别（新 seam 走中立 SPI / `ConnectorCapability`）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。 +4. **memory（现存相关项）**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`memory-keep-only-general-or-requested`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-plugindriven-no-source-specific-code`、`catalog-spi-no-property-parsing-in-fecore`、`catalog-spi-be-java-ext-shared-classpath`、`catalog-spi-plugin-tccl-classloader-gotcha`、`catalog-spi-connector-session-tz-gotcha`、`catalog-spi-history-schema-info-lowercase-nested-names`、`catalog-spi-connector-cache-framework-caffeine-coherence`、`catalog-spi-hms-hiveversionutil-gate-false-positive`、`catalog-spi-tracking-issue`。 +5. **上下文超 30% 即交接**（memory `session-handoff-at-30pct-context`）：找干净节点覆写本 HANDOFF + 通知用户开新 session。 diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index dd03451b2420ad..4131fb23b8dee2 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-06-24** | 当前阶段：**P6.1 ✅ DONE + P6.2 ✅ DONE**（iceberg scan+MVCC+cache+vended 全实现，T11 收口；UT 278/0/1）；**P6.3 写路径 RFC ✅ 评审通过 + P6.3-T01~T09 ✅ DONE = P6.3 DONE**（T01 框架统一·SPI 收口 option B；T02 jdbc thrift 入 planWrite + 删 config-bag 三件套 + `appendExplainInfo` hook；T03 `IcebergConnectorTransaction` 骨架 + `addCommitData` + `getUpdateCnt` + `WriteOperation` 枚举；T04 op 选择 + `IcebergWriterHelper` 等价 + begin* guards + baseSnapshotId 捕获；T05 commit 校验套件 + O5-2 `applyWriteConstraint`〔新 `ConnectorPredicate` SPI default-no-op〕+ V3 DV `removeDeletes`；T06 sink 统一〔INSERT/OVERWRITE〕`IcebergWritePlanProvider` + 写排序 SPI + `getBackendFileType` 接缝〔首动 fe-core/planner〕；T07a DELETE/MERGE sink 方言〔连接器·dormant〕；T07b O5-2 生产半〔fe-core `NereidsToConnectorExpressionConverter` + `WriteConstraintExtractor` + 连接器 `IcebergPredicateConverter` conflict-mode；Option A 忠实 legacy〕；T07c 通用 `RowLevelDmlCommand` 壳·`RowLevelDmlTransform` 注册表·6 instanceof 派发重接·O5-2 dormant〔commit `a61cd9262b9`〕；T08 写路径 parity 审计 wf 40→20 confirmed→11 gap-fill + deviation 中央登记 DV-041..044；T09 收口〔汇总设计 `designs/P6.3-T09-iceberg-write-summary-design.md` + faithfulness 对抗验证 wf 6 verifier+critic 全 CONFIRMED〔1 line-cite 修〕〕= **P6.3 DONE**；fe-core 30/iceberg 389/0/1、checkstyle 0、import-gate 0、0 BE/SPI/fe-core 产品改、iceberg 仍**不在** `SPI_READY_TYPES`），**P6.4 procedures ✅ DONE（T01–T09）**（T01 ✅ SPI 设计 + 用户三签字 [D-062]〔Q1=R-A 分相位 / Q2=S-1 扁平 / §4=4-A 连接器自校验〕；T02 ✅ 2026-06-24 SPI 骨架——新 `connector.api.procedure.{ConnectorProcedureOps,ConnectorProcedureResult}` + `Connector.getProcedureOps()` default-null + `IcebergProcedureOps` dormant 占位 + `IcebergConnector` override；connector-api 37/0；T03 ✅ 2026-06-24 base+factory+arg 框架〔`NamedArguments`/`ArgumentParsers`/`ArgumentParser` 逐字 port，`validate` 抛 `DorisConnectorException` 替 `AnalysisException` error 串字节不变〕→ `connector.iceberg.action` + `IcebergProcedureOps` dispatch 骨架〔`getSupportedProcedures` + `runInAuthScope` body 进 `executeAuthenticated`〕、§4=4-A 连接器自校验、去死 `table` 参 + 去无消费者 `getDescription`；iceberg 401/0/1〔arg 框架 `NamedArguments`/`ArgumentParsers`/`ArgumentParser` 后续移 `fe-foundation` 共享、删连接器副本、`validate` 抛 unchecked `IllegalArgumentException` 两侧 re-wrap〕、checkstyle 0、import-gate 0、0 BE/pom-dM 改、faithfulness wf 4→0 confirmed；T04 ✅ 2026-06-24 港 8 pure-SDK 体〔rollback_to_snapshot/timestamp、set_current_snapshot、cherrypick_snapshot、fast_forward、expire_snapshots、publish_changes、rewrite_manifests〕+ `RewriteManifestExecutor` → `connector.iceberg.action` 接 factory switch；逐字 bug-for-bug〔publish STRING+`"null"`、fast_forward `previous_ref` NULLABLE、not-found try 内外不对称、双 wrap、6×BIGINT〕；**必须改签名** `executeAction(Table,ConnectorSession)`〔rollback_to_timestamp 需会话 TZ，新 `IcebergTimeUtils.msTimeStringToLong` ms 格式+alias+`-1` sentinel〕、cache 失效搬 dispatch 级 `context.getMetaInvalidator()`、SPI/factory 签名不动；iceberg 444/0/1、checkstyle 0、import-gate 0、0 BE/fe-core/pom 改、faithfulness wf 1→0 confirmed/1 refuted+0 critic gaps；**T05 ✅ 2026-06-24** `rewrite_data_files` 规划半 → 新包 `connector.iceberg.rewrite`〔`RewriteDataFilePlanner`/`RewriteDataGroup`/`RewriteResult` 3 类港；WHERE 走 `IcebergPredicateConverter` **conflict-mode**〔用户签字 Option A，DV-T05r-where：不可转节点静默丢=过宽 vs legacy fail-loud；rewrite 语境下 over-approximation 非安全〕；执行半+事务半+bind=T06〕；iceberg 467/0/1、faithfulness wf 8→0 confirmed/0 critic gaps〔8 全 test-coverage 观察，delete-filter 覆盖当场补〕；**T06 🟢 2026-06-24（用户裁 Option 1）**：① 连接器事务 `WriteOperation.REWRITE` 变体〔净 0 新 verb，忠实港 legacy rewrite commit，dormant〕已实现；5-reader recon 证伪设计 §5「pinned snapshot+WHERE 重规划」前提〔over-scan→破正确性 + SPI 模块边界 fe-core 够不到连接器 `RewriteDataGroup` + 侧信道翻闸后死〕→ **②③④（执行半↔规划接线 + 文件级扫描范围 + bind/executor + multi-sink-per-txn）= R-B 推后专门写路径 RFC + 翻闸阻塞 DV-T06r-rb**；iceberg UT **475/0/1**、faithfulness wf 4→0 confirmed〔mutation-check 实证 OCC 测不 pin 显式 `validateFromSnapshot`→诚实修正测试不 overclaim〕；**T07 ✅ 2026-06-24** dispatch rewire——新 fe-core adapter `ConnectorExecuteAction`〔`createAction` PluginDriven 分支返它，`run()` 不变=legacy byte-parity；engine 保 priv+`CommonResultSet` wrap+logRefresh / connector 保 arg+body；`DorisConnectorException`→plain `UserException` re-wrap；WHERE 拒 DV-T07-where〕；fe-core `ConnectorExecuteActionTest` 13/0 + `NereidsParserTest` 73/0；faithfulness wf 5-finder 0 finding，critic 6 类→2 当场修〔空-rows→null 形状 + priv 测断言收紧〕+2 DV〔DV-T07-name-order/exc-contract，T08〕+2 note；0 连接器/BE/pom/CatalogFactory 改）；**T08 ✅ 2026-06-24** parity-UT 审计 + gap-fill + DV 中央登记——对抗 byte-parity 审计 wf〔12 area finder：8 procedure + rewrite-planner + transaction-REWRITE + dispatch + infra；每 finding refute-by-default skeptic + completeness critic〕= 28 confirmed utGap + 2 newDeviation〔**所有前向 DV 审计 accurate=True**〕+ critic 8 跨切 layering 漏报；→ **20 gap-fill UT**〔schema 完整性·error 串字节·bug-for-bug execute 路·auth/cache 接缝；连接器 **494/0/1** + fe-core 双测〕；2 测模型坑实证修〔expire deleteWith `[0,0,0,0,2,0]` / rewrite spec_id 漏 `validate()`→getInt no-op，非产品 bug〕；**DV-045〔🔴 BLOCKER=rewrite 执行半翻闸阻塞〕/046〔correctness-bearing=auth-add+DV-T05r-where〕/047〔perf-cosmetic 批〕中央登记**〔44→47，三层镜像 DV-041..044〕；0 产品码〔唯 1 行 `IcebergProcedureOps` 注释〕、checkstyle 0、import-gate 0、iceberg 仍**不在** `SPI_READY_TYPES`；**T09 ✅ 收口汇总设计 `designs/P6.4-T09-procedure-summary-design.md`（7 节）+ faithfulness 对抗验证 `wf_986bd3db-68b`〔7 verifier refute-by-default + critic→3 真错+1 矛盾修：BaseExecuteAction 非字节不变〔arg-move 改〕/「九 commit 待 push」实为二〔仅 T07/T08 未推，T01–T06 已推 origin=`bdc38b14810`〕/`getFileScanTasksFromContext:498`→`:492`/`:929`/§3 NamedArguments fe-foundation 非 fe-core〕 + gate 重跑实证 iceberg 494/0/1 + fe-core 14/0 = P6.4 DONE**）；**P5 paimon 全完成 ✅**——P5 全量（B0–B7 = 测基建/flavor/normal-read/DDL/sys-tables+MVCC/MTMV桥/时间旅行/**翻闸** + P6 全路径 clean-room review 的全部 deviation fix）squash-合入 `branch-catalog-spi`：**#64446 / `38e7140ce56`**（"migrate to catalog SPI + cutover"，+ `e9c5b3e70ce` 修编译）。paimon 现已在 `SPI_READY_TYPES`，FE 走 SPI 路径。**P5-T29（删 legacy + 5 maven 依赖）已合入 #64653 / `d59ed2f96d9`（fe-core 完全 paimon-SDK-free）；P3b kerberos 收口已合入 #64655 / `e5959e1b53d`。下一 = P6.1 iceberg 元数据 recon**（阶段拆分 spec [tasks/P6](./tasks/P6-iceberg-migration.md)，[D-058]）。前序 P0/P1/P2（#63582/#63641/#64096）+ P3 hybrid（#64143）+ P4（#64253/#64300）+ P5（#64446/#64653）+ P3b（#64655）均已合入。| 项目总进度：**~44%**（按 §一 进度条加权：P0+P1+P2+P4+P5 满 + P3 hybrid 45%，约 10.9/25 周） +> 最后更新：**2026-07-05** | 当前阶段：**P7 hive (+HMS) 迁移（启动中）** —— 工作分支 `catalog-spi-11-hive`。**P0–P6 + P3b 全部合入 `branch-catalog-spi`**：P0 #63582 / P1 #63641 / P2 #64096 / P3 hudi(hybrid) #64143 / P4 #64300 / P5 #64446+#64653 / P3b #64655 / **P6 iceberg 迁移+翻闸+删 legacy #64688 `8b391c7459d`**。下一 = **P7.1 HiveMetadataOps 全功能搬迁**（master plan §3.8 + connectors/hive.md）。| 项目总进度：**~64%**（按 §一 进度条加权：P0+P1+P2+P4+P5+P6 满 + P3 hybrid 45%，约 15.9/25 周） > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -15,11 +15,11 @@ | P3 | hudi 迁移 | 2 周 | ▰▰▰▰▰▱▱▱▱▱ 45% | ✅ hybrid（D-019）批 A–D 已合入 `branch-catalog-spi`（**#64143** squash `5c240dc7a34`）；批 E（live cutover）并入 P7 | [tasks/P3](./tasks/P3-hudi-migration.md) | | **P4** | maxcompute 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（**#64253** T01–T06 适配+翻闸 + **#64300** T07–T09 删 legacy/odps-free；含 #64119 校验迁移）| [tasks/P4](./tasks/P4-maxcompute-migration.md) | | **P5** | paimon 迁移 | 3 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（迁移+翻闸 **#64446** + 删 legacy/maven **#64653** `d59ed2f96d9`；B9 回归用户 docker 覆盖）| [tasks/P5](./tasks/P5-paimon-migration.md) | -| **P6** | iceberg 迁移 | 5 周 | ▰▰▰▰▰▰▱▱▱▱ ~50% | 🟢 **P6.1 DONE + P6.2 DONE + P6.3 DONE + P6.4 DONE（P6.3/P6.4 各 T01–T09 ✅）**：P6.1〔T01–T10〕7-flavor 装配 + 读元数据 parity + per-flavor 校验 + metastore 模块拆分；P6.2〔T01–T11〕scan+MVCC+cache+vended 全实现（UT **278/0/1**）；deviation 中央注册 DV-038〔翻闸阻塞〕/039/040。**P6.3 写路径 RFC ✅ 评审通过**（`a49720820f9`）→ **T01~T09 ✅ DONE = P6.3 DONE**（框架统一·SPI 收口 + jdbc planWrite·config-bag 删 + `IcebergConnectorTransaction` 骨架·op 选择·WriterHelper + commit 校验套件·O5-2·V3 DV + sink 统一〔INSERT/OVERWRITE〕·写排序 SPI + DELETE/MERGE sink 方言〔T07a〕+ O5-2 生产半〔T07b〕+ 通用 `RowLevelDmlCommand` 壳·注册表·6 派发重接〔T07c〕+ parity 审计·DV-041..044 登记〔T08〕+ 收口汇总设计·faithfulness 对抗验证〔T09〕；iceberg UT **389/0/1**）→ **P6.4 procedures DONE**（T01 ✅ SPI 设计 + 用户三签字 [D-062]〔R-A 分相位 / S-1 扁平 / 4-A 连接器自校验〕；T02 ✅ SPI 骨架；T03 ✅ 2026-06-24 base+factory + dispatch 骨架〔`runInAuthScope`〕+ arg 框架移 fe-foundation 共享，iceberg UT **401/0/1**；T04 ✅ 港 8 pure-SDK 体 + `RewriteManifestExecutor`；T05 ✅ `rewrite_data_files` 规划半 → `connector.iceberg.rewrite` 3 类〔WHERE conflict-mode，Option A，DV-T05r-where〕iceberg **467/0/1**；**T06 🟢 2026-06-24（用户裁 Option 1）** ① 连接器事务 `WriteOperation.REWRITE` 变体〔净 0 新 verb，忠实港 legacy rewrite commit，dormant〕实现 + iceberg UT **475/0/1**，**②③④（执行半↔规划接线 + 文件级扫描范围 + bind/executor + multi-sink-per-txn）= R-B 推后专门写路径 RFC**〔recon 证伪设计 §5 over-scan 前提 + SPI 模块边界；翻闸阻塞 DV-T06r-rb〕，faithfulness wf 4→0 confirmed；**T07 ✅ 2026-06-24** dispatch rewire〔新 fe-core adapter `ConnectorExecuteAction`，`createAction` PluginDriven 分支返它·`run()` 不变=legacy byte-parity；engine 保 priv+wrap+logRefresh / connector 保 body；`DorisConnectorException`→plain `UserException` re-wrap；WHERE 拒 DV-T07-where；fe-core `ConnectorExecuteActionTest` 13/0 + `NereidsParserTest` 73/0；faithfulness wf 5-finder 0 finding，critic 6 类→2 当场修+2 DV〔name-order/exc-contract〕+2 note；0 连接器/BE/pom 改〕；T08 ✅ parity-UT 审计 + DV-045/046/047 中央登记〔iceberg 494/0/1〕；**T09 ✅ 收口汇总设计 + faithfulness 对抗验证〔7 verifier+critic→3 真错+1 矛盾修：BaseExecuteAction 非字节不变/「九 commit 待 push」实为二/`:498`→`:492`/`:929`/§3 NamedArguments〕 + gate 重跑实证〔iceberg 494/0/1 + fe-core 14/0〕= P6.4 DONE**）→ **P6.5 sys-table ✅ DONE**（T01 ✅ 2026-06-24 recon+设计+用户二签字：**仅系统表**〔元数据列推迟 P6.6 DV-041 同族〕、**净 0 新 SPI**〔复用 E7 `PluginDrivenSysExternalTable`〕、**fe-core 零改动**；5 偏差不照抄 paimon；**T02 ✅** `IcebergTableHandle` sys 变体〔`sysTableName` 非 transient + `forSystemTable` **保留** snapshot pin〔偏差①〕+ 9 UT〕；**T03 ✅ 2026-06-24** `IcebergConnectorMetadata` 2 override〔`listSupportedSysTables`〔`MetadataTableType.values()` 去 position_deletes 小写 unmodifiable〕+ `getSysTableHandle`〔guard〔null/unknown/position_deletes→empty〕+ 保留 pin + **LAZY 零 catalog 往返**，[D-063]——eager build 移 T04〕+ `isSupportedSysTable` + 11 UT〔3-变异 mutation-check 恰 3 红〕；连接器 **514/0/1**〕；**T04 ✅ 2026-06-24** `getTableSchema`/`getColumnHandles` sys 分支〔新 `loadSysTable`=`executeAuthenticated` 内 `MetadataTableUtils.createMetadataTableInstance`〔决策 B 无新 seam，偏差③〕→ meta-table schema 经 `parseSchema` 透 `enable.mapping.*`〔偏差⑤〕+ 3-arg @snapshot sys 短路〔schema 与快照无关〕+ getColumnHandles sys 分支，[D-064]〕+ 7 UT〔含 seam-identity；RED 5 + 3-变异 mutation-check 验 2 auth guard + 类型线〕；**521/0/1**〕；**T05 ✅ 2026-06-24** `IcebergScanPlanProvider`/`IcebergScanRange` sys split 路〔`planScanInternal` sys guard→`planSystemTableScan`〔`resolveSysTable` 元数据表 + 复用 `buildScan` time-travel→`planFiles`→`SerializationUtil.serializeToBase64`〕；carrier `serializedSplit` 字段 + `populateRangeParams` sys 早返〔仅 `serialized_split`+FORMAT_JNI+`table_level_row_count=-1`，镜像 legacy `setIcebergParams`〕，[D-065]；6 UT〔含 deserialize-round-trip 经 BE 路 + 4-变异 mutation-check〕；潜伏=serialized 字节 BE 兼容 P6.8 e2e 兜底；**527/0/1**〕；**T06 ✅ 2026-06-25** `buildTableDescriptor` hms↔iceberg 分叉〔连接器，覆 base+sys，BE 无感纯 FE parity，闭合 base SCHEMA_TABLE 缺口〕+ `getScanNodeProperties` sys 收口〔[D-065]，跳 dict+ppk 修 unpinned-sys 潜伏崩溃〕+ fe-core engine-name/SHOW-CREATE parity〔[D-066]，用户签字；F2 authTableName 解包亦修 paimon 潜伏 priv 过严〕；recon 纠 HANDOFF 框定 2 处〔buildTableDescriptor 是连接器 SPI 钩子 / SHOW CREATE 输出已 Env+UserAuth 解包〕；3 mutation-check 红证；**532/0/1**〕；T07–T08 待），仍 behind gate | [tasks/P6](./tasks/P6-iceberg-migration.md) | -| P7 | hive (+HMS) 迁移 | 6 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | +| **P6** | iceberg 迁移 | 5 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（迁移+翻闸+删 legacy 原生子系统 **#64688** `8b391c7459d`；P6.1–P6.6 全阶段 + 属性/鉴权全归插件 S1–S10 一次性 squash，685 文件 −23744 行）。⚠️ fe-core `datasource/iceberg/` 尚存 23 个 HMS-iceberg 支撑类，随 P7 hive 迁移删（阶段四）| [tasks/P6](./tasks/P6-iceberg-migration.md) | +| **P7** | hive (+HMS) 迁移 | 6 周 | ▰▱▱▱▱▱▱▱▱▱ 5% | 🚧 启动中（当前活跃阶段；起步 P7.1 HiveMetadataOps）| [connectors/hive.md](./connectors/hive.md) | | P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | -**全局进度：~44%**（25 周计划中已完成约 10.9 周：P0+P1+P2+P4+P5 满 + P3 hybrid 45%；按 §一 进度条加权） +**全局进度：~64%**（25 周计划中已完成约 15.9 周：P0+P1+P2+P4+P5+P6 满 + P3 hybrid 45%；按 §一 进度条加权） --- @@ -35,8 +35,8 @@ | hudi | 🟡（D-005 区分符 + D-020 模型 dispatch 已设计；实现批 E）| 🟨 55%（读路径 dormant + 批 C 测试基线）| ❌（gate 关）| ❌ | 0/0（寄生 hms）| **25%** | [详情](./connectors/hudi.md) | | maxcompute | ✅ | ✅ 100% | ✅ **已合入 #64253** | ✅ **#64300 已删** | ✅ 0/0 | **100%** | [详情](./connectors/maxcompute.md) | | paimon | ✅ | ✅ 100% | ✅ **已入 SPI_READY_TYPES** | ✅ **#64653 已删** | ✅ 热区已清 | **100%** | [详情](./connectors/paimon.md) | -| iceberg | 🟢（P6.1–P6.5 DONE · P6.6 翻闸待）| 🟢 **P6.1+P6.2 全实现**（7-flavor 装配 + 读元数据 parity + per-flavor 校验 + scan/MVCC/cache/vended；UT 278/0/1）；**P6.3 写路径 RFC ✅ + T01~T09 ✅ = P6.3 DONE**（框架统一 + jdbc planWrite·config-bag 删除 + `IcebergConnectorTransaction` 骨架·op 选择·WriterHelper·begin guards + commit 校验套件·O5-2·V3 DV + sink 统一〔INSERT/OVERWRITE〕·写排序 SPI + DELETE/MERGE sink 方言〔T07a〕+ O5-2 生产半〔T07b〕+ 通用 `RowLevelDmlCommand` 壳·注册表·6 派发重接〔T07c〕+ parity 审计·DV-041..044 中央登记〔T08〕+ 收口汇总设计·faithfulness 对抗验证〔T09〕；iceberg UT 389/0/1 + fe-core 30/0）→ **P6.4 procedures DONE**（T01 ✅ SPI 设计+三签字 [D-062]；T02 ✅ SPI 骨架〔`ConnectorProcedureOps` + `Connector.getProcedureOps()` default-null + `IcebergProcedureOps` dormant + override；connector-api 37/0〕；T03 ✅ 2026-06-24 base+factory port + dispatch 骨架〔`runInAuthScope` body 进 auth 作用域；§4=4-A 连接器自校验〕+ arg 框架〔`NamedArguments`/`ArgumentParsers`〕**移 fe-foundation 共享**〔删连接器副本〕，faithfulness wf 4→0 confirmed；T04 ✅ 2026-06-24 港 8 pure-SDK 体 + `RewriteManifestExecutor` 接 factory switch〔逐字 bug-for-bug；**必须改** `executeAction(Table,ConnectorSession)` 透会话 TZ〔新 `IcebergTimeUtils.msTimeStringToLong`〕；cache 失效搬 dispatch 级；faithfulness wf 1→0 confirmed/1 refuted+0 critic gaps〕；T05 ✅ `rewrite_data_files` 规划半 → `connector.iceberg.rewrite`〔WHERE conflict-mode·Option A·DV-T05r-where〕；T06 🟢 ① 连接器事务 `WriteOperation.REWRITE` 变体〔dormant〕+ ②③④=R-B 推后写路径 RFC〔DV-T06r-rb〕，iceberg UT **475/0/1**；**T07 ✅ 2026-06-24** dispatch rewire〔fe-core adapter `ConnectorExecuteAction`·`run()` 不变=legacy byte-parity；engine 保 priv+wrap+logRefresh / connector 保 body；WHERE 拒 DV-T07-where；`ConnectorExecuteActionTest` 13/0 + `NereidsParserTest` 73/0；faithfulness wf 0 finding，critic→2 修+2 DV+2 note；0 连接器/BE/pom 改〕；T08 ✅ parity-UT 审计+DV-045/046/047 登记〔494/0/1〕；**T09 ✅ 收口+faithfulness 对抗验证+gate 重跑实证〔iceberg 494/0/1+fe-core 14/0〕= P6.4 DONE**）/ **P6.5 sys-table ✅ DONE**〔T01 ✅ 2026-06-24 recon+设计+用户二签字：**仅系统表**〔元数据列推迟 P6.6 DV-041 同族〕、**净 0 新 SPI**〔复用 E7 `PluginDrivenSysExternalTable`〕、**fe-core 零改动**；Q2=position_deletes 不上报；5 偏差不照抄 paimon；**T02 ✅** `IcebergTableHandle` sys 变体〔`forSystemTable` 保留 snapshot pin〔偏差①〕+ 9 UT〕；**T03 ✅ 2026-06-24** `IcebergConnectorMetadata` 2 override〔`listSupportedSysTables`〔`MetadataTableType` 去 position_deletes 小写 unmodifiable〕+ `getSysTableHandle`〔guard〔null/unknown/position_deletes→empty〕+ 保留 pin + **LAZY 零 catalog 往返** [D-063]——eager build 移 T04〕+ `isSupportedSysTable` + 11 UT；514/0/1〕；**T04 ✅ 2026-06-24** `getTableSchema`/`getColumnHandles` sys 分支〔新 `loadSysTable`→`MetadataTableUtils.createMetadataTableInstance`〔决策 B/偏差③〕+ mapping flag 透传〔偏差⑤〕+ 3-arg @snapshot sys 短路 + [D-064]；7 UT 含 seam-identity；**521/0/1**〕；**T05 ✅ 2026-06-24** `IcebergScanPlanProvider`/`IcebergScanRange` sys split 路〔`planSystemTableScan`〔`resolveSysTable` 元数据表+复用 `buildScan` time-travel→`serializeToBase64(FileScanTask)`〕+ carrier `serializedSplit` + `populateRangeParams` sys 早返〔serialized_split+FORMAT_JNI+row_count=-1〕，[D-065]；6 UT〔deserialize-round-trip 经 BE 路+4-变异 mutation-check〕；潜伏 serialized 字节 P6.8 e2e 兜底；**527/0/1**〕；**T06 ✅ 2026-06-25** `buildTableDescriptor` hms↔iceberg 分叉〔覆 base+sys〕+ `getScanNodeProperties` sys 收口〔[D-065]〕+ fe-core engine/SHOW-CREATE parity〔[D-066]，用户签字〕；**532/0/1**〕；**T07 ✅ 2026-06-25** 对抗 byte-parity 审计〔8 finder+refute-skeptic+critic，22/19 confirmed〕揭 2 主动偏差→用户裁**现修**〔A sys 时间旅行 guard〔P5 paimon `checkSysTableScanConstraints` 共享 seam〕connector-capability-aware：新 SPI `supportsSystemTableTimeTravel` + iceberg override + fe-core guard 放行 time-travel/branch-tag、@incr 仍拒；B hms 分叉 `equalsIgnoreCase`；[D-067]〕+**9 gap-fill UT**；连接器 **541/0/1** + guard **7/0/0**〔Mut-A/B 红证〕；**DV-048〔correctness〕/049〔perf-cosmetic〕**〕；**T07-续 ✅ 2026-06-25** 残留 8 gap-fill + mutation〔test-6 实证纠 audit spec=BE-residual〕；**T08 ✅ 2026-06-25** 收口汇总设计 `P6.5-T08-systable-summary-design.md` + faithfulness 对抗验证 wf〔18/18 confirmed、0 critic fix〕+ gate 重跑 ⇒ **P6.5 DONE**〕 | ❌（翻闸在 P6.6，DV-038/041/045 + sys 时间旅行 threading 阻塞）| 543 UT（iceberg）+ fe-core sys 10 + guard 8 | 0/49 | **~93%** | [详情](./connectors/iceberg.md) | -| hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **10%** | [详情](./connectors/hive.md) | +| iceberg | ✅ | ✅ 100% | ✅ **已入 SPI_READY_TYPES**（#64688）| ✅ **#64688 已删原生子系统**（HMS-iceberg 23 支撑类随 P7 删）| ✅ 0/0（原生反向 instanceof 已清）| **100%** | [详情](./connectors/iceberg.md) | +| hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **🚧 P7 启动中** | [详情](./connectors/hive.md) | --- @@ -44,11 +44,11 @@ > 状态非 ✅ 的项，按阶段聚合。详细见各阶段 task 文件。 -### P6 — iceberg 迁移（🟢 **P6.1 DONE + P6.2 DONE** [D-058]/[D-059]/[D-060]；P6.1〔T01–T10〕装配+读元数据+校验+metastore 拆分、P6.2〔T01–T11〕scan+MVCC+cache+vended，UT 278/0/1；🎯 **P6.3 写路径 RFC ✅ 评审通过**〔`a49720820f9`〕→ 下一 = 实现 §11 T01–T09。**翻闸阻塞 = DV-038**〔GLOBAL_ROWID + getColumnHandles 共享 fe-core field-id 路径 BE DCHECK，P6.6 前必修〕） +### P7 — hive (+HMS) 迁移（🚧 启动中；最复杂、最后做的连接器。权威计划 master plan §3.8 + connectors/hive.md） -> 策略 = **先在 `fe-connector-iceberg` 实现完整能力（P6.1–P6.5）→ P6.6 一次性翻闸 → P6.7 删 legacy → P6.8 回归**（用户 2026-06-21 签 方案 A / 8 阶段，[D-058]）。翻闸**全有或全无**（`CatalogFactory:104-113`：加入 `SPI_READY_TYPES` 后 scan/write/MVCC/sys-table 全走连接器、seam 无 legacy 回退）⇒ **切忌在 P6.1–P6.5 任何阶段把 iceberg 加进 `SPI_READY_TYPES`**。3 缺失 SPI 各折进首消费阶段：`ConnectorCredentials`(P6.2) / 写路径 RFC(P6.3) / `ConnectorProcedureOps`(P6.4)。 +> 策略 = **先在 `fe-connector-hive`/`fe-connector-hms` 实现完整能力 → 翻闸（hms 入 `SPI_READY_TYPES`）→ 删 fe-core legacy（含 23 个 HMS-iceberg 支撑类 + `datasource/hudi/`，阶段四）→ 回归**（复用 P5/P6 full-adopter 样板）。模块已就绪：`fe-connector-hms` 是 hive/hudi/iceberg-HMS/paimon-HMS 共享元存储库（P3/P5/P6 已在用、稳定）。 > -> **🎯 P6.1 = 连接器地基 + 普通读元数据 + 7 flavor**（含 port DLF 4-file 子树 + wire S3Tables SDK）。recon（`research/p6.1-iceberg-metadata-recon.md`）+ 10-task 拆解（`tasks/P6` §P6.1）+ [D-059]（Q1 DLF port-now / Q2 扩 metastore-spi）完成；**T01（`IcebergCatalogFactory`+`IcebergCatalogOps` seam）/ T02（测试基建）/ T03（metadata rewire）已实现+验证**（27 UT 绿，commit `ae54a2174ff`）；**T08（type-mapping parity，决策无关）已实现+验证 + commit `d41fa4faf3e`**（`TIMESTAMPTZ` 名 + 点分 mapping-flag key + BINARY 无界长度 3 修；36 UT 绿）；**T04（pom 7-flavor 依赖闭包，[D-060]）已实现+验证**（HMS/DLF=复用 `hive-catalog-shade`〔修正 D-059「iceberg-hive-metastore」误述：该 artifact 不存在〕+ AWS SDK v2 child-first + s3tables + metastore-spi + fe/pom.xml dM + plugin-zip thrift 排除；plugin-zip 实查 143 jar、iceberg 全 1.10.1、libthrift 缺席）。下一 = **T05（5 CatalogUtil flavor）**；T05-T07 前须 `MetaStoreProviders.bind` mini-recon（Q2=B 跨 metastore 子线）。 +> **子阶段（connectors/hive.md §子阶段）**：P7.1 HiveMetadataOps 全功能搬迁（2 周）→ P7.2 event pipeline 搬 + `ConnectorMetaInvalidator`（21 event 类，1.5 周）→ P7.3 `HMSTransaction`+`HiveTransactionMgr` ACID 写路径（2 周，**R-002 最大风险**，gate=专门 ACID 集成测试）→ P7.4 DLA 分流改造（`tableFormatType` 区分 hive/hudi/iceberg，D5，0.5 周）→ P7.5 删 fe-core `datasource/hive/` + 31 处反向 instanceof（0.5 周）。**起步无硬前置**（P0 已建 `ConnectorMetaInvalidator`、fe-kerberos P3b 已收口）。**下个 session 起步 = 建 `tasks/P7-hive-migration.md` 阶段拆分 spec + code-grounded recon（信控制流不信注释）。** ### P5 — paimon 迁移（✅ 全完成并合入：迁移+翻闸 #64446 + 删 legacy/maven #64653；B9 回归用户 docker 覆盖） @@ -156,6 +156,8 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-07-05（P6 iceberg 全部完成 + squash-合入 `branch-catalog-spi` #64688 `8b391c7459d` ⇒ P7 hive 启动）** ✅（已合入 upstream）。整条 P6（P6.1–P6.6 迁移/scan/write/procedures/sys-tables/行级 DML + 翻闸 + GSON 迁移 + 属性/鉴权全归插件 S1–S10 + 删 fe-core 原生 iceberg 子系统 −23744 行）一次性 squash 为 **#64688**（685 文件）。iceberg 已入 `SPI_READY_TYPES`，FE 走 SPI 路径。**遗留（P7 接手）**：fe-core `datasource/iceberg/` 尚存 **23 个 HMS-iceberg 支撑类**（`IcebergUtils`/`IcebergMetadataOps`/`source/IcebergScanNode`/`cache/`/`IcebergMvccSnapshot`/…），iceberg-on-HMS 走它们、随 P7 hive 迁移删（阶段四，D5/Q3=B）。**新工作分支 `catalog-spi-11-hive`**；[DEC-FLIP-1]「未 push」铁律随合入解除。**下一 = P7 hive/HMS 迁移**（master plan §3.8 + connectors/hive.md，起步 P7.1）。 + - **2026-06-25（P6.5-T07 ⇒ parity 审计 + 2 现修 + 9 gap-fill + DV 中央登记，TDD+mutation）** 🟡（未 push；连接器 + guard 的 iceberg 分支 pre-flip dormant，guard 修对 paimon 行为不变〔默认 false 保留拒绝〕，零 live iceberg 行为变更）。**对抗 byte-parity 审计 wf** `wf_d530d760-ccf`〔8 area finder 覆 T02–T06 sys 路 + DESCRIBE/SHOW/info_schema；refute-by-default skeptic〔effort=high〕+ completeness critic；31 agent/1.6M token〕= **22 finding/19 confirmed**〔3 refuted 全对〕。**揭出 2 项主动偏差→用户 AskUserQuestion 双裁「现修」（[D-067]，非 DV）**：**A** parity-surface finder + critic **双独立揭出** + 主 session 实证——共享 fe-core `PluginDrivenScanNode.checkSysTableScanConstraints`〔P5 paimon `38e7140ce56`，"Mirrors PaimonScanNode.getProcessedTable"〕无条件拒**任何** `PluginDrivenSysExternalTable` 的 snapshot+scan-params；legacy `IcebergScanNode.createTableScan:569` 对 sys 表 honor `useRef/useSnapshot`〔无 isSystemTable gate〕+ 连接器 T02/T05 保留+honor pin（偏差①）→ 翻闸后 pin **dead-on-arrival**=回归〔DV-038/041 同族〕；**纠 HANDOFF**「偏差①非 DV」分类错。**B** `buildTableDescriptor` `TYPE_HMS.equals(原始值)` 大小写敏感→`type="HMS"` 得 ICEBERG_TABLE vs legacy HIVE_TABLE。**现修 A**（3 文件）：新 SPI `ConnectorScanPlanProvider.supportsSystemTableTimeTravel()` 默认 **false**〔paimon/mc/jdbc/es 继承、零回归〕+ `IcebergScanPlanProvider` override true + fe-core guard capability-aware〔放行 time-travel/branch-tag，**@incr 对所有连接器仍拒**——合成元数据表 incremental 无定义、legacy 静默忽略 guard fail-loud 更安全〕；**⚠️ guard 修仅移除主动 BLOCK**，完整 sys 时间旅行 e2e 还需 query→handle pin 休眠翻闸接线〔DV-041 族〕+ P6.8。**现修 B**：`equalsIgnoreCase` 一行 + 大写 UT。**+9 连接器 gap-fill**〔handle coords-in-identity〔plan-cache key〕/pinned-toString·colhandles auth-scope×2/keyset #969249/empty-sysname·sys location-creds〔BE-403〕·capability·hms 大写·fork〕。**mutation-check**：guard Mut-A〔`=false`→AllowsTimeTravel/AllowsBranchTag 双红〕+Mut-B〔去 @incr 子句→StillRejectsIncr 红〕；hms Mut〔`equalsIgnoreCase→equals`→大写 UT 红〕；每次 `cp`/python 复原 diff IDENTICAL。**验证（重跑 surefire，Rule 12）**：连接器全量 **541/0/1**〔=532+9，0 回归〕；fe-core guard **7/0/0**；checkstyle 0；import-gate 0；`CatalogFactory:51` 未改→iceberg 仍**不在** `SPI_READY_TYPES`。**新 1 D（[D-067]）；新 2 DV（DV-048 correctness-bearing〔F2 paimon priv·serialized 字节潜伏〕/049 perf-cosmetic〔sys split-weight·内部 TableType·position_deletes 文案〕）**。**T07-续 ✅（同日本 session）= 残留 8 gap-fill UT 全落 + mutation**：fe-core〔sys `getMysqlType`="BASE TABLE"〔同测钉 new==legacy `ICEBERG_EXTERNAL_TABLE.toMysqlType`〕/ `getEngine`="iceberg"+`getEngineTableTypeName`="ICEBERG_EXTERNAL_TABLE"〔**assertAll** 两 pin 独立 mutation〕/ position_deletes 通用 not-found〔含 snapshots 正控〕/ 非注册不变式〔sys 类无 `@SerializedName` 声明字段 + discovery key 无 `$`〕/ guard-message 顺序〔scan-params 先于 time-travel〕〕+ 连接器〔sys predicate **作为 residual 带给 BE** + sys split `/dummyPath`〕+ WHY-comment 修。**⚠️ 纠 audit spec（实证）**：sys 元数据列谓词是 **BE-applied residual 非 FE 行裁**——record_count=10 过滤后 FE 行数 2 vs 2 不变，FE 可达观测=反序列化 `task.residual()` 携 `record_count`〔plan-time 裁是 snapshot pin，已另测〕；故 test-6 由「行裁」改「residual 携带」，**非 legacy 偏差**〔legacy 同样 serialize FileScanTask 带 residual 给 BE〕。WHY-comment 修：legacy resolution 是 **case-SENSITIVE** `TableIf.findSysTable` `Map.get` over `IcebergSysTable` 小写键〔`getTableNameWithSysTableName` 不 lower-case suffix〕→连接器 `equalsIgnoreCase` accept 是 production 永不达无害超集；原注释误归给 `MetadataTableType.from`〔后者作用在 metadata-table BUILD 时 `resolveSysTable`，非 resolution gate〕。**mutation-check（全绿→红→`git checkout` 复原 IDENTICAL）**：fe-core 5 变异一次 build〔TableIf PLUGIN mysqltype→null / getEngine+getEngineTableTypeName iceberg case 删〔assertAll 双红〕/ 注入 position_deletes / `@SerializedName` on sysTableName / swap guard 块→time-travel 先〕→ test1–5 全红+1 已知 collateral〔testDelegates size 2→3〕；连接器 2 变异〔sys 丢 filter→residual=true / 改 dummy-path 常量〕→ test6/7 红。**验证（重跑 surefire，Rule 12）**：fe-core `PluginDrivenSysTableTest` 10/0/0 + guard 8/0/0；连接器全量 **543/0/1**〔=541+2，0 回归〕`IcebergScanPlanProviderTest` 67/0 + `IcebergConnectorMetadataSysTableTest` 22/0；checkstyle 0、import-gate 0、`CatalogFactory:51` 未改。**0 新 D/DV**。**live-e2e 未跑**（dormant，P6.8 兜底）。**T08 ✅（同日 ⇒ P6.5 DONE）**：收口汇总设计 `designs/P6.5-T08-systable-summary-design.md`（7 节，仿 P6.3/P6.4-T09）+ **faithfulness 对抗验证 wf** `wf_27596236-5fe`（3 verifier refute-by-default〔fe-core / 连接器+test6 / WHY-comment 链〕+ completeness critic；4 agent/256k token）= **18/18 confirmed、0 refuted、0 critic fix**；critic 经 **iceberg 1.10.1 bytecode** 独立证实 test-6 residual（`BaseFilesTable$ManifestReadTask` 携 row filter 为 per-task residual、`ManifestEvaluator.forRowFilter` 仅 prune manifest 文件非 metadata 行、`rows()` 不 apply residual→FE 行数不变 BE 应用）+ WHY-comment 链 C1–C5 全 confirmed（注释「factually accurate」）。gate 重跑实证 543/0/1 + fe-core 10/0+8/0。**= P6.5 DONE**。**下一 = P6.6 翻闸**（全有或全无，须 holistic 修 DV-038〔读〕/041〔写〕/045〔rewrite〕+ sys 时间旅行 query→handle pin threading）。 - **2026-06-25（P6.5-T06 ⇒ thrift 描述符 hms↔iceberg 分叉 + `getScanNodeProperties` sys 收口 + fe-core engine/SHOW-CREATE parity，TDD）** ✅（未 push；连接器 dormant + fe-core 路 flip 后才激活，零 iceberg 行为变更）。**改动 = 4 产品 + 2 测 + 1 新测类**。**8-agent recon workflow** `wf_aefdfdd7-57e` **纠 HANDOFF 框定 2 处**：① `buildTableDescriptor` 是连接器级 SPI 钩子〔`ConnectorTableOps:187` 默认 null→fe-core `PluginDrivenExternalTable.toThrift:511` 回退 SCHEMA_TABLE〕**非 fe-core 方法**，iceberg 连接器原无覆写→base+sys 都退化 SCHEMA_TABLE；② SHOW CREATE 输出已由 `Env.getDdlStmt:4915`+`UserAuthentication:60` 解包 `PluginDrivenSysExternalTable`〔recon F 初判误称未解包，自核纠正〕，仅 authTableName priv-check 残留。**C1（[D-066] 复现 fork）**：`IcebergConnectorMetadata.buildTableDescriptor` 按 `iceberg.catalog.type=="hms"`→`HIVE_TABLE`+`THiveTable` 否则 `ICEBERG_TABLE`+`TIcebergTable`〔镜像 legacy `toThrift:116-131`，6-arg 描述符+空 props，null-safe〕；SPI 无 handle→单覆写覆 base+sys，仿 paimon；**BE 无感**〔recon G：sys JNI 读只看 scan-range FORMAT_JNI+serialized_split，描述符表类型 HIVE/ICEBERG/SCHEMA 皆合法不崩〕→纯 FE parity+闭合 base 缺口。**C2（[D-065] 收口）**：`getScanNodeProperties` 顶 `isSystemTable()` guard 跳 `path_partition_keys`+`schema_evolution` dict〔保 jni+location 凭据〕，修 unpinned-sys 潜伏崩溃〔`encodeSchemaEvolutionProp`→`buildCurrentSchema:194` meta 列不在 base schema 抛〕；iceberg `resolveTable` 取 base 表故须**显式** guard〔paimon type-driven 隐式跳无法照搬〕。**F1（fe-core，用户签字）**：`PluginDrivenExternalTable.getEngine/getEngineTableTypeName` 加 `case "iceberg"`→"iceberg"/`ICEBERG_EXTERNAL_TABLE`〔paimon 安全，仅 iceberg-typed catalog 命中〕。**F2（fe-core，用户签字）**：`ShowCreateTableCommand.validate():116` authTableName 解包 `PluginDrivenSysExternalTable`→source〔镜像既有 `IcebergSysExternalTable` 分支+`UserAuthentication`；**含 live paimon 潜伏 priv 修**，sys 表 SHOW CREATE 现按 base 授权，严格更宽松同向；**⚠️ 无隔离 UT**——`validate()` 依赖全局单例 `Env`/`ConnectContext`/`AccessManager`，仓内无既有 harness→编译+与 live `UserAuthentication`/`Env` 一致性+P6.8 e2e 兜底〕。**TDD**：C1 新测类 `IcebergBuildTableDescriptorTest` 3 测 + C2 2 测加 `IcebergScanPlanProviderTest` + F1 2 测加 `PluginDrivenExternalTableEngineTest` → RED〔C1 null/NPE；C2 unpinned 抛 Runtime 潜伏崩溃+pinned dict-present；F1 "Plugin"/"PLUGIN_EXTERNAL_TABLE"〕→ GREEN → **mutation-check 3 处**〔A fork 判别 `TYPE_HMS`→`TYPE_REST` swap→hms/rest 双红；B dict guard `if(true)`→unpinned 抛+pinned dict-present 红；C ppk guard `if(true)` 保 dict guard→unpinned 达 `assertFalse(ppk)` 红〔治 trivially-pass 坑〕；`cp` 复原 diff IDENTICAL〕。**验证（重跑 surefire，Rule 12）**：连接器全量 **532/0/1**〔=527+5；`IcebergBuildTableDescriptorTest` 3/0/0、`IcebergScanPlanProviderTest` 63/0/0〕；fe-core `PluginDrivenExternalTableEngineTest` **14/0/0**〔12+2，含 F2 `ShowCreateTableCommand` 编译〕；checkstyle 0；import-gate 0；`CatalogFactory:51` 未改→iceberg 仍**不在** `SPI_READY_TYPES`。**新 1 D（[D-066]）；无新 DV**〔fork 复现+engine/show-create 现修皆消除偏差〕。**消解 T07 预登记 3 项**〔thrift 分叉/engine name/SHOW CREATE 解包〕；残留 T07 DV：内部 `TableType.PLUGIN_EXTERNAL_TABLE`〔user-visible engine/descriptor 已 parity〕/position_deletes 文案/serialized 字节潜伏（T05）。**live-e2e 未跑**（dormant，P6.8 兜底）。**下一 = P6.5-T07**〔parity 审计+DV 中央登记+对抗 parity wf〕。 @@ -246,8 +248,8 @@ | 类型 | 总数 | 最新条目 | 文档 | |---|---|---|---| -| **决策**（D-NNN） | 58 | D-058（最新，P6 iceberg 8 阶段拆分 / 单一翻闸在末）；D-057（P4 MINOR/NIT cleanup scope）；D-045/046/047（P5-B7 restore SHOW PARTITIONS 5 列 / SHOW CREATE LOCATION+PROPERTIES / Hybrid SPI）| [decisions-log.md](./decisions-log.md) | -| **偏差**（DV-NNN） | 37 | DV-036/037（P6-C2 HDFS XML：DLF / disable-cache）；DV-024（P5-B4 BE 描述符 SCHEMA_TABLE→HIVE_TABLE）；DV-023（RFC §10 E7 被 P5-B4 取代）| [deviations-log.md](./deviations-log.md) | +| **决策**（D-NNN） | 73 | D-073（最新，P6.6-C3b-core impl-recon + v3-lineage carrier Option A）；D-072（C3b-core 全量设计 + commit-bridge）；P6 翻闸期大量决策记于 HANDOFF/git | [decisions-log.md](./decisions-log.md) | +| **偏差**（DV-NNN） | 49 | DV-049（P6.5-T07 sys-table perf-cosmetic 批）；DV-048（correctness-bearing）；DV-045/046/047（P6.4-T08 rewrite blocker / auth / perf）| [deviations-log.md](./deviations-log.md) | | **风险**（R-NNN） | 14 | R-014（thrift sink 选择灵活性） | [risks.md](./risks.md) | --- @@ -256,9 +258,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：**P6 阶段拆分设计 + doc-sync（0 产线代码）** —— brainstorm 定 **方案 A / 8 阶段 / 单一翻闸在末**（用户签 [D-058]），firsthand code-grounded 核实翻闸全有或全无（`CatalogFactory:104-113`）、连接器 flavor dispatch 2 缺口（`dlf`/`s3tables`）、3 缺失 SPI（`ConnectorProcedureOps`/`ConnectorCredentials`/写路径 RFC）、iceberg metastore-props 已在 fe-core。产 [tasks/P6-iceberg-migration.md](./tasks/P6-iceberg-migration.md)（phase-level plan）+ 同步刷新 HANDOFF（覆盖）+ 本 PROGRESS（§一/二/三/四/六/七）+ connectors/iceberg + decisions-log D-058；并对账 stale 的 P5/P3b 状态（#64653 / #64655 已合入）。 -- **下一个 session 应做**：**P6.1 元数据 recon + 逐 task 拆解（P6-Tnn）+ 实现**——连接器地基 + 普通读元数据 + 7 flavor（含 port DLF 4-file 子树 + wire S3Tables SDK）。**起步无硬前置**。开场读 master plan §3.7 + [tasks/P6](./tasks/P6-iceberg-migration.md) P6.1 块 + 对照 legacy `datasource/iceberg/` flavor + metadata 读路径真实代码。**切忌在 P6.1–P6.5 任何阶段翻闸**（全有或全无）。 -- **是否需要 handoff**：**是**——本场已**覆盖** rewrite [HANDOFF.md](./HANDOFF.md)（P6.1 元数据 recon 为下一步 + P6 阶段拆分 spec + [D-058]）。 +- **本 session 已完成**：**P6 完成 + 合入实证 + doc-sync（0 产线代码）** —— 核实 git（#64688 `8b391c7459d` 已合入 `branch-catalog-spi`、工作树干净、新分支 `catalog-spi-11-hive`）+ 实证 fe-core `datasource/iceberg/` 尚存 23 个 HMS-iceberg 支撑类（随 P7 阶段四删）；覆写 [HANDOFF.md](./HANDOFF.md) 为 P7 起步 + 同步刷新本 PROGRESS（§一/二/三/四/六/七）+ connectors/iceberg（✅ 合入）+ connectors/hive（🚧 P7）+ tasks/P6·P6.6（✅ 合入）。 +- **下一个 session 应做**：**P7 hive (+HMS) 迁移启动**——建 `tasks/P7-hive-migration.md` 阶段拆分 spec + code-grounded recon，起步 P7.1 HiveMetadataOps 全功能搬迁。开场读 master plan §3.8 + connectors/hive.md + 对照 fe-core `datasource/hive/` 真实代码（信控制流不信注释）。**R-002 ACID 写路径须专门集成测试作 gate。** +- **是否需要 handoff**：**是**——本场已**覆盖** rewrite [HANDOFF.md](./HANDOFF.md)（P6 合入实证 + P7 起步计划 + 23 HMS-iceberg 支撑类遗留）。 - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/connectors/hive.md b/plan-doc/connectors/hive.md index b3fbd2c5a173ae..e5aa5796c2393f 100644 --- a/plan-doc/connectors/hive.md +++ b/plan-doc/connectors/hive.md @@ -10,8 +10,8 @@ | **fe-connector 模块** | `fe/fe-connector/fe-connector-hive/` + `fe/fe-connector/fe-connector-hms/`（共享库）| | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/hive/` | | **共享依赖** | 自身 `fe-connector-hms`；被 hudi/iceberg-HMS/paimon-HMS 依赖 | -| **计划迁移阶段** | **P7**（最复杂，6 周）| -| **当前状态** | ⏸ 未启动 | +| **计划迁移阶段** | **P7**（最复杂，6 周；**当前活跃阶段**，起步 P7.1）| +| **当前状态** | 🚧 P7 启动中（iceberg P6 已合入 #64688 → hive 为下一个活跃迁移目标）；下一 = **P7.1 HiveMetadataOps 全功能搬迁** | | **完成度** | 10%（hive 20% + hms 共享库已立） | | **主 owner** | TBD | @@ -90,6 +90,11 @@ ## 进度日志 +### 2026-07-05（P7 启动 = 当前活跃迁移目标） +- **iceberg P6 已 squash-合入 `branch-catalog-spi`（#64688 `8b391c7459d`）→ hive 成为下一个活跃迁移目标**。工作分支 `catalog-spi-11-hive`。 +- **下个 session 起步**：建 `tasks/P7-hive-migration.md` 阶段拆分 spec + code-grounded recon；起步 P7.1 HiveMetadataOps 全功能搬迁。权威计划 master plan §3.8 + 本文 §子阶段。**R-002 ACID 写路径（P7.3）= 项目最大风险，须专门集成测试作 gate。** +- **P7 连带清理**：删 fe-core `datasource/hive/`（P7.5）+ 23 个 HMS-iceberg 支撑类 + `datasource/hudi/`（阶段四）；hudi 批 E（live cutover）并入 P7。 + ### 2026-05-24 - 跟踪文件建立。当前最复杂的连接器；R-002（ACID 数据不一致）是项目最大风险。 - 注意：hive 是 hudi/iceberg/paimon 共同的底座（通过 HMS 共享库），P7 启动 = 项目核心冲刺。 diff --git a/plan-doc/connectors/iceberg.md b/plan-doc/connectors/iceberg.md index e016af07922341..4566529c32888d 100644 --- a/plan-doc/connectors/iceberg.md +++ b/plan-doc/connectors/iceberg.md @@ -10,9 +10,9 @@ | **fe-connector 模块** | `fe/fe-connector/fe-connector-iceberg/` | | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/` | | **共享依赖** | `fe-connector-hms`（iceberg-HMS-flavor 用） | -| **计划迁移阶段** | **P6**（最大阶段，5 周）| -| **当前状态** | 🟢 **P6.1 DONE + P6.2 DONE + P6.3 DONE + P6.4 DONE（P6.3/P6.4 各 T01~T09 ✅）**：P6.1〔T01–T10〕7-flavor 装配 + 读元数据 parity + per-flavor 校验 + metastore 模块拆分；P6.2〔T01–T11〕scan+MVCC+cache+vended（UT 278/0/1）；**P6.3 写路径 RFC ✅ + T01~T09 = P6.3 DONE**（框架统一·SPI 收口 + jdbc planWrite + `IcebergConnectorTransaction` 骨架·op 选择·WriterHelper·begin guards + commit 校验套件·O5-2·V3 DV + sink 统一〔INSERT/OVERWRITE〕·写排序 SPI + DELETE/MERGE sink 方言〔T07a〕+ O5-2 生产半〔T07b〕+ 通用 `RowLevelDmlCommand` 壳·注册表·6 派发重接·O5-2 dormant〔T07c，commit `a61cd9262b9`〕+ parity 审计·deviation 中央登记 DV-041..044〔T08：审计 wf 40→20 confirmed→11 gap-fill〕+ 收口汇总设计·faithfulness 对抗验证〔T09，`designs/P6.3-T09-iceberg-write-summary-design.md`〕；iceberg UT **389/0/1** + fe-core 30/0）。**P6.4 procedures DONE**（T01 设计+三签字 [D-062] + T02 `ConnectorProcedureOps` SPI 骨架 ✅ + T03 ✅ 2026-06-24 base+factory + dispatch 骨架〔`runInAuthScope` body 进 `executeAuthenticated`〕，§4=4-A 连接器自校验、去死 `table` 参；arg 框架〔`NamedArguments`/`ArgumentParsers`/`ArgumentParser`〕**移 `fe-foundation` 共享**〔引擎+连接器一份，`validate` 抛 unchecked `IllegalArgumentException` 两侧 re-wrap〕；iceberg UT **401/0/1**，faithfulness wf 4→0 confirmed；**T04 ✅ 2026-06-24 港 8 pure-SDK 体 + `RewriteManifestExecutor` 接 factory switch**〔逐字 bug-for-bug；**必须改** `executeAction(Table,ConnectorSession)` 透会话 TZ〔新 `IcebergTimeUtils.msTimeStringToLong`〕；cache 失效搬 dispatch 级；iceberg UT **444/0/1**，faithfulness wf 1→0 confirmed/1 refuted+0 critic gaps〕；**T05 ✅ 2026-06-24 `rewrite_data_files` 规划半 → `connector.iceberg.rewrite`**〔`RewriteDataFilePlanner`/`RewriteDataGroup`/`RewriteResult` 3 类港；WHERE 走 `IcebergPredicateConverter` **conflict-mode**〔用户签字 Option A，DV-T05r-where：不可转节点静默丢=过宽 vs legacy fail-loud〕；执行半+事务半+bind=T06；iceberg UT **467/0/1**，faithfulness wf 8→0 confirmed/0 critic gaps〕；**T06 🟢 2026-06-24（用户裁 Option 1）** ① 连接器事务 `WriteOperation.REWRITE` 变体〔净 0 新 verb，忠实港 legacy rewrite commit·OCC `validateFromSnapshot`·count 访问器，dormant〕实现 + iceberg UT **475/0/1**，**②③④（执行半↔规划接线 + 文件级扫描范围 + bind/executor + multi-sink-per-txn）= R-B 推后专门写路径 RFC**〔5-reader recon 证伪设计 §5「pinned snapshot+WHERE 重规划」前提：over-scan→破正确性 + SPI 模块边界 fe-core 够不到连接器 `RewriteDataGroup` + 侧信道翻闸后死；翻闸阻塞 DV-T06r-rb〕，faithfulness wf 4→0 confirmed〔mutation-check 实证 OCC 测不 pin 显式 `validateFromSnapshot`→诚实修正不 overclaim〕；**T07 ✅ 2026-06-24 dispatch rewire**〔新 fe-core adapter `ConnectorExecuteAction`，`ExecuteActionFactory` 加 PluginDriven→`getProcedureOps()` dormant 分支·保 legacy；`run()` 不变=legacy byte-parity；engine 保 priv+`CommonResultSet` wrap+logRefresh / connector 保 arg+body；`DorisConnectorException`→plain `UserException` re-wrap；WHERE 拒 DV-T07-where；fe-core `ConnectorExecuteActionTest` 13/0 + `NereidsParserTest` 73/0；faithfulness wf 5-finder 0 finding，critic 6 类→2 当场修〔空-rows→null 形状+priv 测断言〕+2 DV〔DV-T07-name-order/exc-contract〕+2 note；0 连接器/BE/pom/CatalogFactory 改〕；**T08 ✅ 2026-06-24 parity-UT 审计 + gap-fill + DV 中央登记**〔对抗 byte-parity 审计 wf 12 area finder = 28 confirmed utGap + 2 newDeviation〔**所有前向 DV accurate=True**〕+ critic 8 跨切；→ **20 gap-fill UT**〔schema 完整性·error 串字节·bug-for-bug execute 路·auth/cache 接缝；连接器 **494/0/1** + fe-core 双测〕；2 测模型坑实证修〔expire `[0,0,0,0,2,0]` / rewrite spec_id 漏 validate〕；**DV-045〔🔴 BLOCKER〕/046〔correctness-bearing〕/047〔perf-cosmetic〕中央登记 44→47**；0 产品码〔唯 1 行注释〕〕；**T09 ✅ 2026-06-24 收口汇总设计 `designs/P6.4-T09-procedure-summary-design.md`〔7 节〕+ faithfulness 对抗验证 `wf_986bd3db-68b`〔7 verifier refute-by-default + critic→3 真错+1 矛盾修：BaseExecuteAction 非字节不变〔arg-move 改〕/「九 commit 待 push」实为二〔仅 T07/T08 未推〕/`getFileScanTasksFromContext:498`→`:492`/`:929`/§3 NamedArguments fe-foundation〕 + gate 重跑实证 iceberg 494/0/1 + fe-core 14/0 = P6.4 DONE**，仍 behind gate）。**翻闸阻塞 = [DV-038]（field-id BE DCHECK）+ [DV-041]（写路径 visitPhysicalConnectorTableSink 缺合成列物化，DV-038 同主题新面）+ [DV-045]（rewrite 执行半接线 R-B，原 DV-T06r-rb）**，P6.6 前必修 | -| **完成度** | ~93%（P6.1+P6.2+P6.3+P6.4 实现完〔T01–T09〕；**P6.5 sys-table ✅ DONE**〔T01 ✅ recon+设计+用户二签字，仅系统表、净 0 新 SPI、fe-core 零改动；**T02 ✅** `IcebergTableHandle` sys 变体〔`sysTableName` 非 transient + `forSystemTable` 保留 snapshot pin + 9 UT〕；**T03 ✅ 2026-06-24** `IcebergConnectorMetadata` 2 override〔`listSupportedSysTables`〔`MetadataTableType.values()` 去 position_deletes 小写 unmodifiable〕+ `getSysTableHandle`〔保留 pin + **LAZY 无 catalog 往返**，[D-063]〕+ 11 UT；**514/0/1**〕；**T04 ✅ 2026-06-24** `getTableSchema`/`getColumnHandles` sys 分支〔新 `loadSysTable`→`MetadataTableUtils.createMetadataTableInstance`〔决策 B/偏差③〕+ meta-table schema 经 `parseSchema` 透 mapping flag〔偏差⑤〕+ 3-arg @snapshot sys 短路 + [D-064]；7 UT 含 seam-identity；**521/0/1**〕；**T05 ✅ 2026-06-24** `IcebergScanPlanProvider`/`IcebergScanRange` sys split 路〔`planSystemTableScan`〔`resolveSysTable` 元数据表 + 复用 `buildScan` time-travel → `serializeToBase64(FileScanTask)`〕 + carrier `serializedSplit` + `populateRangeParams` sys 早返〔serialized_split+FORMAT_JNI+row_count=-1，镜像 legacy `setIcebergParams`〕，[D-065]；6 UT〔含 deserialize-round-trip 经 BE 路 + 4-变异 mutation-check〕；潜伏=serialized 字节 BE 跨版本 P6.8 e2e 兜底；**527/0/1**〕；**T06 ✅ 2026-06-25** `buildTableDescriptor` hms↔iceberg 分叉〔连接器，覆 base+sys，BE 无感纯 FE parity〕 + `getScanNodeProperties` sys 收口〔[D-065]，跳 dict+ppk 修潜伏崩溃〕 + fe-core engine-name/SHOW-CREATE parity〔[D-066]，用户签字〕；3 mutation-check 红证；**532/0/1**〕；**T07 ✅ 2026-06-25** 对抗审计揭 2 主动偏差→用户裁现修〔A sys 时间旅行 guard connector-capability-aware〔新 SPI `supportsSystemTableTimeTravel`〕+ B hms `equalsIgnoreCase`，[D-067]〕 + **17 gap-fill UT〔T07 9 + 续 8〕** + DV-048/049；**T07-续**〔残留 8 gap-fill：sys getMysqlType/engine/engineTableTypeName · position_deletes 通用 not-found · 非注册不变式〔无 `@SerializedName`/无 `$`-key〕· guard-message 顺序 · sys predicate **作 residual 带 BE**〔实证纠 audit spec：元数据列谓词非 FE 行裁〕· sys split `/dummyPath` · WHY-comment〕；**543/0/1** + `PluginDrivenSysTableTest` 10/0 + guard 8/0〕；**T08 ✅** 收口汇总设计 `designs/P6.5-T08-systable-summary-design.md` + faithfulness 对抗验证 wf〔18/18 confirmed、0 critic fix；critic 经 iceberg 1.10.1 bytecode 独立证实 test-6 residual〕+ gate 重跑 ⇒ **P6.5 DONE**〕；**剩** P6.4 `rewrite_data_files` 执行半 ②③④〔R-B 写路径 RFC=DV-045，翻闸阻塞〕/ P6.6 翻闸〔+ sys 时间旅行 query→handle pin threading〕/ P6.7 删 legacy / P6.8 回归）| +| **计划迁移阶段** | **P6**（最大阶段；迁移+翻闸+删原生子系统已合入 `branch-catalog-spi` #64688 `8b391c7459d`）| +| **当前状态** | ✅ 迁移 + 翻闸 + 删 legacy 原生子系统已合入 `branch-catalog-spi`（**#64688** `8b391c7459d`）——iceberg 入 `SPI_READY_TYPES`，FE 走 SPI 路径；P6.1–P6.6 全阶段 + 属性/鉴权全归插件（S1–S10）一次性 squash。⚠️ **iceberg-on-HMS**（`type=hms` / `DlaType.ICEBERG`）仍走 fe-core，`datasource/iceberg/` 尚存 23 个支撑类，随 **P7 hive** 迁移删（阶段四，D5/Q3=B）| +| **完成度** | 100%（原生 iceberg 迁移全阶段完成并合入 #64688；剩 HMS-iceberg 支撑类清理归 P7 阶段四）| | **阶段拆分 spec** | [`tasks/P6-iceberg-migration.md`](../tasks/P6-iceberg-migration.md) | | **主 owner** | TBD | @@ -83,13 +83,17 @@ - 阶段 task：P6（待启动时建） - 决策：D-002, D-005, D-006 -- 偏差：[DV-038]（🔴 翻闸阻塞：GLOBAL_ROWID + getColumnHandles 共享 fe-core field-id 路径 BE DCHECK）、[DV-039]（parity-忠实 HIGH-MEDIUM）、[DV-040]（perf-cosmetic ~36 项批） +- 偏差：[DV-038]（GLOBAL_ROWID + getColumnHandles 共享 fe-core field-id 路径 BE DCHECK；翻闸已随 #64688 完成）、[DV-039]（parity-忠实 HIGH-MEDIUM）、[DV-040]（perf-cosmetic ~36 项批） - 风险：R-003（Procedure SPI 抽象失败）、R-004（classloader）、R-005（nereids 写命令耦合）、R-012（snapshotId 类型） --- ## 进度日志 +### 2026-07-05（P6 完成 + squash-合入 `branch-catalog-spi` #64688 `8b391c7459d`） +- **P6 iceberg 迁移 + 测试全部完成，一次性 squash 为 #64688**（685 文件，−23744 行）：原生 iceberg（元数据/scan/write/procedures/sys-tables/行级 DML）迁到 `fe-connector-iceberg` + 翻闸（入 `SPI_READY_TYPES`）+ GSON 迁移（老镜像→`PluginDriven*`）+ 属性/鉴权全归插件（S1–S10）+ 删 fe-core 原生子系统（`IcebergExternalCatalog`/`IcebergExternalTable`/`IcebergTransaction`/7 flavor/`broker`/`dlf`/`fileio`/`action`/`rewrite`/DML 执行器/planner sink 等）。 +- **遗留（P7 接手）**：fe-core `datasource/iceberg/` 尚存 23 个 HMS-iceberg 支撑类（`IcebergUtils`/`IcebergMetadataOps`/`source/IcebergScanNode`/`cache/`/`IcebergMvccSnapshot`/…），iceberg-on-HMS 走它们、随 P7 hive 迁移删（阶段四）。 + ### 2026-06-25（P6.6-C2 WS-SYNTH-READ：起步 recon 推翻 RFC BE/D7 + classifyColumn SPI 化，TDD+mutation） - **起步对抗 recon（独立读码 + 8-agent 对抗 wf `wf_9bf8730b-05b`/655k token，3 verdict 全 confirmed）证伪/重构 RFC §6.WS-SYNTH-READ，用户 AskUserQuestion 双裁（[D-069]）**：① **BE 已完整处理** SYNTHESIZED/GENERATED（`iceberg_reader.cpp:162-208`/`:444-489`，合成列 `continue` 不触达 `table_schema_change_helper.h` DCHECK；reader 由 `table_format_type=="iceberg"` 选与 FE 节点无关）→ **C2 零 BE 改**（RFC 的 BE「add_not_exist_children」过时）；② **D7 问错方向**——paimon 不触达 GLOBAL_ROWID（被 `MaterializeProbeVisitor.SUPPORT_RELATION_TYPES` 精确类白名单挡，非 classifyColumn），对 paimon 怎样都安全。 - **新挖 RFC 漏的两处翻闸前置项**：**[GAP-A→C5]** 翻闸后 iceberg 表类 `PluginDrivenMvccExternalTable`（与 paimon 同类）掉出 `MaterializeProbeVisitor` allowlist→lazy-top-N 静默失效，须 capability/engine 判别（非 class）；**[GAP-B→随翻闸]** 隐藏列注入（`ICEBERG_ROWID`+v3 row-lineage 现由 legacy `IcebergExternalTable.initSchema:297-301` 注入，翻闸后通用路只从连接器 native schema 建、不注入）。 diff --git a/plan-doc/tasks/P6-iceberg-migration.md b/plan-doc/tasks/P6-iceberg-migration.md index 6d547bc8bdd3af..838307d6762055 100644 --- a/plan-doc/tasks/P6-iceberg-migration.md +++ b/plan-doc/tasks/P6-iceberg-migration.md @@ -8,10 +8,10 @@ ## 元信息 -- **状态**：🟢 规划完成（本 session 2026-06-21 brainstorm，用户签 **方案 A / 8 阶段 / 单一翻闸**）；**下一步 = P6.1 启动**（其 code-grounded recon + 逐 task 拆解）。 +- **状态**：✅ **COMPLETE** —— P6 iceberg 全阶段（P6.1–P6.6 迁移 + 翻闸 + GSON 迁移 + 属性/鉴权全归插件 S1–S10 + 删 fe-core 原生子系统）完成并 squash-合入 `branch-catalog-spi`（**#64688** `8b391c7459d`）。**下一步 = P7 hive/HMS 迁移**（工作分支 `catalog-spi-11-hive`；master plan §3.8 + connectors/hive.md）。遗留：fe-core `datasource/iceberg/` 尚存 23 个 HMS-iceberg 支撑类，随 P7 阶段四删。 - **启动日期**：2026-06-21（阶段拆分设计） - **目标策略**：**先在 `fe-connector-iceberg` 实现完整 iceberg 能力（P6.1–P6.5，全程不翻闸）→ P6.6 一次性翻闸 → P6.7 删 legacy → P6.8 回归**。用户定调（2026-06-21）：翻闸是 per-catalog-type 全有或全无（`CatalogFactory:104-113`），故不做中途/混合翻闸。 -- **工作分支**：`catalog-spi-10-iceberg`（off `branch-catalog-spi` @ `e5959e1b53d`）。PR base = `branch-catalog-spi`，squash 合并（mirror P5-T29 #64653 / P3b #64655）。 +- **工作分支**：`catalog-spi-10-iceberg`（off `branch-catalog-spi` @ `e5959e1b53d`）。PR base = `branch-catalog-spi`，squash 合并（mirror P5-T29 #64653 / P3b #64655）。**✅ 已 squash-合入 #64688 `8b391c7459d`。** - **阻塞**：P6.1 起步**无硬前置**（P3b kerberos 收口 + docker e2e 已清）。子阶段内前置见下 §阶段内前置。 - **阻塞下游**：P7 hive(+HMS) 复用 P6 的写路径 SPI / procedure SPI / DLA 模型；P8 收尾（删 `SPI_READY_TYPES`、删全部反向 instanceof）。 - **主 owner**：@morningman / TBD diff --git a/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md b/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md index 577318fe77847b..0947f1f06f41cd 100644 --- a/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md +++ b/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md @@ -1,7 +1,7 @@ # P6.6 — Iceberg 翻闸前修复任务清单（clean-room 对抗 review 产物） > **来源**：`plan-doc/reviews/P6.6-iceberg-cleanroom-adversarial-review-2026-06-28.md`（wf_4c00d628-5bd，223 agents，每发现独立 refute 验证）。 -> **状态**：⛔ **翻闸 BLOCKED** —— review 推翻了「翻闸代码基本完成」的旧结论，发现 **2 blocker + 11 high + 11 medium + 25 low + 18 info**。**P0 全清 + P1 关键关 + flip-gated e2e 过**之后才能二签翻闸。 +> **状态**：✅ **翻闸 COMPLETE** —— 所有 blocker/high + 关键项已关闭，iceberg 翻闸 + 删 legacy 已 squash-合入 `branch-catalog-spi`（**#64688** `8b391c7459d`）。（下方历史 blocker 清单 = 已归档记录，勿再当活 gate。） > **跟踪方式**：本文件 = master checkbox 清单（逐条 ID 对齐 review 报告，便于"逐步处理"）。每条任务被认领时新建 `plan-doc/tasks/designs/P6.6-FIX---design.md`（沿用 P4-T06e-FIX-* 范式：design→impl→test→独立 commit），并回填本表「设计文档」列 + 勾选状态。`HANDOFF.md` 只记**当前**高层进度（哪一 tier 在做 / 已完成 / 下一步），每 session 覆盖式更新。 > **状态记号**：`☐ TODO` / `◐ WIP` / `☑ DONE` / `⤬ WONTFIX(需用户裁)`。 > **铁律提醒**：① fe-core 不得新增 `if(iceberg)`/`instanceof Iceberg*`/引擎名判别（见 HANDOFF §铁律，新 seam 走中立 SPI / ConnectorCapability）；② 几乎所有发现 **flip-gated e2e 未跑**，每条修完的「验收」含 e2e 项，翻闸后才能真跑（勿谎称已验）；③ 动码前先 grep 全调用方 recon，HANDOFF/review 的行号可能过时。 From c3b450463341441485062eac70d9c819fff948dd Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 5 Jul 2026 23:51:30 +0800 Subject: [PATCH 002/334] [doc](catalog) P7 hive: code-grounded recon + phase-split spec 10-agent code-grounded recon of fe-core datasource/hive (52 files) + a supplemental type-coupling pass. Produces tasks/P7-hive-migration.md (P7.1-P7.5 split, old->new mapping, cutover mechanics, cross-connector deletion ordering, SPI gaps, 8 open decisions) and syncs HANDOFF / PROGRESS / connectors/hive.md. Key facts corrected vs stale plan: reverse instanceof 31->85 (33 files), HMSTransaction 1866->1895, HMSExternalTable 1293->1332. Coverage critic closed an instanceof-only blind spot (CatalogFactory:134, GsonUtils 366/447/471 compat, HudiUtils/IcebergHMSSource type coupling). Confirmed via decisions-log that iceberg/hudi-on-HMS ownership is already settled (D-020 per-table SPI provider delegation + D-019 hudi live cutover into P7) - not re-litigated. No product code, no new decisions. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 33 +++--- plan-doc/PROGRESS.md | 17 +-- plan-doc/connectors/hive.md | 17 ++- plan-doc/tasks/P7-hive-migration.md | 167 ++++++++++++++++++++++++++++ 4 files changed, 201 insertions(+), 33 deletions(-) create mode 100644 plan-doc/tasks/P7-hive-migration.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 87d8cf1e9663cf..254e8569928792 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,9 @@ --- -# 🎯 当前状态（2026-07-05）= **P6 iceberg 迁移 + 测试全部完成，已 squash-合入 `branch-catalog-spi`（#64688 `8b391c7459d`）；下一阶段 = P7 hive/HMS** +# 🎯 当前状态（2026-07-05 下午）= **P7 hive/HMS 迁移已启动：10-agent code-grounded recon + 阶段拆分 spec 完成；下一步 = P7.1 HiveMetadataOps 实现** + +> **本轮（recon + spec session）做了什么**：跑 10-agent code-grounded recon（`.claude/wf-p7-hive-recon.js` + 补充 type-coupling recon，~1.3M token，0 error）核清 fe-core `datasource/hive/` **52 文件**全貌、ACID 写路径、event pipeline、DLA 三分流、**85 处反向 instanceof/33 文件**、跨连接器耦合、翻闸机制。产出 **`tasks/P7-hive-migration.md`**（P7.1–P7.5 阶段拆分 spec + old→new 映射 + SPI 缺口 + 删除排序 + 8 条开放决策）。校正了 plan/HANDOFF 里过时数字（instanceof 31→85、HMSTransaction 1866→1895、HMSExternalTable 1293→1332）。**关键澄清**：recon 标"最大未知 = iceberg/hudi-on-HMS 归属"实为**已定**（D-020 per-table SPI provider 委派 + D-019 hudi live cutover 并入 P7）——**勿重议**。**无代码改动、无新决策**（仅文档）。 > **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： > P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 @@ -27,28 +29,19 @@ --- -# 🚀 下个 session 任务 = **P7 hive (+HMS) 迁移（最复杂、最后做的连接器）** - -> **权威计划**：[`00-connector-migration-master-plan.md` §3.8](./00-connector-migration-master-plan.md)（P7.1–P7.5 子阶段 + 风险）+ [`connectors/hive.md`](./connectors/hive.md)（子阶段/SPI 缺口/特殊性）。**单连接器 13 步 playbook = master plan §4。** -> **模块已就绪**：`fe-connector-hive` + `fe-connector-hms`（后者是 hive/hudi/iceberg-HMS/paimon-HMS 的**共享元存储库**，P3/P5/P6 已在用、稳定）。 - -## P7 为什么是复杂度顶点（master plan §3.8） +# 🚀 下个 session 任务 = **P7.1 实现（HiveMetadataOps → HiveConnectorMetadata + HmsClient 写方法）** -- **HMS 是共同元数据后端**：hive、hudi、iceberg-HMS-flavor、paimon-HMS-flavor 共用。`fe-connector-hms` 须持续稳定。 -- **21 个 metastore event 类 + `MetastoreEventsProcessor`** → 用 P0 已建的 `ConnectorMetaInvalidator` 承接，event pipeline 整体搬进 `fe-connector-hms`（决策 D4）。 -- **`HMSTransaction`（1866 行）+ `HiveTransactionMgr`** —— ACID 写路径，**最难**，须重写。**R-002（ACID 数据不一致，High）= 项目最大风险**；gate = 专门的 ACID 集成测试套件。 -- **`HMSExternalTable`（1293 行）** —— hive/hudi/iceberg 三种 `dlaType` 分流；由 DLA 模型吸收（决策 D5：`ConnectorTableSchema.tableFormatType` 区分）。 -- **31 处反向 `instanceof HMSExternalCatalog/HMSExternalTable`** 散落热路径（`nereids/glue/translator`、`tablefunction/MetadataGenerator`、`AnalyzeTableCommand`、`ShowPartitionsCommand`）。 -- **子阶段（见 connectors/hive.md §子阶段）**：P7.1 HiveMetadataOps 全功能搬迁（2 周）→ P7.2 event pipeline 搬 + `ConnectorMetaInvalidator`（1.5 周）→ P7.3 HMSTransaction+HiveTransactionMgr ACID 写路径（2 周）→ P7.4 DLA 分流改造（0.5 周）→ P7.5 删 fe-core `datasource/hive/` + 31 instanceof（0.5 周）。 -- **连带清理**：P7 收尾要一并删掉上面那 23 个 HMS-iceberg 支撑类 + fe-core `datasource/hudi/`；**hudi 批 E（live cutover）并入 P7**（PROGRESS §一 P3 note）。 -- **Kerberos UGI**：`fe-kerberos`（P3b #64655 已收口）+ `ConnectorContext.executeAuthenticated` 已支持，P7 逐条审查复用（R-004）。 +> **权威计划**：**`tasks/P7-hive-migration.md`**（本轮新建的阶段拆分 spec，含 P7.1–P7.5 + old→new 映射 + SPI 缺口 + 翻闸机制 + 删除排序 + 开放决策）+ [`00-connector-migration-master-plan.md` §3.8](./00-connector-migration-master-plan.md) + [`connectors/hive.md`](./connectors/hive.md)。**单连接器 13 步 playbook = master plan §4。** +> **模块已就绪**：`fe-connector-hive`（"hms" 网关，只读 scan 已立、DDL/txn/stats override=0）+ `fe-connector-hms`（共享读元存储库，无写/txn/lock/col-stats）。 -## P7 起步要点（下个 session 开场） +## P7.1 起步要点（下个 session 开场） -1. **先读**：master plan §3.8 + `connectors/hive.md` + `git show master:` 对照 fe-core `datasource/hive/` 真实代码 recon（**HANDOFF/计划的行号、依赖名、缺口清单可能过时；信控制流不信注释**）。 -2. **建 P7 阶段拆分 spec** `tasks/P7-hive-migration.md`（phase-level plan，镜像 `tasks/P6-iceberg-migration.md`）+ 起步 code-grounded recon（AGENT-PLAYBOOK §7）。 -3. **决策已定（master plan §5，勿重议）**：D4=event pipeline 放 `fe-connector-hms`；D5=DLA 用 `tableFormatType` 区分；写路径 SPI / procedure SPI / DLA 模型复用 P6 已建成果。 -4. **纪律**：每阶段独立 commit + build + test + checkstyle 0 + import-gate 净；**每一轮完成后即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。ACID 写路径必须有独立集成测试作 gate（R-002）。 +1. **先读** `tasks/P7-hive-migration.md`（全）+ 对照 HEAD 真实代码 recon `HiveMetadataOps.java`(425) 全 public 面 + `HiveConnectorMetadata.java`（现 override 0）+ `fe-connector-hms/HmsClient`/`ThriftHmsClient`（**信控制流不信本 spec 行号**）。 +2. **P7.1 范围**：把 `HiveMetadataOps` 的 create/drop db+table、rename、truncate、add/drop partition、col-stats 写回搬进 `HiveConnectorMetadata` + `HmsClient`（写方法留 HmsClient / hive-only 写子接口，别撑大 hudi 共享面）；**no-property-parsing**：file_format/owner/bucket/DLF-guard 全移插件（先确认 `ConnectorCreateTableRequest` 富度，不够就扩）。新增 `ConnectorTableOps.truncateTable`（default throw）。建 `P7.1-Tnn` 逐 task 拆解追加进 spec 文件。 +3. **勿重议的已定决策**（引 decisions-log，spec §已定架构）：**D-004** event→fe-connector-hms；**D-005** tableFormatType 区分（fe-core 热路径不读）；**D-020** per-table SPI provider（hive 网关委派 -iceberg/-hudi）；**D-019** hudi live cutover + 删 legacy 并入 P7；事务桥接 `PluginDrivenTransaction`。 +4. **开放决策（OQ-*）**：到各子阶段 recon 后再**用户签字**（先中文讲背景+示例+推荐、不引任务代号）；本 spec §开放决策已附推荐。 +5. **纪律**：每子阶段独立 commit + build + test + checkstyle 0 + import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。**P7.3 ACID 写路径必须有独立集成测试作 gate（R-002 项目最大风险）**。 +6. **删除排序（最硬约束，spec §跨连接器删除排序）**：`datasource/hive/` 删不掉，直到 `HudiUtils`(5 方法)/`HudiExternalMetaCache`/`HudiScanNode`(extends HiveScanNode)/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 全 retype 到 generic（P7.4）。 --- diff --git a/plan-doc/PROGRESS.md b/plan-doc/PROGRESS.md index 4131fb23b8dee2..1b13aedbe8c880 100644 --- a/plan-doc/PROGRESS.md +++ b/plan-doc/PROGRESS.md @@ -1,6 +1,6 @@ # 📊 项目进度仪表盘 -> 最后更新：**2026-07-05** | 当前阶段：**P7 hive (+HMS) 迁移（启动中）** —— 工作分支 `catalog-spi-11-hive`。**P0–P6 + P3b 全部合入 `branch-catalog-spi`**：P0 #63582 / P1 #63641 / P2 #64096 / P3 hudi(hybrid) #64143 / P4 #64300 / P5 #64446+#64653 / P3b #64655 / **P6 iceberg 迁移+翻闸+删 legacy #64688 `8b391c7459d`**。下一 = **P7.1 HiveMetadataOps 全功能搬迁**（master plan §3.8 + connectors/hive.md）。| 项目总进度：**~64%**（按 §一 进度条加权：P0+P1+P2+P4+P5+P6 满 + P3 hybrid 45%，约 15.9/25 周） +> 最后更新：**2026-07-05（下午）** | 当前阶段：**P7 hive (+HMS) 迁移（进行中：recon + 阶段拆分 spec 完成）** —— 工作分支 `catalog-spi-11-hive`。**P0–P6 + P3b 全部合入 `branch-catalog-spi`**：P0 #63582 / P1 #63641 / P2 #64096 / P3 hudi(hybrid) #64143 / P4 #64300 / P5 #64446+#64653 / P3b #64655 / **P6 iceberg 迁移+翻闸+删 legacy #64688 `8b391c7459d`**。本轮产出 **`tasks/P7-hive-migration.md`**（10-agent code-grounded recon + P7.1–P7.5 阶段拆分）。下一 = **P7.1 HiveMetadataOps 实现**（→ HiveConnectorMetadata + HmsClient 写方法）。| 项目总进度：**~64%**（P0+P1+P2+P4+P5+P6 满 + P3 hybrid 45%，约 15.9/25 周） > [README](./README.md) · [Master Plan](./00-connector-migration-master-plan.md) · [SPI RFC](./01-spi-extensions-rfc.md) · [Decisions](./decisions-log.md) · [Deviations](./deviations-log.md) · [Risks](./risks.md) · [Agent Playbook](./AGENT-PLAYBOOK.md) · [Handoff](./HANDOFF.md) --- @@ -16,7 +16,7 @@ | **P4** | maxcompute 迁移 | 2 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（**#64253** T01–T06 适配+翻闸 + **#64300** T07–T09 删 legacy/odps-free；含 #64119 校验迁移）| [tasks/P4](./tasks/P4-maxcompute-migration.md) | | **P5** | paimon 迁移 | 3 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（迁移+翻闸 **#64446** + 删 legacy/maven **#64653** `d59ed2f96d9`；B9 回归用户 docker 覆盖）| [tasks/P5](./tasks/P5-paimon-migration.md) | | **P6** | iceberg 迁移 | 5 周 | ▰▰▰▰▰▰▰▰▰▰ 100% | ✅ 完成并合入 `branch-catalog-spi`（迁移+翻闸+删 legacy 原生子系统 **#64688** `8b391c7459d`；P6.1–P6.6 全阶段 + 属性/鉴权全归插件 S1–S10 一次性 squash，685 文件 −23744 行）。⚠️ fe-core `datasource/iceberg/` 尚存 23 个 HMS-iceberg 支撑类，随 P7 hive 迁移删（阶段四）| [tasks/P6](./tasks/P6-iceberg-migration.md) | -| **P7** | hive (+HMS) 迁移 | 6 周 | ▰▱▱▱▱▱▱▱▱▱ 5% | 🚧 启动中（当前活跃阶段；起步 P7.1 HiveMetadataOps）| [connectors/hive.md](./connectors/hive.md) | +| **P7** | hive (+HMS) 迁移 | 6 周 | ▰▱▱▱▱▱▱▱▱▱ 8% | 🚧 进行中（recon + 阶段拆分 spec `tasks/P7-hive-migration.md` 立；下一 = P7.1 实现）| [tasks/P7](./tasks/P7-hive-migration.md) · [connectors/hive.md](./connectors/hive.md) | | P8 | 收尾清理 | 2 周 | ▱▱▱▱▱▱▱▱▱▱ 0% | ⏸ 待启动 | — | **全局进度：~64%**（25 周计划中已完成约 15.9 周：P0+P1+P2+P4+P5+P6 满 + P3 hybrid 45%；按 §一 进度条加权） @@ -36,7 +36,7 @@ | maxcompute | ✅ | ✅ 100% | ✅ **已合入 #64253** | ✅ **#64300 已删** | ✅ 0/0 | **100%** | [详情](./connectors/maxcompute.md) | | paimon | ✅ | ✅ 100% | ✅ **已入 SPI_READY_TYPES** | ✅ **#64653 已删** | ✅ 热区已清 | **100%** | [详情](./connectors/paimon.md) | | iceberg | ✅ | ✅ 100% | ✅ **已入 SPI_READY_TYPES**（#64688）| ✅ **#64688 已删原生子系统**（HMS-iceberg 23 支撑类随 P7 删）| ✅ 0/0（原生反向 instanceof 已清）| **100%** | [详情](./connectors/iceberg.md) | -| hive (+hms) | 🟡 | 🟥 20% | ❌ | ❌ | 0/31 | **🚧 P7 启动中** | [详情](./connectors/hive.md) | +| hive (+hms) | 🟡 | 🟥 22% | ❌ | ❌ | 0/85 | **🚧 P7 进行中（recon+spec 立，起步 P7.1）** | [详情](./connectors/hive.md) | --- @@ -48,7 +48,8 @@ > 策略 = **先在 `fe-connector-hive`/`fe-connector-hms` 实现完整能力 → 翻闸（hms 入 `SPI_READY_TYPES`）→ 删 fe-core legacy（含 23 个 HMS-iceberg 支撑类 + `datasource/hudi/`，阶段四）→ 回归**（复用 P5/P6 full-adopter 样板）。模块已就绪：`fe-connector-hms` 是 hive/hudi/iceberg-HMS/paimon-HMS 共享元存储库（P3/P5/P6 已在用、稳定）。 > -> **子阶段（connectors/hive.md §子阶段）**：P7.1 HiveMetadataOps 全功能搬迁（2 周）→ P7.2 event pipeline 搬 + `ConnectorMetaInvalidator`（21 event 类，1.5 周）→ P7.3 `HMSTransaction`+`HiveTransactionMgr` ACID 写路径（2 周，**R-002 最大风险**，gate=专门 ACID 集成测试）→ P7.4 DLA 分流改造（`tableFormatType` 区分 hive/hudi/iceberg，D5，0.5 周）→ P7.5 删 fe-core `datasource/hive/` + 31 处反向 instanceof（0.5 周）。**起步无硬前置**（P0 已建 `ConnectorMetaInvalidator`、fe-kerberos P3b 已收口）。**下个 session 起步 = 建 `tasks/P7-hive-migration.md` 阶段拆分 spec + code-grounded recon（信控制流不信注释）。** +> **子阶段（权威 = `tasks/P7-hive-migration.md`）**：P7.1 HiveMetadataOps 全功能搬迁（2 周）→ P7.2 event pipeline 搬 + `ConnectorMetaInvalidator`（21 event 类，1.5 周）→ P7.3 `HMSTransaction`+`HiveTransactionMgr` ACID 写路径（2 周，**R-002 最大风险**，gate=专门 ACID 集成测试）→ P7.4 DLA 分流改造 + iceberg/hudi-on-HMS 委派（D-020）+ hudi live cutover（D-019）→ P7.5 删 fe-core `datasource/hive/`+`hudi/`+23 HMS-iceberg 类 + 85 处反向 instanceof。**起步无硬前置**（P0 已建 `ConnectorMetaInvalidator`、fe-kerberos P3b 已收口）。 +> **✅ 本轮完成 = 10-agent code-grounded recon + 阶段拆分 spec `tasks/P7-hive-migration.md`**（52 文件分类 + 翻闸机制 CatalogFactory:50/133 + GsonUtils:366/447/471 + 6 文件写路径 retype 链 + 跨连接器删除排序 + 8 条开放决策）。校正过时数字：instanceof 31→**85**、HMSTransaction 1866→**1895**、HMSExternalTable 1293→**1332**。**已定架构勿重议**：D-004/D-005/D-019/D-020 + 事务桥接。**下一 = P7.1 实现**（信控制流不信 spec 行号）。 ### P5 — paimon 迁移（✅ 全完成并合入：迁移+翻闸 #64446 + 删 legacy/maven #64653；B9 回归用户 docker 覆盖） @@ -156,6 +157,8 @@ > 倒序，新内容置顶；超过 14 天的条目移除（git log 保留历史）。 +- **2026-07-05（下午 · P7 启动：code-grounded recon + 阶段拆分 spec）** ✅（纯文档，0 产品码，0 新决策）。跑 **10-agent recon workflow**（`.claude/wf-p7-hive-recon.js`：9 维并行 readers + 1 coverage critic；补 1 路 type-coupling recon）≈1.3M token / 0 error，核清 fe-core `datasource/hive/` **52 文件**分类、ACID 写路径（`HMSTransaction` 1895）、event pipeline（21 类）、DLA 三分流（~19 分支点）、**85 处反向 instanceof/33 文件**、跨连接器耦合、翻闸机制。coverage critic 揪出 instanceof-only 扫描的**结构盲区**（`CatalogFactory:134 new HMSExternalCatalog`、`GsonUtils:366/447/471` 兼容、`HudiUtils` 5 方法、`IcebergHMSSource` 等 type-level 耦合）→ 补充 recon 闭合。**关键澄清**：recon 标"最大未知 = iceberg/hudi-on-HMS 归属"经查 decisions-log **实为已定**——**D-020**（per-table SPI provider，hive 网关委派 -iceberg/-hudi；否决 fe-core 发现期分派）+ **D-019**（hudi live cutover 并入 P7）→ **勿重议**。产出 **`tasks/P7-hive-migration.md`**（元信息/目标/关键事实/已定架构/P7.1–P7.5 拆分/old→new 映射/删除排序/翻闸机制/SPI 缺口/验收门/8 条开放决策 OQ-*/依赖节奏/meta）。校正过时数字（instanceof 31→85、HMSTransaction 1866→1895、HMSExternalTable 1293→1332）。doc-sync：本 PROGRESS + HANDOFF（覆写→P7.1 起步）+ connectors/hive.md。**下一 = P7.1 实现**（`HiveMetadataOps`→`HiveConnectorMetadata`+`HmsClient` 写方法；no-property-parsing；`ConnectorTableOps.truncateTable` 加 default）。 + - **2026-07-05（P6 iceberg 全部完成 + squash-合入 `branch-catalog-spi` #64688 `8b391c7459d` ⇒ P7 hive 启动）** ✅（已合入 upstream）。整条 P6（P6.1–P6.6 迁移/scan/write/procedures/sys-tables/行级 DML + 翻闸 + GSON 迁移 + 属性/鉴权全归插件 S1–S10 + 删 fe-core 原生 iceberg 子系统 −23744 行）一次性 squash 为 **#64688**（685 文件）。iceberg 已入 `SPI_READY_TYPES`，FE 走 SPI 路径。**遗留（P7 接手）**：fe-core `datasource/iceberg/` 尚存 **23 个 HMS-iceberg 支撑类**（`IcebergUtils`/`IcebergMetadataOps`/`source/IcebergScanNode`/`cache/`/`IcebergMvccSnapshot`/…），iceberg-on-HMS 走它们、随 P7 hive 迁移删（阶段四，D5/Q3=B）。**新工作分支 `catalog-spi-11-hive`**；[DEC-FLIP-1]「未 push」铁律随合入解除。**下一 = P7 hive/HMS 迁移**（master plan §3.8 + connectors/hive.md，起步 P7.1）。 - **2026-06-25（P6.5-T07 ⇒ parity 审计 + 2 现修 + 9 gap-fill + DV 中央登记，TDD+mutation）** 🟡（未 push；连接器 + guard 的 iceberg 分支 pre-flip dormant，guard 修对 paimon 行为不变〔默认 false 保留拒绝〕，零 live iceberg 行为变更）。**对抗 byte-parity 审计 wf** `wf_d530d760-ccf`〔8 area finder 覆 T02–T06 sys 路 + DESCRIBE/SHOW/info_schema；refute-by-default skeptic〔effort=high〕+ completeness critic；31 agent/1.6M token〕= **22 finding/19 confirmed**〔3 refuted 全对〕。**揭出 2 项主动偏差→用户 AskUserQuestion 双裁「现修」（[D-067]，非 DV）**：**A** parity-surface finder + critic **双独立揭出** + 主 session 实证——共享 fe-core `PluginDrivenScanNode.checkSysTableScanConstraints`〔P5 paimon `38e7140ce56`，"Mirrors PaimonScanNode.getProcessedTable"〕无条件拒**任何** `PluginDrivenSysExternalTable` 的 snapshot+scan-params；legacy `IcebergScanNode.createTableScan:569` 对 sys 表 honor `useRef/useSnapshot`〔无 isSystemTable gate〕+ 连接器 T02/T05 保留+honor pin（偏差①）→ 翻闸后 pin **dead-on-arrival**=回归〔DV-038/041 同族〕；**纠 HANDOFF**「偏差①非 DV」分类错。**B** `buildTableDescriptor` `TYPE_HMS.equals(原始值)` 大小写敏感→`type="HMS"` 得 ICEBERG_TABLE vs legacy HIVE_TABLE。**现修 A**（3 文件）：新 SPI `ConnectorScanPlanProvider.supportsSystemTableTimeTravel()` 默认 **false**〔paimon/mc/jdbc/es 继承、零回归〕+ `IcebergScanPlanProvider` override true + fe-core guard capability-aware〔放行 time-travel/branch-tag，**@incr 对所有连接器仍拒**——合成元数据表 incremental 无定义、legacy 静默忽略 guard fail-loud 更安全〕；**⚠️ guard 修仅移除主动 BLOCK**，完整 sys 时间旅行 e2e 还需 query→handle pin 休眠翻闸接线〔DV-041 族〕+ P6.8。**现修 B**：`equalsIgnoreCase` 一行 + 大写 UT。**+9 连接器 gap-fill**〔handle coords-in-identity〔plan-cache key〕/pinned-toString·colhandles auth-scope×2/keyset #969249/empty-sysname·sys location-creds〔BE-403〕·capability·hms 大写·fork〕。**mutation-check**：guard Mut-A〔`=false`→AllowsTimeTravel/AllowsBranchTag 双红〕+Mut-B〔去 @incr 子句→StillRejectsIncr 红〕；hms Mut〔`equalsIgnoreCase→equals`→大写 UT 红〕；每次 `cp`/python 复原 diff IDENTICAL。**验证（重跑 surefire，Rule 12）**：连接器全量 **541/0/1**〔=532+9，0 回归〕；fe-core guard **7/0/0**；checkstyle 0；import-gate 0；`CatalogFactory:51` 未改→iceberg 仍**不在** `SPI_READY_TYPES`。**新 1 D（[D-067]）；新 2 DV（DV-048 correctness-bearing〔F2 paimon priv·serialized 字节潜伏〕/049 perf-cosmetic〔sys split-weight·内部 TableType·position_deletes 文案〕）**。**T07-续 ✅（同日本 session）= 残留 8 gap-fill UT 全落 + mutation**：fe-core〔sys `getMysqlType`="BASE TABLE"〔同测钉 new==legacy `ICEBERG_EXTERNAL_TABLE.toMysqlType`〕/ `getEngine`="iceberg"+`getEngineTableTypeName`="ICEBERG_EXTERNAL_TABLE"〔**assertAll** 两 pin 独立 mutation〕/ position_deletes 通用 not-found〔含 snapshots 正控〕/ 非注册不变式〔sys 类无 `@SerializedName` 声明字段 + discovery key 无 `$`〕/ guard-message 顺序〔scan-params 先于 time-travel〕〕+ 连接器〔sys predicate **作为 residual 带给 BE** + sys split `/dummyPath`〕+ WHY-comment 修。**⚠️ 纠 audit spec（实证）**：sys 元数据列谓词是 **BE-applied residual 非 FE 行裁**——record_count=10 过滤后 FE 行数 2 vs 2 不变，FE 可达观测=反序列化 `task.residual()` 携 `record_count`〔plan-time 裁是 snapshot pin，已另测〕；故 test-6 由「行裁」改「residual 携带」，**非 legacy 偏差**〔legacy 同样 serialize FileScanTask 带 residual 给 BE〕。WHY-comment 修：legacy resolution 是 **case-SENSITIVE** `TableIf.findSysTable` `Map.get` over `IcebergSysTable` 小写键〔`getTableNameWithSysTableName` 不 lower-case suffix〕→连接器 `equalsIgnoreCase` accept 是 production 永不达无害超集；原注释误归给 `MetadataTableType.from`〔后者作用在 metadata-table BUILD 时 `resolveSysTable`，非 resolution gate〕。**mutation-check（全绿→红→`git checkout` 复原 IDENTICAL）**：fe-core 5 变异一次 build〔TableIf PLUGIN mysqltype→null / getEngine+getEngineTableTypeName iceberg case 删〔assertAll 双红〕/ 注入 position_deletes / `@SerializedName` on sysTableName / swap guard 块→time-travel 先〕→ test1–5 全红+1 已知 collateral〔testDelegates size 2→3〕；连接器 2 变异〔sys 丢 filter→residual=true / 改 dummy-path 常量〕→ test6/7 红。**验证（重跑 surefire，Rule 12）**：fe-core `PluginDrivenSysTableTest` 10/0/0 + guard 8/0/0；连接器全量 **543/0/1**〔=541+2，0 回归〕`IcebergScanPlanProviderTest` 67/0 + `IcebergConnectorMetadataSysTableTest` 22/0；checkstyle 0、import-gate 0、`CatalogFactory:51` 未改。**0 新 D/DV**。**live-e2e 未跑**（dormant，P6.8 兜底）。**T08 ✅（同日 ⇒ P6.5 DONE）**：收口汇总设计 `designs/P6.5-T08-systable-summary-design.md`（7 节，仿 P6.3/P6.4-T09）+ **faithfulness 对抗验证 wf** `wf_27596236-5fe`（3 verifier refute-by-default〔fe-core / 连接器+test6 / WHY-comment 链〕+ completeness critic；4 agent/256k token）= **18/18 confirmed、0 refuted、0 critic fix**；critic 经 **iceberg 1.10.1 bytecode** 独立证实 test-6 residual（`BaseFilesTable$ManifestReadTask` 携 row filter 为 per-task residual、`ManifestEvaluator.forRowFilter` 仅 prune manifest 文件非 metadata 行、`rows()` 不 apply residual→FE 行数不变 BE 应用）+ WHY-comment 链 C1–C5 全 confirmed（注释「factually accurate」）。gate 重跑实证 543/0/1 + fe-core 10/0+8/0。**= P6.5 DONE**。**下一 = P6.6 翻闸**（全有或全无，须 holistic 修 DV-038〔读〕/041〔写〕/045〔rewrite〕+ sys 时间旅行 query→handle pin threading）。 @@ -258,9 +261,9 @@ > 当本项目通过 Claude Code 这类 LLM agent 推进时，跟踪当前 session 状态、handoff 状况和 context 健康度。 -- **本 session 已完成**：**P6 完成 + 合入实证 + doc-sync（0 产线代码）** —— 核实 git（#64688 `8b391c7459d` 已合入 `branch-catalog-spi`、工作树干净、新分支 `catalog-spi-11-hive`）+ 实证 fe-core `datasource/iceberg/` 尚存 23 个 HMS-iceberg 支撑类（随 P7 阶段四删）；覆写 [HANDOFF.md](./HANDOFF.md) 为 P7 起步 + 同步刷新本 PROGRESS（§一/二/三/四/六/七）+ connectors/iceberg（✅ 合入）+ connectors/hive（🚧 P7）+ tasks/P6·P6.6（✅ 合入）。 -- **下一个 session 应做**：**P7 hive (+HMS) 迁移启动**——建 `tasks/P7-hive-migration.md` 阶段拆分 spec + code-grounded recon，起步 P7.1 HiveMetadataOps 全功能搬迁。开场读 master plan §3.8 + connectors/hive.md + 对照 fe-core `datasource/hive/` 真实代码（信控制流不信注释）。**R-002 ACID 写路径须专门集成测试作 gate。** -- **是否需要 handoff**：**是**——本场已**覆盖** rewrite [HANDOFF.md](./HANDOFF.md)（P6 合入实证 + P7 起步计划 + 23 HMS-iceberg 支撑类遗留）。 +- **本 session 已完成**：**P7 启动 = 10-agent code-grounded recon + 阶段拆分 spec（0 产线代码、0 新决策）** —— 建 `tasks/P7-hive-migration.md`（P7.1–P7.5 + old→new 映射 + 翻闸机制 + 跨连接器删除排序 + 8 条开放决策）；coverage-critic 闭合 instanceof-only 扫描盲区（type-level 耦合）；查 decisions-log 确认 D-019/D-020 已定 iceberg/hudi-on-HMS 归属（勿重议）；doc-sync 刷新本 PROGRESS + 覆写 HANDOFF（→P7.1 起步）+ connectors/hive.md。 +- **下一个 session 应做**：**P7.1 实现** —— 读 `tasks/P7-hive-migration.md` 全 + 对照 HEAD 核读 `HiveMetadataOps`(425)/`HiveConnectorMetadata`(override 0)/`HmsClient`；把 DDL/partition/col-stats 写端搬进 `HiveConnectorMetadata`+`HmsClient`（no-property-parsing；`ConnectorTableOps.truncateTable` 加 default）；建 `P7.1-Tnn` 逐 task 追加 spec。**信控制流不信 spec 行号。** +- **是否需要 handoff**：**是**——本场已**覆写** [HANDOFF.md](./HANDOFF.md)（recon+spec 完成实证 + P7.1 起步 6 要点 + 删除排序约束）。 - **协作规范**：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)（context 预算、subagent 使用、handoff 触发条件） --- diff --git a/plan-doc/connectors/hive.md b/plan-doc/connectors/hive.md index e5aa5796c2393f..3bae1dc74c82e6 100644 --- a/plan-doc/connectors/hive.md +++ b/plan-doc/connectors/hive.md @@ -10,9 +10,9 @@ | **fe-connector 模块** | `fe/fe-connector/fe-connector-hive/` + `fe/fe-connector/fe-connector-hms/`（共享库）| | **fe-core 旧路径** | `fe/fe-core/src/main/java/org/apache/doris/datasource/hive/` | | **共享依赖** | 自身 `fe-connector-hms`；被 hudi/iceberg-HMS/paimon-HMS 依赖 | -| **计划迁移阶段** | **P7**（最复杂，6 周；**当前活跃阶段**，起步 P7.1）| -| **当前状态** | 🚧 P7 启动中（iceberg P6 已合入 #64688 → hive 为下一个活跃迁移目标）；下一 = **P7.1 HiveMetadataOps 全功能搬迁** | -| **完成度** | 10%（hive 20% + hms 共享库已立） | +| **计划迁移阶段** | **P7**（最复杂，6 周；**当前活跃阶段**，phase-split spec 已立，起步 P7.1）| +| **当前状态** | 🚧 P7 进行中：**code-grounded recon（10-agent）+ 阶段拆分 spec `tasks/P7-hive-migration.md` 已完成**；下一 = **P7.1 HiveMetadataOps 全功能搬迁（实现）** | +| **完成度** | 12%（recon+spec 立 + hive scan 只读已立 + hms 共享读库已立；DDL/txn/event/stats 端 0） | | **主 owner** | TBD | --- @@ -21,9 +21,9 @@ | 步骤 | 状态 | 备注 | |---|---|---| -| 1 | 🟥 | fe-core 30 个顶层 + `event/`（21 个）+ `source/`（HiveScanNode 等） | -| 2 | 🟥 | fe-connector-hive 12 个文件（scan path + handles）；fe-connector-hms 9 个文件 | -| 3 | ⏳ | 反向 instanceof：**31 处**（最高）| +| 1 | 🟥 | fe-core **52** 文件（29 顶层 + `event/` 21 + `source/` 2）| +| 2 | 🟨 | fe-connector-hive 12 文件（**只读 scan 已立**，DDL/txn/stats override=0）；fe-connector-hms 9 文件（**共享读库**，无写/txn/lock/col-stats）| +| 3 | ⏳ | 反向 instanceof/cast：**85 occurrence / 33 文件**（最高；plan 旧记 31 已校正）+ ~7 处 type-level 耦合（CatalogFactory/GsonUtils/HudiUtils/IcebergHMSSource…）| | 4 | ⏳ | `HiveMetadataOps` 全功能未迁；P7.1 重头 | | 5 | ⏳ | | | 6 | ✅ | META-INF/services 已注册（HiveConnectorProvider）；hms 共享库无 service 注册 | @@ -90,6 +90,11 @@ ## 进度日志 +### 2026-07-05（下午 · P7 recon + 阶段拆分 spec 完成） +- **10-agent code-grounded recon**（`wf-p7-hive-recon` + 补充 type-coupling recon，~1.3M token）核清 52 文件分类、ACID 写路径、event pipeline、DLA 三分流、85 处反向 instanceof、跨连接器耦合 + 翻闸机制（CatalogFactory:50/133 + GsonUtils:366/447/471 兼容 + 6 文件写路径 retype 链 + 删除排序）。校正过时数字（instanceof 31→85、HMSTransaction 1866→1895、HMSExternalTable 1293→1332）。 +- **关键澄清**：recon 标"最大未知"= iceberg/hudi-on-HMS 归属，实为**已定** —— D-020（per-table SPI provider，hive 网关委派 -iceberg/-hudi）+ D-019（hudi live cutover 并入 P7）。故本阶段目标含删 `datasource/hudi/` + 23 HMS-iceberg 类。 +- **产出** `tasks/P7-hive-migration.md`（阶段拆分 spec，P7.1–P7.5 + old→new 映射 + SPI 缺口 + 8 条开放决策待各子阶段签字）。**下一 = P7.1 实现**（HiveMetadataOps → HiveConnectorMetadata + HmsClient 写方法）。 + ### 2026-07-05（P7 启动 = 当前活跃迁移目标） - **iceberg P6 已 squash-合入 `branch-catalog-spi`（#64688 `8b391c7459d`）→ hive 成为下一个活跃迁移目标**。工作分支 `catalog-spi-11-hive`。 - **下个 session 起步**：建 `tasks/P7-hive-migration.md` 阶段拆分 spec + code-grounded recon；起步 P7.1 HiveMetadataOps 全功能搬迁。权威计划 master plan §3.8 + 本文 §子阶段。**R-002 ACID 写路径（P7.3）= 项目最大风险，须专门集成测试作 gate。** diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md new file mode 100644 index 00000000000000..3efa67c5919fe0 --- /dev/null +++ b/plan-doc/tasks/P7-hive-migration.md @@ -0,0 +1,167 @@ +# P7 — hive (+HMS) 迁移（最复杂、最后一个连接器；先在 fe-connector 实现完整能力 → 分子阶段翻闸 → 删 legacy） + +> 阶段拆分 spec（phase-level plan），镜像 `P6-iceberg-migration.md`。各子阶段的逐 task 拆解（P7.x-Tnn）在**进入该子阶段时**做 code-grounded recon 后追加到本文件末尾。 +> 本 spec 的"关键事实/映射/翻闸机制"来自 2026-07-05 的 10-agent code-grounded recon（工作流 `wf-p7-hive-recon` + 补充 type-coupling recon），已对照 HEAD 真实代码校正 HANDOFF/master-plan/connectors 里的过时数字。 + +--- + +## 元信息 + +| 项 | 值 | +|---|---| +| catalog type 名 | `hms`（`CATALOG_TYPE_PROP=hms`）| +| 目标模块 | `fe/fe-connector/fe-connector-hive/`（= "hms" 网关连接器）+ `fe/fe-connector/fe-connector-hms/`（共享元存储库）| +| fe-core 旧路径 | `fe/fe-core/src/main/java/org/apache/doris/datasource/hive/`（**52** 个文件：29 顶层 + `event/` 21 + `source/` 2）| +| 工作分支 | `catalog-spi-11-hive`（off `branch-catalog-spi` @ `8b391c7459d`）| +| PR base / 合并方式 | `branch-catalog-spi`，**squash**（复用 P5-T29 #64653 / P6 #64688 范式）| +| tracking issue | apache/doris#65185（PR 须引用）| +| 估时 | 6 周（master plan §3.8）| +| 主 owner | TBD | + +--- + +## 阶段目标（终态） + +1. `hms` catalog 走 SPI 路径：`CatalogFactory` 不再 `new HMSExternalCatalog`；catalog/db/table 退化为 `PluginDrivenExternalCatalog` / `PluginDrivenExternalDatabase` / `PluginDrivenExternalTable`。 +2. fe-core **零** source-specific 代码：删净 `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名字符串判别（**85 处 occurrence / 33 文件** + 补充 recon 揪出的 ~7 处 type-level 耦合）。 +3. fe-core 不解析任何 hive 属性（file-format/SerDe/ACID/staging/broker.name 全部移到插件；no-property-parsing 铁律）。 +4. hive/hudi/iceberg-on-HMS 三格式经 **per-table SPI provider**（D-020）分流；hudi live cutover + 删 fe-core `datasource/hudi/`（D-019）并入本阶段。 +5. 删除 fe-core `datasource/hive/` 整目录 + P6 遗留的 23 个 HMS-iceberg 支撑类 + `datasource/hudi/`。 +6. ACID 写路径重写为连接器 `ConnectorTransaction`（E4），有独立集成测试作 gate（R-002，项目最大风险）。 + +--- + +## 关键事实（2026-07-05 code-grounded recon） + +**规模（校正过时数字）**：`HMSTransaction` **1895** 行（plan 写 1866）、`HMSExternalTable` **1332** 行（plan 写 1293）、`HiveMetadataOps` 425 行、`MetastoreEventsProcessor` 357 行、`HiveTransactionMgr` 55 行、`HMSExternalCatalog` 250 行。反向 `instanceof/cast` = **85 occurrence / 33 文件**（plan 写 31）。fe-core test 引用 HMS 类型 = **22 文件**（迁移须预算）。 + +**连接器现状**：`fe-connector-hive`（12 主类）= **只读 scan 已立**（`HiveScanPlanProvider`/`HiveScanRange`/`HiveFileFormat`/`HiveTableFormatDetector`/`HiveTableType`/`HiveTextProperties`/handles）；`HiveConnectorMetadata` **override 了 0 个** DDL/txn/stats/partition SPI（全继承默认 throw）。`fe-connector-hms`（9 主类）= 共享读元存储客户端（`HmsClient`/`ThriftHmsClient`/`HmsClientConfig`/`HmsConfHelper`/`Hms*Info`/`HmsTypeMapping`）+ vendored `HiveVersionUtil` + `HiveMetaStoreClient` shim；**无写/txn/lock/col-stats 方法**。P3/P5/P6 已在用 `fe-connector-hms`（稳定）。 + +**HMS 是异构 catalog**：一个 `hms` catalog 下同时有 plain-hive（非 MVCC，时间戳新鲜度）+ iceberg-on-HMS / hudi-on-HMS（MVCC，snapshot 新鲜度）。今天靠 `HMSExternalTable` **单类 + 惰性探测 `dlaType`** + 处处 `switch(dlaType)`（recon 数出 ~19 个分支点）承载三者；`HMSDlaTable` 策略只抽出了 MTMV/partition 面，其余（schema/stats/rowcount/file-format/sys-table/mvcc/toThrift）仍是 `HMSExternalTable` 内联分支，**且 ICEBERG/HUDI 分支直接调 fe-core 的 `IcebergUtils`/`HudiUtils` 子系统 → 它们是那些 P6 遗留子系统的最后 live 消费者**（删除排序的核心约束，见下）。 + +--- + +## 已定架构（**勿重议**，实现即可 — 引 decisions-log） + +| 决策 | 结论 | 对 P7 的含义 | +|---|---|---| +| **D-004** | HMS event pipeline 放 `fe-connector-hms`，经 `ConnectorMetaInvalidator` 回调 | P7.2 把 21 event 类 + processor 搬入 hms 库 | +| **D-005** | hive/hudi/iceberg-on-HMS 用 `ConnectorTableSchema.tableFormatType` 区分（值 `"HIVE"`/`"HUDI"`/`"ICEBERG"`，连接器探测后填充）| P7.4 DLA 退化；tableFormatType 作 opaque 串逐字上报、**fe-core 热路径不读**（不得 `if(format==...)`）| +| **D-020** | 单 `hms` catalog 多格式 scan 路由 = **方案 B（per-table SPI provider）**：`ConnectorMetadata` 新增向后兼容 default `getScanPlanProvider(handle)`（默认 null→回落 per-catalog）；注册 `"hms"` 的连接器（=`HiveConnectorProvider`/fe-connector-hive）override 之，按 `handle.getTableType()` **委派** Hudi/Iceberg provider | fe-connector-hive = "hms" 网关，**依赖 `-hudi`/`-iceberg` 模块**做委派；**否决**了"fe-core 发现期分派"和"hive 内嵌 iceberg/hudi SDK" | +| **D-019** | P3 hudi hybrid 把 live cutover（fe-core 消费 tableFormatType per-table 分流 + gate flip `SPI_READY_TYPES` 加 hms/hudi + 删 legacy `datasource/hudi/` + 完整增量/time-travel）**推入 P7** | hudi 批 E 并入本阶段（P7.4/P7.5）| +| **D-003** | 旧 `*ExternalCatalog` 子类全部删除，不留中间形态 | `HMSExternalCatalog/Database/Table` 删除 | +| 事务模型（D-022/24/25 + "A 桥接"）| 连接器 `ConnectorTransaction` 为单一事实源；fe-core 通用写编排经 `PluginDrivenTransaction` 桥接 | P7.3 HMSTransaction 重表达于 E4，**不新增** ConnectorMetadata 写 SPI（写/stats/partition 方法留在 HmsClient，由 hive 的 ConnectorTransaction.commit 驱动）| + +--- + +## 阶段拆分（P7.1 – P7.5，master plan §3.8 + recon 细化） + +> 节奏：串行为主（P7.1 是地基，P7.4 依赖 P7.1/P7.3 的连接器能力齐备，P7.5 依赖全部）。每子阶段 = 独立 commit + build + test + checkstyle 0 + import-gate 净；**每轮完成即更新 HANDOFF + commit**。 + +### P7.1 — HiveMetadataOps 全功能搬迁（DDL / partition / statistics 写端）— 2 周 +把 `HiveMetadataOps`(425) 的 create/drop db+table、rename、truncate、add/drop partition、column-stats 写回搬进 `HiveConnectorMetadata` + `HmsClient`（加写方法，或 hive-only 写子接口避免撑大 hudi 共享面）。E1 CreateTable（identity partition + bucket）、E8 col-stats 写回、E10 listPartitions。**no-property-parsing**：file_format/owner/bucket/DLF-guard/LIST-partition 校验全部移入插件（须确认 `ConnectorCreateTableRequest` 富到能重建 `HiveTableMetadata`，否则扩之）。 + +### P7.2 — event pipeline 搬入 fe-connector-hms（D-004）— 1.5 周 +21 event 类 + `MetastoreEventsProcessor` 搬入 hms 库，经 `ConnectorMetaInvalidator` 交付失效。**核心 fork（见开放决策 OQ-EVT）**：保留事件驱动的**结构化 register/unregister**（需新 SPI seam 回 fe-core）还是**降级为纯 invalidate + 惰性 re-list**（只复用现有 invalidator，但改 master/slave 语义 + 丢预填缓存）。连带：poller loop 位置（全搬 vs fe-core 留薄 driver 管 master/slave + edit-log）、MetaId/`ExternalMetaIdMgr` 是否 HMS 还需要（`genIdByName` 确定性可能使其冗余）、partition-name 粒度失效（现 SPI 只带 values→降级为 table 级）、R-010 线程泄漏 + TCCL pin。 + +### P7.3 — HMSTransaction + 写路径重写（ACID，最难，R-002）— 2 周 +`HMSTransaction`(1895) + `HiveTransactionMgr`(55) 重表达于 `ConnectorTransaction`（E4）+ hms 库的 txn/writeId/lock/commit 方法。**写路径是 6 文件耦合链**（须一起 retype `HMSExternalTable`→generic）：`BindSink.bindHiveTableSink` → `LogicalHiveTableSink` → `PhysicalHiveTableSink` → `PhysicalPlanTranslator:569 (new HiveTableSink)` → legacy `planner/HiveTableSink` → `HiveInsertExecutor`。读侧 ACID（delete-delta，`AcidUtil.getAcidState` + valid-write-ids）须移入插件（否则 txn 表静默错读——`HiveScanRange.populateTransactionalHiveParams` 现只填 dir 不填 fileNames，是 stub）。**reader-txn 生命周期缺 SPI seam**（现由 `QeProcessorImpl:210` 硬调 `Env.getHiveTransactionMgr` 的 query-finish 回调 + 共享锁无 heartbeat）——须加中立 per-query finish 回调或收进 scan-provider teardown（见 OQ-RTX）。**gate = 独立 ACID 集成测试套件**。 + +### P7.4 — DLA 分流改造 + iceberg/hudi-on-HMS 委派 + hudi live cutover（D-005/D-019/D-020）— 0.5 周（实际含 hudi 会更重） +`HMSExternalTable`→`PluginDrivenExternalTable`（plain，非 MVCC）/ iceberg-hudi-on-HMS 表经 per-table `getScanPlanProvider(handle)` 委派给 `-iceberg`/`-hudi` 连接器。异构 catalog 的**表类抉择**（见 OQ-HET）+ tableFormatType 须 thread 进 `PluginDrivenSchemaCacheValue`（现被 `toSchemaCacheValue` 丢弃）。31→85 处 instanceof/switch 的 planner/nereids/stats/tvf 侧改为中立 capability / per-table trait。plain-hive 的**时间戳新鲜度** MTMV（非 snapshot-id）须 `PluginDrivenMvccExternalTable` 支持（现 `getTableSnapshot` 只返 snapshot-id）。 + +### P7.5 — 删 fe-core datasource/hive + hudi + 23 HMS-iceberg 类 + 翻闸收口 — 0.5 周（实际更重） +gate flip + 删目录 + 删所有 instanceof + 常量搬迁 + GsonUtils 兼容（见翻闸机制）。**受删除排序约束**（见下）。 + +--- + +## 跨连接器删除排序（**本阶段最硬约束**） + +`datasource/hive/` **不能删**，直到以下非-hive 消费者全部 retype 到 generic table（否则编译断）： + +| fe-core 消费者 | 依赖的 hive 类 | 何时解绑 | +|---|---|---| +| `datasource/hudi/HudiUtils`（5 方法带 `HMSExternalTable` 参，:259–423，用 `getHudiClient`/`useHiveSyncPartition`）| HMSExternalTable | hudi 迁入插件（P7.4）| +| `datasource/hudi/HudiExternalMetaCache`（`findHudiTable` cast HMSExternalCatalog.getClient）| HMSExternalCatalog/Table | hudi 迁入插件（P7.4）| +| `datasource/hudi/HudiScanNode extends HiveScanNode`；`HudiSchemaCacheValue extends HMSSchemaCacheValue` | HiveScanNode/HMSSchemaCacheValue | hudi 迁入插件 → 决定共享基类是搬 hms 库还是随 hudi（OQ-SHARE）| +| `datasource/iceberg/source/IcebergHMSSource`（field+ctor `HMSExternalTable`，:30/:34）| HMSExternalTable | iceberg-on-HMS 走 per-table provider（P7.4）| +| `statistics/HMSAnalysisTask`（field + `setTable(HMSExternalTable)`）| HMSExternalTable | col-stats E8 中立化（P7.4）| +| `statistics/util/StatisticsUtil.getIcebergColumnStats(org.apache.iceberg.Table)` | iceberg SDK in fe-core | iceberg-on-HMS 走 iceberg 连接器（P7.4）| + +**含义**：P7.4 必须把 hudi + iceberg-on-HMS + hudi-on-HMS 全部切到插件路径，P7.5 才能删 `datasource/hive/`。`fe-connector-hive` 依赖 `-iceberg`/`-hudi`（D-020）是委派前提。 + +--- + +## 翻闸机制（cutover mechanics，实测行号） + +1. **CatalogFactory**：`SPI_READY_TYPES`（:50）加 `"hms"`；删 `case "hms"`（:133–134 `new HMSExternalCatalog`）+ import。iceberg/paimon 已是此形态（其 case 已删）。 +2. **GsonUtils 兼容（元数据 image/edit-log 回放 HAZARD，3 factory）**：把 `registerSubtype` → `registerCompatibleSubtype`： + - :366 `HMSExternalCatalog` → `PluginDrivenExternalCatalog` + - :447 `HMSExternalDatabase` → `PluginDrivenExternalDatabase` + - :471 `HMSExternalTable` → **plain** `PluginDrivenExternalTable`（hive **非 MVCC**，区别于 paimon/iceberg 的 `PluginDrivenMvccExternalTable`，:494） + - **一条 `"HMSExternalTable"` 兼容行覆盖 hive+hudi-on-HMS+iceberg-on-HMS**（三者历史都持久化为同一 tag；格式判别 load 后由 tableFormatType 承接）。precedent：:388 `PaimonHMSExternalCatalog` / :399 `IcebergHMSExternalCatalog`。须加 hive gson-compat replay 单测（仿 `IcebergGsonCompatReplayTest`/`PaimonGsonCompatReplayTest`）。 +3. **常量搬迁（删类前）**：`HMSExternalCatalog.BIND_BROKER_NAME="broker.name"`（`ExternalCatalog:1320` 基类读它，generic 路径，`DefaultConnectorContext:304` 也调）、`HIVE_STAGING_DIR`/`DEFAULT_STAGING_BASE_DIR`（`HiveTableSink:173`）→ 移插件/属性侧，基类 `bindBrokerName` 改读 fe-core-local 常量或经 properties thread。 +4. **写路径 6 文件 retype**（P7.3）：见上。 + +--- + +## SPI 缺口（consolidated，按子阶段） + +**P7.1**：ConnectorTableOps 加 `truncateTable`（default throw，hive override→HMS native truncate）；`ConnectorCreateTableRequest` 富化（bucket cols/count、LIST partition col names、per-col default、DLF flag）；force `dropDatabase` cascade 语义下沉连接器；写/stats/partition 方法**留 HmsClient**（不上 ConnectorMetadata），由 P7.3 的 txn 驱动。 + +**P7.2**：`ConnectorMetaInvalidator` 三缺口——(a) 结构化 register/unregister seam（或降级 invalidate，OQ-EVT）；(b) partition-**name** 粒度失效（现只带 values→降级 table 级）+ 批量（modified+added）；(c) master/slave 角色 + master 转发 + edit-log/MetaId（须 fe-core 留 driver 或加 ConnectorContext seam）。R-010 线程生命周期 + TCCL pin。 + +**P7.3**：reader-txn 生命周期回调（query-finish，中立、所有连接器可用，替 `QeProcessorImpl`→`Env` 硬耦合）；write-begin context（isOverwrite/fileType/writePath 进 `ConnectorWriteHandle.getWriteContext`，仿 iceberg `IcebergWriteContext`，finishInsert 折进 commit）；post-commit **选择性** partition 失效 + follower edit-log（确认 `invalidatePartition` 是否 fan-out edit-log，否则退化 full-table）；连接器共享 Executor（异步 rename）；ACID 读 delete-delta（dir+fileNames）；共享锁 heartbeat（OQ-LOCK，默认保持现状 no-heartbeat）。 + +**P7.4**：`tableFormatType` thread 进 `PluginDrivenSchemaCacheValue` + getter（**只用于 split 路由/capability 派生，不用于 planner 分支**）；异构 catalog per-table MVCC-vs-plain 表类（OQ-HET）；plain-hive 时间戳新鲜度（`Freshness.TIMESTAMP` + `MTMVMaxTimestampSnapshot` 语义）；per-table file-scan trait（top-N lazy-mat / nested-column-prune，hive 按 format、iceberg 无条件）；`SUPPORTS_SQL_CACHE` capability（否则 hive 翻闸后静默丢 SQL cache）；`SUPPORTS_SAMPLE_ANALYZE` capability；partition_values TVF 中立化（`Map>` + capability gate）；ACID kind 作 `HiveTableHandle` field/capability（连接器持 `AcidUtils`）。 + +**P7.5**：`ExternalMetaCacheRouteResolver` 硬编 `instanceof HMSExternalCatalog→{hive,hudi,iceberg}`——加 capability 返回 cache-engine id 集（否则退化 default，静默断跨格式失效）。 + +--- + +## 验收门（per 子阶段，逐项细化在各子阶段 recon 时定） + +- 编译 `BUILD SUCCESS`（fe-core 只依赖 fe-connector-api；连接器注意 optional-shade `HiveConf`/hadoop-common）。 +- checkstyle 0（`mvn -pl : checkstyle:check` **不带 -am**）。 +- import-gate 净（`tools/check-connector-imports.sh`；HMS `HiveVersionUtil` 命中=误报，见 memory）。 +- 连接器单测（无 Mockito，真 fake/recording）+ fe-core 单测（Mockito）。 +- **P7.3 硬门**：独立 ACID 集成测试套件（INSERT INTO / INSERT OVERWRITE / 分区写 / delete-delta 读 / rollback / 多 FE 失效），R-002 缓解。 +- 翻闸门：gson-compat replay 单测（老 image tag 反序列化）+ image 兼容回归。 +- 端到端：docker 重部署类加载冒烟（TCCL split-brain，见 memory）。 + +--- + +## 开放决策（待各子阶段 recon 后**用户签字**；此处附推荐，勿在本 session 拍板） + +- **OQ-HET（P7.4，异构 catalog 表类）**：一个 hms catalog 混装 plain-hive（非 MVCC）+ iceberg/hudi-on-HMS（MVCC），而表的 Java 类今天在**注册进 cache 时**（读表前，为 SHOW TABLES 快）就定。方案 (a) 一律建 `PluginDrivenMvccExternalTable`、plain-hive 退化为 empty/timestamp 新鲜度；(b) 惰性到首读再定类/行为。**推荐 (a)**（避免 eager 全 catalog load；MVCC 类对 plain-hive 做 trivial no-snapshot）。 +- **OQ-EVT（P7.2，事件模型）**：结构化 register/unregister（新 SPI seam）vs 纯 invalidate + 惰性 re-list（只复用现 invalidator，改 master/slave 语义 + 丢预填缓存）。**推荐**：pipeline 类搬插件，但 role-aware polling driver + edit-log 留 fe-core 薄壳（cleaner，少新 SPI）；结构事件优先降级 invalidate（惰性 reload），仅 partition-name 粒度确需扩 SPI。 +- **OQ-RTX（P7.3，reader-txn 生命周期）**：加通用 per-query finish 回调（fe-core 为所有连接器调）vs 收进 scan-provider teardown。**推荐**：通用 query-lifecycle 回调（替 `QeProcessorImpl`→`Env` 硬耦合，其他连接器也可用）。 +- **OQ-ACID-WRITE（P7.3，写 ACID 范围）**：今天 full-ACID **表**的 INSERT 是硬拒（`InsertIntoTableCommand:553`），非-ACID INSERT INTO/OVERWRITE 走 HMSTransaction，ACID **读**（delete-delta）支持。**推荐：迁移行为保持**——非-ACID 写 + ACID 读迁移到位，full-ACID 写**继续拒**（不在本阶段引入净新 ACID 写能力，控 R-002）。 +- **OQ-SHOWCREATE（P7.4/P7.5，可见行为）**：hive `SHOW CREATE TABLE` 今天吐原生 Hive DDL（`HiveMetaStoreClientHelper.showCreateTable`）。加"render full DDL" SPI 逐字保 vs 接受 generic `Env.getDdlStmt`（可见输出变化）。**需产品签字**；推荐加 SPI 保持向后兼容。 +- **OQ-SHARE（P7.4/P7.5，共享基类去向）**：`HiveScanNode`/`HiveSplit`/`HivePartition`/`HMSSchemaCacheValue` 被 hudi 继承。搬 `fe-connector-hms` 共享库并 re-point hudi，还是随各连接器复制。**推荐**搬 hms 共享库（单副本，对齐 D-004）。 +- **OQ-LOCK（P7.3）**：读侧共享 HMS 锁无 heartbeat。**推荐**：迁移保持现状（不静默改），若加 heartbeat 单列。 +- **OQ-COLSTATS（P7.1/P7.4，E8）**：hive col-stats 保 HMS-metadata 快路径（读 spark col-stats + NUM_ROWS，不扫描；需扩 ConnectorStatisticsOps 加 per-column 读）vs 降级为 generic SQL-based analyze（同 iceberg/paimon，简单但丢快路径）。**推荐**：扩 SPI 保快路径（hive 大表 analyze 性能敏感）。 + +--- + +## 阶段依赖 + 节奏 + +``` +P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写方法) + │ │ + └──→ P7.2 (event，可与 P7.1 并行) + ↓ + P7.4 (DLA 退化 + hudi/iceberg-on-HMS 委派，依赖 P7.1/P7.3 连接器能力齐 + hudi 迁移) + ↓ + P7.5 (删 legacy + 翻闸收口，依赖全部 + 删除排序解绑) +``` +每子阶段单独 PR？否——沿 P6 范式**整阶段一次 squash 合入**（未 push 铁律已随 #64688 解除，P7 走常规流程）。子阶段间在工作分支上串行 commit。 + +--- + +## 给下一个 agent 的 meta + +- **起步 P7.1**：读 `HiveMetadataOps.java`(425) 全 public 面 + `HiveConnectorMetadata.java`（现 override 0）+ `HmsClient`/`ThriftHmsClient`；对照 gap（recon R4/R7 已列，但**信 HEAD 代码不信本 spec 行号**）。建 `P7.1-Tnn` 逐 task 拆解追加本文件。 +- **recon 存档**：完整 10-agent recon 结构化结果 + 补充 type-coupling recon 在本 session 的 job tmp（`p7-recon-raw.json` / `p7-recon-digest.md`，**ephemeral**）；关键结论已蒸馏进本 spec + `connectors/hive.md`。若需重跑：`.claude/wf-p7-hive-recon.js`。 +- **铁律复读**：fe-core 不得新增 `if(hive)`/`instanceof HMSExternal*`/引擎名判别（memory `catalog-spi-plugindriven-no-source-specific-code`）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`）；history_schema_info nested 名 lowercase（memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 +- **决策纪律**：D-004/005/019/020 + 事务桥接**已定勿重议**；OQ-* 到各子阶段 recon 后再用户签字（先中文讲背景+示例+推荐，不引任务代号）。 From c8ee3e9d99fe68460b665e0f3b216e084f1c20f4 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 00:13:15 +0800 Subject: [PATCH 003/334] =?UTF-8?q?[doc](catalog)=20P7.1=20hive:=20code-gr?= =?UTF-8?q?ounded=20recon=20+=20T01=E2=80=93T11=20task=20breakdown=20+=20l?= =?UTF-8?q?ock=20config-threading=20decision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P7.1 起步 recon（3 并行 agent + 直读 HEAD）核清并校正 spec： - 通用 DDL 桥 = PluginDrivenExternalCatalog（逐条 override + 内联 cache/editlog；连接器只实现纯 SPI）。 - TRUNCATE 硬缺口：SPI 无 truncateTable、桥未 override → 须加 seam + 桥 override（复用 OP_TRUNCATE_TABLE）。 - shared converter 逐字拷 properties（不 parse），但丢每列默认值（恒 null → 破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value。 - stats/partition 写 4 法仅 HMSTransaction 消费 → 推 P7.3（本阶段加即死代码）；rename hive 今天不支持 → 保持 SPI default throw。 - 模板 = IcebergConnectorMetadata（HMS-backed + DLF guard + 插件侧 SchemaBuilder）。 产出 tasks/P7-hive-migration.md 末尾 P7.1 逐 task 拆解（T01–T11，范围=仅 DDL 写路径）+ recon 结论块。 唯一阻塞决策已由用户裁定：全局建表默认（默认文件格式 / 是否允许分桶表）迁到插件后取值来源 = 方案 A （fe-core 建连接器时把两个全局 Config 值注入为连接器属性默认，插件只读属性、零行为回归）。 无代码改动（仅文档 + 决策）。 Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 22 +++++++++--------- plan-doc/tasks/P7-hive-migration.md | 35 +++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+), 11 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 254e8569928792..b9d9a3270232ef 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,9 +5,9 @@ --- -# 🎯 当前状态（2026-07-05 下午）= **P7 hive/HMS 迁移已启动：10-agent code-grounded recon + 阶段拆分 spec 完成；下一步 = P7.1 HiveMetadataOps 实现** +# 🎯 当前状态（2026-07-06）= **P7.1 起步 recon + 逐 task 拆解 + 唯一阻塞决策已裁定；下一步 = 按 P7.1-T01…T11 写代码（尚无代码改动）** -> **本轮（recon + spec session）做了什么**：跑 10-agent code-grounded recon（`.claude/wf-p7-hive-recon.js` + 补充 type-coupling recon，~1.3M token，0 error）核清 fe-core `datasource/hive/` **52 文件**全貌、ACID 写路径、event pipeline、DLA 三分流、**85 处反向 instanceof/33 文件**、跨连接器耦合、翻闸机制。产出 **`tasks/P7-hive-migration.md`**（P7.1–P7.5 阶段拆分 spec + old→new 映射 + SPI 缺口 + 删除排序 + 8 条开放决策）。校正了 plan/HANDOFF 里过时数字（instanceof 31→85、HMSTransaction 1866→1895、HMSExternalTable 1293→1332）。**关键澄清**：recon 标"最大未知 = iceberg/hudi-on-HMS 归属"实为**已定**（D-020 per-table SPI provider 委派 + D-019 hudi live cutover 并入 P7）——**勿重议**。**无代码改动、无新决策**（仅文档）。 +> **本轮（P7.1 recon + 拆解 session）做了什么**：对 P7.1 范围做 code-grounded recon（3 并行 agent + 直读 HEAD，~225k subagent token）核清 4 件事并**校正了 spec**：(1) 通用 DDL 桥 = `PluginDrivenExternalCatalog`（非独立 MetadataOps，逐条 override + 内联 cache/editlog 钩子，连接器只实现纯 SPI）；(2) **TRUNCATE 是硬缺口**（SPI 无 `truncateTable`、桥未 override）；(3) shared converter `CreateTableInfoToConnectorRequestConverter` 逐字拷 properties（✅ 不 parse）但**丢每列默认值**（恒 null，破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value 定义；(4) stats/partition 写 4 法**仅 HMSTransaction 消费**→**推 P7.3**（本阶段加进来即死代码），rename **hive 今天不支持**→保持 SPI default throw。产出 **`tasks/P7-hive-migration.md` 末尾 P7.1 逐 task 拆解（T01–T11）+ recon 结论块**。**唯一阻塞决策 OQ-HIVE-CFG 已由用户裁定 = 方案 A**（fe-core 建连接器时把 `hive_default_file_format`/`enable_create_hive_bucket_table` 两个全局 Config 值注入为连接器属性默认，插件只读属性、零行为回归）。**本轮无代码改动**（仅文档 + 决策）。 > **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： > P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 @@ -29,18 +29,18 @@ --- -# 🚀 下个 session 任务 = **P7.1 实现（HiveMetadataOps → HiveConnectorMetadata + HmsClient 写方法）** +# 🚀 下个 session 任务 = **执行 P7.1 实现（按 spec 末尾 P7.1-T01…T11；recon+决策已就绪，可直接写代码）** -> **权威计划**：**`tasks/P7-hive-migration.md`**（本轮新建的阶段拆分 spec，含 P7.1–P7.5 + old→new 映射 + SPI 缺口 + 翻闸机制 + 删除排序 + 开放决策）+ [`00-connector-migration-master-plan.md` §3.8](./00-connector-migration-master-plan.md) + [`connectors/hive.md`](./connectors/hive.md)。**单连接器 13 步 playbook = master plan §4。** -> **模块已就绪**：`fe-connector-hive`（"hms" 网关，只读 scan 已立、DDL/txn/stats override=0）+ `fe-connector-hms`（共享读元存储库，无写/txn/lock/col-stats）。 +> **权威计划**：**`tasks/P7-hive-migration.md`**——尤其**文件末尾「P7.1 逐 task 拆解」块**（本轮新建，含 recon 结论 + T01–T11 + OQ-HIVE-CFG=方案A 裁定）+ 上半 P7.1–P7.5 spec + [`connectors/hive.md`](./connectors/hive.md)。**单连接器 13 步 playbook = master plan §4。** +> **模块已就绪**：`fe-connector-hive`（"hms" 网关，只读 scan 已立、DDL override=0）+ `fe-connector-hms`（共享读元存储库，**只读、无任何写方法**）。**通用桥 = `PluginDrivenExternalCatalog`**（DDL 命令逐条 override + 内联 cache/editlog，连接器只实现纯 SPI 方法）。**模板 = `IcebergConnectorMetadata`**（最接近：HMS-backed + DLF guard + location cleanup + 插件侧 SchemaBuilder）。 -## P7.1 起步要点（下个 session 开场） +## P7.1 执行要点（下个 session 开场） -1. **先读** `tasks/P7-hive-migration.md`（全）+ 对照 HEAD 真实代码 recon `HiveMetadataOps.java`(425) 全 public 面 + `HiveConnectorMetadata.java`（现 override 0）+ `fe-connector-hms/HmsClient`/`ThriftHmsClient`（**信控制流不信本 spec 行号**）。 -2. **P7.1 范围**：把 `HiveMetadataOps` 的 create/drop db+table、rename、truncate、add/drop partition、col-stats 写回搬进 `HiveConnectorMetadata` + `HmsClient`（写方法留 HmsClient / hive-only 写子接口，别撑大 hudi 共享面）；**no-property-parsing**：file_format/owner/bucket/DLF-guard 全移插件（先确认 `ConnectorCreateTableRequest` 富度，不够就扩）。新增 `ConnectorTableOps.truncateTable`（default throw）。建 `P7.1-Tnn` 逐 task 拆解追加进 spec 文件。 -3. **勿重议的已定决策**（引 decisions-log，spec §已定架构）：**D-004** event→fe-connector-hms；**D-005** tableFormatType 区分（fe-core 热路径不读）；**D-020** per-table SPI provider（hive 网关委派 -iceberg/-hudi）；**D-019** hudi live cutover + 删 legacy 并入 P7；事务桥接 `PluginDrivenTransaction`。 -4. **开放决策（OQ-*）**：到各子阶段 recon 后再**用户签字**（先中文讲背景+示例+推荐、不引任务代号）；本 spec §开放决策已附推荐。 -5. **纪律**：每子阶段独立 commit + build + test + checkstyle 0 + import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。**P7.3 ACID 写路径必须有独立集成测试作 gate（R-002 项目最大风险）**。 +1. **先读** spec 末尾「P7.1 逐 task 拆解」块（recon 结论 + T01–T11）。按 **T01(truncate SPI seam+桥 override) → T02(hms 写 DTO) → T03(hms converter，HiveUtil 等价插件侧) → T04(HmsClient 写方法 DDL-only) → T05(HiveConnectorMetadata DB DDL) → T06(createTable) → T07(dropTable+truncate；rename 保持 default throw) → T08(shared converter 带列默认值) → T09(config 注入=方案A) → T10(连接器单测) → T11(守门+交接)** 顺序推。**信 HEAD 控制流不信 spec 行号。** +2. **本阶段范围 = 仅 DDL 写路径**：create/drop db+table + truncate。**stats/partition 写 4 法（updateTableStatistics/updatePartitionStatistics/addPartitions/dropPartition）推 P7.3**（仅 HMSTransaction 消费，本阶段加即死代码）。**rename 不实现**（hive 今天不支持，保持 SPI default throw）。**no-property-parsing**：file_format 默认/owner/bucket-gate/DLF-guard/transactional-reject 全插件侧（`ConnectorSession.getUser()`+`getCatalogProperties()`+`ConnectorColumn.defaultValue` 已够，无须扩 SPI，除 truncate seam + T08 列默认值 carrier）。 +3. **OQ-HIVE-CFG 已裁定 = 方案 A**（T09）：fe-core 组装 `hms` 连接器属性处，把 `Config.hive_default_file_format`/`Config.enable_create_hive_bucket_table` 当前值注入为属性默认（仅用户未显式设时），**插件只读属性、不碰 `common.Config`**。注入 locus 待 recon。 +4. **勿重议的已定决策**：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransaction`。后续子阶段 OQ-* 到 recon 后再用户签字（中文讲背景+示例+推荐、不引任务代号）。 +5. **纪律**：每 task/每条 fix 独立 commit + build + test + checkstyle 0（不带 -am）+ import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。P7.1 **非 live**（翻闸在 P7.5），验收靠**连接器单测**（无 Mockito、recording fake `IMetaStoreClient`），无 e2e DDL。**P7.3 ACID 写路径必须有独立集成测试作 gate（R-002 项目最大风险）**。 6. **删除排序（最硬约束，spec §跨连接器删除排序）**：`datasource/hive/` 删不掉，直到 `HudiUtils`(5 方法)/`HudiExternalMetaCache`/`HudiScanNode`(extends HiveScanNode)/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 全 retype 到 generic（P7.4）。 --- diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index 3efa67c5919fe0..df3e10fed5a598 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -165,3 +165,38 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 - **recon 存档**：完整 10-agent recon 结构化结果 + 补充 type-coupling recon 在本 session 的 job tmp（`p7-recon-raw.json` / `p7-recon-digest.md`，**ephemeral**）；关键结论已蒸馏进本 spec + `connectors/hive.md`。若需重跑：`.claude/wf-p7-hive-recon.js`。 - **铁律复读**：fe-core 不得新增 `if(hive)`/`instanceof HMSExternal*`/引擎名判别（memory `catalog-spi-plugindriven-no-source-specific-code`）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`）；history_schema_info nested 名 lowercase（memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 - **决策纪律**：D-004/005/019/020 + 事务桥接**已定勿重议**；OQ-* 到各子阶段 recon 后再用户签字（先中文讲背景+示例+推荐，不引任务代号）。 + +--- + +# P7.1 逐 task 拆解（code-grounded recon 完成 2026-07-06；3-agent + 直读 HEAD） + +> **recon 结论（已对照 HEAD 校正 spec）**： +> - **通用桥 = `PluginDrivenExternalCatalog`**（非独立 MetadataOps；`metadataOps==null`）。它逐条 override DDL 命令，`connector.getMetadata(session)` 拿 `ConnectorMetadata`，并**内联** cache/editlog 记账（替代旧 `ExternalMetadataOps.afterXxx()`）。→ **fe-core 侧 cache/editlog 钩子留在桥里，连接器只实现纯 SPI 方法。** +> - DDL 流向（桥 → SPI）：CREATE TABLE→`createTable(session, request)`（db=remote 名、table=SQL 名）；CREATE DB→`createDatabase(session, dbName, props)`；DROP DB→`dropDatabase(session, remoteName, ifExists, force)`；DROP TABLE→`dropTable(session, handle)`（handle 预解析，viewExists 时走 dropView）；RENAME→`renameTable(session, handle, newName)`。**IF [NOT] EXISTS 桥侧（fe-core）判定**，SPI 方法不带该 flag。 +> - **TRUNCATE = 硬缺口**：桥**未** override `truncateTable`，落到 base `ExternalCatalog.truncateTable`(:1353) → `metadataOps==null` 抛 "Truncate table is not supported"。SPI 上**根本无** `truncateTable` 方法。→ 须**加 SPI seam + 桥 override**。editlog `OP_TRUNCATE_TABLE`/`logTruncateTable(TruncateTableInfo)`/`replayTruncateTable` 已存在（复用；external 失效等价旧 `afterTruncateTable`=`refreshTableInternal(updateTime)`）。 +> - **converter = `CreateTableInfoToConnectorRequestConverter.convert(info, dbName)`**：properties **逐字**拷贝（不 parse ✅）；columns/partitionSpec(style+fields)/bucketSpec(cols+count+algorithm)/comment/ifNotExists/external 均已带。**但丢两项**：(1) **每列默认值**恒传 `null`（注释："not exposed via public getter … until SPI gains a typed carrier"）——**破坏 hive `createTableWithConstraints` 列默认值路径 + DLF guard**；(2) **LIST/RANGE partition value 定义**恒空（`initialValues=[]`）——只留 partition 列名。bucket `algorithm` 桥给 `doris_default`/`doris_random`（非 `hive_hash`；插件按此判 hash-only 即可）。 +> - **模板 = `IcebergConnectorMetadata`**（最接近：HMS-backed、DLF guard、location cleanup、插件侧 `IcebergSchemaBuilder.buildTableProperties(requestProps, catalogProps)` 装配、`context.executeAuthenticated` 包裹）。paimon 亦可参。**no-property-parsing 实践**：fe-core 从不看 property map，全插件侧解释。 +> - **调用方分布（agent C）**：`updateTableStatistics`/`updatePartitionStatistics`/`addPartitions`/`dropPartition` = **仅 `HMSTransaction` 消费**（stats/partition 写是 INSERT-commit 后回写 HMS 行数/分区，非 DDL、非 ANALYZE）。→ **本阶段不搬这 4 个 + 不动 HMSTransaction**；它们的 HmsClient 写原语**推到 P7.3** 随 HMSTransaction 落地（否则 P7.1 加进来即死代码，违背 surgical/minimal）。ANALYZE 列统计走 `HMSAnalysisTask`（读 `getTableColumnStatistics` + Doris 内部统计表）= P7.4。 +> - **rename 现状**：`HMSCachedClient` 无 rename/alterTable、`HiveMetadataOps` 无 renameImpl → **hive 今天不支持 ALTER TABLE RENAME**。→ 本阶段 `HiveConnectorMetadata.renameTable` **保持 SPI default throw**（等价现状），**不实现**。（翻闸后行为与今一致。实现前二次确认 `ExternalMetadataOps` rename 默认行为。） +> - **config 缺口**：DLF guard 插件可自足（`HmsClientConfig.METASTORE_TYPE_KEY="hive.metastore.type"` + `METASTORE_TYPE_DLF="dlf"` 已在，连接器已收到 catalog props）。但 `Config.hive_default_file_format`(="orc") + `Config.enable_create_hive_bucket_table`(=false) 是 **FE 全局 master-mutable Config、非 catalog 属性**，插件读不到、无等价物 → **见开放决策 OQ-HIVE-CFG（待用户签字）**。 +> - **SPI 面已足**：`ConnectorSession.getUser()`（owner 默认）、`getCatalogProperties()`（DLF 判别）、`ConnectorColumn` 已带 `defaultValue/nullable/comment`、`ConnectorPartitionSpec.Style{IDENTITY,TRANSFORM,LIST,RANGE}`+fields、`ConnectorBucketSpec`(cols+count+algo)、`ConnectorSchemaOps` 已有 `supportsCreateDatabase()`/`createDatabase`/`dropDatabase(force)` seam。→ **除 `truncateTable` + 列默认值 carrier 外，SPI 无须扩。** + +## 范围（本阶段 = **DDL 写路径**，不含 txn/stats 写原语；不翻闸 = 非 live，验收靠连接器单测） + +| task | 模块 | 内容 | +|---|---|---| +| **P7.1-T01** | fe-connector-api + fe-core | **truncateTable SPI seam**：`ConnectorTableOps.truncateTable(session, handle, List partitions)` default throw；`PluginDrivenExternalCatalog.truncateTable(...)` override（解析 handle→`metadata.truncateTable`→cache 失效 `refreshTableInternal(updateTime)` + editlog，复用 `OP_TRUNCATE_TABLE`；对齐 base `ExternalCatalog.truncateTable` 签名 `(db, tbl, PartitionNamesInfo, forceDrop, updateTime)`）。实现前二次确认 external truncate 的 editlog/binlog 路径细节。 | +| **P7.1-T02** | fe-connector-hms | **写 DTO（SPI-clean，仿 `HmsTableInfo`/`HmsDatabaseInfo` 风格，无 fe-core `Column`/`NameMapping`）**：table-create spec（db、table、location?、columns=`ConnectorColumn`、partitionKeys、bucketCols、numBuckets、fileFormat、comment、properties + 列默认值）；db-create spec（db、locationUri、comment、properties）。（partition/stats DTO 推 P7.3。） | +| **P7.1-T03** | fe-connector-hms | **converter（HiveUtil 等价，插件侧、fe-core-free）**：`toHiveTable`（storage-descriptor / serde / input-output-format per orc/parquet/text、compress、`tableType=MANAGED_TABLE`、`DORIS_VERSION` 常量本地化、owner/comment props、列默认值→`SQLDefaultConstraint`）、`toHiveDatabase`。**Doris→Hive 类型映射**：复用/反转 `HmsTypeMapping` 或加 `dorisTypeToHiveType` 等价（断 `HiveMetaStoreClientHelper.dorisTypeToHiveType`）。断耦：`ExternalCatalog.DORIS_VERSION`、`HiveProperties`、text 压缩 **session var**（`ConnectorSession.getProperty` 线程进）、`AnalysisException`。 | +| **P7.1-T04** | fe-connector-hms | **HmsClient 写方法（DDL only）**：`HmsClient` 接口 + `ThriftHmsClient`（走现成 `execute(...)` + auth）加 `createDatabase`、`dropDatabase`、`createTable`(含 default-constraint 分支)、`dropTable`、`truncateTable(db,tbl,List)`。（addPartitions/dropPartition/updateTableStatistics/updatePartitionStatistics 推 P7.3。） | +| **P7.1-T05** | fe-connector-hive | **HiveConnectorMetadata DB DDL**：override `supportsCreateDatabase()=true`、`createDatabase`（插件侧解析 `location` prop）、`dropDatabase(force)`（force 时枚举 drop 表，仿 iceberg/paimon cascade）。 | +| **P7.1-T06** | fe-connector-hive | **HiveConnectorMetadata createTable**（模板 iceberg）：由 request + catalog props 装 table-create spec；**插件侧 parse**：file_format 默认（OQ-HIVE-CFG）、owner 默认（`session.getUser()`）、transactional 建表拒绝、DLF 列默认值 guard（catalog prop + 列默认值）、bucket gate（OQ-HIVE-CFG）+ hash-only（algorithm≠random）、LIST-only（style==RANGE 拒）、partition-value 表达式拒绝（converter gap，见 T08）。→ `hmsClient.createTable`。 | +| **P7.1-T07** | fe-connector-hive | **HiveConnectorMetadata dropTable + truncateTable**：`dropTable(handle)`（先 `getTable` 判 `AcidUtils.isTransactionalTable` 拒事务表）→`hmsClient.dropTable`；`truncateTable(handle, partitions)`→`hmsClient.truncateTable`。**renameTable 保持 default throw（不实现，parity）。** | +| **P7.1-T08** | fe-core（shared converter） | **converter 带上每列默认值**：给 `CreateTableInfo`/列定义加 public getter，`CreateTableInfoToConnectorRequestConverter.convertColumns` 把默认值填进 `ConnectorColumn.defaultValue`（现恒 null）。additive、对 iceberg/paimon 安全（它们忽略）。（partition-value 表达式拒绝：评估是否顺带 thread `initialValues` 或退化为静默忽略——实现时定，倾向最小改动＝插件按 style 判别 + 记录降级。） | +| **P7.1-T09** | 依 OQ-HIVE-CFG | **config threading**：按用户裁定落地 `hive_default_file_format` + `enable_create_hive_bucket_table` 进插件。 | +| **P7.1-T10** | fe-connector-hms/hive 单测（无 Mockito，recording fake IMetaStoreClient） | HmsClient 写方法（create/drop db+table、truncate）；HiveConnectorMetadata DDL（createTable 装配正确性：file_format 默认/owner/bucket gate/DLF guard/partition；dropTable 事务表拒绝；createDatabase location；dropDatabase cascade）；converter 列默认值。 | +| **P7.1-T11** | 守门 + 交接 | build SUCCESS（fe-core 仅依赖 fe-connector-api；连接器注意 optional-shade HiveConf/hadoop-common）+ checkstyle 0 + import-gate 净 + 单测过；更新 HANDOFF + commit。**非 live**（翻闸 P7.5），故本阶段无 e2e DDL。 | + +## P7.1 开放决策（待用户签字） + +- **OQ-HIVE-CFG（config threading）—— ✅ 用户已裁定 2026-07-06 = 方案 A**：`hive_default_file_format`(默认 orc) + `enable_create_hive_bucket_table`(默认 false) 是 FE 全局 master-mutable Config、非 catalog 属性，插件读不到。**结论：方案 A** —— fe-core **建连接器时**把这两个全局 Config 的当前值**注入为连接器属性默认**（仅当用户未在 catalog 显式设置时；键名如 `hive.default.file.format`/`hive.enable.create.bucket.table`），插件只读属性。保留全局配置生效 + 允许 per-catalog 覆盖 + 插件不碰 `org.apache.doris.common.Config`，行为零回归。（备选 B＝纯 per-catalog 属性对齐 Trino、C＝硬编码，均**否决**：B 改用户可见行为、C 丢可调性。）→ 见 T09；注入点 = fe-core 组装 `hms` 连接器属性处（须 recon 具体 locus，勿在插件侧读 Config）。 From 18f42477d6ae0a52fbdc32d75f17e22a32911dee Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 03:33:33 +0800 Subject: [PATCH 004/334] [refactor](catalog) P7.1 hive: add TRUNCATE TABLE SPI seam + PluginDriven bridge route The generic SPI DDL bridge (PluginDrivenExternalCatalog) had no truncate route: the base ExternalCatalog.truncateTable throws "not supported" when metadataOps == null, so every SPI catalog (iceberg/paimon today) rejects TRUNCATE. Hive needs it after the P7 cutover. - ConnectorTableOps.truncateTable(session, handle, partitions): new default-throw seam (partitions null/empty = whole table). Default preserves iceberg/paimon "not supported". - PluginDrivenExternalCatalog.truncateTable(...): resolves the table by REMOTE names, dispatches to metadata.truncateTable, then emits the same TruncateTableInfo edit log the base op writes and refreshes the local table cache via RefreshManager.refreshTableInternal (mirroring legacy HiveMetadataOps.afterTruncateTable). forceDrop/rawTruncateSql carry no external semantics and are ignored, matching the legacy path. - PluginDrivenExternalCatalog.replayTruncateTable(info): refreshes the cache on follower replay (the base delegates to metadataOps.afterTruncateTable, a no-op for PluginDriven). No connector overrides truncateTable yet (HiveConnectorMetadata comes in a later P7.1 task), so behavior is unchanged pre-flip; verified by compile + checkstyle 0 + import-gate. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../connector/api/ConnectorTableOps.java | 15 +++++ .../PluginDrivenExternalCatalog.java | 58 +++++++++++++++++++ 2 files changed, 73 insertions(+) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java index 61000c4699cdf8..26b27341a99abc 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java @@ -188,6 +188,21 @@ default void renameTable(ConnectorSession session, "RENAME TABLE not supported"); } + /** + * Truncates the table identified by {@code handle}. When {@code partitions} is non-empty only those + * partitions are truncated; {@code null} / empty truncates the whole table. + * + *

Connectors that support {@code TRUNCATE TABLE} override this. The default throws, matching the + * pre-flip behavior of the generic bridge (which had no truncate route for the SPI path).

+ * + * @throws DorisConnectorException if the connector does not support truncate + */ + default void truncateTable(ConnectorSession session, + ConnectorTableHandle handle, List partitions) { + throw new DorisConnectorException( + "TRUNCATE TABLE not supported"); + } + /** * Adds a column to the table at the given position. * diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index 6fe91dd0b71bba..c57ee69da972cb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -25,6 +25,7 @@ import org.apache.doris.catalog.info.CreateOrReplaceTagInfo; import org.apache.doris.catalog.info.DropBranchInfo; import org.apache.doris.catalog.info.DropTagInfo; +import org.apache.doris.catalog.info.PartitionNamesInfo; import org.apache.doris.catalog.info.TableNameInfo; import org.apache.doris.common.DdlException; import org.apache.doris.common.ErrorCode; @@ -52,6 +53,7 @@ import org.apache.doris.persist.CreateDbInfo; import org.apache.doris.persist.DropDbInfo; import org.apache.doris.persist.DropInfo; +import org.apache.doris.persist.TruncateTableInfo; import org.apache.doris.qe.ConnectContext; import org.apache.doris.transaction.PluginDrivenTransactionManager; @@ -609,6 +611,62 @@ public void renameTable(String dbName, String oldTableName, String newTableName) afterExternalRename(dbName, oldTableName, newTableName); } + /** + * Routes {@code TRUNCATE TABLE} through the SPI's {@code ConnectorTableOps.truncateTable(session, handle, + * partitions)} instead of the base {@link ExternalCatalog#truncateTable} (which throws on + * {@code metadataOps == null}). + * + *

Resolves the table by REMOTE names for the connector (like {@link #dropTable}); {@code partitions} is + * {@code null} for a whole-table truncate or the named partitions otherwise. On success it emits the same + * {@link TruncateTableInfo} edit log the base op writes and refreshes the local table cache (mirroring legacy + * {@code HiveMetadataOps.afterTruncateTable -> RefreshManager.refreshTableInternal}); followers refresh via + * {@link #replayTruncateTable}. {@code forceDrop} / {@code rawTruncateSql} carry no external semantics (the + * connector truncates the remote table directly) and are ignored, matching the legacy path.

+ */ + @Override + public void truncateTable(String dbName, String tableName, PartitionNamesInfo partitionNamesInfo, + boolean forceDrop, String rawTruncateSql) throws DdlException { + makeSureInitialized(); + ExternalDatabase db = getDbNullable(dbName); + if (db == null) { + throw new DdlException("Failed to get database: '" + dbName + "' in catalog: " + getName()); + } + ExternalTable dorisTable = db.getTableNullable(tableName); + if (dorisTable == null) { + throw new DdlException("Failed to get table: '" + tableName + "' in database: " + dbName); + } + List partitions = partitionNamesInfo == null ? null : partitionNamesInfo.getPartitionNames(); + ConnectorSession session = buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorTableHandle handle = resolveAlterHandle(dorisTable, session, metadata); + try { + metadata.truncateTable(session, handle, partitions); + } catch (DorisConnectorException e) { + throw new DdlException(e.getMessage(), e); + } + long updateTime = System.currentTimeMillis(); + // Cache refresh + edit log use the LOCAL db/table names for follower-replay parity (only the + // connector-bound handle is remote-resolved), mirroring base ExternalCatalog.truncateTable. + Env.getCurrentEnv().getRefreshManager().refreshTableInternal(db, dorisTable, updateTime); + Env.getCurrentEnv().getEditLog().logTruncateTable( + new TruncateTableInfo(getName(), dbName, tableName, partitions, updateTime)); + LOG.info("finished to truncate table {}.{}.{}", getName(), dbName, tableName); + } + + /** + * Refreshes the local table cache on edit-log replay of a connector-driven truncate. The base + * {@link ExternalCatalog#replayTruncateTable} delegates to {@code metadataOps.afterTruncateTable}, which is a + * no-op for PluginDriven ({@code metadataOps == null}); this override re-resolves the cached table by the + * replayed LOCAL names and runs {@code refreshTableInternal} (the same effect the master path applied), + * mirroring legacy {@code HiveMetadataOps.afterTruncateTable}. + */ + @Override + public void replayTruncateTable(TruncateTableInfo info) { + getDbForReplay(info.getDb()).ifPresent(db -> + db.getTableForReplay(info.getTable()).ifPresent(tbl -> + Env.getCurrentEnv().getRefreshManager().refreshTableInternal(db, tbl, info.getUpdateTime()))); + } + /** * Routes {@code ALTER TABLE ... ADD/DROP/RENAME/MODIFY/REORDER COLUMN} through the SPI's * {@code ConnectorTableOps} column-evolution methods instead of the legacy {@code metadataOps} path used From 72baf6075c2a5486b52f92dcac989910b05231cb Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 03:35:23 +0800 Subject: [PATCH 005/334] =?UTF-8?q?[doc](catalog)=20P7.1=20hive:=20mark=20?= =?UTF-8?q?T01=20done=20+=20stash=20T03=20porting=20pointers;=20handoff=20?= =?UTF-8?q?=E2=86=92=20T02?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Update rolling handoff + task spec: T01 (TRUNCATE SPI seam) done & green (c0222977ebd); next session resumes at T02 (plugin write DTOs) → T03/T04 (write client). Archive the T03 converter porting pointers (HiveUtil line refs + the fe-core couplings to break: dorisTypeToHiveType inverse over ConnectorType, DORIS_VERSION const, HiveProperties, hiveTextCompression session var, format/compression tables) so the next session does not re-recon. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 12 +++++++++--- plan-doc/tasks/P7-hive-migration.md | 14 ++++++++++++++ 2 files changed, 23 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index b9d9a3270232ef..20fe6123fa5a69 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,9 +5,15 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.1 起步 recon + 逐 task 拆解 + 唯一阻塞决策已裁定；下一步 = 按 P7.1-T01…T11 写代码（尚无代码改动）** +# 🎯 当前状态（2026-07-06）= **P7.1 进行中：recon+拆解+决策已定，T01（TRUNCATE SPI seam）已实现+全绿+提交；下一步 = T02→T04 插件写客户端** -> **本轮（P7.1 recon + 拆解 session）做了什么**：对 P7.1 范围做 code-grounded recon（3 并行 agent + 直读 HEAD，~225k subagent token）核清 4 件事并**校正了 spec**：(1) 通用 DDL 桥 = `PluginDrivenExternalCatalog`（非独立 MetadataOps，逐条 override + 内联 cache/editlog 钩子，连接器只实现纯 SPI）；(2) **TRUNCATE 是硬缺口**（SPI 无 `truncateTable`、桥未 override）；(3) shared converter `CreateTableInfoToConnectorRequestConverter` 逐字拷 properties（✅ 不 parse）但**丢每列默认值**（恒 null，破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value 定义；(4) stats/partition 写 4 法**仅 HMSTransaction 消费**→**推 P7.3**（本阶段加进来即死代码），rename **hive 今天不支持**→保持 SPI default throw。产出 **`tasks/P7-hive-migration.md` 末尾 P7.1 逐 task 拆解（T01–T11）+ recon 结论块**。**唯一阻塞决策 OQ-HIVE-CFG 已由用户裁定 = 方案 A**（fe-core 建连接器时把 `hive_default_file_format`/`enable_create_hive_bucket_table` 两个全局 Config 值注入为连接器属性默认，插件只读属性、零行为回归）。**本轮无代码改动**（仅文档 + 决策）。 +> **本轮（P7.1 recon + 拆解 + T01 session）做了什么**： +> 1. **P7.1 code-grounded recon**（3 并行 agent + 直读 HEAD，~225k subagent token）核清并**校正 spec** 4 件事：(1) 通用 DDL 桥 = `PluginDrivenExternalCatalog`（非独立 MetadataOps，逐条 override + 内联 cache/editlog，连接器只实现纯 SPI）；(2) **TRUNCATE 硬缺口**（SPI 无 `truncateTable`、桥未 override）；(3) shared converter 逐字拷 properties（✅ 不 parse）但**丢每列默认值**（恒 null → 破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value；(4) stats/partition 写 4 法**仅 HMSTransaction 消费**→推 P7.3，rename **hive 今天不支持**→保持 SPI default throw。 +> 2. 产出 **`tasks/P7-hive-migration.md` 末尾 P7.1 逐 task 拆解（T01–T11）+ recon 结论块**。 +> 3. **决策 OQ-HIVE-CFG 用户裁定 = 方案 A**（fe-core 建连接器时注入两个全局 Config 值为属性默认，插件只读属性、零行为回归）。 +> 4. **T01 已实现并全绿提交（`c0222977ebd`）**：加 `ConnectorTableOps.truncateTable(session, handle, partitions)` default-throw seam + `PluginDrivenExternalCatalog.truncateTable`（remote 解析 → `metadata.truncateTable` → `TruncateTableInfo` editlog + `refreshTableInternal` 缓存刷新，仿 legacy `afterTruncateTable`）+ `replayTruncateTable`（follower 缓存刷新）。compile SUCCESS + checkstyle 0（fe-connector-api,fe-core）+ import-gate 净。**尚无连接器 override truncate（等 T07），翻闸前行为不变。** +> +> **T02–T11 未动**（下一 session 接手；T03 类型映射/转换器移植最精细，故留新 context 谨慎做，不在长 session 尾仓促）。 > **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： > P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 @@ -36,7 +42,7 @@ ## P7.1 执行要点（下个 session 开场） -1. **先读** spec 末尾「P7.1 逐 task 拆解」块（recon 结论 + T01–T11）。按 **T01(truncate SPI seam+桥 override) → T02(hms 写 DTO) → T03(hms converter，HiveUtil 等价插件侧) → T04(HmsClient 写方法 DDL-only) → T05(HiveConnectorMetadata DB DDL) → T06(createTable) → T07(dropTable+truncate；rename 保持 default throw) → T08(shared converter 带列默认值) → T09(config 注入=方案A) → T10(连接器单测) → T11(守门+交接)** 顺序推。**信 HEAD 控制流不信 spec 行号。** +1. **先读** spec 末尾「P7.1 逐 task 拆解」块 + 「T03 移植指针」块（recon 存档，勿再 recon）。**T01 已完成**（truncate SPI seam + 桥 override，`c0222977ebd`）。**从 T02 起接**：T02(hms 写 DTO) → T03(hms converter，HiveUtil 等价插件侧，**最精细**，指针见 spec) → T04(HmsClient 写方法 DDL-only) → T05(HiveConnectorMetadata DB DDL) → T06(createTable) → T07(dropTable+truncate override；rename 保持 default throw) → T08(shared converter 带列默认值) → T09(config 注入=方案A) → T10(连接器单测) → T11(守门+交接)。**信 HEAD 控制流不信 spec 行号。** T02+T03+T04 = 一个自洽单元（插件写客户端），可合并一次 commit。 2. **本阶段范围 = 仅 DDL 写路径**：create/drop db+table + truncate。**stats/partition 写 4 法（updateTableStatistics/updatePartitionStatistics/addPartitions/dropPartition）推 P7.3**（仅 HMSTransaction 消费，本阶段加即死代码）。**rename 不实现**（hive 今天不支持，保持 SPI default throw）。**no-property-parsing**：file_format 默认/owner/bucket-gate/DLF-guard/transactional-reject 全插件侧（`ConnectorSession.getUser()`+`getCatalogProperties()`+`ConnectorColumn.defaultValue` 已够，无须扩 SPI，除 truncate seam + T08 列默认值 carrier）。 3. **OQ-HIVE-CFG 已裁定 = 方案 A**（T09）：fe-core 组装 `hms` 连接器属性处，把 `Config.hive_default_file_format`/`Config.enable_create_hive_bucket_table` 当前值注入为属性默认（仅用户未显式设时），**插件只读属性、不碰 `common.Config`**。注入 locus 待 recon。 4. **勿重议的已定决策**：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransaction`。后续子阶段 OQ-* 到 recon 后再用户签字（中文讲背景+示例+推荐、不引任务代号）。 diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index df3e10fed5a598..259923f54b90e7 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -199,4 +199,18 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 ## P7.1 开放决策（待用户签字） +## P7.1 实现进度 + T03 移植指针（recon 存档，勿再 recon） + +- **T01 ✅ 已提交 `c0222977ebd`**：`ConnectorTableOps.truncateTable(session, handle, partitions)` default-throw + `PluginDrivenExternalCatalog.truncateTable`/`replayTruncateTable`（editlog `TruncateTableInfo` + `refreshTableInternal`；base 签名 = `truncateTable(String db, String tbl, PartitionNamesInfo, boolean forceDrop, String rawTruncateSql)`，`forceDrop`/`rawTruncateSql` 无 external 语义故忽略）。全绿。 +- **T02–T11 未动**（下 session 从 T02 起）。 +- **T03（converter，最精细）移植指针（HEAD 行号，信控制流）**：源在 `datasource/hive/HiveUtil.java` —— `toHiveTable`(:227)、`setCompressType`(:259)、`toHiveStorageDesc`(:286)、`setFileFormat`(:301)、`toHiveSchema`(:327)、`toHiveDatabase`(:346)。**必须断的 fe-core 耦合**： + - `HiveMetaStoreClientHelper.dorisTypeToHiveType(Type)`（`HiveMetaStoreClientHelper.java:529`，walk scalar/array/map/struct → hive 类型串；char(len)/decimal(p,s)；date←DATE/DATEV2、timestamp←DATETIME/DATETIMEV2；VARCHAR/STRING→string）→ **插件侧要写 `ConnectorType` 版逆映射**（`HmsTypeMapping` 是正向 hive-str→ConnectorType，需补逆向；~80 行对称）。 + - `ExternalCatalog.DORIS_VERSION` / `DORIS_VERSION_VALUE` → 插件本地常量。 + - `HiveProperties.setTableProperties(table, props)` → 核实其语义（大概率 `table.setParameters`）后插件内联。 + - text 压缩默认 `ConnectContext.get().getSessionVariable().hiveTextCompression()` → 经 `ConnectorSession`（`getProperty`/session 属性）线程进，别在插件 import fe-core。 + - `AnalysisException` → `DorisConnectorException`/`IllegalArgumentException`。 + - format 表（orc/parquet/text 的 input/output format + serde，见 :301-325）+ 压缩默认（parquet→snappy、orc→zlib、text→session var）逐字照搬。 + - `toHiveTable` 里：`tableType="MANAGED_TABLE"`、`props.put(DORIS_VERSION,…)`、`props.put("comment", …)`、`owner`→`table.setOwner`、createTime=`(int)(System.currentTimeMillis()*1000)`（注意原代码此处有整型溢出但**照搬保持一致**）、sd.parameters 里 `tag="doris external hive table"`。 + - columns = **全列**（数据+分区），`toHiveSchema` 用 partitionKeys 名集切分数据列/分区列。 + - **OQ-HIVE-CFG（config threading）—— ✅ 用户已裁定 2026-07-06 = 方案 A**：`hive_default_file_format`(默认 orc) + `enable_create_hive_bucket_table`(默认 false) 是 FE 全局 master-mutable Config、非 catalog 属性，插件读不到。**结论：方案 A** —— fe-core **建连接器时**把这两个全局 Config 的当前值**注入为连接器属性默认**（仅当用户未在 catalog 显式设置时；键名如 `hive.default.file.format`/`hive.enable.create.bucket.table`），插件只读属性。保留全局配置生效 + 允许 per-catalog 覆盖 + 插件不碰 `org.apache.doris.common.Config`，行为零回归。（备选 B＝纯 per-catalog 属性对齐 Trino、C＝硬编码，均**否决**：B 改用户可见行为、C 丢可调性。）→ 见 T09；注入点 = fe-core 组装 `hms` 连接器属性处（须 recon 具体 locus，勿在插件侧读 Config）。 From 33d87402f39f26a1f89a65dbf5c3a4cb2f08b1dc Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 06:43:49 +0800 Subject: [PATCH 006/334] [refactor](catalog) P7.1 hive: add plugin-side HMS DDL write client (create/drop db+table, truncate) + converter Builds the SPI-clean write client in fe-connector-hms so the hive connector can issue CREATE/DROP DATABASE, CREATE/DROP TABLE and TRUNCATE without fe-core. The metadata layer (HiveConnectorMetadata overrides), config threading and the recording-fake client tests land in follow-up steps; this stays non-live (no cutover until the hive connector is flipped on). - HmsCreateTableRequest / HmsCreateDatabaseRequest: write-side specs mirroring legacy HiveTableMetadata / HiveDatabaseMetadata, connector-api + JDK types only. Columns carry data + partition columns; per-column defaults ride on ConnectorColumn.getDefaultValue(). Text-compression default and doris.version are threaded on the request (the plugin must not import fe-core Config/Version). - HmsTypeMapping.toHiveTypeString: reverse of the read mapping, SPI-clean equivalent of HiveMetaStoreClientHelper.dorisTypeToHiveType. Switches on the Doris PrimitiveType names ConnectorColumnConverter emits; unsupported types throw (parity with legacy) rather than emit a bogus type. - HmsWriteConverter: faithful port of HiveUtil.toHiveTable/toHiveDatabase plus HiveProperties.setTableProperties (serde vs table property split). Per-format input/output/serde + compression defaults copied verbatim; MANAGED_TABLE type and the "doris external hive table" storage tag preserved. - HmsClient: add createDatabase/dropDatabase/createTable/dropTable/truncateTable as default-throwing seams so read-only implementations (partition-pruning test fakes) keep compiling; ThriftHmsClient overrides all five via the existing execute(...) auth+pool framework, building SQLDefaultConstraint from column defaults (equivalent to legacy createTableWithConstraints). Verify: fe-connector-hms compile SUCCESS + checkstyle 0 (main+test) + check-connector-imports clean + 28 unit tests green (reverse type mapping + converter). No connector overrides these yet, so behavior is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../apache/doris/connector/hms/HmsClient.java | 60 ++++ .../hms/HmsCreateDatabaseRequest.java | 71 +++++ .../connector/hms/HmsCreateTableRequest.java | 237 +++++++++++++++ .../doris/connector/hms/HmsTypeMapping.java | 96 ++++++ .../connector/hms/HmsWriteConverter.java | 278 ++++++++++++++++++ .../doris/connector/hms/ThriftHmsClient.java | 71 +++++ .../connector/hms/HmsTypeMappingTest.java | 85 ++++++ .../connector/hms/HmsWriteConverterTest.java | 225 ++++++++++++++ 8 files changed, 1123 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateDatabaseRequest.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateTableRequest.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java index 26b2263bb99fda..af728e9c433eeb 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java @@ -135,4 +135,64 @@ List getPartitions(String dbName, String tableName, */ HmsPartitionInfo getPartition(String dbName, String tableName, List values); + + // ========== Phase 3: DDL / write operations ========== + // + // Optional operations: default to throwing so read-only implementations (the partition-pruning + // test fakes, and connectors that never issue DDL) need not implement them. The production + // {@link ThriftHmsClient} overrides all of them. + + /** + * Create a database. + * + * @param request database create spec + * @throws HmsClientException if the operation fails + */ + default void createDatabase(HmsCreateDatabaseRequest request) { + throw new UnsupportedOperationException("createDatabase is not supported by this client"); + } + + /** + * Drop a database. Callers must have already handled IF EXISTS / cascade semantics. + * + * @param dbName database name + * @throws HmsClientException if the operation fails + */ + default void dropDatabase(String dbName) { + throw new UnsupportedOperationException("dropDatabase is not supported by this client"); + } + + /** + * Create a table. When any column carries a default value, it is registered as a Hive default + * constraint (equivalent to legacy {@code createTableWithConstraints}). + * + * @param request table create spec + * @throws HmsClientException if the table already exists or the operation fails + */ + default void createTable(HmsCreateTableRequest request) { + throw new UnsupportedOperationException("createTable is not supported by this client"); + } + + /** + * Drop a table. Callers must have already handled IF EXISTS and transactional-table rejection. + * + * @param dbName database name + * @param tableName table name + * @throws HmsClientException if the operation fails + */ + default void dropTable(String dbName, String tableName) { + throw new UnsupportedOperationException("dropTable is not supported by this client"); + } + + /** + * Truncate a table, or the given partitions of it. + * + * @param dbName database name + * @param tableName table name + * @param partitions partition names to truncate; empty/null truncates the whole table + * @throws HmsClientException if the operation fails + */ + default void truncateTable(String dbName, String tableName, List partitions) { + throw new UnsupportedOperationException("truncateTable is not supported by this client"); + } } diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateDatabaseRequest.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateDatabaseRequest.java new file mode 100644 index 00000000000000..b89f50fcc15ead --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateDatabaseRequest.java @@ -0,0 +1,71 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import java.util.Collections; +import java.util.Map; +import java.util.Objects; + +/** + * Write-side spec describing a database to create in the metastore. + * + *

The write-side counterpart of the read DTO {@link HmsDatabaseInfo}. It is + * SPI-clean (only JDK types) and mirrors fe-core's {@code HiveDatabaseMetadata} + * without depending on any fe-core class. A connector plugin assembles it from + * a CREATE DATABASE request and hands it to {@link HmsClient#createDatabase}.

+ */ +public final class HmsCreateDatabaseRequest { + + private final String dbName; + private final String locationUri; + private final String comment; + private final Map properties; + + public HmsCreateDatabaseRequest(String dbName, String locationUri, + String comment, Map properties) { + this.dbName = Objects.requireNonNull(dbName, "dbName"); + this.locationUri = locationUri; + this.comment = comment; + this.properties = properties == null + ? Collections.emptyMap() + : Collections.unmodifiableMap(properties); + } + + public String getDbName() { + return dbName; + } + + /** Optional database location URI; {@code null} when the metastore should pick a default. */ + public String getLocationUri() { + return locationUri; + } + + /** Database comment; never {@code null} (empty when unset). */ + public String getComment() { + return comment == null ? "" : comment; + } + + public Map getProperties() { + return properties; + } + + @Override + public String toString() { + return "HmsCreateDatabaseRequest{" + dbName + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateTableRequest.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateTableRequest.java new file mode 100644 index 00000000000000..8a9994b0728bab --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCreateTableRequest.java @@ -0,0 +1,237 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.api.ConnectorColumn; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * Write-side spec describing a Hive-compatible table to create in the metastore. + * + *

The write-side counterpart of the read DTO {@link HmsTableInfo}. It is + * SPI-clean (connector-api + JDK types only) and mirrors fe-core's + * {@code HiveTableMetadata} without depending on any fe-core class. A connector + * plugin assembles it from a CREATE TABLE request (resolving file format, owner, + * location, bucketing and the text-compression default on its side) and hands it + * to {@link HmsClient#createTable}.

+ * + *

{@link #getColumns()} carries all columns (data + partition, in + * declaration order); {@link #getPartitionKeys()} lists the names of the ones that + * are partition keys. The write converter splits them apart, matching legacy + * {@code HiveUtil.toHiveSchema}. Per-column default values ride on the + * {@link ConnectorColumn#getDefaultValue()} of each column.

+ */ +public final class HmsCreateTableRequest { + + private final String dbName; + private final String tableName; + private final String location; + private final List columns; + private final List partitionKeys; + private final List bucketCols; + private final int numBuckets; + private final String fileFormat; + private final String comment; + private final Map properties; + private final String defaultTextCompression; + private final String dorisVersion; + + private HmsCreateTableRequest(Builder b) { + this.dbName = Objects.requireNonNull(b.dbName, "dbName"); + this.tableName = Objects.requireNonNull(b.tableName, "tableName"); + this.location = b.location; + this.columns = b.columns == null + ? Collections.emptyList() + : Collections.unmodifiableList(b.columns); + this.partitionKeys = b.partitionKeys == null + ? Collections.emptyList() + : Collections.unmodifiableList(b.partitionKeys); + this.bucketCols = b.bucketCols == null + ? Collections.emptyList() + : Collections.unmodifiableList(b.bucketCols); + this.numBuckets = b.numBuckets; + this.fileFormat = Objects.requireNonNull(b.fileFormat, "fileFormat"); + this.comment = b.comment; + this.properties = b.properties == null + ? Collections.emptyMap() + : Collections.unmodifiableMap(b.properties); + this.defaultTextCompression = b.defaultTextCompression; + this.dorisVersion = b.dorisVersion; + } + + public String getDbName() { + return dbName; + } + + public String getTableName() { + return tableName; + } + + /** Optional table location; {@code null} when the metastore should pick a default under the db. */ + public String getLocation() { + return location; + } + + /** All columns (data + partition keys) in declaration order. */ + public List getColumns() { + return columns; + } + + /** Names of the columns that are partition keys (subset of {@link #getColumns()} names). */ + public List getPartitionKeys() { + return partitionKeys; + } + + public List getBucketCols() { + return bucketCols; + } + + public int getNumBuckets() { + return numBuckets; + } + + /** File format string: one of "orc", "parquet", "text" (case-insensitive). */ + public String getFileFormat() { + return fileFormat; + } + + /** Table comment; never {@code null} (empty when unset). */ + public String getComment() { + return comment == null ? "" : comment; + } + + public Map getProperties() { + return properties; + } + + /** + * The compression to fall back to for a {@code text} table when the user did not set a + * {@code compression} property. The connector resolves it from its session (legacy read the + * {@code hive_text_compression} session variable); {@code null} falls back to "plain". + */ + public String getDefaultTextCompression() { + return defaultTextCompression; + } + + /** + * The value stamped into the {@code doris.version} table parameter. The connector sources it from + * its injected properties (fe-core has the build version; the plugin must not import it). When + * {@code null}/empty the write converter omits the parameter. + */ + public String getDorisVersion() { + return dorisVersion; + } + + @Override + public String toString() { + return "HmsCreateTableRequest{" + dbName + "." + tableName + "}"; + } + + public static Builder builder() { + return new Builder(); + } + + /** + * Builder for {@link HmsCreateTableRequest}. + */ + public static final class Builder { + private String dbName; + private String tableName; + private String location; + private List columns; + private List partitionKeys; + private List bucketCols; + private int numBuckets; + private String fileFormat; + private String comment; + private Map properties; + private String defaultTextCompression; + private String dorisVersion; + + private Builder() { + } + + public Builder dbName(String val) { + this.dbName = val; + return this; + } + + public Builder tableName(String val) { + this.tableName = val; + return this; + } + + public Builder location(String val) { + this.location = val; + return this; + } + + public Builder columns(List val) { + this.columns = val; + return this; + } + + public Builder partitionKeys(List val) { + this.partitionKeys = val; + return this; + } + + public Builder bucketCols(List val) { + this.bucketCols = val; + return this; + } + + public Builder numBuckets(int val) { + this.numBuckets = val; + return this; + } + + public Builder fileFormat(String val) { + this.fileFormat = val; + return this; + } + + public Builder comment(String val) { + this.comment = val; + return this; + } + + public Builder properties(Map val) { + this.properties = val; + return this; + } + + public Builder defaultTextCompression(String val) { + this.defaultTextCompression = val; + return this; + } + + public Builder dorisVersion(String val) { + this.dorisVersion = val; + return this; + } + + public HmsCreateTableRequest build() { + return new HmsCreateTableRequest(this); + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTypeMapping.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTypeMapping.java index 45f10809356c00..a4df2a7a8df0e0 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTypeMapping.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTypeMapping.java @@ -21,6 +21,7 @@ import java.util.ArrayList; import java.util.List; +import java.util.Locale; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -195,6 +196,101 @@ private static ConnectorType toConnectorTypeInternal(String lowerType, return ConnectorType.of("UNSUPPORTED"); } + /** + * Convert a {@link ConnectorType} to its Hive type string. This is the reverse direction of + * {@link #toConnectorType(String, Options)} for the shapes a Doris CREATE TABLE emits, and the + * SPI-clean equivalent of fe-core's {@code HiveMetaStoreClientHelper.dorisTypeToHiveType()}. + * + *

Scalar type names are Doris {@code PrimitiveType} names, matching what fe-core's + * {@code ConnectorColumnConverter.toConnectorType} emits ({@code PrimitiveType.toString()}); complex + * types use "ARRAY"/"MAP"/"STRUCT" with populated children. CHAR carries its length in the precision + * field (mirroring the create-request encoding). VARCHAR and STRING both map to Hive {@code string}, + * matching legacy.

+ * + * @throws IllegalArgumentException for a type Hive tables cannot represent — matching legacy, which + * threw for the same unsupported primitive/complex shapes. + */ + public static String toHiveTypeString(ConnectorType type) { + String name = type.getTypeName().toUpperCase(Locale.ROOT); + switch (name) { + case "ARRAY": { + List children = type.getChildren(); + if (children.isEmpty()) { + throw new IllegalArgumentException("Unsupported type conversion of " + type); + } + return "array<" + toHiveTypeString(children.get(0)) + ">"; + } + case "MAP": { + List children = type.getChildren(); + if (children.size() < 2) { + throw new IllegalArgumentException("Unsupported type conversion of " + type); + } + return "map<" + toHiveTypeString(children.get(0)) + "," + + toHiveTypeString(children.get(1)) + ">"; + } + case "STRUCT": { + List children = type.getChildren(); + List fieldNames = type.getFieldNames(); + StringBuilder sb = new StringBuilder("struct<"); + for (int i = 0; i < children.size(); i++) { + if (i > 0) { + sb.append(","); + } + String fieldName = i < fieldNames.size() ? fieldNames.get(i) : "col" + i; + sb.append(fieldName).append(":").append(toHiveTypeString(children.get(i))); + } + sb.append(">"); + return sb.toString(); + } + default: + return scalarToHiveTypeString(name, type); + } + } + + private static String scalarToHiveTypeString(String name, ConnectorType type) { + switch (name) { + case "BOOLEAN": + return "boolean"; + case "TINYINT": + return "tinyint"; + case "SMALLINT": + return "smallint"; + case "INT": + return "int"; + case "BIGINT": + return "bigint"; + case "DATE": + case "DATEV2": + return "date"; + case "DATETIME": + case "DATETIMEV2": + return "timestamp"; + case "FLOAT": + return "float"; + case "DOUBLE": + return "double"; + case "CHAR": + return "char(" + type.getPrecision() + ")"; + case "VARCHAR": + case "STRING": + return "string"; + case "DECIMALV2": + case "DECIMAL32": + case "DECIMAL64": + case "DECIMAL128": + case "DECIMAL256": + case "DECIMALV3": { + int precision = type.getPrecision(); + if (precision == 0) { + precision = DEFAULT_DECIMAL_PRECISION; + } + return "decimal(" + precision + "," + type.getScale() + ")"; + } + default: + throw new IllegalArgumentException("Unsupported type conversion of " + type); + } + } + /** * Find the index of the next top-level comma separator in a * comma-separated nested type string. Respects angle-bracket and diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java new file mode 100644 index 00000000000000..87e924c442b6d2 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java @@ -0,0 +1,278 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.api.ConnectorColumn; + +import org.apache.hadoop.hive.metastore.api.Database; +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.PrincipalType; +import org.apache.hadoop.hive.metastore.api.SerDeInfo; +import org.apache.hadoop.hive.metastore.api.StorageDescriptor; +import org.apache.hadoop.hive.metastore.api.Table; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.Set; + +/** + * Builds Hive metastore API objects ({@link Table}, {@link Database}) from the connector-SPI write specs. + * + *

This is the SPI-clean equivalent of fe-core's {@code HiveUtil.toHiveTable}/{@code toHiveDatabase} + * (plus the serde/table property split from {@code HiveProperties.setTableProperties}). It takes only + * {@link HmsCreateTableRequest}/{@link HmsCreateDatabaseRequest} and connector-api types, with no fe-core + * dependency. Doris→Hive type strings come from {@link HmsTypeMapping#toHiveTypeString}.

+ * + *

Behavior is a faithful port of the legacy code: storage-descriptor / serde / input-output-format per + * orc/parquet/text, per-format compression defaults, {@code MANAGED_TABLE} table type, and the + * {@code "doris external hive table"} storage-descriptor tag.

+ */ +public final class HmsWriteConverter { + + /** Table parameter key stamping which Doris build created the table (legacy {@code DORIS_VERSION}). */ + static final String DORIS_VERSION_KEY = "doris.version"; + /** User-facing compression property key consumed while building the table (removed afterwards). */ + static final String COMPRESSION_KEY = "compression"; + /** Fallback text compression when neither the request nor a session default supplies one. */ + static final String DEFAULT_TEXT_COMPRESSION = "plain"; + + private static final Set SUPPORTED_ORC_COMPRESSIONS = + unmodifiableSet("plain", "zlib", "snappy", "zstd", "lz4"); + private static final Set SUPPORTED_PARQUET_COMPRESSIONS = + unmodifiableSet("plain", "snappy", "zstd", "lz4"); + private static final Set SUPPORTED_TEXT_COMPRESSIONS = + unmodifiableSet("plain", "gzip", "zstd", "bzip2", "lz4", "snappy"); + + // Property keys that belong on the SerDe (not the table). Mirrors fe-core HiveProperties.HIVE_SERDE_PROPERTIES. + private static final Set HIVE_SERDE_PROPERTIES = unmodifiableSet( + "field.delim", + "colelction.delim", // Hive 2 (legacy metastore typo, kept for parity) + "collection.delim", // Hive 3 + "separatorChar", // OpenCSVSerde.SEPARATORCHAR + "serialization.format", + "line.delim", + "quoteChar", // OpenCSVSerde.QUOTECHAR + "mapkey.delim", + "escape.delim", + "escapeChar", // OpenCSVSerde.ESCAPECHAR + "serialization.null.format", + "skip.header.line.count", + "skip.footer.line.count"); + + private HmsWriteConverter() { + } + + /** + * Build a Hive metastore {@link Table} from a create-table request. Faithful port of legacy + * {@code HiveUtil.toHiveTable}. + */ + public static Table toHiveTable(HmsCreateTableRequest req) { + Objects.requireNonNull(req.getDbName(), "Hive database name should be not null"); + Objects.requireNonNull(req.getTableName(), "Hive table name should be not null"); + Table table = new Table(); + table.setDbName(req.getDbName()); + table.setTableName(req.getTableName()); + // Legacy overflows the millis cast to int before multiplying; kept verbatim for byte-for-byte parity. + int createTime = (int) System.currentTimeMillis() * 1000; + table.setCreateTime(createTime); + table.setLastAccessTime(createTime); + Set partitionSet = new HashSet<>(req.getPartitionKeys()); + SchemaSplit schema = toHiveSchema(req.getColumns(), partitionSet); + + table.setSd(toHiveStorageDesc(schema.dataColumns, req.getBucketCols(), req.getNumBuckets(), + req.getFileFormat(), req.getLocation())); + table.setPartitionKeys(schema.partitionColumns); + + table.setTableType("MANAGED_TABLE"); + Map props = new HashMap<>(req.getProperties()); + // Legacy always stamps DORIS_VERSION; the plugin cannot compute the build version (no fe-core import), + // so it is threaded on the request and stamped only when supplied. See HmsCreateTableRequest#getDorisVersion. + if (req.getDorisVersion() != null && !req.getDorisVersion().isEmpty()) { + props.put(DORIS_VERSION_KEY, req.getDorisVersion()); + } + setCompressType(req, props); + // set hive table comment by table properties + props.put("comment", req.getComment()); + if (props.containsKey("owner")) { + table.setOwner(props.get("owner")); + } + setTableProperties(table, props); + return table; + } + + /** + * Build a Hive metastore {@link Database} from a create-database request. Faithful port of legacy + * {@code HiveUtil.toHiveDatabase}. + */ + public static Database toHiveDatabase(HmsCreateDatabaseRequest req) { + Database database = new Database(); + database.setName(req.getDbName()); + String locationUri = req.getLocationUri(); + if (locationUri != null && !locationUri.isEmpty()) { + database.setLocationUri(locationUri); + } + Map props = new HashMap<>(req.getProperties()); + database.setParameters(props); + database.setDescription(req.getComment()); + if (props.containsKey("owner")) { + database.setOwnerName(props.get("owner")); + database.setOwnerType(PrincipalType.USER); + } + return database; + } + + private static void setCompressType(HmsCreateTableRequest req, Map props) { + String fileFormat = req.getFileFormat(); + String compression = props.get(COMPRESSION_KEY); + // on HMS, default orc compression type is zlib and default parquet compression type is snappy. + if (fileFormat.equalsIgnoreCase("parquet")) { + if (isNotEmpty(compression) && !SUPPORTED_PARQUET_COMPRESSIONS.contains(compression)) { + throw new IllegalArgumentException("Unsupported parquet compression type " + compression); + } + props.putIfAbsent("parquet.compression", isEmpty(compression) ? "snappy" : compression); + } else if (fileFormat.equalsIgnoreCase("orc")) { + if (isNotEmpty(compression) && !SUPPORTED_ORC_COMPRESSIONS.contains(compression)) { + throw new IllegalArgumentException("Unsupported orc compression type " + compression); + } + props.putIfAbsent("orc.compress", isEmpty(compression) ? "zlib" : compression); + } else if (fileFormat.equalsIgnoreCase("text")) { + if (isNotEmpty(compression) && !SUPPORTED_TEXT_COMPRESSIONS.contains(compression)) { + throw new IllegalArgumentException("Unsupported text compression type " + compression); + } + String textDefault = isEmpty(req.getDefaultTextCompression()) + ? DEFAULT_TEXT_COMPRESSION : req.getDefaultTextCompression(); + props.putIfAbsent("text.compression", isEmpty(compression) ? textDefault : compression); + } else { + throw new IllegalArgumentException("Compression is not supported on " + fileFormat); + } + // remove if exists + props.remove(COMPRESSION_KEY); + } + + private static StorageDescriptor toHiveStorageDesc(List columns, + List bucketCols, int numBuckets, String fileFormat, String location) { + StorageDescriptor sd = new StorageDescriptor(); + sd.setCols(columns); + setFileFormat(fileFormat, sd); + if (location != null) { + sd.setLocation(location); + } + sd.setBucketCols(bucketCols); + sd.setNumBuckets(numBuckets); + Map parameters = new HashMap<>(); + parameters.put("tag", "doris external hive table"); + sd.setParameters(parameters); + return sd; + } + + private static void setFileFormat(String fileFormat, StorageDescriptor sd) { + String inputFormat; + String outputFormat; + String serDe; + if (fileFormat.equalsIgnoreCase("orc")) { + inputFormat = "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + outputFormat = "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat"; + serDe = "org.apache.hadoop.hive.ql.io.orc.OrcSerde"; + } else if (fileFormat.equalsIgnoreCase("parquet")) { + inputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + outputFormat = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat"; + serDe = "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"; + } else if (fileFormat.equalsIgnoreCase("text")) { + inputFormat = "org.apache.hadoop.mapred.TextInputFormat"; + outputFormat = "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"; + serDe = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; + } else { + throw new IllegalArgumentException("Creating table with an unsupported file format: " + fileFormat); + } + SerDeInfo serDeInfo = new SerDeInfo(); + serDeInfo.setSerializationLib(serDe); + sd.setSerdeInfo(serDeInfo); + sd.setInputFormat(inputFormat); + sd.setOutputFormat(outputFormat); + } + + private static SchemaSplit toHiveSchema(List columns, Set partitionSet) { + List hiveCols = new ArrayList<>(); + List hiveParts = new ArrayList<>(); + for (ConnectorColumn column : columns) { + FieldSchema hiveFieldSchema = new FieldSchema(); + hiveFieldSchema.setType(HmsTypeMapping.toHiveTypeString(column.getType())); + hiveFieldSchema.setName(column.getName()); + hiveFieldSchema.setComment(column.getComment()); + if (partitionSet.contains(column.getName())) { + hiveParts.add(hiveFieldSchema); + } else { + hiveCols.add(hiveFieldSchema); + } + } + return new SchemaSplit(hiveCols, hiveParts); + } + + // Faithful port of fe-core HiveProperties.setTableProperties: serde keys land on the SerDe params, + // everything else on the table params. + private static void setTableProperties(Table table, Map properties) { + Map serdeProps = new HashMap<>(); + Map tblProps = new HashMap<>(); + for (String k : properties.keySet()) { + if (HIVE_SERDE_PROPERTIES.contains(k)) { + serdeProps.put(k, properties.get(k)); + } else { + tblProps.put(k, properties.get(k)); + } + } + if (table.getParameters() == null) { + table.setParameters(tblProps); + } else { + table.getParameters().putAll(tblProps); + } + if (table.getSd().getSerdeInfo().getParameters() == null) { + table.getSd().getSerdeInfo().setParameters(serdeProps); + } else { + table.getSd().getSerdeInfo().getParameters().putAll(serdeProps); + } + } + + private static boolean isEmpty(String s) { + return s == null || s.isEmpty(); + } + + private static boolean isNotEmpty(String s) { + return !isEmpty(s); + } + + private static Set unmodifiableSet(String... values) { + return Collections.unmodifiableSet(new HashSet<>(Arrays.asList(values))); + } + + /** Data columns and partition-key columns split out of the full column list. */ + private static final class SchemaSplit { + private final List dataColumns; + private final List partitionColumns; + + private SchemaSplit(List dataColumns, List partitionColumns) { + this.dataColumns = dataColumns; + this.partitionColumns = partitionColumns; + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 401c9170dab16b..00dd40d65b98e2 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -34,6 +34,7 @@ import org.apache.hadoop.hive.metastore.api.FieldSchema; import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.metastore.api.Partition; +import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint; import org.apache.hadoop.hive.metastore.api.StorageDescriptor; import org.apache.hadoop.hive.metastore.api.Table; import org.apache.logging.log4j.LogManager; @@ -212,6 +213,76 @@ public HmsPartitionInfo getPartition(String dbName, String tableName, }); } + // ========== Phase 3: DDL / write operations ========== + + @Override + public void createDatabase(HmsCreateDatabaseRequest request) { + execute(client -> { + client.createDatabase(HmsWriteConverter.toHiveDatabase(request)); + return null; + }); + } + + @Override + public void dropDatabase(String dbName) { + execute(client -> { + client.dropDatabase(dbName); + return null; + }); + } + + @Override + public void createTable(HmsCreateTableRequest request) { + if (tableExists(request.getDbName(), request.getTableName())) { + throw new HmsClientException("Table '" + request.getTableName() + + "' has existed in '" + request.getDbName() + "'."); + } + Table hiveTable = HmsWriteConverter.toHiveTable(request); + List defaults = buildDefaultConstraints(request); + execute(client -> { + if (!defaults.isEmpty()) { + // foreignKeys, uniqueConstraints, notNullConstraints, defaultConstraints, checkConstraints + client.createTableWithConstraints(hiveTable, null, null, null, null, defaults, null); + } else { + client.createTable(hiveTable); + } + return null; + }); + } + + @Override + public void dropTable(String dbName, String tableName) { + execute(client -> { + client.dropTable(dbName, tableName); + return null; + }); + } + + @Override + public void truncateTable(String dbName, String tableName, List partitions) { + execute(client -> { + client.truncateTable(dbName, tableName, partitions); + return null; + }); + } + + private static List buildDefaultConstraints(HmsCreateTableRequest request) { + List defaults = new ArrayList<>(); + for (ConnectorColumn column : request.getColumns()) { + String defaultValue = column.getDefaultValue(); + if (defaultValue != null) { + SQLDefaultConstraint dv = new SQLDefaultConstraint(); + dv.setTable_db(request.getDbName()); + dv.setTable_name(request.getTableName()); + dv.setColumn_name(column.getName()); + dv.setDefault_value(defaultValue); + dv.setDc_name(column.getName() + "_dv_constraint"); + defaults.add(dv); + } + } + return defaults; + } + @Override public synchronized void close() throws IOException { if (closed) { diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java index 4c63afae1f8925..8247b4349301dd 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsTypeMappingTest.java @@ -159,4 +159,89 @@ public void testFindNextNestedFieldRespectsNesting() { // No top-level comma -> returns the length. Assertions.assertEquals(3, HmsTypeMapping.findNextNestedField("int")); } + + // ==================== reverse mapping: ConnectorType -> Hive type string ==================== + // + // WHY: toHiveTypeString is the CREATE TABLE direction, the SPI-clean equivalent of fe-core + // HiveMetaStoreClientHelper.dorisTypeToHiveType. Its input type names are Doris PrimitiveType + // names (what ConnectorColumnConverter.toConnectorType emits via PrimitiveType.toString()). A + // wrong reverse mapping silently mistypes every column of a table Doris creates in Hive, so + // these pin the exact Hive string per Doris type — especially the ones that intentionally + // collapse (VARCHAR->string) or drop parameters (timestamp), and the unsupported ones that + // must throw rather than emit a bogus type. + + private static String hive(ConnectorType type) { + return HmsTypeMapping.toHiveTypeString(type); + } + + @Test + public void testToHiveTypeStringPrimitives() { + Assertions.assertEquals("boolean", hive(ConnectorType.of("BOOLEAN"))); + Assertions.assertEquals("tinyint", hive(ConnectorType.of("TINYINT"))); + Assertions.assertEquals("smallint", hive(ConnectorType.of("SMALLINT"))); + Assertions.assertEquals("int", hive(ConnectorType.of("INT"))); + Assertions.assertEquals("bigint", hive(ConnectorType.of("BIGINT"))); + Assertions.assertEquals("float", hive(ConnectorType.of("FLOAT"))); + Assertions.assertEquals("double", hive(ConnectorType.of("DOUBLE"))); + Assertions.assertEquals("string", hive(ConnectorType.of("STRING"))); + } + + @Test + public void testToHiveTypeStringDateAndTimestampVariants() { + // Both the legacy and V2 date/datetime primitives collapse to Hive date/timestamp. + Assertions.assertEquals("date", hive(ConnectorType.of("DATE"))); + Assertions.assertEquals("date", hive(ConnectorType.of("DATEV2"))); + Assertions.assertEquals("timestamp", hive(ConnectorType.of("DATETIME"))); + // The datetime scale carried on the ConnectorType is intentionally dropped (Hive timestamp has none). + Assertions.assertEquals("timestamp", hive(ConnectorType.of("DATETIMEV2", 6, -1))); + } + + @Test + public void testToHiveTypeStringCharVarcharString() { + // CHAR carries its length in the precision field (create-request encoding). + Assertions.assertEquals("char(10)", hive(ConnectorType.of("CHAR", 10, 0))); + // VARCHAR intentionally maps to Hive string (parity with legacy dorisTypeToHiveType). + Assertions.assertEquals("string", hive(ConnectorType.of("VARCHAR", 255, 0))); + Assertions.assertEquals("string", hive(ConnectorType.of("STRING"))); + } + + @Test + public void testToHiveTypeStringDecimalVariantsAndDefault() { + // Every Doris decimal storage width maps to Hive decimal(p,s). + Assertions.assertEquals("decimal(10,2)", hive(ConnectorType.of("DECIMAL64", 10, 2))); + Assertions.assertEquals("decimal(38,10)", hive(ConnectorType.of("DECIMAL128", 38, 10))); + Assertions.assertEquals("decimal(9,0)", hive(ConnectorType.of("DECIMALV2", 9, 0))); + Assertions.assertEquals("decimal(76,0)", hive(ConnectorType.of("DECIMAL256", 76, 0))); + // A read-origin DECIMALV3 name is accepted too. + Assertions.assertEquals("decimal(5,3)", hive(ConnectorType.of("DECIMALV3", 5, 3))); + // Precision 0 falls back to the default precision 9 (parity with legacy). + Assertions.assertEquals("decimal(9,0)", hive(ConnectorType.of("DECIMAL32", 0, 0))); + } + + @Test + public void testToHiveTypeStringComplexIncludingNested() { + Assertions.assertEquals("array", hive(ConnectorType.arrayOf(ConnectorType.of("INT")))); + Assertions.assertEquals("map", + hive(ConnectorType.mapOf(ConnectorType.of("STRING"), ConnectorType.of("BIGINT")))); + Assertions.assertEquals("struct", + hive(ConnectorType.structOf(Arrays.asList("a", "b"), + Arrays.asList(ConnectorType.of("INT"), ConnectorType.of("STRING"))))); + // Nested: an array of structs of a map. + ConnectorType nested = ConnectorType.arrayOf( + ConnectorType.structOf(Arrays.asList("m"), + Arrays.asList(ConnectorType.mapOf(ConnectorType.of("STRING"), + ConnectorType.of("INT"))))); + Assertions.assertEquals("array>>", hive(nested)); + } + + @Test + public void testToHiveTypeStringUnsupportedThrows() { + // Types Hive tables cannot represent must throw, not emit a bogus type string. + Assertions.assertThrows(IllegalArgumentException.class, + () -> hive(ConnectorType.of("LARGEINT"))); + Assertions.assertThrows(IllegalArgumentException.class, + () -> hive(ConnectorType.of("IPV4"))); + Assertions.assertThrows(IllegalArgumentException.class, + () -> hive(ConnectorType.of("JSONB"))); + } } diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java new file mode 100644 index 00000000000000..65abb48be337b2 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java @@ -0,0 +1,225 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; + +import org.apache.hadoop.hive.metastore.api.Database; +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.PrincipalType; +import org.apache.hadoop.hive.metastore.api.StorageDescriptor; +import org.apache.hadoop.hive.metastore.api.Table; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Tests {@link HmsWriteConverter} — the CREATE TABLE / CREATE DATABASE direction, the SPI-clean + * equivalent of fe-core {@code HiveUtil.toHiveTable}/{@code toHiveDatabase} plus + * {@code HiveProperties.setTableProperties}. + * + *

WHY: this converter decides exactly what metastore object Doris writes when a user creates a + * Hive table. Bugs here silently produce an unreadable table (wrong serde/format), lose the + * data/partition-column split, or misplace serde properties. These pin the storage descriptor, + * the per-format compression defaults, the property split, and the column split — the behavior + * the connector's DDL path depends on (Rule 9: encode the contract).

+ */ +public class HmsWriteConverterTest { + + private static ConnectorColumn col(String name, String type) { + return new ConnectorColumn(name, ConnectorType.of(type), null, true, null); + } + + private static HmsCreateTableRequest.Builder baseTable(String fileFormat, List columns, + List partitionKeys) { + return HmsCreateTableRequest.builder() + .dbName("db") + .tableName("t") + .columns(columns) + .partitionKeys(partitionKeys) + .fileFormat(fileFormat) + .properties(new HashMap<>()); + } + + @Test + public void testOrcTableFormatAndColumnSplit() { + List columns = Arrays.asList( + col("id", "INT"), + col("name", "STRING"), + col("dt", "DATEV2")); + Table table = HmsWriteConverter.toHiveTable( + baseTable("orc", columns, Collections.singletonList("dt")) + .location("hdfs://ns/db/t") + .dorisVersion("2.1.0-abc123") + .comment("hello") + .properties(mutableMap("owner", "alice")) + .build()); + + Assertions.assertEquals("db", table.getDbName()); + Assertions.assertEquals("t", table.getTableName()); + Assertions.assertEquals("MANAGED_TABLE", table.getTableType()); + Assertions.assertEquals("alice", table.getOwner()); + + // Data columns exclude the partition key; partition key is carried separately. + StorageDescriptor sd = table.getSd(); + Assertions.assertEquals(Arrays.asList("id", "name"), names(sd.getCols())); + Assertions.assertEquals(Arrays.asList("int", "string"), types(sd.getCols())); + Assertions.assertEquals(Collections.singletonList("dt"), names(table.getPartitionKeys())); + Assertions.assertEquals(Collections.singletonList("date"), types(table.getPartitionKeys())); + + // ORC storage formats + serde. + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat", sd.getInputFormat()); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat", sd.getOutputFormat()); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.orc.OrcSerde", + sd.getSerdeInfo().getSerializationLib()); + Assertions.assertEquals("hdfs://ns/db/t", sd.getLocation()); + Assertions.assertEquals("doris external hive table", sd.getParameters().get("tag")); + + // Table params: doris.version stamped, comment set, default ORC compression applied. + Assertions.assertEquals("2.1.0-abc123", table.getParameters().get("doris.version")); + Assertions.assertEquals("hello", table.getParameters().get("comment")); + Assertions.assertEquals("zlib", table.getParameters().get("orc.compress")); + } + + @Test + public void testParquetDefaultCompression() { + Table table = HmsWriteConverter.toHiveTable( + baseTable("parquet", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).build()); + StorageDescriptor sd = table.getSd(); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat", + sd.getInputFormat()); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe", + sd.getSerdeInfo().getSerializationLib()); + Assertions.assertEquals("snappy", table.getParameters().get("parquet.compression")); + } + + @Test + public void testTextCompressionUsesRequestDefaultThenPlainFallback() { + // The connector-resolved session default flows through for a text table. + Table withDefault = HmsWriteConverter.toHiveTable( + baseTable("text", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).defaultTextCompression("gzip").build()); + Assertions.assertEquals("org.apache.hadoop.mapred.TextInputFormat", + withDefault.getSd().getInputFormat()); + Assertions.assertEquals("gzip", withDefault.getParameters().get("text.compression")); + + // No request default -> "plain" fallback (matches the legacy session-var default). + Table noDefault = HmsWriteConverter.toHiveTable( + baseTable("text", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).build()); + Assertions.assertEquals("plain", noDefault.getParameters().get("text.compression")); + } + + @Test + public void testExplicitCompressionHonoredAndKeyRemoved() { + Table table = HmsWriteConverter.toHiveTable( + baseTable("orc", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).properties(mutableMap("compression", "snappy")).build()); + Assertions.assertEquals("snappy", table.getParameters().get("orc.compress")); + // The transient "compression" property must not leak onto the table. + Assertions.assertFalse(table.getParameters().containsKey("compression")); + } + + @Test + public void testUnsupportedCompressionAndFormatThrow() { + Assertions.assertThrows(IllegalArgumentException.class, () -> + HmsWriteConverter.toHiveTable( + baseTable("orc", Collections.singletonList(col("id", "INT")), + Collections.emptyList()) + .properties(mutableMap("compression", "bogus")).build())); + Assertions.assertThrows(IllegalArgumentException.class, () -> + HmsWriteConverter.toHiveTable( + baseTable("avro", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).build())); + } + + @Test + public void testSerdePropertiesSplitFromTableProperties() { + Map props = mutableMap("field.delim", ","); + props.put("my.custom", "v"); + Table table = HmsWriteConverter.toHiveTable( + baseTable("text", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).properties(props).build()); + // field.delim is a serde property -> goes to the SerDe params, not the table params. + Assertions.assertEquals(",", table.getSd().getSerdeInfo().getParameters().get("field.delim")); + Assertions.assertFalse(table.getParameters().containsKey("field.delim")); + // A non-serde property stays on the table. + Assertions.assertEquals("v", table.getParameters().get("my.custom")); + } + + @Test + public void testDorisVersionOmittedWhenAbsent() { + Table table = HmsWriteConverter.toHiveTable( + baseTable("orc", Collections.singletonList(col("id", "INT")), + Collections.emptyList()).build()); + Assertions.assertFalse(table.getParameters().containsKey("doris.version")); + } + + @Test + public void testBucketingCarriedToStorageDescriptor() { + Table table = HmsWriteConverter.toHiveTable( + baseTable("orc", Collections.singletonList(col("id", "INT")), + Collections.emptyList()) + .bucketCols(Collections.singletonList("id")).numBuckets(8).build()); + Assertions.assertEquals(Collections.singletonList("id"), table.getSd().getBucketCols()); + Assertions.assertEquals(8, table.getSd().getNumBuckets()); + } + + @Test + public void testToHiveDatabase() { + Database db = HmsWriteConverter.toHiveDatabase(new HmsCreateDatabaseRequest( + "mydb", "hdfs://ns/mydb", "a comment", mutableMap("owner", "bob"))); + Assertions.assertEquals("mydb", db.getName()); + Assertions.assertEquals("hdfs://ns/mydb", db.getLocationUri()); + Assertions.assertEquals("a comment", db.getDescription()); + Assertions.assertEquals("bob", db.getOwnerName()); + Assertions.assertEquals(PrincipalType.USER, db.getOwnerType()); + } + + @Test + public void testToHiveDatabaseNoLocationNoOwner() { + Database db = HmsWriteConverter.toHiveDatabase(new HmsCreateDatabaseRequest( + "mydb", null, null, new HashMap<>())); + Assertions.assertEquals("mydb", db.getName()); + Assertions.assertFalse(db.isSetLocationUri()); + Assertions.assertNull(db.getOwnerName()); + // Comment normalizes to empty string, never null. + Assertions.assertEquals("", db.getDescription()); + } + + private static List names(List schemas) { + return schemas.stream().map(FieldSchema::getName).collect(java.util.stream.Collectors.toList()); + } + + private static List types(List schemas) { + return schemas.stream().map(FieldSchema::getType).collect(java.util.stream.Collectors.toList()); + } + + private static Map mutableMap(String k, String v) { + Map m = new HashMap<>(); + m.put(k, v); + return m; + } +} From fa35fb08b45cbc6638c5fc032e51bb6e4b3d671e Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 06:45:43 +0800 Subject: [PATCH 007/334] =?UTF-8?q?[doc](catalog)=20P7.1=20hive:=20mark=20?= =?UTF-8?q?T02=E2=80=93T04=20done=20(plugin=20write=20client);=20handoff?= =?UTF-8?q?=20=E2=86=92=20T05?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the plugin-side HMS DDL write client (fdf577e7946) as complete in the task breakdown + rolling handoff, and point the next unit at T05→T07 (HiveConnectorMetadata DDL overrides) with the two request fields the metadata layer / config-injection locus must populate (defaultTextCompression, dorisVersion). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 16 ++++++++-------- plan-doc/tasks/P7-hive-migration.md | 7 ++++++- 2 files changed, 14 insertions(+), 9 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 20fe6123fa5a69..f662987c36d947 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,15 +5,15 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.1 进行中：recon+拆解+决策已定，T01（TRUNCATE SPI seam）已实现+全绿+提交；下一步 = T02→T04 插件写客户端** +# 🎯 当前状态（2026-07-06）= **P7.1 进行中：T01（TRUNCATE SPI seam）+ T02–T04（插件写客户端）已实现+全绿+提交；下一步 = T05→T07（HiveConnectorMetadata DDL override）+ T08/T09/T10/T11** -> **本轮（P7.1 recon + 拆解 + T01 session）做了什么**： -> 1. **P7.1 code-grounded recon**（3 并行 agent + 直读 HEAD，~225k subagent token）核清并**校正 spec** 4 件事：(1) 通用 DDL 桥 = `PluginDrivenExternalCatalog`（非独立 MetadataOps，逐条 override + 内联 cache/editlog，连接器只实现纯 SPI）；(2) **TRUNCATE 硬缺口**（SPI 无 `truncateTable`、桥未 override）；(3) shared converter 逐字拷 properties（✅ 不 parse）但**丢每列默认值**（恒 null → 破坏 hive 列默认值 + DLF guard）+ 丢 LIST/RANGE partition value；(4) stats/partition 写 4 法**仅 HMSTransaction 消费**→推 P7.3，rename **hive 今天不支持**→保持 SPI default throw。 -> 2. 产出 **`tasks/P7-hive-migration.md` 末尾 P7.1 逐 task 拆解（T01–T11）+ recon 结论块**。 -> 3. **决策 OQ-HIVE-CFG 用户裁定 = 方案 A**（fe-core 建连接器时注入两个全局 Config 值为属性默认，插件只读属性、零行为回归）。 -> 4. **T01 已实现并全绿提交（`c0222977ebd`）**：加 `ConnectorTableOps.truncateTable(session, handle, partitions)` default-throw seam + `PluginDrivenExternalCatalog.truncateTable`（remote 解析 → `metadata.truncateTable` → `TruncateTableInfo` editlog + `refreshTableInternal` 缓存刷新，仿 legacy `afterTruncateTable`）+ `replayTruncateTable`（follower 缓存刷新）。compile SUCCESS + checkstyle 0（fe-connector-api,fe-core）+ import-gate 净。**尚无连接器 override truncate（等 T07），翻闸前行为不变。** +> **本轮（P7.1 T02+T03+T04 session）做了什么** —— 提交 `fdf577e7946`，**插件写客户端一个自洽单元，全在 `fe-connector-hms`**： +> 1. **T02 写 DTO**：`HmsCreateTableRequest`（builder：全列[数据+分区]、partitionKeys 名、bucketCols/numBuckets、fileFormat、comment、properties、`defaultTextCompression`、`dorisVersion`；列默认值走 `ConnectorColumn.getDefaultValue()`）+ `HmsCreateDatabaseRequest`。仿读侧 `HmsTableInfo`/`HmsDatabaseInfo`，SPI-clean。 +> 2. **T03 转换器**：`HmsTypeMapping.toHiveTypeString(ConnectorType)`（反向类型映射，等价 `dorisTypeToHiveType`；switch 于 `PrimitiveType.toString()` 名，VARCHAR→string、datetime 丢 scale、decimal p==0→9、不支持类型 throw）+ `HmsWriteConverter.toHiveTable/toHiveDatabase`（忠实移植 `HiveUtil` + `HiveProperties.setTableProperties` serde/table 属性切分；in/out/serde+压缩默认逐字照搬；`createTime` 溢出照搬）。**断耦**：`DORIS_VERSION_VALUE` 经 request 线程进（插件禁 import fe-common `Version`——import gate 禁 `org.apache.doris.common`）、text 压缩默认经 request（T06 从 session 取）、`AnalysisException`→`IllegalArgumentException`、JDK-only。 +> 3. **T04 写方法**：`HmsClient` 加 5 个写方法为 **default-throw seam**（3 个读侧 fake 不破）；`ThriftHmsClient` 用现成 `execute(...)`（auth+pool）override 全部，`createTable` 从列默认值建 `SQLDefaultConstraint`→`createTableWithConstraints`，等价 legacy `ThriftHMSCachedClient`。 +> 4. **验收**：fe-connector-hms compile SUCCESS + checkstyle 0（main+test）+ import-gate 净 + **28 单测绿**（反向映射 + 转换器，无 fake；client-dispatch/HiveConnectorMetadata 单测留 T10 recording-fake）。**无连接器 override 这些写方法（等 T05–T07），翻闸前行为不变。** > -> **T02–T11 未动**（下一 session 接手；T03 类型映射/转换器移植最精细，故留新 context 谨慎做，不在长 session 尾仓促）。 +> **T05–T11 未动**（下一单元：`fe-connector-hive` 的 `HiveConnectorMetadata` DDL override + shared converter 列默认值 + config 注入 + 单测 + 守门）。T05–T07 需研读模板 `IcebergConnectorMetadata` 与 SPI base（`ConnectorSchemaOps`/`ConnectorTableOps`），故为下一自洽单元。 > **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： > P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 @@ -42,7 +42,7 @@ ## P7.1 执行要点（下个 session 开场） -1. **先读** spec 末尾「P7.1 逐 task 拆解」块 + 「T03 移植指针」块（recon 存档，勿再 recon）。**T01 已完成**（truncate SPI seam + 桥 override，`c0222977ebd`）。**从 T02 起接**：T02(hms 写 DTO) → T03(hms converter，HiveUtil 等价插件侧，**最精细**，指针见 spec) → T04(HmsClient 写方法 DDL-only) → T05(HiveConnectorMetadata DB DDL) → T06(createTable) → T07(dropTable+truncate override；rename 保持 default throw) → T08(shared converter 带列默认值) → T09(config 注入=方案A) → T10(连接器单测) → T11(守门+交接)。**信 HEAD 控制流不信 spec 行号。** T02+T03+T04 = 一个自洽单元（插件写客户端），可合并一次 commit。 +1. **先读** spec 末尾「P7.1 逐 task 拆解」块 + 「P7.1 实现进度」块（recon 存档，勿再 recon）。**T01–T04 已完成**（T01 truncate SPI seam + 桥 override `c0222977ebd`；T02+T03+T04 插件写客户端 `fdf577e7946`）。**从 T05 起接**：T05(HiveConnectorMetadata DB DDL) → T06(createTable) → T07(dropTable+truncate override；rename 保持 default throw) → T08(shared converter 带列默认值) → T09(config 注入=方案A) → T10(连接器单测，recording-fake `IMetaStoreClient`) → T11(守门+交接)。**信 HEAD 控制流不信 spec 行号。** T05→T07 = 下一自洽单元（连接器 DDL override，模板 `IcebergConnectorMetadata`）。**注意**：T06 须从 `ConnectorSession` 取 text 压缩默认填 `HmsCreateTableRequest.defaultTextCompression`；T09 注入 locus 须同时把 `doris.version`（值）填 `HmsCreateTableRequest.dorisVersion`（插件已留字段，值缺时转换器省略该 tag）。 2. **本阶段范围 = 仅 DDL 写路径**：create/drop db+table + truncate。**stats/partition 写 4 法（updateTableStatistics/updatePartitionStatistics/addPartitions/dropPartition）推 P7.3**（仅 HMSTransaction 消费，本阶段加即死代码）。**rename 不实现**（hive 今天不支持，保持 SPI default throw）。**no-property-parsing**：file_format 默认/owner/bucket-gate/DLF-guard/transactional-reject 全插件侧（`ConnectorSession.getUser()`+`getCatalogProperties()`+`ConnectorColumn.defaultValue` 已够，无须扩 SPI，除 truncate seam + T08 列默认值 carrier）。 3. **OQ-HIVE-CFG 已裁定 = 方案 A**（T09）：fe-core 组装 `hms` 连接器属性处，把 `Config.hive_default_file_format`/`Config.enable_create_hive_bucket_table` 当前值注入为属性默认（仅用户未显式设时），**插件只读属性、不碰 `common.Config`**。注入 locus 待 recon。 4. **勿重议的已定决策**：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransaction`。后续子阶段 OQ-* 到 recon 后再用户签字（中文讲背景+示例+推荐、不引任务代号）。 diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index 259923f54b90e7..bc4f8b9c3cc962 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -202,7 +202,12 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 ## P7.1 实现进度 + T03 移植指针（recon 存档，勿再 recon） - **T01 ✅ 已提交 `c0222977ebd`**：`ConnectorTableOps.truncateTable(session, handle, partitions)` default-throw + `PluginDrivenExternalCatalog.truncateTable`/`replayTruncateTable`（editlog `TruncateTableInfo` + `refreshTableInternal`；base 签名 = `truncateTable(String db, String tbl, PartitionNamesInfo, boolean forceDrop, String rawTruncateSql)`，`forceDrop`/`rawTruncateSql` 无 external 语义故忽略）。全绿。 -- **T02–T11 未动**（下 session 从 T02 起）。 +- **T02+T03+T04 ✅ 已提交 `fdf577e7946`（插件写客户端，一个自洽单元）**，全在 `fe-connector-hms`： + - **T02**：`HmsCreateTableRequest`（builder；columns=全列[数据+分区]、partitionKeys=名、bucketCols/numBuckets、fileFormat、comment、properties、`defaultTextCompression`、`dorisVersion`；列默认值走 `ConnectorColumn.getDefaultValue()`）+ `HmsCreateDatabaseRequest`（dbName/locationUri/comment/properties）。仿读侧 `HmsTableInfo`/`HmsDatabaseInfo` 风格，SPI-clean。 + - **T03**：`HmsTypeMapping.toHiveTypeString(ConnectorType)`（反向映射，等价 `HiveMetaStoreClientHelper.dorisTypeToHiveType`；switch 于 `PrimitiveType.toString()` 名——`ConnectorColumnConverter.toConnectorType` 之输出；VARCHAR→string、datetime 丢 scale、decimal p==0→9、不支持类型 throw）+ `HmsWriteConverter.toHiveTable/toHiveDatabase`（忠实移植 `HiveUtil` + `HiveProperties.setTableProperties` 的 serde/table 属性切分；orc/parquet/text 的 in/out/serde+压缩默认逐字照搬；`createTime=(int)millis*1000` 溢出照搬；MANAGED_TABLE、`doris external hive table` tag）。**断耦**：DORIS_VERSION_VALUE 经 request 线程进（插件不能 import fe-common `Version`）、text 压缩默认经 request（T06 从 session 取）、`AnalysisException`→`IllegalArgumentException`、JDK-only（无 guava/commons）。 + - **T04**：`HmsClient` 加 5 个写方法（createDatabase/dropDatabase/createTable/dropTable/truncateTable）为 **default-throw seam**（读侧 fake 不受影响）；`ThriftHmsClient` 用现成 `execute(...)`（auth+pool）override 全部，`createTable` 从列默认值建 `SQLDefaultConstraint`→`createTableWithConstraints`（否则 `createTable`），等价 legacy。 + - **验收**：fe-connector-hms compile SUCCESS + checkstyle 0（main+test）+ import-gate 净 + 28 单测绿（反向类型映射 + 转换器；client-dispatch/HiveConnectorMetadata 单测留 T10 recording-fake）。**无连接器 override，行为不变。** +- **下一单元 = T05→T07（`HiveConnectorMetadata` DDL override）+ T08（shared converter 带列默认值）+ T09（config 注入=方案A）+ T10（recording-fake 单测）+ T11（守门+交接）。** T06 需从 `ConnectorSession` 取 text 压缩默认填 request、并把 `doris.version`（值）与两个全局 Config 一起在注入 locus（T09）落地填 request；模板 = `IcebergConnectorMetadata`。**信 HEAD 控制流不信本 spec 行号。** - **T03（converter，最精细）移植指针（HEAD 行号，信控制流）**：源在 `datasource/hive/HiveUtil.java` —— `toHiveTable`(:227)、`setCompressType`(:259)、`toHiveStorageDesc`(:286)、`setFileFormat`(:301)、`toHiveSchema`(:327)、`toHiveDatabase`(:346)。**必须断的 fe-core 耦合**： - `HiveMetaStoreClientHelper.dorisTypeToHiveType(Type)`（`HiveMetaStoreClientHelper.java:529`，walk scalar/array/map/struct → hive 类型串；char(len)/decimal(p,s)；date←DATE/DATEV2、timestamp←DATETIME/DATETIMEV2；VARCHAR/STRING→string）→ **插件侧要写 `ConnectorType` 版逆映射**（`HmsTypeMapping` 是正向 hive-str→ConnectorType，需补逆向；~80 行对称）。 - `ExternalCatalog.DORIS_VERSION` / `DORIS_VERSION_VALUE` → 插件本地常量。 From fb1aad5b7a517af37e215d025010866be67f1500 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 08:10:56 +0800 Subject: [PATCH 008/334] [refactor](catalog) P7.1 hive T08: enrich CREATE TABLE request with column defaults + explicit-partition-values signal Thread two pieces of the neutral CREATE TABLE request that the hive plugin needs to reproduce legacy HiveMetadataOps.createTableImpl parity; both are additive and ignored by connectors that build their schema from name/type/nullable/comment only (iceberg/paimon/maxcompute build-time paths do not read either). - Per-column default value: expose ColumnDefinition.getDefaultValueString() (Column.getDefaultValue() equivalent, via DefaultValue.getValue()) and thread it onto ConnectorColumn.defaultValue in CreateTableInfoToConnectorRequestConverter (was hardcoded null). Hive builds metastore default constraints from it and gates its DLF catalog on per-column defaults. - Explicit-partition-values signal: the converter drops LIST/RANGE partition value expressions (still not lowered), so add ConnectorPartitionSpec.hasExplicitPartitionValues() and set it from PartitionTableInfo.getPartitionDefs(). Lets hive keep rejecting `PARTITION BY LIST(dt) (PARTITION p VALUES IN (...))` (external tables discover partitions from the data layout) instead of silently ignoring it. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../api/ddl/ConnectorPartitionSpec.java | 27 +++++++++++++++++-- ...eTableInfoToConnectorRequestConverter.java | 17 +++++++----- .../plans/commands/info/ColumnDefinition.java | 12 +++++++++ 3 files changed, 48 insertions(+), 8 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java index 2414661f3ed87f..8397fe271d3f84 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorPartitionSpec.java @@ -46,10 +46,18 @@ public enum Style { private final Style style; private final List fields; private final List initialValues; + private final boolean hasExplicitPartitionValues; public ConnectorPartitionSpec(Style style, List fields, List initialValues) { + this(style, fields, initialValues, false); + } + + public ConnectorPartitionSpec(Style style, + List fields, + List initialValues, + boolean hasExplicitPartitionValues) { this.style = Objects.requireNonNull(style, "style"); this.fields = fields == null ? Collections.emptyList() @@ -57,6 +65,7 @@ public ConnectorPartitionSpec(Style style, this.initialValues = initialValues == null ? Collections.emptyList() : Collections.unmodifiableList(initialValues); + this.hasExplicitPartitionValues = hasExplicitPartitionValues; } public Style getStyle() { @@ -71,6 +80,18 @@ public List getInitialValues() { return initialValues; } + /** + * Whether the CREATE TABLE declared explicit partition value definitions (e.g. + * {@code PARTITION BY LIST(dt) (PARTITION p1 VALUES IN ('a'))}). The neutral converter does not lower + * those value expressions into {@link #getInitialValues()} (it stays empty), so this flag preserves the + * information a connector needs to reject them: Hive external tables discover partitions from the data + * layout and reject explicit partition values (legacy parity). Connectors that accept explicit partition + * definitions ignore this flag. + */ + public boolean hasExplicitPartitionValues() { + return hasExplicitPartitionValues; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -81,19 +102,21 @@ public boolean equals(Object o) { } ConnectorPartitionSpec that = (ConnectorPartitionSpec) o; return style == that.style + && hasExplicitPartitionValues == that.hasExplicitPartitionValues && fields.equals(that.fields) && initialValues.equals(that.initialValues); } @Override public int hashCode() { - return Objects.hash(style, fields, initialValues); + return Objects.hash(style, fields, initialValues, hasExplicitPartitionValues); } @Override public String toString() { return "ConnectorPartitionSpec{style=" + style + ", fields=" + fields - + ", initialValues=" + initialValues.size() + "}"; + + ", initialValues=" + initialValues.size() + + ", hasExplicitPartitionValues=" + hasExplicitPartitionValues + "}"; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java b/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java index 6070f75a6a3965..d5b3aa375dfd8c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/ddl/CreateTableInfoToConnectorRequestConverter.java @@ -93,12 +93,13 @@ private static List convertColumns( // cannot store them (MaxCompute); see MaxComputeConnectorMetadata.validateColumns. boolean isAggregated = d.getAggType() != null && d.getAggType() != AggregateType.NONE; - // Default value is not exposed via a public getter on ColumnDefinition - // (private Optional); pass null until the SPI gains a - // typed default-value carrier. See HANDOFF open issues. + // Thread the catalog-level default value string (null when the column has no DEFAULT clause). + // Hive builds metastore default constraints from it and gates its DLF catalog on per-column + // defaults; connectors that ignore create-time defaults (iceberg/paimon/maxcompute build their + // schema from name/type/nullable/comment only) are unaffected. out.add(new ConnectorColumn( d.getName(), type, d.getComment(), - d.isNullable(), null, d.isKey(), d.getAutoIncInitValue() != -1, + d.isNullable(), d.getDefaultValueString(), d.isKey(), d.getAutoIncInitValue() != -1, isAggregated)); } return out; @@ -140,8 +141,12 @@ private static ConnectorPartitionSpec convertPartition( // require full analysis to flatten into List>. Connectors // that need the initial values today read the Doris PartitionDesc // directly; this converter passes an empty list and leaves richer - // lowering for a follow-up. - return new ConnectorPartitionSpec(style, fields, Collections.emptyList()); + // lowering for a follow-up. The presence flag is still threaded so a + // connector that rejects explicit partition values (Hive external tables + // discover partitions from the data layout) can fail loud (legacy parity). + boolean hasExplicitValues = info.getPartitionDefs() != null + && !info.getPartitionDefs().isEmpty(); + return new ConnectorPartitionSpec(style, fields, Collections.emptyList(), hasExplicitValues); } private static boolean hasAnyTransform(List exprs) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java index 4be6315f079fdb..98196358a2094f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/ColumnDefinition.java @@ -183,6 +183,18 @@ public boolean hasDefaultValue() { return defaultValue.isPresent(); } + /** + * Returns the column's default value as the catalog-level string (the same value the translated + * {@link org.apache.doris.catalog.Column#getDefaultValue()} carries), or {@code null} when the column + * has no default. Exposed so {@code CreateTableInfoToConnectorRequestConverter} can thread it onto + * {@code ConnectorColumn.defaultValue} for connectors (Hive) that build metastore default constraints + * and gate DDL on per-column defaults; connectors that ignore create-time defaults (iceberg/paimon/ + * maxcompute) are unaffected. + */ + public String getDefaultValueString() { + return defaultValue.map(DefaultValue::getValue).orElse(null); + } + public boolean isVisible() { return isVisible; } From 1fcc93d8003464eb645a8ab1d0c2b841350e6c41 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 08:11:06 +0800 Subject: [PATCH 009/334] [refactor](catalog) P7.1 hive T09: thread hive CREATE TABLE global-Config defaults via connector environment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fe-connector-hive plugin cannot read FE Config, so surface the two FE-global CREATE TABLE toggles + the build version through the connector environment channel (DefaultConnectorContext.buildEnvironment) — the same "C4" mechanism already used for hive_metastore_client_timeout_second. Chosen over injecting them into the catalog property map so they are NOT persisted into the catalog image/edit-log, do not appear in SHOW CREATE CATALOG, and refresh from current Config on each connector rebuild. - hive_default_file_format <- Config.hive_default_file_format - enable_create_hive_bucket_table <- Config.enable_create_hive_bucket_table - doris_version <- Version build id (legacy ExternalCatalog.DORIS_VERSION_VALUE) A user-set per-table file_format property still wins (resolved plugin-side); the plugin reads these keys via ConnectorContext.getEnvironment(). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../doris/connector/DefaultConnectorContext.java | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index c86a3ebd1f6949..41e03da2af1e68 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -23,6 +23,7 @@ import org.apache.doris.common.CatalogConfigFileUtils; import org.apache.doris.common.Config; import org.apache.doris.common.EnvUtils; +import org.apache.doris.common.Version; import org.apache.doris.common.util.LocationPath; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; import org.apache.doris.connector.spi.ConnectorBrokerAddress; @@ -419,6 +420,15 @@ private static Map buildEnvironment() { // applied by HmsMetaStoreProperties.toHiveConfOverrides when the user has not overridden it. env.put("hive_metastore_client_timeout_second", String.valueOf(Config.hive_metastore_client_timeout_second)); + // Hive CREATE TABLE defaults (P7.1): the fe-connector-hive plugin cannot read FE Config, so the two + // FE-global CREATE TABLE toggles are threaded through the environment (not persisted into the catalog + // property map) and applied by HiveConnectorMetadata.createTable when the user did not override them. + // Keys must stay byte-identical to the reads in HiveConnectorProperties. + env.put("hive_default_file_format", Config.hive_default_file_format); + env.put("enable_create_hive_bucket_table", String.valueOf(Config.enable_create_hive_bucket_table)); + // Build version stamped into a created Hive table's doris.version parameter (legacy + // ExternalCatalog.DORIS_VERSION_VALUE); the plugin cannot import fe-common Version. + env.put("doris_version", Version.DORIS_BUILD_VERSION + "-" + Version.DORIS_BUILD_SHORT_HASH); // The trino-connector plugin runs in an isolated classloader and cannot read FE // Config (it would see its own bundled copy with default values). Pass the // configured plugin dir through the engine environment instead. From ea33042cd25f61355700278c942b8cd9781548e3 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 08:11:24 +0800 Subject: [PATCH 010/334] [refactor](catalog) P7.1 hive T05-T07+T10: HiveConnectorMetadata DDL override (create/drop db+table, truncate) + tests MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Port legacy HiveMetadataOps create/drop database, create/drop/truncate table into the plugin so the "hms" SPI path reproduces its behavior (still off — cutover is P7.5). All property interpretation stays plugin-side (fe-core parses no hive properties). - T05 database: supportsCreateDatabase()=true; createDatabase (location -> URI, comment -> description, rest -> params); dropDatabase(force) cascades table drops then the db. - T06 createTable (faithful port of createTableImpl): owner default (session.getUser()), transactional-create reject, file_format default (env, user prop wins), doris.-prefixed round-trip params, LIST-only partitions + explicit-partition-values reject, DLF per-column-default guard, bucket gate (env enable + hash-only), doris.version + text compression default threaded onto the write request. - T07 dropTable (transactional-table reject, via handle params = AcidUtils parity without a hive-exec dep) + truncateTable; renameTable intentionally left as the SPI default throw (hive has no rename). HiveConnector now passes ConnectorContext to the metadata. - T10 tests: recording HmsClient fake + fake session/context (no Mockito), 20 DDL cases covering the guards, defaults, round-trip params, and force cascade. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../doris/connector/hive/HiveConnector.java | 2 +- .../connector/hive/HiveConnectorMetadata.java | 263 ++++++++- .../hive/HiveConnectorProperties.java | 36 ++ .../connector/hive/FakeConnectorContext.java | 64 ++ .../hive/HiveConnectorMetadataDdlTest.java | 553 ++++++++++++++++++ ...ConnectorMetadataPartitionPruningTest.java | 2 +- 6 files changed, 917 insertions(+), 3 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeConnectorContext.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataDdlTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 1f30e8f242c3d6..d489f974c2af68 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -53,7 +53,7 @@ public HiveConnector(Map properties, ConnectorContext context) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - return new HiveConnectorMetadata(getOrCreateClient(), properties); + return new HiveConnectorMetadata(getOrCreateClient(), properties, context); } @Override diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 906236a44cb12f..4f2116dec864ac 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -21,6 +21,11 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.ddl.ConnectorPartitionField; +import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.pushdown.ConnectorAnd; @@ -31,10 +36,14 @@ import org.apache.doris.connector.api.pushdown.ConnectorLiteral; import org.apache.doris.connector.api.pushdown.FilterApplicationResult; import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.HmsClientException; +import org.apache.doris.connector.hms.HmsCreateDatabaseRequest; +import org.apache.doris.connector.hms.HmsCreateTableRequest; import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; import org.apache.doris.connector.hms.HmsTypeMapping; +import org.apache.doris.connector.spi.ConnectorContext; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -44,6 +53,7 @@ import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Optional; import java.util.Set; @@ -69,10 +79,16 @@ public class HiveConnectorMetadata implements ConnectorMetadata { private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; + // Carries the fe-core-injected environment (getEnvironment()) with the FE-global CREATE TABLE defaults + // (hive_default_file_format / enable_create_hive_bucket_table / doris_version) that the plugin cannot + // read from FE Config. The default getEnvironment() is an empty map, so direct-construction tests that + // pass a bare context degrade to the hard-coded fallbacks in createTable. + private final ConnectorContext context; - public HiveConnectorMetadata(HmsClient hmsClient, Map properties) { + public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context) { this.hmsClient = hmsClient; this.properties = properties; + this.context = context; this.typeMappingOptions = buildTypeMappingOptions(properties); } @@ -233,6 +249,251 @@ public Optional> applyFilter( newHandle, constraint.getExpression(), false)); } + // ========== ConnectorSchemaOps: DDL writes (create/drop database) ========== + + /** + * Hive supports CREATE DATABASE. Declaring it lets {@code PluginDrivenExternalCatalog.createDb} consult + * the remote database existence for IF NOT EXISTS (the SPI default {@code false} would skip that check). + */ + @Override + public boolean supportsCreateDatabase() { + return true; + } + + /** + * Creates a Hive database, mirroring legacy {@code HiveMetadataOps.createDbImpl}: the {@code location} + * property becomes the database location URI (and is dropped from the parameter map), the {@code comment} + * property becomes the description, and the remaining properties become database parameters. Existence / + * IF NOT EXISTS is resolved upstream by {@code PluginDrivenExternalCatalog.createDb}. + */ + @Override + public void createDatabase(ConnectorSession session, String dbName, Map dbProperties) { + Map params = new HashMap<>(dbProperties); + String location = params.remove(HiveConnectorProperties.CREATE_LOCATION); + String comment = params.getOrDefault(HiveConnectorProperties.CREATE_COMMENT, ""); + try { + hmsClient.createDatabase(new HmsCreateDatabaseRequest(dbName, location, comment, params)); + } catch (HmsClientException e) { + throw new DorisConnectorException( + "Failed to create Hive database " + dbName + ": " + e.getMessage(), e); + } + } + + /** + * Drops a Hive database, mirroring legacy {@code HiveMetadataOps.dropDbImpl}: with {@code force} every + * table in the database is dropped first (a table that vanished remotely is skipped; a transactional table + * is rejected exactly as a direct DROP TABLE would be), then the database itself. Existence / IF EXISTS is + * resolved upstream by {@code PluginDrivenExternalCatalog.dropDb}, so {@code ifExists} is accepted for SPI + * parity but not re-checked here. + */ + @Override + public void dropDatabase(ConnectorSession session, String dbName, boolean ifExists, boolean force) { + try { + if (force) { + for (String tableName : hmsClient.listTables(dbName)) { + HmsTableInfo tableInfo; + try { + tableInfo = hmsClient.getTable(dbName, tableName); + } catch (HmsClientException e) { + // The table disappeared between listing and load (dropped out-of-band); skip it, + // mirroring legacy dropDbImpl which swallowed getTableOrDdlException and continued. + LOG.warn("failed to load table {}.{} during force drop database: {}", + dbName, tableName, e.getMessage()); + continue; + } + dropTableChecked(dbName, tableName, tableInfo.getParameters()); + } + } + hmsClient.dropDatabase(dbName); + } catch (HmsClientException e) { + throw new DorisConnectorException( + "Failed to drop Hive database " + dbName + ": " + e.getMessage(), e); + } + } + + // ========== ConnectorTableOps: DDL writes (create/drop/truncate table) ========== + + /** + * Creates a Hive table, a faithful port of legacy {@code HiveMetadataOps.createTableImpl}. All property + * interpretation happens here (plugin side); fe-core does not parse hive properties. Existence / + * IF NOT EXISTS is resolved upstream by {@code PluginDrivenExternalCatalog.createTable}. + */ + @Override + public void createTable(ConnectorSession session, ConnectorCreateTableRequest request) { + // Working copy of the user CREATE TABLE properties; the default owner is added here (legacy added it + // to the same map before deriving the metastore parameters). + Map userProps = new HashMap<>(request.getProperties()); + if (session.getUser() != null) { + userProps.putIfAbsent(HiveConnectorProperties.CREATE_OWNER, session.getUser()); + } + // Reject a transactional table create (legacy parity: a hive transactional table only appears to + // accept inserts). Matches legacy's case-sensitive "transactional" key check. + String transactional = userProps.get(HiveConnectorProperties.CREATE_TRANSACTIONAL); + if (transactional != null && transactional.equalsIgnoreCase("true")) { + throw new DorisConnectorException("Not support create hive transactional table."); + } + Map env = context.getEnvironment(); + String fileFormat = userProps.getOrDefault(HiveConnectorProperties.CREATE_FILE_FORMAT, + env.getOrDefault(HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT, + HiveConnectorProperties.DEFAULT_FILE_FORMAT)); + + // Metastore table parameters: lower-case every key and stamp the file_format / location keys under a + // doris. prefix so they round-trip (legacy HiveMetadataOps ddlProps loop). + Map tableParams = new HashMap<>(); + for (Map.Entry entry : userProps.entrySet()) { + String key = entry.getKey().toLowerCase(Locale.ROOT); + if (HiveConnectorProperties.DORIS_HIVE_KEYS.contains(key)) { + tableParams.put(HiveConnectorProperties.DORIS_PROP_PREFIX + key, entry.getValue()); + } else { + tableParams.put(key, entry.getValue()); + } + } + + // Partition columns: LIST only (reject RANGE), and reject explicit partition value definitions + // (hive external tables discover partitions from the data layout). Legacy parity. + List partitionColNames = new ArrayList<>(); + ConnectorPartitionSpec partitionSpec = request.getPartitionSpec(); + if (partitionSpec != null) { + if (partitionSpec.getStyle() == ConnectorPartitionSpec.Style.RANGE) { + throw new DorisConnectorException("Only support 'LIST' partition type in hive catalog."); + } + for (ConnectorPartitionField field : partitionSpec.getFields()) { + partitionColNames.add(field.getColumnName()); + } + if (partitionSpec.hasExplicitPartitionValues()) { + throw new DorisConnectorException( + "Partition values expressions is not supported in hive catalog."); + } + } + + // DLF catalogs reject per-column default values (legacy parity). + if (HmsClientConfig.METASTORE_TYPE_DLF.equals(properties.get(HmsClientConfig.METASTORE_TYPE_KEY))) { + for (ConnectorColumn column : request.getColumns()) { + if (column.getDefaultValue() != null) { + throw new DorisConnectorException("Default values are not supported with `DLF` catalog."); + } + } + } + + HmsCreateTableRequest.Builder builder = HmsCreateTableRequest.builder() + .dbName(request.getDbName()) + .tableName(request.getTableName()) + .location(userProps.get(HiveConnectorProperties.CREATE_LOCATION)) + .columns(request.getColumns()) + .partitionKeys(partitionColNames) + .fileFormat(fileFormat) + .comment(request.getComment()) + .properties(tableParams) + .defaultTextCompression(resolveTextCompressionDefault(session)) + .dorisVersion(env.get(HiveConnectorProperties.ENV_DORIS_VERSION)); + + // Bucketing: gated on the FE-global toggle, and hive supports hash bucketing only. Legacy checks the + // enable gate first, then the hash requirement. + ConnectorBucketSpec bucketSpec = request.getBucketSpec(); + if (bucketSpec != null) { + boolean bucketEnabled = Boolean.parseBoolean(env.getOrDefault( + HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE, "false")); + if (!bucketEnabled) { + throw new DorisConnectorException( + "Create hive bucket table need set enable_create_hive_bucket_table to true"); + } + if (HiveConnectorProperties.BUCKET_ALGO_RANDOM.equals(bucketSpec.getAlgorithm())) { + throw new DorisConnectorException("External hive table only supports hash bucketing"); + } + builder.bucketCols(bucketSpec.getColumns()).numBuckets(bucketSpec.getNumBuckets()); + } + + try { + hmsClient.createTable(builder.build()); + } catch (HmsClientException | IllegalArgumentException e) { + throw new DorisConnectorException("Failed to create Hive table " + + request.getDbName() + "." + request.getTableName() + ": " + e.getMessage(), e); + } + } + + /** + * Drops a Hive table, mirroring legacy {@code HiveMetadataOps.dropTableImpl}: a transactional table is + * rejected. {@code PluginDrivenExternalCatalog} has already resolved the handle / IF EXISTS upstream and + * routed a view DROP elsewhere. + */ + @Override + public void dropTable(ConnectorSession session, ConnectorTableHandle handle) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + try { + // The handle was just built by the bridge's getTableHandle (which loaded the table), so its + // parameters carry the transactional flag; reuse them instead of re-fetching, matching legacy's + // AcidUtils.isTransactionalTable(client.getTable(...)) check. + dropTableChecked(hiveHandle.getDbName(), hiveHandle.getTableName(), + hiveHandle.getTableParameters()); + } catch (HmsClientException e) { + throw new DorisConnectorException("Failed to drop Hive table " + + hiveHandle.getDbName() + "." + hiveHandle.getTableName() + ": " + e.getMessage(), e); + } + } + + /** + * Truncates a Hive table, or the given partitions of it, mirroring legacy + * {@code HiveMetadataOps.truncateTableImpl}. {@code partitions} is {@code null}/empty for a whole-table + * truncate. + */ + @Override + public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, List partitions) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + try { + hmsClient.truncateTable(hiveHandle.getDbName(), hiveHandle.getTableName(), partitions); + } catch (HmsClientException e) { + throw new DorisConnectorException("Failed to truncate Hive table " + + hiveHandle.getDbName() + "." + hiveHandle.getTableName() + ": " + e.getMessage(), e); + } + } + + // RENAME TABLE is intentionally NOT overridden: hive does not support ALTER TABLE RENAME (legacy + // HMSCachedClient has no rename), so the SPI default throw ("RENAME TABLE not supported") preserves the + // pre-flip behavior. + + /** + * Drops {@code dbName.tableName} after rejecting a transactional table, mirroring legacy + * {@code HiveMetadataOps.dropTableImpl}. Shared by the direct DROP TABLE and the force DROP DATABASE + * cascade. + */ + private void dropTableChecked(String dbName, String tableName, Map tableParameters) { + if (isTransactionalTable(tableParameters)) { + throw new DorisConnectorException("Not support drop hive transactional table."); + } + hmsClient.dropTable(dbName, tableName); + } + + /** + * Whether the metastore table parameters mark the table transactional, replicating Hive's + * {@code AcidUtils.isTransactionalTable} (case-insensitive "true" under the "transactional" key, with the + * upper-cased key as a fallback) without pulling in the hive-exec dependency. + */ + private static boolean isTransactionalTable(Map tableParameters) { + if (tableParameters == null) { + return false; + } + String value = tableParameters.get(HiveConnectorProperties.CREATE_TRANSACTIONAL); + if (value == null) { + value = tableParameters.get( + HiveConnectorProperties.CREATE_TRANSACTIONAL.toUpperCase(Locale.ROOT)); + } + return "true".equalsIgnoreCase(value); + } + + /** + * Resolves the compression a {@code text} table falls back to when the user set no {@code compression} + * property, replicating legacy {@code SessionVariable.hiveTextCompression()} (the "uncompressed" alias maps + * to "plain"). The value rides on the request; the write converter only consults it for a text table. + */ + private static String resolveTextCompressionDefault(ConnectorSession session) { + String textCompression = session.getSessionProperties() + .get(HiveConnectorProperties.SESSION_HIVE_TEXT_COMPRESSION); + if (HiveConnectorProperties.TEXT_COMPRESSION_UNCOMPRESSED.equals(textCompression)) { + return HiveConnectorProperties.TEXT_COMPRESSION_PLAIN; + } + return textCompression; + } + // ========== Internal helpers ========== private List buildColumns(HmsTableInfo tableInfo) { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java index 0d1a5ee48c6129..ed3f99f2aee867 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java @@ -17,7 +17,11 @@ package org.apache.doris.connector.hive; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashSet; import java.util.Map; +import java.util.Set; /** * Property constants for Hive connector configuration. @@ -52,6 +56,38 @@ private HiveConnectorProperties() { public static final String ENABLE_MAPPING_BINARY_AS_STRING = "enable_mapping_binary_as_string"; public static final String ENABLE_MAPPING_TIMESTAMP_TZ = "enable_mapping_timestamp_tz"; + // -- CREATE TABLE / DATABASE property keys (legacy HiveMetadataOps) -- + public static final String CREATE_FILE_FORMAT = "file_format"; + public static final String CREATE_LOCATION = "location"; + public static final String CREATE_OWNER = "owner"; + public static final String CREATE_COMMENT = "comment"; + public static final String CREATE_TRANSACTIONAL = "transactional"; + /** + * Property keys that legacy {@code HiveMetadataOps} stamps into the metastore table parameters under a + * {@code doris.} prefix (so they round-trip). Mirrors legacy {@code HiveMetadataOps.DORIS_HIVE_KEYS}. + */ + public static final Set DORIS_HIVE_KEYS = Collections.unmodifiableSet( + new HashSet<>(Arrays.asList(CREATE_FILE_FORMAT, CREATE_LOCATION))); + public static final String DORIS_PROP_PREFIX = "doris."; + + // -- environment keys threaded from fe-core DefaultConnectorContext (must stay byte-identical there) -- + public static final String ENV_HIVE_DEFAULT_FILE_FORMAT = "hive_default_file_format"; + public static final String ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE = "enable_create_hive_bucket_table"; + public static final String ENV_DORIS_VERSION = "doris_version"; + /** Fallback default file format, matching legacy {@code Config.hive_default_file_format} default. */ + public static final String DEFAULT_FILE_FORMAT = "orc"; + + // -- session variable read for a text table's compression default (legacy hive_text_compression) -- + public static final String SESSION_HIVE_TEXT_COMPRESSION = "hive_text_compression"; + public static final String TEXT_COMPRESSION_UNCOMPRESSED = "uncompressed"; + public static final String TEXT_COMPRESSION_PLAIN = "plain"; + + /** + * Bucket algorithm string produced by {@code CreateTableInfoToConnectorRequestConverter} for a + * random (non-hash) distribution. Hive external tables only support hash bucketing. + */ + public static final String BUCKET_ALGO_RANDOM = "doris_random"; + /** * Parse an integer property with a default value. */ diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeConnectorContext.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeConnectorContext.java new file mode 100644 index 00000000000000..d1baf0771dbb26 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeConnectorContext.java @@ -0,0 +1,64 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.spi.ConnectorContext; + +import java.util.Collections; +import java.util.Map; + +/** + * Minimal {@link ConnectorContext} test double: carries a fixed catalog identity and an environment map (the + * channel through which fe-core threads the FE-global CREATE TABLE defaults). Everything else uses the + * interface defaults. + */ +public class FakeConnectorContext implements ConnectorContext { + + private final String catalogName; + private final long catalogId; + private final Map environment; + + public FakeConnectorContext() { + this("test_catalog", 0L, Collections.emptyMap()); + } + + public FakeConnectorContext(Map environment) { + this("test_catalog", 0L, environment); + } + + public FakeConnectorContext(String catalogName, long catalogId, Map environment) { + this.catalogName = catalogName; + this.catalogId = catalogId; + this.environment = environment == null ? Collections.emptyMap() : environment; + } + + @Override + public String getCatalogName() { + return catalogName; + } + + @Override + public long getCatalogId() { + return catalogId; + } + + @Override + public Map getEnvironment() { + return environment; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataDdlTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataDdlTest.java new file mode 100644 index 00000000000000..fa04daeab15a5d --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataDdlTest.java @@ -0,0 +1,553 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; +import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; +import org.apache.doris.connector.api.ddl.ConnectorPartitionField; +import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; +import org.apache.doris.connector.hms.HmsClientException; +import org.apache.doris.connector.hms.HmsCreateDatabaseRequest; +import org.apache.doris.connector.hms.HmsCreateTableRequest; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; + +/** + * DDL tests for {@link HiveConnectorMetadata}'s create/drop database, create/drop/truncate table overrides + * (P7.1). They run offline against a recording {@link org.apache.doris.connector.hms.HmsClient} fake (no + * live metastore, no Mockito), asserting the neutral request is turned into the same metastore write spec the + * legacy {@code HiveMetadataOps.createTableImpl}/{@code createDbImpl}/{@code dropTableImpl}/ + * {@code truncateTableImpl} produced: file-format / owner defaults, the {@code doris.}-prefixed round-trip + * parameters, the bucket gate, the transactional-table rejections, and the partition rules. + */ +public class HiveConnectorMetadataDdlTest { + + private static final String CATALOG_USER = "hive_user"; + + // ==================== createTable: file format + owner + doris.version ==================== + + @Test + public void createTableUsesEnvDefaultFileFormatAndStampsOwnerAndVersion() { + RecordingHmsClient client = new RecordingHmsClient(); + // WHY: with no file_format property the connector must fall back to the FE-global + // hive_default_file_format threaded through the environment (legacy Config.hive_default_file_format), + // stamp the connecting user as the owner (legacy set owner from ConnectContext), and stamp the build + // version threaded via the environment. MUTATION: dropping any of the three assertions' sources + // (env fallback / owner default / doris_version) flips it red. + Map env = new LinkedHashMap<>(); + env.put(HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT, "parquet"); + env.put(HiveConnectorProperties.ENV_DORIS_VERSION, "9.9-deadbeef"); + metadata(client, Collections.emptyMap(), env) + .createTable(session(), request().build()); + + HmsCreateTableRequest req = client.lastCreateTable; + Assertions.assertNotNull(req); + Assertions.assertEquals("parquet", req.getFileFormat()); + Assertions.assertEquals(CATALOG_USER, req.getProperties().get("owner")); + Assertions.assertEquals("9.9-deadbeef", req.getDorisVersion()); + } + + @Test + public void createTableUserFileFormatOverridesEnvAndRoundTripsUnderDorisPrefix() { + RecordingHmsClient client = new RecordingHmsClient(); + Map props = new LinkedHashMap<>(); + props.put("file_format", "orc"); + props.put("location", "s3://bucket/t"); + props.put("some_key", "v"); + Map env = Collections.singletonMap( + HiveConnectorProperties.ENV_HIVE_DEFAULT_FILE_FORMAT, "parquet"); + + // WHY: a user-set file_format wins over the env default; and file_format/location must round-trip as + // metastore parameters under a doris. prefix while an ordinary property keeps its plain key (legacy + // ddlProps loop). location is ALSO surfaced as the storage-descriptor location. MUTATION: dropping + // the doris. prefix, or not honoring the user file_format, flips these. + metadata(client, Collections.emptyMap(), env) + .createTable(session(), request().properties(props).build()); + + HmsCreateTableRequest req = client.lastCreateTable; + Assertions.assertEquals("orc", req.getFileFormat()); + Assertions.assertEquals("s3://bucket/t", req.getLocation()); + Assertions.assertEquals("orc", req.getProperties().get("doris.file_format")); + Assertions.assertEquals("s3://bucket/t", req.getProperties().get("doris.location")); + Assertions.assertEquals("v", req.getProperties().get("some_key")); + } + + @Test + public void createTableFallsBackToOrcWhenEnvMissing() { + RecordingHmsClient client = new RecordingHmsClient(); + // WHY: a direct-construction context with no environment (getEnvironment() empty) must still create a + // table, degrading to the hard-coded orc default (matches Config.hive_default_file_format's default). + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(session(), request().build()); + Assertions.assertEquals("orc", client.lastCreateTable.getFileFormat()); + } + + // ==================== createTable: transactional rejection ==================== + + @Test + public void createTableRejectsTransactional() { + RecordingHmsClient client = new RecordingHmsClient(); + Map props = Collections.singletonMap("transactional", "TRUE"); + // WHY: legacy rejects creating a hive transactional table (it only appears to accept inserts). The + // value check is case-insensitive. MUTATION: dropping the reject lets the create through -> the seam + // records a createTable and the assertThrows fails. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(session(), request().properties(props).build())); + Assertions.assertTrue(ex.getMessage().contains("transactional")); + Assertions.assertNull(client.lastCreateTable, "reject must happen before the metastore create"); + } + + // ==================== createTable: bucketing gate ==================== + + @Test + public void createTableBucketRejectedWhenGloballyDisabled() { + RecordingHmsClient client = new RecordingHmsClient(); + Map env = Collections.singletonMap( + HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE, "false"); + ConnectorBucketSpec bucket = new ConnectorBucketSpec( + Collections.singletonList("id"), 8, "doris_default"); + // WHY: bucketed hive tables require the FE-global enable_create_hive_bucket_table toggle (default + // off). The gate is checked BEFORE the hash requirement (legacy order). MUTATION: skipping the gate + // lets a bucket table through. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), env) + .createTable(session(), request().bucketSpec(bucket).build())); + Assertions.assertTrue(ex.getMessage().contains("enable_create_hive_bucket_table")); + } + + @Test + public void createTableBucketRejectsNonHashDistribution() { + RecordingHmsClient client = new RecordingHmsClient(); + Map env = Collections.singletonMap( + HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE, "true"); + ConnectorBucketSpec random = new ConnectorBucketSpec( + Collections.singletonList("id"), 8, HiveConnectorProperties.BUCKET_ALGO_RANDOM); + // WHY: hive external tables only support hash bucketing; a random distribution is rejected AFTER the + // enable gate passed. MUTATION: accepting random creates an unsupported table. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), env) + .createTable(session(), request().bucketSpec(random).build())); + Assertions.assertTrue(ex.getMessage().contains("hash bucketing")); + } + + @Test + public void createTableHashBucketThreadsColsAndCount() { + RecordingHmsClient client = new RecordingHmsClient(); + Map env = Collections.singletonMap( + HiveConnectorProperties.ENV_ENABLE_CREATE_HIVE_BUCKET_TABLE, "true"); + ConnectorBucketSpec hash = new ConnectorBucketSpec( + Collections.singletonList("id"), 16, "doris_default"); + // WHY: an enabled hash bucket spec must reach the write spec with its columns + count intact. + metadata(client, Collections.emptyMap(), env) + .createTable(session(), request().bucketSpec(hash).build()); + Assertions.assertEquals(Collections.singletonList("id"), client.lastCreateTable.getBucketCols()); + Assertions.assertEquals(16, client.lastCreateTable.getNumBuckets()); + } + + // ==================== createTable: partitioning ==================== + + @Test + public void createTableRejectsRangePartition() { + RecordingHmsClient client = new RecordingHmsClient(); + ConnectorPartitionSpec range = new ConnectorPartitionSpec( + ConnectorPartitionSpec.Style.RANGE, + Collections.singletonList(new ConnectorPartitionField("dt", "identity", + Collections.emptyList())), + Collections.emptyList()); + // WHY: hive supports only LIST-style partitioning (legacy rejected RANGE). MUTATION: accepting RANGE + // would build an invalid hive partition spec. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(session(), request().partitionSpec(range).build())); + Assertions.assertTrue(ex.getMessage().contains("'LIST' partition type")); + } + + @Test + public void createTableRejectsExplicitPartitionValues() { + RecordingHmsClient client = new RecordingHmsClient(); + ConnectorPartitionSpec listWithValues = new ConnectorPartitionSpec( + ConnectorPartitionSpec.Style.LIST, + Collections.singletonList(new ConnectorPartitionField("dt", "identity", + Collections.emptyList())), + Collections.emptyList(), + true /* hasExplicitPartitionValues */); + // WHY: a hive external table discovers partitions from the data layout, so explicit partition value + // definitions are rejected (legacy parity). The neutral converter drops the value expressions but + // threads this presence flag so the rejection survives. MUTATION: ignoring the flag turns a hard + // error into a silent success. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(session(), request().partitionSpec(listWithValues).build())); + Assertions.assertTrue(ex.getMessage().contains("Partition values expressions")); + } + + @Test + public void createTableListPartitionThreadsPartitionKeys() { + RecordingHmsClient client = new RecordingHmsClient(); + ConnectorPartitionSpec list = new ConnectorPartitionSpec( + ConnectorPartitionSpec.Style.LIST, + Arrays.asList( + new ConnectorPartitionField("dt", "identity", Collections.emptyList()), + new ConnectorPartitionField("region", "identity", Collections.emptyList())), + Collections.emptyList()); + // WHY: the LIST partition columns become the metastore partition keys (order preserved). MUTATION: + // dropping the field-name threading yields a non-partitioned table. + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(session(), request().partitionSpec(list).build()); + Assertions.assertEquals(Arrays.asList("dt", "region"), + client.lastCreateTable.getPartitionKeys()); + } + + // ==================== createTable: DLF default-value guard ==================== + + @Test + public void createTableRejectsColumnDefaultsOnDlfCatalog() { + RecordingHmsClient client = new RecordingHmsClient(); + Map catalogProps = Collections.singletonMap("hive.metastore.type", "dlf"); + List cols = Arrays.asList( + new ConnectorColumn("id", ConnectorType.of("INT"), null, false, null), + new ConnectorColumn("v", ConnectorType.of("INT"), null, true, "5")); + // WHY: DLF catalogs cannot honor per-column default values, so a create carrying one is rejected + // (legacy parity). This depends on the shared converter now threading the default value onto the + // column. MUTATION: dropping the DLF guard, or the column default reaching the connector, flips it. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, catalogProps, Collections.emptyMap()) + .createTable(session(), request().columns(cols).build())); + Assertions.assertTrue(ex.getMessage().contains("DLF")); + } + + @Test + public void createTableAllowsColumnDefaultsOnNonDlfCatalog() { + RecordingHmsClient client = new RecordingHmsClient(); + List cols = Arrays.asList( + new ConnectorColumn("id", ConnectorType.of("INT"), null, false, null), + new ConnectorColumn("v", ConnectorType.of("INT"), null, true, "5")); + // WHY: a plain HMS catalog keeps column defaults; the guard must only fire for DLF. MUTATION: an + // unconditional guard would wrongly reject this. + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(session(), request().columns(cols).build()); + Assertions.assertNotNull(client.lastCreateTable); + } + + // ==================== createTable: text compression default ==================== + + @Test + public void createTableThreadsTextCompressionSessionDefaultMappingUncompressedToPlain() { + RecordingHmsClient client = new RecordingHmsClient(); + Map sessionProps = Collections.singletonMap( + HiveConnectorProperties.SESSION_HIVE_TEXT_COMPRESSION, "uncompressed"); + // WHY: a text table's fallback compression comes from the hive_text_compression session variable, and + // legacy maps the "uncompressed" alias to "plain". MUTATION: skipping the alias mapping would thread + // "uncompressed" (an unsupported metastore value). + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createTable(sessionWith(sessionProps), request().build()); + Assertions.assertEquals("plain", client.lastCreateTable.getDefaultTextCompression()); + } + + // ==================== createDatabase ==================== + + @Test + public void createDatabaseSplitsLocationCommentAndParams() { + RecordingHmsClient client = new RecordingHmsClient(); + Map props = new LinkedHashMap<>(); + props.put("location", "s3://bucket/db"); + props.put("comment", "my db"); + props.put("k", "v"); + // WHY (legacy createDbImpl): the location property becomes the db location URI and is REMOVED from the + // parameters; the comment becomes the description; the rest stay as db parameters. MUTATION: leaving + // location in the parameters, or dropping the description, flips these. + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .createDatabase(session(), "db1", props); + HmsCreateDatabaseRequest req = client.lastCreateDatabase; + Assertions.assertEquals("db1", req.getDbName()); + Assertions.assertEquals("s3://bucket/db", req.getLocationUri()); + Assertions.assertEquals("my db", req.getComment()); + Assertions.assertFalse(req.getProperties().containsKey("location"), + "location must be removed from db parameters"); + Assertions.assertEquals("v", req.getProperties().get("k")); + } + + // ==================== dropTable / truncateTable ==================== + + @Test + public void dropTableRejectsTransactionalTable() { + RecordingHmsClient client = new RecordingHmsClient(); + HiveTableHandle handle = new HiveTableHandle.Builder("db1", "t1", HiveTableType.HIVE) + .tableParameters(Collections.singletonMap("transactional", "true")) + .build(); + // WHY: legacy dropTableImpl rejects dropping a hive transactional table (via + // AcidUtils.isTransactionalTable). MUTATION: dropping the check lets the drop through -> the seam + // records a dropTable and assertThrows fails. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .dropTable(session(), handle)); + Assertions.assertTrue(ex.getMessage().contains("transactional")); + Assertions.assertTrue(client.log.isEmpty(), "reject must happen before the metastore drop"); + } + + @Test + public void dropTableDelegatesForNonTransactionalTable() { + RecordingHmsClient client = new RecordingHmsClient(); + HiveTableHandle handle = new HiveTableHandle.Builder("db1", "t1", HiveTableType.HIVE) + .tableParameters(Collections.emptyMap()) + .build(); + metadata(client, Collections.emptyMap(), Collections.emptyMap()).dropTable(session(), handle); + Assertions.assertEquals(Collections.singletonList("dropTable:db1.t1"), client.log); + } + + @Test + public void truncateTableDelegatesWithPartitions() { + RecordingHmsClient client = new RecordingHmsClient(); + HiveTableHandle handle = new HiveTableHandle.Builder("db1", "t1", HiveTableType.HIVE).build(); + List partitions = Collections.singletonList("dt=2024-01-01"); + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .truncateTable(session(), handle, partitions); + Assertions.assertEquals( + Collections.singletonList("truncateTable:db1.t1:[dt=2024-01-01]"), client.log); + } + + // ==================== dropDatabase (force cascade) ==================== + + @Test + public void dropDatabaseForceCascadesTableDropsThenDb() { + RecordingHmsClient client = new RecordingHmsClient(); + client.tables.put("t1", tableInfo("db1", "t1", Collections.emptyMap())); + client.tables.put("t2", tableInfo("db1", "t2", Collections.emptyMap())); + // WHY (legacy dropDbImpl with force): every table is dropped first, then the database. MUTATION: + // dropping the cascade would call dropDatabase on a non-empty database. + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .dropDatabase(session(), "db1", false, true); + Assertions.assertEquals( + Arrays.asList("dropTable:db1.t1", "dropTable:db1.t2", "dropDatabase:db1"), client.log); + } + + @Test + public void dropDatabaseForceRejectsTransactionalTableInCascade() { + RecordingHmsClient client = new RecordingHmsClient(); + client.tables.put("t1", tableInfo("db1", "t1", + Collections.singletonMap("transactional", "true"))); + // WHY: the force cascade drops each table through the SAME transactional check as a direct DROP TABLE, + // so a transactional table aborts the whole force drop (legacy dropTableImpl propagated the error). + // MUTATION: cascading without the check would silently drop a transactional table. + Assertions.assertThrows(DorisConnectorException.class, + () -> metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .dropDatabase(session(), "db1", false, true)); + Assertions.assertTrue(client.log.isEmpty(), "transactional table must abort before any drop"); + } + + @Test + public void dropDatabaseNonForceJustDropsDb() { + RecordingHmsClient client = new RecordingHmsClient(); + client.tables.put("t1", tableInfo("db1", "t1", Collections.emptyMap())); + // WHY: without force the tables are NOT cascaded (legacy left the metastore to reject a non-empty db). + metadata(client, Collections.emptyMap(), Collections.emptyMap()) + .dropDatabase(session(), "db1", false, false); + Assertions.assertEquals(Collections.singletonList("dropDatabase:db1"), client.log); + } + + // ==================== helpers ==================== + + private static HiveConnectorMetadata metadata(RecordingHmsClient client, + Map catalogProps, Map env) { + return new HiveConnectorMetadata(client, catalogProps, new FakeConnectorContext(env)); + } + + private static ConnectorCreateTableRequest.Builder request() { + List cols = Arrays.asList( + new ConnectorColumn("id", ConnectorType.of("INT"), "id", false, null), + new ConnectorColumn("dt", ConnectorType.of("STRING"), null, true, null)); + return ConnectorCreateTableRequest.builder() + .dbName("db1") + .tableName("t1") + .columns(cols) + .comment("t comment") + .properties(new LinkedHashMap<>()); + } + + private static ConnectorSession session() { + return sessionWith(Collections.emptyMap()); + } + + private static ConnectorSession sessionWith(Map sessionProps) { + return new FakeSession(CATALOG_USER, sessionProps); + } + + private static HmsTableInfo tableInfo(String db, String table, Map params) { + return HmsTableInfo.builder().dbName(db).tableName(table).parameters(params).build(); + } + + /** Minimal recording {@link org.apache.doris.connector.hms.HmsClient}: records writes, serves canned reads. */ + private static final class RecordingHmsClient implements org.apache.doris.connector.hms.HmsClient { + private final List log = new ArrayList<>(); + private final Map tables = new LinkedHashMap<>(); + private HmsCreateTableRequest lastCreateTable; + private HmsCreateDatabaseRequest lastCreateDatabase; + + @Override + public List listDatabases() { + return Collections.emptyList(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new HmsClientException("no database: " + dbName); + } + + @Override + public List listTables(String dbName) { + return new ArrayList<>(tables.keySet()); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return tables.containsKey(tableName); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + HmsTableInfo info = tables.get(tableName); + if (info == null) { + throw new HmsClientException("no table: " + tableName); + } + return info; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return Collections.emptyList(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + return Collections.emptyList(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new HmsClientException("no partition"); + } + + @Override + public void createDatabase(HmsCreateDatabaseRequest request) { + lastCreateDatabase = request; + log.add("createDatabase:" + request.getDbName()); + } + + @Override + public void dropDatabase(String dbName) { + log.add("dropDatabase:" + dbName); + } + + @Override + public void createTable(HmsCreateTableRequest request) { + lastCreateTable = request; + log.add("createTable:" + request.getDbName() + "." + request.getTableName()); + } + + @Override + public void dropTable(String dbName, String tableName) { + log.add("dropTable:" + dbName + "." + tableName); + } + + @Override + public void truncateTable(String dbName, String tableName, List partitions) { + log.add("truncateTable:" + dbName + "." + tableName + ":" + partitions); + } + + @Override + public void close() { + } + } + + /** Minimal {@link ConnectorSession}: fixed user + a configurable session-property map. */ + private static final class FakeSession implements ConnectorSession { + private final String user; + private final Map sessionProperties; + + private FakeSession(String user, Map sessionProperties) { + this.user = user; + this.sessionProperties = sessionProperties; + } + + @Override + public String getQueryId() { + return "q"; + } + + @Override + public String getUser() { + return user; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return 0L; + } + + @Override + public String getCatalogName() { + return "hive_catalog"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + + @Override + public Map getSessionProperties() { + return sessionProperties; + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java index 51380bcf58e89b..ccde4c41f635ef 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java @@ -141,7 +141,7 @@ public void testUnpartitionedTableIsNotTouched() { private Optional> applyFilter( HiveTableHandle handle, ConnectorExpression expr) { HiveConnectorMetadata metadata = new HiveConnectorMetadata( - new FakeHmsClient(PARTITIONS), Collections.emptyMap()); + new FakeHmsClient(PARTITIONS), Collections.emptyMap(), new FakeConnectorContext()); return metadata.applyFilter(null, handle, new ConnectorFilterConstraint(expr)); } From 96e9f365bb3edfd425bd0c2e1e98d26c4de2b726 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 08:14:58 +0800 Subject: [PATCH 011/334] =?UTF-8?q?[doc](catalog)=20P7.1=20hive=20T11:=20m?= =?UTF-8?q?ark=20P7.1=20complete=20(DDL=20write=20path)=20+=20record=202?= =?UTF-8?q?=20decisions;=20handoff=20=E2=86=92=20P7.2/P7.3?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit P7.1 (hive DDL metadata write path: create/drop db+table, truncate) is done and green across fe-core + fe-connector-hive (28 unit tests, checkstyle 0, import-gate clean; non-live, cutover is P7.5). Records the two user-signed decisions this round — config threading via the connector environment channel (not the persisted catalog property map), and keeping the legacy "explicit partition values not supported" rejection (via a threaded flag, not silent ignore) — and repoints the next session at P7.2 (events) / P7.3 (HMSTransaction, critical path, R-002). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 35 +++++++++++++++-------------- plan-doc/tasks/P7-hive-migration.md | 9 +++++++- 2 files changed, 26 insertions(+), 18 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index f662987c36d947..425879b7e58c7d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,15 +5,16 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.1 进行中：T01（TRUNCATE SPI seam）+ T02–T04（插件写客户端）已实现+全绿+提交；下一步 = T05→T07（HiveConnectorMetadata DDL override）+ T08/T09/T10/T11** - -> **本轮（P7.1 T02+T03+T04 session）做了什么** —— 提交 `fdf577e7946`，**插件写客户端一个自洽单元，全在 `fe-connector-hms`**： -> 1. **T02 写 DTO**：`HmsCreateTableRequest`（builder：全列[数据+分区]、partitionKeys 名、bucketCols/numBuckets、fileFormat、comment、properties、`defaultTextCompression`、`dorisVersion`；列默认值走 `ConnectorColumn.getDefaultValue()`）+ `HmsCreateDatabaseRequest`。仿读侧 `HmsTableInfo`/`HmsDatabaseInfo`，SPI-clean。 -> 2. **T03 转换器**：`HmsTypeMapping.toHiveTypeString(ConnectorType)`（反向类型映射，等价 `dorisTypeToHiveType`；switch 于 `PrimitiveType.toString()` 名，VARCHAR→string、datetime 丢 scale、decimal p==0→9、不支持类型 throw）+ `HmsWriteConverter.toHiveTable/toHiveDatabase`（忠实移植 `HiveUtil` + `HiveProperties.setTableProperties` serde/table 属性切分；in/out/serde+压缩默认逐字照搬；`createTime` 溢出照搬）。**断耦**：`DORIS_VERSION_VALUE` 经 request 线程进（插件禁 import fe-common `Version`——import gate 禁 `org.apache.doris.common`）、text 压缩默认经 request（T06 从 session 取）、`AnalysisException`→`IllegalArgumentException`、JDK-only。 -> 3. **T04 写方法**：`HmsClient` 加 5 个写方法为 **default-throw seam**（3 个读侧 fake 不破）；`ThriftHmsClient` 用现成 `execute(...)`（auth+pool）override 全部，`createTable` 从列默认值建 `SQLDefaultConstraint`→`createTableWithConstraints`，等价 legacy `ThriftHMSCachedClient`。 -> 4. **验收**：fe-connector-hms compile SUCCESS + checkstyle 0（main+test）+ import-gate 净 + **28 单测绿**（反向映射 + 转换器，无 fake；client-dispatch/HiveConnectorMetadata 单测留 T10 recording-fake）。**无连接器 override 这些写方法（等 T05–T07），翻闸前行为不变。** +# 🎯 当前状态（2026-07-06）= **P7.1 完成：hive DDL 写路径（create/drop db+table、truncate）全部落地 + 全绿 + 提交（非 live，翻闸在 P7.5）；下一步 = 进入 P7.2（event）或 P7.3（txn，关键路径）** + +> **本轮（P7.1 T05–T11 收官 session）做了什么** —— 3 个实现 commit（`4a92…` T08 / `0713…` T09 / `c17b…` T05–T07+T10）+ 本 HANDOFF： +> 1. **T05–T07 插件 DDL override**（`c17b…`，`fe-connector-hive`）：`HiveConnectorMetadata` 加 `supportsCreateDatabase/createDatabase/dropDatabase(force cascade)/createTable/dropTable/truncateTable`——忠实移植 legacy `HiveMetadataOps`。`createTable` 全套插件侧解析：owner 默认（`session.getUser()`）、transactional 建表拒绝、file_format 默认（env，用户 per-table 属性胜）、`doris.`-前缀 round-trip 参数、LIST-only 分区 + 显式分区取值拒绝、DLF 列默认值 guard、bucket gate（env enable + hash-only）、`doris.version`/text 压缩默认线程进写请求。事务表拒绝用 handle 的 `tableParameters` 复刻 `AcidUtils.isTransactionalTable`（**不引 hive-exec**）。**`renameTable` 保持 SPI default throw**（hive 无 rename，parity）。`HiveConnector` 传 `ConnectorContext`。 +> 2. **T08 shared converter 富化**（`4a92…`，fe-core+api）：`ColumnDefinition.getDefaultValueString()` + converter 填 `ConnectorColumn.defaultValue`（原恒 null）；`ConnectorPartitionSpec.hasExplicitPartitionValues()` + converter 从 `PartitionTableInfo.getPartitionDefs()` 置位。**已核实 additive**：iceberg create `buildSchema` 不读默认值（读默认值的三处都在 ADD COLUMN 路径），paimon/maxcompute create 不读——只 hive 消费。 +> 3. **T09 config threading**（`0713…`，fe-core）：`DefaultConnectorContext.buildEnvironment` 加 `hive_default_file_format`/`enable_create_hive_bucket_table`/`doris_version` 三键，**走运行环境**（非写 catalog 属性）。 +> 4. **T10 单测**（并入 `c17b…`）：`HiveConnectorMetadataDdlTest`（20 条，recording-fake `HmsClient` + fake session/context，无 Mockito）覆盖各 guard/默认值/round-trip/force cascade；修既存 pruning 测 ctor。 +> 5. **验收全绿**：fe-core compile + fe-connector-hive test-compile SUCCESS；单测 20+8=28 条 0 fail/0 skip；checkstyle 0（api+hive+core）；import-gate 净。**非 live**（翻闸在 P7.5），无 e2e DDL（符合本阶段验收门）。 > -> **T05–T11 未动**（下一单元：`fe-connector-hive` 的 `HiveConnectorMetadata` DDL override + shared converter 列默认值 + config 注入 + 单测 + 守门）。T05–T07 需研读模板 `IcebergConnectorMetadata` 与 SPI base（`ConnectorSchemaOps`/`ConnectorTableOps`），故为下一自洽单元。 +> **本轮两个用户签字决策（2026-07-06，见 spec「P7.1 实现进度」块）**：① **config threading 机制 = 走运行环境**（非写 catalog 属性；在方案 A 内的机制细化——不落镜像/edit-log、不污染 SHOW CREATE、随全局配置刷新；有现成 `hive_metastore_client_timeout_second` 先例）。**注意**：legacy-exact（全局默认 + per-table 覆盖），**未**新增 per-catalog 覆盖属性。② **显式分区取值 = 保持报错**（converter thread 标志，hive 照 legacy 报错，非静默忽略）。 > **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： > P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 @@ -35,18 +36,18 @@ --- -# 🚀 下个 session 任务 = **执行 P7.1 实现（按 spec 末尾 P7.1-T01…T11；recon+决策已就绪，可直接写代码）** +# 🚀 下个 session 任务 = **进入 P7.2（event pipeline，可与 P7.3 并行）或 P7.3（HMSTransaction/写路径，关键路径、最难、R-002）；先 code-grounded recon → 用户签字 OQ-* → 再写代码** -> **权威计划**：**`tasks/P7-hive-migration.md`**——尤其**文件末尾「P7.1 逐 task 拆解」块**（本轮新建，含 recon 结论 + T01–T11 + OQ-HIVE-CFG=方案A 裁定）+ 上半 P7.1–P7.5 spec + [`connectors/hive.md`](./connectors/hive.md)。**单连接器 13 步 playbook = master plan §4。** -> **模块已就绪**：`fe-connector-hive`（"hms" 网关，只读 scan 已立、DDL override=0）+ `fe-connector-hms`（共享读元存储库，**只读、无任何写方法**）。**通用桥 = `PluginDrivenExternalCatalog`**（DDL 命令逐条 override + 内联 cache/editlog，连接器只实现纯 SPI 方法）。**模板 = `IcebergConnectorMetadata`**（最接近：HMS-backed + DLF guard + location cleanup + 插件侧 SchemaBuilder）。 +> **权威计划**：**`tasks/P7-hive-migration.md`**——P7.2/P7.3 子阶段 spec（上半）+「跨连接器删除排序」+「SPI 缺口（按子阶段）」块 +「开放决策 OQ-*」块 + [`connectors/hive.md`](./connectors/hive.md)。**「P7.1 逐 task 拆解」块已标 ✅ 完成（勿重做）。** 流程样板照搬 P7.1：recon → 中文讲背景+示例+推荐 → 用户签字 OQ-* → 逐 task commit+build+test+checkstyle。 +> **P7.1 已交付的地基**：`fe-connector-hms` 有完整 DDL 写客户端（create/drop db+table、truncate）；`fe-connector-hive` `HiveConnectorMetadata` DDL override 齐；env channel 已通（`DefaultConnectorContext.buildEnvironment` → `ConnectorContext.getEnvironment()`）；shared converter 已带列默认值 + 显式分区标志。**通用桥 = `PluginDrivenExternalCatalog`**；**模板 = `IcebergConnectorMetadata`**。 -## P7.1 执行要点（下个 session 开场） +## 下一子阶段选择 + 开场要点 -1. **先读** spec 末尾「P7.1 逐 task 拆解」块 + 「P7.1 实现进度」块（recon 存档，勿再 recon）。**T01–T04 已完成**（T01 truncate SPI seam + 桥 override `c0222977ebd`；T02+T03+T04 插件写客户端 `fdf577e7946`）。**从 T05 起接**：T05(HiveConnectorMetadata DB DDL) → T06(createTable) → T07(dropTable+truncate override；rename 保持 default throw) → T08(shared converter 带列默认值) → T09(config 注入=方案A) → T10(连接器单测，recording-fake `IMetaStoreClient`) → T11(守门+交接)。**信 HEAD 控制流不信 spec 行号。** T05→T07 = 下一自洽单元（连接器 DDL override，模板 `IcebergConnectorMetadata`）。**注意**：T06 须从 `ConnectorSession` 取 text 压缩默认填 `HmsCreateTableRequest.defaultTextCompression`；T09 注入 locus 须同时把 `doris.version`（值）填 `HmsCreateTableRequest.dorisVersion`（插件已留字段，值缺时转换器省略该 tag）。 -2. **本阶段范围 = 仅 DDL 写路径**：create/drop db+table + truncate。**stats/partition 写 4 法（updateTableStatistics/updatePartitionStatistics/addPartitions/dropPartition）推 P7.3**（仅 HMSTransaction 消费，本阶段加即死代码）。**rename 不实现**（hive 今天不支持，保持 SPI default throw）。**no-property-parsing**：file_format 默认/owner/bucket-gate/DLF-guard/transactional-reject 全插件侧（`ConnectorSession.getUser()`+`getCatalogProperties()`+`ConnectorColumn.defaultValue` 已够，无须扩 SPI，除 truncate seam + T08 列默认值 carrier）。 -3. **OQ-HIVE-CFG 已裁定 = 方案 A**（T09）：fe-core 组装 `hms` 连接器属性处，把 `Config.hive_default_file_format`/`Config.enable_create_hive_bucket_table` 当前值注入为属性默认（仅用户未显式设时），**插件只读属性、不碰 `common.Config`**。注入 locus 待 recon。 -4. **勿重议的已定决策**：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransaction`。后续子阶段 OQ-* 到 recon 后再用户签字（中文讲背景+示例+推荐、不引任务代号）。 -5. **纪律**：每 task/每条 fix 独立 commit + build + test + checkstyle 0（不带 -am）+ import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。P7.1 **非 live**（翻闸在 P7.5），验收靠**连接器单测**（无 Mockito、recording fake `IMetaStoreClient`），无 e2e DDL。**P7.3 ACID 写路径必须有独立集成测试作 gate（R-002 项目最大风险）**。 +1. **关键路径 = P7.3**（P7.1→P7.3→P7.4→P7.5）；P7.2（event）是可并行侧支。二选一按精力：P7.3 最难（ACID 写路径 6 文件耦合链 + `HMSTransaction` 1895 行重表达，R-002 项目最大风险，**硬门 = 独立 ACID 集成测试套件**）；P7.2 较独立（21 event 类 + processor 搬 `fe-connector-hms`，核心 fork = OQ-EVT）。 +2. **进入子阶段先 recon**（**信 HEAD 控制流不信 spec 行号**）：读该子阶段 spec +「SPI 缺口」对应块 + 真实代码，产出逐 task 拆解追加 spec 末尾（仿「P7.1 逐 task 拆解」块）。**OQ-* 到 recon 后用户签字**（中文讲背景+示例+推荐、不引任务代号）——P7.3 = OQ-RTX/OQ-ACID-WRITE/OQ-LOCK；P7.2 = OQ-EVT。 +3. **P7.3 需补的写原语**（随 `HMSTransaction` 落地才非死代码）：`HmsClient` 加 `updateTableStatistics/updatePartitionStatistics/addPartitions/dropPartition` + txn/writeId/lock/commit 方法。写路径 6 文件耦合链 retype `HMSExternalTable`→generic：`BindSink`→`LogicalHiveTableSink`→`PhysicalHiveTableSink`→`PhysicalPlanTranslator:569`→`planner/HiveTableSink`→`HiveInsertExecutor`。读侧 ACID（delete-delta）须移入插件。 +4. **勿重议的已定决策**：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransaction`。 +5. **纪律**：每 task/每条 fix 独立 commit + build + test + checkstyle 0（不带 -am）+ import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。**P7.3 ACID 写路径必须有独立集成测试作 gate（R-002 项目最大风险）**。 6. **删除排序（最硬约束，spec §跨连接器删除排序）**：`datasource/hive/` 删不掉，直到 `HudiUtils`(5 方法)/`HudiExternalMetaCache`/`HudiScanNode`(extends HiveScanNode)/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 全 retype 到 generic（P7.4）。 --- diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index bc4f8b9c3cc962..cf853a020d8a82 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -207,7 +207,14 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 - **T03**：`HmsTypeMapping.toHiveTypeString(ConnectorType)`（反向映射，等价 `HiveMetaStoreClientHelper.dorisTypeToHiveType`；switch 于 `PrimitiveType.toString()` 名——`ConnectorColumnConverter.toConnectorType` 之输出；VARCHAR→string、datetime 丢 scale、decimal p==0→9、不支持类型 throw）+ `HmsWriteConverter.toHiveTable/toHiveDatabase`（忠实移植 `HiveUtil` + `HiveProperties.setTableProperties` 的 serde/table 属性切分；orc/parquet/text 的 in/out/serde+压缩默认逐字照搬；`createTime=(int)millis*1000` 溢出照搬；MANAGED_TABLE、`doris external hive table` tag）。**断耦**：DORIS_VERSION_VALUE 经 request 线程进（插件不能 import fe-common `Version`）、text 压缩默认经 request（T06 从 session 取）、`AnalysisException`→`IllegalArgumentException`、JDK-only（无 guava/commons）。 - **T04**：`HmsClient` 加 5 个写方法（createDatabase/dropDatabase/createTable/dropTable/truncateTable）为 **default-throw seam**（读侧 fake 不受影响）；`ThriftHmsClient` 用现成 `execute(...)`（auth+pool）override 全部，`createTable` 从列默认值建 `SQLDefaultConstraint`→`createTableWithConstraints`（否则 `createTable`），等价 legacy。 - **验收**：fe-connector-hms compile SUCCESS + checkstyle 0（main+test）+ import-gate 净 + 28 单测绿（反向类型映射 + 转换器；client-dispatch/HiveConnectorMetadata 单测留 T10 recording-fake）。**无连接器 override，行为不变。** -- **下一单元 = T05→T07（`HiveConnectorMetadata` DDL override）+ T08（shared converter 带列默认值）+ T09（config 注入=方案A）+ T10（recording-fake 单测）+ T11（守门+交接）。** T06 需从 `ConnectorSession` 取 text 压缩默认填 request、并把 `doris.version`（值）与两个全局 Config 一起在注入 locus（T09）落地填 request；模板 = `IcebergConnectorMetadata`。**信 HEAD 控制流不信本 spec 行号。** +- **T05–T11 ✅ 全部完成（2026-07-06）——P7.1 收官。** 三个实现 commit： + - `4a92…` **T08**：`ColumnDefinition.getDefaultValueString()` + converter 把列默认值填进 `ConnectorColumn.defaultValue`（原恒 null）；`ConnectorPartitionSpec.hasExplicitPartitionValues()` + converter 从 `PartitionTableInfo.getPartitionDefs()` 置位（保留 hive 对显式分区取值的拒绝，见下决策）。additive、iceberg/paimon/maxcompute 建表路径不读默认值故不受影响（**已核实**：iceberg `buildSchema` 不读默认值；`IcebergCatalogOps`/`IcebergConnectorMetadata:1180` 读默认值的三处都在 **ADD COLUMN** 路径、非 create 转换器）。 + - `0713…` **T09**：`DefaultConnectorContext.buildEnvironment` 加 `hive_default_file_format`/`enable_create_hive_bucket_table`/`doris_version` 三键（**走运行环境，非写进 catalog 属性**——见决策）。 + - `c17b…` **T05–T07+T10**：`HiveConnectorMetadata` 加 `supportsCreateDatabase/createDatabase/dropDatabase(force cascade)/createTable/dropTable/truncateTable`（`renameTable` 保持 SPI default throw = hive 无 rename，parity）+ `HiveConnector` 传 `ConnectorContext` + `HiveConnectorProperties` 常量 + 20 条 recording-fake DDL 单测（无 Mockito）+ 修既存 pruning 测 ctor。事务表拒绝用 handle 的 `tableParameters` 复刻 `AcidUtils.isTransactionalTable`（不引 hive-exec）。 +- **验收全绿**：fe-core compile SUCCESS；fe-connector-hive test-compile SUCCESS；单测 20+8=28 条 0 fail/0 skip；checkstyle 0（api+hive+core）；import-gate 净。**非 live**（翻闸在 P7.5），无 e2e DDL。 +- **本轮两个用户签字决策（2026-07-06）**： + 1. **config threading 机制 = 走运行环境（environment channel），非写 catalog 属性**（在方案 A"注入默认"大方向内的机制细化）。理由：不落进元数据镜像/edit-log、不污染 `SHOW CREATE CATALOG`、每次重建连接器按当前全局配置刷新；有现成先例（`hive_metastore_client_timeout_second` 同法）。per-table `file_format` 仍胜（插件侧解析）。**注意**：这是 legacy-exact（全局默认 + per-table 覆盖），**未**新增 per-catalog 覆盖属性（legacy 也无；如需可后续加）。 + 2. **显式分区取值 = 保持报错**（非静默忽略）：converter 丢弃取值表达式但 thread `hasExplicitPartitionValues` 标志，hive 侧照 legacy 报 "Partition values expressions is not supported in hive catalog." - **T03（converter，最精细）移植指针（HEAD 行号，信控制流）**：源在 `datasource/hive/HiveUtil.java` —— `toHiveTable`(:227)、`setCompressType`(:259)、`toHiveStorageDesc`(:286)、`setFileFormat`(:301)、`toHiveSchema`(:327)、`toHiveDatabase`(:346)。**必须断的 fe-core 耦合**： - `HiveMetaStoreClientHelper.dorisTypeToHiveType(Type)`（`HiveMetaStoreClientHelper.java:529`，walk scalar/array/map/struct → hive 类型串；char(len)/decimal(p,s)；date←DATE/DATEV2、timestamp←DATETIME/DATETIMEV2；VARCHAR/STRING→string）→ **插件侧要写 `ConnectorType` 版逆映射**（`HmsTypeMapping` 是正向 hive-str→ConnectorType，需补逆向；~80 行对称）。 - `ExternalCatalog.DORIS_VERSION` / `DORIS_VERSION_VALUE` → 插件本地常量。 From 7510a79ee8ae861e84048de6d29d3bff4cbbd5b7 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 09:11:02 +0800 Subject: [PATCH 012/334] =?UTF-8?q?[doc](catalog)=20P7.3=20hive:=20recon?= =?UTF-8?q?=20+=203=20signed=20decisions=20+=20T01=E2=80=93T11=20task=20br?= =?UTF-8?q?eakdown=20(no=20impl=20code)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Enter the hive transaction/write-path subphase (critical path, R-002). Code-grounded recon (4 parallel agents, HEAD-verified) + user sign-off on three open decisions; append the P7.3 task breakdown to the phase spec and roll the HANDOFF forward. No implementation code this session. Key recon findings (baked into tasks/P7-hive-migration.md): - The generic write path already exists (built during iceberg P6): PhysicalConnectorTableSink -> PluginDrivenTableSink -> PluginDrivenInsertExecutor + ConnectorTransaction/ConnectorWritePlanProvider /PluginDrivenTransactionManager. fe-core bridge needs ZERO changes. P7.3 = fold hive INSERT into it + mirror IcebergConnectorTransaction/WriteContext. - Missing HMS write primitives (addPartitions/dropPartition/stats + ACID openTxn/allocateWriteId/lock/commitTxn/abortTxn/getValidWriteIds) all already exist on the vendored HiveMetaStoreClient -> just surface them on HmsClient + one-line execute() wrappers; no new thrift plumbing. - Latent regression to fix (T07): plugin HiveScanRange.populateTransactional HiveParams fills the delete-delta directory but drops file names (old fe-core filled both) -> transactional tables would silently mis-read once hive is flipped. Signed decisions (spec OQ section marked done): - OQ-RTX = add a neutral per-query finish callback in fe-core (Trino-aligned), replacing the hard QeProcessorImpl:210 -> Env.getHiveTransactionMgr call; hive read-txn manager moves off Env into the plugin. - OQ-ACID-WRITE = preserve current behavior (non-ACID write + ACID read migrated; full-ACID table write stays rejected) to control R-002. - OQ-LOCK = keep as-is (read-side shared HMS lock, no heartbeat). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 41 ++++++++++++----------- plan-doc/tasks/P7-hive-migration.md | 52 +++++++++++++++++++++++++++-- 2 files changed, 71 insertions(+), 22 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 425879b7e58c7d..d8a13b9d269c52 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,16 +5,18 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.1 完成：hive DDL 写路径（create/drop db+table、truncate）全部落地 + 全绿 + 提交（非 live，翻闸在 P7.5）；下一步 = 进入 P7.2（event）或 P7.3（txn，关键路径）** - -> **本轮（P7.1 T05–T11 收官 session）做了什么** —— 3 个实现 commit（`4a92…` T08 / `0713…` T09 / `c17b…` T05–T07+T10）+ 本 HANDOFF： -> 1. **T05–T07 插件 DDL override**（`c17b…`，`fe-connector-hive`）：`HiveConnectorMetadata` 加 `supportsCreateDatabase/createDatabase/dropDatabase(force cascade)/createTable/dropTable/truncateTable`——忠实移植 legacy `HiveMetadataOps`。`createTable` 全套插件侧解析：owner 默认（`session.getUser()`）、transactional 建表拒绝、file_format 默认（env，用户 per-table 属性胜）、`doris.`-前缀 round-trip 参数、LIST-only 分区 + 显式分区取值拒绝、DLF 列默认值 guard、bucket gate（env enable + hash-only）、`doris.version`/text 压缩默认线程进写请求。事务表拒绝用 handle 的 `tableParameters` 复刻 `AcidUtils.isTransactionalTable`（**不引 hive-exec**）。**`renameTable` 保持 SPI default throw**（hive 无 rename，parity）。`HiveConnector` 传 `ConnectorContext`。 -> 2. **T08 shared converter 富化**（`4a92…`，fe-core+api）：`ColumnDefinition.getDefaultValueString()` + converter 填 `ConnectorColumn.defaultValue`（原恒 null）；`ConnectorPartitionSpec.hasExplicitPartitionValues()` + converter 从 `PartitionTableInfo.getPartitionDefs()` 置位。**已核实 additive**：iceberg create `buildSchema` 不读默认值（读默认值的三处都在 ADD COLUMN 路径），paimon/maxcompute create 不读——只 hive 消费。 -> 3. **T09 config threading**（`0713…`，fe-core）：`DefaultConnectorContext.buildEnvironment` 加 `hive_default_file_format`/`enable_create_hive_bucket_table`/`doris_version` 三键，**走运行环境**（非写 catalog 属性）。 -> 4. **T10 单测**（并入 `c17b…`）：`HiveConnectorMetadataDdlTest`（20 条，recording-fake `HmsClient` + fake session/context，无 Mockito）覆盖各 guard/默认值/round-trip/force cascade；修既存 pruning 测 ctor。 -> 5. **验收全绿**：fe-core compile + fe-connector-hive test-compile SUCCESS；单测 20+8=28 条 0 fail/0 skip；checkstyle 0（api+hive+core）；import-gate 净。**非 live**（翻闸在 P7.5），无 e2e DDL（符合本阶段验收门）。 +# 🎯 当前状态（2026-07-06）= **P7.1（DDL 写）已完成合入前状态；P7.3（写/事务，关键路径）已完成 recon + 3 项决策 + 逐 task 拆解（写进 spec，无实现代码）；下一步 = 实现 P7.3-T01…T11** + +> **本轮（P7.3 开场 recon + 设计 session）做了什么** —— **无实现代码**，产出 = P7.3 逐 task 拆解 + 决策，全部写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（OQ 段已标 ✅）： +> 1. **选定子阶段**：用户在 P7.2（event）/ P7.3（写/事务）二选一中选 **P7.3（关键路径、最难、R-002）**。 +> 2. **4-agent 并行 code-grounded recon**（直读 HEAD）：写引擎 `HMSTransaction` / 6 文件写链 / 读侧 ACID / 目标 SPI+桥接。 +> 3. **关键去风险发现**：**通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）。⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，**非从零发明 seam**。 +> 4. **缺的 HMS 写原语底层全有**（vendored `HiveMetaStoreClient` 已实现 addPartitions/stats/ACID 全套）→ 缺的只是 `HmsClient` 接口开口 + 一行 `execute` 转发。老写引擎写原语已干净隔离在 `hiveOps` 后。 +> 5. **⚠️ 揪出一处已存真实回归（必修，T07）**：插件 `HiveScanRange.populateTransactionalHiveParams` 处理 ACID delete-delta **只填目录、丢文件名**（老 fe-core 两者都填）→ 翻闸后事务表静默读错。 > -> **本轮两个用户签字决策（2026-07-06，见 spec「P7.1 实现进度」块）**：① **config threading 机制 = 走运行环境**（非写 catalog 属性；在方案 A 内的机制细化——不落镜像/edit-log、不污染 SHOW CREATE、随全局配置刷新；有现成 `hive_metastore_client_timeout_second` 先例）。**注意**：legacy-exact（全局默认 + per-table 覆盖），**未**新增 per-catalog 覆盖属性。② **显式分区取值 = 保持报错**（converter thread 标志，hive 照 legacy 报错，非静默忽略）。 +> **本轮三项决策（2026-07-06，spec OQ 段已标 ✅）**：① **OQ-RTX = 主干加通用 per-query finish 回调**（连接器无关，替 `QeProcessorImpl:210→Env.getCurrentHiveTransactionMgr` 硬编，对齐 Trino；hive 读事务管理器随之从 `Env` 摘掉入插件）；② **OQ-ACID-WRITE = 迁移行为保持**（非-ACID 写 + ACID 读迁到位，full-ACID 写继续硬拒，控 R-002）；③ **OQ-LOCK = 保持现状**（读侧共享 HMS 锁无 heartbeat，原样保留）。 +> +> **P7.1 交付的地基（已在工作分支）**：`fe-connector-hms` 有 DDL 写客户端（create/drop db+table、truncate，default-throw seam + `ThriftHmsClient` 实现）；`fe-connector-hive` `HiveConnectorMetadata` DDL override 齐（txn/planWrite override 仍 0）；env channel 通；shared converter 带列默认值 + 显式分区标志。 > **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： > P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 @@ -36,19 +38,20 @@ --- -# 🚀 下个 session 任务 = **进入 P7.2（event pipeline，可与 P7.3 并行）或 P7.3（HMSTransaction/写路径，关键路径、最难、R-002）；先 code-grounded recon → 用户签字 OQ-* → 再写代码** +# 🚀 下个 session 任务 = **实现 P7.3（写/事务，关键路径）——照 spec「P7.3 逐 task 拆解」块 T01–T11 逐 task commit+build+test+checkstyle；recon+决策已完成，直接写代码** -> **权威计划**：**`tasks/P7-hive-migration.md`**——P7.2/P7.3 子阶段 spec（上半）+「跨连接器删除排序」+「SPI 缺口（按子阶段）」块 +「开放决策 OQ-*」块 + [`connectors/hive.md`](./connectors/hive.md)。**「P7.1 逐 task 拆解」块已标 ✅ 完成（勿重做）。** 流程样板照搬 P7.1：recon → 中文讲背景+示例+推荐 → 用户签字 OQ-* → 逐 task commit+build+test+checkstyle。 -> **P7.1 已交付的地基**：`fe-connector-hms` 有完整 DDL 写客户端（create/drop db+table、truncate）；`fe-connector-hive` `HiveConnectorMetadata` DDL override 齐；env channel 已通（`DefaultConnectorContext.buildEnvironment` → `ConnectorContext.getEnvironment()`）；shared converter 已带列默认值 + 显式分区标志。**通用桥 = `PluginDrivenExternalCatalog`**；**模板 = `IcebergConnectorMetadata`**。 +> **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 -## 下一子阶段选择 + 开场要点 +## 开场要点（P7.3 实现） -1. **关键路径 = P7.3**（P7.1→P7.3→P7.4→P7.5）；P7.2（event）是可并行侧支。二选一按精力：P7.3 最难（ACID 写路径 6 文件耦合链 + `HMSTransaction` 1895 行重表达，R-002 项目最大风险，**硬门 = 独立 ACID 集成测试套件**）；P7.2 较独立（21 event 类 + processor 搬 `fe-connector-hms`，核心 fork = OQ-EVT）。 -2. **进入子阶段先 recon**（**信 HEAD 控制流不信 spec 行号**）：读该子阶段 spec +「SPI 缺口」对应块 + 真实代码，产出逐 task 拆解追加 spec 末尾（仿「P7.1 逐 task 拆解」块）。**OQ-* 到 recon 后用户签字**（中文讲背景+示例+推荐、不引任务代号）——P7.3 = OQ-RTX/OQ-ACID-WRITE/OQ-LOCK；P7.2 = OQ-EVT。 -3. **P7.3 需补的写原语**（随 `HMSTransaction` 落地才非死代码）：`HmsClient` 加 `updateTableStatistics/updatePartitionStatistics/addPartitions/dropPartition` + txn/writeId/lock/commit 方法。写路径 6 文件耦合链 retype `HMSExternalTable`→generic：`BindSink`→`LogicalHiveTableSink`→`PhysicalHiveTableSink`→`PhysicalPlanTranslator:569`→`planner/HiveTableSink`→`HiveInsertExecutor`。读侧 ACID（delete-delta）须移入插件。 -4. **勿重议的已定决策**：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransaction`。 -5. **纪律**：每 task/每条 fix 独立 commit + build + test + checkstyle 0（不带 -am）+ import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。**P7.3 ACID 写路径必须有独立集成测试作 gate（R-002 项目最大风险）**。 -6. **删除排序（最硬约束，spec §跨连接器删除排序）**：`datasource/hive/` 删不掉，直到 `HudiUtils`(5 方法)/`HudiExternalMetaCache`/`HudiScanNode`(extends HiveScanNode)/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 全 retype 到 generic（P7.4）。 +1. **不发明中立 seam**：通用写入通路现成（iceberg P6 建）。P7.3 = 把 hive INSERT 折进现成通路 + 照抄 iceberg 事务。 +2. **落地顺序**（spec 已排）：**组1 写原语（T01/T02）与组2 连接器事务（T03–T05）同批 commit**（P7.1 教训：写原语不得先于事务作死代码，签名由实际调用反推）；组3 6 文件 retype（T06）依赖组2 provider/txn 就位；组4 读侧 ACID（T07，含**回归修复**）+ 组5 读事务中立 seam（T08）相对独立、可并行。 +3. **3 硬耦合结**（成为插件事务时必断）：注入 profile/queryId（去 `ConnectContext.get()`）；thrift `THivePartitionUpdate` 簇随插件走；fs 抽象须暴露对象存储 MPU complete/abort（`SpiSwitchingFileSystem.forPath→ObjFileSystem`）。 +4. **范围锁定**：full-ACID **表**写继续硬拒（OQ-ACID-WRITE）；读侧共享锁**不加 heartbeat**（OQ-LOCK）。 +5. **硬门 = 独立 ACID 集成测试套件**（T10，R-002 项目最大风险）：INSERT/OVERWRITE/分区写/delete-delta 读/rollback/多 FE 失效。⚠️端到端写测试需插件写路径 live，但翻闸在 P7.5 → 须本地/scoped 翻闸跑该套件（或与 P7.5 联跑），**勿静默跳过**（Rule 12）。 +6. **勿重议的已定决策**（P7 全局）：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransactionManager`（已存，零改动）。 +7. **纪律**：每 task 独立 commit + build + test + checkstyle 0（不带 -am）+ import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。 +8. **删除排序（最硬约束，spec §跨连接器删除排序，P7.4/P7.5 才触发）**：`datasource/hive/` 删不掉，直到 `HudiUtils`/`HudiExternalMetaCache`/`HudiScanNode`/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 全 retype 到 generic。 --- diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index cf853a020d8a82..f8a2a610afcdb8 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -135,11 +135,11 @@ gate flip + 删目录 + 删所有 instanceof + 常量搬迁 + GsonUtils 兼容 - **OQ-HET（P7.4，异构 catalog 表类）**：一个 hms catalog 混装 plain-hive（非 MVCC）+ iceberg/hudi-on-HMS（MVCC），而表的 Java 类今天在**注册进 cache 时**（读表前，为 SHOW TABLES 快）就定。方案 (a) 一律建 `PluginDrivenMvccExternalTable`、plain-hive 退化为 empty/timestamp 新鲜度；(b) 惰性到首读再定类/行为。**推荐 (a)**（避免 eager 全 catalog load；MVCC 类对 plain-hive 做 trivial no-snapshot）。 - **OQ-EVT（P7.2，事件模型）**：结构化 register/unregister（新 SPI seam）vs 纯 invalidate + 惰性 re-list（只复用现 invalidator，改 master/slave 语义 + 丢预填缓存）。**推荐**：pipeline 类搬插件，但 role-aware polling driver + edit-log 留 fe-core 薄壳（cleaner，少新 SPI）；结构事件优先降级 invalidate（惰性 reload），仅 partition-name 粒度确需扩 SPI。 -- **OQ-RTX（P7.3，reader-txn 生命周期）**：加通用 per-query finish 回调（fe-core 为所有连接器调）vs 收进 scan-provider teardown。**推荐**：通用 query-lifecycle 回调（替 `QeProcessorImpl`→`Env` 硬耦合，其他连接器也可用）。 -- **OQ-ACID-WRITE（P7.3，写 ACID 范围）**：今天 full-ACID **表**的 INSERT 是硬拒（`InsertIntoTableCommand:553`），非-ACID INSERT INTO/OVERWRITE 走 HMSTransaction，ACID **读**（delete-delta）支持。**推荐：迁移行为保持**——非-ACID 写 + ACID 读迁移到位，full-ACID 写**继续拒**（不在本阶段引入净新 ACID 写能力，控 R-002）。 +- **OQ-RTX（P7.3，reader-txn 生命周期）—— ✅ 用户签字 2026-07-06 = 方案 (a)**：加**通用 per-query finish 回调**（fe-core 为所有连接器调），替 `QeProcessorImpl`→`Env` 硬耦合。（否决 scan-provider teardown：provider 生命周期是 planning、拿不到 query-finish 信号、多 scan node 共享同一读事务，等于重造 a。） +- **OQ-ACID-WRITE（P7.3，写 ACID 范围）—— ✅ 用户签字 2026-07-06 = 方案 (a)迁移行为保持**：今天 full-ACID **表**的 INSERT 是硬拒（`InsertIntoTableCommand:553`），非-ACID INSERT INTO/OVERWRITE 走 HMSTransaction，ACID **读**（delete-delta）支持。**结论**：非-ACID 写 + ACID 读迁移到位，full-ACID 写**继续拒**（不在本阶段引入净新 ACID 写能力，控 R-002）。 - **OQ-SHOWCREATE（P7.4/P7.5，可见行为）**：hive `SHOW CREATE TABLE` 今天吐原生 Hive DDL（`HiveMetaStoreClientHelper.showCreateTable`）。加"render full DDL" SPI 逐字保 vs 接受 generic `Env.getDdlStmt`（可见输出变化）。**需产品签字**；推荐加 SPI 保持向后兼容。 - **OQ-SHARE（P7.4/P7.5，共享基类去向）**：`HiveScanNode`/`HiveSplit`/`HivePartition`/`HMSSchemaCacheValue` 被 hudi 继承。搬 `fe-connector-hms` 共享库并 re-point hudi，还是随各连接器复制。**推荐**搬 hms 共享库（单副本，对齐 D-004）。 -- **OQ-LOCK（P7.3）**：读侧共享 HMS 锁无 heartbeat。**推荐**：迁移保持现状（不静默改），若加 heartbeat 单列。 +- **OQ-LOCK（P7.3）—— ✅ 用户签字 2026-07-06 = 保持现状**：读侧共享 HMS 锁无 heartbeat；迁移**原样保留**（不静默改行为）；如需 heartbeat 作迁移后独立增强、不混入本阶段。 - **OQ-COLSTATS（P7.1/P7.4，E8）**：hive col-stats 保 HMS-metadata 快路径（读 spark col-stats + NUM_ROWS，不扫描；需扩 ConnectorStatisticsOps 加 per-column 读）vs 降级为 generic SQL-based analyze（同 iceberg/paimon，简单但丢快路径）。**推荐**：扩 SPI 保快路径（hive 大表 analyze 性能敏感）。 --- @@ -226,3 +226,49 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 - columns = **全列**（数据+分区），`toHiveSchema` 用 partitionKeys 名集切分数据列/分区列。 - **OQ-HIVE-CFG（config threading）—— ✅ 用户已裁定 2026-07-06 = 方案 A**：`hive_default_file_format`(默认 orc) + `enable_create_hive_bucket_table`(默认 false) 是 FE 全局 master-mutable Config、非 catalog 属性，插件读不到。**结论：方案 A** —— fe-core **建连接器时**把这两个全局 Config 的当前值**注入为连接器属性默认**（仅当用户未在 catalog 显式设置时；键名如 `hive.default.file.format`/`hive.enable.create.bucket.table`），插件只读属性。保留全局配置生效 + 允许 per-catalog 覆盖 + 插件不碰 `org.apache.doris.common.Config`，行为零回归。（备选 B＝纯 per-catalog 属性对齐 Trino、C＝硬编码，均**否决**：B 改用户可见行为、C 丢可调性。）→ 见 T09；注入点 = fe-core 组装 `hms` 连接器属性处（须 recon 具体 locus，勿在插件侧读 Config）。 + +--- + +# P7.3 逐 task 拆解（code-grounded recon 完成 2026-07-06；4-agent 并行直读 HEAD） + +> **recon 最重要结论（决定难度与形状）**： +> - **通用写入通路已现成**（iceberg P6 建的）：`UnboundConnectorTableSink → Physical/LogicalConnectorTableSink →`（translator `visitPhysicalConnectorTableSink`，`PhysicalPlanTranslator:671`）`→ PluginDrivenTableSink`（`:734`，连接器 `ConnectorWritePlanProvider.planWrite` 出 thrift 方言，fe-core 不再自建 `THiveTableSink`）；dispatch `InsertIntoTableCommand:565-595 → PluginDrivenInsertExecutor`。执行器已把每个 `HMSTransaction` 位点换成 SPI：`beginTransaction`(`:74-83`)=`writeOps.beginTransaction`+`PluginDrivenTransactionManager.begin(connectorTx)`；`finalizeSink`(`:91-104`)在 `planWrite` 前 `setCurrentTransaction`；`doBeforeCommit`(`:127`)读 `getUpdateCnt`；commit/rollback 走 base `onComplete`/`onFail`。**BE→FE 回传已 source-agnostic**（`LoadProcessor:233 CommitDataSerializer.feed → Transaction.addCommitData(byte[])`）。**⇒ 本阶段不发明中立 seam，只把 hive INSERT 折进现成通路 + 照抄 iceberg。fe-core 事务桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` 零改动。** +> - **模板 = `IcebergConnectorTransaction`**（`fe-connector-iceberg`，implements `ConnectorTransaction` `:118`）：begin 于 `IcebergConnectorMetadata.beginTransaction`(`:1318`)=`new IcebergConnectorTransaction(session.allocateTransactionId(), ...)`；`planWrite`(`:154`)拉 `session.getCurrentTransaction()`、建 `IcebergWriteContext`(immutable，op/overwrite/staticPartitionValues)、调 `beginWrite`(txn `:193-222`，`context.executeAuthenticated` 下 load 表 + per-op guard + `beginLock/writeStarted` 幂等)；**`commit()`(`:438`) 折进 finishInsert**（`buildPendingOperation` `:457` 按 writeOperation 选 append/overwrite/rowDelta）；`addCommitData`(`:315`)反序列化 commit 片段；`rollback`(`:1104`)/`close`(`:1113`) no-op。 +> - **`ConnectorTransaction` SPI 面**（`fe-connector-api handle/ConnectorTransaction.java`，extends `ConnectorTransactionHandle`+`Closeable` `:36`）：`getTransactionId()`(`:39`，兼作 Doris 全局 txnId)、`commit()`(`:47`)、`rollback()`(`:53`)、`close()`(`:57`)、default `addCommitData(byte[])`(`:69`，逐 BE 片段喂入累积)、default `getUpdateCnt()`(`:99`)、`profileLabel()`(`:127`)。**begin 不在此，在 `ConnectorWriteOps`**。`ConnectorWriteHandle`=每次写请求（`getTableHandle/getColumns/isOverwrite/getWriteContext():Map/getWriteOperation/getSortInfo/getBranchName`），连接器 `planWrite` 内读它建 thrift sink + per-op begin context。 +> - **缺的 HMS 写原语底层全有**：`HmsClient` 现只到 DDL（`createDatabase/dropDatabase/createTable/dropTable/truncateTable`，接口 default-throw `:151-197` + `ThriftHmsClient` 真实现 `:218-267`）。**缺**：`addPartitions`/`dropPartition`/`updateTableStatistics`/`updatePartitionStatistics` + ACID `openTxn`/`allocateWriteId`/`acquireSharedLock`/`commitTxn`/`abortTxn`/`getValidWriteIds`。但 `ThriftHmsClient` 已实例化的 vendored `HiveMetaStoreClient` **全部已实现**（`add_partitions:917/928`、`dropPartition(s):1306+`、`updateTableColumnStatistics:2214`、`updatePartitionColumnStatistics:2222`、`getTableColumnStatistics:2251`、`getValidWriteIds:2741`、`openTxn:2755`、`commitTxn:2808`、`abortTxns:2827`、`allocateTableWriteId:2859/2864`、`lock/checkLock/unlock:2885-2898`、`heartbeat*:2915/2925`）。**⇒ 缺的写原语纯是"接口开口 + 一行 `execute(...)` 转发"，无新 thrift 管道。** +> - **老写引擎写原语是干净隔离的**：`HMSTransaction`(1895) 真正碰 metastore 的调用都过 `hiveOps`(`HiveMetadataOps`)/`HMSCachedClient`（`updateTableStatistics:547`、`updatePartitionStatistics:545`、`addPartitions`(批 20)`:608`、`dropPartition:625`；接口声明 `HMSCachedClient.java:95-114`，delegate `HiveMetadataOps:403-424`）。剩下是**动作 diff 状态机 + `HmsCommitter`**（`tableActions`/`partitionActions`、`finishInsertTable:216`、`prepare*/doCommit:341/abort/rollback:182`、staging/rename/对象存储 MPU）——整体成为 hive 的 `ConnectorTransaction`。 +> - **写侧两个"事务管理器"辨析**：`transaction/HiveTransactionManager`（写侧，extends `AbstractExternalTransactionManager`，`createTransaction` new `HMSTransaction` `:40`，Env 耦合在 base）**被通用 `PluginDrivenTransactionManager` 取代 → 删除，不搬**。`datasource/hive/HiveTransactionMgr`（**读侧** ACID，55 行，`register→openTxn`、`deregister→commitTxn`）是另一回事，随读侧 ACID 入插件。 +> - **3 个硬耦合结（成为插件 `ConnectorTransaction` 时必断）**：① `HMSTransaction` ctor 从 `ConnectContext.get().getExecutor()` 抓 `SummaryProfile`(`:142-144`) + queryId ⇒ 改注入（profile 可选/no-op、queryId 经 write context）；② FE↔BE 契约 thrift `THivePartitionUpdate`/`THiveLocationParams`/`TS3MPUPendingUpload`/`TUpdateMode`/`TFileType`(`:37-41`) 须随插件走（不留 fe-core thrift）；③ 对象存储 MPU 收尾向下转型 `SpiSwitchingFileSystem.forPath → ObjFileSystem.completeMultipartUpload/abortMultipartUpload`(`:1886/:1587`) ⇒ 插件拿到的 fs 抽象须仍暴露 MPU complete/abort。 +> - **⚠️ 已存真实回归（读侧，必修）**：新插件 `HiveScanRange.populateTransactionalHiveParams`(`:150-178`) 处理 ACID delete-delta 时**只 `setDirectoryLocation`(`:168-170`)、丢弃 `setFileNames`**（builder 明明把文件名编码进 `dir|file1,file2` `:258-269`，populate 步把 `|` 后丢了）。老 fe-core `HiveScanNode.setScanParams` 两者都填（`:509-510`）。**不修 → 翻闸后事务表静默读错**。delete-delta 文件名源在 `AcidUtil.getAcidState`（`AcidInfo/DeleteDeltaInfo`，`AcidInfo.java:70-85`）。 + +## 本阶段两项用户签字决策（2026-07-06）+ 一项保持现状 + +1. **OQ-RTX = 方案 (a)**：主干加**通用 per-query finish 回调**（连接器无关，任何连接器注册查询结束钩子），替 `QeProcessorImpl.unregisterQuery:210 → Env.getCurrentHiveTransactionMgr().deregister` 的 hive 专属硬编。对齐 Trino 的 query/txn 生命周期回调模型。hive 读事务管理器随之从 `Env` 摘掉、入插件。 +2. **OQ-ACID-WRITE = 方案 (a) 迁移行为保持**：非-ACID 表 INSERT/OVERWRITE 写 + ACID 表读（delete-delta）迁到位；**full-ACID 表写继续硬拒**（不引净新 ACID 写能力，控 R-002）。 +3. **OQ-LOCK = 保持现状**：读侧共享 HMS 锁无 heartbeat，原样保留。 + +## 范围（本阶段 = 非-ACID 写路径 + ACID 读，全部落插件 + 中立化 6 文件写链 + 读事务生命周期中立 seam；硬门 = 独立 ACID 集成测试） + +> 依赖：写原语（组1）是地基；连接器事务（组2）与写计划一体；6 文件 retype（组3）依赖组2 的 provider/txn 就位才能删 legacy sink；读侧 ACID（组4）+ 读事务 seam（组5）相对独立、可与组2/3 并行。**P7.1 教训**：stats/partition 写原语**不得**先于事务作为死代码加入 → 组1 与组2 同批落地（签名由组2 的实际调用定）。 + +| task | 模块 | 内容 | +|---|---|---| +| **P7.3-T01** | fe-connector-hms | **写原语加 `HmsClient` SPI seam**（default-throw，仿 DDL `:151-197`）：`addPartitions`、`dropPartition`、`updateTableStatistics`、`updatePartitionStatistics` + ACID `openTxn`、`allocateWriteId`、`acquireSharedLock`、`commitTxn`、`abortTxn`、`getValidWriteIds`。+ 请求 DTO（fe-core-free，仿 `HmsCreateTableRequest` 风格）：partition-spec、column/partition-stats、txn/lock 参数。**签名由 T04/T05 的实际调用点反推定，避免死代码**（与组2同批 commit）。 | +| **P7.3-T02** | fe-connector-hms | **`ThriftHmsClient` 实现**：每个写原语一行 `execute(client -> client....)`（auth+pool，仿 `:218-267`）转发到 vendored `HiveMetaStoreClient`（addPartitions 批 20、stats 用 `updateTable/PartitionColumnStatistics`、ACID 用 `openTxn/allocateTableWriteId/lock/checkLock/commitTxn/abortTxns/getValidWriteIds`）。 | +| **P7.3-T03** | fe-connector-hive | **`HiveWriteContext`**（immutable，peer of `IcebergWriteContext`）：`writeOperation`、`isOverwrite`、`staticPartitionValues`、`writePath`、`fileType`（从 `ConnectorWriteHandle` 建）。 | +| **P7.3-T04** | fe-connector-hive | **`HiveConnectorTransaction implements ConnectorTransaction`**（peer of `IcebergConnectorTransaction`）：移植 `HMSTransaction` 动作 diff 状态机 + `HmsCommitter`（`finishInsertTable`/`prepare*`/`doCommit`/`abort`/`rollback`/staging-rename/MPU）。持 `HmsClient` + ACID txn/writeId/lock 状态；`addCommitData` 反序列化 `THivePartitionUpdate` 累积；`commit()`=（按 op）addPartitions/updateStatistics/rename-staging/`commitTxn`；`rollback()`=`abortTxn`+清 staging/MPU。**断 3 硬结**（注入 profile/queryId、thrift 随插件、fs 抽象暴露 MPU）。**full-ACID 写在此硬拒**（OQ-ACID-WRITE）。 | +| **P7.3-T05** | fe-connector-hive | **`HiveConnectorMetadata.beginTransaction` + `planWrite`（+ `ConnectorWritePlanProvider`）**（照 iceberg metadata `:1318`/`:154`）：`beginTransaction`=`new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient, context)`；`planWrite` 拉 `getCurrentTransaction()`、建 `HiveWriteContext`、调 begin。写计划 = 移植 `HiveTableSink.bindDataSink`（staging temp path、serde、partition values、format/压缩）产出 `THiveTableSink`。 | +| **P7.3-T06** | fe-core（6 文件写链 retype） | `HMSExternalTable`→中立 `ExternalTable`/`PluginDrivenExternalTable`，hive INSERT 折进 `PhysicalConnectorTableSink`/`PluginDrivenTableSink`/`PluginDrivenInsertExecutor`：删/退化 `BindSink.bindHiveTableSink`(`:660`)、`LogicalHiveTableSink`、`PhysicalHiveTableSink`、`PhysicalPlanTranslator.visitPhysicalHiveTableSink`(`:565`,`new HiveTableSink :569`)、`planner/HiveTableSink`、`HiveInsertExecutor` + dispatch `InsertIntoTableCommand:552,561`。连接器-specific staging/serde/partition + `doAfterCommit` 分区级 cache-refresh 移插件。（`PhysicalBaseExternalTableSink` 基字段已中立 `:44-45`，主要是 ctor 参数 + 收窄 cast。）| +| **P7.3-T07** | fe-core + 插件（读侧 ACID） | 移 `AcidUtil`/`HiveTransaction`(读侧)/`AcidInfo`/`DeleteDeltaInfo` 入插件；valid-write-ids 线程；**修 delete-delta `setFileNames` 回归**（`HiveScanRange.populateTransactionalHiveParams` 填回文件名）。锁保持无 heartbeat（OQ-LOCK）。 | +| **P7.3-T08** | fe-core（中立 seam） | **通用 per-query finish 回调**（OQ-RTX a）：fe-core 加 queryId→回调登记表，`QeProcessorImpl.unregisterQuery`(`:210`) 统一 drain（含异常路径清理），替 `Env.getCurrentHiveTransactionMgr().deregister` 硬编。hive 读事务管理器（`HiveTransactionMgr`）从 `Env`（`Env.java:568/865/1097/1101`）摘掉、入插件，经 seam 注册/注销。**连接器无关**（其他连接器可用）。 | +| **P7.3-T09** | fe-connector 单测（无 Mockito，recording-fake `IMetaStoreClient`） | 写原语 dispatch（T01/T02）；`HiveConnectorTransaction` commit/rollback 状态机（append/new/overwrite 分类、abort、full-ACID 拒绝）；`HiveWriteContext`；ACID 读 delete-delta（含**回归修复**断言：文件名被填回）。 | +| **P7.3-T10** | 集成测试（硬门，R-002） | **独立 ACID 集成测试套件**：INSERT INTO / INSERT OVERWRITE / 分区写 / delete-delta 读 / rollback / 多 FE 失效。⚠️**逻辑门槛**：端到端写测试需插件写路径 live，但翻闸在 P7.5——须在 P7.3 收尾用**本地/scoped 翻闸**跑该套件（或与 P7.5 翻闸联跑）；实现时定，勿静默跳过（Rule 12）。 | +| **P7.3-T11** | 守门 + 交接 | build SUCCESS + checkstyle 0 + import-gate 净 + 单测过 + ACID 集成套件过；更新 HANDOFF + commit。 | + +## P7.3 移植指针（HEAD 行号，信控制流不信本行号） + +- **写引擎源** `datasource/hive/HMSTransaction.java`：implements `transaction.Transaction`(`:87`)；ctor`(HiveMetadataOps, SpiSwitchingFileSystem, Executor)`(`:138`)；`beginInsertTable(HiveInsertCommandContext)`(`:205`，读 queryId/isOverwrite/fileType/writePath)；`addCommitData`(`:406`，反序列化 `THivePartitionUpdate`)；`finishInsertTable(NameMapping)`(`:216`，分类 APPEND/NEW/OVERWRITE，NEW 做 HMS 存在探测降级 APPEND `:297`)；`commit→doCommit`(`:341`，建 `HmsCommitter`→prepare→`doCommit`；throw 则 abort+rollback `:387-388`)；`rollback`(`:182`)；`HmsCommitter.updateStatisticsExecutor=newFixedThreadPool(16)`(`:1196`，`finally` shut `:1631`)；异步 rename 走注入 `fileSystemExecutor`（`FileSystemUtil.asyncRename*` `:1822/:1833`）。 +- **写计划源** `planner/HiveTableSink.java`：`bindDataSink()`(`:92`) 建 `THiveTableSink`——列 PARTITION_KEY/REGULAR 标签(`:102-116`)、已存分区 `setPartitionValues`(`:201-236`)、bucket(`:121-124`)、format+压缩(`:126-128,178-199`)、location（S3 就地 / 否则 staging temp `createTempPath` 用 `HMSExternalCatalog.HIVE_STAGING_DIR` `:131-176`）、serde(`:243-263`)、hadoop conf(`:164`)。isOverwrite/writePath/fileType 经 `HiveInsertCommandContext`(`:26-51`) round-trip。 +- **执行器驱动点** `HiveInsertExecutor`：`beforeExec`(`:61-69`)取 `(HMSTransaction) transactionManager.getTransaction(txnId)` 调 `beginInsertTable`；`doBeforeCommit`(`:72-80`)`getUpdateCnt`+`finishInsertTable`+取 `getHivePartitionUpdates`；`doAfterCommit`(`:82-120`)选择性/全量 cache refresh + edit log。commit/rollback 由 base `BaseExternalTableInsertExecutor.onComplete/onFail`(`:115/124/186/192`)。 +- **读侧 ACID 源** `AcidUtil.getAcidState`(`:223-420`，纯目录列举复刻 hive `AcidUtils`)：valid-write-ids 预序列化于 `txnValidIds` 键 `hive.txn.valid.txns`/`hive.txn.valid.writeids`(`:50-51`，`ThriftHMSCachedClient.getValidWriteIds:572-609` 产)；delete-delta **含文件名**入 `AcidInfo/DeleteDeltaInfo`(`:392-395,415`)。读事务开于 `HiveScanNode.doInitialize`(`:137-142`)→`Env.getCurrentHiveTransactionMgr().register`→`openTxn`(`HiveTransaction.java:76`)；共享锁惰性于 split 生成 `HiveTransaction.getValidWriteIds`→`acquireSharedLock(...5000)`(`:68`)；关于 `QeProcessorImpl.unregisterQuery:210`→`deregister`→`commitTxn`（早释放 split 失败 `HiveScanNode:312`）。**无 heartbeat**。 +- **iceberg 模板对读**：`IcebergConnectorTransaction.java`、`IcebergConnectorMetadata.java:154/:1318`、`PluginDrivenInsertExecutor.java:74`、`PluginDrivenTransactionManager.java:65`、`ConnectorTransaction.java`/`ConnectorWriteHandle.java`。 From 8c3467d4c4879aae36561c85f757dd9c1345d226 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 09:27:50 +0800 Subject: [PATCH 013/334] [fix](catalog) P7.3 hive T07 (partial): restore ACID delete-delta file names in HiveScanRange The plugin scan-range builder encodes each delete-delta as "dir|file1,file2" (HiveScanRange.Builder#acidInfo), but populateTransactionalHiveParams only decoded the directory and dropped everything after '|'. The old fe-core path (HiveScanNode.setScanParams) set BOTH directoryLocation and fileNames on TTransactionalHiveDeleteDeltaDesc; without the file names the BE cannot locate the delete records and silently under-deletes on transactional Hive reads. Parse the file names after '|' and set them; a directory-only encoding (no '|') still sets the directory and carries no file names. Add HiveScanRangeAcidTest covering the encode/decode round-trip, the directory-only case, and the non-transactional (no params emitted) case. This is the isolated must-fix regression called out in the P7.3 recon; the remaining T07 work (moving AcidUtil/HiveTransaction/AcidInfo into the plugin and wiring the producer that calls acidInfo) is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../doris/connector/hive/HiveScanRange.java | 17 ++- .../connector/hive/HiveScanRangeAcidTest.java | 106 ++++++++++++++++++ 2 files changed, 120 insertions(+), 3 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanRangeAcidTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java index 610bfeaeb88631..2e9863c34958cd 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanRange.java @@ -25,6 +25,7 @@ import org.apache.doris.thrift.TTransactionalHiveDesc; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -165,9 +166,19 @@ private void populateTransactionalHiveParams(TTableFormatFileDesc formatDesc) { if (deltaStr != null) { TTransactionalHiveDeleteDeltaDesc delta = new TTransactionalHiveDeleteDeltaDesc(); - delta.setDirectoryLocation(deltaStr.contains("|") - ? deltaStr.substring(0, deltaStr.indexOf('|')) - : deltaStr); + // Encoded as "dir|file1,file2" (see Builder#acidInfo). BE needs BOTH the + // delete-delta directory AND the file names to correctly apply row deletes; + // dropping the file names silently under-deletes on ACID reads. + int sep = deltaStr.indexOf('|'); + if (sep >= 0) { + delta.setDirectoryLocation(deltaStr.substring(0, sep)); + String fileNamesPart = deltaStr.substring(sep + 1); + if (!fileNamesPart.isEmpty()) { + delta.setFileNames(Arrays.asList(fileNamesPart.split(","))); + } + } else { + delta.setDirectoryLocation(deltaStr); + } deltas.add(delta); } } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanRangeAcidTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanRangeAcidTest.java new file mode 100644 index 00000000000000..5ba592feb1967e --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanRangeAcidTest.java @@ -0,0 +1,106 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.thrift.TFileRangeDesc; +import org.apache.doris.thrift.TTableFormatFileDesc; +import org.apache.doris.thrift.TTransactionalHiveDeleteDeltaDesc; +import org.apache.doris.thrift.TTransactionalHiveDesc; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.List; + +/** + * Tests that {@link HiveScanRange} carries ACID delete-delta info to the BE. + * + *

WHY: for transactional Hive tables the BE applies row-level deletes by reading the + * delete-delta files listed per partition. The delta descriptor needs BOTH the directory + * AND the file names within it. If the file names are dropped, the BE cannot locate the + * delete records and silently under-deletes — returning rows that were logically deleted. + * These tests pin the "dir|file1,file2" encode/decode round-trip end to end.

+ */ +public class HiveScanRangeAcidTest { + + @Test + public void testDeleteDeltaCarriesDirectoryAndFileNames() { + HiveScanRange range = HiveScanRange.builder() + .path("/tbl/delta_0000005_0000005/bucket_00000") + .acidInfo("/tbl/p=1", Arrays.asList( + "/tbl/p=1/delete_delta_0000003_0000003|bucket_00000,bucket_00001", + "/tbl/p=1/delete_delta_0000004_0000004|bucket_00000")) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + range.populateRangeParams(formatDesc, new TFileRangeDesc()); + + Assertions.assertTrue(formatDesc.isSetTransactionalHiveParams(), + "transactional_hive range must emit transactional params"); + TTransactionalHiveDesc txnDesc = formatDesc.getTransactionalHiveParams(); + Assertions.assertEquals("/tbl/p=1", txnDesc.getPartition()); + + List deltas = txnDesc.getDeleteDeltas(); + Assertions.assertEquals(2, deltas.size()); + + TTransactionalHiveDeleteDeltaDesc first = deltas.get(0); + Assertions.assertEquals("/tbl/p=1/delete_delta_0000003_0000003", + first.getDirectoryLocation()); + // The regression: file names must survive the "dir|file1,file2" round-trip. + Assertions.assertEquals(Arrays.asList("bucket_00000", "bucket_00001"), + first.getFileNames()); + + TTransactionalHiveDeleteDeltaDesc second = deltas.get(1); + Assertions.assertEquals("/tbl/p=1/delete_delta_0000004_0000004", + second.getDirectoryLocation()); + Assertions.assertEquals(Arrays.asList("bucket_00000"), second.getFileNames()); + } + + @Test + public void testDeleteDeltaWithoutFileNamesLeavesFileNamesUnset() { + // A directory-only encoding (no '|') must still set the directory and simply + // carry no file names, rather than mis-parsing the whole string as a directory + // with a bogus trailing file name. + HiveScanRange range = HiveScanRange.builder() + .path("/tbl/base_0000005/bucket_00000") + .acidInfo("/tbl/p=1", Arrays.asList("/tbl/p=1/delete_delta_dir_only")) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + range.populateRangeParams(formatDesc, new TFileRangeDesc()); + + TTransactionalHiveDeleteDeltaDesc delta = + formatDesc.getTransactionalHiveParams().getDeleteDeltas().get(0); + Assertions.assertEquals("/tbl/p=1/delete_delta_dir_only", delta.getDirectoryLocation()); + Assertions.assertFalse(delta.isSetFileNames()); + } + + @Test + public void testNonTransactionalRangeEmitsNoTransactionalParams() { + HiveScanRange range = HiveScanRange.builder() + .path("/tbl/000000_0") + .fileFormat("parquet") + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + range.populateRangeParams(formatDesc, new TFileRangeDesc()); + + Assertions.assertFalse(formatDesc.isSetTransactionalHiveParams()); + } +} From e48111789f1676bbb03848b3a2d62bc16235ca6f Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 09:30:26 +0800 Subject: [PATCH 014/334] [doc](catalog) P7.3 hive: record T07 delete-delta regression fix; refresh HANDOFF for next session Mark the isolated ACID delete-delta file-name regression as landed (c30fa15d99a) in both HANDOFF and the P7.3 task table; note the remaining T07 work (move AcidUtil/HiveTransaction/AcidInfo/DeleteDeltaInfo into the plugin + wire the producer) and keep T01-T05 (atomic write/transaction batch) as the next task. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 24 +++++++++++++----------- plan-doc/tasks/P7-hive-migration.md | 2 +- 2 files changed, 14 insertions(+), 12 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d8a13b9d269c52..f44393d4adde07 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,16 +5,18 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.1（DDL 写）已完成合入前状态；P7.3（写/事务，关键路径）已完成 recon + 3 项决策 + 逐 task 拆解（写进 spec，无实现代码）；下一步 = 实现 P7.3-T01…T11** - -> **本轮（P7.3 开场 recon + 设计 session）做了什么** —— **无实现代码**，产出 = P7.3 逐 task 拆解 + 决策，全部写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（OQ 段已标 ✅）： -> 1. **选定子阶段**：用户在 P7.2（event）/ P7.3（写/事务）二选一中选 **P7.3（关键路径、最难、R-002）**。 -> 2. **4-agent 并行 code-grounded recon**（直读 HEAD）：写引擎 `HMSTransaction` / 6 文件写链 / 读侧 ACID / 目标 SPI+桥接。 -> 3. **关键去风险发现**：**通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）。⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，**非从零发明 seam**。 -> 4. **缺的 HMS 写原语底层全有**（vendored `HiveMetaStoreClient` 已实现 addPartitions/stats/ACID 全套）→ 缺的只是 `HmsClient` 接口开口 + 一行 `execute` 转发。老写引擎写原语已干净隔离在 `hiveOps` 后。 -> 5. **⚠️ 揪出一处已存真实回归（必修，T07）**：插件 `HiveScanRange.populateTransactionalHiveParams` 处理 ACID delete-delta **只填目录、丢文件名**（老 fe-core 两者都填）→ 翻闸后事务表静默读错。 +# 🎯 当前状态（2026-07-06）= **P7.3 已落地首个独立可交付（T07 delete-delta 回归修复，已 commit `c30fa15d99a`）；核心写/事务批（T01–T05）+ 6 文件 retype（T06）+ 读侧 ACID 搬迁（T07 剩余）+ 中立 seam（T08）+ 测试/门（T09–T11）待做** + +> **本轮（P7.3 实现开场）做了什么** —— **1 个独立 commit**（`c30fa15d99a`），只做了 spec 揪出的那处**已存真实回归**（T07 的隔离必修部分）： +> - **修 `HiveScanRange.populateTransactionalHiveParams`**（`fe-connector-hive`）：builder `acidInfo` 把 delete-delta 编码成 `dir|file1,file2`，但 populate 步只 decode 目录、`|` 后文件名整段丢弃 → 翻闸后事务表 BE 定位不到删除记录、**静默少删**（老 fe-core `HiveScanNode.setScanParams` 两者都填）。改成解析 `|` 后逗号分隔文件名并 `setFileNames`；纯目录编码（无 `|`）仍只填目录、不带文件名。 +> - **加单测** `HiveScanRangeAcidTest`（3 例，无 Mockito）：encode/decode 往返（断言文件名回来了）+ 纯目录例 + 非事务表不发 params 例。build SUCCESS / 3 pass / checkstyle 0 / import-gate 净。 +> - **为何只做这一处**：T01–T05 是**互锁原子批**（写原语签名须由事务实际调用点反推，不得先作死代码落地——P7.1 教训），无更小的可独立编译切片；照 spec 移植 1895 行 `HMSTransaction` + `HiveTableSink.bindDataSink` 是**多 session** 工作。T07 这处回归是唯一「独立、有界、本 session 可完整交付+验证」的片。 +> - **T07 剩余未动**：`AcidUtil`/读侧 `HiveTransaction`/`AcidInfo`/`DeleteDeltaInfo` 搬入插件 + 调用 `acidInfo` 的生产者接线，仍在 fe-core，未迁。 > -> **本轮三项决策（2026-07-06，spec OQ 段已标 ✅）**：① **OQ-RTX = 主干加通用 per-query finish 回调**（连接器无关，替 `QeProcessorImpl:210→Env.getCurrentHiveTransactionMgr` 硬编，对齐 Trino；hive 读事务管理器随之从 `Env` 摘掉入插件）；② **OQ-ACID-WRITE = 迁移行为保持**（非-ACID 写 + ACID 读迁到位，full-ACID 写继续硬拒，控 R-002）；③ **OQ-LOCK = 保持现状**（读侧共享 HMS 锁无 heartbeat，原样保留）。 +> **上一轮（P7.3 recon+设计）已定的产出仍有效**（写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块 + OQ 段 ✅）： +> - **通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，非从零发明 seam。 +> - **缺的 HMS 写原语底层全有**（vendored `HiveMetaStoreClient` 已实现 addPartitions/stats/ACID 全套）→ 缺的只是 `HmsClient` 接口开口 + 一行 `execute` 转发。 +> - **三项决策（2026-07-06，spec OQ 段 ✅）**：① **OQ-RTX = 主干加通用 per-query finish 回调**（连接器无关，替 `QeProcessorImpl:210→Env.getCurrentHiveTransactionMgr` 硬编，对齐 Trino）；② **OQ-ACID-WRITE = 迁移行为保持**（非-ACID 写 + ACID 读迁到位，full-ACID 写继续硬拒，控 R-002）；③ **OQ-LOCK = 保持现状**（读侧共享 HMS 锁无 heartbeat）。 > > **P7.1 交付的地基（已在工作分支）**：`fe-connector-hms` 有 DDL 写客户端（create/drop db+table、truncate，default-throw seam + `ThriftHmsClient` 实现）；`fe-connector-hive` `HiveConnectorMetadata` DDL override 齐（txn/planWrite override 仍 0）；env channel 通；shared converter 带列默认值 + 显式分区标志。 @@ -38,14 +40,14 @@ --- -# 🚀 下个 session 任务 = **实现 P7.3（写/事务，关键路径）——照 spec「P7.3 逐 task 拆解」块 T01–T11 逐 task commit+build+test+checkstyle；recon+决策已完成，直接写代码** +# 🚀 下个 session 任务 = **实现 P7.3 核心写/事务批（T01–T05 原子批 → T06 retype → T07 剩余 + T08 → T09–T11）；T07 回归修复已单独落地（`c30fa15d99a`），勿重做** > **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 ## 开场要点（P7.3 实现） 1. **不发明中立 seam**：通用写入通路现成（iceberg P6 建）。P7.3 = 把 hive INSERT 折进现成通路 + 照抄 iceberg 事务。 -2. **落地顺序**（spec 已排）：**组1 写原语（T01/T02）与组2 连接器事务（T03–T05）同批 commit**（P7.1 教训：写原语不得先于事务作死代码，签名由实际调用反推）；组3 6 文件 retype（T06）依赖组2 provider/txn 就位；组4 读侧 ACID（T07，含**回归修复**）+ 组5 读事务中立 seam（T08）相对独立、可并行。 +2. **落地顺序**（spec 已排）：**组1 写原语（T01/T02）与组2 连接器事务（T03–T05）同批 commit**（P7.1 教训：写原语不得先于事务作死代码，签名由实际调用反推）；组3 6 文件 retype（T06）依赖组2 provider/txn 就位；组4 读侧 ACID（T07）+ 组5 读事务中立 seam（T08）相对独立、可并行。**T07 内的 delete-delta 回归修复已单独落地（`c30fa15d99a`）**；T07 剩余 = 搬 `AcidUtil`/读侧 `HiveTransaction`/`AcidInfo`/`DeleteDeltaInfo` 入插件 + 接生产者（调 `HiveScanRange.Builder#acidInfo`），届时复用已修好的 populate。 3. **3 硬耦合结**（成为插件事务时必断）：注入 profile/queryId（去 `ConnectContext.get()`）；thrift `THivePartitionUpdate` 簇随插件走；fs 抽象须暴露对象存储 MPU complete/abort（`SpiSwitchingFileSystem.forPath→ObjFileSystem`）。 4. **范围锁定**：full-ACID **表**写继续硬拒（OQ-ACID-WRITE）；读侧共享锁**不加 heartbeat**（OQ-LOCK）。 5. **硬门 = 独立 ACID 集成测试套件**（T10，R-002 项目最大风险）：INSERT/OVERWRITE/分区写/delete-delta 读/rollback/多 FE 失效。⚠️端到端写测试需插件写路径 live，但翻闸在 P7.5 → 须本地/scoped 翻闸跑该套件（或与 P7.5 联跑），**勿静默跳过**（Rule 12）。 diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index f8a2a610afcdb8..caef33a733046b 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -259,7 +259,7 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 | **P7.3-T04** | fe-connector-hive | **`HiveConnectorTransaction implements ConnectorTransaction`**（peer of `IcebergConnectorTransaction`）：移植 `HMSTransaction` 动作 diff 状态机 + `HmsCommitter`（`finishInsertTable`/`prepare*`/`doCommit`/`abort`/`rollback`/staging-rename/MPU）。持 `HmsClient` + ACID txn/writeId/lock 状态；`addCommitData` 反序列化 `THivePartitionUpdate` 累积；`commit()`=（按 op）addPartitions/updateStatistics/rename-staging/`commitTxn`；`rollback()`=`abortTxn`+清 staging/MPU。**断 3 硬结**（注入 profile/queryId、thrift 随插件、fs 抽象暴露 MPU）。**full-ACID 写在此硬拒**（OQ-ACID-WRITE）。 | | **P7.3-T05** | fe-connector-hive | **`HiveConnectorMetadata.beginTransaction` + `planWrite`（+ `ConnectorWritePlanProvider`）**（照 iceberg metadata `:1318`/`:154`）：`beginTransaction`=`new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient, context)`；`planWrite` 拉 `getCurrentTransaction()`、建 `HiveWriteContext`、调 begin。写计划 = 移植 `HiveTableSink.bindDataSink`（staging temp path、serde、partition values、format/压缩）产出 `THiveTableSink`。 | | **P7.3-T06** | fe-core（6 文件写链 retype） | `HMSExternalTable`→中立 `ExternalTable`/`PluginDrivenExternalTable`，hive INSERT 折进 `PhysicalConnectorTableSink`/`PluginDrivenTableSink`/`PluginDrivenInsertExecutor`：删/退化 `BindSink.bindHiveTableSink`(`:660`)、`LogicalHiveTableSink`、`PhysicalHiveTableSink`、`PhysicalPlanTranslator.visitPhysicalHiveTableSink`(`:565`,`new HiveTableSink :569`)、`planner/HiveTableSink`、`HiveInsertExecutor` + dispatch `InsertIntoTableCommand:552,561`。连接器-specific staging/serde/partition + `doAfterCommit` 分区级 cache-refresh 移插件。（`PhysicalBaseExternalTableSink` 基字段已中立 `:44-45`，主要是 ctor 参数 + 收窄 cast。）| -| **P7.3-T07** | fe-core + 插件（读侧 ACID） | 移 `AcidUtil`/`HiveTransaction`(读侧)/`AcidInfo`/`DeleteDeltaInfo` 入插件；valid-write-ids 线程；**修 delete-delta `setFileNames` 回归**（`HiveScanRange.populateTransactionalHiveParams` 填回文件名）。锁保持无 heartbeat（OQ-LOCK）。 | +| **P7.3-T07** | fe-core + 插件（读侧 ACID） | 移 `AcidUtil`/`HiveTransaction`(读侧)/`AcidInfo`/`DeleteDeltaInfo` 入插件；valid-write-ids 线程；~~**修 delete-delta `setFileNames` 回归**（`HiveScanRange.populateTransactionalHiveParams` 填回文件名）~~ **✅ 已单独落地 `c30fa15d99a`**（+ `HiveScanRangeAcidTest`）。锁保持无 heartbeat（OQ-LOCK）。**剩余**：搬 4 个读侧 ACID 类入插件 + 接生产者（调 `Builder#acidInfo` 产 `dir\|file1,file2`）。 | | **P7.3-T08** | fe-core（中立 seam） | **通用 per-query finish 回调**（OQ-RTX a）：fe-core 加 queryId→回调登记表，`QeProcessorImpl.unregisterQuery`(`:210`) 统一 drain（含异常路径清理），替 `Env.getCurrentHiveTransactionMgr().deregister` 硬编。hive 读事务管理器（`HiveTransactionMgr`）从 `Env`（`Env.java:568/865/1097/1101`）摘掉、入插件，经 seam 注册/注销。**连接器无关**（其他连接器可用）。 | | **P7.3-T09** | fe-connector 单测（无 Mockito，recording-fake `IMetaStoreClient`） | 写原语 dispatch（T01/T02）；`HiveConnectorTransaction` commit/rollback 状态机（append/new/overwrite 分类、abort、full-ACID 拒绝）；`HiveWriteContext`；ACID 读 delete-delta（含**回归修复**断言：文件名被填回）。 | | **P7.3-T10** | 集成测试（硬门，R-002） | **独立 ACID 集成测试套件**：INSERT INTO / INSERT OVERWRITE / 分区写 / delete-delta 读 / rollback / 多 FE 失效。⚠️**逻辑门槛**：端到端写测试需插件写路径 live，但翻闸在 P7.5——须在 P7.3 收尾用**本地/scoped 翻闸**跑该套件（或与 P7.5 翻闸联跑）；实现时定，勿静默跳过（Rule 12）。 | From 785ae1d7dda0a730031b9b6a2a977f5281b839b5 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 10:13:30 +0800 Subject: [PATCH 015/334] [refactor](catalog) P7.3 hive T08 (partial): connector-agnostic query-finish callback seam Replace the hive-specific hook hardcoded in the generic query-cleanup path (QeProcessorImpl.unregisterQuery -> Env.getCurrentHiveTransactionMgr().deregister) with a connector-agnostic QueryFinishCallbackRegistry: a connector registers a per-query cleanup callback and fe-core drains all callbacks exactly once at query finish. The hive read path now registers its read-transaction commit through the seam instead of relying on fe-core naming it. Aligns with the engine-driven query/transaction lifecycle model (e.g. Trino); implements OQ-RTX = (a). - QueryFinishCallbackRegistry: register / runAndClear (idempotent, exception-isolated). - QeProcessor: add registerQueryFinishCallback SPI; QeProcessorImpl drains generically. - HiveScanNode.doInitialize: register the deregister callback via the seam. - 6 unit tests (no Mockito): run-once, no-op-when-empty, order, idempotency, failure isolation, per-query scoping. Behavior preserved: drain key printId(queryId) matches the callback key hiveTransaction.getQueryId() (same equality the old direct deregister relied on); the early lock-release path (HiveScanNode) still commits eagerly and the finish callback then no-ops via idempotent deregister. HiveTransactionMgr stays in Env for now (still used by the legacy fe-core hive read path); it moves into the plugin later with the read-side ACID migration. Build SUCCESS / checkstyle 0 / import-gate net / 6 tests pass. Non-live (cutover P7.5). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- .../datasource/hive/source/HiveScanNode.java | 7 ++ .../java/org/apache/doris/qe/QeProcessor.java | 8 ++ .../org/apache/doris/qe/QeProcessorImpl.java | 11 +- .../doris/qe/QueryFinishCallbackRegistry.java | 76 ++++++++++++ .../qe/QueryFinishCallbackRegistryTest.java | 114 ++++++++++++++++++ 5 files changed, 214 insertions(+), 2 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/qe/QueryFinishCallbackRegistry.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/qe/QueryFinishCallbackRegistryTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java index 5d1bb33f59ae61..3474a25eb865f6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/source/HiveScanNode.java @@ -51,6 +51,7 @@ import org.apache.doris.planner.PlanNodeId; import org.apache.doris.planner.ScanContext; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.QeProcessorImpl; import org.apache.doris.qe.SessionVariable; import org.apache.doris.spi.Split; import org.apache.doris.thrift.TColumnCategory; @@ -138,6 +139,12 @@ protected void doInitialize() throws UserException { this.hiveTransaction = new HiveTransaction(DebugUtil.printId(ConnectContext.get().queryId()), ConnectContext.get().getQualifiedUser(), hmsTable, hmsTable.isFullAcidTable()); Env.getCurrentHiveTransactionMgr().register(hiveTransaction); + // Commit this hive read transaction when the query finishes, registered via the + // connector-agnostic query-finish callback so the generic query-cleanup path + // (QeProcessorImpl.unregisterQuery) no longer hardcodes a hive-specific hook. + String txnQueryId = hiveTransaction.getQueryId(); + QeProcessorImpl.INSTANCE.registerQueryFinishCallback(txnQueryId, + () -> Env.getCurrentHiveTransactionMgr().deregister(txnQueryId)); skipCheckingAcidVersionFile = sessionVariable.skipCheckingAcidVersionFile; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessor.java index c5aff2c9d5c11b..39a56deed6977b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessor.java @@ -37,6 +37,14 @@ public interface QeProcessor { void unregisterQuery(TUniqueId queryId); + /** + * Register a callback to run when the given query finishes (is unregistered). + * Connector-agnostic: lets a connector hook query completion (for example to + * commit a read transaction / release a metastore lock) without fe-core + * naming the connector in its generic query-cleanup path. + */ + void registerQueryFinishCallback(String queryId, Runnable callback); + Map getQueryStatistics(); String getCurrentQueryByQueryId(TUniqueId queryId); diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java index ff023aeb9394de..2c4202c3078757 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/QeProcessorImpl.java @@ -57,6 +57,7 @@ public final class QeProcessorImpl implements QeProcessor { private Map queryToInstancesNum; private Map userToInstancesCount; private ExecutorService writeProfileExecutor; + private final QueryFinishCallbackRegistry queryFinishCallbackRegistry = new QueryFinishCallbackRegistry(); public static final QeProcessor INSTANCE; @@ -206,8 +207,14 @@ public void unregisterQuery(TUniqueId queryId) { } } - // commit hive tranaction if needed - Env.getCurrentHiveTransactionMgr().deregister(DebugUtil.printId(queryId)); + // Run connector-registered query-finish callbacks (e.g. committing a hive + // read transaction). Connector-agnostic: fe-core names no source here. + queryFinishCallbackRegistry.runAndClear(DebugUtil.printId(queryId)); + } + + @Override + public void registerQueryFinishCallback(String queryId, Runnable callback) { + queryFinishCallbackRegistry.register(queryId, callback); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/QueryFinishCallbackRegistry.java b/fe/fe-core/src/main/java/org/apache/doris/qe/QueryFinishCallbackRegistry.java new file mode 100644 index 00000000000000..1dda4f34b94198 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/QueryFinishCallbackRegistry.java @@ -0,0 +1,76 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.qe; + +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; + +/** + * Connector-agnostic registry of callbacks to run when a query finishes. + * + *

fe-core owns the query lifecycle: when a query is unregistered (see + * {@link QeProcessorImpl#unregisterQuery}), all callbacks registered for that + * query id are run exactly once and then removed. This lets any connector hook + * query completion (for example committing a hive read transaction or releasing + * a metastore lock) without fe-core naming the connector in its generic + * query-cleanup path, aligning with the engine-driven query/transaction + * lifecycle used by systems such as Trino. + * + *

Callbacks are best-effort: a failure in one callback is logged and does + * not prevent the remaining callbacks (or the rest of query cleanup) from + * running. + */ +public class QueryFinishCallbackRegistry { + private static final Logger LOG = LogManager.getLogger(QueryFinishCallbackRegistry.class); + + private final Map> callbacks = new ConcurrentHashMap<>(); + + /** + * Register a callback to run when the query with the given id finishes. + * Multiple callbacks may be registered for one query; they run in + * registration order. + */ + public void register(String queryId, Runnable callback) { + callbacks.computeIfAbsent(queryId, k -> new CopyOnWriteArrayList<>()).add(callback); + } + + /** + * Run and remove all callbacks registered for the given query. Idempotent: + * a second call, or a call for a query with no callbacks, is a no-op. + * Exceptions thrown by an individual callback are isolated so that one + * connector's failing cleanup cannot block another's. + */ + public void runAndClear(String queryId) { + List queryCallbacks = callbacks.remove(queryId); + if (queryCallbacks == null) { + return; + } + for (Runnable callback : queryCallbacks) { + try { + callback.run(); + } catch (Exception e) { + LOG.warn("query finish callback failed for query {}", queryId, e); + } + } + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/QueryFinishCallbackRegistryTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/QueryFinishCallbackRegistryTest.java new file mode 100644 index 00000000000000..df9ede51dcbc95 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/qe/QueryFinishCallbackRegistryTest.java @@ -0,0 +1,114 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.qe; + +import com.google.common.collect.Lists; +import org.junit.Assert; +import org.junit.Test; + +import java.util.List; +import java.util.concurrent.atomic.AtomicInteger; + +public class QueryFinishCallbackRegistryTest { + + // A query-finish callback must actually run so connector cleanup + // (e.g. committing a hive read transaction) happens at query end. + @Test + public void testRunAndClearRunsRegisteredCallback() { + QueryFinishCallbackRegistry registry = new QueryFinishCallbackRegistry(); + AtomicInteger runs = new AtomicInteger(0); + registry.register("q1", runs::incrementAndGet); + + registry.runAndClear("q1"); + + Assert.assertEquals(1, runs.get()); + } + + // The generic query-cleanup path drains this registry for every query, + // most of which register nothing; draining an unknown query must be a + // harmless no-op (never throw). + @Test + public void testRunAndClearWithNoCallbackIsNoOp() { + QueryFinishCallbackRegistry registry = new QueryFinishCallbackRegistry(); + registry.runAndClear("unknown-query"); + } + + // A single query may open several transactional scans; all their cleanup + // callbacks must run, in the order they were registered. + @Test + public void testMultipleCallbacksRunInRegistrationOrder() { + QueryFinishCallbackRegistry registry = new QueryFinishCallbackRegistry(); + List order = Lists.newArrayList(); + registry.register("q1", () -> order.add(1)); + registry.register("q1", () -> order.add(2)); + registry.register("q1", () -> order.add(3)); + + registry.runAndClear("q1"); + + Assert.assertEquals(Lists.newArrayList(1, 2, 3), order); + } + + // The early lock-release path can drain a query before its final drain at + // unregisterQuery. The second drain must be a no-op so cleanup runs once. + @Test + public void testRunAndClearIsIdempotent() { + QueryFinishCallbackRegistry registry = new QueryFinishCallbackRegistry(); + AtomicInteger runs = new AtomicInteger(0); + registry.register("q1", runs::incrementAndGet); + + registry.runAndClear("q1"); + registry.runAndClear("q1"); + + Assert.assertEquals(1, runs.get()); + } + + // One connector's failing cleanup must not block another's, nor break the + // rest of query teardown: exceptions are isolated and runAndClear returns. + @Test + public void testFailingCallbackIsIsolated() { + QueryFinishCallbackRegistry registry = new QueryFinishCallbackRegistry(); + AtomicInteger runs = new AtomicInteger(0); + registry.register("q1", () -> { + throw new RuntimeException("boom"); + }); + registry.register("q1", runs::incrementAndGet); + + registry.runAndClear("q1"); + + Assert.assertEquals(1, runs.get()); + } + + // Cleanup is scoped to the finishing query: draining one query must not run + // or discard callbacks registered for a different query. + @Test + public void testCallbacksAreScopedPerQuery() { + QueryFinishCallbackRegistry registry = new QueryFinishCallbackRegistry(); + AtomicInteger q1Runs = new AtomicInteger(0); + AtomicInteger q2Runs = new AtomicInteger(0); + registry.register("q1", q1Runs::incrementAndGet); + registry.register("q2", q2Runs::incrementAndGet); + + registry.runAndClear("q1"); + + Assert.assertEquals(1, q1Runs.get()); + Assert.assertEquals(0, q2Runs.get()); + + registry.runAndClear("q2"); + Assert.assertEquals(1, q2Runs.get()); + } +} From 32148229458a6cc7a699f029c919666caa45d148 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 10:16:58 +0800 Subject: [PATCH 016/334] [doc](catalog) P7.3 hive T08 (partial): record query-finish seam landing + refresh HANDOFF; correct T07 read-side ACID boundary - Mark T08 connector-agnostic query-finish seam as landed (21aa30683dc); HiveTransactionMgr stays in Env, moves to plugin later with T07. - Record this session's recon correction to T07: the read-side ACID *produce* half is not an independent slice -- it is entangled with fe-core filesystem/ StorageProperties/FileCacheValue and shares ACID HmsClient primitives with the write batch (T01/T02); the plugin scan path currently skips ACID dirs, so producing ACID reads needs new scan-descent logic. Should advance with the write batch, not in parallel. - Roll HANDOFF forward for next session (atomic write/txn batch remains the core). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 21 ++++++++++++--------- plan-doc/tasks/P7-hive-migration.md | 4 ++-- 2 files changed, 14 insertions(+), 11 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index f44393d4adde07..6111bd138c881b 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,13 +5,16 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 已落地首个独立可交付（T07 delete-delta 回归修复，已 commit `c30fa15d99a`）；核心写/事务批（T01–T05）+ 6 文件 retype（T06）+ 读侧 ACID 搬迁（T07 剩余）+ 中立 seam（T08）+ 测试/门（T09–T11）待做** - -> **本轮（P7.3 实现开场）做了什么** —— **1 个独立 commit**（`c30fa15d99a`），只做了 spec 揪出的那处**已存真实回归**（T07 的隔离必修部分）： -> - **修 `HiveScanRange.populateTransactionalHiveParams`**（`fe-connector-hive`）：builder `acidInfo` 把 delete-delta 编码成 `dir|file1,file2`，但 populate 步只 decode 目录、`|` 后文件名整段丢弃 → 翻闸后事务表 BE 定位不到删除记录、**静默少删**（老 fe-core `HiveScanNode.setScanParams` 两者都填）。改成解析 `|` 后逗号分隔文件名并 `setFileNames`；纯目录编码（无 `|`）仍只填目录、不带文件名。 -> - **加单测** `HiveScanRangeAcidTest`（3 例，无 Mockito）：encode/decode 往返（断言文件名回来了）+ 纯目录例 + 非事务表不发 params 例。build SUCCESS / 3 pass / checkstyle 0 / import-gate 净。 -> - **为何只做这一处**：T01–T05 是**互锁原子批**（写原语签名须由事务实际调用点反推，不得先作死代码落地——P7.1 教训），无更小的可独立编译切片；照 spec 移植 1895 行 `HMSTransaction` + `HiveTableSink.bindDataSink` 是**多 session** 工作。T07 这处回归是唯一「独立、有界、本 session 可完整交付+验证」的片。 -> - **T07 剩余未动**：`AcidUtil`/读侧 `HiveTransaction`/`AcidInfo`/`DeleteDeltaInfo` 搬入插件 + 调用 `acidInfo` 的生产者接线，仍在 fe-core，未迁。 +# 🎯 当前状态（2026-07-06）= **P7.3 已落地中立读事务生命周期 seam（T08 连接器无关部分，commit `21aa30683dc`）+ 前序 delete-delta 回归修复（T07 隔离必修，commit `c30fa15d99a`）；核心写/事务批（T01–T05）+ 6 文件 retype（T06）+ 读侧 ACID 生产半搬迁（T07 剩余，本轮侦察已知与写批耦合）+ HiveTransactionMgr 入插件（T08 剩余）+ 测试/门（T09–T11）待做** + +> **本轮做了什么** —— **1 实现 commit（`21aa30683dc`）+ 1 文档 commit**，落地 T08 的**中立读事务生命周期 seam**（OQ-RTX=a 的连接器无关部分）： +> - **新 `QueryFinishCallbackRegistry`**（fe-core `qe/`）：`register(queryId, Runnable)` / `runAndClear(queryId)`（幂等 + 单回调异常隔离，一个连接器清理失败不阻断其它）。 +> - **`QeProcessor` 加 `registerQueryFinishCallback` SPI**；`QeProcessorImpl.unregisterQuery` 原 `:210` 硬编 `Env.getCurrentHiveTransactionMgr().deregister(...)`（**每条查询都点名 hive** 的通用清理代码）改为通用 drain `runAndClear(printId(queryId))` → **通用查询生命周期代码不再点名任何数据源**（架构铁律进展；对齐 Trino 引擎驱动的 query/txn 生命周期）。 +> - **`HiveScanNode.doInitialize`** 经 seam 注册 deregister 回调。**行为保持**：drain key `printId(queryId)` == 回调 key `hiveTransaction.getQueryId()`（老 deregister 依赖的同一等式）；早释放路径仍即时 commit、finish 回调再走幂等 deregister 即 no-op → 单次 commit。 +> - **6 单测（无 Mockito）**：run-once / 空查询 no-op / 多回调按序 / 幂等 / 失败隔离 / per-query 作用域。**build SUCCESS / checkstyle 0 / import-gate 净 / 6 pass**。非 live（翻闸 P7.5）。 +> - **T08 剩余**：`HiveTransactionMgr` **仍留 `Env`**（legacy fe-core 读路径 `HiveScanNode` 在用）；入插件须待 T07 读侧 ACID 迁移（届时插件经**同一 seam** 注册），故与 T07 同批。 +> - **为何本轮只做 T08（用户拍板）**：T01–T05 是**互锁原子批**（写原语签名由事务实际调用反推，不得先作死代码——P7.1 教训），照 spec 移植 1895 行 `HMSTransaction`+`HiveTableSink.bindDataSink` 是**多 session** 且中途拿不出可编译可提交结果。T08 中立 seam 是本 session 唯一「干净、有界、可独立编译+提交+测试」的片。 +> - **⚠️ 本轮侦察修正 T07 边界（已写进 spec T07 行，勿低估）**：读侧 ACID 的**消费/解码半已在插件做完+单测**（`HiveScanRange`，`.acidInfo(` 目前无生产者调用）；**生产半仍是活且与写批耦合**——`AcidUtil.getAcidState`(427) 返回 fe-core 缓存内类 `FileCacheValue`、拖入 Doris `filesystem`/`StorageProperties`/`LocationPath`/`HivePartition`；生产实调在 `HiveExternalMetaCache:816`（非 `HiveScanNode`）；`HiveTransaction` 的 `openTxn`/`getValidWriteIds`/`acquireSharedLock` **与写路径共用同批 ACID HmsClient 原语（T01/T02）**；插件 `HiveScanPlanProvider` 现**整段跳过 ACID 目录**，产 ACID 读须新增扫描下潜逻辑。⇒ **T07 生产半宜与 T01/T02 写批同批推进，非独立并行小片**。 > > **上一轮（P7.3 recon+设计）已定的产出仍有效**（写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块 + OQ 段 ✅）： > - **通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，非从零发明 seam。 @@ -40,14 +43,14 @@ --- -# 🚀 下个 session 任务 = **实现 P7.3 核心写/事务批（T01–T05 原子批 → T06 retype → T07 剩余 + T08 → T09–T11）；T07 回归修复已单独落地（`c30fa15d99a`），勿重做** +# 🚀 下个 session 任务 = **实现 P7.3 核心写/事务批（T01–T05 原子批 → T06 retype → T07 剩余 + T08 剩余 → T09–T11）；T07 delete-delta 回归修复（`c30fa15d99a`）+ T08 中立 seam（`21aa30683dc`）已落地，勿重做** > **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 ## 开场要点（P7.3 实现） 1. **不发明中立 seam**：通用写入通路现成（iceberg P6 建）。P7.3 = 把 hive INSERT 折进现成通路 + 照抄 iceberg 事务。 -2. **落地顺序**（spec 已排）：**组1 写原语（T01/T02）与组2 连接器事务（T03–T05）同批 commit**（P7.1 教训：写原语不得先于事务作死代码，签名由实际调用反推）；组3 6 文件 retype（T06）依赖组2 provider/txn 就位；组4 读侧 ACID（T07）+ 组5 读事务中立 seam（T08）相对独立、可并行。**T07 内的 delete-delta 回归修复已单独落地（`c30fa15d99a`）**；T07 剩余 = 搬 `AcidUtil`/读侧 `HiveTransaction`/`AcidInfo`/`DeleteDeltaInfo` 入插件 + 接生产者（调 `HiveScanRange.Builder#acidInfo`），届时复用已修好的 populate。 +2. **落地顺序**（spec 已排 + 本轮修正）：**组1 写原语（T01/T02）与组2 连接器事务（T03–T05）同批 commit**（P7.1 教训：写原语不得先于事务作死代码，签名由实际调用反推）；组3 6 文件 retype（T06）依赖组2 provider/txn 就位。**组5 读事务中立 seam（T08）的连接器无关部分已落地（`21aa30683dc`）**——通用 `QueryFinishCallbackRegistry` + `QeProcessorImpl` 通用 drain 替 hive 硬编 + `HiveScanNode` 经 seam 注册；其剩余（`HiveTransactionMgr` 入插件）随 T07 走。**⚠️ 组4 读侧 ACID（T07）经本轮侦察已知非独立小切片**：消费半已在插件+单测，生产半（`AcidUtil.getAcidState` + 插件扫描下潜 + `openTxn`/`getValidWriteIds`）拖入 fe-core `filesystem`/`StorageProperties`/`FileCacheValue` 且**与写批共用 ACID HmsClient 原语**→ 宜与组1/组2 写批**同批**推进，非并行独立片（详见 spec T07 行 ⚠️ 段）。T07 delete-delta 回归修复已落地（`c30fa15d99a`），复用已修好的 populate。 3. **3 硬耦合结**（成为插件事务时必断）：注入 profile/queryId（去 `ConnectContext.get()`）；thrift `THivePartitionUpdate` 簇随插件走；fs 抽象须暴露对象存储 MPU complete/abort（`SpiSwitchingFileSystem.forPath→ObjFileSystem`）。 4. **范围锁定**：full-ACID **表**写继续硬拒（OQ-ACID-WRITE）；读侧共享锁**不加 heartbeat**（OQ-LOCK）。 5. **硬门 = 独立 ACID 集成测试套件**（T10，R-002 项目最大风险）：INSERT/OVERWRITE/分区写/delete-delta 读/rollback/多 FE 失效。⚠️端到端写测试需插件写路径 live，但翻闸在 P7.5 → 须本地/scoped 翻闸跑该套件（或与 P7.5 联跑），**勿静默跳过**（Rule 12）。 diff --git a/plan-doc/tasks/P7-hive-migration.md b/plan-doc/tasks/P7-hive-migration.md index caef33a733046b..449465e2471ac5 100644 --- a/plan-doc/tasks/P7-hive-migration.md +++ b/plan-doc/tasks/P7-hive-migration.md @@ -259,8 +259,8 @@ P7.1 (metadata 地基) ──→ P7.3 (写/txn，依赖 P7.1 的 HmsClient 写 | **P7.3-T04** | fe-connector-hive | **`HiveConnectorTransaction implements ConnectorTransaction`**（peer of `IcebergConnectorTransaction`）：移植 `HMSTransaction` 动作 diff 状态机 + `HmsCommitter`（`finishInsertTable`/`prepare*`/`doCommit`/`abort`/`rollback`/staging-rename/MPU）。持 `HmsClient` + ACID txn/writeId/lock 状态；`addCommitData` 反序列化 `THivePartitionUpdate` 累积；`commit()`=（按 op）addPartitions/updateStatistics/rename-staging/`commitTxn`；`rollback()`=`abortTxn`+清 staging/MPU。**断 3 硬结**（注入 profile/queryId、thrift 随插件、fs 抽象暴露 MPU）。**full-ACID 写在此硬拒**（OQ-ACID-WRITE）。 | | **P7.3-T05** | fe-connector-hive | **`HiveConnectorMetadata.beginTransaction` + `planWrite`（+ `ConnectorWritePlanProvider`）**（照 iceberg metadata `:1318`/`:154`）：`beginTransaction`=`new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient, context)`；`planWrite` 拉 `getCurrentTransaction()`、建 `HiveWriteContext`、调 begin。写计划 = 移植 `HiveTableSink.bindDataSink`（staging temp path、serde、partition values、format/压缩）产出 `THiveTableSink`。 | | **P7.3-T06** | fe-core（6 文件写链 retype） | `HMSExternalTable`→中立 `ExternalTable`/`PluginDrivenExternalTable`，hive INSERT 折进 `PhysicalConnectorTableSink`/`PluginDrivenTableSink`/`PluginDrivenInsertExecutor`：删/退化 `BindSink.bindHiveTableSink`(`:660`)、`LogicalHiveTableSink`、`PhysicalHiveTableSink`、`PhysicalPlanTranslator.visitPhysicalHiveTableSink`(`:565`,`new HiveTableSink :569`)、`planner/HiveTableSink`、`HiveInsertExecutor` + dispatch `InsertIntoTableCommand:552,561`。连接器-specific staging/serde/partition + `doAfterCommit` 分区级 cache-refresh 移插件。（`PhysicalBaseExternalTableSink` 基字段已中立 `:44-45`，主要是 ctor 参数 + 收窄 cast。）| -| **P7.3-T07** | fe-core + 插件（读侧 ACID） | 移 `AcidUtil`/`HiveTransaction`(读侧)/`AcidInfo`/`DeleteDeltaInfo` 入插件；valid-write-ids 线程；~~**修 delete-delta `setFileNames` 回归**（`HiveScanRange.populateTransactionalHiveParams` 填回文件名）~~ **✅ 已单独落地 `c30fa15d99a`**（+ `HiveScanRangeAcidTest`）。锁保持无 heartbeat（OQ-LOCK）。**剩余**：搬 4 个读侧 ACID 类入插件 + 接生产者（调 `Builder#acidInfo` 产 `dir\|file1,file2`）。 | -| **P7.3-T08** | fe-core（中立 seam） | **通用 per-query finish 回调**（OQ-RTX a）：fe-core 加 queryId→回调登记表，`QeProcessorImpl.unregisterQuery`(`:210`) 统一 drain（含异常路径清理），替 `Env.getCurrentHiveTransactionMgr().deregister` 硬编。hive 读事务管理器（`HiveTransactionMgr`）从 `Env`（`Env.java:568/865/1097/1101`）摘掉、入插件，经 seam 注册/注销。**连接器无关**（其他连接器可用）。 | +| **P7.3-T07** | fe-core + 插件（读侧 ACID） | 移 `AcidUtil`/`HiveTransaction`(读侧)/`AcidInfo`/`DeleteDeltaInfo` 入插件；valid-write-ids 线程；~~**修 delete-delta `setFileNames` 回归**（`HiveScanRange.populateTransactionalHiveParams` 填回文件名）~~ **✅ 已单独落地 `c30fa15d99a`**（+ `HiveScanRangeAcidTest`）。锁保持无 heartbeat（OQ-LOCK）。**剩余**：搬 4 个读侧 ACID 类入插件 + 接生产者（调 `Builder#acidInfo` 产 `dir\|file1,file2`）。 **⚠️ 2026-07-06 侦察修正边界（比原表述更纠缠，勿低估）**：消费/解码半已在插件做完+单测（`HiveScanRange.acidInfo`/`populateTransactionalHiveParams`，`.acidInfo(` 目前**无生产者调用**）；**生产半才是活**——(1) `AcidUtil.getAcidState`(427 行) 返回 fe-core 缓存内类 `HiveExternalMetaCache.FileCacheValue`、拖入 Doris `filesystem`（`FileSystem`/`FileSystemTransferUtil`/`FileEntry`/`Location`）+ `StorageProperties`+`LocationPath`+`HivePartition`；(2) 生产接线**不在** `HiveScanNode`，实调在 `HiveExternalMetaCache.java:816`（`doAs(() -> AcidUtil.getAcidState(...))`）；(3) `HiveTransaction`(读侧) 绑 `HMSExternalTable`/`HMSExternalCatalog`/`HMSCachedClient`+全局 `HiveTransactionMgr`，其 `openTxn`/`getValidWriteIds`/`acquireSharedLock` **与写路径共用同一批 ACID HmsClient 原语（T01/T02）**→ T07 生产半与写批耦合；(4) 插件 `HiveScanPlanProvider` 现**整段跳过 ACID 目录**（跳目录 + `_`/`.` 前缀名），产 ACID 读须**新增插件扫描下潜逻辑**（base_/delta_/delete_delta_ + valid-write-ids），非仅搬 4 类。⇒ T07 生产半宜与 T01/T02 写原语同批推进，非独立小切片。 | +| **P7.3-T08** | fe-core（中立 seam） | **通用 per-query finish 回调**（OQ-RTX a）：fe-core 加 queryId→回调登记表，`QeProcessorImpl.unregisterQuery`(`:210`) 统一 drain（含异常路径清理），替 `Env.getCurrentHiveTransactionMgr().deregister` 硬编。hive 读事务管理器（`HiveTransactionMgr`）从 `Env`（`Env.java:568/865/1097/1101`）摘掉、入插件，经 seam 注册/注销。**连接器无关**（其他连接器可用）。 **✅ 中立 seam 已落地 `21aa30683dc`**：新 `QueryFinishCallbackRegistry`（`register`/`runAndClear`，幂等+异常隔离）+ `QeProcessor.registerQueryFinishCallback` SPI + `QeProcessorImpl.unregisterQuery` 通用 drain（替原 `:210` 的 `Env.getCurrentHiveTransactionMgr().deregister` 硬编）+ `HiveScanNode.doInitialize` 经 seam 注册 deregister 回调；6 单测（无 Mockito）。**剩余**：`HiveTransactionMgr` **仍留 `Env`**（legacy fe-core 读路径 `HiveScanNode` 在用），随 **T07 读侧 ACID 迁移**一并入插件（届时插件经同一 seam 注册）。 | | **P7.3-T09** | fe-connector 单测（无 Mockito，recording-fake `IMetaStoreClient`） | 写原语 dispatch（T01/T02）；`HiveConnectorTransaction` commit/rollback 状态机（append/new/overwrite 分类、abort、full-ACID 拒绝）；`HiveWriteContext`；ACID 读 delete-delta（含**回归修复**断言：文件名被填回）。 | | **P7.3-T10** | 集成测试（硬门，R-002） | **独立 ACID 集成测试套件**：INSERT INTO / INSERT OVERWRITE / 分区写 / delete-delta 读 / rollback / 多 FE 失效。⚠️**逻辑门槛**：端到端写测试需插件写路径 live，但翻闸在 P7.5——须在 P7.3 收尾用**本地/scoped 翻闸**跑该套件（或与 P7.5 翻闸联跑）；实现时定，勿静默跳过（Rule 12）。 | | **P7.3-T11** | 守门 + 交接 | build SUCCESS + checkstyle 0 + import-gate 净 + 单测过 + ACID 集成套件过；更新 HANDOFF + commit。 | From 4311999bcd03151cff98461e5c75b09e8bcc4056 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 12:02:10 +0800 Subject: [PATCH 017/334] [doc](catalog) P7.3 hive: HEAD-grounded recon + executable write/txn implementation design (no impl code) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Converts the P7.3 write/transaction batch from spec-level task breakdown into a single executable design doc + ordered build sequence, after a 6-agent HEAD-accurate porting recon (HMSTransaction, iceberg template, SPI surface, HmsClient, read-side ACID). Key recon corrections (now locked in the design): - Write path is a FOUR-class split mirroring iceberg (HiveConnectorTransaction + immutable HiveWriteContext + separate HiveWritePlanProvider + one-line HiveConnectorMetadata.beginTransaction); planWrite is NOT in the metadata. - fe-core insert/txn bridge needs ZERO changes; profileLabel()="HMS" reuses the existing TransactionType.HMS enum. - Read-side ACID (production half) is NOT entangled with the write-transaction machinery; the feared fe-core drag drops out at the plugin boundary — only the pure directory-name parse + hive-common ValidWriteIdList algorithm ports. - hive is not in SPI_READY_TYPES, so the whole plugin batch is dormant/safe (compiles + unit-tested, zero live routing), same as iceberg/paimon pre-cutover. - Cutover (fe-core retype, catalog identity flip, Env HiveTransactionMgr removal, SPI_READY flip, delete legacy) is explicitly OUT of this batch (later P7.4/P7.5). Delivery: user decided the whole plugin-side batch lands as ONE atomic commit, built internally in compilable dependency-order increments INC-1..INC-5 that are not individually committed. Design doc §6 is the resumable build order; §8 is the atomic-commit management strategy across sessions. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 8 +- ....3-hive-write-txn-implementation-design.md | 247 ++++++++++++++++++ 2 files changed, 253 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 6111bd138c881b..7e281114bdbc22 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,11 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 已落地中立读事务生命周期 seam（T08 连接器无关部分，commit `21aa30683dc`）+ 前序 delete-delta 回归修复（T07 隔离必修，commit `c30fa15d99a`）；核心写/事务批（T01–T05）+ 6 文件 retype（T06）+ 读侧 ACID 生产半搬迁（T07 剩余，本轮侦察已知与写批耦合）+ HiveTransactionMgr 入插件（T08 剩余）+ 测试/门（T09–T11）待做** +# 🎯 当前状态（2026-07-06 晚）= **P7.3 写/事务批已完成 HEAD 级侦察 + 可执行实现设计（新文档 [`P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md)）；用户已拍板「整批作为一次原子提交」，实现拆成 5 个可编译增量 INC-1..INC-5（内部构建顺序、逐增量编译+单测过、但不单独 commit，最后一次性提交）；下一步 = 起手实现 INC-1（HmsClient 读/写原语 + DTO + 录制式假客户端单测）。前序 seam/回归已落地（T08 `21aa30683dc`、T07 delete-delta `c30fa15d99a`），勿重做** + +> **⚠️ 权威实现计划已切换到设计文档**：本轮把 spec 的 T01–T11 拆解**收敛成一份可直接执行的设计**（4 类拆分修正、精确签名、3 硬耦合断法、INC-1..INC-5 构建顺序、逐增量测试计划、原子提交管理策略）。**下个 session 直接照 `tasks/P7.3-hive-write-txn-implementation-design.md` 的 §6 构建顺序执行**，spec 的 T01–T11 表仅作背景。 +> +> **本轮侦察的关键修正（写进设计文档，勿再纠结）**：① **写路径是「4 类拆分」**——`HiveConnectorTransaction` + 不可变 `HiveWriteContext` + **独立** `HiveWritePlanProvider`（planWrite **不在** metadata，spec 说法有误）+ `HiveConnectorMetadata.beginTransaction` 一行工厂，全对齐 iceberg。② **fe-core 桥零改动**已复核（`profileLabel()="HMS"` 复用既存 `TransactionType.HMS` 枚举）。③ **读侧 ACID（T07 生产半）并不与写事务机器耦合**——`HMSTransaction` 全程不调 `openTxn/commitTxn/getValidWriteIds`，四个读原语仅读侧用；所谓「拖入 fe-core」是浅的（插件已用裸 Hadoop FS + 自有 `HiveScanRange`，`FileCacheValue/FileSystemTransferUtil/LocationPath/StorageProperties/HivePartition/AcidInfo` 在边界处全落地），真正搬的只有纯目录名解析 + `hive-common ValidWriteIdList` 算法。④ hive **不在** `SPI_READY_TYPES`（`{jdbc,es,trino-connector,max_compute,paimon,iceberg}`）——整批**天然 dormant/安全**，编译+单测但零线上路由，同 iceberg/paimon 翻闸前范式。⑤ **翻闸/fe-core retype/Env 摘 `HiveTransactionMgr`/删 legacy 均不在本原子批**（属后续 P7.4/P7.5，另起一次原子提交）。 > **本轮做了什么** —— **1 实现 commit（`21aa30683dc`）+ 1 文档 commit**，落地 T08 的**中立读事务生命周期 seam**（OQ-RTX=a 的连接器无关部分）： > - **新 `QueryFinishCallbackRegistry`**（fe-core `qe/`）：`register(queryId, Runnable)` / `runAndClear(queryId)`（幂等 + 单回调异常隔离，一个连接器清理失败不阻断其它）。 @@ -43,7 +47,7 @@ --- -# 🚀 下个 session 任务 = **实现 P7.3 核心写/事务批（T01–T05 原子批 → T06 retype → T07 剩余 + T08 剩余 → T09–T11）；T07 delete-delta 回归修复（`c30fa15d99a`）+ T08 中立 seam（`21aa30683dc`）已落地，勿重做** +# 🚀 下个 session 任务 = **照 [`tasks/P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md) §6 构建顺序起手 INC-1（HmsClient 原语+DTO+录制式假客户端单测）→ INC-2..INC-5；全批作为一次原子提交（用户拍板），逐增量保持树可编译+单测过、但不单独 commit** > **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md new file mode 100644 index 00000000000000..e98879bacd426d --- /dev/null +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -0,0 +1,247 @@ +# P7.3 — Hive Write/Transaction Plugin Batch — Implementation Design + +> **Status**: DESIGN (pre-implementation). Research complete (HEAD-accurate 6-agent recon, archived reasoning below). +> **Authoritative recon**: `plan-doc/tasks/P7-hive-migration.md` §"P7.3 逐 task 拆解" + the 2026-07-06 porting-map workflow (readers: HMSTransaction, write-plan/T06, iceberg-template, SPI-surface, HmsClient, read-ACID). +> **Trust HEAD control flow, not the line numbers quoted here** (they were HEAD-accurate on 2026-07-06; re-verify before each edit). +> **Delivery constraint (user decision 2026-07-06)**: this whole plugin-side batch lands as **ONE atomic commit**. Built internally in **compilable dependency-order increments** (below) that are *never individually git-committed*; the working tree stays compilable at every stopping point so the multi-session port is resumable, and the single feature commit happens only when the full batch is green. Design-doc / HANDOFF commits are separate and allowed. + +--- + +## 1. Goal / Non-goals / Scope + +**Goal**: Build, entirely inside the connector plugins (`fe-connector-hms` + `fe-connector-hive`), the **complete but dormant** capability for: +- Hive non-ACID **INSERT / INSERT OVERWRITE** (partitioned + non-partitioned) write, as a plugin `ConnectorTransaction` mirroring `IcebergConnectorTransaction`. +- Hive **ACID-table READ** (base/delta/delete-delta directory descent + snapshot isolation), producing the `HiveScanRange.acidInfo(...)` fragments whose consumer half already landed. + +**Dormant = not live**: `HIVE` stays **out of `CatalogFactory.SPI_READY_TYPES`** (currently `{jdbc, es, trino-connector, max_compute, paimon, iceberg}`). Nothing routes live hive reads/writes through the new code until the cutover. This is the iceberg/paimon pre-cutover pattern verbatim. + +**Non-goals (explicitly OUT of this atomic batch — later P7.4/P7.5 cutover, a separate atomic commit)**: +- fe-core write-chain retype / cutover (delete `LogicalHiveTableSink`/`PhysicalHiveTableSink`/`planner.HiveTableSink`/`HiveInsertExecutor`, route hive INSERT through the generic `PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor`). *(was called T06)* +- Catalog identity flip `HMSExternalCatalog → PluginDrivenExternalCatalog` / `HMSExternalTable → PluginDrivenExternalTable` (prereq for the generic path's hard-casts). +- Removing `HiveTransactionMgr` from `Env` (touches the **live** fe-core `HiveScanNode`) and rewiring the live read path. +- Adding `HIVE` to `SPI_READY_TYPES`. +- Deleting fe-core `datasource/hive`, `datasource/hudi`, the 23 HMS-iceberg support classes. + +**Out of feature scope (signed decisions)**: +- **full-ACID table WRITE stays hard-rejected** (OQ-ACID-WRITE). Only non-ACID INSERT/OVERWRITE writes. *(full-ACID READ is in scope — must still be detected to pick the correct file filter + emit delete-delta merge.)* +- No DELETE / UPDATE / MERGE / REWRITE / branch / MoR machinery (iceberg-only; the hive commit `switch` stays `{INSERT, OVERWRITE}`). +- Read-side shared HMS lock keeps **no heartbeat** (OQ-LOCK). + +--- + +## 2. Locked decisions (do not re-litigate during implementation) + +| # | Decision | Rationale | +|---|---|---| +| **D1** | **Four-class split**, mirroring iceberg: `HiveConnectorTransaction implements ConnectorTransaction` + immutable `HiveWriteContext` + `HiveWritePlanProvider implements ConnectorWritePlanProvider` + one-line `HiveConnectorMetadata.beginTransaction`. **`planWrite` is NOT in the metadata.** | Corrects the spec (which said planWrite is in metadata). Iceberg `planWrite` lives in `IcebergWritePlanProvider`, metadata has only the begin factory. | +| **D2** | `profileLabel()` returns **`"HMS"`**. | `TransactionType.HMS` already exists in fe-core → **zero fe-core change**. `"HIVE"` would fall back to `UNKNOWN` or force a fe-core enum edit. | +| **D3** | `getUpdateCnt()` = **sum of `THivePartitionUpdate.getRowCount()`**. | Preserves legacy `HMSTransaction.getUpdateCnt()` behavior (migration-faithful). | +| **D4** | **Drop query profiling** in the plugin (no `ConnectContext` / `SummaryProfile`). `queryId` arrives via `HiveWriteContext`. Collapse each `wrapper*WithProfileSummary` to its plain body. | Iceberg dropped profiling identically. Removes the `org.apache.doris.qe` / `common.profile` coupling (#1) with zero behavior loss beyond profile timings. | +| **D5** | **Auth**: metastore calls are auth'd **for free** inside `ThriftHmsClient.execute()` (it already does `doAs` + system-CL TCCL pin). Only the **plugin-owned async fs/rename/MPU tasks** must each be wrapped in `context.executeAuthenticated`. **No JVM-wide worker-pool primer** (hive fans work onto its *own* pool, not a shared SDK pool — unlike iceberg's `ThreadPools.getWorkerPool()`). | Recon: iceberg's `pinIcebergWorkerPoolToPluginClassLoader` is SDK-specific. Per-task wrap is correct for hive. See coupling #3. | +| **D6** | **MPU downcast**: replace `((SpiSwitchingFileSystem) fs).forPath(path)` with a **plugin-side `fe-filesystem-spi` FileSystem** built from `context.getStorageProperties()`; keep the `instanceof ObjFileSystem` guard + `objFs.completeMultipartUpload` (complete) and `objFs.getObjStorage().abortMultipartUpload` (abort). | `SpiSwitchingFileSystem` is fe-core; `ObjFileSystem`/`ObjStorage` are plugin-visible (`org.apache.doris.filesystem.spi`). | +| **D7** | full-ACID **write** hard-reject in the transaction begin-guard; full-ACID **read** detected + `FullAcidFileFilter` + delete-delta merge. | OQ-ACID-WRITE. Do not conflate "reject full-acid write" with "skip full-acid read". | +| **D8** | `isTransactional` / `isFullAcid` derived **plugin-side** from `HiveTableHandle.getTableParameters()` (the raw HMS props: `transactional`, `transactional_properties`). | fe-core parses no properties (project rule). Mirrors `HMSExternalTable.isHiveTransactionalTable/isFullAcidTable`. | +| **D9** | `rollback()` **deletes staging files + aborts MPU** (port legacy `pruneAndDeleteStagingDirectories` + `abortMultiUploads`). NOT a no-op. | Hive writes data files to staging **before** commit (unlike iceberg's manifest-only staging). Genuine divergence from the iceberg template. | +| **D10** | The transaction uses the **plugin `HmsClient` SPI**, not fe-core `HiveMetadataOps`/`HMSCachedClient`. Write-primitive DTOs are plugin-side (`connector-api` + JDK types). | Keeps the plugin fe-core-free. `HmsClient` is the metastore seam. | +| **D11** | thrift `THivePartitionUpdate` (commit fragment) + `THiveTableSink` / `TDataSinkType.HIVE_TABLE_SINK` (sink) travel with the plugin as-is. | The shared thrift module is already compile-visible to `fe-connector-hive` (`HiveScanRange` imports `org.apache.doris.thrift.*`). Verified: `HIVE_TABLE_SINK=13`, structs at `DataSinks.thrift:365/393`. | +| **D12** | Read-txn primitives are a **strict subset** of the write-primitive HmsClient surface and are used **only by the read side** (`HMSTransaction` never calls `openTxn/commitTxn/getValidWriteIds`). Surface all HmsClient primitives once (read + write) in this batch. | Recon-confirmed by grep. Avoids double-surfacing. | + +--- + +## 3. Architecture + +### 3.1 Write path (dormant) + +``` +(cutover, later) UnboundConnectorTableSink → LogicalConnectorTableSink → PhysicalConnectorTableSink + → PluginDrivenTableSink → PluginDrivenInsertExecutor [fe-core bridge, ZERO changes] + │ beginTransaction() ──────────────► HiveConnectorMetadata.beginTransaction (D1 one-liner) + │ └─► new HiveConnectorTransaction(txnId, hmsClient, context) + │ finalizeSink → planWrite() ──────────► HiveWritePlanProvider.planWrite + │ ├─ transaction.beginWrite(session, db, table, HiveWriteContext) + │ └─ buildSink() → TDataSink(HIVE_TABLE_SINK).setHiveTableSink(THiveTableSink) + │ BE reports fragments → addCommitData(byte[]) ─► deserialize THivePartitionUpdate, accumulate + │ doBeforeCommit → getUpdateCnt() ─────► Σ rowCount (D3) + └─ onComplete → commit() / onFail → rollback() ─► HiveConnectorTransaction +``` + +`HiveConnectorTransaction` internals (the genuinely-atomic core — the ported `HMSTransaction`): +- **Accumulator**: `List` fed by `addCommitData` (deserialize `TDeserializer(TBinaryProtocol)`, `synchronized`). +- **Classification state machine** (`finishInsertTable`, run inside `commit()` or `beginWrite`-then-commit — see §5.1): `tableActions` / `partitionActions` maps of `Action` (`ADD/ALTER/DROP/DROP_PRESERVE_DATA/INSERT_EXISTING/MERGE`); classify each write as **APPEND / NEW / OVERWRITE**; **NEW→APPEND downgrade** on HMS existence probe (`partitionExists`) for Doris cache misses. +- **`HmsCommitter`** (`prepare*` → `doCommit` → `abort`/`rollback`): staging→target renames (async, plugin-owned `Executor`), object-store **MPU complete/abort** (D6), HMS `addPartitions` (batch 20) + `updateTable/PartitionStatistics` (parallel on owned `newFixedThreadPool(16)`, shut down in `close()`). +- SPI methods: `getTransactionId` / `commit` / `rollback` / `close` / `addCommitData` / `getUpdateCnt` / `profileLabel="HMS"`. Inherits all iceberg/maxcompute-only defaults untouched. + +### 3.2 Read path (dormant) + +``` +(live path today = fe-core HiveScanNode; plugin path dormant until read cutover) +HiveScanPlanProvider.planScan + └─ (for transactional partitions) NEW ACID descent [ports AcidUtil.getAcidState PURE algorithm] + ├─ plugin HiveTransaction.openTxn / acquireSharedLock / getValidWriteIds [HmsClient primitives] + ├─ glob partition dir → parse base_/delta_/delete_delta_ names → write-id snapshot filter + ├─ emit HiveScanRange per surviving base/delta bucket_ file (FullAcidFileFilter | InsertOnlyFileFilter) + └─ HiveScanRange.Builder.acidInfo(partitionLocation, deltaDescriptors) [producer — consumer already landed] + plugin read-txn manager (keyed by queryId) registered via QueryFinishCallbackRegistry seam (already landed) + └─ query finish → commitTxn (lock release) +``` + +The read side is **NOT entangled** with the write-transaction machinery — it only shares the four HmsClient primitives. `getAcidState`'s feared fe-core drag (`FileCacheValue`/`FileSystemTransferUtil`/`LocationPath`/`StorageProperties`/`HivePartition`/`AcidInfo`) **drops out at the plugin boundary**: the plugin already lists files with raw `org.apache.hadoop.fs.FileSystem` and emits `HiveScanRange`, so only the **pure name-parsing + `hive-common ValidWriteIdList/ValidTxnList`** algorithm ports. + +--- + +## 4. Interfaces / APIs (exact signatures) + +### 4.1 `HmsClient` additions (`fe-connector-hms`, `default`-throw block mirroring the DDL block; `ThriftHmsClient` overrides each via `execute(client -> …)`) + +**Read-side ACID primitives** (signatures FIXED by the vendored `IMetaStoreClient` — no back-derivation): +```java +long openTxn(String user); +void commitTxn(long txnId); +Map getValidWriteIds(String fullTableName, long currentTransactionId); +void acquireSharedLock(String queryId, long txnId, String user, + String dbName, String tableName, List partitionNames, long timeoutMs); +``` +- `getValidWriteIds`: `client.getValidTxns()` then `client.getValidWriteIds(singletonList(fullTableName), validTxns.toString())`; assemble the two keys `hive.txn.valid.txns` / `hive.txn.valid.writeids` (define as **local constants**, no fe-core `AcidUtil` import) with the same graceful max-watermark `ValidReadTxnList`/`ValidReaderWriteIdList` fallback as fe-core `ThriftHMSCachedClient`. +- `acquireSharedLock`: build a `SHARED_READ` `LockRequest` (one `LockComponent` per table + per partition) + `checkLock` poll loop up to `timeoutMs`. +- Neutralize fe-core `TableNameInfo` → `(dbName, tableName)`. + +**Write-side primitives** (DTOs back-derived from HEAD `HMSTransaction` — a 1:1 migration target, not a guess): +```java +void addPartitions(String dbName, String tableName, List partitions); // batch 20 inside impl +void updateTableStatistics(String dbName, String tableName, + Function update); +void updatePartitionStatistics(String dbName, String tableName, String partitionName, + Function update); +boolean dropPartition(String dbName, String tableName, List partitionValues, boolean deleteData); +boolean partitionExists(String dbName, String tableName, List partitionValues); // not-found-tolerant probe → NEW→APPEND +``` +New plugin DTOs (`connector-api` + JDK only): `HmsPartitionWithStatistics`, `HmsPartitionStatistics` (fields from HEAD `HivePartitionWithStatistics`/`HivePartitionStatistics`/`CommonStatistics`). Extend `HmsWriteConverter` with `toHivePartitions(...)` / `toColumnStatistics(...)` (pure, unit-tested like the existing converter). + +### 4.2 `HiveWriteContext` (immutable, package-private, peer of `IcebergWriteContext`) +```java +final WriteOperation writeOperation; +final boolean overwrite; +final Map staticPartitionValues; // defensive copy +final String queryId; // replaces ConnectContext (D4) +final TFileType fileType; // FILE_S3 vs staging (from bindDataSink logic) +final String writePath; // staging temp path +// getters + isStaticPartitionOverwrite() = overwrite && !staticPartitionValues.isEmpty() +// DROP iceberg's branchName / readSnapshotId +``` + +### 4.3 `HiveConnectorMetadata.beginTransaction` (one-liner, D1) +```java +@Override public ConnectorTransaction beginTransaction(ConnectorSession session) { + return new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient /* or hive metastore seam */, context); +} +``` +Also override `ConnectorWriteOps.beginTransaction` entry if separate (recon: `beginTransaction` seam is on `ConnectorWriteOps`, reached from executor `:81`). + +### 4.4 `HiveWritePlanProvider implements ConnectorWritePlanProvider` +```java +@Override public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + HiveTableHandle t = (HiveTableHandle) handle.getTableHandle(); + HiveConnectorTransaction tx = currentTransaction(session); // session.getCurrentTransaction() or throw + HiveWriteContext ctx = buildWriteContext(handle); // INSERT→OVERWRITE promotion via isOverwrite() + tx.beginWrite(session, t.getDbName(), t.getTableName(), ctx); + THiveTableSink sink = buildSink(t, handle, ctx); // = ported bindDataSink + return new ConnectorSinkPlan(new TDataSink(TDataSinkType.HIVE_TABLE_SINK).setHiveTableSink(sink)); +} +@Override public EnumSet supportedOperations() { return EnumSet.of(INSERT, OVERWRITE); } +// capability gates (requiresParallelWrite / requiresFullSchemaWriteOrder / requiresPartitionLocalSort / +// requiresMaterializeStaticPartitionValues) set to match hive's file/partition write model. +``` +`buildSink` = byte-faithful port of `planner.HiveTableSink.bindDataSink` (§5.2). Register via `HiveConnector.getWritePlanProvider()`. + +### 4.5 fe-core bridge — **ZERO changes** (verified) +`PluginDrivenInsertExecutor` + `PluginDrivenTransactionManager` + `BaseExternalTableInsertExecutor` + `ConnectorSinkPlan` + the SPI interfaces are 100% source-agnostic. `profileLabel="HMS"` maps to the existing `TransactionType.HMS`. No fe-core edit in this batch. + +--- + +## 5. Porting detail (the hard parts) + +### 5.1 SPI reshape — where classification runs +Legacy `HMSTransaction` has `beginInsertTable(ctx)` + `finishInsertTable(NameMapping)` + `commit()`; the SPI has none of these. Mirror iceberg: add a **non-SPI `beginWrite(session, db, table, HiveWriteContext)`** (called by `planWrite`) that loads the table under `context.executeAuthenticated` and captures op-context; run the **classification inside `commit()`** (or at first `beginWrite`) before `HmsCommitter`. `NameMapping`/target identity + `isOverwrite`/`fileType`/`writePath`/`queryId` arrive via `HiveWriteContext`, **not** `ConnectContext`. + +### 5.2 `buildSink` (largest surface) — port of `bindDataSink` +Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE); existing-partition `THivePartition` list; bucket info; format + compression; **location** (FILE_S3 in-place vs staging temp path created here and carried on the transaction, **not** written back to a ctx); serde props; `hadoopConfig` from `context.getStorageProperties().toBackendProperties().toMap()` + `context.vendStorageCredentials` overlay; `setOverwrite(handle.isOverwrite())` + `setStaticPartitionValues`. Location/creds via the same `context` seams iceberg uses (`getBackendFileType` / `normalizeStorageUri` / `getBrokerAddresses` for `FILE_BROKER`). + +### 5.3 The three hard couplings — break plan +1. **`ConnectContext` profile/queryId** → **D4**: drop profiling; `queryId` via `HiveWriteContext`. Result: zero `org.apache.doris.qe`/`common.profile` imports in the plugin. +2. **thrift `THivePartitionUpdate` cluster** → **D11**: non-issue, thrift module already compile-visible; travels as-is. `addCommitData` deserializes identically to iceberg's `TIcebergCommitData` path. +3. **`SpiSwitchingFileSystem` MPU downcast** → **D6** (the genuinely hard one): plugin-side FileSystem provider from `context.getStorageProperties()`; plugin-owned async-rename `Executor` (shut in `close()`); per-task `context.executeAuthenticated` (D5). Preserve the `isMockedPartitionUpdate` guard and the `s3:///` URI rebuild. + +### 5.4 Read-side ACID — what ports +- **Pure algorithm** into `fe-connector-hive` on directory **names** + `hive-common Valid*` only: `parseBase` / `parseDelta` / `isValidBase` / `isValidMetaDataFile` / `FullAcidFileFilter` (bucket_ prefix & not `_flush_length`) / `InsertOnlyFileFilter` (accept-all) / the best-base + working-delta **selection loop**. +- **`HiveScanPlanProvider`**: replace the wholesale ACID-dir skip (`if (status.isDirectory()) continue;`) with the descent for transactional partitions; emit `HiveScanRange…acidInfo(partitionLocation, deltaDescriptors)` per surviving base/delta bucket file. +- **plugin `HiveTransaction`** (read-side): `openTxn` at scan init, `acquireSharedLock` + `getValidWriteIds`, `commitTxn` at finish, using the §4.1 primitives. Bindings unwound: `HMSExternalTable`/`HMSExternalCatalog`/`HMSCachedClient`/`TableNameInfo` → `(db, table)` strings + `HmsClient`. +- **plugin read-txn manager** keyed by `queryId`, registered via the already-landed `QueryFinishCallbackRegistry` seam (do the live Env removal at cutover, not here — keep the manager dormant/plugin-owned now). +- **`HiveTableHandle`**: derive `isTransactional` / `isFullAcid` from `getTableParameters()` (D8). + +--- + +## 6. Build order (ordered TODO — compilable increments → ONE atomic commit) + +> Each increment leaves the working tree **compiling + unit-tests green**, but is **NOT git-committed**. Accumulate all, then one atomic feature commit at the end (§8). Verify per increment: build the touched module(s) (`-am`), `checkstyle:check` (no `-am`), `tools/check-connector-imports.sh` clean, run the new tests. + +- [ ] **INC-1 — HmsClient primitive surface + DTOs + converters + recording-fake harness** (`fe-connector-hms`) + - [ ] `HmsClient`: add read-ACID (4) + write (5) `default`-throw methods (§4.1), mirroring the DDL block. + - [ ] DTOs `HmsPartitionWithStatistics`, `HmsPartitionStatistics` (connector-api/JDK only). + - [ ] `ThriftHmsClient`: override each via `execute(client -> …)` (openTxn/commitTxn one-liners; getValidWriteIds two-key + fallback; acquireSharedLock SHARED_READ + checkLock poll; addPartitions batch-20 + converter; stats via `updateTable/PartitionColumnStatistics`; dropPartition/partitionExists not-found-tolerant). + - [ ] `HmsWriteConverter`: `toHivePartitions` / `toColumnStatistics`. + - [ ] **Tests**: `ThriftHmsClientAcidTest` (recording-fake `IMetaStoreClient` via package-private ctor + `MetaStoreClientProvider`, `poolSize<=0`) asserts each forward; `HmsWriteConverterTest` additions. +- [ ] **INC-2 — hive support leaf types + pure helpers into `fe-connector-hive`** + - [ ] Pure path helpers (`isSubDirectory`/`getImmediateChildPath`/`sameFileSystem`/`normalizePath`/`pathsEqual`) + `mergePartitions` (pure `THivePartitionUpdate` merge). + - [ ] Plugin equivalents / co-move for `NameMapping`, `CommonStatistics`, `HivePartition`/`HivePartitionStatistics`/`HivePartitionWithStatistics` **as needed by the transaction** (prefer plugin DTOs from INC-1; co-move only what has no plugin peer). **Verify what already exists plugin-side before moving.** + - [ ] **Tests**: pure helper + mergePartitions unit tests. +- [ ] **INC-3 — `HiveWriteContext` + `HiveConnectorTransaction` core + `beginTransaction`** (`fe-connector-hive`) — the genuinely-atomic unit + - [ ] `HiveWriteContext` (§4.2). + - [ ] `HiveConnectorTransaction`: classification state machine (APPEND/NEW/OVERWRITE + probe downgrade) + `HmsCommitter` (prepare*/doCommit/abort/rollback + staging rename + MPU) + SPI methods. Break couplings #1/#3 (D4/D5/D6). full-ACID write reject (D7). rollback deletes staging + aborts MPU (D9). + - [ ] `HiveConnectorMetadata.beginTransaction` one-liner (+ `ConnectorWriteOps.beginTransaction`). + - [ ] **Tests**: `HiveConnectorTransactionTest` (recording-fake `HmsClient` + fake `FileSystem`): classification cases, commit dispatch (add_partitions + staging rename + MPU complete), rollback (staging delete + MPU abort), addCommitData+getUpdateCnt, full-ACID-write reject. +- [ ] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive`) + - [ ] `HiveWritePlanProvider` (§4.4) + `buildSink` (§5.2) + `HiveConnector.getWritePlanProvider()` + `supportedOperations` + capability gates. Re-home LZO-INSERT reject + transactional-table reject as begin-guards. + - [ ] **Tests**: `HiveWritePlanProviderTest` against a fake `ConnectorWriteHandle` — asserts emitted `THiveTableSink` (PARTITION_KEY/REGULAR order, bucket, compression-per-format, staging-vs-in-place location, serde), INSERT→OVERWRITE promotion, begin-guard rejects. +- [ ] **INC-5 — read-side ACID producer + plugin read-txn lifecycle (dormant)** (`fe-connector-hive`) + - [ ] Port `getAcidState` **pure** algorithm (§5.4). Plugin `HiveTransaction` + read-txn manager. `HiveScanPlanProvider` descent + `acidInfo(...)` producer wiring. `HiveTableHandle` `isTransactional`/`isFullAcid` from `tableParameters` (D8). + - [ ] **Tests**: `HiveAcidDescentTest` (pure name-parse + write-id filter → surviving files + delete-delta descriptors) + confirm existing `HiveScanRangeAcidTest` still green. +- [ ] **FINAL — single atomic feature commit** of INC-1..INC-5 (§8) + separate HANDOFF/doc commit. + +**Dependency graph**: INC-1 → {INC-3 (write primitives), INC-5 (read primitives)}; INC-2 → INC-3; INC-3 → INC-4; INC-5 independent of INC-3/4 (only needs INC-1). Recommended order: INC-1 → INC-2 → INC-3 → INC-4 → INC-5 (write spine first, read last), or INC-1 → INC-5 (read) → INC-2/3/4 (write) if preferring to bank the low-risk read side first. + +--- + +## 7. Test plan (unit, no Mockito on connector side) + +- **INC-1**: `ThriftHmsClientAcidTest` (recording-fake `IMetaStoreClient`), `HmsWriteConverterTest`. +- **INC-3**: `HiveConnectorTransactionTest` (recording-fake `HmsClient` + fake `FileSystem`) — classification / commit / rollback / addCommitData / full-acid-write reject. +- **INC-4**: `HiveWritePlanProviderTest` (fake `ConnectorWriteHandle`) — sink byte-shape + promotion + guards. +- **INC-5**: `HiveAcidDescentTest` (pure) + `HiveScanRangeAcidTest` (regression, already green). +- **Integration (the R-002 hard gate, deferred to cutover)**: INSERT/OVERWRITE/partition write/delete-delta read/rollback/multi-FE invalidation. Needs a **live** write path → runs at the P7.4/P7.5 cutover (local/scoped SPI_READY flip). **Do not silently skip** (Rule 12) — tracked as the cutover gate. + +--- + +## 8. Atomic-commit management (how the multi-session port stays safe) + +- Each increment compiles + tests green **before moving on**, but is **not committed**. The uncommitted working tree is the WIP carrier across sessions. +- **This design doc + HANDOFF are the durable cross-session artifacts** (the feature code is uncommitted until the end). At every session boundary: (a) leave the tree compiling, (b) update HANDOFF with exactly which increments are done / in-progress / remaining + the compile state, (c) commit only doc/HANDOFF. +- **Final commit** (one, path-whitelisted `git add` — never `-A`): the full INC-1..INC-5 diff. Message per repo convention + `Co-Authored-By` / `Claude-Session`. This is P7.3's plugin-side deliverable; the cutover is a later separate commit/PR. +- If a session ends mid-increment (tree not compiling): HANDOFF must say so loudly (Rule 12) and name the exact next edit to restore compilation. + +--- + +## 9. Risks / open items carried into implementation + +- **R1 TCCL/doAs on plugin-owned pools** (D5): stats pool + async-rename pool threads don't inherit the commit-thread pin. Metastore calls self-pin inside `ThriftHmsClient.doAs`; **fs/rename/MPU tasks must each be wrapped in `context.executeAuthenticated`**. Do not copy iceberg's JVM-wide primer. (memory `catalog-spi-plugin-tccl-classloader-gotcha`.) +- **R2 fe-core leakage**: run `tools/check-connector-imports.sh` every increment. `HiveVersionUtil` hit is a known false positive — do not touch (memory `catalog-spi-hms-hiveversionutil-gate-false-positive`). +- **R3 MPU is NEW plugin capability**: scan path uses only raw Hadoop FS today; object-store complete/abort needs an `fe-filesystem-spi ObjFileSystem` from `context.getStorageProperties()`. Largest new-code risk in INC-3. +- **R4 DTO shape drift** (write primitives back-derived): converter unit tests must exercise exactly the fields the transaction passes, so a missed field surfaces as a test gap now, not a signature break in INC-3. +- **R5 `getValidWriteIds` correctness**: replicate fe-core's subtlety (recent-snapshot `getValidTxns()` vs `currentTransactionId` into `TxnUtils.createValidTxnWriteIdList`) + keep the max-watermark fallback (version-incompatible HMS degrades instead of failing the scan). +- **R6 co-move blast radius (INC-2)**: verify how much of `datasource.hive` genuinely must move vs. already has a plugin peer, before moving — over-moving risks dragging fe-core deps. **Read plugin-side first.** +- **Alternatives considered**: (a) slice into 5 separate commits — rejected by user (atomic chosen); the increments above are the internal build order of that one commit. (b) fold `planWrite` into metadata — rejected (D1, iceberg puts it in a separate provider). + +--- + +## 10. Rollout + +Dormant now (hive not in `SPI_READY_TYPES`). Cutover = later phase (P7.4/P7.5): fe-core write-chain retype + catalog identity flip + `HiveTransactionMgr` Env removal + read-path rewire + add `HIVE` to `SPI_READY_TYPES` + run the ACID integration gate + delete legacy `datasource/hive`(+`hudi`+23 HMS-iceberg classes, respecting the cross-connector delete ordering). From 094094902d13dd6917faae9dccdeaf58465f5c64 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 13:06:26 +0800 Subject: [PATCH 018/334] [doc](catalog) P7.3 hive: record INC-1 done (HmsClient write/ACID primitives, uncommitted in tree) + refresh HANDOFF MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Atomic-batch increment INC-1 is implemented and green in the working tree but intentionally NOT committed (the batch lands as one feature commit once INC-1..INC-5 are all green; the uncommitted tree is the cross-session WIP carrier per the design doc §8). INC-1 (fe-connector-hms): 9 HmsClient metastore primitives (5 write: addPartitions / updateTable+PartitionStatistics / dropPartition / partitionExists; 4 read-ACID: openTxn / commitTxn / getValidWriteIds / acquireSharedLock) + 4 DTOs (HmsCommonStatistics, HmsPartitionStatistics, HmsPartitionWithStatistics, HmsAcidConstants) + HmsWriteConverter partition/ stats helpers + a recording-fake IMetaStoreClient unit harness. Module test SUCCESS (49 hms cases), checkstyle 0, connector import-gate clean; verified by a 3-lens adversarial review (0 production defects; 2 test-coverage gaps fixed). This commit updates only the design-doc INC-1 checklist (marked done, with the four faithful-migration deviations) and the rolling HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 25 +++++++++++-------- ....3-hive-write-txn-implementation-design.md | 13 +++++----- 2 files changed, 22 insertions(+), 16 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 7e281114bdbc22..7cec01588ea743 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,20 +5,23 @@ --- -# 🎯 当前状态（2026-07-06 晚）= **P7.3 写/事务批已完成 HEAD 级侦察 + 可执行实现设计（新文档 [`P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md)）；用户已拍板「整批作为一次原子提交」，实现拆成 5 个可编译增量 INC-1..INC-5（内部构建顺序、逐增量编译+单测过、但不单独 commit，最后一次性提交）；下一步 = 起手实现 INC-1（HmsClient 读/写原语 + DTO + 录制式假客户端单测）。前序 seam/回归已落地（T08 `21aa30683dc`、T07 delete-delta `c30fa15d99a`），勿重做** +# 🎯 当前状态（2026-07-07）= **P7.3 原子批 INC-1 已实现并通过（HmsClient 9 原语 + 4 DTO + 转换器 + 录制式假客户端单测）——但按用户拍板的「整批一次原子提交」，INC-1 代码仍 UNCOMMITTED 在工作树里（是跨 session 的 WIP 载体，勿 commit、勿 revert、勿 `git checkout`）；本轮只 commit 了本 HANDOFF + 设计文档。验证：`fe-connector-hms` 模块测试 SUCCESS（49 hms 用例：16 新 ACID/写 + 15 转换器 + 18 既存类型映射）、checkstyle 0、import-gate 净。下一步 = INC-2（把 hive 支撑叶子类型 + 纯 helper 挪进 `fe-connector-hive`；先读插件侧已有什么再挪，R6）。前序 seam/回归已落地（T08 `21aa30683dc`、T07 delete-delta `c30fa15d99a`），勿重做** + +> **⚠️ 工作树有未提交的 INC-1 代码**：`git status` 会显示 `fe/fe-connector/fe-connector-hms/` 下 4 新文件 + 4 改文件（`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` + 2 test）。这是**故意的**（原子批模型，见设计文档 §8）——下个 session 在此基础上继续 INC-2..INC-5，全绿后才一次性 feature commit。**若工作树被清就得照设计文档 §6 INC-1 块重建**（该块已勾选 [x] 并记录了精确签名与 4 处偏差）。 > **⚠️ 权威实现计划已切换到设计文档**：本轮把 spec 的 T01–T11 拆解**收敛成一份可直接执行的设计**（4 类拆分修正、精确签名、3 硬耦合断法、INC-1..INC-5 构建顺序、逐增量测试计划、原子提交管理策略）。**下个 session 直接照 `tasks/P7.3-hive-write-txn-implementation-design.md` 的 §6 构建顺序执行**，spec 的 T01–T11 表仅作背景。 > > **本轮侦察的关键修正（写进设计文档，勿再纠结）**：① **写路径是「4 类拆分」**——`HiveConnectorTransaction` + 不可变 `HiveWriteContext` + **独立** `HiveWritePlanProvider`（planWrite **不在** metadata，spec 说法有误）+ `HiveConnectorMetadata.beginTransaction` 一行工厂，全对齐 iceberg。② **fe-core 桥零改动**已复核（`profileLabel()="HMS"` 复用既存 `TransactionType.HMS` 枚举）。③ **读侧 ACID（T07 生产半）并不与写事务机器耦合**——`HMSTransaction` 全程不调 `openTxn/commitTxn/getValidWriteIds`，四个读原语仅读侧用；所谓「拖入 fe-core」是浅的（插件已用裸 Hadoop FS + 自有 `HiveScanRange`，`FileCacheValue/FileSystemTransferUtil/LocationPath/StorageProperties/HivePartition/AcidInfo` 在边界处全落地），真正搬的只有纯目录名解析 + `hive-common ValidWriteIdList` 算法。④ hive **不在** `SPI_READY_TYPES`（`{jdbc,es,trino-connector,max_compute,paimon,iceberg}`）——整批**天然 dormant/安全**，编译+单测但零线上路由，同 iceberg/paimon 翻闸前范式。⑤ **翻闸/fe-core retype/Env 摘 `HiveTransactionMgr`/删 legacy 均不在本原子批**（属后续 P7.4/P7.5，另起一次原子提交）。 -> **本轮做了什么** —— **1 实现 commit（`21aa30683dc`）+ 1 文档 commit**，落地 T08 的**中立读事务生命周期 seam**（OQ-RTX=a 的连接器无关部分）： -> - **新 `QueryFinishCallbackRegistry`**（fe-core `qe/`）：`register(queryId, Runnable)` / `runAndClear(queryId)`（幂等 + 单回调异常隔离，一个连接器清理失败不阻断其它）。 -> - **`QeProcessor` 加 `registerQueryFinishCallback` SPI**；`QeProcessorImpl.unregisterQuery` 原 `:210` 硬编 `Env.getCurrentHiveTransactionMgr().deregister(...)`（**每条查询都点名 hive** 的通用清理代码）改为通用 drain `runAndClear(printId(queryId))` → **通用查询生命周期代码不再点名任何数据源**（架构铁律进展；对齐 Trino 引擎驱动的 query/txn 生命周期）。 -> - **`HiveScanNode.doInitialize`** 经 seam 注册 deregister 回调。**行为保持**：drain key `printId(queryId)` == 回调 key `hiveTransaction.getQueryId()`（老 deregister 依赖的同一等式）；早释放路径仍即时 commit、finish 回调再走幂等 deregister 即 no-op → 单次 commit。 -> - **6 单测（无 Mockito）**：run-once / 空查询 no-op / 多回调按序 / 幂等 / 失败隔离 / per-query 作用域。**build SUCCESS / checkstyle 0 / import-gate 净 / 6 pass**。非 live（翻闸 P7.5）。 -> - **T08 剩余**：`HiveTransactionMgr` **仍留 `Env`**（legacy fe-core 读路径 `HiveScanNode` 在用）；入插件须待 T07 读侧 ACID 迁移（届时插件经**同一 seam** 注册），故与 T07 同批。 -> - **为何本轮只做 T08（用户拍板）**：T01–T05 是**互锁原子批**（写原语签名由事务实际调用反推，不得先作死代码——P7.1 教训），照 spec 移植 1895 行 `HMSTransaction`+`HiveTableSink.bindDataSink` 是**多 session** 且中途拿不出可编译可提交结果。T08 中立 seam 是本 session 唯一「干净、有界、可独立编译+提交+测试」的片。 -> - **⚠️ 本轮侦察修正 T07 边界（已写进 spec T07 行，勿低估）**：读侧 ACID 的**消费/解码半已在插件做完+单测**（`HiveScanRange`，`.acidInfo(` 目前无生产者调用）；**生产半仍是活且与写批耦合**——`AcidUtil.getAcidState`(427) 返回 fe-core 缓存内类 `FileCacheValue`、拖入 Doris `filesystem`/`StorageProperties`/`LocationPath`/`HivePartition`；生产实调在 `HiveExternalMetaCache:816`（非 `HiveScanNode`）；`HiveTransaction` 的 `openTxn`/`getValidWriteIds`/`acquireSharedLock` **与写路径共用同批 ACID HmsClient 原语（T01/T02）**；插件 `HiveScanPlanProvider` 现**整段跳过 ACID 目录**，产 ACID 读须新增扫描下潜逻辑。⇒ **T07 生产半宜与 T01/T02 写批同批推进，非独立并行小片**。 +> **本轮做了什么（INC-1，未 commit）** —— 照设计文档 §6 起手实现原子批第一增量：**在 `fe-connector-hms` 建齐 9 个 HmsClient 元存储原语 + 4 个 DTO + 转换器扩展 + 录制式假客户端单测**。 +> - **`HmsClient` 接口**：Phase-3 补 5 写原语（addPartitions / updateTable+PartitionStatistics / dropPartition / partitionExists），新 Phase-4 补 4 读-ACID 原语（openTxn / commitTxn / getValidWriteIds / acquireSharedLock），均照既有 DDL 块 `default`-throw 写法。 +> - **4 新 DTO**：`HmsCommonStatistics`（port `CommonStatistics`）、`HmsPartitionStatistics`（port `HivePartitionStatistics`，**丢弃 column-stats map**——写路径恒空且是唯一 fe-core 拖拽）、`HmsPartitionWithStatistics`（builder；**打平** `HivePartition` 的 SD 字段，绕开 fe-core 类型）、`HmsAcidConstants`（两个 BE 契约 key，INC-5 复用）。均 connector-api/JDK/hive-metastore-api only。 +> - **`ThriftHmsClient` 实现**：每个原语 `execute(client -> …)`（auth/TCCL 由 `execute→doAs` 免费提供）。getValidWriteIds 成功路径（`getValidTxns()` **无参**快照 + 两 key）+ 版本不兼容时 max-watermark 降级 fallback；acquireSharedLock 用 `LockRequestBuilder`/`LockComponentBuilder`（SHARED_READ + `checkLock` 忙轮询，无 heartbeat=OQ-LOCK）；**addPartitions batch-20 移进 client**（JDK subList，不引 guava）；统计走 **`alter_table`/`alter_partition` 参数重建**（非 `SetPartitionsStatsRequest`）；partitionExists **新写**=getPartition 在 lambda 内吞 `NoSuchObjectException`（not-found 不 taint 池化 client）。 +> - **`HmsWriteConverter` 扩展**：`toHivePartitions` + `toStatisticsParameters`（=`HiveUtil.updateStatisticsParameters`，spec 里叫 `toColumnStatistics` **系误名**，其实是基础统计参数 numRows/totalSize/numFiles，非 ColumnStatistics）+ `toPartitionStatistics` + 私有 `emptyToNull`（去 guava）。 +> - **单测（无 Mockito，JDK `Proxy` 录制式假 `IMetaStoreClient`）**：`ThriftHmsClientWriteAcidTest` 16 例（转发/batch-20 **全覆盖**校验 union==p0..p44 而非仅调用次数/getValidWriteIds **成功**（辨别水位，证明确实走成功支非 fallback）+fallback/锁 acquire·poll·timeout/统计重建/partitionExists 真假/歧义名拒绝）；`HmsWriteConverterTest` +5。**pom 加 2 个 test-scoped 依赖**（`hadoop-mapreduce-client-core`→`JobConf`、`commons-lang`→`HiveConf` 静态初始化，因 ctor 会 new HiveConf；照 iceberg/paimon 先例）。 +> - **验证**：module test SUCCESS（49 hms 用例）、checkstyle 0、import-gate 净。**3-lens 对抗 review + 逐条 verify**：0 生产缺陷、2 测试覆盖缺口（batch 只校验次数、getValidWriteIds 成功路径无覆盖）——均已补测修掉。 +> - **4 处对 §4.1 的偏差（均忠实迁移、零行为变化，已写进设计文档 §6 INC-1 块）**：batch-20 在 client 非 committer；`toColumnStatistics`→`toStatisticsParameters`/`toPartitionStatistics`；column-stats map 丢弃；partitionExists 新写。 +> - **前序已落地（历史 commit，勿重做）**：T08 连接器无关读事务 seam（`QueryFinishCallbackRegistry` + `QeProcessorImpl` 通用 drain 替 hive 硬编 + `HiveScanNode` 经 seam 注册，`21aa30683dc`）；T07 delete-delta 回归修复（`c30fa15d99a`）。T08 剩余（`HiveTransactionMgr` 出 `Env` 入插件）与 T07 读侧 ACID（INC-5）同批。 > > **上一轮（P7.3 recon+设计）已定的产出仍有效**（写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块 + OQ 段 ✅）： > - **通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，非从零发明 seam。 @@ -47,7 +50,9 @@ --- -# 🚀 下个 session 任务 = **照 [`tasks/P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md) §6 构建顺序起手 INC-1（HmsClient 原语+DTO+录制式假客户端单测）→ INC-2..INC-5；全批作为一次原子提交（用户拍板），逐增量保持树可编译+单测过、但不单独 commit** +# 🚀 下个 session 任务 = **照 [`tasks/P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md) §6 续做 INC-2（hive 支撑叶子类型 + 纯 helper 挪进 `fe-connector-hive`）→ INC-3（`HiveWriteContext`+`HiveConnectorTransaction` 核心，最硬的一块）→ INC-4（`HiveWritePlanProvider`+`buildSink`）→ INC-5（读侧 ACID 生产半，可用 INC-1 的 `HmsAcidConstants`/read 原语）。INC-1 已在工作树（未 commit）；全批全绿后一次性 feature commit** + +> **INC-2 起步先做（R6，设计文档 §6 明确警告）**：**先读 `fe-connector-hive` 插件侧已有什么**，再决定 `datasource.hive` 里哪些叶子类型真需要挪 vs 已有插件对等物——过度搬运会拖入 fe-core 依赖。纯 helper（path 工具 + `mergePartitions`）与 DTO 优先复用 INC-1 已建的 `Hms*` 类型；只 co-move 无插件对等物的。INC-2/INC-3 的写原语 DTO 应直接用 INC-1 的 `HmsPartitionWithStatistics`/`HmsPartitionStatistics`/`HmsCommonStatistics`（committer 侧 batch-20 已在 client 内，故 committer 只需一次 `hmsClient.addPartitions(db,table,全部)`）。 > **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index e98879bacd426d..cf4507accd0f06 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -184,12 +184,13 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE > Each increment leaves the working tree **compiling + unit-tests green**, but is **NOT git-committed**. Accumulate all, then one atomic feature commit at the end (§8). Verify per increment: build the touched module(s) (`-am`), `checkstyle:check` (no `-am`), `tools/check-connector-imports.sh` clean, run the new tests. -- [ ] **INC-1 — HmsClient primitive surface + DTOs + converters + recording-fake harness** (`fe-connector-hms`) - - [ ] `HmsClient`: add read-ACID (4) + write (5) `default`-throw methods (§4.1), mirroring the DDL block. - - [ ] DTOs `HmsPartitionWithStatistics`, `HmsPartitionStatistics` (connector-api/JDK only). - - [ ] `ThriftHmsClient`: override each via `execute(client -> …)` (openTxn/commitTxn one-liners; getValidWriteIds two-key + fallback; acquireSharedLock SHARED_READ + checkLock poll; addPartitions batch-20 + converter; stats via `updateTable/PartitionColumnStatistics`; dropPartition/partitionExists not-found-tolerant). - - [ ] `HmsWriteConverter`: `toHivePartitions` / `toColumnStatistics`. - - [ ] **Tests**: `ThriftHmsClientAcidTest` (recording-fake `IMetaStoreClient` via package-private ctor + `MetaStoreClientProvider`, `poolSize<=0`) asserts each forward; `HmsWriteConverterTest` additions. +- [x] **INC-1 — HmsClient primitive surface + DTOs + converters + recording-fake harness** (`fe-connector-hms`) — **DONE (uncommitted, in working tree; module test SUCCESS 49 hms tests, checkstyle 0, import-gate clean)** + - [x] `HmsClient`: added write (5) `default`-throw methods under Phase-3 header + read-ACID (4) under a new Phase-4 header, mirroring the DDL block. `+import java.util.function.Function`. + - [x] DTOs: `HmsCommonStatistics` (port of `CommonStatistics`), `HmsPartitionStatistics` (port of `HivePartitionStatistics` — **column-stats map DROPPED**, dead on write path + it was the only fe-core drag), `HmsPartitionWithStatistics` (builder; **flattened** `HivePartition` SD fields: partitionValues/location/columns:`FieldSchema`/inputFormat/outputFormat/serde/parameters — avoids fe-core `HivePartition`), `HmsAcidConstants` (the two BE-contract keys, reused by INC-5). + - [x] `ThriftHmsClient`: overrode each via `execute(client -> …)`. openTxn/commitTxn one-liners; getValidWriteIds = success (getValidTxns **no-arg** snapshot + two keys) with the max-watermark `ValidReadTxnList`/`ValidReaderWriteIdList` fallback in the method body (execute wraps the size!=1 throw); acquireSharedLock via `LockRequestBuilder`/`LockComponentBuilder` `setShared()`+`SELECT`+`isTransactional` + `checkLock` poll (busy-poll, no heartbeat — OQ-LOCK); **addPartitions batch-20 moved INTO the client** (JDK subList, no guava) — legacy batched in the committer; stats via **`alter_table`/`alter_partition` param rebuild** (NOT `SetPartitionsStatsRequest`/`ColumnStatistics` — legacy uses params); dropPartition returns the metastore boolean; partitionExists **written fresh** = getPartition catching `NoSuchObjectException` **inside** the execute-lambda (so the pooled client is not tainted on not-found). + - [x] `HmsWriteConverter`: added `toHivePartitions` + `toStatisticsParameters` (**renamed** from spec's `toColumnStatistics` — it is `HiveUtil.updateStatisticsParameters`: numRows/totalSize/numFiles + CDH flag) + `toPartitionStatistics` (`HiveUtil.toHivePartitionStatistics`) + private `emptyToNull` (drops guava `Strings.emptyToNull`). + - [x] **Tests**: `ThriftHmsClientWriteAcidTest` (recording-fake `IMetaStoreClient` via JDK `Proxy` + package-private ctor + `MetaStoreClientProvider` + `poolSize=0`; 16 tests) — forwards, batching **coverage** (union==p0..p44, not just call count), getValidWriteIds **success** (distinctive watermark, provably not fallback) + fallback, lock acquire/poll/timeout, stats rebuild, partitionExists true/false, ambiguous-name reject. `HmsWriteConverterTest` +5. **pom.xml: +2 test-scoped deps** (`hadoop-mapreduce-client-core` for `JobConf`, `commons-lang` for `HiveConf` static init — mirrors iceberg/paimon; `HiveConf` is built in the `ThriftHmsClient` ctor, so instantiating it in a unit test needs them). + - **Deviations vs §4.1 (all faithful-migration, no behavior change): batch-20 in client not committer; `toColumnStatistics`→`toStatisticsParameters`/`toPartitionStatistics` (param math, not `ColumnStatistics`); column-stats map dropped; `partitionExists` written fresh. Adversarial review (3 lenses × verify): 0 production defects, 2 test-coverage gaps (both fixed above).** - [ ] **INC-2 — hive support leaf types + pure helpers into `fe-connector-hive`** - [ ] Pure path helpers (`isSubDirectory`/`getImmediateChildPath`/`sameFileSystem`/`normalizePath`/`pathsEqual`) + `mergePartitions` (pure `THivePartitionUpdate` merge). - [ ] Plugin equivalents / co-move for `NameMapping`, `CommonStatistics`, `HivePartition`/`HivePartitionStatistics`/`HivePartitionWithStatistics` **as needed by the transaction** (prefer plugin DTOs from INC-1; co-move only what has no plugin peer). **Verify what already exists plugin-side before moving.** From a0739a3bac74270178b70a84cd474130b913b384 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 13:47:56 +0800 Subject: [PATCH 019/334] [doc](catalog) P7.3 hive: record INC-2 done (write-path pure helpers + NameMapping in fe-connector-hive, uncommitted) + refresh HANDOFF INC-2 lands, inside fe-connector-hive (uncommitted in the working tree per the atomic-batch model), the minimal write-path leaf material the connector transaction needs: - HiveWriteUtils (new, package-private): byte-faithful port of the pure path helpers (isSubDirectory/getImmediateChildPath/pathsEqual + private sameFileSystem/normalizeUriPart/normalizePath) and mergePartitions from HEAD HMSTransaction. fe-core-free (Hadoop Path + java.net.URI + shared thrift only; no guava/lombok). mergePartitions made static. - NameMapping (new, co-move): JDK-only plugin copy of fe-core datasource.NameMapping (no lombok/guava), the action-map key of the connector transaction. Only leaf type co-moved -- it has no plugin peer. R6 resolution after reading plugin-side first: the statistics trio already has INC-1 plugin peers (HmsCommonStatistics/HmsPartitionStatistics/ HmsPartitionWithStatistics, with full merge/reduce ops), so they are NOT co-moved; HivePartition is deferred to INC-3 (its fe-core Column drag is dead on the write path, and HmsPartitionWithStatistics supersedes its SD role, so the PartitionAndMore descriptor shape is an INC-3 body-port decision). Tests: HiveWriteUtilsTest (12) + NameMappingTest (4), each pinning the contract's intent. fe-connector-hive build SUCCESS, 16 new tests green, checkstyle 0, import-gate clean. Zero changes to existing files. Only this doc/HANDOFF commit is made; INC-1+INC-2 code stays uncommitted (single atomic feature commit at end of batch). Next: INC-3 (HiveWriteContext + HiveConnectorTransaction core + beginTransaction). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01BxdCRs2TtGXbwxwiL6Jggz --- plan-doc/HANDOFF.md | 29 ++++++++++--------- ....3-hive-write-txn-implementation-design.md | 11 +++---- 2 files changed, 22 insertions(+), 18 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 7cec01588ea743..6e60ebd9f736e2 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,23 +5,22 @@ --- -# 🎯 当前状态（2026-07-07）= **P7.3 原子批 INC-1 已实现并通过（HmsClient 9 原语 + 4 DTO + 转换器 + 录制式假客户端单测）——但按用户拍板的「整批一次原子提交」，INC-1 代码仍 UNCOMMITTED 在工作树里（是跨 session 的 WIP 载体，勿 commit、勿 revert、勿 `git checkout`）；本轮只 commit 了本 HANDOFF + 设计文档。验证：`fe-connector-hms` 模块测试 SUCCESS（49 hms 用例：16 新 ACID/写 + 15 转换器 + 18 既存类型映射）、checkstyle 0、import-gate 净。下一步 = INC-2（把 hive 支撑叶子类型 + 纯 helper 挪进 `fe-connector-hive`；先读插件侧已有什么再挪，R6）。前序 seam/回归已落地（T08 `21aa30683dc`、T07 delete-delta `c30fa15d99a`），勿重做** +# 🎯 当前状态（2026-07-07）= **P7.3 原子批 INC-1 + INC-2 均已实现并通过——按用户拍板的「整批一次原子提交」，两批代码都仍 UNCOMMITTED 在工作树里（跨 session 的 WIP 载体，勿 commit、勿 revert、勿 `git checkout`）；本轮只 commit 了本 HANDOFF + 设计文档。INC-2 验证：`fe-connector-hive` 模块 build SUCCESS、16 新单测全绿（`HiveWriteUtilsTest` 12 + `NameMappingTest` 4）、checkstyle 0、import-gate 净。下一步 = INC-3（`HiveWriteContext` + `HiveConnectorTransaction` 核心 + `beginTransaction`——最硬的一块，含分类状态机 + `HmsCommitter` + 断三硬耦合 D4/D5/D6 + rollback 删 staging/abort MPU）。前序 seam/回归已落地（T08 `21aa30683dc`、T07 delete-delta `c30fa15d99a`），勿重做** -> **⚠️ 工作树有未提交的 INC-1 代码**：`git status` 会显示 `fe/fe-connector/fe-connector-hms/` 下 4 新文件 + 4 改文件（`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` + 2 test）。这是**故意的**（原子批模型，见设计文档 §8）——下个 session 在此基础上继续 INC-2..INC-5，全绿后才一次性 feature commit。**若工作树被清就得照设计文档 §6 INC-1 块重建**（该块已勾选 [x] 并记录了精确签名与 4 处偏差）。 +> **⚠️ 工作树有未提交的 INC-1 + INC-2 代码**：`git status` 会显示——INC-1：`fe/fe-connector/fe-connector-hms/` 下 4 新文件 + 4 改文件（`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` + 2 test）；**INC-2：`fe-connector-hive` 下 4 新文件**（`HiveWriteUtils.java` + `NameMapping.java` + `HiveWriteUtilsTest.java` + `NameMappingTest.java`，**零改动既有文件**）。这是**故意的**（原子批模型，见设计文档 §8）——下个 session 在此基础上继续 INC-3..INC-5，全绿后才一次性 feature commit。**若工作树被清就得照设计文档 §6 INC-1/INC-2 块重建**（两块已勾选 [x] 并记录了精确签名/偏差/叶子类型裁决）。 > **⚠️ 权威实现计划已切换到设计文档**：本轮把 spec 的 T01–T11 拆解**收敛成一份可直接执行的设计**（4 类拆分修正、精确签名、3 硬耦合断法、INC-1..INC-5 构建顺序、逐增量测试计划、原子提交管理策略）。**下个 session 直接照 `tasks/P7.3-hive-write-txn-implementation-design.md` 的 §6 构建顺序执行**，spec 的 T01–T11 表仅作背景。 > > **本轮侦察的关键修正（写进设计文档，勿再纠结）**：① **写路径是「4 类拆分」**——`HiveConnectorTransaction` + 不可变 `HiveWriteContext` + **独立** `HiveWritePlanProvider`（planWrite **不在** metadata，spec 说法有误）+ `HiveConnectorMetadata.beginTransaction` 一行工厂，全对齐 iceberg。② **fe-core 桥零改动**已复核（`profileLabel()="HMS"` 复用既存 `TransactionType.HMS` 枚举）。③ **读侧 ACID（T07 生产半）并不与写事务机器耦合**——`HMSTransaction` 全程不调 `openTxn/commitTxn/getValidWriteIds`，四个读原语仅读侧用；所谓「拖入 fe-core」是浅的（插件已用裸 Hadoop FS + 自有 `HiveScanRange`，`FileCacheValue/FileSystemTransferUtil/LocationPath/StorageProperties/HivePartition/AcidInfo` 在边界处全落地），真正搬的只有纯目录名解析 + `hive-common ValidWriteIdList` 算法。④ hive **不在** `SPI_READY_TYPES`（`{jdbc,es,trino-connector,max_compute,paimon,iceberg}`）——整批**天然 dormant/安全**，编译+单测但零线上路由，同 iceberg/paimon 翻闸前范式。⑤ **翻闸/fe-core retype/Env 摘 `HiveTransactionMgr`/删 legacy 均不在本原子批**（属后续 P7.4/P7.5，另起一次原子提交）。 -> **本轮做了什么（INC-1，未 commit）** —— 照设计文档 §6 起手实现原子批第一增量：**在 `fe-connector-hms` 建齐 9 个 HmsClient 元存储原语 + 4 个 DTO + 转换器扩展 + 录制式假客户端单测**。 -> - **`HmsClient` 接口**：Phase-3 补 5 写原语（addPartitions / updateTable+PartitionStatistics / dropPartition / partitionExists），新 Phase-4 补 4 读-ACID 原语（openTxn / commitTxn / getValidWriteIds / acquireSharedLock），均照既有 DDL 块 `default`-throw 写法。 -> - **4 新 DTO**：`HmsCommonStatistics`（port `CommonStatistics`）、`HmsPartitionStatistics`（port `HivePartitionStatistics`，**丢弃 column-stats map**——写路径恒空且是唯一 fe-core 拖拽）、`HmsPartitionWithStatistics`（builder；**打平** `HivePartition` 的 SD 字段，绕开 fe-core 类型）、`HmsAcidConstants`（两个 BE 契约 key，INC-5 复用）。均 connector-api/JDK/hive-metastore-api only。 -> - **`ThriftHmsClient` 实现**：每个原语 `execute(client -> …)`（auth/TCCL 由 `execute→doAs` 免费提供）。getValidWriteIds 成功路径（`getValidTxns()` **无参**快照 + 两 key）+ 版本不兼容时 max-watermark 降级 fallback；acquireSharedLock 用 `LockRequestBuilder`/`LockComponentBuilder`（SHARED_READ + `checkLock` 忙轮询，无 heartbeat=OQ-LOCK）；**addPartitions batch-20 移进 client**（JDK subList，不引 guava）；统计走 **`alter_table`/`alter_partition` 参数重建**（非 `SetPartitionsStatsRequest`）；partitionExists **新写**=getPartition 在 lambda 内吞 `NoSuchObjectException`（not-found 不 taint 池化 client）。 -> - **`HmsWriteConverter` 扩展**：`toHivePartitions` + `toStatisticsParameters`（=`HiveUtil.updateStatisticsParameters`，spec 里叫 `toColumnStatistics` **系误名**，其实是基础统计参数 numRows/totalSize/numFiles，非 ColumnStatistics）+ `toPartitionStatistics` + 私有 `emptyToNull`（去 guava）。 -> - **单测（无 Mockito，JDK `Proxy` 录制式假 `IMetaStoreClient`）**：`ThriftHmsClientWriteAcidTest` 16 例（转发/batch-20 **全覆盖**校验 union==p0..p44 而非仅调用次数/getValidWriteIds **成功**（辨别水位，证明确实走成功支非 fallback）+fallback/锁 acquire·poll·timeout/统计重建/partitionExists 真假/歧义名拒绝）；`HmsWriteConverterTest` +5。**pom 加 2 个 test-scoped 依赖**（`hadoop-mapreduce-client-core`→`JobConf`、`commons-lang`→`HiveConf` 静态初始化，因 ctor 会 new HiveConf；照 iceberg/paimon 先例）。 -> - **验证**：module test SUCCESS（49 hms 用例）、checkstyle 0、import-gate 净。**3-lens 对抗 review + 逐条 verify**：0 生产缺陷、2 测试覆盖缺口（batch 只校验次数、getValidWriteIds 成功路径无覆盖）——均已补测修掉。 -> - **4 处对 §4.1 的偏差（均忠实迁移、零行为变化，已写进设计文档 §6 INC-1 块）**：batch-20 在 client 非 committer；`toColumnStatistics`→`toStatisticsParameters`/`toPartitionStatistics`；column-stats map 丢弃；partitionExists 新写。 -> - **前序已落地（历史 commit，勿重做）**：T08 连接器无关读事务 seam（`QueryFinishCallbackRegistry` + `QeProcessorImpl` 通用 drain 替 hive 硬编 + `HiveScanNode` 经 seam 注册，`21aa30683dc`）；T07 delete-delta 回归修复（`c30fa15d99a`）。T08 剩余（`HiveTransactionMgr` 出 `Env` 入插件）与 T07 读侧 ACID（INC-5）同批。 +> **本轮做了什么（INC-2，未 commit）** —— 照设计文档 §6 续做原子批第二增量：**在 `fe-connector-hive` 落地写路径纯 helper + 唯一无插件对等物的叶子标识类**（先读插件侧 R6 侦察，把「搬什么」收敛到最小）。 +> - **`HiveWriteUtils`（新，package-private）**：纯路径 helper（`isSubDirectory`/`getImmediateChildPath`/`pathsEqual` 包级可见供 INC-3 调；`sameFileSystem`/`normalizeUriPart`/`normalizePath` 私有）+ `mergePartitions`（纯 `THivePartitionUpdate` 合并，改 static）。**逐字忠实**移植 HEAD `HMSTransaction`（1654–1737 + 153–168）；fe-core-free（仅 Hadoop `Path` + `java.net.URI` + 共享 thrift；**无 guava/lombok**）。 +> - **`NameMapping`（新，co-move）**：fe-core `datasource.NameMapping` 的 JDK-only 插件副本（去 lombok `@Getter`/guava `@VisibleForTesting`，手写 getter + equals/hashCode/toString）。它是事务分类 action-map 的 **key**、且无插件对等物，故唯一 co-move。 +> - **R6 叶子类型裁决（读插件侧后收敛）**：统计三件套**不搬**（INC-1 已有插件对等物且带全套运算：`CommonStatistics`→`HmsCommonStatistics`、`HivePartitionStatistics`→`HmsPartitionStatistics`、`HivePartitionWithStatistics`→`HmsPartitionWithStatistics`）；**`HivePartition` 故意推迟到 INC-3**——其 fe-core `Column` 拖拽（`getPartitionName(List)`）经查**写路径恒不调用**，且 INC-1 打平的 `HmsPartitionWithStatistics` 已取代其 SD 角色，`PartitionAndMore` 描述子形态是 INC-3 主体移植时的决定，现在预造副本属投机（Rule 2）。 +> - **单测（JUnit5，无 Mockito）**：`HiveWriteUtilsTest` 12 例（mergePartitions 求和/拼接/多名保留/pending-upload 聚合/null 安全；isSubDirectory happy/相等/兄弟前缀/跨 FS/null；getImmediateChildPath 首层/直接子/非子；pathsEqual 归一/跨 FS/null——每例锚定**为什么**：commit 计数、staging 清理边界）+ `NameMappingTest` 4 例（值 equals/hashCode、**按值做 map key** 解析、任一字段差异破坏相等、全名拼接——key 语义关键：坏 key 会把同表 action 拆桶丢写）。 +> - **验证**：`fe-connector-hive` build SUCCESS、16 新单测全绿、checkstyle 0、import-gate 净（仅 `HiveVersionUtil` 已知误报，与本轮新文件无关；新文件零 fe-core import）。本轮**零改动既有文件**（纯新增 4 文件），故不会破坏既有测试。 +> - **INC-1（上一轮，未 commit）已落地**（详见设计文档 §6 INC-1 块）：`fe-connector-hms` 9 个 HmsClient 元存储原语 + 4 DTO（`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsAcidConstants`）+ `HmsWriteConverter` 扩展 + `ThriftHmsClientWriteAcidTest` 16 例 + `HmsWriteConverterTest` +5（module test SUCCESS 49 例、checkstyle 0）。 +> - **前序已落地（历史 commit，勿重做）**：T08 连接器无关读事务 seam（`21aa30683dc`）；T07 delete-delta 回归修复（`c30fa15d99a`）。T08 剩余（`HiveTransactionMgr` 出 `Env` 入插件）与 T07 读侧 ACID（INC-5）同批。 > > **上一轮（P7.3 recon+设计）已定的产出仍有效**（写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块 + OQ 段 ✅）： > - **通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，非从零发明 seam。 @@ -50,9 +49,13 @@ --- -# 🚀 下个 session 任务 = **照 [`tasks/P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md) §6 续做 INC-2（hive 支撑叶子类型 + 纯 helper 挪进 `fe-connector-hive`）→ INC-3（`HiveWriteContext`+`HiveConnectorTransaction` 核心，最硬的一块）→ INC-4（`HiveWritePlanProvider`+`buildSink`）→ INC-5（读侧 ACID 生产半，可用 INC-1 的 `HmsAcidConstants`/read 原语）。INC-1 已在工作树（未 commit）；全批全绿后一次性 feature commit** +# 🚀 下个 session 任务 = **照 [`tasks/P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md) §6 续做 INC-3（`HiveWriteContext` + `HiveConnectorTransaction` 核心 + `beginTransaction`，最硬的一块）→ INC-4（`HiveWritePlanProvider`+`buildSink`）→ INC-5（读侧 ACID 生产半，可用 INC-1 的 `HmsAcidConstants`/read 原语）。INC-1+INC-2 已在工作树（未 commit）；全批全绿后一次性 feature commit** -> **INC-2 起步先做（R6，设计文档 §6 明确警告）**：**先读 `fe-connector-hive` 插件侧已有什么**，再决定 `datasource.hive` 里哪些叶子类型真需要挪 vs 已有插件对等物——过度搬运会拖入 fe-core 依赖。纯 helper（path 工具 + `mergePartitions`）与 DTO 优先复用 INC-1 已建的 `Hms*` 类型；只 co-move 无插件对等物的。INC-2/INC-3 的写原语 DTO 应直接用 INC-1 的 `HmsPartitionWithStatistics`/`HmsPartitionStatistics`/`HmsCommonStatistics`（committer 侧 batch-20 已在 client 内，故 committer 只需一次 `hmsClient.addPartitions(db,table,全部)`）。 +> **INC-3 起步先做**：**先读 HEAD `HMSTransaction` 的 `commit()`/分类状态机/`HmsCommitter` 控制流**（信 HEAD 不信行号）+ 对照 `IcebergConnectorTransaction` 模板。写原语 DTO 直接用 INC-1 的 `HmsPartitionWithStatistics`/`HmsPartitionStatistics`/`HmsCommonStatistics`（committer 侧 batch-20 已在 client 内，故 committer 只需一次 `hmsClient.addPartitions(db,table,全部)`）；action-map 用 INC-2 的插件 `NameMapping` 当 key；纯 path helper/合并用 INC-2 的 `HiveWriteUtils`。 +> +> **INC-3 的三个硬点**（设计文档 §5.3 + R1/R3）：① 断 `ConnectContext` profile/queryId → queryId 走 `HiveWriteContext`（D4，丢 profiling）；② MPU downcast（R3 最大新代码风险）→ 用 `context.getStorageProperties()` 建 `fe-filesystem-spi` FileSystem、`instanceof ObjFileSystem` guard + `completeMultipartUpload`/`abortMultipartUpload`（D6）；③ 插件自有 async-rename `Executor` + stats `newFixedThreadPool(16)`，其上每个 fs/rename/MPU task **各自** `context.executeAuthenticated` 包裹（D5/R1，**勿**抄 iceberg 的 JVM-wide primer）。**范围**：full-ACID 写 begin-guard 硬拒（D7）；rollback 删 staging + abort MPU 非 no-op（D9）；commit switch 只 `{INSERT, OVERWRITE}`。 +> +> **INC-3 还需（从 INC-2 推迟过来）**：决定 `PartitionAndMore`/`TableAndMore` 描述子形态——重塑到 `HmsPartitionWithStatistics`（INC-1 打平 SD）+ 内联字段，**不**复制 fe-core `HivePartition`（其 `Column` 拖拽方法写路径已死，INC-2 已验）。 > **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index cf4507accd0f06..a47629c6b5836e 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -191,13 +191,14 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] `HmsWriteConverter`: added `toHivePartitions` + `toStatisticsParameters` (**renamed** from spec's `toColumnStatistics` — it is `HiveUtil.updateStatisticsParameters`: numRows/totalSize/numFiles + CDH flag) + `toPartitionStatistics` (`HiveUtil.toHivePartitionStatistics`) + private `emptyToNull` (drops guava `Strings.emptyToNull`). - [x] **Tests**: `ThriftHmsClientWriteAcidTest` (recording-fake `IMetaStoreClient` via JDK `Proxy` + package-private ctor + `MetaStoreClientProvider` + `poolSize=0`; 16 tests) — forwards, batching **coverage** (union==p0..p44, not just call count), getValidWriteIds **success** (distinctive watermark, provably not fallback) + fallback, lock acquire/poll/timeout, stats rebuild, partitionExists true/false, ambiguous-name reject. `HmsWriteConverterTest` +5. **pom.xml: +2 test-scoped deps** (`hadoop-mapreduce-client-core` for `JobConf`, `commons-lang` for `HiveConf` static init — mirrors iceberg/paimon; `HiveConf` is built in the `ThriftHmsClient` ctor, so instantiating it in a unit test needs them). - **Deviations vs §4.1 (all faithful-migration, no behavior change): batch-20 in client not committer; `toColumnStatistics`→`toStatisticsParameters`/`toPartitionStatistics` (param math, not `ColumnStatistics`); column-stats map dropped; `partitionExists` written fresh. Adversarial review (3 lenses × verify): 0 production defects, 2 test-coverage gaps (both fixed above).** -- [ ] **INC-2 — hive support leaf types + pure helpers into `fe-connector-hive`** - - [ ] Pure path helpers (`isSubDirectory`/`getImmediateChildPath`/`sameFileSystem`/`normalizePath`/`pathsEqual`) + `mergePartitions` (pure `THivePartitionUpdate` merge). - - [ ] Plugin equivalents / co-move for `NameMapping`, `CommonStatistics`, `HivePartition`/`HivePartitionStatistics`/`HivePartitionWithStatistics` **as needed by the transaction** (prefer plugin DTOs from INC-1; co-move only what has no plugin peer). **Verify what already exists plugin-side before moving.** - - [ ] **Tests**: pure helper + mergePartitions unit tests. +- [x] **INC-2 — hive support leaf types + pure helpers into `fe-connector-hive`** — **DONE (uncommitted, in working tree; module build SUCCESS, 16 new tests green, checkstyle 0, import-gate clean)** + - [x] Pure path helpers + `mergePartitions` → new package-private `HiveWriteUtils` (`fe-connector-hive`): `isSubDirectory`/`getImmediateChildPath`/`pathsEqual` (package-private, INC-3-callable) + private `sameFileSystem`/`normalizeUriPart`/`normalizePath` + `mergePartitions` (pure `THivePartitionUpdate` merge, static). **Byte-faithful** port of HEAD `HMSTransaction` (1654–1737 + 153–168); fe-core-free (only Hadoop `Path` + `java.net.URI` + shared thrift; **no guava/lombok**). `mergePartitions` made static (used no instance state). + - [x] **Leaf-type resolution after reading plugin-side first (R6):** the design's "co-move only what has no plugin peer" resolved to co-moving **only `NameMapping`** (`fe-connector-hive`, JDK-only copy of fe-core `datasource.NameMapping`, dropping lombok `@Getter`/guava `@VisibleForTesting`) — it is the transaction's per-table/per-partition action-map **key** and has no plugin peer. **NOT co-moved** (already have plugin peers from INC-1): `CommonStatistics`→`HmsCommonStatistics`, `HivePartitionStatistics`→`HmsPartitionStatistics` (both carry the full `EMPTY`/`fromCommonStatistics`/`merge`/`reduce`/`ReduceOperator` ops the txn uses), `HivePartitionWithStatistics`→`HmsPartitionWithStatistics` (flattened SD). **`HivePartition` deliberately deferred to INC-3** (not a leaf co-move): its fe-core `Column` drag (`getPartitionName(List)`) is **dead on the write path** — verified never called (the `getPartitionName()` hits are `PartitionAndMore.getPartitionName()`), and INC-1's flattened `HmsPartitionWithStatistics` already supersedes its SD role, so `PartitionAndMore`'s descriptor shape (Hms-types vs. a trimmed struct) is an INC-3 body-port decision — pre-creating a `HivePartition` copy now would be speculative (Rule 2) and risk rework. + - [x] **Tests**: `HiveWriteUtilsTest` (12: mergePartitions sum/concat/distinct-names/pending-upload-aggregate/null-safe; isSubDirectory happy/equal/sibling-prefix/cross-FS/null; getImmediateChildPath first-level/direct/non-child; pathsEqual normalize/cross-FS/null — each pins the *why*, e.g. commit accounting, staging-cleanup boundary) + `NameMappingTest` (4: value equals/hashCode, **map-key-by-value** resolution, any-field-difference breaks equality, full-name join — key semantics are load-bearing since a broken key splits one table's actions and drops writes). - [ ] **INC-3 — `HiveWriteContext` + `HiveConnectorTransaction` core + `beginTransaction`** (`fe-connector-hive`) — the genuinely-atomic unit - [ ] `HiveWriteContext` (§4.2). - - [ ] `HiveConnectorTransaction`: classification state machine (APPEND/NEW/OVERWRITE + probe downgrade) + `HmsCommitter` (prepare*/doCommit/abort/rollback + staging rename + MPU) + SPI methods. Break couplings #1/#3 (D4/D5/D6). full-ACID write reject (D7). rollback deletes staging + aborts MPU (D9). + - [ ] `HiveConnectorTransaction`: classification state machine (APPEND/NEW/OVERWRITE + probe downgrade) + `HmsCommitter` (prepare*/doCommit/abort/rollback + staging rename + MPU) + SPI methods. Break couplings #1/#3 (D4/D5/D6). full-ACID write reject (D7). rollback deletes staging + aborts MPU (D9). **Action maps key by INC-2's plugin `NameMapping`.** + - [ ] **Partition-descriptor shaping (deferred from INC-2):** decide `PartitionAndMore`/`TableAndMore` descriptor form — reshape onto `HmsPartitionWithStatistics` (INC-1 flattened SD) + inline fields rather than copying fe-core `HivePartition` (its `Column`-dragging `getPartitionName` is dead on write; verified in INC-2). `addPartitions` already consumes `List` with batch-20 inside the client, so the committer builds `HmsPartitionWithStatistics` directly. - [ ] `HiveConnectorMetadata.beginTransaction` one-liner (+ `ConnectorWriteOps.beginTransaction`). - [ ] **Tests**: `HiveConnectorTransactionTest` (recording-fake `HmsClient` + fake `FileSystem`): classification cases, commit dispatch (add_partitions + staging rename + MPU complete), rollback (staging delete + MPU abort), addCommitData+getUpdateCnt, full-ACID-write reject. - [ ] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive`) From 3f38790456ada99ff16ab0831f30982a13bdb320 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 17:21:11 +0800 Subject: [PATCH 020/334] [doc](catalog) P7.3 hive INC-3: record main port DONE+green (uncommitted) + land port-map + refresh HANDOFF MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INC-3 = fe-core HMSTransaction(1895) faithfully ported to plugin HiveConnectorTransaction(1697) + HiveWriteContext(89) + beginTransaction one-liner + HiveWriteUtils equalsIgnoreSchemeIfOneIsS3/toPartitionValues + HiveConnectorTransactionTest(9, FS-free half) + FakeConnectorContext. All feature code stays UNCOMMITTED in the working tree (atomic-batch model, design §8); this commit is docs only. Verified independently: fe-connector-hive module 62 tests green (no regression), checkstyle 0, import-gate clean, compile+test BUILD SUCCESS; riskiest spans (D6 MPU/FS build, commit/rollback order, classification / NEW->APPEND downgrade, full-ACID-write guard) spot-checked faithful. Remaining INC-3: FS-heavy committer tests (via resolveObjectStoreFileSystem seam) + full adversarial fidelity review; then INC-4, INC-5, one atomic feature commit. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 122 ++-- plan-doc/tasks/P7.3-INC-3-portmap.md | 595 ++++++++++++++++++ ....3-hive-write-txn-implementation-design.md | 15 +- 3 files changed, 654 insertions(+), 78 deletions(-) create mode 100644 plan-doc/tasks/P7.3-INC-3-portmap.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 6e60ebd9f736e2..90c4100344791c 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,95 +5,73 @@ --- -# 🎯 当前状态（2026-07-07）= **P7.3 原子批 INC-1 + INC-2 均已实现并通过——按用户拍板的「整批一次原子提交」，两批代码都仍 UNCOMMITTED 在工作树里（跨 session 的 WIP 载体，勿 commit、勿 revert、勿 `git checkout`）；本轮只 commit 了本 HANDOFF + 设计文档。INC-2 验证：`fe-connector-hive` 模块 build SUCCESS、16 新单测全绿（`HiveWriteUtilsTest` 12 + `NameMappingTest` 4）、checkstyle 0、import-gate 净。下一步 = INC-3（`HiveWriteContext` + `HiveConnectorTransaction` 核心 + `beginTransaction`——最硬的一块，含分类状态机 + `HmsCommitter` + 断三硬耦合 D4/D5/D6 + rollback 删 staging/abort MPU）。前序 seam/回归已落地（T08 `21aa30683dc`、T07 delete-delta `c30fa15d99a`），勿重做** - -> **⚠️ 工作树有未提交的 INC-1 + INC-2 代码**：`git status` 会显示——INC-1：`fe/fe-connector/fe-connector-hms/` 下 4 新文件 + 4 改文件（`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` + 2 test）；**INC-2：`fe-connector-hive` 下 4 新文件**（`HiveWriteUtils.java` + `NameMapping.java` + `HiveWriteUtilsTest.java` + `NameMappingTest.java`，**零改动既有文件**）。这是**故意的**（原子批模型，见设计文档 §8）——下个 session 在此基础上继续 INC-3..INC-5，全绿后才一次性 feature commit。**若工作树被清就得照设计文档 §6 INC-1/INC-2 块重建**（两块已勾选 [x] 并记录了精确签名/偏差/叶子类型裁决）。 - -> **⚠️ 权威实现计划已切换到设计文档**：本轮把 spec 的 T01–T11 拆解**收敛成一份可直接执行的设计**（4 类拆分修正、精确签名、3 硬耦合断法、INC-1..INC-5 构建顺序、逐增量测试计划、原子提交管理策略）。**下个 session 直接照 `tasks/P7.3-hive-write-txn-implementation-design.md` 的 §6 构建顺序执行**，spec 的 T01–T11 表仅作背景。 -> -> **本轮侦察的关键修正（写进设计文档，勿再纠结）**：① **写路径是「4 类拆分」**——`HiveConnectorTransaction` + 不可变 `HiveWriteContext` + **独立** `HiveWritePlanProvider`（planWrite **不在** metadata，spec 说法有误）+ `HiveConnectorMetadata.beginTransaction` 一行工厂，全对齐 iceberg。② **fe-core 桥零改动**已复核（`profileLabel()="HMS"` 复用既存 `TransactionType.HMS` 枚举）。③ **读侧 ACID（T07 生产半）并不与写事务机器耦合**——`HMSTransaction` 全程不调 `openTxn/commitTxn/getValidWriteIds`，四个读原语仅读侧用；所谓「拖入 fe-core」是浅的（插件已用裸 Hadoop FS + 自有 `HiveScanRange`，`FileCacheValue/FileSystemTransferUtil/LocationPath/StorageProperties/HivePartition/AcidInfo` 在边界处全落地），真正搬的只有纯目录名解析 + `hive-common ValidWriteIdList` 算法。④ hive **不在** `SPI_READY_TYPES`（`{jdbc,es,trino-connector,max_compute,paimon,iceberg}`）——整批**天然 dormant/安全**，编译+单测但零线上路由，同 iceberg/paimon 翻闸前范式。⑤ **翻闸/fe-core retype/Env 摘 `HiveTransactionMgr`/删 legacy 均不在本原子批**（属后续 P7.4/P7.5，另起一次原子提交）。 - -> **本轮做了什么（INC-2，未 commit）** —— 照设计文档 §6 续做原子批第二增量：**在 `fe-connector-hive` 落地写路径纯 helper + 唯一无插件对等物的叶子标识类**（先读插件侧 R6 侦察，把「搬什么」收敛到最小）。 -> - **`HiveWriteUtils`（新，package-private）**：纯路径 helper（`isSubDirectory`/`getImmediateChildPath`/`pathsEqual` 包级可见供 INC-3 调；`sameFileSystem`/`normalizeUriPart`/`normalizePath` 私有）+ `mergePartitions`（纯 `THivePartitionUpdate` 合并，改 static）。**逐字忠实**移植 HEAD `HMSTransaction`（1654–1737 + 153–168）；fe-core-free（仅 Hadoop `Path` + `java.net.URI` + 共享 thrift；**无 guava/lombok**）。 -> - **`NameMapping`（新，co-move）**：fe-core `datasource.NameMapping` 的 JDK-only 插件副本（去 lombok `@Getter`/guava `@VisibleForTesting`，手写 getter + equals/hashCode/toString）。它是事务分类 action-map 的 **key**、且无插件对等物，故唯一 co-move。 -> - **R6 叶子类型裁决（读插件侧后收敛）**：统计三件套**不搬**（INC-1 已有插件对等物且带全套运算：`CommonStatistics`→`HmsCommonStatistics`、`HivePartitionStatistics`→`HmsPartitionStatistics`、`HivePartitionWithStatistics`→`HmsPartitionWithStatistics`）；**`HivePartition` 故意推迟到 INC-3**——其 fe-core `Column` 拖拽（`getPartitionName(List)`）经查**写路径恒不调用**，且 INC-1 打平的 `HmsPartitionWithStatistics` 已取代其 SD 角色，`PartitionAndMore` 描述子形态是 INC-3 主体移植时的决定，现在预造副本属投机（Rule 2）。 -> - **单测（JUnit5，无 Mockito）**：`HiveWriteUtilsTest` 12 例（mergePartitions 求和/拼接/多名保留/pending-upload 聚合/null 安全；isSubDirectory happy/相等/兄弟前缀/跨 FS/null；getImmediateChildPath 首层/直接子/非子；pathsEqual 归一/跨 FS/null——每例锚定**为什么**：commit 计数、staging 清理边界）+ `NameMappingTest` 4 例（值 equals/hashCode、**按值做 map key** 解析、任一字段差异破坏相等、全名拼接——key 语义关键：坏 key 会把同表 action 拆桶丢写）。 -> - **验证**：`fe-connector-hive` build SUCCESS、16 新单测全绿、checkstyle 0、import-gate 净（仅 `HiveVersionUtil` 已知误报，与本轮新文件无关；新文件零 fe-core import）。本轮**零改动既有文件**（纯新增 4 文件），故不会破坏既有测试。 -> - **INC-1（上一轮，未 commit）已落地**（详见设计文档 §6 INC-1 块）：`fe-connector-hms` 9 个 HmsClient 元存储原语 + 4 DTO（`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsAcidConstants`）+ `HmsWriteConverter` 扩展 + `ThriftHmsClientWriteAcidTest` 16 例 + `HmsWriteConverterTest` +5（module test SUCCESS 49 例、checkstyle 0）。 -> - **前序已落地（历史 commit，勿重做）**：T08 连接器无关读事务 seam（`21aa30683dc`）；T07 delete-delta 回归修复（`c30fa15d99a`）。T08 剩余（`HiveTransactionMgr` 出 `Env` 入插件）与 T07 读侧 ACID（INC-5）同批。 -> -> **上一轮（P7.3 recon+设计）已定的产出仍有效**（写进 `tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块 + OQ 段 ✅）： -> - **通用写入通路已由 iceberg P6 建好**（`PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` + `ConnectorTransaction`/`ConnectorWritePlanProvider`/`PluginDrivenTransactionManager`，**fe-core 桥零改动**）⇒ P7.3 = 把 hive INSERT 折进现成通路 + 照抄 `IcebergConnectorTransaction`/`IcebergWriteContext`，非从零发明 seam。 -> - **缺的 HMS 写原语底层全有**（vendored `HiveMetaStoreClient` 已实现 addPartitions/stats/ACID 全套）→ 缺的只是 `HmsClient` 接口开口 + 一行 `execute` 转发。 -> - **三项决策（2026-07-06，spec OQ 段 ✅）**：① **OQ-RTX = 主干加通用 per-query finish 回调**（连接器无关，替 `QeProcessorImpl:210→Env.getCurrentHiveTransactionMgr` 硬编，对齐 Trino）；② **OQ-ACID-WRITE = 迁移行为保持**（非-ACID 写 + ACID 读迁到位，full-ACID 写继续硬拒，控 R-002）；③ **OQ-LOCK = 保持现状**（读侧共享 HMS 锁无 heartbeat）。 -> -> **P7.1 交付的地基（已在工作分支）**：`fe-connector-hms` 有 DDL 写客户端（create/drop db+table、truncate，default-throw seam + `ThriftHmsClient` 实现）；`fe-connector-hive` `HiveConnectorMetadata` DDL override 齐（txn/planWrite override 仍 0）；env channel 通；shared converter 带列默认值 + 显式分区标志。 - -> **整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`**： -> P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi(hybrid) #64143 · P4 maxcompute #64300 · P5 paimon 迁移+翻闸 #64446 + 删 legacy #64653 · P3b kerberos→fe-kerberos #64655 · **P6 iceberg #64688（本轮收官）**。 -> -> **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`，工作树干净）。P6 期间所有"未 push（[DEC-FLIP-1] 铁律）"的翻闸/删死码/属性迁移工作**已全部随 #64688 一次性合入**——**该铁律现已解除**，P7 走常规「连接器内实现 → 翻闸 → 删 legacy → squash-合入」流程（复用 P5/P6 样板）。 - -## #64688 做了什么（685 文件，+79738/−23744） - -把原生 iceberg（元数据/scan/write/procedures/sys-tables/行级 DML）整体迁到自包含 `fe-connector-iceberg` + 翻闸（iceberg 入 `CatalogFactory.SPI_READY_TYPES`）+ 删 fe-core 原生 iceberg 子系统： -- **删除**：`IcebergExternalCatalog`/`IcebergExternalTable`/`IcebergTransaction`/`IcebergNereidsUtils`、7 catalog flavor + factory、`broker/`·`dlf/`·`fileio/`·`action/`·`rewrite/`·`helper/`、四个 DML 执行器、planner 三 sink、`IcebergTransactionManager` + fe-core Iceberg/Paimon 元存储属性簇（S7，−4914 行）等。 -- **翻闸 + GSON 迁移**：旧 8 catalog 变体 + db + table 标签 `registerCompatibleSubtype`→`PluginDriven*`（保升级老集群反序列化）。 -- **属性/鉴权全归插件**（S1–S10，用户 2026-07-05 架构裁定）：fe-core 不再解析任何属性；已迁连接器（iceberg+paimon）走「插件解析 → BE thrift → 回传 fe-core」。详见 memory `catalog-spi-no-property-parsing-in-fecore`。 - -## ⚠️ 关键遗留（P7 必须接手）= fe-core `datasource/iceberg/` 还剩 **23 个 HMS-iceberg 支撑类** - -`#64688` 删的是**原生 iceberg 子系统**；但 **iceberg-on-HMS**（`type=hms` 下 `DlaType.ICEBERG` 的表）仍走 fe-core，因此以下 23 文件**故意保活**（decision D5 / Q3=B：HMS-iceberg 随 hive 一起迁）： -`IcebergUtils`、`IcebergMetadataOps`、`source/IcebergScanNode`(+`IcebergHMSSource`/`IcebergSource`/`IcebergSplit`/…)、`cache/`、`IcebergMvccSnapshot`、`IcebergSchema/Snapshot/Partition*CacheValue`、`profile/IcebergMetricsReporter`、`DorisTypeToIcebergType`、`IcebergCatalogConstants`。 -→ **P7 hive 迁移把 HMS-iceberg 挪到连接器路径后，这 23 文件才能删（阶段四）**。同理 fe-core `datasource/hudi/` 与 `datasource/hive/` 也在 P7 范围。 +# 🎯 当前状态（2026-07-07）= **P7.3 原子批 INC-1 + INC-2 + INC-3(主体) 均已实现并全绿——INC-3 把 fe-core `HMSTransaction`(1895 行) 忠实移植成插件 `HiveConnectorTransaction`(1697 行) + `HiveWriteContext`(89) + `HiveConnectorMetadata.beginTransaction` 一行工厂 + `HiveWriteUtils` 增 `equalsIgnoreSchemeIfOneIsS3`(GAP-11)/`toPartitionValues`(GAP-12) + `HiveConnectorTransactionTest`(9 例全绿) + `FakeConnectorContext`(测试 ctx double)。按用户拍板「整批一次原子提交」，全部仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit 了 HANDOFF + 设计文档 + 新落地的 port-map。** + +> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 独立 `compile` BUILD SUCCESS；全模块 `test` **62 测试 0 失败 0 错误 BUILD SUCCESS（含新 `HiveConnectorTransactionTest` 9、既有 hive/hms 全绿、无回归）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（仅 `HiveVersionUtil` 已知误报）。侦察侧我**逐行核对了最险的几段**（D6 MPU/FS 构建 `getFileSystem`/`resolveObjectStoreFileSystem`/`objCommit`；`commit()`/`rollback()` 顺序；分类状态机 `finishInsertTable`/NEW→APPEND 降级；full-ACID 拒写守卫 D7/D8），确为忠实移植。 + +> **⚠️ 本轮的过程事故（已恢复，无损）**：INC-3 的实现最初交给一个后台实现 agent（照 port-map 写代码），它写完**主体代码并使其编译通过**、正开始写测试时**撞上账号级 session limit（resets 5pm Asia/Shanghai）而中断**。我接手后：确认主体代码 `compile` + checkstyle + import-gate 全绿、逐行 spot-check 忠实、然后**亲手补齐了 FS-free 单测**（利用 `finishInsertTable` 包级可见 + `HmsClient` 方法多为 default-throw，绕开重型 FS 假件）。因 session limit 是共享池，本轮起**未再起新的多 agent 编排**，改在主循环内经济地推进。 + +> **⚠️ 工作树未提交的 INC-1 + INC-2 + INC-3 代码**（`git status` 会显示）： +> - INC-1（`fe-connector-hms`）：`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsPartitionInfo` 新 + 2 test。 +> - INC-2（`fe-connector-hive`）：`HiveWriteUtils.java` + `NameMapping.java`（+ 2 test）。 +> - **INC-3（`fe-connector-hive`，本轮）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1697) + `HiveConnectorTransactionTest.java`(351) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(+15，`beginTransaction` override) + `pom.xml`(+11，`fe-filesystem-spi` scope `provided`) + `HiveWriteUtils.java`（追加 `equalsIgnoreSchemeIfOneIsS3`/`toPartitionValues` 两个纯 helper + 单测）。 +> 这是**故意的**（原子批模型，设计文档 §8）——下个 session 在此基础上做 INC-3 收尾 + INC-4/INC-5，全批全绿后才一次性 feature commit。**若工作树被清就照 port-map 重建**。 + +> **权威实现依据**（信 HEAD 控制流，不信行号）： +> - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 +> - **`tasks/P7.3-INC-3-portmap.md`（本轮新落地，595 行）**：逐行移植地图 + §7 12 处 GAP + **§9 已核实的实现前检查**（GAP-11/12 结论、D6 MPU map 重载确认、FS 构建/鉴权/iceberg 模板签名全确认）。INC-4/INC-5 也可续用其读侧 §5.4 + 测试计划 §6。 +> - 移植源 = HEAD `fe/fe-core/.../datasource/hive/HMSTransaction.java`；模板 = `fe-connector-iceberg/.../IcebergConnectorTransaction.java`+`IcebergWriteContext.java`。 --- -# 🚀 下个 session 任务 = **照 [`tasks/P7.3-hive-write-txn-implementation-design.md`](./tasks/P7.3-hive-write-txn-implementation-design.md) §6 续做 INC-3（`HiveWriteContext` + `HiveConnectorTransaction` 核心 + `beginTransaction`，最硬的一块）→ INC-4（`HiveWritePlanProvider`+`buildSink`）→ INC-5（读侧 ACID 生产半，可用 INC-1 的 `HmsAcidConstants`/read 原语）。INC-1+INC-2 已在工作树（未 commit）；全批全绿后一次性 feature commit** +# 🚀 下个 session 任务 = **INC-3 收尾 →（全绿后）INC-4 → INC-5，全批一次原子 feature commit** + +> **① INC-3 收尾（两件，本轮故意未做——session-limit 下保守停在干净节点）**： +> - **(a) 提交器 FS 重活的单测**（port-map §6 里 commit/rollback 那半）：commit 派发（`addPartitions` 一次 + staging→target 改名 + MPU `completeMultipartUpload`）、rollback（MPU `abortMultipartUpload` + 删 staging + 幂等）。**注入路径已备好**：`HiveConnectorTransaction.resolveObjectStoreFileSystem(StorageProperties)` 是 `protected`，测试**子类化**它返回一个假 `ObjFileSystem`（记录 `completeMultipartUpload`/`getObjStorage().abortMultipartUpload`）+ 假 `org.apache.doris.filesystem.FileSystem`（记录 `rename`/`renameDirectory`/`delete`/`mkdirs`/`exists`）。本轮的 FS-free 测试已覆盖分类/降级/拒写/getUpdateCnt/SPI-identity，故这步是**补齐 FS 那一半**。 +> - **(b) 全量对抗性忠实度复审（clean-room，memory `clean-room-adversarial-review-pref`）**：我本轮只 spot-check 了最险几段（已确认忠实），但**尚未做多 agent 全量对抗复审**。宜派多 agent 各拿 `HiveConnectorTransaction` 与 HEAD `HMSTransaction` 逐维度对比（提交顺序 / `HmsCommitter` prepare*/doCommit/abort/rollback / MPU / D5 鉴权包裹 / 那 12 处 GAP / `Action` ctor 的 DROP-data==null 语义 / `convertToInsertExistingPartitionAction` 的 map/list index quirk 是否忠实），专挑分歧，再核对。**⚠️ session limit 是共享池——起多 agent 前先确认额度已恢复（5pm 后）**。 + +> **② INC-4 —— `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities**（`fe-connector-hive`，port-map §4.4/§5.2）：`planWrite` 调 `tx.beginWrite(...)` + `buildSink`（= 忠实移植 HEAD `planner.HiveTableSink.bindDataSink`：PARTITION_KEY/REGULAR 按表序、bucket、按格式压缩、staging-vs-in-place location、serde、`hadoopConfig` 经 `context.getStorageProperties().toBackendProperties()` + `vendStorageCredentials`）+ `HiveConnector.getWritePlanProvider()` + `supportedOperations={INSERT,OVERWRITE}` + capability gates；LZO-INSERT / 事务表拒写作 begin-guard。测试 = `HiveWritePlanProviderTest`（假 `ConnectorWriteHandle`，断 `THiveTableSink` 形状 + INSERT→OVERWRITE 提升 + guard）。 -> **INC-3 起步先做**：**先读 HEAD `HMSTransaction` 的 `commit()`/分类状态机/`HmsCommitter` 控制流**（信 HEAD 不信行号）+ 对照 `IcebergConnectorTransaction` 模板。写原语 DTO 直接用 INC-1 的 `HmsPartitionWithStatistics`/`HmsPartitionStatistics`/`HmsCommonStatistics`（committer 侧 batch-20 已在 client 内，故 committer 只需一次 `hmsClient.addPartitions(db,table,全部)`）；action-map 用 INC-2 的插件 `NameMapping` 当 key；纯 path helper/合并用 INC-2 的 `HiveWriteUtils`。 -> -> **INC-3 的三个硬点**（设计文档 §5.3 + R1/R3）：① 断 `ConnectContext` profile/queryId → queryId 走 `HiveWriteContext`（D4，丢 profiling）；② MPU downcast（R3 最大新代码风险）→ 用 `context.getStorageProperties()` 建 `fe-filesystem-spi` FileSystem、`instanceof ObjFileSystem` guard + `completeMultipartUpload`/`abortMultipartUpload`（D6）；③ 插件自有 async-rename `Executor` + stats `newFixedThreadPool(16)`，其上每个 fs/rename/MPU task **各自** `context.executeAuthenticated` 包裹（D5/R1，**勿**抄 iceberg 的 JVM-wide primer）。**范围**：full-ACID 写 begin-guard 硬拒（D7）；rollback 删 staging + abort MPU 非 no-op（D9）；commit switch 只 `{INSERT, OVERWRITE}`。 -> -> **INC-3 还需（从 INC-2 推迟过来）**：决定 `PartitionAndMore`/`TableAndMore` 描述子形态——重塑到 `HmsPartitionWithStatistics`（INC-1 打平 SD）+ 内联字段，**不**复制 fe-core `HivePartition`（其 `Column` 拖拽方法写路径已死，INC-2 已验）。 +> **③ INC-5 —— 读侧 ACID 生产半 + 插件读事务生命周期（dormant）**（`fe-connector-hive`，port-map §5.4）：移植 `getAcidState` **纯**目录名解析 + `hive-common Valid*` 算法；插件 `HiveTransaction`（openTxn/acquireSharedLock/getValidWriteIds/commitTxn，用 INC-1 已落地的读原语）；`HiveScanPlanProvider` 下潜 + `HiveScanRange.acidInfo(...)` 生产端接线；`HiveTableHandle` 的 `isTransactional`/`isFullAcid` 从 `getTableParameters()` 派生（D8）。测试 = `HiveAcidDescentTest` + 既有 `HiveScanRangeAcidTest` 保绿。**依赖仅 INC-1，独立于 INC-3/4**，也可先做。 -> **权威计划**：`tasks/P7-hive-migration.md` 末尾「**P7.3 逐 task 拆解**」块（recon 结论 + 3 决策 + T01–T11 任务表 + 移植指针 + HEAD 行号）。**信 HEAD 控制流不信本文/spec 行号。** 模板 = `IcebergConnectorTransaction`/`IcebergWriteContext`/`IcebergConnectorMetadata.beginTransaction+planWrite`；fe-core 桥 `PluginDrivenTransactionManager`/`PluginDrivenInsertExecutor` **零改动**。 +> **④ FINAL** —— INC-1..INC-5 全绿后**一次原子 feature commit**（path-whitelist `git add`，勿 `-A`）+ HANDOFF/doc 单独 commit。 -## 开场要点（P7.3 实现） +## 开场要点（承接） -1. **不发明中立 seam**：通用写入通路现成（iceberg P6 建）。P7.3 = 把 hive INSERT 折进现成通路 + 照抄 iceberg 事务。 -2. **落地顺序**（spec 已排 + 本轮修正）：**组1 写原语（T01/T02）与组2 连接器事务（T03–T05）同批 commit**（P7.1 教训：写原语不得先于事务作死代码，签名由实际调用反推）；组3 6 文件 retype（T06）依赖组2 provider/txn 就位。**组5 读事务中立 seam（T08）的连接器无关部分已落地（`21aa30683dc`）**——通用 `QueryFinishCallbackRegistry` + `QeProcessorImpl` 通用 drain 替 hive 硬编 + `HiveScanNode` 经 seam 注册；其剩余（`HiveTransactionMgr` 入插件）随 T07 走。**⚠️ 组4 读侧 ACID（T07）经本轮侦察已知非独立小切片**：消费半已在插件+单测，生产半（`AcidUtil.getAcidState` + 插件扫描下潜 + `openTxn`/`getValidWriteIds`）拖入 fe-core `filesystem`/`StorageProperties`/`FileCacheValue` 且**与写批共用 ACID HmsClient 原语**→ 宜与组1/组2 写批**同批**推进，非并行独立片（详见 spec T07 行 ⚠️ 段）。T07 delete-delta 回归修复已落地（`c30fa15d99a`），复用已修好的 populate。 -3. **3 硬耦合结**（成为插件事务时必断）：注入 profile/queryId（去 `ConnectContext.get()`）；thrift `THivePartitionUpdate` 簇随插件走；fs 抽象须暴露对象存储 MPU complete/abort（`SpiSwitchingFileSystem.forPath→ObjFileSystem`）。 -4. **范围锁定**：full-ACID **表**写继续硬拒（OQ-ACID-WRITE）；读侧共享锁**不加 heartbeat**（OQ-LOCK）。 -5. **硬门 = 独立 ACID 集成测试套件**（T10，R-002 项目最大风险）：INSERT/OVERWRITE/分区写/delete-delta 读/rollback/多 FE 失效。⚠️端到端写测试需插件写路径 live，但翻闸在 P7.5 → 须本地/scoped 翻闸跑该套件（或与 P7.5 联跑），**勿静默跳过**（Rule 12）。 -6. **勿重议的已定决策**（P7 全局）：**D-004** event→fe-connector-hms；**D-005** tableFormatType；**D-020** per-table SPI provider；**D-019** hudi live cutover 并入 P7；事务桥接 `PluginDrivenTransactionManager`（已存，零改动）。 -7. **纪律**：每 task 独立 commit + build + test + checkstyle 0（不带 -am）+ import-gate 净；**每轮完成即更新本 HANDOFF + commit**（memory `handoff-discipline-per-phase`）。 -8. **删除排序（最硬约束，spec §跨连接器删除排序，P7.4/P7.5 才触发）**：`datasource/hive/` 删不掉，直到 `HudiUtils`/`HudiExternalMetaCache`/`HudiScanNode`/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 全 retype 到 generic。 +1. **起步先读**本文顶部 🎯 段 + `tasks/P7.3-INC-3-portmap.md`（尤其 §9 已核实检查 + §6 测试计划）+ 设计文档 §6 构建顺序。**信 HEAD 控制流不信行号**——每处编辑前重读对应 HEAD 段。 +2. **主体已在，勿重写**：INC-3 的 `HiveConnectorTransaction` 已编译 + 单测 + checkstyle + import-gate 全绿且经 spot-check 忠实。下步是**加 FS 测试 + 全量复审**，不是重造。 +3. **范围锁定（勿重议）**：hive **不在** `SPI_READY_TYPES`（整批天然 dormant，编译+单测但零线上路由）；翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 删 legacy **均属后续 P7.4/P7.5，另起原子批**。full-ACID **写**继续硬拒（D7），full-ACID **读**在范围（INC-5）。 +4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 写路径 → P7.4/P7.5 翻闸时跑，勿静默跳过——Rule 12）。 +5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 --- # 📦 分支 / Commit 须知 - **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**（复用 P5-T29 #64653 / P6 #64688 范式）。 -- **公开 tracking issue = apache/doris#65185**（catalog-SPI 迁移 umbrella，大步骤勾选清单 + 连接器表）；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 +- **公开 tracking issue = apache/doris#65185**（catalog-SPI 迁移 umbrella）；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 - **⚠️ path-whitelist `git add`，严禁 `git add -A`**（scrub `regression-test/conf/regression-conf.groovy` 明文 key + `*.bak` + scratch `.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...` + `plan-doc/reviews/P5-paimon-rereview3-*`；`.claude/` 是 memory、非仓内）。 -- commit message：见 `git log` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 任务清单 + 设计文档单独 commit。 +- commit message：见 `git log` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 任务清单 + 设计文档 + port-map 单独 commit。 # ⚙️ 操作须知（复用） -- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**fe-core 只依赖 `fe-connector-api`**。**fe-connector-hive/hms 单独 build 注意 optional-shade 依赖**（如 HiveConf；参照 paimon `package` 而非 `test-compile` 的坑）。 -- **⚠️ checkstyle 别加 `-am`**：`-am` 把 `fe-common`（大量既存 error）拖进假红 → `mvn -pl : checkstyle:check`（不带 -am）。 -- **⚠️ bash 工具默认 timeout 120s**：fe-core build 超时 → 调 `timeout` ~590000ms 或后台跑（全模块 ~2min）。**后台 task 通知的 "exit code" 是末尾 echo/df 的、非 maven 的**——读 LOG 里 `BUILD SUCCESS`/`MAVEN_EXIT=` 行或 surefire XML（`tests=`/`failures=`），别信通知 exit。 -- **⚠️ maven 经管道 `$?` 是管道尾的** → 用 `${PIPESTATUS[0]}` 或 grep `BUILD SUCCESS`。**mutation/review 后 commit 前务必 fresh recompile**（stale `.class` 假红坑）。 -- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`。**HMS `HiveVersionUtil` import 命中 = 误报非违规**（`fe-connector-hms` 内 vendored 同名副本，非 fe-core）——勿改，详见 memory `catalog-spi-hms-hiveversionutil-gate-false-positive`。 -- **连接器测试无 Mockito**（真 InMemoryCatalog/Recording fakes）；**fe-core 用 Mockito**（`CALLS_REAL_METHODS` + `Deencapsulation.setField`；`anyString()` 不匹配 null）。详见 memory `catalog-spi-fe-core-test-infra`。 -- **cwd 会被 harness 重置** → 一律绝对路径。 -- **⚠️ 环境**：`/mnt/disk1` 紧（2.0T，~96% used）。**起步先 `df -h /mnt/disk1`**；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 +- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**checkstyle 别加 `-am`**：`mvn -pl : checkstyle:check`。 +- **⚠️ bash 工具默认 timeout 120s**：fe-connector-hive 全模块 build/test ~2min → 调 `timeout` ~580000ms。**后台 task 通知的 "exit code" 是末尾 echo 的、非 maven 的**——读 LOG 的 `BUILD SUCCESS` 行或 surefire XML（`Tests run=/Failures=/Errors=`），别信通知 exit。maven 经管道 `$?` 是管道尾的 → grep `BUILD SUCCESS`。**改代码后 commit 前务必 fresh recompile**（stale `.class` 假红）。 +- **连接器测试无 Mockito**（真 recording fakes；本轮 `HiveConnectorTransactionTest` 即用手写 recording `HmsClient` fake + `FakeConnectorContext`，`HmsClient` 多数 Phase-3+ 方法 default-throw、fake 只覆盖用到的）。checkstyle **禁 static import**（用 `Assertions.assertX`）、**扫 test 源**。 +- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 +- **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 紧**（2.0T ~82% used，360G free）；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够；本轮实现 agent 即在主树工作）。 # 🧠 起步必读 -1. **本文档顶部 🎯/🚀 段** + master plan [§3.8](./00-connector-migration-master-plan.md)（P7 战略）+ [`connectors/hive.md`](./connectors/hive.md)（P7 子阶段/SPI 缺口/特殊性）+ master plan §4（13 步 playbook）。 -2. **样板**：`tasks/P5-paimon-migration.md`（full-adopter + 翻闸 + 删 legacy 全流程）；`tasks/P6-iceberg-migration.md`（阶段拆分 spec 范式，P7 spec 照此建）。 -3. **铁律**：fe-core 不得新增 `if(hive)` / `instanceof HMSExternal*` / 引擎名字符串判别（新 seam 走中立 SPI / `ConnectorCapability`）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。 -4. **memory（现存相关项）**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`memory-keep-only-general-or-requested`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-plugindriven-no-source-specific-code`、`catalog-spi-no-property-parsing-in-fecore`、`catalog-spi-be-java-ext-shared-classpath`、`catalog-spi-plugin-tccl-classloader-gotcha`、`catalog-spi-connector-session-tz-gotcha`、`catalog-spi-history-schema-info-lowercase-nested-names`、`catalog-spi-connector-cache-framework-caffeine-coherence`、`catalog-spi-hms-hiveversionutil-gate-false-positive`、`catalog-spi-tracking-issue`。 -5. **上下文超 30% 即交接**（memory `session-handoff-at-30pct-context`）：找干净节点覆写本 HANDOFF + 通知用户开新 session。 +1. 本文档顶部 🎯/🚀 段 + `tasks/P7.3-INC-3-portmap.md` + 设计文档 `tasks/P7.3-hive-write-txn-implementation-design.md` §6。 +2. **样板**：`tasks/P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`tasks/P6-iceberg-migration.md`（阶段拆分范式）。模板事务 = `IcebergConnectorTransaction`/`IcebergWriteContext`。 +3. **铁律**：fe-core 不得新增 `if(hive)`/`instanceof HMSExternal*`/引擎名判别；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；插件跨边界须 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 +4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`memory-keep-only-general-or-requested`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-plugin-tccl-classloader-gotcha`、`catalog-spi-hms-hiveversionutil-gate-false-positive`、`catalog-spi-tracking-issue`。 + +--- + +## 背景：#64688（P6 iceberg 收官，已合入 `branch-catalog-spi`）+ P7 关键遗留 + +整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`：P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi #64143 · P4 maxcompute #64300 · P5 paimon #64446+#64653 · P3b kerberos #64655 · **P6 iceberg #64688**。#64688 把原生 iceberg 整体迁到自包含 `fe-connector-iceberg` + 翻闸 + 删 fe-core 原生 iceberg 子系统 + 属性/鉴权全归插件（用户 2026-07-05 架构裁定，memory `catalog-spi-no-property-parsing-in-fecore`）。 + +**⚠️ P7 必须接手的遗留**：`#64688` 删的是原生 iceberg；但 **iceberg-on-HMS**（`type=hms` 下 `DlaType.ICEBERG`）仍走 fe-core，故 fe-core `datasource/iceberg/` 还**故意保活 23 个 HMS-iceberg 支撑类**（`IcebergUtils`/`IcebergMetadataOps`/`source/IcebergScanNode`+…/`cache/`/`IcebergMvccSnapshot`/… ）。→ P7 hive 迁移把 HMS-iceberg 挪到连接器路径后，这 23 文件才能删（P7.4/P7.5 阶段四）。同理 fe-core `datasource/hudi/`、`datasource/hive/` 也在 P7 范围。**删除排序（最硬约束）**：`datasource/hive/` 删不掉，直到 `HudiUtils`/`HudiScanNode`/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 等全 retype 到 generic。 diff --git a/plan-doc/tasks/P7.3-INC-3-portmap.md b/plan-doc/tasks/P7.3-INC-3-portmap.md new file mode 100644 index 00000000000000..6702d197f02f1f --- /dev/null +++ b/plan-doc/tasks/P7.3-INC-3-portmap.md @@ -0,0 +1,595 @@ +# INC-3 PORT MAP — HiveWriteContext + HiveConnectorTransaction + beginTransaction + +> Synthesis of 5 reader reports, reconciled against REAL in-tree signatures (working tree, branch `catalog-spi-11-hive`, 2026-07-06/07). Design authority = `plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md` §2 (D1–D12), §4, §5, §6. Port source = HEAD `fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java` (1895 lines). Template = `fe-connector-iceberg/.../IcebergConnectorTransaction.java` + `IcebergWriteContext.java`. +> **Trust HEAD control flow, not line numbers.** INC-1 (`fe-connector-hms`) + INC-2 (`fe-connector-hive`) are DONE, uncommitted in tree. This file drives INC-3. + +--- + +## 0. FILES TO CREATE + WIRING + POM (design §4, §6) + +### New files (all in `fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/`) +| file | top-level type | visibility | notes | +|---|---|---|---| +| `HiveWriteContext.java` | `final class HiveWriteContext` | **package-private** (mirror `IcebergWriteContext`, no `public`) | immutable value object | +| `HiveConnectorTransaction.java` | `public class HiveConnectorTransaction implements ConnectorTransaction` | **public** (constructed by `HiveConnectorMetadata.beginTransaction`) | contains inner `HmsCommitter` + `Action`/`ActionType` + `TableAndMore`/`PartitionAndMore` + task classes + `UncompletedMpuPendingUpload` | + +### Edited files +| file | change | +|---|---| +| `HiveConnectorMetadata.java` (fe-connector-hive) | add `@Override public ConnectorTransaction beginTransaction(ConnectorSession session)` (one-liner, §5) | +| `HmsWriteConverter.java` (fe-connector-hms) | **OPTIONAL** — add a public `List toFieldSchemas(List cols)` helper IF you don't do the ConnectorColumn→FieldSchema conversion inline (see §5.2 / GAP-4) | +| `fe/fe-connector/fe-connector-hive/pom.xml` | add `fe-filesystem-spi` dependency, **scope `provided`** (see below) | + +### `beginTransaction` wiring point — METADATA, not a separate writeOps class +`ConnectorMetadata extends ConnectorWriteOps`; `ConnectorWriteOps.beginTransaction(ConnectorSession)` is a `default`-throw. `HiveConnectorMetadata implements ConnectorMetadata` → the override lands **once** on `HiveConnectorMetadata`. It already holds the two ctor args (`private final HmsClient hmsClient;` line 79, `private final ConnectorContext context;` line 86). Reached at runtime from `PluginDrivenInsertExecutor.beginTransaction()` via `writeOps = connector.getMetadata(session)` then `writeOps.beginTransaction(session)`. There is NO separate writeOps object. (`planWrite` is NOT here — it goes to `HiveWritePlanProvider` in INC-4, per D1.) + +### pom addition (fe-connector-hive/pom.xml) — mirror the `fe-thrift` `provided` block (pom lines 59–62) +```xml + + ${project.groupId} + fe-filesystem-spi + ${project.version} + provided + +``` +**MUST be `provided`, never bundled** — `ObjFileSystem`/`ObjStorage`/`FileSystemProvider` (module `fe-filesystem-spi`) come from the shared/parent classpath at runtime; a bundled second copy makes the plugin's `ObjFileSystem.class != runtime instance.class` → `instanceof` fails, downcast throws CCE (memory `catalog-spi-plugin-tccl-classloader-gotcha`). `FileSystem`/`Location`/`UploadPartResult`/`FileSystemUtil`/`StorageProperties`/`StorageKind` are ALREADY on compile CP transitively (fe-connector-spi → fe-filesystem-api). Only the `.spi.*` types need this new dep. + +### Recommended INC-3 internal build sub-order (each sub-step compiles) +1. **`HiveWriteContext`** (§2) — leaf value object; depends only on `WriteOperation` (`org.apache.doris.connector.api.handle.WriteOperation`) + `TFileType` (`org.apache.doris.thrift.TFileType`). Compiles alone. +2. **`HiveConnectorTransaction` skeleton** — class decl + fields + ctor + the 7 SPI overrides with `commit()`/`rollback()` as stubs (`getTransactionId`, `close`(empty), `addCommitData`, `getUpdateCnt`, `profileLabel`="HMS"). Compiles. +3. **Inner types** — `ActionType` enum, `Action`, `TableAndMore`, `PartitionAndMore`, task classes (`UpdateStatisticsTask`, `AddPartitionsTask`, `DirectoryCleanUpTask`, `RenameDirectoryTask`, `DeleteRecursivelyResult`), `UncompletedMpuPendingUpload`. Compiles. +4. **Classification** — `beginWrite` + `finishInsertTable` + state helpers (`getTable`, `finishChangingExistingTable`, `createTable`, `dropTable`, `checkNoPartitionAction`, `createAndAddPartition`, `addPartition`, `dropPartition`, `convertToInsertExistingPartitionAction`). Compiles. +5. **`HmsCommitter`** — fields + executor lifecycle + `prepare*`/`doCommit`/`abort`/`rollback` + FS-walk helpers + MPU blocks; wire real `commit()`/`rollback()` to it. Compiles. +6. **`beginTransaction`** one-liner on `HiveConnectorMetadata` (+ ConnectorColumn→FieldSchema helper if chosen). Compiles. +7. **`HiveConnectorTransactionTest`** (§6). + +--- + +## 1. `HiveWriteContext` (§4.2, template = `IcebergWriteContext`) + +`final class HiveWriteContext` — package-private. All fields `private final`, immutable, defensive-copy the Map. Package-private getters (no `public`). + +```java +final class HiveWriteContext { + private final WriteOperation writeOperation; // org.apache.doris.connector.api.handle.WriteOperation + private final boolean overwrite; + private final Map staticPartitionValues; // defensive copy in ctor + private final String queryId; // replaces ConnectContext (D4) + private final TFileType fileType; // org.apache.doris.thrift.TFileType — FILE_S3 vs staging + private final String writePath; // staging temp path (empty/unused when FILE_S3) + + HiveWriteContext(WriteOperation writeOperation, boolean overwrite, + Map staticPartitionValues, String queryId, + TFileType fileType, String writePath) { + this.writeOperation = writeOperation; + this.overwrite = overwrite; + this.staticPartitionValues = staticPartitionValues == null + ? Collections.emptyMap() : new HashMap<>(staticPartitionValues); // defensive copy (iceberg pattern) + this.queryId = queryId; + this.fileType = fileType; + this.writePath = writePath; + } + + WriteOperation getWriteOperation() { return writeOperation; } + boolean isOverwrite() { return overwrite; } + Map getStaticPartitionValues() { return staticPartitionValues; } // internal ref, no copy on read + boolean isStaticPartitionOverwrite() { return overwrite && !staticPartitionValues.isEmpty(); } // derived + String getQueryId() { return queryId; } + TFileType getFileType() { return fileType; } + String getWritePath() { return writePath; } +} +``` +**Divergence from `IcebergWriteContext`**: DROP iceberg's `branchName` (Optional) + `readSnapshotId` (long); ADD hive's `queryId`/`fileType`/`writePath` (no iceberg precedent — hive-specific, per §4.2). Keep the immutable-value-object + defensive-copied-Map + telescoping/single-ctor + package-private-getters shape verbatim. (INC-4 constructs this in `buildWriteContext`; INC-3 only consumes it via `beginWrite`.) + +--- + +## 2. `HiveConnectorTransaction` — SPI SHELL + +### 2.1 Class decl + imports +```java +public class HiveConnectorTransaction implements ConnectorTransaction +``` +SPI/engine imports: +- `org.apache.doris.connector.api.handle.ConnectorTransaction` +- `org.apache.doris.connector.api.handle.WriteOperation` +- `org.apache.doris.connector.api.DorisConnectorException` (RuntimeException; ctors `(String)`, `(String,Throwable)`) +- `org.apache.doris.connector.api.ConnectorSession` +- `org.apache.doris.connector.spi.ConnectorContext` +- `org.apache.doris.connector.hms.HmsClient`, `HmsTableInfo`, `HmsPartitionInfo`, `HmsPartitionWithStatistics`, `HmsPartitionStatistics`, `HmsCommonStatistics`, `HmsWriteConverter`, `HmsAcidConstants` +- `org.apache.doris.connector.hive.NameMapping`, `HiveWriteUtils` (package-private, same package — no import) +- thrift: `org.apache.doris.thrift.{TFileType, THivePartitionUpdate, THiveLocationParams, TS3MPUPendingUpload, TUpdateMode}` +- filesystem: `org.apache.doris.filesystem.{FileSystem, Location, UploadPartResult, FileEntry, FileSystemUtil}`; `org.apache.doris.filesystem.spi.{ObjFileSystem, ObjStorage, FileSystemProvider}`; `org.apache.doris.filesystem.properties.{StorageProperties, StorageKind}` +- hive-metastore-api: `org.apache.hadoop.hive.metastore.api.FieldSchema` +- hadoop: `org.apache.hadoop.fs.Path` +- 3rd-party: guava (`Lists`, `Iterables`, `ImmutableList`, `Preconditions`, `Verify`), `io.airlift.concurrent.MoreFutures`, thrift `TDeserializer`/`TException`/`protocol.TBinaryProtocol`, log4j. + +**DROP entirely (fe-core, D4/D10)**: `org.apache.doris.common.Pair` (→ use JDK `AbstractMap.SimpleImmutableEntry` or guava), `org.apache.doris.common.profile.SummaryProfile`, `org.apache.doris.qe.ConnectContext`, `org.apache.doris.transaction.Transaction`, `org.apache.doris.datasource.*` (NameMapping/CommonStatistics/statistics), all `org.apache.doris.fs.SpiSwitchingFileSystem`, `org.apache.doris.foundation.util.PathUtils` (use `HiveWriteUtils.pathsEqual`/`isSubDirectory`/`getImmediateChildPath` instead). + +### 2.2 Fields (replacing HMSTransaction's fe-core fields) +| field | type | init | replaces (HEAD) | +|---|---|---|---| +| `LOG` | `static final Logger` | `LogManager.getLogger(HiveConnectorTransaction.class)` | same | +| `transactionId` | `private final long` | ctor | (NEW — SPI id, from `session.allocateTransactionId()`) | +| `hmsClient` | `private final HmsClient` | ctor | `HiveMetadataOps hiveOps` (D10) | +| `context` | `private final ConnectorContext` | ctor | (NEW — `executeAuthenticated` + `getStorageProperties` + `getCatalogId`) | +| `fileSystemExecutor` | `private final Executor` | **created in ctor** (see below) | ctor-injected in HEAD → now plugin-owned | +| `fs` | `private volatile FileSystem` | built lazily in `HmsCommitter` from `context.getStorageProperties()` (D6/§4) | `SpiSwitchingFileSystem fs` (D6) | +| `nameMapping` | `private NameMapping` | set in `beginWrite` | (was passed per-call; now one per txn) | +| `hmsTableInfo` | `private volatile HmsTableInfo` | set in `beginWrite` | `Table` (fe-core hive-api Table) | +| `queryId` | `private String` | `beginWrite` (from ctx) | `beginInsertTable` | +| `isOverwrite` | `private boolean` | `beginWrite` | same | +| `fileType` | `private TFileType` | `beginWrite` | same | +| `stagingDirectory` | `private Optional` | `beginWrite` | same | +| `isMockedPartitionUpdate` | `private boolean` | `false` | same | +| `hivePartitionUpdates` | `private final List` | `Lists.newArrayList()` | same (accumulator) | +| `tableActions` | `private final Map>` | `new HashMap<>()` | key type → plugin `NameMapping` | +| `partitionActions` | `private final Map, Action>>` | `new HashMap<>()` | key → plugin `NameMapping`; inner key `List` partitionValues | +| `uncompletedMpuPendingUploads` | `private final Set` | `new HashSet<>()` | same | +| `hmsCommitter` | `private HmsCommitter` | null | same | +| **DROP** `summaryProfile` | — | — | D4 (no profiling) | +| **DROP** `tableColumns` | — | — | dead in HEAD (declared, never read) | + +**Executor**: HEAD injected `fileSystemExecutor` via ctor. Plugin OWNS it (no fe-core to inject). Create in ctor: `this.fileSystemExecutor = Executors.newFixedThreadPool(N, )` and **shut it down in `close()`** (see §3.8 lifecycle). This is the async-rename/MPU pool. The stats pool (`newFixedThreadPool(16)`) is created inside `HmsCommitter` (D-faithful) and shut in its `shutdownExecutorService()`. + +### 2.3 Constructor (analogue of `IcebergConnectorTransaction(long, IcebergCatalogOps, ConnectorContext)`) +```java +public HiveConnectorTransaction(long transactionId, HmsClient hmsClient, ConnectorContext context) { + this.transactionId = transactionId; + this.hmsClient = hmsClient; + this.context = context; + this.fileSystemExecutor = Executors.newFixedThreadPool(/*N*/16, threadFactory("hive-write-fs-%d")); +} +``` +DROP the HEAD ctor's `ConnectContext.get().getExecutor().getSummaryProfile()` block (D4). + +### 2.4 SPI @Override method set (from `ConnectorTransaction` — abstract 4 + optional overrides) +| method | body | +|---|---| +| `@Override public long getTransactionId()` | `return transactionId;` | +| `@Override public void commit()` | classification + committer (§3.7) | +| `@Override public void rollback()` | D9 (§3.12) — idempotent | +| `@Override public void close()` | shut down `fileSystemExecutor` (and ensure stats pool shut if committer exists); no `throws` (narrows Closeable) | +| `@Override public void addCommitData(byte[] commitFragment)` | deserialize + accumulate (§2.5) | +| `@Override public long getUpdateCnt()` | `return hivePartitionUpdates.stream().mapToLong(THivePartitionUpdate::getRowCount).sum();` (D3) | +| `@Override public String profileLabel()` | `return "HMS";` (D2 — maps to `TransactionType.HMS`; ANY other string → UNKNOWN) | + +**Do NOT override**: `supportsWriteBlockAllocation`, `allocateWriteBlockRange`, `applyWriteConstraint`, `registerRewriteSourceFiles`, `getRewriteAddedDataFilesCount` — keep the interface defaults (hive is not maxcompute/iceberg). Confirmed against the interface: abstract = `{getTransactionId, commit, rollback, close}`; the rest are `default`. + +Non-SPI method (called by INC-4 `HiveWritePlanProvider.planWrite`): +```java +public void beginWrite(ConnectorSession session, String db, String tableName, HiveWriteContext ctx) +``` +(§3.5. Analogue of iceberg `beginWrite`.) + +### 2.5 `addCommitData` — mirror iceberg/HEAD deserialization exactly +```java +@Override +public void addCommitData(byte[] commitFragment) { + THivePartitionUpdate pu = new THivePartitionUpdate(); + try { + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(pu, commitFragment); + } catch (TException e) { + throw new DorisConnectorException("failed to deserialize Hive partition update", e); + } + synchronized (this) { + hivePartitionUpdates.add(pu); + } +} +``` +(HEAD wraps in `RuntimeException`; iceberg/maxcompute use `DorisConnectorException` — use `DorisConnectorException` to match the SPI template. Accumulation = single `List` synced on `this`.) + +--- + +## 3. `HiveConnectorTransaction` — CORE + +### 3.1 Enums + `Action` (HEAD lines 912–957, copy verbatim, plugin-safe) +```java +private enum ActionType { DROP, DROP_PRESERVE_DATA, ADD, ALTER, INSERT_EXISTING, MERGE } + +public static class Action { + private final ActionType type; + private final T data; + public Action(ActionType type, T data) { + this.type = type; + if (type == ActionType.DROP || type == ActionType.DROP_PRESERVE_DATA) { + Preconditions.checkArgument(data == null, "data not null"); + } else { + this.data = requireNonNull(data, "data is null"); // requireNonNull + } + this.type = type; this.data = data; // (match HEAD exact assignment) + } + public ActionType getType() { return type; } + public T getData() { Preconditions.checkState(type != ActionType.DROP, ...); return data; } +} +``` +`MERGE` is declared but never produced (only appears in switch default-throw branches). Keep for faithfulness. + +### 3.2 Classification maps — keyed by plugin `NameMapping` +- `tableActions: Map>` +- `partitionActions: Map, Action>>` (inner key = `HiveUtil.toPartitionValues(partitionName)` — verify a plugin `toPartitionValues` exists in fe-connector-hive; if not, port the pure `name=val/...` split. Reader flagged HiveUtil as "verify it carries `toPartitionValues` + `convertToNamePartitionMap`"). + +`NameMapping` (INC-2, `public final class`): ctor `NameMapping(long ctlId, String localDbName, String localTblName, String remoteDbName, String remoteTblName)`; getters `getCtlId()/getLocalDbName/getLocalTblName/getRemoteDbName/getRemoteTblName/getFullLocalName/getFullRemoteName`; `equals`/`hashCode` over all 5 fields. Build **one** per transaction in `beginWrite`: +```java +this.nameMapping = new NameMapping(context.getCatalogId(), db, tableName, db, tableName); +``` +`ctlId = context.getCatalogId()` (**both `ConnectorContext.getCatalogId()` and `ConnectorSession.getCatalogId()` exist — GAP-6 dissolved**). Local names set == remote (HMS local==remote; only `getFullLocalName` uses local, for logging). HMS calls use `getRemoteDbName()/getRemoteTblName()`. + +### 3.3 `beginWrite` (non-SPI; template = iceberg `beginWrite`; folds HEAD `beginInsertTable` + table load + D7 guard) +```java +public void beginWrite(ConnectorSession session, String db, String tableName, HiveWriteContext ctx) { + this.queryId = ctx.getQueryId(); + this.isOverwrite = ctx.isOverwrite(); + this.fileType = ctx.getFileType(); + this.stagingDirectory = (fileType == TFileType.FILE_S3) + ? Optional.empty() : Optional.of(ctx.getWritePath()); + this.nameMapping = new NameMapping(context.getCatalogId(), db, tableName, db, tableName); + try { + context.executeAuthenticated(() -> { // D5 — auth-wrap the metastore load + HmsTableInfo t = hmsClient.getTable(db, tableName); + rejectFullAcidWrite(t.getParameters()); // D7 — see §3.11 + this.hmsTableInfo = t; + return null; + }); + } catch (Exception e) { + throw new DorisConnectorException("Failed to begin write for hive table " + tableName + ": " + e.getMessage(), e); + } +} +``` +(HEAD `beginInsertTable` reads only queryId/isOverwrite/fileType/stagingDirectory. The table load + full-acid guard are pulled forward here because that is the only pre-commit point with the table — see §3.11.) + +### 3.4 `finishInsertTable` (HEAD 216–339) — runs INSIDE `commit()` (§3.7), after all `addCommitData` +Uses accumulated `hivePartitionUpdates` + `nameMapping` + `hmsTableInfo`. Reshaped fe-core→plugin: +1. `HmsTableInfo table = getTable(nameMapping)` (§3.6 helper; returns `HmsTableInfo`, NOT hive-api Table). +2. **Mocked-overwrite path**: if `hivePartitionUpdates.isEmpty() && isOverwrite && table.getPartitionKeys().isEmpty()` → set `isMockedPartitionUpdate = true`; build one empty `THivePartitionUpdate` (OVERWRITE, size 0, rows 0, empty fileNames). S3: attach one empty `TS3MPUPendingUpload` + `location.writePath = table.getLocation()`. non-S3: `fs.mkdirs(Location.of(stagingDir))` then `location.writePath = stagingDir`. **(`table.getPartitionKeysSize()==0` → `table.getPartitionKeys().isEmpty()`; `table.getSd().getLocation()` → `table.getLocation()`.)** +3. `mergedPUs = HiveWriteUtils.mergePartitions(hivePartitionUpdates)` (INC-2 static; sums fileSize/rowCount, unions s3MpuPendingUploads + fileNames). +4. `collectUncompletedMpuPendingUploads(mergedPUs)` (port pure helper; fills `uncompletedMpuPendingUploads`). +5. Per PU: `HmsPartitionStatistics.fromCommonStatistics(rowCount, fileNamesSize, fileSize)` **(GAP-9: order rowCount, fileCount, totalBytes — fileCount is MIDDLE)**; `writePath = pu.getLocation().getWritePath()`. +6. **Unpartitioned** (`table.getPartitionKeys().isEmpty()`): assert exactly 1 PU. `APPEND`→`finishChangingExistingTable(INSERT_EXISTING,...)`; `OVERWRITE`→`dropTable(nm)` then `createTable(nm, table, ...)` (net action = ALTER via the DROP→ALTER transition); else throw. +7. **Partitioned**: per PU by `pu.getUpdateMode()`: + - `APPEND` → add to `insertExistsPartitions`. + - `NEW` → **NEW→APPEND downgrade probe**: `partitionValues = toPartitionValues(pu.getName())`; `if (hmsClient.partitionExists(nm.getRemoteDbName(), nm.getRemoteTblName(), partitionValues))` → treat as APPEND (`insertExistsPartitions.add`); else `createAndAddPartition(..., dropFirst=false)`. Empty/null name → warn+skip. **(HEAD used `hiveOps.getClient().getPartition(...)!=null`; replace with the not-found-tolerant `partitionExists` primitive — INC-1 wrote it fresh to not taint the pooled client.)** + - `OVERWRITE` → `createAndAddPartition(..., dropFirst=true)`. Empty name → warn+skip. + - else throw. +8. If `insertExistsPartitions` non-empty → `convertToInsertExistingPartitionAction(nameMapping, insertExistsPartitions)`. + +**`convertToInsertExistingPartitionAction`** (HEAD 430–513): `Iterables.partition(partitions, 100)`; guard pre-existing partitionAction (DROP→throw "Not found"; ADD/ALTER/INSERT_EXISTING/MERGE→throw "Inserting into a partition that were added…not supported"); fetch real partitions via `hmsClient.getPartitions(nm.getRemoteDbName(), nm.getRemoteTblName(), partitionNames)` → `List`; null/missing → throw "Not found partition from hms"; build the INSERT_EXISTING `PartitionAndMore` from `HmsPartitionInfo` (targetPath = `hmsPartitionInfo.getLocation()`; **GAP-5: `HmsPartitionInfo.getSerializationLib()` if later reshaped to write DTO → `.serde(...)`**; but INSERT_EXISTING does not addPartition, so cols/formats are not needed here — only targetPath/partitionValues/stats). Store `Action(INSERT_EXISTING, PartitionAndMore(...))` keyed by partitionValues. **Carry HEAD's map/list index quirk as-is** (`partitionsByNamesMap.size()` bound vs `partitionNames.get(i)`). + +### 3.5 State-transition helpers (all `synchronized`, HEAD 959–1191, fe-core→plugin) +- `getTable(NameMapping)` → return tableAction data table (now `HmsTableInfo`) or `hmsClient.getTable(nm.getRemoteDbName(), nm.getRemoteTblName())`; DROP/DROP_PRESERVE_DATA→throw. **Return type `HmsTableInfo`, not hive-api `Table`.** +- `finishChangingExistingTable(ActionType, NameMapping, String location, List fileNames, HmsPartitionStatistics, THivePartitionUpdate)` → null→`Action(actionType, new TableAndMore(table, location, fileNames, stats, pu))`; DROP→throw; ADD/ALTER/INSERT_EXISTING/MERGE→throw "not supported"; DROP_PRESERVE_DATA→fallthrough. +- `createTable(NameMapping, HmsTableInfo, String location, List fileNames, HmsPartitionStatistics, THivePartitionUpdate)` → `checkNoPartitionAction`; null→ADD; prior DROP→ALTER; else throw. +- `dropTable(NameMapping)` → `checkNoPartitionAction`; null|ALTER→DROP; DROP→throw; ADD/ALTER/INSERT_EXISTING/MERGE→throw. +- `checkNoPartitionAction(NameMapping)` → throw if partitionActions non-empty for table. +- `createAndAddPartition(NameMapping, HmsTableInfo, List partitionValues, String writePath, THivePartitionUpdate pu, HmsPartitionStatistics, boolean dropFirst)` → `pathForHMS = (fileType==FILE_S3) ? writePath : pu.getLocation().getTargetPath()`; build `PartitionAndMore` (inline: partitionValues + targetPath=pathForHMS + currentLocation=writePath + partitionName + fileNames + stats + pu); if `dropFirst`→`dropPartition(nm, partitionValues, deleteData=true)`; then `addPartition(nm, partitionAndMore, ...)`. +- `addPartition(NameMapping, PartitionAndMore, ...)` → null→ADD; DROP/DROP_PRESERVE_DATA→ALTER; else throw "already exists". +- `dropPartition(NameMapping, List partitionValues, boolean deleteData)` → null→ DROP (deleteData) or DROP_PRESERVE_DATA; DROP*→throw; ADD/ALTER/INSERT_EXISTING/MERGE→throw. + +### 3.6 Descriptor reshaping — `TableAndMore` / `PartitionAndMore` (§4 requirement; design §6 "reshape onto HmsPartitionWithStatistics + inline; do NOT copy fe-core HivePartition") +**`TableAndMore`** (`private static class`): +```java +private final HmsTableInfo table; // was hive-api Table +private final String currentLocation; +private final List fileNames; +private final HmsPartitionStatistics statisticsUpdate; +private final THivePartitionUpdate hivePartitionUpdate; // public thrift field s3_mpu_pending_uploads accessed directly +// ctor requireNonNull all; getters getTable/getCurrentLocation/getFileNames/getStatisticsUpdate/getHivePartitionUpdate +``` +Committer reads `tableAndMore.getTable().getLocation()` as targetPath (was `.getSd().getLocation()`), and `hivePartitionUpdate.getS3MpuPendingUploads()` for MPU. + +**`PartitionAndMore`** (`private static class`) — **HivePartition field collapsed to inline fields** (do NOT copy fe-core `HivePartition`; its `Column`-dragging `getPartitionName(List)` is dead on write, verified INC-2): +```java +private final List partitionValues; // was partition.getPartitionValues() +private final String targetPath; // was partition.getPath() (= pathForHMS for ADD, existing loc for INSERT_EXISTING) +private final String currentLocation; // writePath (staging) +private final String partitionName; +private final List fileNames; +private final HmsPartitionStatistics statisticsUpdate; +private final THivePartitionUpdate hivePartitionUpdate; +// getters getPartitionValues/getTargetPath/getCurrentLocation/getPartitionName/getFileNames/getStatisticsUpdate/getHivePartitionUpdate +``` +For the **ADD** case, the committer builds the `HmsPartitionWithStatistics` at commit time in `prepareAddPartition` from these inline fields + the table's cols/formats (re-fetched via `getTable(nm)` → `HmsTableInfo`; see §5.2 for ConnectorColumn→FieldSchema). This mirrors HEAD's "rebuild partition SD from table SD at commit." + +### 3.7 `commit()` ordered flow (HEAD `commit()`→`doCommit()` 148–397) +```java +@Override +public void commit() { + try { + // 1. classification (finishInsertTable) — HEAD ran this from the executor; here it runs at commit + finishInsertTable(nameMapping); // populates tableActions / partitionActions + + hmsCommitter = new HmsCommitter(); + // 2. dispatch table actions + for (Map.Entry> e : tableActions.entrySet()) { + switch (e.getValue().getType()) { + case INSERT_EXISTING: hmsCommitter.prepareInsertExistingTable(e.getKey(), e.getValue().getData()); break; + case ALTER: hmsCommitter.prepareAlterTable(e.getKey(), e.getValue().getData()); break; + default: throw new UnsupportedOperationException(...); + } + } + // 3. dispatch partition actions + for (Map.Entry, Action>> te : partitionActions.entrySet()) { + for (Map.Entry, Action> pe : te.getValue().entrySet()) { + switch (pe.getValue().getType()) { + case INSERT_EXISTING: hmsCommitter.prepareInsertExistPartition(te.getKey(), pe.getValue().getData()); break; + case ADD: hmsCommitter.prepareAddPartition(te.getKey(), pe.getValue().getData()); break; + case ALTER: hmsCommitter.prepareAlterPartition(te.getKey(), pe.getValue().getData()); break; + default: throw new UnsupportedOperationException(...); + } + } + } + // 4. + hmsCommitter.doCommit(); + } catch (Throwable t) { + LOG.warn("Rolling back due to commit failure", t); + try { hmsCommitter.abort(); hmsCommitter.rollback(); } + catch (RuntimeException e) { t.addSuppressed(e); } + throw t; // (wrap non-RuntimeException in DorisConnectorException if needed to satisfy commit() unchecked contract) + } finally { + if (hmsCommitter != null) { + hmsCommitter.runClearPathsForFinish(); + hmsCommitter.shutdownExecutorService(); + } + } +} +``` +Note: `commit()` throws only unchecked. HEAD rethrows `Throwable t`; ensure the final escape is a `RuntimeException`/`DorisConnectorException`. (Iceberg wraps its whole commit body in `try/catch(Exception)→DorisConnectorException` — you may do the same outer wrap, but preserve the abort→rollback-on-failure ordering above.) The metastore calls inside `doAddPartitionsTask`/`doUpdateStatisticsTasks` are auth'd for free inside `ThriftHmsClient.execute()` (D5) — no extra `executeAuthenticated` needed for them; only the fs/rename/MPU tasks need wrapping (§3.10). + +### 3.8 `HmsCommitter` (inner NON-static class, HEAD 1192–1651) — method-by-method with REAL INC-1 calls +**Fields**: +```java +final List updateStatisticsTasks = new ArrayList<>(); +ExecutorService updateStatisticsExecutor = Executors.newFixedThreadPool(16); // STATS pool — created at HmsCommitter ctor, shut in shutdownExecutorService() +final AddPartitionsTask addPartitionsTask = new AddPartitionsTask(); +final AtomicBoolean fileSystemTaskCancelled = new AtomicBoolean(false); +final List> asyncFileSystemTaskFutures = new ArrayList<>(); +final Queue directoryCleanUpTasksForAbort = new ConcurrentLinkedQueue<>(); +final List renameDirectoryTasksForAbort = new ArrayList<>(); +final List clearDirsForFinish = new ArrayList<>(); +final List s3cleanWhenSuccess = new ArrayList<>(); +``` +The **async-rename Executor is the outer `fileSystemExecutor`** (plugin-owned, created in the transaction ctor; NOT owned by HmsCommitter). Tasks submit via `FileSystemUtil.asyncRenameFiles/asyncRenameDir(fs, fileSystemExecutor, asyncFileSystemTaskFutures, fileSystemTaskCancelled, orig, dest, ...)` and via `CompletableFuture.runAsync(..., fileSystemExecutor)`. Awaited via `MoreFutures.getFutureValue(future, RuntimeException.class)`. + +**Executor lifecycle**: STATS pool (`updateStatisticsExecutor`) — created at `HmsCommitter` construction, `shutdownExecutorService()` (HEAD 1631): `shutdown()`; `awaitTermination(60s)`; `shutdownNow()`; `awaitTermination(60s)`; interrupt-handling. Async-rename pool (`fileSystemExecutor`) — created in txn ctor, shut in `HiveConnectorTransaction.close()`. + +**Methods** (each `wrapper*WithProfileSummary` collapses to its plain body per D4 — drop all `summaryProfile.ifPresent(...)`): +- `cancelUnStartedAsyncFileSystemTask()` → `fileSystemTaskCancelled.set(true)`. +- `undoUpdateStatisticsTasks()` → per task `CompletableFuture.runAsync(() -> task.undo(hmsClient, nameMapping), updateStatisticsExecutor)`; await all; clear. +- `undoAddPartitionsTask()` → if empty return; else `addPartitionsTask.rollback(hmsClient)`; clear. +- `waitForAsyncFileSystemTaskSuppressThrowable()` → `future.get()` swallowing throwables; clear. +- `prepareInsertExistingTable(nm, TableAndMore)` → `targetPath = tableAndMore.getTable().getLocation()`, `writePath = currentLocation`; `needRename = !HiveWriteUtils.pathsEqual(targetPath, writePath)` (HEAD used `PathUtils.equalsIgnoreSchemeIfOneIsS3`; `HiveWriteUtils.pathsEqual` is the plugin port — verify equivalence, else port `equalsIgnoreSchemeIfOneIsS3` into HiveWriteUtils); if needRename→async rename files; else if `getHivePartitionUpdate().getS3MpuPendingUploads()` non-empty→`objCommit(...)`; add `DirectoryCleanUpTask(targetPath, false)`; add `UpdateStatisticsTask(nm, Optional.empty(), stats, merge=true)`. +- `prepareAlterTable(nm, TableAndMore)` → HEAD 1302 legacy/sub-dir branch (uses `HiveWriteUtils.isSubDirectory`/`getImmediateChildPath` + temp rename `_temp__` recorded in `renameDirectoryTasksForAbort`/`clearDirsForFinish`/`DirectoryCleanUpTask(targetPath,true)`); else s3 mpu → `s3cleanWhenSuccess.add(targetPath)`+`objCommit`; `UpdateStatisticsTask(..., merge=false)`. +- `prepareAddPartition(nm, PartitionAndMore)` → `targetPath = partitionAndMore.getTargetPath()`; if `!pathsEqual(targetPath, writePath)`→`DirectoryCleanUpTask(targetPath,true)`+`FileSystemUtil.asyncRenameDir(...)`; else s3 mpu→objCommit. **Build `HmsPartitionWithStatistics`** (§5.2) from inline fields + `getTable(nm)` cols/formats/serde + targetPath location; `addPartitionsTask.addPartition(nm, hmsPartitionWithStatistics)`. +- `prepareInsertExistPartition(nm, PartitionAndMore)` → `DirectoryCleanUpTask(targetPath,false)`; if `!pathsEqual`→async rename files; else s3 mpu→objCommit; `UpdateStatisticsTask(nm, Optional.of(partitionName), stats, merge=true)`. +- `prepareAlterPartition(nm, PartitionAndMore)` → mirror `prepareAlterTable` legacy branch; `UpdateStatisticsTask(nm, Optional.of(partitionName), stats, merge=false)`. +- `runDirectoryClearUpTasksForAbort()` → per task `recursiveDeleteItems(path, deleteEmptyDir, reverse=false)`; clear. +- `runRenameDirTasksForAbort()` → if `fs.exists(Location.of(from))`→`fs.renameDirectory(Location.of(from), Location.of(to), noop)`; clear. +- `runClearPathsForFinish()` → per path `fs.delete(Location.of(path), true)` (was `wrapperDeleteDirWithProfileSummary`). +- `runS3cleanWhenSuccess()` → per path `recursiveDeleteItems(new Path(path), false, reverse=true)`. +- `waitForAsyncFileSystemTasks()` → await all futures via `MoreFutures.getFutureValue(future, RuntimeException.class)`. +- `doAddPartitionsTask()` → if `!addPartitionsTask.isEmpty()` `addPartitionsTask.run(hmsClient)`. **NO committer-side batching** — `Lists.partition(...,20)` is INSIDE `ThriftHmsClient.addPartitions` (INC-1). The task calls `hmsClient.addPartitions(nm.getRemoteDbName(), nm.getRemoteTblName(), allPartitions)` ONCE. +- `doUpdateStatisticsTasks()` → per task `CompletableFuture.runAsync(() -> task.run(hmsClient, nameMapping), updateStatisticsExecutor)`; collect suppressed (cap 5); await; throw aggregated `RuntimeException` if any. +- `pruneAndDeleteStagingDirectories()` → `stagingDirectory.ifPresent(v -> recursiveDeleteItems(new Path(v), true, false))`. +- `abortMultiUploads()` — §3.9. +- `doCommit()` → `waitForAsyncFileSystemTasks(); runS3cleanWhenSuccess(); doAddPartitionsTask(); doUpdateStatisticsTasks(); pruneAndDeleteStagingDirectories();`. +- `abort()` → `cancelUnStartedAsyncFileSystemTask(); undoUpdateStatisticsTasks(); undoAddPartitionsTask(); waitForAsyncFileSystemTaskSuppressThrowable(); runDirectoryClearUpTasksForAbort(); runRenameDirTasksForAbort();`. +- `rollback()` → `pruneAndDeleteStagingDirectories(); abortMultiUploads();` await+clear `asyncFileSystemTaskFutures`. +- `shutdownExecutorService()` → shut `updateStatisticsExecutor` (16-pool). + +**Task classes** (top-level `private static`, HmsClient/NameMapping-parameterized): +- `UpdateStatisticsTask`: fields `NameMapping nameMapping; Optional partitionName; HmsPartitionStatistics updatePartitionStat; boolean merge; boolean done`. + - `run(HmsClient c, NameMapping nm)` → `if (partitionName.isPresent()) c.updatePartitionStatistics(nm.getRemoteDbName(), nm.getRemoteTblName(), partitionName.get(), this::updateStatistics); else c.updateTableStatistics(nm.getRemoteDbName(), nm.getRemoteTblName(), this::updateStatistics);` set done. + - `undo(HmsClient c, NameMapping nm)` → mirror with `this::resetStatistics`. + - `updateStatistics(HmsPartitionStatistics cur)` → `merge ? HmsPartitionStatistics.merge(cur, updatePartitionStat) : updatePartitionStat`. + - `resetStatistics(HmsPartitionStatistics cur)` → `HmsPartitionStatistics.reduce(cur, updatePartitionStat, HmsCommonStatistics.ReduceOperator.SUBTRACT)`. **(GAP-2: `ReduceOperator` lives on `HmsCommonStatistics`, not `HmsPartitionStatistics`.)** + - `getDescription()` uses `nameMapping.getFullLocalName()`/`getFullRemoteName()`. + - **Signature note**: `HmsClient.updateTableStatistics(String db, String tbl, Function)` and `updatePartitionStatistics(String db, String tbl, String partitionName, Function<...>)` — take **(db, tbl) strings**, NOT NameMapping. `this::updateStatistics`/`this::resetStatistics` are the `Function` args. +- `AddPartitionsTask`: fields `List partitions; List> createdPartitionValues; NameMapping nameMapping` (store nm — HmsPartitionWithStatistics has no NameMapping, unlike HEAD's HivePartition). + - `addPartition(NameMapping nm, HmsPartitionWithStatistics p)` → set nameMapping, add p. + - `run(HmsClient c)` → `c.addPartitions(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), partitions);` then record all `p.getPartitionValues()` into createdPartitionValues. + - `rollback(HmsClient c)` → per created value `c.dropPartition(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), values, false)`; collect failures. +- `DirectoryCleanUpTask` (ctor `(String path, boolean deleteEmptyDir)` → `new Path(path)`), `DeleteRecursivelyResult`, `RenameDirectoryTask` (`String renameFrom; renameTo`), `UncompletedMpuPendingUpload` (`final TS3MPUPendingUpload s3MPUPendingUpload; final String path;` + equals/hashCode over both). + +**FS-walk helpers** (outer class, all use `fs`, port HEAD verbatim with `Location.of` + `fs.delete/exists/listFiles/listDirectories`): `recursiveDeleteItems`, `recursiveDeleteFiles`, `doRecursiveDeleteFiles` (uses `queryId` prefix + `reverse ^ fileName.startsWith(queryId)` selector), `deleteIfExists`, `deleteDirectoryIfExists`, `deleteTargetPathContents`, `ensureDirectory`. + +### 3.9 D6 — MPU complete/abort recipe (EXACT plugin-side signatures) +**(a) Build the FileSystem** (replace `SpiSwitchingFileSystem` field + `.forPath(path)`). Build ONCE lazily, hold on `fs`, close in `HiveConnectorTransaction.close()`: +```java +StorageProperties objSp = context.getStorageProperties().stream() // List + .filter(sp -> sp.kind() == StorageKind.OBJECT_STORAGE) // org.apache.doris.filesystem.properties.StorageKind + .findFirst().orElseThrow(() -> new DorisConnectorException("no object-store StorageProperties for MPU")); +FileSystem fs = resolveObjectStoreFileSystem(objSp); // ISOLATE discovery behind ONE method (see §8 risk) +// resolveObjectStoreFileSystem: iterate ServiceLoader, first p.supports(objSp.rawProperties()) +// → p.create(objSp.rawProperties()) (yields an ObjFileSystem for object stores; S3FileSystem extends ...ObjFileSystem) +``` +`ObjFileSystem.completeMultipartUpload(String, String, Map) throws IOException`; `TS3MPUPendingUpload.getEtags()` returns `Map` (CONFIRMED) → the map overload. Single-backend hive write → one FS, no per-path `forPath` routing needed. Staging renames/deletes use this same `fs`. + +**(b) `instanceof ObjFileSystem` guard** (`org.apache.doris.filesystem.spi.ObjFileSystem`): +```java +if (!(fs instanceof ObjFileSystem)) { + throw new RuntimeException("Expected ObjFileSystem for MPU at '" + path + "', got: " + fs.getClass().getSimpleName()); +} +ObjFileSystem objFs = (ObjFileSystem) fs; +``` +(abort block keeps HEAD's soft skip: `if (!(fs instanceof ObjFileSystem)) { LOG.warn(...); continue; }`.) + +**(c) `objCommit` — completeMultipartUpload** (HEAD 1860–1894). Signature: +```java +private void objCommit(Executor fileSystemExecutor, List> asyncFileSystemTaskFutures, + AtomicBoolean fileSystemTaskCancelled, THivePartitionUpdate hivePartitionUpdate, String path) +``` +Body: `if (isMockedPartitionUpdate) return;` (guard BEFORE resolving fs); resolve/cast `objFs`; per `TS3MPUPendingUpload s3mpu` in `hivePartitionUpdate.getS3MpuPendingUploads()`: +```java +CompletableFuture f = CompletableFuture.runAsync(() -> { + if (fileSystemTaskCancelled.get()) return; + String remotePath = "s3://" + s3mpu.getBucket() + "/" + s3mpu.getKey(); + try { + context.executeAuthenticated(() -> { // D5 wrap + objFs.completeMultipartUpload(remotePath, s3mpu.getUploadId(), s3mpu.getEtags()); + return null; + }); + } catch (Exception e) { throw new RuntimeException(e); } + uncompletedMpuPendingUploads.remove(new UncompletedMpuPendingUpload(s3mpu, path)); +}, fileSystemExecutor); +asyncFileSystemTaskFutures.add(f); +``` + +**(d) `abortMultiUploads` — abortMultipartUpload** (HEAD 1564–1594): +```java +if (uncompletedMpuPendingUploads.isEmpty()) return; +for (UncompletedMpuPendingUpload u : uncompletedMpuPendingUploads) { + if (!(fs instanceof ObjFileSystem)) { LOG.warn(...); continue; } + ObjFileSystem objFs = (ObjFileSystem) fs; + TS3MPUPendingUpload mpu = u.s3MPUPendingUpload; + String remotePath = "s3://" + mpu.getBucket() + "/" + mpu.getKey(); + CompletableFuture f = CompletableFuture.runAsync(() -> { + try { + context.executeAuthenticated(() -> { // D5 wrap + objFs.getObjStorage().abortMultipartUpload(remotePath, mpu.getUploadId()); // ObjStorage.abortMultipartUpload(String,String) + return null; + }); + } catch (Exception e) { LOG.warn("abort mpu failed", e); } + }, fileSystemExecutor); + asyncFileSystemTaskFutures.add(f); +} +uncompletedMpuPendingUploads.clear(); +``` +Keep the raw `"s3://" + bucket + "/" + key` concatenation (HEAD does exactly this; no `ObjectStorageUri` object). + +### 3.10 D5 — `executeAuthenticated` wrap points (exactly these; metastore calls do NOT need it) +- `beginWrite`: the `hmsClient.getTable` load (metastore self-auths, but the load happens before commit and iceberg wraps its load — wrapping is harmless and matches template; optional since `ThriftHmsClient.execute` self-pins). +- Every **async fs/rename/MPU task body** on `fileSystemExecutor` and `updateStatisticsExecutor` threads: wrap `objFs.completeMultipartUpload` / `objFs.getObjStorage().abortMultipartUpload` / `fs.rename` / `fs.renameDirectory` / `fs.delete` in `context.executeAuthenticated(() -> {...; return null;})` inside a `try/catch(Exception e)` (checked `throws Exception`). **Reason**: plugin-owned pool threads don't inherit the caller pin (memory `catalog-spi-plugin-tccl-classloader-gotcha` locus #2/#3). +- `FileSystemUtil.asyncRenameFiles/asyncRenameDir` wrap NO auth internally → for a Kerberos catalog either (i) reimplement the async loop with `context.executeAuthenticated` inside each `runAsync`, or (ii) hand them an auth-wrapping `Executor`. **Do NOT** copy iceberg's JVM-wide `pinIcebergWorkerPoolToPluginClassLoader` primer (D5 — hive fans onto its own pool). +- **NOT wrapped**: `hmsClient.addPartitions/updateTableStatistics/updatePartitionStatistics/dropPartition/partitionExists/getTable/getPartitions/partitionExists` — `ThriftHmsClient.execute()` already does `doAs` + system-CL TCCL pin (D5). + +### 3.11 D7 — full-acid-WRITE reject location +No begin-guard exists in HEAD. The only pre-commit point with the table is the load. **Locus**: `rejectFullAcidWrite(Map tableParameters)` called inside `beginWrite` right after `hmsClient.getTable` (§3.3). Derive `isTransactional`/`isFullAcid` plugin-side from the raw HMS params (D8) using `HmsAcidConstants` keys (`transactional`, `transactional_properties`) — mirror `HiveConnectorMetadata.isTransactionalTable(Map)` (private static, already exists — the `transactional` half) and add the `transactional_properties`==`insert_only`? / full-acid half. Throw `DorisConnectorException("Cannot write to a full-ACID Hive table ...")` for full-acid tables. (Full-acid READ stays in scope — INC-5 — do not conflate.) `HiveTableHandle.getTableParameters()` also exposes these params for INC-4's begin-guard; INC-3's guard uses `HmsTableInfo.getParameters()`. + +### 3.12 D9 — `rollback()` (NOT a no-op; deletes staging + aborts MPU) +```java +@Override +public void rollback() { + if (hmsCommitter == null) { + collectUncompletedMpuPendingUploads(hivePartitionUpdates); // fill the set from raw updates + if (uncompletedMpuPendingUploads.isEmpty()) return; + hmsCommitter = new HmsCommitter(); + try { hmsCommitter.rollback(); } finally { hmsCommitter.shutdownExecutorService(); } + } else { + try { hmsCommitter.abort(); hmsCommitter.rollback(); } finally { hmsCommitter.shutdownExecutorService(); } + } +} +``` +`HmsCommitter.rollback()` = `pruneAndDeleteStagingDirectories()` (delete staging files/dirs via `fs.delete(Location, recursive)`) + `abortMultiUploads()` (§3.9d) + await/clear futures. Idempotent-safe (guard on `hmsCommitter==null`; empty-set early return). `close()` must also cover the async-rename pool shutdown. + +--- + +## 4. EVERY fe-core COUPLING → PLUGIN REPLACEMENT (consolidated) +| # | HEAD fe-core symbol | usage | plugin replacement | +|---|---|---|---| +| 1 | `ConnectContext` (ctor) | fetch SummaryProfile | **DROP** (D4); queryId via `HiveWriteContext` | +| 2 | `SummaryProfile` / `summaryProfile` (~15 sites) | profile timings | **DROP** all `summaryProfile.ifPresent(...)` (D4); collapse each `wrapper*WithProfileSummary` to plain body | +| 3 | `HiveMetadataOps hiveOps` (+`.getClient()`) | getTable/getPartition/getPartitions/updateTable+PartitionStatistics/addPartitions/dropPartition | `HmsClient` (D10). Read: `getTable`→`HmsTableInfo`, `getPartitions`→`List`; write: 5 primitives (db,tbl strings) | +| 4 | `HMSCachedClient` | via `hiveOps.getClient()` | folded into `HmsClient` | +| 5 | `HivePartition` (built 3×) | partitionValues/path/SD | **inline fields in PartitionAndMore** + `HmsPartitionWithStatistics` for addPartition (do NOT copy) | +| 6 | `HivePartitionStatistics` | fromCommonStatistics/merge/reduce | `HmsPartitionStatistics` (fromCommonStatistics(rowCount,fileCount,bytes)/merge/reduce) | +| 7 | `HivePartitionWithStatistics` | ctor(name,partition,stats) | `HmsPartitionWithStatistics` **builder** (`.name/.partitionValues/.location/.columns/.inputFormat/.outputFormat/.serde/.parameters/.statistics/.build()`) | +| 8 | `CommonStatistics.ReduceOperator.SUBTRACT` | reduce op | `HmsCommonStatistics.ReduceOperator.SUBTRACT` (GAP-2) | +| 9 | `NameMapping` (fe-core datasource) | map keys + remote/full names | `org.apache.doris.connector.hive.NameMapping` (INC-2) | +| 10 | `HiveInsertCommandContext` | beginInsertTable param | `HiveWriteContext` (INC-3) | +| 11 | `HiveUtil.toPartitionValues/convertToNamePartitionMap` | partition name parse | plugin HiveUtil (verify presence in fe-connector-hive; else port pure split) | +| 12 | `FileSystem` (org.apache.doris.filesystem, fe-core alias in HEAD) | mkdirs/exists/list/delete/rename | `org.apache.doris.filesystem.FileSystem` (fe-filesystem-api, already on CP) via `Location.of` | +| 13 | `SpiSwitchingFileSystem` (+`.forPath`) | per-path FS + MPU downcast | plugin-built `FileSystem` from `context.getStorageProperties()` via `FileSystemProvider` (D6, §3.9a) | +| 14 | `ObjFileSystem`/`ObjStorage` | complete/abort MPU | `org.apache.doris.filesystem.spi.{ObjFileSystem,ObjStorage}` (add `fe-filesystem-spi` provided) | +| 15 | `FileSystemUtil.asyncRenameFiles/asyncRenameDir` | async rename | `org.apache.doris.filesystem.FileSystemUtil` (fe-filesystem-api, already on CP) — but wrap auth (§3.10) | +| 16 | `PathUtils.equalsIgnoreSchemeIfOneIsS3` | path compare | `HiveWriteUtils.pathsEqual` (INC-2) — **verify semantic equivalence**, else port the S3-scheme-tolerant compare into HiveWriteUtils | +| 17 | `FileEntry`/`Location` | file uri | `org.apache.doris.filesystem.{FileEntry,Location}` (already on CP) | +| 18 | `Transaction` (implements) | commit/rollback/addCommitData/getUpdateCnt | `implements ConnectorTransaction` | +| 19 | `Pair` (common) | Pair.of/first/second | JDK `Map.Entry`/`SimpleImmutableEntry` or guava | +| 20 | thrift cluster | keep | `org.apache.doris.thrift.*` unchanged (D11) | + +--- + +## 5. `beginTransaction` WIRING + COLUMN CONVERSION + +### 5.1 `HiveConnectorMetadata.beginTransaction` (one-liner, D1/§4.3) +```java +@Override +public ConnectorTransaction beginTransaction(ConnectorSession session) { + return new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient, context); +} +``` +`session.allocateTransactionId()` (engine-global id, becomes `getTransactionId()` + the txn-registry key — MUST NOT mint own). `hmsClient` + `context` are metadata's existing final fields (lines 79/86). Overrides the `ConnectorWriteOps.beginTransaction` default. NO separate ConnectorWriteOps object. + +### 5.2 ConnectorColumn → FieldSchema (GAP-4 — needed for `HmsPartitionWithStatistics.columns(...)` on ADD) +`HmsClient.getTable` → `HmsTableInfo`; `HmsTableInfo.getColumns()` returns `List`, but `HmsPartitionWithStatistics.columns(...)` needs `List` (consumed by `HmsWriteConverter.toHivePartitionStorageDesc` → `sd.setCols`). The existing `HmsWriteConverter.toHiveSchema(List, Set)` is **private**. Two options: +- **(recommended, inline)** In `prepareAddPartition`, convert via the public `HmsTypeMapping.toHiveTypeString(ConnectorType)`: + ```java + List cols = table.getColumns().stream() + .map(c -> new FieldSchema(c.getName(), HmsTypeMapping.toHiveTypeString(c.getType()), c.getComment())) + .collect(Collectors.toList()); + ``` +- (alt) add a public `HmsWriteConverter.toFieldSchemas(List)` (touches fe-connector-hms — still inside the atomic batch). +Then: +```java +HmsPartitionWithStatistics p = HmsPartitionWithStatistics.builder() + .name(partitionAndMore.getPartitionName()) + .partitionValues(partitionAndMore.getPartitionValues()) + .location(partitionAndMore.getTargetPath()) + .columns(cols) + .inputFormat(table.getInputFormat()) + .outputFormat(table.getOutputFormat()) + .serde(table.getSerializationLib()) // GAP-5: HmsTableInfo/HmsPartitionInfo use getSerializationLib() → .serde(...) + .parameters() + .statistics(partitionAndMore.getStatisticsUpdate()) + .build(); +``` +(Only the **ADD** case needs columns; INSERT_EXISTING/ALTER do not addPartition.) + +--- + +## 6. TEST PLAN — `HiveConnectorTransactionTest` (JUnit5, NO Mockito; recording-fake `HmsClient` + fake `FileSystem`) +Recording-fake `HmsClient`: implement the interface (all read methods + the 5 write primitives), record calls (`addPartitions`/`updateTableStatistics`/`updatePartitionStatistics`/`dropPartition`/`partitionExists`/`getTable`/`getPartitions`), return canned `HmsTableInfo`/`HmsPartitionInfo`. Fake `FileSystem`: implement `org.apache.doris.filesystem.FileSystem` recording `rename`/`renameDirectory`/`delete`/`mkdirs`/`exists`; for MPU tests provide a fake `ObjFileSystem` subclass recording `completeMultipartUpload`/`getObjStorage().abortMultipartUpload` (inject via a test seam that overrides `resolveObjectStoreFileSystem`). +Cases (each pins the WHY, Rule 9): +1. **Classification**: (a) unpartitioned APPEND → INSERT_EXISTING table action; (b) unpartitioned OVERWRITE → ALTER (DROP→ALTER transition); (c) partitioned NEW where `partitionExists`=false → ADD; (d) partitioned NEW where `partitionExists`=true → **NEW→APPEND downgrade** → INSERT_EXISTING (assert `addPartitions` NOT called, `updatePartitionStatistics` called); (e) partitioned OVERWRITE → ADD with dropFirst (assert `dropPartition(...,true)` then addPartition). Assert the action maps keyed by the right `NameMapping`/partitionValues. +2. **Commit dispatch**: full commit → recording-fake sees `addPartitions(db,tbl,allPartitions)` ONCE (batch-20 is inside the real client, so the fake sees the full list), staging→target `fs.renameDirectory`/rename invoked, MPU `completeMultipartUpload(remotePath, uploadId, etags)` invoked with `s3://bucket/key`. Assert commit order (fs rename → s3clean → addPartitions → updateStatistics → prune staging). +3. **Rollback**: with committer null but pending MPU → `abortMultipartUpload` called + staging deleted; with committer set → abort (undo stats via SUBTRACT + dropPartition of created values) then rollback (staging delete + MPU abort). Assert idempotent (second `rollback()` no-throw). +4. **addCommitData + getUpdateCnt**: feed 2–3 serialized `THivePartitionUpdate` (via `TSerializer`), assert `getUpdateCnt()` == Σ rowCount (D3) and accumulation order. +5. **full-ACID-write reject** (D7): `beginWrite` on a table whose params carry `transactional=true` (+ full-acid `transactional_properties`) → `DorisConnectorException`. Also assert a plain `transactional=true` insert-only vs full-acid distinction matches D8. +6. **profileLabel** == `"HMS"`; **getTransactionId** returns the ctor id. + +(Integration write/read gate is deferred to P7.4/P7.5 cutover — do NOT run here; tracked as the R-002 gate, Rule 12.) + +--- + +## 7. GAP LIST (design §4 assumptions vs REAL in-tree signatures) +- **GAP-1**: §4.1 `toColumnStatistics` does NOT exist → real: `HmsWriteConverter.toStatisticsParameters(Map,HmsCommonStatistics)` + `toPartitionStatistics(Map)`. Committer typically calls NEITHER (stats params are stamped inside `ThriftHmsClient.addPartitions`/stats overrides). +- **GAP-2**: `ReduceOperator` lives on `HmsCommonStatistics` (`ADD/SUBTRACT/MIN/MAX`), NOT `HmsPartitionStatistics`. Use `HmsCommonStatistics.ReduceOperator.SUBTRACT`. +- **GAP-3 (getTable return type)**: `HmsClient.getTable(db,tbl)` returns **`HmsTableInfo`** (columns `List`, `getLocation()`, `getInputFormat/getOutputFormat/getSerializationLib/getParameters/getSdParameters`), NOT hive-api `Table`. `table.getPartitionKeysSize()==0` → `getPartitionKeys().isEmpty()`; `table.getSd().getLocation()` → `getLocation()`. `TableAndMore.table` field type = `HmsTableInfo`. +- **GAP-4 (columns conversion)**: to fill `HmsPartitionWithStatistics.columns(List)` from `HmsTableInfo.getColumns()` (`List`), convert via public `HmsTypeMapping.toHiveTypeString(ConnectorType)` inline (or add a public `HmsWriteConverter.toFieldSchemas`). `toHiveSchema` is private — do not rely on it. +- **GAP-5 (serde name)**: read DTOs `HmsTableInfo`/`HmsPartitionInfo` expose `getSerializationLib()`; write DTO builder is `.serde(...)`/`getSerde()`. Map `getSerializationLib()` → `.serde(...)`. +- **GAP-6 DISSOLVED**: Reader 4's GAP-6 was wrong — BOTH `ConnectorSession.getCatalogId()` (line 43) AND `ConnectorContext.getCatalogId()` (line 38) exist. `NameMapping` ctlId = `context.getCatalogId()`. (equals/hashCode only needs intra-txn consistency; one NameMapping per txn.) +- **GAP-7 (batch-20 location)**: `Lists.partition(...,20)` is INSIDE `ThriftHmsClient.addPartitions` (INC-1), NOT the committer. `AddPartitionsTask.run` calls `hmsClient.addPartitions(db, tbl, ALL)` once. +- **GAP-8 (probe API)**: HEAD's NEW→APPEND probe used `hiveOps.getClient().getPartition(...)!=null`; use the not-found-tolerant `hmsClient.partitionExists(db, tbl, values)` primitive instead. +- **GAP-9 (fromCommonStatistics arg order)**: `HmsPartitionStatistics.fromCommonStatistics(long rowCount, long fileCount, long totalFileBytes)` — fileCount is the MIDDLE arg. +- **GAP-10 (write primitives take strings)**: all 5 write/stats primitives take `(String dbName, String tableName, ...)`, NOT `NameMapping`. Pass `nm.getRemoteDbName()`/`nm.getRemoteTblName()`. +- **GAP-11 (pathsEqual vs equalsIgnoreSchemeIfOneIsS3)**: HEAD's `needRename` used `PathUtils.equalsIgnoreSchemeIfOneIsS3`; INC-2's `HiveWriteUtils.pathsEqual` is a normalized compare (two nulls equal). **Verify it reproduces the S3-scheme-tolerant semantics**; if not, port `equalsIgnoreSchemeIfOneIsS3` into `HiveWriteUtils`. +- **GAP-12 (HiveUtil.toPartitionValues)**: HEAD used `HiveUtil.toPartitionValues`/`convertToNamePartitionMap` (fe-core). Verify a plugin equivalent exists in `fe-connector-hive`; if absent, port the pure `k=v/...` split. + +--- + +## 8. RISKS carried into implementation (locked by D-decisions; not human-blocking now) +- **R3/D6 provider-discovery (deferred to cutover)**: production `FileSystemProvider` discovery is fe-core/directory-plugin mediated (`FileSystemPluginManager` = classpath ServiceLoader **+** directory-loaded providers in a separate classloader). A plugin-side `ServiceLoader.load(FileSystemProvider.class)` sees ONLY providers on the hive plugin's own/parent classpath → directory-loaded object-store providers may NOT be discovered. **Mitigation**: isolate discovery behind one method (`resolveObjectStoreFileSystem(StorageProperties)`) so it is swappable; the batch is DORMANT (hive not in `SPI_READY_TYPES`) and unit tests inject a fake FS, so the risk does not bite until the P7.4/P7.5 integration gate. See §8 open decision below (D6 locks plugin-side; this is the one place recon flagged a template divergence with a material trade-off). +- **R1/D5 auth on plugin pools**: fs/rename/MPU tasks must each be `context.executeAuthenticated`-wrapped (§3.10). Metastore self-auths. No JVM-wide primer. +- **R4 DTO shape drift**: converter/committer unit tests must exercise exactly the fields the transaction passes (columns/formats/serde/stats), so a missed field is a test gap now, not a signature break. +- Import gate `tools/check-connector-imports.sh` each sub-step; `HiveVersionUtil` hit = known false positive (do not touch). + +--- + +## 9. VERIFIED PRE-IMPLEMENTATION CHECKS (resolved by orchestrator 2026-07-07 — TRUST THESE OVER §7 where they conflict) +- **GAP-11 RESOLVED — `pathsEqual` is NOT equivalent to `equalsIgnoreSchemeIfOneIsS3`.** `HiveWriteUtils.pathsEqual` calls `sameFileSystem(leftUri,rightUri)` which returns false when schemes differ (e.g. `s3://` vs `s3a://`, or s3 vs scheme-less). HEAD's `needRename` used `PathUtils.equalsIgnoreSchemeIfOneIsS3`, which deliberately treats an s3-schemed path equal to the same path with a different/absent scheme. **ACTION**: add a NEW package-private helper to `HiveWriteUtils` — `equalsIgnoreSchemeIfOneIsS3(String,String)` — a byte-faithful port of HEAD `PathUtils.equalsIgnoreSchemeIfOneIsS3` (find it: `fe/fe-core/.../foundation/util/PathUtils.java` or wherever HEAD defines it), add a unit test for it in `HiveWriteUtilsTest`, and use it for the committer's `needRename`/`needRename`-style checks (`prepareInsertExistingTable`/`prepareAddPartition`/`prepareInsertExistPartition`). Keep `pathsEqual` for exact-equality uses only. +- **GAP-12 RESOLVED — no plugin partition-name parser exists.** No `HiveUtil.toPartitionValues`/`convertToNamePartitionMap` in `fe-connector-hive`. **ACTION**: port the pure `col=val/col2=val2` → `List` (values in order) split as a package-private static helper (in `HiveWriteUtils` or a private method in the txn). Byte-faithful to HEAD `HiveUtil.toPartitionValues` (URL-decode each `=`-split value as HEAD does — verify HEAD uses `FileUtils.unescapePathName`/`MetaStoreUtils`). Add a small unit test. +- **D6 completeMultipartUpload — MAP OVERLOAD CONFIRMED (port §3.9c verbatim).** `ObjFileSystem.completeMultipartUpload(String remotePath, String uploadId, Map etags) throws IOException` EXISTS (converts to sorted `UploadPartResult` list internally). `TS3MPUPendingUpload.getEtags()` = thrift `map` = `Map`. HEAD's block (HMSTransaction ~1876-1893) ALREADY calls `objFs.completeMultipartUpload(remotePath, s3mpu.getUploadId(), s3mpu.getEtags())` + catches `java.io.IOException` — copy verbatim; ONLY change how `objFs` is obtained (build plugin FS from `context.getStorageProperties()` instead of `((SpiSwitchingFileSystem)fs).forPath(path)`). +- **D6 abort CONFIRMED**: `objFs.getObjStorage().abortMultipartUpload(String remotePath, String uploadId) throws IOException` (`ObjStorage.abortMultipartUpload`). `getObjStorage()` returns `ObjStorage`. +- **D6 FS-build API CONFIRMED**: `StorageProperties.kind()` returns `StorageKind`; `StorageKind.OBJECT_STORAGE` exists. `ConnectorContext.getStorageProperties()` returns `List` (default method, line ~292). `FileSystemProvider` (`org.apache.doris.filesystem.spi`, `interface FileSystemProvider

extends PluginFactory`): `boolean supports(Map properties)` + `default FileSystem create(Map properties) throws IOException` (also a typed `create(P)`; use the `Map` one). Recipe: `ServiceLoader.load(FileSystemProvider.class)` → first `p.supports(objSp.rawProperties())` → `p.create(objSp.rawProperties())` → cast to `ObjFileSystem`. **ISOLATE behind one method `resolveObjectStoreFileSystem(StorageProperties)` (swappable + test-injectable)** — production ServiceLoader discovery of directory-loaded providers is a cutover-time concern (§8/R3), unit tests inject a fake, batch is dormant. +- **auth CONFIRMED**: `ConnectorContext.executeAuthenticated(Callable task) throws Exception` (default method, line ~94). Wrap each fs/rename/MPU task body: `context.executeAuthenticated(() -> { ...; return null; })` inside `try { } catch (Exception e) { ... }`. +- **iceberg template ctor CONFIRMED**: `IcebergConnectorTransaction(long transactionId, IcebergCatalogOps catalogOps, ConnectorContext context)`; uses `context.executeAuthenticated(() -> {...})` for table load; `close()` at ~1114. Hive analogue: `HiveConnectorTransaction(long, HmsClient, ConnectorContext)`. +- **beginTransaction wiring CONFIRMED**: `HiveConnectorMetadata` already holds `private final HmsClient hmsClient;` + `private final ConnectorContext context;`. Override `ConnectorWriteOps.beginTransaction(ConnectorSession)` default (one-liner) directly on `HiveConnectorMetadata`. diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index a47629c6b5836e..0ae597cfd58a29 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -195,12 +195,15 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] Pure path helpers + `mergePartitions` → new package-private `HiveWriteUtils` (`fe-connector-hive`): `isSubDirectory`/`getImmediateChildPath`/`pathsEqual` (package-private, INC-3-callable) + private `sameFileSystem`/`normalizeUriPart`/`normalizePath` + `mergePartitions` (pure `THivePartitionUpdate` merge, static). **Byte-faithful** port of HEAD `HMSTransaction` (1654–1737 + 153–168); fe-core-free (only Hadoop `Path` + `java.net.URI` + shared thrift; **no guava/lombok**). `mergePartitions` made static (used no instance state). - [x] **Leaf-type resolution after reading plugin-side first (R6):** the design's "co-move only what has no plugin peer" resolved to co-moving **only `NameMapping`** (`fe-connector-hive`, JDK-only copy of fe-core `datasource.NameMapping`, dropping lombok `@Getter`/guava `@VisibleForTesting`) — it is the transaction's per-table/per-partition action-map **key** and has no plugin peer. **NOT co-moved** (already have plugin peers from INC-1): `CommonStatistics`→`HmsCommonStatistics`, `HivePartitionStatistics`→`HmsPartitionStatistics` (both carry the full `EMPTY`/`fromCommonStatistics`/`merge`/`reduce`/`ReduceOperator` ops the txn uses), `HivePartitionWithStatistics`→`HmsPartitionWithStatistics` (flattened SD). **`HivePartition` deliberately deferred to INC-3** (not a leaf co-move): its fe-core `Column` drag (`getPartitionName(List)`) is **dead on the write path** — verified never called (the `getPartitionName()` hits are `PartitionAndMore.getPartitionName()`), and INC-1's flattened `HmsPartitionWithStatistics` already supersedes its SD role, so `PartitionAndMore`'s descriptor shape (Hms-types vs. a trimmed struct) is an INC-3 body-port decision — pre-creating a `HivePartition` copy now would be speculative (Rule 2) and risk rework. - [x] **Tests**: `HiveWriteUtilsTest` (12: mergePartitions sum/concat/distinct-names/pending-upload-aggregate/null-safe; isSubDirectory happy/equal/sibling-prefix/cross-FS/null; getImmediateChildPath first-level/direct/non-child; pathsEqual normalize/cross-FS/null — each pins the *why*, e.g. commit accounting, staging-cleanup boundary) + `NameMappingTest` (4: value equals/hashCode, **map-key-by-value** resolution, any-field-difference breaks equality, full-name join — key semantics are load-bearing since a broken key splits one table's actions and drops writes). -- [ ] **INC-3 — `HiveWriteContext` + `HiveConnectorTransaction` core + `beginTransaction`** (`fe-connector-hive`) — the genuinely-atomic unit - - [ ] `HiveWriteContext` (§4.2). - - [ ] `HiveConnectorTransaction`: classification state machine (APPEND/NEW/OVERWRITE + probe downgrade) + `HmsCommitter` (prepare*/doCommit/abort/rollback + staging rename + MPU) + SPI methods. Break couplings #1/#3 (D4/D5/D6). full-ACID write reject (D7). rollback deletes staging + aborts MPU (D9). **Action maps key by INC-2's plugin `NameMapping`.** - - [ ] **Partition-descriptor shaping (deferred from INC-2):** decide `PartitionAndMore`/`TableAndMore` descriptor form — reshape onto `HmsPartitionWithStatistics` (INC-1 flattened SD) + inline fields rather than copying fe-core `HivePartition` (its `Column`-dragging `getPartitionName` is dead on write; verified in INC-2). `addPartitions` already consumes `List` with batch-20 inside the client, so the committer builds `HmsPartitionWithStatistics` directly. - - [ ] `HiveConnectorMetadata.beginTransaction` one-liner (+ `ConnectorWriteOps.beginTransaction`). - - [ ] **Tests**: `HiveConnectorTransactionTest` (recording-fake `HmsClient` + fake `FileSystem`): classification cases, commit dispatch (add_partitions + staging rename + MPU complete), rollback (staging delete + MPU abort), addCommitData+getUpdateCnt, full-ACID-write reject. +- [~] **INC-3 — `HiveWriteContext` + `HiveConnectorTransaction` core + `beginTransaction`** (`fe-connector-hive`) — **MAIN DONE (uncommitted, in tree; `fe-connector-hive` module 62 tests green incl new 9, checkstyle 0, import-gate clean, `compile`+`test` BUILD SUCCESS, riskiest spans spot-checked faithful). Port map = `P7.3-INC-3-portmap.md` (595 lines, incl §9 verified checks).** Built by an implementation agent (interrupted by an account session-limit after the main code compiled) then finished/verified by the orchestrator. + - [x] `HiveWriteContext` (§4.2) — 89 lines, package-private immutable value object. + - [x] `HiveConnectorTransaction` (1697 lines): classification state machine (APPEND/NEW/OVERWRITE + `partitionExists` probe downgrade) + `HmsCommitter` (prepare*/doCommit/abort/rollback + staging rename + MPU) + 7 SPI overrides. Couplings #1/#3 broken (D4 profiling dropped / D5 per-task `executeAuthenticated` / D6 plugin FS via `protected resolveObjectStoreFileSystem` + `objCommit`). full-ACID write reject in `beginWrite` (D7/D8, plugin-derived from raw params). rollback deletes staging + aborts MPU (D9). Action maps keyed by INC-2 `NameMapping`. + - [x] **GAP-11/GAP-12 resolved in `HiveWriteUtils`**: added `equalsIgnoreSchemeIfOneIsS3` (the rename check — `pathsEqual` is NOT scheme-tolerant) + `toPartitionValues` (partition-name split) + unit tests. + - [x] **Partition-descriptor shaping**: `TableAndMore`/`PartitionAndMore` as inline fields + `HmsTableInfo`/`HmsPartitionStatistics` (no fe-core `HivePartition`); ADD builds `HmsPartitionWithStatistics` in `prepareAddPartition` (`ConnectorColumn`→`FieldSchema` via inline `toFieldSchemas`/`HmsTypeMapping`). + - [x] `HiveConnectorMetadata.beginTransaction` one-liner (overrides `ConnectorWriteOps.beginTransaction` default; +11 pom `fe-filesystem-spi` `provided`). + - [x] **Tests (FS-free half, GREEN — 9 cases)**: `HiveConnectorTransactionTest` + `FakeConnectorContext` — classification / **NEW→APPEND downgrade** / full-ACID-write reject (D7) vs insert-only allow (D8) / addCommitData+getUpdateCnt=Σ (D3) / garbage-bytes reject / profileLabel="HMS" (D2) / getTransactionId. Uses a recording-fake `HmsClient` (most Phase-3+ methods are `default`-throw → fake overrides only what's used); `finishInsertTable` is package-private so classification is exercised without a filesystem. + - [ ] **REMAINING (a): FS-heavy committer tests** — commit dispatch (`addPartitions` once + staging→target rename + MPU `completeMultipartUpload`) + rollback (MPU `abortMultipartUpload` + staging delete + idempotency). Inject a fake `ObjFileSystem` + fake `FileSystem` by subclassing `HiveConnectorTransaction` and overriding `protected resolveObjectStoreFileSystem(StorageProperties)`. + - [ ] **REMAINING (b): full adversarial fidelity review** (clean-room, memory `clean-room-adversarial-review-pref`) — orchestrator spot-checked the riskiest spans (D6/commit/rollback/classification/guards = faithful) but has NOT run the multi-agent line-by-line-vs-HEAD review. **⚠️ session-limit is a shared pool — confirm quota recovered before fanning out agents.** - [ ] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive`) - [ ] `HiveWritePlanProvider` (§4.4) + `buildSink` (§5.2) + `HiveConnector.getWritePlanProvider()` + `supportedOperations` + capability gates. Re-home LZO-INSERT reject + transactional-table reject as begin-guards. - [ ] **Tests**: `HiveWritePlanProviderTest` against a fake `ConnectorWriteHandle` — asserts emitted `THiveTableSink` (PARTITION_KEY/REGULAR order, bucket, compression-per-format, staging-vs-in-place location, serde), INSERT→OVERWRITE promotion, begin-guard rejects. From e05615d21d4cff0e5905534f05a150d64212094f Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 17:50:50 +0800 Subject: [PATCH 021/334] [doc](catalog) P7.3 hive INC-3: clean-room fidelity review DONE (11/12 faithful) + dup-key fail-loud fix + refresh HANDOFF 12-agent adversarial clean-room review of HiveConnectorTransaction vs HEAD HMSTransaction: 11/12 dimensions faithful; 4 low discrepancies, only 1 CONFIRMED defect -- convertToInsertExistingPartitionAction silently deduped via HashMap.put where HEAD's Collectors.toMap fails loud on a duplicate key (silent dropped write vs abort, Rule 12). Fixed with two put(...)!=null -> IllegalStateException guards + testDuplicatePartitionValuesFromHmsFailsLoud. Other 3 adjudicated BENIGN (getTable begin-snapshot; redundant auth wrap; dropped fe-core doNothing() debug-point test seam). Code fix rides in the uncommitted INC-1..INC-3 WIP batch (atomic model, design doc S8); this commit is docs only. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 19 +++++++++---------- ....3-hive-write-txn-implementation-design.md | 5 +++-- 2 files changed, 12 insertions(+), 12 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 90c4100344791c..87787411e6a91d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,17 +5,16 @@ --- -# 🎯 当前状态（2026-07-07）= **P7.3 原子批 INC-1 + INC-2 + INC-3(主体) 均已实现并全绿——INC-3 把 fe-core `HMSTransaction`(1895 行) 忠实移植成插件 `HiveConnectorTransaction`(1697 行) + `HiveWriteContext`(89) + `HiveConnectorMetadata.beginTransaction` 一行工厂 + `HiveWriteUtils` 增 `equalsIgnoreSchemeIfOneIsS3`(GAP-11)/`toPartitionValues`(GAP-12) + `HiveConnectorTransactionTest`(9 例全绿) + `FakeConnectorContext`(测试 ctx double)。按用户拍板「整批一次原子提交」，全部仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit 了 HANDOFF + 设计文档 + 新落地的 port-map。** +# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1 + INC-2 + INC-3(主体) 全绿 + INC-3 的全量对抗式忠实度复审已跑完(收尾 b) + 复审查出的唯一缺陷已修+加测+验证。INC-3 只剩 FS 重活单测(收尾 a) → 然后 INC-4 / INC-5。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit HANDOFF + 设计文档。** -> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 独立 `compile` BUILD SUCCESS；全模块 `test` **62 测试 0 失败 0 错误 BUILD SUCCESS（含新 `HiveConnectorTransactionTest` 9、既有 hive/hms 全绿、无回归）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（仅 `HiveVersionUtil` 已知误报）。侦察侧我**逐行核对了最险的几段**（D6 MPU/FS 构建 `getFileSystem`/`resolveObjectStoreFileSystem`/`objCommit`；`commit()`/`rollback()` 顺序；分类状态机 `finishInsertTable`/NEW→APPEND 降级；full-ACID 拒写守卫 D7/D8），确为忠实移植。 +> **本轮做了什么（复审 + 修缺陷）**：跑了 12 路净室对抗复审（workflow `hive-txn-fidelity-review`）——12 个 finder（code-first、不给"忠实"先验）各拿 `HiveConnectorTransaction` 一个维度逐行对照 HEAD `HMSTransaction`，每条分歧再派对抗性怀疑者 verify。**结论：12 维中 11 维完全忠实；共 4 条分歧全为 low；只 1 条 CONFIRMED 缺陷**（`convertToInsertExistingPartitionAction` 用 `HashMap.put` 静默去重，HEAD 的 `Collectors.toMap` 遇重复键抛异常——静默去重会让 `size()` 界的循环丢掉一个分区的 INSERT_EXISTING 动作 = 静默丢写，违反 Rule 12）。**已修**：两处 `put(...) != null` 抛 `IllegalStateException("Duplicate key ...")`，忠实复刻 HEAD；加测 `testDuplicatePartitionValuesFromHmsFailsLoud`。另 3 条判 BENIGN（`getTable` 复用 begin 期快照=单表同真值；`beginWrite` 对自 pin 的 `getTable` 多包一层 `executeAuthenticated`=无害冗余；HEAD `doCommit()` 末尾 `doNothing()` 被删=空的 fe-core debug-point 测试注入桩，连接器不能 import `DebugPointUtil`、且只在延后的集成门有意义→**有意删除，已记录**）。有 1 个 verifier 死于 StructuredOutput 重试上限（`doNothing` 那条），已人工判 BENIGN。 -> **⚠️ 本轮的过程事故（已恢复，无损）**：INC-3 的实现最初交给一个后台实现 agent（照 port-map 写代码），它写完**主体代码并使其编译通过**、正开始写测试时**撞上账号级 session limit（resets 5pm Asia/Shanghai）而中断**。我接手后：确认主体代码 `compile` + checkstyle + import-gate 全绿、逐行 spot-check 忠实、然后**亲手补齐了 FS-free 单测**（利用 `finishInsertTable` 包级可见 + `HmsClient` 方法多为 default-throw，绕开重型 FS 假件）。因 session limit 是共享池，本轮起**未再起新的多 agent 编排**，改在主循环内经济地推进。 +> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `test` **63 测试 0 失败 0 错误 BUILD SUCCESS（原 62 + 新 dup 测试 1；hive/hms 全绿、无回归）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（仅 `HiveVersionUtil` 已知误报，gate 现已自动 skip）。 -> **⚠️ 工作树未提交的 INC-1 + INC-2 + INC-3 代码**（`git status` 会显示）： +> **⚠️ 工作树未提交的 INC-1 + INC-2 + INC-3 代码**（`git status` 会显示；原子批模型，设计文档 §8——**若工作树被清就照 port-map 重建**）： > - INC-1（`fe-connector-hms`）：`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsPartitionInfo` 新 + 2 test。 > - INC-2（`fe-connector-hive`）：`HiveWriteUtils.java` + `NameMapping.java`（+ 2 test）。 -> - **INC-3（`fe-connector-hive`，本轮）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1697) + `HiveConnectorTransactionTest.java`(351) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(+15，`beginTransaction` override) + `pom.xml`(+11，`fe-filesystem-spi` scope `provided`) + `HiveWriteUtils.java`（追加 `equalsIgnoreSchemeIfOneIsS3`/`toPartitionValues` 两个纯 helper + 单测）。 -> 这是**故意的**（原子批模型，设计文档 §8）——下个 session 在此基础上做 INC-3 收尾 + INC-4/INC-5，全批全绿后才一次性 feature commit。**若工作树被清就照 port-map 重建**。 +> - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含本轮 dup-key 修复) + `HiveConnectorTransactionTest.java`(377，含新 dup 测试) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(+15，`beginTransaction` override) + `pom.xml`(+11，`fe-filesystem-spi` scope `provided`) + `HiveWriteUtils.java`（`equalsIgnoreSchemeIfOneIsS3`/`toPartitionValues` + 单测）。 > **权威实现依据**（信 HEAD 控制流，不信行号）： > - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 @@ -26,9 +25,9 @@ # 🚀 下个 session 任务 = **INC-3 收尾 →（全绿后）INC-4 → INC-5，全批一次原子 feature commit** -> **① INC-3 收尾（两件，本轮故意未做——session-limit 下保守停在干净节点）**： -> - **(a) 提交器 FS 重活的单测**（port-map §6 里 commit/rollback 那半）：commit 派发（`addPartitions` 一次 + staging→target 改名 + MPU `completeMultipartUpload`）、rollback（MPU `abortMultipartUpload` + 删 staging + 幂等）。**注入路径已备好**：`HiveConnectorTransaction.resolveObjectStoreFileSystem(StorageProperties)` 是 `protected`，测试**子类化**它返回一个假 `ObjFileSystem`（记录 `completeMultipartUpload`/`getObjStorage().abortMultipartUpload`）+ 假 `org.apache.doris.filesystem.FileSystem`（记录 `rename`/`renameDirectory`/`delete`/`mkdirs`/`exists`）。本轮的 FS-free 测试已覆盖分类/降级/拒写/getUpdateCnt/SPI-identity，故这步是**补齐 FS 那一半**。 -> - **(b) 全量对抗性忠实度复审（clean-room，memory `clean-room-adversarial-review-pref`）**：我本轮只 spot-check 了最险几段（已确认忠实），但**尚未做多 agent 全量对抗复审**。宜派多 agent 各拿 `HiveConnectorTransaction` 与 HEAD `HMSTransaction` 逐维度对比（提交顺序 / `HmsCommitter` prepare*/doCommit/abort/rollback / MPU / D5 鉴权包裹 / 那 12 处 GAP / `Action` ctor 的 DROP-data==null 语义 / `convertToInsertExistingPartitionAction` 的 map/list index quirk 是否忠实），专挑分歧，再核对。**⚠️ session limit 是共享池——起多 agent 前先确认额度已恢复（5pm 后）**。 +> **① INC-3 收尾 —— 只剩 (a) FS 重活单测（(b) 复审已完成，见 🎯）**： +> - **(a) 提交器 FS 重活的单测**：commit 的 MPU `completeMultipartUpload` + rollback 的 MPU `abortMultipartUpload` + 幂等（第二次 `rollback()` 不抛）。**注入 seam + 假件形状已确定**（设计文档 §6 INC-3 (a) 有完整 scoping）：子类化 `HiveConnectorTransaction`、override `protected resolveObjectStoreFileSystem(StorageProperties)` 返回假 `ObjFileSystem`（`abstract class implements FileSystem`，ctor 收 `ObjStorage`；其**具体** `completeMultipartUpload(String,String,Map)` 转 `List` 后委派 `objStorage.completeMultipartUpload(...)`，`getObjStorage()` 返回 ctor 参数——故**一个 recording 假 `ObjStorage` 同时记录 complete+abort**）。**关键简化**：沿用现有 `TFileType.FILE_S3` ctx → `stagingDirectory=Optional.empty()` → `pruneAndDeleteStagingDirectories` 空转、目录遍历 FS 方法（`listFiles`/`delete`）**不触发**，故只需假 MPU 面。触发：MPU-complete = 无分区 APPEND + PU 带 `s3MpuPendingUploads` 且 `targetPath==writePath`（needRename=false → `objCommit`）；MPU-abort = committer-null 路径的 `rollback()`（`collectUncompletedMpuPendingUploads` 从原始 updates 收集 → `abortMultiUploads`）。若要覆盖 staging→target 改名 + `addPartitions` 一次，另用 non-S3 staging ctx（需再假 `FileSystem.rename`/`renameDirectory`）。 +> - **(b) 全量对抗性忠实度复审 —— ✅ 已完成本轮**（12 路净室对抗，结论见 🎯：11/12 忠实，唯一 low 缺陷已修+测）。无需再跑；INC-4/INC-5 落地后可对新代码再起一轮同款复审。 > **② INC-4 —— `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities**（`fe-connector-hive`，port-map §4.4/§5.2）：`planWrite` 调 `tx.beginWrite(...)` + `buildSink`（= 忠实移植 HEAD `planner.HiveTableSink.bindDataSink`：PARTITION_KEY/REGULAR 按表序、bucket、按格式压缩、staging-vs-in-place location、serde、`hadoopConfig` 经 `context.getStorageProperties().toBackendProperties()` + `vendStorageCredentials`）+ `HiveConnector.getWritePlanProvider()` + `supportedOperations={INSERT,OVERWRITE}` + capability gates；LZO-INSERT / 事务表拒写作 begin-guard。测试 = `HiveWritePlanProviderTest`（假 `ConnectorWriteHandle`，断 `THiveTableSink` 形状 + INSERT→OVERWRITE 提升 + guard）。 @@ -39,7 +38,7 @@ ## 开场要点（承接） 1. **起步先读**本文顶部 🎯 段 + `tasks/P7.3-INC-3-portmap.md`（尤其 §9 已核实检查 + §6 测试计划）+ 设计文档 §6 构建顺序。**信 HEAD 控制流不信行号**——每处编辑前重读对应 HEAD 段。 -2. **主体已在，勿重写**：INC-3 的 `HiveConnectorTransaction` 已编译 + 单测 + checkstyle + import-gate 全绿且经 spot-check 忠实。下步是**加 FS 测试 + 全量复审**，不是重造。 +2. **主体已在 + 已全量复审，勿重写**：INC-3 的 `HiveConnectorTransaction` 已编译 + 63 单测 + checkstyle + import-gate 全绿，且**已过 12 路净室对抗复审（11/12 忠实，唯一 low 缺陷已修+测）**。下步是**加 FS 单测(收尾 a)** 然后 **INC-4 → INC-5**，不是重造、也不用再复审现有代码。 3. **范围锁定（勿重议）**：hive **不在** `SPI_READY_TYPES`（整批天然 dormant，编译+单测但零线上路由）；翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 删 legacy **均属后续 P7.4/P7.5，另起原子批**。full-ACID **写**继续硬拒（D7），full-ACID **读**在范围（INC-5）。 4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 写路径 → P7.4/P7.5 翻闸时跑，勿静默跳过——Rule 12）。 5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index 0ae597cfd58a29..e5965caf02de80 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -202,8 +202,9 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] **Partition-descriptor shaping**: `TableAndMore`/`PartitionAndMore` as inline fields + `HmsTableInfo`/`HmsPartitionStatistics` (no fe-core `HivePartition`); ADD builds `HmsPartitionWithStatistics` in `prepareAddPartition` (`ConnectorColumn`→`FieldSchema` via inline `toFieldSchemas`/`HmsTypeMapping`). - [x] `HiveConnectorMetadata.beginTransaction` one-liner (overrides `ConnectorWriteOps.beginTransaction` default; +11 pom `fe-filesystem-spi` `provided`). - [x] **Tests (FS-free half, GREEN — 9 cases)**: `HiveConnectorTransactionTest` + `FakeConnectorContext` — classification / **NEW→APPEND downgrade** / full-ACID-write reject (D7) vs insert-only allow (D8) / addCommitData+getUpdateCnt=Σ (D3) / garbage-bytes reject / profileLabel="HMS" (D2) / getTransactionId. Uses a recording-fake `HmsClient` (most Phase-3+ methods are `default`-throw → fake overrides only what's used); `finishInsertTable` is package-private so classification is exercised without a filesystem. - - [ ] **REMAINING (a): FS-heavy committer tests** — commit dispatch (`addPartitions` once + staging→target rename + MPU `completeMultipartUpload`) + rollback (MPU `abortMultipartUpload` + staging delete + idempotency). Inject a fake `ObjFileSystem` + fake `FileSystem` by subclassing `HiveConnectorTransaction` and overriding `protected resolveObjectStoreFileSystem(StorageProperties)`. - - [ ] **REMAINING (b): full adversarial fidelity review** (clean-room, memory `clean-room-adversarial-review-pref`) — orchestrator spot-checked the riskiest spans (D6/commit/rollback/classification/guards = faithful) but has NOT run the multi-agent line-by-line-vs-HEAD review. **⚠️ session-limit is a shared pool — confirm quota recovered before fanning out agents.** + - [x] **CONFIRMED-DEFECT FIX (from the review below) + test — GREEN**: `convertToInsertExistingPartitionAction` now fails loud on a duplicate key (mirrors HEAD `HiveUtil.convertToNamePartitionMap`'s `Collectors.toMap`): `if (partitionsByValues.put(...) != null) throw IllegalStateException("Duplicate key ...")` for duplicate HMS partition values, and `if (p != null && partitionsByNamesMap.put(...) != null) throw` for a repeated requested name. The prior `HashMap.put` silently overwrote → the `size()`-bounded loop dropped a trailing partition's INSERT_EXISTING action = silent write loss (Rule 12 violation). Test `testDuplicatePartitionValuesFromHmsFailsLoud` (dup-valued HMS response → `IllegalStateException`). `fe-connector-hive` now **63 tests green**, checkstyle 0, import-gate clean, `compile`+`test` BUILD SUCCESS. + - [ ] **REMAINING (a): FS-heavy committer tests** — commit dispatch (MPU `completeMultipartUpload`, and `addPartitions` once + staging→target rename for a non-S3 staging ctx) + rollback (MPU `abortMultipartUpload` + staging delete + idempotency). **Scoped**: subclass `HiveConnectorTransaction`, override `protected resolveObjectStoreFileSystem(StorageProperties)` to return a fake `ObjFileSystem` (`abstract class implements FileSystem`, ctor takes `ObjStorage`; its concrete `completeMultipartUpload(String,String,Map)` converts→`List` and delegates to `objStorage.completeMultipartUpload(...,List)`; `getObjStorage()` returns the ctor arg — so a recording fake `ObjStorage` captures BOTH complete+abort). **Simplification**: keep the existing `TFileType.FILE_S3` ctx → `stagingDirectory = Optional.empty()` → `pruneAndDeleteStagingDirectories` is a no-op and directory-walk FS methods (`listFiles`/`delete`) are NOT hit, so only the MPU surface needs faking. MPU-complete: unpartitioned APPEND with `s3MpuPendingUploads` set and `targetPath==writePath` (needRename false → `objCommit`). MPU-abort: `rollback()` on the committer-null path (`collectUncompletedMpuPendingUploads` from the raw updates → `abortMultiUploads`). + - [x] **REMAINING (b): full adversarial fidelity review — DONE (clean-room, 12-agent, memory `clean-room-adversarial-review-pref`)**. Ran `hive-txn-fidelity-review` workflow: 12 finders (code-first, un-primed) each diffed one dimension of `HiveConnectorTransaction` vs HEAD `HMSTransaction`, then an adversarial skeptic verified each discrepancy. **Result: 11/12 dimensions faithful; 4 raw discrepancies, all severity `low`; only 1 CONFIRMED defect** (the dup-key fail-loud above, now fixed). Other 3: `getTable()` reuses begin-time snapshot (BENIGN — same-truth in single-table txn), `beginWrite` double-wraps self-pinning `getTable` in `executeAuthenticated` (BENIGN — harmless redundancy), and HEAD's trailing `doNothing()` in `doCommit()` dropped (BENIGN — an **empty fe-core debug-point test seam**, `// only for regression test and unit test to throw exception`; re-adding needs fe-core `DebugPointUtil` which connectors may not import, and it only matters at the deferred integration gate → **intentional drop, recorded**). One verifier died on the `doNothing()` candidate (StructuredOutput retry cap); adjudicated by hand → BENIGN. - [ ] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive`) - [ ] `HiveWritePlanProvider` (§4.4) + `buildSink` (§5.2) + `HiveConnector.getWritePlanProvider()` + `supportedOperations` + capability gates. Re-home LZO-INSERT reject + transactional-table reject as begin-guards. - [ ] **Tests**: `HiveWritePlanProviderTest` against a fake `ConnectorWriteHandle` — asserts emitted `THiveTableSink` (PARTITION_KEY/REGULAR order, bucket, compression-per-format, staging-vs-in-place location, serde), INSERT→OVERWRITE promotion, begin-guard rejects. From 9569710dd3d67bdc6da6bf7758fe95ff9cd547e2 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 18:27:49 +0800 Subject: [PATCH 022/334] =?UTF-8?q?[doc](catalog)=20P7.3=20hive=20INC-3:?= =?UTF-8?q?=20FS-heavy=20committer=20tests=20DONE=20=E2=86=92=20INC-3=20co?= =?UTF-8?q?mplete;=20refresh=20HANDOFF=20+=20design=20doc?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Added 4 committer filesystem unit tests to HiveConnectorTransactionTest (injected via a resolveObjectStoreFileSystem override returning a recording ObjFileSystem/ObjStorage): MPU completeMultipartUpload on commit (ETags sorted by part number), MPU abortMultipartUpload on rollback, rollback idempotency (second rollback no-throw, abort exactly once), and addPartitions called once with the full list + SD columns rebuilt from the table (GAP-7 / GAP-4). fe-connector-hive now 67 tests green (was 63), checkstyle 0, import-gate clean. INC-3 is fully done (main + fidelity review + FS tests); next is INC-4 -> INC-5. Feature code stays uncommitted (atomic-batch model); this commit is docs only. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 18 ++++++++---------- ...7.3-hive-write-txn-implementation-design.md | 4 ++-- 2 files changed, 10 insertions(+), 12 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 87787411e6a91d..fdbe930a2b11af 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,16 +5,16 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1 + INC-2 + INC-3(主体) 全绿 + INC-3 的全量对抗式忠实度复审已跑完(收尾 b) + 复审查出的唯一缺陷已修+加测+验证。INC-3 只剩 FS 重活单测(收尾 a) → 然后 INC-4 / INC-5。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit HANDOFF + 设计文档。** +# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1 + INC-2 + INC-3 全部完成（主体 + 复审 + FS 重活单测全绿）。下一步 = INC-4 → INC-5。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit HANDOFF + 设计文档。** -> **本轮做了什么（复审 + 修缺陷）**：跑了 12 路净室对抗复审（workflow `hive-txn-fidelity-review`）——12 个 finder（code-first、不给"忠实"先验）各拿 `HiveConnectorTransaction` 一个维度逐行对照 HEAD `HMSTransaction`，每条分歧再派对抗性怀疑者 verify。**结论：12 维中 11 维完全忠实；共 4 条分歧全为 low；只 1 条 CONFIRMED 缺陷**（`convertToInsertExistingPartitionAction` 用 `HashMap.put` 静默去重，HEAD 的 `Collectors.toMap` 遇重复键抛异常——静默去重会让 `size()` 界的循环丢掉一个分区的 INSERT_EXISTING 动作 = 静默丢写，违反 Rule 12）。**已修**：两处 `put(...) != null` 抛 `IllegalStateException("Duplicate key ...")`，忠实复刻 HEAD；加测 `testDuplicatePartitionValuesFromHmsFailsLoud`。另 3 条判 BENIGN（`getTable` 复用 begin 期快照=单表同真值；`beginWrite` 对自 pin 的 `getTable` 多包一层 `executeAuthenticated`=无害冗余；HEAD `doCommit()` 末尾 `doNothing()` 被删=空的 fe-core debug-point 测试注入桩，连接器不能 import `DebugPointUtil`、且只在延后的集成门有意义→**有意删除，已记录**）。有 1 个 verifier 死于 StructuredOutput 重试上限（`doNothing` 那条），已人工判 BENIGN。 +> **本轮做了什么（INC-3 收尾 a：提交器 FS 重活单测）**：给 `HiveConnectorTransactionTest` 加了 4 个 committer FS 单测，覆盖 D6/D9 对象存储那半（净室复审 b 已在上一轮做完，本轮无需再跑）。注入方式 = 匿名子类 override `resolveObjectStoreFileSystem(...)` 返回一个 recording 假 `ObjFileSystem`（内含 recording 假 `ObjStorage`，同时记录 complete+abort；假件的非 MPU FS 方法一律 fail-loud，走到即测试失败）。4 例：① commit 时 MPU `completeMultipartUpload`（无分区 APPEND、`targetPath==writePath` 走 `objCommit`；并断言 ETag 按 part 号排序）；② `rollback()`（committer-null 路径）MPU `abortMultipartUpload`；③ rollback **幂等**（第二次 `rollback()` 不抛、且 abort 恰好一次）；④ `addPartitions` **只调一次**且带整表分区 SD（GAP-7 批处理在 client 内、GAP-4 列从表 schema 重建）——第④例 FILE_S3 下 `targetPath==writePath` 不触发任何 FS，故连假件都不需要。同步把测试类头 Javadoc 与设计文档 §6 INC-3 (a) 勾成 done。 -> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `test` **63 测试 0 失败 0 错误 BUILD SUCCESS（原 62 + 新 dup 测试 1；hive/hms 全绿、无回归）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（仅 `HiveVersionUtil` 已知误报，gate 现已自动 skip）。 +> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `test` **67 测试 0 失败 0 错误 BUILD SUCCESS（原 63 + 新 4；hive/hms 全绿、无回归）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（仅 `HiveVersionUtil` 已知误报，gate 现已自动 skip）。 > **⚠️ 工作树未提交的 INC-1 + INC-2 + INC-3 代码**（`git status` 会显示；原子批模型，设计文档 §8——**若工作树被清就照 port-map 重建**）： > - INC-1（`fe-connector-hms`）：`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsPartitionInfo` 新 + 2 test。 > - INC-2（`fe-connector-hive`）：`HiveWriteUtils.java` + `NameMapping.java`（+ 2 test）。 -> - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含本轮 dup-key 修复) + `HiveConnectorTransactionTest.java`(377，含新 dup 测试) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(+15，`beginTransaction` override) + `pom.xml`(+11，`fe-filesystem-spi` scope `provided`) + `HiveWriteUtils.java`（`equalsIgnoreSchemeIfOneIsS3`/`toPartitionValues` + 单测）。 +> - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含 dup-key 修复) + `HiveConnectorTransactionTest.java`(654，14 例：10 分类/SPI + 4 committer FS) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(+15，`beginTransaction` override) + `pom.xml`(+11，`fe-filesystem-spi` scope `provided`) + `HiveWriteUtils.java`（`equalsIgnoreSchemeIfOneIsS3`/`toPartitionValues` + 单测）。 > **权威实现依据**（信 HEAD 控制流，不信行号）： > - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 @@ -23,11 +23,9 @@ --- -# 🚀 下个 session 任务 = **INC-3 收尾 →（全绿后）INC-4 → INC-5，全批一次原子 feature commit** +# 🚀 下个 session 任务 = **INC-4 → INC-5，（全绿后）全批一次原子 feature commit** -> **① INC-3 收尾 —— 只剩 (a) FS 重活单测（(b) 复审已完成，见 🎯）**： -> - **(a) 提交器 FS 重活的单测**：commit 的 MPU `completeMultipartUpload` + rollback 的 MPU `abortMultipartUpload` + 幂等（第二次 `rollback()` 不抛）。**注入 seam + 假件形状已确定**（设计文档 §6 INC-3 (a) 有完整 scoping）：子类化 `HiveConnectorTransaction`、override `protected resolveObjectStoreFileSystem(StorageProperties)` 返回假 `ObjFileSystem`（`abstract class implements FileSystem`，ctor 收 `ObjStorage`；其**具体** `completeMultipartUpload(String,String,Map)` 转 `List` 后委派 `objStorage.completeMultipartUpload(...)`，`getObjStorage()` 返回 ctor 参数——故**一个 recording 假 `ObjStorage` 同时记录 complete+abort**）。**关键简化**：沿用现有 `TFileType.FILE_S3` ctx → `stagingDirectory=Optional.empty()` → `pruneAndDeleteStagingDirectories` 空转、目录遍历 FS 方法（`listFiles`/`delete`）**不触发**，故只需假 MPU 面。触发：MPU-complete = 无分区 APPEND + PU 带 `s3MpuPendingUploads` 且 `targetPath==writePath`（needRename=false → `objCommit`）；MPU-abort = committer-null 路径的 `rollback()`（`collectUncompletedMpuPendingUploads` 从原始 updates 收集 → `abortMultiUploads`）。若要覆盖 staging→target 改名 + `addPartitions` 一次，另用 non-S3 staging ctx（需再假 `FileSystem.rename`/`renameDirectory`）。 -> - **(b) 全量对抗性忠实度复审 —— ✅ 已完成本轮**（12 路净室对抗，结论见 🎯：11/12 忠实，唯一 low 缺陷已修+测）。无需再跑；INC-4/INC-5 落地后可对新代码再起一轮同款复审。 +> **① INC-3 —— ✅ 全部完成**（主体 + 复审 b + FS 重活单测 a，见 🎯；`fe-connector-hive` 67 测试全绿）。下步是 **INC-4 → INC-5**，不再回炒 INC-3。INC-4/INC-5 落地后可对**新代码**再起一轮同款净室对抗复审。 > **② INC-4 —— `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities**（`fe-connector-hive`，port-map §4.4/§5.2）：`planWrite` 调 `tx.beginWrite(...)` + `buildSink`（= 忠实移植 HEAD `planner.HiveTableSink.bindDataSink`：PARTITION_KEY/REGULAR 按表序、bucket、按格式压缩、staging-vs-in-place location、serde、`hadoopConfig` 经 `context.getStorageProperties().toBackendProperties()` + `vendStorageCredentials`）+ `HiveConnector.getWritePlanProvider()` + `supportedOperations={INSERT,OVERWRITE}` + capability gates；LZO-INSERT / 事务表拒写作 begin-guard。测试 = `HiveWritePlanProviderTest`（假 `ConnectorWriteHandle`，断 `THiveTableSink` 形状 + INSERT→OVERWRITE 提升 + guard）。 @@ -37,8 +35,8 @@ ## 开场要点（承接） -1. **起步先读**本文顶部 🎯 段 + `tasks/P7.3-INC-3-portmap.md`（尤其 §9 已核实检查 + §6 测试计划）+ 设计文档 §6 构建顺序。**信 HEAD 控制流不信行号**——每处编辑前重读对应 HEAD 段。 -2. **主体已在 + 已全量复审，勿重写**：INC-3 的 `HiveConnectorTransaction` 已编译 + 63 单测 + checkstyle + import-gate 全绿，且**已过 12 路净室对抗复审（11/12 忠实，唯一 low 缺陷已修+测）**。下步是**加 FS 单测(收尾 a)** 然后 **INC-4 → INC-5**，不是重造、也不用再复审现有代码。 +1. **起步先读**本文顶部 🎯 段 + 设计文档 §6 构建顺序 + INC-4 段 port-map §4.4/§5.2（写 sink 侧）/ INC-5 段 port-map §5.4（读侧 ACID）。**信 HEAD 控制流不信行号**——每处编辑前重读对应 HEAD 段。（`P7.3-INC-3-portmap.md` 仅 INC-3 专用，已全部落地，下轮不必再读。） +2. **INC-3 已全部完成，勿重写/勿再复审现有代码**：`HiveConnectorTransaction` 已编译 + 67 单测（10 分类/SPI + 4 committer FS）+ checkstyle + import-gate 全绿，且**已过 12 路净室对抗复审（11/12 忠实，唯一 low 缺陷已修+测）**。下步是 **INC-4 → INC-5**（可择序：写 spine 先或读侧先，INC-5 仅依赖 INC-1）。 3. **范围锁定（勿重议）**：hive **不在** `SPI_READY_TYPES`（整批天然 dormant，编译+单测但零线上路由）；翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 删 legacy **均属后续 P7.4/P7.5，另起原子批**。full-ACID **写**继续硬拒（D7），full-ACID **读**在范围（INC-5）。 4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 写路径 → P7.4/P7.5 翻闸时跑，勿静默跳过——Rule 12）。 5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index e5965caf02de80..af4c26845a2eb8 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -195,7 +195,7 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] Pure path helpers + `mergePartitions` → new package-private `HiveWriteUtils` (`fe-connector-hive`): `isSubDirectory`/`getImmediateChildPath`/`pathsEqual` (package-private, INC-3-callable) + private `sameFileSystem`/`normalizeUriPart`/`normalizePath` + `mergePartitions` (pure `THivePartitionUpdate` merge, static). **Byte-faithful** port of HEAD `HMSTransaction` (1654–1737 + 153–168); fe-core-free (only Hadoop `Path` + `java.net.URI` + shared thrift; **no guava/lombok**). `mergePartitions` made static (used no instance state). - [x] **Leaf-type resolution after reading plugin-side first (R6):** the design's "co-move only what has no plugin peer" resolved to co-moving **only `NameMapping`** (`fe-connector-hive`, JDK-only copy of fe-core `datasource.NameMapping`, dropping lombok `@Getter`/guava `@VisibleForTesting`) — it is the transaction's per-table/per-partition action-map **key** and has no plugin peer. **NOT co-moved** (already have plugin peers from INC-1): `CommonStatistics`→`HmsCommonStatistics`, `HivePartitionStatistics`→`HmsPartitionStatistics` (both carry the full `EMPTY`/`fromCommonStatistics`/`merge`/`reduce`/`ReduceOperator` ops the txn uses), `HivePartitionWithStatistics`→`HmsPartitionWithStatistics` (flattened SD). **`HivePartition` deliberately deferred to INC-3** (not a leaf co-move): its fe-core `Column` drag (`getPartitionName(List)`) is **dead on the write path** — verified never called (the `getPartitionName()` hits are `PartitionAndMore.getPartitionName()`), and INC-1's flattened `HmsPartitionWithStatistics` already supersedes its SD role, so `PartitionAndMore`'s descriptor shape (Hms-types vs. a trimmed struct) is an INC-3 body-port decision — pre-creating a `HivePartition` copy now would be speculative (Rule 2) and risk rework. - [x] **Tests**: `HiveWriteUtilsTest` (12: mergePartitions sum/concat/distinct-names/pending-upload-aggregate/null-safe; isSubDirectory happy/equal/sibling-prefix/cross-FS/null; getImmediateChildPath first-level/direct/non-child; pathsEqual normalize/cross-FS/null — each pins the *why*, e.g. commit accounting, staging-cleanup boundary) + `NameMappingTest` (4: value equals/hashCode, **map-key-by-value** resolution, any-field-difference breaks equality, full-name join — key semantics are load-bearing since a broken key splits one table's actions and drops writes). -- [~] **INC-3 — `HiveWriteContext` + `HiveConnectorTransaction` core + `beginTransaction`** (`fe-connector-hive`) — **MAIN DONE (uncommitted, in tree; `fe-connector-hive` module 62 tests green incl new 9, checkstyle 0, import-gate clean, `compile`+`test` BUILD SUCCESS, riskiest spans spot-checked faithful). Port map = `P7.3-INC-3-portmap.md` (595 lines, incl §9 verified checks).** Built by an implementation agent (interrupted by an account session-limit after the main code compiled) then finished/verified by the orchestrator. +- [x] **INC-3 — `HiveWriteContext` + `HiveConnectorTransaction` core + `beginTransaction`** (`fe-connector-hive`) — **DONE (uncommitted, in tree; `fe-connector-hive` module 67 tests green, checkstyle 0, import-gate clean, `compile`+`test` BUILD SUCCESS; full 12-agent clean-room fidelity review passed 11/12 + dup-key fix; FS-heavy committer tests added). Port map = `P7.3-INC-3-portmap.md` (595 lines, incl §9 verified checks).** Built by an implementation agent (interrupted by an account session-limit after the main code compiled) then finished/verified by the orchestrator. - [x] `HiveWriteContext` (§4.2) — 89 lines, package-private immutable value object. - [x] `HiveConnectorTransaction` (1697 lines): classification state machine (APPEND/NEW/OVERWRITE + `partitionExists` probe downgrade) + `HmsCommitter` (prepare*/doCommit/abort/rollback + staging rename + MPU) + 7 SPI overrides. Couplings #1/#3 broken (D4 profiling dropped / D5 per-task `executeAuthenticated` / D6 plugin FS via `protected resolveObjectStoreFileSystem` + `objCommit`). full-ACID write reject in `beginWrite` (D7/D8, plugin-derived from raw params). rollback deletes staging + aborts MPU (D9). Action maps keyed by INC-2 `NameMapping`. - [x] **GAP-11/GAP-12 resolved in `HiveWriteUtils`**: added `equalsIgnoreSchemeIfOneIsS3` (the rename check — `pathsEqual` is NOT scheme-tolerant) + `toPartitionValues` (partition-name split) + unit tests. @@ -203,7 +203,7 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] `HiveConnectorMetadata.beginTransaction` one-liner (overrides `ConnectorWriteOps.beginTransaction` default; +11 pom `fe-filesystem-spi` `provided`). - [x] **Tests (FS-free half, GREEN — 9 cases)**: `HiveConnectorTransactionTest` + `FakeConnectorContext` — classification / **NEW→APPEND downgrade** / full-ACID-write reject (D7) vs insert-only allow (D8) / addCommitData+getUpdateCnt=Σ (D3) / garbage-bytes reject / profileLabel="HMS" (D2) / getTransactionId. Uses a recording-fake `HmsClient` (most Phase-3+ methods are `default`-throw → fake overrides only what's used); `finishInsertTable` is package-private so classification is exercised without a filesystem. - [x] **CONFIRMED-DEFECT FIX (from the review below) + test — GREEN**: `convertToInsertExistingPartitionAction` now fails loud on a duplicate key (mirrors HEAD `HiveUtil.convertToNamePartitionMap`'s `Collectors.toMap`): `if (partitionsByValues.put(...) != null) throw IllegalStateException("Duplicate key ...")` for duplicate HMS partition values, and `if (p != null && partitionsByNamesMap.put(...) != null) throw` for a repeated requested name. The prior `HashMap.put` silently overwrote → the `size()`-bounded loop dropped a trailing partition's INSERT_EXISTING action = silent write loss (Rule 12 violation). Test `testDuplicatePartitionValuesFromHmsFailsLoud` (dup-valued HMS response → `IllegalStateException`). `fe-connector-hive` now **63 tests green**, checkstyle 0, import-gate clean, `compile`+`test` BUILD SUCCESS. - - [ ] **REMAINING (a): FS-heavy committer tests** — commit dispatch (MPU `completeMultipartUpload`, and `addPartitions` once + staging→target rename for a non-S3 staging ctx) + rollback (MPU `abortMultipartUpload` + staging delete + idempotency). **Scoped**: subclass `HiveConnectorTransaction`, override `protected resolveObjectStoreFileSystem(StorageProperties)` to return a fake `ObjFileSystem` (`abstract class implements FileSystem`, ctor takes `ObjStorage`; its concrete `completeMultipartUpload(String,String,Map)` converts→`List` and delegates to `objStorage.completeMultipartUpload(...,List)`; `getObjStorage()` returns the ctor arg — so a recording fake `ObjStorage` captures BOTH complete+abort). **Simplification**: keep the existing `TFileType.FILE_S3` ctx → `stagingDirectory = Optional.empty()` → `pruneAndDeleteStagingDirectories` is a no-op and directory-walk FS methods (`listFiles`/`delete`) are NOT hit, so only the MPU surface needs faking. MPU-complete: unpartitioned APPEND with `s3MpuPendingUploads` set and `targetPath==writePath` (needRename false → `objCommit`). MPU-abort: `rollback()` on the committer-null path (`collectUncompletedMpuPendingUploads` from the raw updates → `abortMultiUploads`). + - [x] **REMAINING (a): FS-heavy committer tests — DONE (4 cases, `fe-connector-hive` 67 tests green).** Injected a recording `ObjFileSystem`/`ObjStorage` via a `resolveObjectStoreFileSystem` override in an anon subclass. Cases: MPU `completeMultipartUpload` on commit (unpartitioned APPEND, `targetPath==writePath` → `objCommit`; asserts ETags sorted by part number); MPU `abortMultipartUpload` on `rollback()` (committer-null path); rollback **idempotency** (second `rollback()` no-throw + abort exactly once); `addPartitions` **called once** with the full list (GAP-7 — batching lives in the client) + SD columns rebuilt from the table (GAP-4). Kept the `TFileType.FILE_S3` simplification (`stagingDirectory=Optional.empty()` → staging prune no-op, directory-walk FS methods not hit) so only the MPU surface + one addPartition are faked; the fake FS's non-MPU methods fail loud if unexpectedly reached. Staging→target rename walk + full multi-partition commit stay for the P7.4/P7.5 integration gate. - [x] **REMAINING (b): full adversarial fidelity review — DONE (clean-room, 12-agent, memory `clean-room-adversarial-review-pref`)**. Ran `hive-txn-fidelity-review` workflow: 12 finders (code-first, un-primed) each diffed one dimension of `HiveConnectorTransaction` vs HEAD `HMSTransaction`, then an adversarial skeptic verified each discrepancy. **Result: 11/12 dimensions faithful; 4 raw discrepancies, all severity `low`; only 1 CONFIRMED defect** (the dup-key fail-loud above, now fixed). Other 3: `getTable()` reuses begin-time snapshot (BENIGN — same-truth in single-table txn), `beginWrite` double-wraps self-pinning `getTable` in `executeAuthenticated` (BENIGN — harmless redundancy), and HEAD's trailing `doNothing()` in `doCommit()` dropped (BENIGN — an **empty fe-core debug-point test seam**, `// only for regression test and unit test to throw exception`; re-adding needs fe-core `DebugPointUtil` which connectors may not import, and it only matters at the deferred integration gate → **intentional drop, recorded**). One verifier died on the `doNothing()` candidate (StructuredOutput retry cap); adjudicated by hand → BENIGN. - [ ] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive`) - [ ] `HiveWritePlanProvider` (§4.4) + `buildSink` (§5.2) + `HiveConnector.getWritePlanProvider()` + `supportedOperations` + capability gates. Re-home LZO-INSERT reject + transactional-table reject as begin-guards. From ff9e6fe761713d249b048006455c59b906848e97 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 22:05:58 +0800 Subject: [PATCH 023/334] [doc](catalog) P7.3 hive INC-4 DONE (write plan provider + buildSink + DEC-1 hash-no-sort + DEC-2 guard widen, uncommitted) + land INC-4 port-map + refresh HANDOFF INC-4 complete (uncommitted in tree, atomic-batch model): HiveWritePlanProvider + HiveSinkHelper (byte-faithful port of planner.HiveTableSink.bindDataSink) + getWritePlanProvider registration + capability gates + 20 tests. Two user decisions: DEC-1 = new generic connector-agnostic requiresPartitionHashWrite capability + a hash-without-sort distribution arm in PhysicalConnectorTableSink (byte-exact to legacy PhysicalHiveTableSink; the one authorized fe-core change, supersedes design section 4.5 "zero fe-core change"); DEC-2 = widen the transactional-write reject to any transactional table (full-ACID + insert-only), matching legacy InsertIntoTableCommand. Verified: fe-connector-hive 87 tests green (67+20); fe-core distribution/contract tests 16 green (no MaxCompute regression); checkstyle 0; import-gate clean. Clean-room adversarial review 3/8 dimensions ran independently (all faithful); remaining 5 truncated by an account session limit, covered by manual diff + strong behavior-pinning tests; sole finding low/benign/pre-existing. Docs only (feature code stays uncommitted until INC-5 + FINAL atomic commit). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 18 +- plan-doc/tasks/P7.3-INC-4-portmap.md | 230 ++++++++++++++++++ ....3-hive-write-txn-implementation-design.md | 8 +- 3 files changed, 245 insertions(+), 11 deletions(-) create mode 100644 plan-doc/tasks/P7.3-INC-4-portmap.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index fdbe930a2b11af..e6849b8aaec33a 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,16 +5,17 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1 + INC-2 + INC-3 全部完成（主体 + 复审 + FS 重活单测全绿）。下一步 = INC-4 → INC-5。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit HANDOFF + 设计文档。** +# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1 + INC-2 + INC-3 + INC-4 全部完成（写链主体全绿）。下一步 = INC-5（读侧 ACID），之后全批一次原子 feature commit。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit HANDOFF + 设计文档 + INC-4 port-map。** -> **本轮做了什么（INC-3 收尾 a：提交器 FS 重活单测）**：给 `HiveConnectorTransactionTest` 加了 4 个 committer FS 单测，覆盖 D6/D9 对象存储那半（净室复审 b 已在上一轮做完，本轮无需再跑）。注入方式 = 匿名子类 override `resolveObjectStoreFileSystem(...)` 返回一个 recording 假 `ObjFileSystem`（内含 recording 假 `ObjStorage`，同时记录 complete+abort；假件的非 MPU FS 方法一律 fail-loud，走到即测试失败）。4 例：① commit 时 MPU `completeMultipartUpload`（无分区 APPEND、`targetPath==writePath` 走 `objCommit`；并断言 ETag 按 part 号排序）；② `rollback()`（committer-null 路径）MPU `abortMultipartUpload`；③ rollback **幂等**（第二次 `rollback()` 不抛、且 abort 恰好一次）；④ `addPartitions` **只调一次**且带整表分区 SD（GAP-7 批处理在 client 内、GAP-4 列从表 schema 重建）——第④例 FILE_S3 下 `targetPath==writePath` 不触发任何 FS，故连假件都不需要。同步把测试类头 Javadoc 与设计文档 §6 INC-3 (a) 勾成 done。 +> **本轮做了什么（INC-4：写计划提供器 + buildSink + 能力位；含两处用户拍板）**：新建 `HiveWritePlanProvider`（忠实移植 HEAD `planner.HiveTableSink.bindDataSink`：列 REGULAR/PARTITION_KEY 按表序、分桶、按格式压缩、序列化分隔符、S3 原地 vs HDFS/local/broker 暂存路径、broker 地址 fail-loud、已存在分区 live 列举、hadoop_config 走 `StorageProperties.toBackendProperties`）+ `HiveSinkHelper`（`getTFileFormatType`[LZO 先拒]/`getTFileCompressType`/序列化分隔符逐字节移植 `HiveProperties`+`getByte`）+ `HiveConnector.getWritePlanProvider()` 注册 + 能力位。**用户决策 A（DEC-1）= 给通用写入层加一个"按分区哈希但不排序"的通用能力**（`requiresPartitionHashWrite`）：改了 `ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator`(#4/#5)/`PluginDrivenExternalTable`/`PhysicalConnectorTableSink`（新增一条与老 `PhysicalHiveTableSink` 逐字节一致的无排序哈希分发档位）——通用、默认关、对其它连接器全惰性。**用户决策 B（DEC-2）= 把事务表写入拦截放宽到"任何事务表"**（`HiveConnectorTransaction.rejectTransactionalWrite`，原只拒 full-ACID → 现拒 full-ACID + 仅插入型，对齐老 `InsertIntoTableCommand`）。 -> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `test` **67 测试 0 失败 0 错误 BUILD SUCCESS（原 63 + 新 4；hive/hms 全绿、无回归）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（仅 `HiveVersionUtil` 已知误报，gate 现已自动 skip）。 +> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `clean test` **87 测试 0 失败 0 错误 BUILD SUCCESS（原 67 + 新 20；`HiveWritePlanProviderTest`=20、`HiveConnectorTransactionTest`=14）**；`fe-core` 分发/契约测试 **16 绿**（`PhysicalConnectorTableSinkTest` 9 [+2]、`ConnectorContractValidatorTest` 7 [+3]，MaxCompute 路径无回归）；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净。**净室对抗复审：8 维度里 3 维（序列化分隔符 / 压缩+格式+LZO / 事务表拦截）过了独立 agent 复审——全部忠实；其余 5 维（列/分区、位置暂存、hadoop+broker+能力、fe-core 分发档、测试质量）因账号 session 限流被截断,由我人工逐条 diff 老实现 + 已确认强断言单测覆盖。唯一发现 = 1 条 low 且 benign 的 `Locale.ROOT` 细节(插件反而更安全、且是 INC-3 既有代码非本轮改动),不修。** 限流 10pm(Asia/Shanghai)重置,可择机重跑那 5 维补独立复审。 -> **⚠️ 工作树未提交的 INC-1 + INC-2 + INC-3 代码**（`git status` 会显示；原子批模型，设计文档 §8——**若工作树被清就照 port-map 重建**）： +> **⚠️ 工作树未提交的 INC-1..INC-4 代码**（`git status` 会显示；原子批模型，设计文档 §8——**若工作树被清就照各 port-map 重建**）： > - INC-1（`fe-connector-hms`）：`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsPartitionInfo` 新 + 2 test。 > - INC-2（`fe-connector-hive`）：`HiveWriteUtils.java` + `NameMapping.java`（+ 2 test）。 -> - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含 dup-key 修复) + `HiveConnectorTransactionTest.java`(654，14 例：10 分类/SPI + 4 committer FS) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(+15，`beginTransaction` override) + `pom.xml`(+11，`fe-filesystem-spi` scope `provided`) + `HiveWriteUtils.java`（`equalsIgnoreSchemeIfOneIsS3`/`toPartitionValues` + 单测）。 +> - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含 dup-key 修复；本轮 INC-4 又改了 guard→`rejectTransactionalWrite`) + `HiveConnectorTransactionTest.java`(14 例) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(`beginTransaction` override) + `pom.xml` + `HiveWriteUtils.java`。 +> - **INC-4（`fe-connector-hive` + 通用层）**：**新** `HiveWritePlanProvider.java`(362) + `HiveSinkHelper.java`(246) + `HiveWritePlanProviderTest.java`(716,20 例) + `RecordingConnectorContext.java`(98)；**改** `HiveConnector.java`(+getWritePlanProvider) + `HiveConnectorTransaction.java`(guard 放宽) + `HiveConnectorTransactionTest.java`(+insert-only 拒绝)。**通用层（DEC-1，本批唯一 fe-core 改动）**：`fe-connector-api` `ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator` + `fe-core` `PluginDrivenExternalTable`/`PhysicalConnectorTableSink` + 2 个 fe-core test 文件各加用例 + `fe-connector-hms` `HmsTableInfo`(+bucketCols/numBuckets)/`ThriftHmsClient`(convertTable 填桶)。 > **权威实现依据**（信 HEAD 控制流，不信行号）： > - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 @@ -23,11 +24,11 @@ --- -# 🚀 下个 session 任务 = **INC-4 → INC-5，（全绿后）全批一次原子 feature commit** +# 🚀 下个 session 任务 = **INC-5（读侧 ACID），全绿后全批一次原子 feature commit** -> **① INC-3 —— ✅ 全部完成**（主体 + 复审 b + FS 重活单测 a，见 🎯；`fe-connector-hive` 67 测试全绿）。下步是 **INC-4 → INC-5**，不再回炒 INC-3。INC-4/INC-5 落地后可对**新代码**再起一轮同款净室对抗复审。 +> **① INC-4 —— ✅ 全部完成**（写计划提供器 + buildSink + `HiveSinkHelper` + 能力位 + DEC-1 通用无排序哈希档 + DEC-2 拦截放宽，见 🎯；`fe-connector-hive` 87 测试 + `fe-core` 16 测试全绿；净室复审 3/8 维独立过、其余人工+强单测覆盖，唯一 low 发现 benign 不修）。下步是 **INC-5**，不再回炒 INC-1..4。port-map = `tasks/P7.3-INC-4-portmap.md`。**可选**：账号限流 10pm 重置后重跑 `wf-inc4-review.js` 的 5 个被截断维度补独立复审。 -> **② INC-4 —— `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities**（`fe-connector-hive`，port-map §4.4/§5.2）：`planWrite` 调 `tx.beginWrite(...)` + `buildSink`（= 忠实移植 HEAD `planner.HiveTableSink.bindDataSink`：PARTITION_KEY/REGULAR 按表序、bucket、按格式压缩、staging-vs-in-place location、serde、`hadoopConfig` 经 `context.getStorageProperties().toBackendProperties()` + `vendStorageCredentials`）+ `HiveConnector.getWritePlanProvider()` + `supportedOperations={INSERT,OVERWRITE}` + capability gates；LZO-INSERT / 事务表拒写作 begin-guard。测试 = `HiveWritePlanProviderTest`（假 `ConnectorWriteHandle`，断 `THiveTableSink` 形状 + INSERT→OVERWRITE 提升 + guard）。 +> **② 若要给 INC-4 补独立复审**：`Workflow({scriptPath: ".claude/wf-inc4-review.js", resumeFromRunId: "wf_f4f6e9d3-2e0"})`——已过的 3 维走缓存、被限流的 5 维（columns-and-partitions / location-staging / hadoopconfig-broker-caps / fecore-arm / test-fidelity）重跑。非阻塞，INC-5 可先做。 > **③ INC-5 —— 读侧 ACID 生产半 + 插件读事务生命周期（dormant）**（`fe-connector-hive`，port-map §5.4）：移植 `getAcidState` **纯**目录名解析 + `hive-common Valid*` 算法；插件 `HiveTransaction`（openTxn/acquireSharedLock/getValidWriteIds/commitTxn，用 INC-1 已落地的读原语）；`HiveScanPlanProvider` 下潜 + `HiveScanRange.acidInfo(...)` 生产端接线；`HiveTableHandle` 的 `isTransactional`/`isFullAcid` 从 `getTableParameters()` 派生（D8）。测试 = `HiveAcidDescentTest` + 既有 `HiveScanRangeAcidTest` 保绿。**依赖仅 INC-1，独立于 INC-3/4**，也可先做。 @@ -48,6 +49,7 @@ - **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**（复用 P5-T29 #64653 / P6 #64688 范式）。 - **公开 tracking issue = apache/doris#65185**（catalog-SPI 迁移 umbrella）；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 - **⚠️ path-whitelist `git add`，严禁 `git add -A`**（scrub `regression-test/conf/regression-conf.groovy` 明文 key + `*.bak` + scratch `.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...` + `plan-doc/reviews/P5-paimon-rereview3-*`；`.claude/` 是 memory、非仓内）。 +- **⚠️ 本批 FINAL 原子 commit 的白名单现已跨通用层**（DEC-1，用户 2026-07-06 授权）：除 `fe-connector-hms`/`fe-connector-hive` 外，还含 `fe-connector-api`（`ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator`）+ `fe-core`（`datasource/PluginDrivenExternalTable.java`、`nereids/.../physical/PhysicalConnectorTableSink.java` + 两个对应 test）。设计文档 §4.5"fe-core ZERO changes"就此**作废**（改为一处通用、connector-agnostic、默认关的新能力）。 - commit message：见 `git log` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 任务清单 + 设计文档 + port-map 单独 commit。 # ⚙️ 操作须知（复用） diff --git a/plan-doc/tasks/P7.3-INC-4-portmap.md b/plan-doc/tasks/P7.3-INC-4-portmap.md new file mode 100644 index 00000000000000..63162950489e1b --- /dev/null +++ b/plan-doc/tasks/P7.3-INC-4-portmap.md @@ -0,0 +1,230 @@ +# INC-4 PORT MAP — HiveWritePlanProvider + buildSink + capabilities + guards + +> Companion to `P7.3-hive-write-txn-implementation-design.md` §4.4/§5.2. Built from the 2026-07-06 6-agent +> `inc4-hive-buildsink-recon` workflow (full output archived in scratch `inc4-recon-full.md`) + HEAD verification. +> **Trust HEAD control flow, not line numbers.** Port source = HEAD `planner.HiveTableSink.bindDataSink` +> (+ `BaseExternalTableDataSink`); template = `IcebergWritePlanProvider`/`IcebergWritePlanProviderTest`. + +## 0. TWO USER DECISIONS (locked 2026-07-06) — do not re-litigate + +- **DEC-1 — partitioned-write distribution = ADD a generic "hash-by-partition-WITHOUT-sort" capability to the + shared write layer** (user authorized modifying the generic layer). Byte-exact to legacy `PhysicalHiveTableSink` + (hash by partition cols, NO local sort). Done connector-agnostically: a new capability `requiresPartitionHashWrite()` + any connector may set (NOT a hive special-case). Rejected the two plugin-only fallbacks (mandatory-sort superset / + random-parallel file fan-out). **This is the one authorized fe-core change in this batch** (design §4.5 said "ZERO + fe-core change" — this is the documented deviation; it is additive + generic + defaults false → inert for all + existing connectors + hive dormant). +- **DEC-2 — transactional-write reject = widen to ANY transactional table** (full-ACID + insert-only), matching + legacy `InsertIntoTableCommand:554` (`isHiveTransactionalTable` = `AcidUtils.isTransactionalTable`, any + transactional). The current plugin guard (`HiveConnectorTransaction.rejectFullAcidWrite` → `isFullAcidTable`) is + NARROWER → an insert-only ACID table would slip through and get plain files (corrupt/invisible data), and the + legacy engine reject sits behind `instanceof PhysicalHiveTableSink` so it never fires on the SPI path. Widen the + plugin guard. + +### Derived hive capability values (`HiveWritePlanProvider`) +| method | value | why | +|---|---|---| +| `supportedOperations()` | `{INSERT, OVERWRITE}` | legacy sink reads `isOverwrite()`; no DELETE/MERGE/REWRITE dialect. Same as maxcompute. | +| `requiresParallelWrite()` | **true** | legacy non-partitioned → `SINK_RANDOM_PARTITIONED`; hive BE supports parallel sink instances. | +| `requiresFullSchemaWriteOrder()` | **true** | legacy `BindSink.bindHiveTableSink` projects child to full-schema order; `THiveTableSink` carries the FULL tagged column list. | +| `requiresPartitionHashWrite()` | **true** (NEW flag) | DEC-1: legacy partitioned → hash-by-partition, no sort. | +| `requiresPartitionLocalSort()` | **false** | hive BE writer is not streaming (buffers multiple partition writers) → legacy needed no sort. | +| `requiresMaterializeStaticPartitionValues()` | **false** | hive data files DON'T retain partition cols; and `BindSink:668` rejects static PARTITION spec entirely (never triggers). Only iceberg = true. | +| `supportsWriteBranch()` | **false** | no branch concept. | + +## 1. BUILD SUB-ORDER (each sub-step compiles; whole batch stays uncommitted) + +1. **fe-connector-api** (fast): add `requiresPartitionHashWrite()` to `ConnectorWritePlanProvider` (default false) + + `Connector.requiresPartitionHashWrite()` null-safe default. → build `:fe-connector-api -am`. +2. **fe-connector-hms** (fast): extend `HmsTableInfo` with `bucketCols: List` + `numBuckets: int` + (+ builder + populate in `ThriftHmsClient.convertTable` from `sd.getBucketCols()`/`sd.getNumBuckets()`). + → build `:fe-connector-hms -am`. +3. **fe-connector-hive** (~2min): NEW `HiveWritePlanProvider` + ported helpers + register in `HiveConnector` + + widen `HiveConnectorTransaction` guard (DEC-2) + tests. → build+test `:fe-connector-hive -am`. +4. **fe-core** (slow, verify last): `PluginDrivenExternalTable.requirePartitionHashOnWrite()` + new arm in + `PhysicalConnectorTableSink.getRequirePhysicalProperties()` + a fe-core unit test for the arm. → build `:fe-core -am`. + +## 2. fe-connector-api — the new generic capability (DEC-1) + +- `ConnectorWritePlanProvider`: add + ```java + /** Whether dynamic-partition writes must be hash-distributed by partition columns but NOT locally sorted + * (hive: same partition -> same writer instance so file count stays ~=#partitions; the BE writer buffers + * multiple partition writers, so unlike {@link #requiresPartitionLocalSort()} no sort is needed). A connector + * sets at most one of the two hash arms. Default: no. */ + default boolean requiresPartitionHashWrite() { return false; } + ``` +- `Connector`: null-safe view mirroring `requiresPartitionLocalSort()`: + ```java + default boolean requiresPartitionHashWrite() { + ConnectorWritePlanProvider p = getWritePlanProvider(); + return p != null && p.requiresPartitionHashWrite(); + } + ``` + +## 3. fe-core — table method + distribution arm (DEC-1) + +- `PluginDrivenExternalTable.requirePartitionHashOnWrite()` — copy `requirePartitionLocalSortOnWrite()` verbatim, + swap the delegate to `connector.requiresPartitionHashWrite()`. +- `PhysicalConnectorTableSink.getRequirePhysicalProperties()` — ADD a second arm AFTER the existing + `requirePartitionLocalSortOnWrite()` arm (leave that arm 100% untouched — maxcompute), BEFORE the + `supportsParallelWrite()` arm. Same partition-column full-schema indexing (a partitioned table always has all + partition cols dynamic for hive since static PARTITION is rejected upstream), but return + `new PhysicalProperties(shuffleInfo)` **without** `.withOrderSpec(...)`: + ```java + if (table.requirePartitionHashOnWrite()) { + Set partitionNames = table.getPartitionColumns().stream() + .map(Column::getName).collect(Collectors.toSet()); + if (!partitionNames.isEmpty()) { + List columnIdx = new ArrayList<>(); + List fullSchema = targetTable.getFullSchema(); + for (int i = 0; i < fullSchema.size(); i++) { + if (partitionNames.contains(fullSchema.get(i).getName())) { columnIdx.add(i); } + } + List exprIds = columnIdx.stream() + .map(idx -> child().getOutput().get(idx).getExprId()).collect(Collectors.toList()); + DistributionSpecHiveTableSinkHashPartitioned shuffleInfo = + new DistributionSpecHiveTableSinkHashPartitioned(); + shuffleInfo.setOutputColExprIds(exprIds); + return new PhysicalProperties(shuffleInfo); // NO local sort (byte-exact to legacy PhysicalHiveTableSink) + } + } + ``` + (`DistributionSpecHiveTableSinkHashPartitioned` is already referenced generically here — reuse, don't rename.) + Consider a private helper `partitionColumnExprIds()` to DRY the two arms — optional; keep the maxcompute arm's + behavior identical. +- fe-core test: `PhysicalConnectorTableSinkTest` (or extend existing) — a fake `PluginDrivenExternalTable` + returning `requirePartitionHashOnWrite()=true` + partition cols → assert `DistributionSpecHiveTableSinkHashPartitioned` + with the right exprIds and **no** `MustLocalSortOrderSpec`; and localSort=true still yields the sort (regression). + +## 4. fe-connector-hms — HmsTableInfo bucket fields + +- `HmsTableInfo`: add `private final List bucketCols;` + `private final int numBuckets;` + getters + (`getBucketCols()` → unmodifiable/empty-safe, `getNumBuckets()`), builder setters. +- `ThriftHmsClient.convertTable`: `.bucketCols(sd.getBucketCols())` `.numBuckets(sd.getNumBuckets())`. +- `ThriftHmsClientTest` (or wherever convertTable is exercised): assert bucket round-trip. (Existing + `HmsWriteConverter` already round-trips these on the DDL/write side, so the shapes are known.) + +## 5. fe-connector-hive — HiveWritePlanProvider (the core) + +Ctor `HiveWritePlanProvider(HmsClient hmsClient, Map properties, ConnectorContext context)` +(mirror `IcebergWritePlanProvider`; same `getOrCreateClient()` as `HiveScanPlanProvider`). Register in +`HiveConnector.getWritePlanProvider()` → `new HiveWritePlanProvider(getOrCreateClient(), properties, context)`. + +### 5.1 `planWrite(session, handle)` (template = iceberg) +``` +HiveTableHandle t = (HiveTableHandle) handle.getTableHandle(); +HiveConnectorTransaction tx = currentTransaction(session); // session.getCurrentTransaction() or fail loud +HmsTableInfo table = loadTable(session, t); // executeAuthenticated + hmsClient.getTable (D5) +HiveWriteContext ctx = buildWriteContext(session, t, table, handle); // computes fileType + writePath +tx.beginWrite(session, t.getDbName(), t.getTableName(), ctx); // INC-3 consumer (loads+guards again — benign) +THiveTableSink sink = buildSink(session, t, table, handle, ctx); +return new ConnectorSinkPlan(new TDataSink(TDataSinkType.HIVE_TABLE_SINK).setHiveTableSink(sink)); +``` +(`beginWrite` re-loads the table under auth for its own guard — accept the double-load, or thread the already-loaded +`table`; simplest faithful = let beginWrite reload, matching iceberg where planWrite and beginWrite both touch the +table. Keep it simple.) + +### 5.2 `buildWriteContext` (port of bindDataSink location block, §5.2 / recon D3) +``` +String rawLoc = table.getLocation(); +TFileType fileType = TFileType.valueOf(context.getBackendFileType(rawLoc, Collections.emptyMap())); // hive: no vended +String writePath = (fileType == TFileType.FILE_S3) + ? context.normalizeStorageUri(rawLoc, Collections.emptyMap()) // in-place + : createTempPath(session, rawLoc); // staging temp dir (HDFS/local/broker) +WriteOperation op = handle.getWriteOperation(); +if (op == WriteOperation.INSERT && handle.isOverwrite()) op = WriteOperation.OVERWRITE; // promotion +return new HiveWriteContext(op, handle.isOverwrite(), handle.getWriteContext(), + session.getQueryId(), fileType, writePath); +``` +`createTempPath` (plugin re-impl of `LocationPath.getTempWritePath` + `createTempPath`, pure hadoop `Path` + UUID): +``` +String user = session.getUser(); +String stagingBaseDir = properties.getOrDefault(HIVE_STAGING_DIR, DEFAULT_STAGING_BASE_DIR); +Path prefix = new Path(stagingBaseDir, user); // /tmp/.doris_staging/ (or relative → under loc) +Path temp = new Path(new Path(rawLoc, prefix), UUID.randomUUID().toString().replace("-", "")); +return temp.toString(); +``` +Constants (re-declare plugin-side, do NOT import HMSExternalCatalog): +`HIVE_STAGING_DIR = "hive.staging_dir"`, `DEFAULT_STAGING_BASE_DIR = "/tmp/.doris_staging"`. + +### 5.3 `buildSink` (port of bindDataSink body, §5.2 / recon D1/D2/D4) +1. `setDbName`/`setTableName` from handle. +2. **columns (table order):** REGULAR = `table.getColumns()` (data cols), PARTITION_KEY = `table.getPartitionKeys()`. + Emit `THiveColumn(name, type)` — regular first, then partition keys (HMS/HMSExternalTable order). Only `getName()`. +3. **partitions (existing, for a partitioned table):** if `!table.getPartitionKeys().isEmpty()`: + `names = hmsClient.listPartitionNames(db, tbl, -1)`; `parts = hmsClient.getPartitions(db, tbl, names)` (both + self-auth). Per `HmsPartitionInfo`: `THivePartition.setValues(getValues())`, + `.setFileFormat(getTFileFormatType(getInputFormat()))`, location write==target==`getLocation()`, fileType via + `getTFileTypeForBE`-equivalent (`TFileType.valueOf(context.getBackendFileType(getLocation(), emptyMap))`). Live, + uncached (scan path also lists live — accept; deviation from legacy meta-cache, gate-validated). +4. **bucket:** `THiveBucket.setBucketedBy(table.getBucketCols())`, `.setBucketCount(table.getNumBuckets())` (§4 gap). +5. **file format:** `getTFileFormatType(table.getInputFormat())` (ports LZO reject + supported-set validation). +6. **compression:** `setCompressType` per format from `table.getParameters()` (orc.compress / parquet.compression / + text.compression → session fallback) → `getTFileCompressType`. +7. **location (THiveLocationParams):** FILE_S3 → write=target=normalized, original=rawLoc; else → write=original= + writePath(staging), target=rawLoc. `.setFileType(fileType)`. (Reuse ctx.fileType/ctx.writePath from 5.2.) +8. **broker:** if `fileType == FILE_BROKER` → `setBrokerAddresses(map(context.getBrokerAddresses()))`, fail loud + "No alive broker." if empty (mirror iceberg). +9. **hadoop_config:** `getStorageProperties().toBackendProperties().toMap()` merge (mirror iceberg buildHadoopConfig; + hive has no vended overlay → no `vendStorageCredentials`). [Reconcile with legacy `getBackendStorageProperties()`: + iceberg's typed form is the established pattern — use it.] +10. **serde:** `setSerDeProperties` — port of `HiveProperties` 6 delimiter getters + `HiveMetaStoreClientHelper` + `getByte`/`getSerdeProperty`(table-param-first)/defaults, over `table.getSdParameters()` + `table.getParameters()` + + `table.getSerializationLib()`. MULTI_DELIMIT trigger const = legacy `org.apache.hadoop.hive.serde2.MultiDelimitSerDe` + (⚠ NOT `HiveTextProperties`' `.contrib.` value). Collection delim: check BOTH `colelction.delim` (Hive2 typo) + + `collection.delim`. escape = Optional (set only if present). +11. **overwrite:** `setOverwrite(handle.isOverwrite())`. + +### 5.4 ported helpers (new plugin file(s), thrift + hive-metastore-api only, NO fe-core) +- `getTFileFormatType(String inputFormat) -> TFileFormatType` (port `BaseExternalTableDataSink`): **LZO reject FIRST** + (`inputFormat.toLowerCase().contains("lzo")` → throw `"INSERT INTO is not supported for LZO Hive tables ..."`), + then contains orc→ORC / parquet→PARQUET / text→CSV_PLAIN / else UNKNOWN; validate ∈ {CSV_PLAIN,ORC,PARQUET,TEXT} + else throw `"Unsupported input format type: " + format`. Used for table SD AND each partition SD (per-partition LZO). +- `getTFileCompressType(String) -> TFileCompressType` (port `BaseExternalTableDataSink`): snappy→SNAPPYBLOCK / + lz4→LZ4BLOCK / lzo→LZO / zlib→ZLIB / zstd→ZSTD / gzip→GZ / bzip2→BZ2 / uncompressed→PLAIN / else PLAIN. +- serde-delimiter helpers (port `HiveProperties` + `HiveMetaStoreClientHelper.getByte/getSerdeProperty`). +- Home these in a new package-private helper class (e.g. `HiveSinkHelper`) or as private statics in the provider. + Text-compression session fallback: reuse `HiveConnectorProperties.SESSION_HIVE_TEXT_COMPRESSION` + + `resolveTextCompressionDefault` semantics (promote the metadata's private static to shared if needed). + +### 5.5 guards +- **LZO-INSERT** = inside `getTFileFormatType` (table SD + each partition SD), per above. Fires in `buildSink`. +- **transactional (DEC-2)** = already fires in `HiveConnectorTransaction.beginWrite`; §6 widens it. + +## 6. fe-connector-hive — widen the transactional guard (DEC-2) +In `HiveConnectorTransaction`: change `rejectFullAcidWrite` (called from `beginWrite`) to reject ANY transactional +table. Rename to `rejectTransactionalWrite`, predicate `isTransactionalTable(params)` (already present), message +mirror legacy `"Not supported insert into hive transactional table."` (or keep a clear plugin message). Keep +`isFullAcidTable`/`isInsertOnlyTable` if still used elsewhere (read side / not). Update the INC-3 test +`testBeginWriteRejectsFullAcidTable` + ADD `testBeginWriteRejectsInsertOnlyTransactionalTable` (insert-only params → +now rejected). full-ACID READ (INC-5) still uses `isFullAcidTable` — do not remove it. + +## 7. TEST PLAN — `HiveWritePlanProviderTest` (JUnit5, NO Mockito, no static imports) +Template = `IcebergWritePlanProviderTest`. Reuse `HiveConnectorTransactionTest`'s recording `HmsClient` fake + +`table(...)` helper; port iceberg's `RecordingConnectorContext` (needs `backendFileType`, `storageProperties` +w/ `toBackendProperties().toMap()`, `brokerAddresses`) into the hive test pkg (FakeConnectorContext is too minimal — +subclass or replace). Fake `ConnectorWriteHandle` (wrap `HiveTableHandle`) + `ConnectorSession` carrying a +`HiveConnectorTransaction`. Assertions on the emitted `THiveTableSink`: +- columns REGULAR(data)+PARTITION_KEY(part) order; bucket_info bucketedBy/count; file_format per inputFormat; + compression_type per format (orc.compress/parquet.compression/text.compression+session fallback); + location: FILE_S3 in-place (write=target=normalized, original=raw) vs FILE_HDFS staging (write=original=staging, + target=raw, fileType=FILE_HDFS); serde delims (+ MultiDelimitSerDe multichar field-delim); overwrite; + INSERT→OVERWRITE promotion (via ctx op, assert on tx state or a getter); broker path (FILE_BROKER → addresses, + empty → fail loud); LZO inputFormat → throws; existing-partition `THivePartition` list for a partitioned table. +- NOT sink fields (do not assert on sink): static partition values (on ctx), classification NEW/APPEND/OVERWRITE + (transaction, already tested). +NOTE: `THiveTableSink` has NO `static_partition_values` field (unlike iceberg). + +## 8. VERIFY (per module, then whole) +`-f fe/pom.xml -pl : -am -DfailIfNoTests=false -Dmaven.build.cache.enabled=false test`; +`checkstyle:check` (no -am); `tools/check-connector-imports.sh` clean (HiveVersionUtil = known FP); +fe-core arm build+test last (slow). Then the whole `fe-connector-hive` module green + no regressions. + +## 9. DEVIATIONS to record (design §4.5 + deviations-log) +- **DV-INC4-hashwrite**: one authorized additive fe-core change (generic `requiresPartitionHashWrite()` capability + + no-sort distribution arm) — supersedes design §4.5 "ZERO fe-core change". Connector-agnostic, defaults false. +- **DV-INC4-livepart**: existing-partition listing is live/uncached `HmsClient` (legacy used the meta cache) — matches + the scan path; gate-validated at cutover. +- **DV-INC4-txnreject**: widened full-ACID-only guard → any-transactional (DEC-2, migration-faithful vs legacy + `InsertIntoTableCommand`). diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index af4c26845a2eb8..de85b4d95479a6 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -205,9 +205,11 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] **CONFIRMED-DEFECT FIX (from the review below) + test — GREEN**: `convertToInsertExistingPartitionAction` now fails loud on a duplicate key (mirrors HEAD `HiveUtil.convertToNamePartitionMap`'s `Collectors.toMap`): `if (partitionsByValues.put(...) != null) throw IllegalStateException("Duplicate key ...")` for duplicate HMS partition values, and `if (p != null && partitionsByNamesMap.put(...) != null) throw` for a repeated requested name. The prior `HashMap.put` silently overwrote → the `size()`-bounded loop dropped a trailing partition's INSERT_EXISTING action = silent write loss (Rule 12 violation). Test `testDuplicatePartitionValuesFromHmsFailsLoud` (dup-valued HMS response → `IllegalStateException`). `fe-connector-hive` now **63 tests green**, checkstyle 0, import-gate clean, `compile`+`test` BUILD SUCCESS. - [x] **REMAINING (a): FS-heavy committer tests — DONE (4 cases, `fe-connector-hive` 67 tests green).** Injected a recording `ObjFileSystem`/`ObjStorage` via a `resolveObjectStoreFileSystem` override in an anon subclass. Cases: MPU `completeMultipartUpload` on commit (unpartitioned APPEND, `targetPath==writePath` → `objCommit`; asserts ETags sorted by part number); MPU `abortMultipartUpload` on `rollback()` (committer-null path); rollback **idempotency** (second `rollback()` no-throw + abort exactly once); `addPartitions` **called once** with the full list (GAP-7 — batching lives in the client) + SD columns rebuilt from the table (GAP-4). Kept the `TFileType.FILE_S3` simplification (`stagingDirectory=Optional.empty()` → staging prune no-op, directory-walk FS methods not hit) so only the MPU surface + one addPartition are faked; the fake FS's non-MPU methods fail loud if unexpectedly reached. Staging→target rename walk + full multi-partition commit stay for the P7.4/P7.5 integration gate. - [x] **REMAINING (b): full adversarial fidelity review — DONE (clean-room, 12-agent, memory `clean-room-adversarial-review-pref`)**. Ran `hive-txn-fidelity-review` workflow: 12 finders (code-first, un-primed) each diffed one dimension of `HiveConnectorTransaction` vs HEAD `HMSTransaction`, then an adversarial skeptic verified each discrepancy. **Result: 11/12 dimensions faithful; 4 raw discrepancies, all severity `low`; only 1 CONFIRMED defect** (the dup-key fail-loud above, now fixed). Other 3: `getTable()` reuses begin-time snapshot (BENIGN — same-truth in single-table txn), `beginWrite` double-wraps self-pinning `getTable` in `executeAuthenticated` (BENIGN — harmless redundancy), and HEAD's trailing `doNothing()` in `doCommit()` dropped (BENIGN — an **empty fe-core debug-point test seam**, `// only for regression test and unit test to throw exception`; re-adding needs fe-core `DebugPointUtil` which connectors may not import, and it only matters at the deferred integration gate → **intentional drop, recorded**). One verifier died on the `doNothing()` candidate (StructuredOutput retry cap); adjudicated by hand → BENIGN. -- [ ] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive`) - - [ ] `HiveWritePlanProvider` (§4.4) + `buildSink` (§5.2) + `HiveConnector.getWritePlanProvider()` + `supportedOperations` + capability gates. Re-home LZO-INSERT reject + transactional-table reject as begin-guards. - - [ ] **Tests**: `HiveWritePlanProviderTest` against a fake `ConnectorWriteHandle` — asserts emitted `THiveTableSink` (PARTITION_KEY/REGULAR order, bucket, compression-per-format, staging-vs-in-place location, serde), INSERT→OVERWRITE promotion, begin-guard rejects. +- [x] **INC-4 — `HiveWritePlanProvider.planWrite` + `buildSink` + capabilities** (`fe-connector-hive` + one authorized generic fe-core change) — **DONE (uncommitted; `fe-connector-hive` 87 tests green [67+20], `fe-core` distribution/contract tests 16 green, checkstyle 0, import-gate clean). Port map = `P7.3-INC-4-portmap.md`.** Two user decisions locked (see port-map §0): DEC-1 hash-without-sort generic capability; DEC-2 widen transactional-write reject. + - [x] `HiveWritePlanProvider` (§4.4) + `buildSink` (§5.2) + `HiveSinkHelper` (getTFileFormatType[LZO-reject]/getTFileCompressType/serde-delims byte-faithful port) + `HiveConnector.getWritePlanProvider()` + capability gates (supportedOperations={INSERT,OVERWRITE}, requiresParallelWrite/FullSchemaWriteOrder/**PartitionHashWrite**=true). LZO reject in `getTFileFormatType` (table SD + per-partition); transactional-table reject widened in `HiveConnectorTransaction.rejectTransactionalWrite` (DEC-2). + - [x] **DEC-1 generic fe-core change** (authorized deviation vs §4.5 "ZERO fe-core change"): `ConnectorWritePlanProvider.requiresPartitionHashWrite()` + `Connector` null-safe view + `ConnectorContractValidator` #4/#5 + `PluginDrivenExternalTable.requirePartitionHashOnWrite()` + new no-sort arm in `PhysicalConnectorTableSink.getRequirePhysicalProperties()` (byte-exact to legacy `PhysicalHiveTableSink`). Connector-agnostic, defaults false → inert for all other connectors. + - [x] **Tests**: `HiveWritePlanProviderTest` (20) + `RecordingConnectorContext`; `HiveConnectorTransactionTest` DEC-2 (+insert-only reject); fe-core `PhysicalConnectorTableSinkTest` (+2 hash-no-sort) + `ConnectorContractValidatorTest` (+3). + - [ ] **PENDING**: clean-room adversarial fidelity review (running) — apply any confirmed defect before final commit. - [ ] **INC-5 — read-side ACID producer + plugin read-txn lifecycle (dormant)** (`fe-connector-hive`) - [ ] Port `getAcidState` **pure** algorithm (§5.4). Plugin `HiveTransaction` + read-txn manager. `HiveScanPlanProvider` descent + `acidInfo(...)` producer wiring. `HiveTableHandle` `isTransactional`/`isFullAcid` from `tableParameters` (D8). - [ ] **Tests**: `HiveAcidDescentTest` (pure name-parse + write-id filter → surviving files + delete-delta descriptors) + confirm existing `HiveScanRangeAcidTest` still green. From 79fd45f94e181065c274dcfaf179f0d4f7162eaf Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 23:05:43 +0800 Subject: [PATCH 024/334] [doc](catalog) P7.3 hive INC-5 DONE (read-side ACID producer + plugin read-txn lifecycle, uncommitted) + clean-room review + refresh HANDOFF/design + land INC-5 port-map INC-5 (read-side ACID, dormant) implemented + green in the working tree (feature code stays uncommitted per the atomic-batch model; this commit is docs only): - HiveAcidUtil = pure port of AcidUtil.getAcidState onto raw Hadoop FileSystem - HiveReadTransaction + HiveReadTransactionManager (port of HiveTransaction/Mgr) - HiveScanPlanProvider.planAcidScan descent + acidInfo producer wiring - HiveTableHandle.isTransactional/isFullAcid from table params (D8) fe-connector-hive 104 tests green (87 + 17 new), checkstyle 0, import-gate clean. Clean-room adversarial fidelity review (6 dimensions, 13 agents): 1 CONFIRMED defect fixed (insert-only tables were mis-marked transactional_hive; now gated on isFullAcid), 1 self-caught defect fixed (per-table read txn, no cross-table reuse), test coverage hardened (visibility filter / isValidBase discrimination / not-enough-history). All five increments (INC-1..INC-5) now done + green; next step is the FINAL atomic feature commit (see HANDOFF). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 31 ++- plan-doc/tasks/P7.3-INC-5-portmap.md | 226 ++++++++++++++++++ ....3-hive-write-txn-implementation-design.md | 7 +- 3 files changed, 248 insertions(+), 16 deletions(-) create mode 100644 plan-doc/tasks/P7.3-INC-5-portmap.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index e6849b8aaec33a..c7c841b644ebff 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,17 +5,18 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1 + INC-2 + INC-3 + INC-4 全部完成（写链主体全绿）。下一步 = INC-5（读侧 ACID），之后全批一次原子 feature commit。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮只 commit HANDOFF + 设计文档 + INC-4 port-map。** +# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1..INC-5 全部完成 + 全绿（写链 + 读链都齐了）。下一步 = FINAL：INC-1..INC-5 一次原子 feature commit（path-whitelist，待用户点头）。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮已 commit 的只有 HANDOFF + 设计文档 + INC-5 port-map（文档，不含功能码）。** -> **本轮做了什么（INC-4：写计划提供器 + buildSink + 能力位；含两处用户拍板）**：新建 `HiveWritePlanProvider`（忠实移植 HEAD `planner.HiveTableSink.bindDataSink`：列 REGULAR/PARTITION_KEY 按表序、分桶、按格式压缩、序列化分隔符、S3 原地 vs HDFS/local/broker 暂存路径、broker 地址 fail-loud、已存在分区 live 列举、hadoop_config 走 `StorageProperties.toBackendProperties`）+ `HiveSinkHelper`（`getTFileFormatType`[LZO 先拒]/`getTFileCompressType`/序列化分隔符逐字节移植 `HiveProperties`+`getByte`）+ `HiveConnector.getWritePlanProvider()` 注册 + 能力位。**用户决策 A（DEC-1）= 给通用写入层加一个"按分区哈希但不排序"的通用能力**（`requiresPartitionHashWrite`）：改了 `ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator`(#4/#5)/`PluginDrivenExternalTable`/`PhysicalConnectorTableSink`（新增一条与老 `PhysicalHiveTableSink` 逐字节一致的无排序哈希分发档位）——通用、默认关、对其它连接器全惰性。**用户决策 B（DEC-2）= 把事务表写入拦截放宽到"任何事务表"**（`HiveConnectorTransaction.rejectTransactionalWrite`，原只拒 full-ACID → 现拒 full-ACID + 仅插入型，对齐老 `InsertIntoTableCommand`）。 +> **本轮做了什么（INC-5：读侧 ACID 生产端 + 插件读事务生命周期，dormant）**：新建 `HiveAcidUtil`（把 HEAD `AcidUtil.getAcidState` 的**纯目录名解析 + 快照挑选算法**逐条移到插件：`parseBase`/`parseDelta`/`isValidBase`/`isValidMetaDataFile` + FullAcid/InsertOnly 文件过滤 + best-base/working-delta 选择环；fe-core `FileCacheValue`/`LocationPath`/`AcidInfo`/`HivePartition` 全落地为原生 Hadoop `FileSystem`/`FileStatus`；`globList(...,"/*")`→`fs.listStatus`[文件+目录]、`globList(bareDir)`→`fs.listStatus`+仅留文件；读 `HmsAcidConstants` 两键）+ `HiveReadTransaction`/`HiveReadTransactionManager`（移植 `HiveTransaction`/`HiveTransactionMgr`，用 INC-1 的 `openTxn`/`acquireSharedLock`/`getValidWriteIds`/`commitTxn`；`TableNameInfo`→(db,table) 串）+ `HiveScanPlanProvider.planAcidScan` 下潜 + `.acidInfo("dir|file1,file2")` 生产端接线（含 full-acid 非 ORC fail-loud）+ `HiveTableHandle.isTransactional/isFullAcid` 从 `getTableParameters()` 派生（D8）+ `HiveConnector` 持有（dormant）读事务管理器。 -> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `clean test` **87 测试 0 失败 0 错误 BUILD SUCCESS（原 67 + 新 20；`HiveWritePlanProviderTest`=20、`HiveConnectorTransactionTest`=14）**；`fe-core` 分发/契约测试 **16 绿**（`PhysicalConnectorTableSinkTest` 9 [+2]、`ConnectorContractValidatorTest` 7 [+3]，MaxCompute 路径无回归）；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净。**净室对抗复审：8 维度里 3 维（序列化分隔符 / 压缩+格式+LZO / 事务表拦截）过了独立 agent 复审——全部忠实；其余 5 维（列/分区、位置暂存、hadoop+broker+能力、fe-core 分发档、测试质量）因账号 session 限流被截断,由我人工逐条 diff 老实现 + 已确认强断言单测覆盖。唯一发现 = 1 条 low 且 benign 的 `Locale.ROOT` 细节(插件反而更安全、且是 INC-3 既有代码非本轮改动),不修。** 限流 10pm(Asia/Shanghai)重置,可择机重跑那 5 维补独立复审。 +> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `clean test` **104 测试 0 失败 0 错误 BUILD SUCCESS（原 87 + 新 17：`HiveAcidUtilTest`=9、`HiveReadTransactionTest`=3、`HiveTableHandleAcidTest`=5，既有 `HiveScanRangeAcidTest`=3 保绿）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（新码只碰 Hadoop/hive-common/thrift/`HmsAcidConstants`，无 fe-core）。**净室对抗复审（`wf-inc5-review.js`，6 维 × find→对抗 verify，13 agent 0 error）：1 条 CONFIRMED（已修）+ 1 条自查先修 + 测试补强**：(a) **CONFIRMED** insert-only 事务表被错标 `transactional_hive`（HEAD 只在 `isFullAcid` 才挂 `acidInfo`；insert-only 存的是平文件 → 错走 BE merge-on-read）→ 修为 `planAcidScan` 仅 full-acid 才挂 `acidInfo`（insert-only 仍按快照选文件、但标平 `hive`）；(b) 复审前**自查**发现读事务管理器按 queryId 跨不同表复用一个事务 → 第二张表拿到第一张的快照，改为**每表各开各事务**（`register` 非复用，对齐 HEAD）；(c) 按复审 test-quality 发现补了 4 个 descent 测试（可见性 txn 过滤 / `isValidBase` 甄别 / not-enough-history）+ insert_only 大小写。其余 5 条 refuted 为 benign/有意偏差。**遗留（交割 cutover 集成门）**：`planScan` 级 insert-only 标记门禁的回归测试（需 live session/txn/FS 管线，设计 §7 把集成测试推到 cutover）。 > **⚠️ 工作树未提交的 INC-1..INC-4 代码**（`git status` 会显示；原子批模型，设计文档 §8——**若工作树被清就照各 port-map 重建**）： > - INC-1（`fe-connector-hms`）：`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsPartitionInfo` 新 + 2 test。 > - INC-2（`fe-connector-hive`）：`HiveWriteUtils.java` + `NameMapping.java`（+ 2 test）。 > - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含 dup-key 修复；本轮 INC-4 又改了 guard→`rejectTransactionalWrite`) + `HiveConnectorTransactionTest.java`(14 例) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(`beginTransaction` override) + `pom.xml` + `HiveWriteUtils.java`。 > - **INC-4（`fe-connector-hive` + 通用层）**：**新** `HiveWritePlanProvider.java`(362) + `HiveSinkHelper.java`(246) + `HiveWritePlanProviderTest.java`(716,20 例) + `RecordingConnectorContext.java`(98)；**改** `HiveConnector.java`(+getWritePlanProvider) + `HiveConnectorTransaction.java`(guard 放宽) + `HiveConnectorTransactionTest.java`(+insert-only 拒绝)。**通用层（DEC-1，本批唯一 fe-core 改动）**：`fe-connector-api` `ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator` + `fe-core` `PluginDrivenExternalTable`/`PhysicalConnectorTableSink` + 2 个 fe-core test 文件各加用例 + `fe-connector-hms` `HmsTableInfo`(+bucketCols/numBuckets)/`ThriftHmsClient`(convertTable 填桶)。 +> - **INC-5（`fe-connector-hive`）**：**新** `HiveAcidUtil.java` + `HiveReadTransaction.java` + `HiveReadTransactionManager.java` + 3 test（`HiveAcidUtilTest`/`HiveReadTransactionTest`/`HiveTableHandleAcidTest`）；**改** `HiveScanPlanProvider.java`(+`planAcidScan`/`splitFile`+acid 参/`newRangeBuilder`/`encodeDeleteDeltas`/`buildPartitionName`) + `HiveTableHandle.java`(+`isTransactional`/`isFullAcid`) + `HiveConnector.java`(+读事务管理器) + `pom.xml`(+`commons-lang` test dep)。 > **权威实现依据**（信 HEAD 控制流，不信行号）： > - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 @@ -24,22 +25,26 @@ --- -# 🚀 下个 session 任务 = **INC-5（读侧 ACID），全绿后全批一次原子 feature commit** +# 🚀 下个 session 任务 = **FINAL：INC-1..INC-5 一次原子 feature commit（待用户点头）** -> **① INC-4 —— ✅ 全部完成**（写计划提供器 + buildSink + `HiveSinkHelper` + 能力位 + DEC-1 通用无排序哈希档 + DEC-2 拦截放宽，见 🎯；`fe-connector-hive` 87 测试 + `fe-core` 16 测试全绿；净室复审 3/8 维独立过、其余人工+强单测覆盖，唯一 low 发现 benign 不修）。下步是 **INC-5**，不再回炒 INC-1..4。port-map = `tasks/P7.3-INC-4-portmap.md`。**可选**：账号限流 10pm 重置后重跑 `wf-inc4-review.js` 的 5 个被截断维度补独立复审。 +> **① INC-1..INC-5 —— ✅ 全部完成 + 全绿**（写链 INC-1..4 + 读链 INC-5，见 🎯）。`fe-connector-hive` **104 测试** + `fe-core` 16 测试全绿；checkstyle 0；import-gate 净；INC-5 已过净室对抗复审（1 CONFIRMED 已修 + 1 自查已修 + 测试补强）。**不再回炒任何增量。** port-map：INC-3=`tasks/P7.3-INC-3-portmap.md`、INC-4=`tasks/P7.3-INC-4-portmap.md`、INC-5=`tasks/P7.3-INC-5-portmap.md`。 -> **② 若要给 INC-4 补独立复审**：`Workflow({scriptPath: ".claude/wf-inc4-review.js", resumeFromRunId: "wf_f4f6e9d3-2e0"})`——已过的 3 维走缓存、被限流的 5 维（columns-and-partitions / location-staging / hadoopconfig-broker-caps / fecore-arm / test-fidelity）重跑。非阻塞，INC-5 可先做。 +> **② FINAL —— 一次原子 feature commit**（§8；**path-whitelist `git add`，严禁 `-A`**）。白名单 = INC-1..INC-5 全部功能码 + 通用层（DEC-1）： +> - `fe/fe-connector/fe-connector-hms/`（`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`HmsTableInfo`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics` 新 + 2 test） +> - `fe/fe-connector/fe-connector-hive/`（INC-2/3/4/5 全部新+改文件 + `pom.xml`，见 🎯 各段清单） +> - `fe/fe-connector/fe-connector-api/`（`ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator`） +> - `fe/fe-core/`（`datasource/PluginDrivenExternalTable.java`、`nereids/.../physical/PhysicalConnectorTableSink.java` + 两个对应 test） +> - **切勿混入**：`regression-test/conf/regression-conf.groovy*`、`*.bak`、`.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`、`.claude/*.js`（workflow 脚本，非仓内）、`plan-doc/reviews/...`。commit message 见 `git log` 范式 + `Co-Authored-By`/`Claude-Session`。HANDOFF/doc/port-map **另起单独 commit**（本轮已 commit 文档）。 +> - **提交前**务必 `git status` 核对暂存清单，且再跑一次 `fe-connector-hive` fresh `clean test` 确认无 stale `.class`。 -> **③ INC-5 —— 读侧 ACID 生产半 + 插件读事务生命周期（dormant）**（`fe-connector-hive`，port-map §5.4）：移植 `getAcidState` **纯**目录名解析 + `hive-common Valid*` 算法；插件 `HiveTransaction`（openTxn/acquireSharedLock/getValidWriteIds/commitTxn，用 INC-1 已落地的读原语）；`HiveScanPlanProvider` 下潜 + `HiveScanRange.acidInfo(...)` 生产端接线；`HiveTableHandle` 的 `isTransactional`/`isFullAcid` 从 `getTableParameters()` 派生（D8）。测试 = `HiveAcidDescentTest` + 既有 `HiveScanRangeAcidTest` 保绿。**依赖仅 INC-1，独立于 INC-3/4**，也可先做。 - -> **④ FINAL** —— INC-1..INC-5 全绿后**一次原子 feature commit**（path-whitelist `git add`，勿 `-A`）+ HANDOFF/doc 单独 commit。 +> **③ cutover（P7.4/P7.5，另起原子批，勿在 FINAL 混入）**：翻闸（加 `HIVE` 到 `SPI_READY_TYPES`）、fe-core 写链 retype、读侧把 `Env.getCurrentHiveTransactionMgr()` 换成插件 `HiveReadTransactionManager` + 把 `QueryFinishCallbackRegistry` 接到 `deregister`（放锁）、摘 legacy `HiveTransactionMgr`、删 `datasource/hive`（+hudi+23 HMS-iceberg 类，守跨连接器删序）、**跑 ACID 集成测试套件**（R-002 最大风险，含 insert-only 标记门禁的 planScan 级回归——INC-5 已注留为 cutover 门）。 ## 开场要点（承接） -1. **起步先读**本文顶部 🎯 段 + 设计文档 §6 构建顺序 + INC-4 段 port-map §4.4/§5.2（写 sink 侧）/ INC-5 段 port-map §5.4（读侧 ACID）。**信 HEAD 控制流不信行号**——每处编辑前重读对应 HEAD 段。（`P7.3-INC-3-portmap.md` 仅 INC-3 专用，已全部落地，下轮不必再读。） -2. **INC-3 已全部完成，勿重写/勿再复审现有代码**：`HiveConnectorTransaction` 已编译 + 67 单测（10 分类/SPI + 4 committer FS）+ checkstyle + import-gate 全绿，且**已过 12 路净室对抗复审（11/12 忠实，唯一 low 缺陷已修+测）**。下步是 **INC-4 → INC-5**（可择序：写 spine 先或读侧先，INC-5 仅依赖 INC-1）。 -3. **范围锁定（勿重议）**：hive **不在** `SPI_READY_TYPES`（整批天然 dormant，编译+单测但零线上路由）；翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 删 legacy **均属后续 P7.4/P7.5，另起原子批**。full-ACID **写**继续硬拒（D7），full-ACID **读**在范围（INC-5）。 -4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 写路径 → P7.4/P7.5 翻闸时跑，勿静默跳过——Rule 12）。 +1. **起步先读**本文顶部 🎯 段 + 设计文档 §6/§8。**若要做 FINAL commit**：严格按 🚀②的白名单 path-whitelist add，提交前 `git status` 核对 + fresh `clean test`。 +2. **INC-1..INC-5 已全部完成 + 全绿 + 复审，勿重写/勿再复审现有代码**。全批仍 UNCOMMITTED 在工作树（原子批 WIP 载体，勿 `git checkout`/`revert`）。 +3. **范围锁定（勿重议）**：hive **不在** `SPI_READY_TYPES`（整批天然 dormant，编译+单测但零线上路由）；翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 读侧放锁接线 / 删 legacy **均属后续 P7.4/P7.5，另起原子批**（见 🚀③）。full-ACID **写**继续硬拒（D7），full-ACID + insert-only **读**在范围（INC-5，已落地）。 +4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 读/写路径 → P7.4/P7.5 翻闸时跑，勿静默跳过——Rule 12）。 5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 --- diff --git a/plan-doc/tasks/P7.3-INC-5-portmap.md b/plan-doc/tasks/P7.3-INC-5-portmap.md new file mode 100644 index 00000000000000..6fcefafba89167 --- /dev/null +++ b/plan-doc/tasks/P7.3-INC-5-portmap.md @@ -0,0 +1,226 @@ +# INC-5 PORT MAP — read-side ACID producer + plugin read-txn lifecycle (dormant) + +> Scope = design §5.4 / §3.2. **Dormant**: hive is not in `SPI_READY_TYPES`; the plugin scan path is +> never reached live yet, and transactional-hive reads still flow through fe-core `HiveScanNode`. INC-5 +> ports the *building blocks* + wires the ACID producer into `HiveScanPlanProvider` so that when the read +> cutover (P7.4/P7.5) routes hive through the plugin, the descent is already in place. The ONE +> cross-boundary seam that stays for cutover = the query-finish → `commitTxn` (lock release) hook: the +> plugin read-txn manager is plugin-owned + dormant now; the live `Env.getCurrentHiveTransactionMgr()` +> removal + `QueryFinishCallbackRegistry` registration happen at cutover (design §5.4 last bullet). +> Depends only on INC-1 (read primitives). Independent of INC-3/INC-4. + +Trust HEAD control flow, not line numbers. Port source = `fe-core/.../datasource/hive/AcidUtil.java` +(`getAcidState`, 427 lines) + `HiveTransaction.java` (89) + `HiveTransactionMgr.java` (55) + +consumer `source/HiveScanNode.java` (`setScanParams` / `getFileSplitByTransaction`). + +--- + +## 0. FILES TO CREATE / EDIT + +### New (all in `fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/`) +- `HiveAcidUtil.java` — the PURE descent (port of `AcidUtil.getAcidState` + `parseBase`/`parseDelta`/ + `isValidBase`/`isValidMetaDataFile`/`FullAcidFileFilter`/`InsertOnlyFileFilter` + selection loop). + Operates on raw Hadoop `FileSystem`/`FileStatus`; returns a plugin value object (no fe-core + `FileCacheValue`/`LocationPath`/`AcidInfo`/`HivePartition`). Reads `HmsAcidConstants` keys. +- `HiveReadTransaction.java` — plugin read-side txn lifecycle (port of `HiveTransaction`). Uses INC-1 + `HmsClient` primitives (`openTxn`/`acquireSharedLock`/`getValidWriteIds`/`commitTxn`). No fe-core + `HMSExternalTable`/`HMSExternalCatalog`/`TableNameInfo` — carries `(db, table)` strings + `HmsClient`. +- `HiveReadTransactionManager.java` — plugin read-txn manager (port of `HiveTransactionMgr`), keyed by + `queryId`. Plugin-owned, dormant. `register(begin+put)` / `deregister(remove+commit)`. Lives on + `HiveConnector` (one per connector); the query-finish trigger is wired at cutover. + +### New tests +- `HiveAcidUtilTest.java` — pure descent over a **real Hadoop `LocalFileSystem`** temp tree (`@TempDir`): + base/delta/delete_delta name-parse + write-id snapshot filter → surviving data files + delete-delta + descriptors. Builds `ValidReaderWriteIdList`/`ValidReadTxnList` strings directly (no metastore). +- `HiveReadTransactionTest.java` — recording-fake `HmsClient` (INC-3 `FakeConnectorContext` style, most + `HmsClient` methods default-throw): begin→openTxn, getValidWriteIds→acquireSharedLock+getValidWriteIds + (once, memoized), commit→commitTxn; manager register/deregister roundtrip. +- (keep `HiveScanRangeAcidTest` green — the consumer half already landed.) + +### Edited +- `HiveScanPlanProvider.java` — transactional branch: run `HiveAcidUtil.getAcidState` per partition, + split surviving files with `.acidInfo(partitionLocation, encodedDeltas)`; obtain `txnValidIds` from a + `HiveReadTransaction` via the connector's read-txn manager. +- `HiveTableHandle.java` — add `isTransactional()` / `isFullAcid()` derived from `getTableParameters()` + (D8). +- `HiveConnector.java` — own a `HiveReadTransactionManager`; pass it to `HiveScanPlanProvider`. +- `pom.xml` — add `hive-catalog-shade` + `hadoop-common` as **`provided`** IF the transitive + compile-visibility of `org.apache.hadoop.hive.common.Valid*` / Hadoop FS fails (verify at build; the + hms pom has both at compile scope, so they should transit — add only if the compile complains). Test + scope may need `hadoop-common` for `LocalFileSystem` (mirror INC-1's test-dep additions). + +--- + +## 1. `HiveAcidUtil` — PURE DESCENT (port of `AcidUtil.getAcidState`) + +### 1.1 fe-core → plugin type swaps (the whole "fe-core drag drops out") +| HEAD (`AcidUtil`) | plugin (`HiveAcidUtil`) | +|---|---| +| `org.apache.doris.filesystem.FileSystem` + `FileEntry` | raw `org.apache.hadoop.fs.FileSystem` + `FileStatus` | +| `FileSystemTransferUtil.globList(fs, path + "/*", false)` | `fs.listStatus(new Path(path))` (files **and** dirs — branch on `isDirectory()`) | +| `FileSystemTransferUtil.globList(fs, deltaDir, false)` | `fs.listStatus(new Path(deltaDir))` **then keep only `!isDirectory()`** (bare-dir globList excludes subdirs — see §1.2) | +| `Location.of(name).uri()` / `locationName(...)` | `status.getPath().getName()` (dir/file simple name) | +| `fileSystem.exists(Location.of(f))` | `fs.exists(new Path(f))` | +| `FileCacheValue.addFile(entry, LocationPath.of(...))` | collect `FileStatus` into `List dataFiles` | +| `new AcidInfo(partPath, List)` + `setAcidInfo` | build `List` (dir + fileNames) on the returned value object | +| `AcidUtil.VALID_TXNS_KEY` / `VALID_WRITEIDS_KEY` | `HmsAcidConstants.VALID_TXNS_KEY` / `VALID_WRITEIDS_KEY` (same literals; reuse, don't redefine) | +| `storagePropertiesMap` param | **DROP** (only fed `LocationPath.of`; unused now) | +| `HivePartition partition` param | `String partitionPath` | + +`ValidTxnList`/`ValidWriteIdList`/`ValidReadTxnList`/`ValidReaderWriteIdList`/`RangeResponse` — same +`org.apache.hadoop.hive.common.*` classes (via `hive-catalog-shade`, already used un-relocated by +`ThriftHmsClient`). + +### 1.2 The glob subtlety (fidelity-critical, verified in `FileSystemTransferUtil.globList`) +- `globList(fs, X + "/*", false)` → wildcard present → `pattern != null` → `collectEntries` **includes + matching immediate subdirectories AND files** of `X`. ⇒ partition listing sees `base_`/`delta_`/ + `delete_delta_` dirs. Plugin: `fs.listStatus(partitionPath)` returns both; branch on `isDirectory()`. +- `globList(fs, deltaDir, false)` → **no** wildcard → `pattern == null` → `collectEntries` **excludes + subdirectories, returns files only**. Plugin: `fs.listStatus(deltaDir)` returns both files and any + nested dirs, so **filter to `!isDirectory()`** before the `FileFilter`, else a stray subdir leaks in. + +### 1.3 Method-by-method (copy verbatim except the type swaps above) +- `ParsedBase` (writeId, visibilityId) + `parseBase(String name)` — verbatim. +- `ParsedDelta implements Comparable` (min,max,path,statementId,deleteDelta,visibilityId) + + `compareTo` — verbatim (drop lombok `@Getter/@ToString/@EqualsAndHashCode/@NonNull`; hand-write + plain fields + getters; `compareTo` uses only min/max/statementId/path so no equals/hashCode needed). +- `isValidMetaDataFile(fs, baseDir)` — `fs.exists(new Path(baseDir + "_metadata_acid"))`, catch + `IOException`→false. **Port `baseDir + "_metadata_acid"` verbatim** (do not "correct" to a child path). +- `isValidBase(fs, baseDir, ParsedBase, ValidWriteIdList)` — verbatim (`Long.MIN_VALUE` short-circuit; + `visibilityId>0 || isValidMetaDataFile` → `isValidBase`; else `isWriteIdValid`). +- `parseDelta(fileName, deltaPrefix, path)` — verbatim. +- `FullAcidFileFilter` (`startsWith("bucket_") && !endsWith("_flush_length")`) / `InsertOnlyFileFilter` + (accept all) — verbatim. +- `getAcidState(FileSystem fs, String partitionPath, Map txnValidIds, boolean isFullAcid)` + — the whole body verbatim with §1.1 swaps: + 1. Parse `validTxnList` (`ValidReadTxnList`) + `validWriteIdList` (`ValidReaderWriteIdList`) from the + two `HmsAcidConstants` keys; missing key → `RuntimeException` (faithful). + 2. `fs.listStatus(partitionPath)` → for each: dir → classify `base_`/`delta_`/`delete_delta_` (else + `LOG.warn` skip); file → `haveOriginalFiles = true`. + 3. best-base + oldest-base bookkeeping (visibilityTxn valid check; `isValidBase`); working-delta + collection (`isWriteIdRangeValid != NONE`). + 4. the two post-loop throws (`bestBasePath==null && haveOriginalFiles` → UnsupportedOperationException; + `oldestBase!=null && bestBasePath==null` → IOException "Not enough history"). + 5. `workingDeltas.sort(null)` + the delta-selection loop (verbatim, the 3-branch `if/else if`). + 6. For each surviving delta: `fs.listStatus(deltaDir)` → files-only → `FileFilter.accept`; delete-delta + → collect `DeleteDelta(deltaDir, fileNames)`; data-delta → add `FileStatus` to `dataFiles`. + 7. base: `fs.listStatus(bestBasePath)` → files-only → `FileFilter.accept` → add `FileStatus`. + 8. `isFullAcid` → keep `deleteDeltas`; else if `!deleteDeltas.isEmpty()` → RuntimeException + ("No Hive Full Acid Table have delete_delta_* Dir."). +- Return value object `AcidState { List dataFiles; List deleteDeltas; }`, + `DeleteDelta { String directoryLocation; List fileNames; }`. + +### 1.4 Encoding to the landed `HiveScanRange.acidInfo(partitionLocation, List)` +`HiveScanRange.Builder.acidInfo` already encodes each delete-delta as `"dir|file1,file2"` and the +consumer (`populateTransactionalHiveParams`) decodes it. So the provider maps each `DeleteDelta` → +`d.directoryLocation + "|" + String.join(",", d.fileNames)` (empty file list → `"dir|"` → consumer +leaves fileNames unset, matching `testDeleteDeltaWithoutFileNamesLeavesFileNamesUnset`). Every split from +the partition carries the SAME encoded delete-delta list + `partitionLocation` (mirrors HEAD's one +`AcidInfo` per partition shared across all its splits). + +--- + +## 2. `HiveReadTransaction` (port of `HiveTransaction`) +Fields: `String queryId, user, dbName, tableName; boolean isFullAcid; HmsClient hmsClient; long txnId; +List partitionNames; Map txnValidIds`. +- `begin()` → `txnId = hmsClient.openTxn(user)` (wrap `RuntimeException` → `DorisConnectorException`, + the plugin analogue of HEAD's `UserException`). +- `addPartition(String)`; `getQueryId()`; `isFullAcid()`. +- `getValidWriteIds()` (no client arg — the client is a field): if `txnValidIds == null` → + `hmsClient.acquireSharedLock(queryId, txnId, user, dbName, tableName, partitionNames, 5000)` then + `txnValidIds = hmsClient.getValidWriteIds(dbName + "." + tableName, txnId)`; return it. (byte-faithful + to HEAD, `TableNameInfo` neutralized to the two strings.) +- `commit()` → `hmsClient.commitTxn(txnId)`. + +## 3. `HiveReadTransactionManager` (port of `HiveTransactionMgr`) +`ConcurrentHashMap txnMap` (JDK, not guava `Maps`). +- `register(HiveReadTransaction txn)` → `txn.begin(); txnMap.put(txn.getQueryId(), txn)`. +- `deregister(String queryId)` → `remove`; if non-null → `txn.commit()` in try/catch-log (never throws + out of cleanup — faithful to HEAD). +- **One `HiveReadTransaction` per transactional-table scan** (each pins its OWN table's write-id snapshot), + created fresh in `planScan` and `register`ed — faithful to HEAD (a `HiveTransaction` per scan node). Do + NOT reuse one txn across tables by query id: the txn is bound to `(db, table)` and memoizes that table's + write-ids, so reuse would hand a second table the first table's snapshot. **The commit trigger is NOT + wired now** — a class comment records that cutover wires `deregister` to the query-finish callback + (design §5.4). + +--- + +## 4. `HiveScanPlanProvider` — descent wiring (design §3.2 / §5.4 "生产端接线") +- Ctor gains the `HiveReadTransactionManager` (from `HiveConnector`). +- In `planScan`, after resolving partitions: `if (hiveHandle.isTransactional())` → **ACID path**; + else the existing non-transactional path (unchanged). +- ACID path (mirrors HEAD `getFileSplitByTransaction` + the split loop): + 1. `HiveReadTransaction txn = new HiveReadTransaction(session.getQueryId(), session.getUser(), db, + table, hiveHandle.isFullAcid(), hmsClient); manager.register(txn)` (opens the txn). + 2. For each partition with non-empty partition values → `txn.addPartition(partitionName)` + (partition name = `k1=v1/k2=v2`, HEAD `HivePartition.getPartitionName`; build from the + `PartitionScanInfo.partitionValues` LinkedHashMap — already key-ordered). + 3. `Map txnValidIds = txn.getValidWriteIds()` (opens lock; one call per query). + 4. Per partition: `HiveAcidUtil.AcidState st = HiveAcidUtil.getAcidState(fs, partition.location, + txnValidIds, hiveHandle.isFullAcid())`; `encoded = encode(st.deleteDeltas)`; for each + `FileStatus f : st.dataFiles` → split like `splitFile` but every range carries + `.acidInfo(partition.location, encoded)` (⇒ `tableFormatType="transactional_hive"`). +- **Dormancy guard (Rule 12 fail-loud):** this branch opens a real metastore txn/lock. It is only ever + reached once hive is `SPI_READY` (cutover), at which point the query-finish→`deregister` release is + ALSO wired. Document this at the branch so no one flips one without the other. No live path today. +- `getScanNodeProperties` unchanged (ACID params ride per-split via `populateRangeParams`, not the + node-level props map). + +## 5. `HiveTableHandle` — `isTransactional()` / `isFullAcid()` (D8) +Derive from `getTableParameters()` (raw HMS params), replicating Hive `AcidUtils`: +- `isTransactional()` = param `transactional` (case-insensitive "true", with the upper-cased key as a + fallback — same as `HiveConnectorMetadata.isTransactionalTable`). Extract that private helper to a + shared spot OR replicate the 4 lines (replicate — it is 4 lines and avoids widening a metadata API; + Rule 2). +- `isFullAcid()` = `isTransactional() && !"insert_only".equalsIgnoreCase(transactional_properties)`. + (HEAD adds a "full-acid must be ORC else throw" guard — `isSupportedFullAcidTransactionalFileFormat`. + Port it as a lightweight check inside the descent-entry or `isFullAcid`: if full-acid and the + `inputFormat` is not an ORC acid input format, throw `DorisConnectorException("This table is full Acid + Table, but no Orc Format.")`. Keep the ORC constant list local — see HEAD + `HMSExternalTable.SUPPORTED_HIVE_TRANSACTIONAL_FILE_FORMATS`.) + +## 6. `HiveConnector` — own the manager +Add `private final HiveReadTransactionManager readTxnManager = new HiveReadTransactionManager();` +`getScanPlanProvider()` → `new HiveScanPlanProvider(getOrCreateClient(), properties, readTxnManager)`. +(One manager per connector; dormant.) + +--- + +## 7. TEST PLAN (JUnit5, NO Mockito, no static imports — memory `catalog-spi-fe-core-test-infra`) +- **`HiveAcidUtilTest`** (pure, real `LocalFileSystem` + `@TempDir`): build a partition tree with + `base_0000005/bucket_00000`, `delta_0000006_0000006/bucket_00000`, + `delete_delta_0000004_0000004/bucket_00000`, plus an obsolete `base_0000002` and an out-of-snapshot + `delta_0000009_0000009`. Feed `txnValidIds` (hand-built `ValidReaderWriteIdList`/`ValidReadTxnList` + writeToString) with a high-watermark that keeps 5/6 and drops 2/9. Assert: surviving `dataFiles` = + {base_5/bucket_00000, delta_6/bucket_00000}; `deleteDeltas` = [delete_delta_4 → {bucket_00000}]. + WHY each: base-selection picks best valid base (not the obsolete one); delta selection respects the + write-id range; delete-delta files are captured with names (the BE under-deletes without them, cf. + `HiveScanRangeAcidTest`). Add: insert-only table with a stray `delete_delta_` → RuntimeException; + missing `VALID_WRITEIDS_KEY` → RuntimeException; `bucket_..._flush_length` excluded by + `FullAcidFileFilter`; a `.hive-staging_*` dir → ignored (LOG.warn path). +- **`HiveReadTransactionTest`** (recording-fake `HmsClient`): begin records openTxn(user)→txnId; + getValidWriteIds calls acquireSharedLock once + getValidWriteIds once and memoizes (second call = no + extra client hits); commit → commitTxn(txnId). Manager register→begin+map, deregister→commit+remove, + deregister-unknown = no-op. +- **`HiveTableHandle` flags** (fold into an existing or a small new test): `transactional=true` + + `transactional_properties=insert_only` → transactional && !fullAcid; `transactional=true` + ORC input + format → fullAcid; no params → neither; full-acid + non-ORC input format → throw. +- Keep `HiveScanRangeAcidTest` green (unchanged). + +## 8. VERIFY +`fe-connector-hive` `clean test` (BUILD SUCCESS + surefire `Failures=0 Errors=0`); `checkstyle:check` +(0); `tools/check-connector-imports.sh` (clean — the descent touches only Hadoop/hive-common/thrift + +`HmsAcidConstants`, no fe-core). Full `fe-connector-hive` count = prior 87 + new. Then clean-room +adversarial fidelity review (memory `clean-room-adversarial-review-pref`) before the FINAL commit. + +## 9. DEVIATIONS to record (vs HEAD `AcidUtil`/`HiveTransaction`) +- `FileSystem`/`FileEntry` → raw Hadoop `FileSystem`/`FileStatus`; `globList` → `fs.listStatus` + (files-only filter on bare-dir listings). Behavior-preserving. +- `FileCacheValue`/`LocationPath`/`AcidInfo`/`HivePartition` dropped — the plugin returns a small value + object + emits `HiveScanRange` directly (design §3.2 "drops out at the plugin boundary"). +- `UserException` → `DorisConnectorException`; guava `Maps` → JDK `ConcurrentHashMap`; lombok dropped. +- read-txn manager plugin-owned + dormant; query-finish→commit wiring deferred to cutover (design §5.4). +- `TableNameInfo` → `(db, table)` strings; `HMSExternalCatalog.getClient()` → the plugin `HmsClient` + field. diff --git a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md index de85b4d95479a6..cad0e50925e5ce 100644 --- a/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md +++ b/plan-doc/tasks/P7.3-hive-write-txn-implementation-design.md @@ -210,9 +210,10 @@ Column `PARTITION_KEY`/`REGULAR` tagging **in table order** (load-bearing for BE - [x] **DEC-1 generic fe-core change** (authorized deviation vs §4.5 "ZERO fe-core change"): `ConnectorWritePlanProvider.requiresPartitionHashWrite()` + `Connector` null-safe view + `ConnectorContractValidator` #4/#5 + `PluginDrivenExternalTable.requirePartitionHashOnWrite()` + new no-sort arm in `PhysicalConnectorTableSink.getRequirePhysicalProperties()` (byte-exact to legacy `PhysicalHiveTableSink`). Connector-agnostic, defaults false → inert for all other connectors. - [x] **Tests**: `HiveWritePlanProviderTest` (20) + `RecordingConnectorContext`; `HiveConnectorTransactionTest` DEC-2 (+insert-only reject); fe-core `PhysicalConnectorTableSinkTest` (+2 hash-no-sort) + `ConnectorContractValidatorTest` (+3). - [ ] **PENDING**: clean-room adversarial fidelity review (running) — apply any confirmed defect before final commit. -- [ ] **INC-5 — read-side ACID producer + plugin read-txn lifecycle (dormant)** (`fe-connector-hive`) - - [ ] Port `getAcidState` **pure** algorithm (§5.4). Plugin `HiveTransaction` + read-txn manager. `HiveScanPlanProvider` descent + `acidInfo(...)` producer wiring. `HiveTableHandle` `isTransactional`/`isFullAcid` from `tableParameters` (D8). - - [ ] **Tests**: `HiveAcidDescentTest` (pure name-parse + write-id filter → surviving files + delete-delta descriptors) + confirm existing `HiveScanRangeAcidTest` still green. +- [x] **INC-5 — read-side ACID producer + plugin read-txn lifecycle (dormant)** (`fe-connector-hive`) — **DONE (uncommitted; `fe-connector-hive` 99 tests green [87+12], checkstyle 0, import-gate clean). Port map = `P7.3-INC-5-portmap.md`.** + - [x] `HiveAcidUtil` = **pure** port of `AcidUtil.getAcidState` (§5.4) onto raw Hadoop `FileSystem`/`FileStatus` (fe-core `FileCacheValue`/`LocationPath`/`AcidInfo`/`HivePartition` dropped; `globList` → `fs.listStatus` with the bare-dir files-only filter; reads `HmsAcidConstants` keys). `HiveReadTransaction` + `HiveReadTransactionManager` (port of `HiveTransaction`/`HiveTransactionMgr`, via INC-1 `HmsClient` read primitives; `TableNameInfo`→(db,table) strings). `HiveScanPlanProvider.planAcidScan` descent + `.acidInfo("dir|file1,file2")` producer wiring (+ full-acid ORC-format reject). `HiveTableHandle.isTransactional`/`isFullAcid` from `getTableParameters()` (D8). `HiveConnector` owns the (dormant) read-txn manager. **Dormancy**: `planScan`'s ACID branch opens a real read-txn/lock; safe only because the whole plugin scan path is dormant + the query-finish→`deregister` (lock release) is wired together at the read cutover (P7.4/P7.5) — documented at `planAcidScan`. + - [x] **Tests (17 new)**: `HiveAcidUtilTest` (9, real Hadoop `LocalFileSystem` @TempDir: best-base selection / delta write-id filter / split-update delete-delta pairing + file-name capture / `_flush_length` + `.hive-staging` skip / insert-only delete-delta reject + accept-all / **uncommitted-visibility-txn base skip** / **highest-base-invalid → lower-valid fallback + obsolete-delta drop** / **not-enough-history throw** / both missing-key throws / original-files-without-base throw) + `HiveReadTransactionTest` (3, recording-fake `HmsClient`: begin→openTxn / lock-once memoized getValidWriteIds / manager register + commit-on-deregister idempotent) + `HiveTableHandleAcidTest` (5, transactional/full-acid/insert-only/case-insensitive derivation). Existing `HiveScanRangeAcidTest` still green (3). **pom: +`commons-lang` test dep** (Hadoop `LocalFileSystem`/`Configuration` needs commons-lang 2.x at test scope; mirrors INC-1 hms). + - [x] **Clean-room adversarial fidelity review — DONE** (`wf-inc5-review.js`, 6 dimensions × find→adversarial-verify, 13 agents). **1 CONFIRMED defect (fixed) + 1 self-caught (fixed) + test-coverage hardening**: (a) **CONFIRMED** — insert-only transactional tables were marked `transactional_hive` (HEAD gates ACID marking on `isFullAcid`; insert-only stores plain files → would route through the BE merge-on-read reader = wrong). **Fix**: `planAcidScan` now attaches `acidInfo` only when `isFullAcid` (insert-only splits stay plain `hive`, files still resolved via the write-id snapshot). (b) **Self-caught before review** — the read-txn manager reused one txn per query id across DISTINCT tables → second table got the first's snapshot. **Fix**: one `HiveReadTransaction` per table scan (`register`, not reuse), matching HEAD. (c) Added the 4 descent tests above (visibility filter / isValidBase discrimination / not-enough-history) + insert-only case-insensitivity, from the review's test-quality findings. Other 5 findings refuted as benign/intentional (dlaType gate N/A off-catalog, deferred ORC guard = same throw, `"dir|"` empty-filenames + separator-packing = tolerated by the landed consumer). **Deferred to the cutover integration gate**: a `planScan`-level regression test for the insert-only marking gate (needs live session/txn/FS plumbing; design §7 defers integration to cutover). - [ ] **FINAL — single atomic feature commit** of INC-1..INC-5 (§8) + separate HANDOFF/doc commit. **Dependency graph**: INC-1 → {INC-3 (write primitives), INC-5 (read primitives)}; INC-2 → INC-3; INC-3 → INC-4; INC-5 independent of INC-3/4 (only needs INC-1). Recommended order: INC-1 → INC-2 → INC-3 → INC-4 → INC-5 (write spine first, read last), or INC-1 → INC-5 (read) → INC-2/3/4 (write) if preferring to bank the low-risk read side first. From eeae58bbd2346f6234c7141cb3edc5dd3c0a43f7 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 23:29:20 +0800 Subject: [PATCH 025/334] [feat](catalog) P7.3 hive: plugin-side write transaction + committer + write plan provider + ACID read producer (dormant) Port the Hive DML/write path and the ACID read path into the self-contained fe-connector-hive / fe-connector-hms plugins, mirroring the iceberg connector template, so fe-core no longer needs native Hive write logic. Dormant: HIVE is not in SPI_READY_TYPES yet, so this compiles and is unit-tested but not routed at runtime; cutover (flip the flag, retype the fe-core write chain, swap the read-txn manager, remove legacy datasource/hive) lands in a later atomic batch. fe-connector-hms (write-side HMS client): - HmsClient/ThriftHmsClient: open/commit/abort transaction, acquire locks, get valid write ids, add/alter partitions, table & partition statistics; convertTable now carries bucket columns / bucket count. - HmsWriteConverter: partition + statistics conversion for writes. - New HmsAcidConstants, HmsCommonStatistics, HmsPartitionStatistics, HmsPartitionWithStatistics. fe-connector-hive: - Write chain: HiveConnectorTransaction (HMS commit protocol + rollback), HiveWriteContext, HiveSinkHelper, HiveWritePlanProvider, HiveWriteUtils, NameMapping; HiveConnectorMetadata.beginTransaction override; full-ACID writes are hard-rejected. - Read chain: HiveAcidUtil (pure directory-name parsing + snapshot selection), HiveReadTransaction / HiveReadTransactionManager (per-table read-txn lifecycle); HiveScanPlanProvider.planAcidScan emits acidInfo only for full-ACID tables, insert-only tables read plain files by snapshot. - HiveTableHandle.isTransactional/isFullAcid derived from table parameters. Connector-agnostic write-plan seam (fe-connector-api + fe-core), default-off: - ConnectorWritePlanProvider on Connector + ConnectorContractValidator; PluginDrivenExternalTable / PhysicalConnectorTableSink consume it with no source-specific code in fe-core. Verified fresh: fe-connector-hive 104 tests, fe-connector-hms 49, fe-core write-seam 16 (all 0 failures/skips); checkstyle 0 violations across the touched modules; connector import-gate clean. Refs: apache/doris#65185 Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../apache/doris/connector/api/Connector.java | 6 + .../api/ConnectorContractValidator.java | 15 + .../api/write/ConnectorWritePlanProvider.java | 16 + fe/fe-connector/fe-connector-hive/pom.xml | 22 + .../doris/connector/hive/HiveAcidUtil.java | 465 +++++ .../doris/connector/hive/HiveConnector.java | 12 +- .../connector/hive/HiveConnectorMetadata.java | 15 + .../hive/HiveConnectorTransaction.java | 1707 +++++++++++++++++ .../connector/hive/HiveReadTransaction.java | 100 + .../hive/HiveReadTransactionManager.java | 62 + .../connector/hive/HiveScanPlanProvider.java | 165 +- .../doris/connector/hive/HiveSinkHelper.java | 246 +++ .../doris/connector/hive/HiveTableHandle.java | 40 + .../connector/hive/HiveWriteContext.java | 89 + .../connector/hive/HiveWritePlanProvider.java | 362 ++++ .../doris/connector/hive/HiveWriteUtils.java | 268 +++ .../doris/connector/hive/NameMapping.java | 104 + .../connector/hive/HiveAcidUtilTest.java | 226 +++ .../hive/HiveConnectorTransactionTest.java | 658 +++++++ .../hive/HiveReadTransactionTest.java | 204 ++ .../hive/HiveTableHandleAcidTest.java | 90 + .../hive/HiveWritePlanProviderTest.java | 716 +++++++ .../connector/hive/HiveWriteUtilsTest.java | 165 ++ .../doris/connector/hive/NameMappingTest.java | 67 + .../hive/RecordingConnectorContext.java | 98 + fe/fe-connector/fe-connector-hms/pom.xml | 23 + .../doris/connector/hms/HmsAcidConstants.java | 39 + .../apache/doris/connector/hms/HmsClient.java | 139 +- .../connector/hms/HmsCommonStatistics.java | 86 + .../connector/hms/HmsPartitionStatistics.java | 71 + .../hms/HmsPartitionWithStatistics.java | 175 ++ .../doris/connector/hms/HmsTableInfo.java | 28 + .../connector/hms/HmsWriteConverter.java | 80 + .../doris/connector/hms/ThriftHmsClient.java | 211 +- .../connector/hms/HmsWriteConverterTest.java | 110 ++ .../hms/ThriftHmsClientWriteAcidTest.java | 455 +++++ .../datasource/PluginDrivenExternalTable.java | 16 + .../physical/PhysicalConnectorTableSink.java | 28 + .../ConnectorContractValidatorTest.java | 51 + .../PhysicalConnectorTableSinkTest.java | 60 + 40 files changed, 7456 insertions(+), 34 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransaction.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransactionManager.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveSinkHelper.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteContext.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWritePlanProvider.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/NameMapping.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableHandleAcidTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWritePlanProviderTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWriteUtilsTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/NameMappingTest.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/RecordingConnectorContext.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsAcidConstants.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCommonStatistics.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionStatistics.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionWithStatistics.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientWriteAcidTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index c7f2ff10420fbc..a1a1811dc34f5d 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -89,6 +89,12 @@ default boolean requiresPartitionLocalSort() { return p != null && p.requiresPartitionLocalSort(); } + /** Null-safe view of {@link ConnectorWritePlanProvider#requiresPartitionHashWrite()}. No provider ⇒ false. */ + default boolean requiresPartitionHashWrite() { + ConnectorWritePlanProvider p = getWritePlanProvider(); + return p != null && p.requiresPartitionHashWrite(); + } + /** * Null-safe view of {@link ConnectorWritePlanProvider#requiresMaterializeStaticPartitionValues()}. No * provider ⇒ false. diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java index fc89fe6c56cfbd..eeacdf232b6441 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java @@ -52,5 +52,20 @@ public static void validate(Connector connector, String catalogType) { + "' declares requiresPartitionLocalSort without requiresParallelWrite" + " AND requiresFullSchemaWriteOrder"); } + // #4 partition-hash-write (hash without sort) likewise implies parallel write AND full-schema write + // order (the sink indexes partition columns by full-schema position and distributes in parallel). + if (connector.requiresPartitionHashWrite() + && !(connector.requiresParallelWrite() && connector.requiresFullSchemaWriteOrder())) { + throw new IllegalStateException("Connector '" + catalogType + + "' declares requiresPartitionHashWrite without requiresParallelWrite" + + " AND requiresFullSchemaWriteOrder"); + } + // #5 the two hash arms are mutually exclusive: the engine checks local-sort first, so declaring both + // would silently ignore the hash-without-sort request. Fail loud instead. + if (connector.requiresPartitionLocalSort() && connector.requiresPartitionHashWrite()) { + throw new IllegalStateException("Connector '" + catalogType + + "' declares both requiresPartitionLocalSort and requiresPartitionHashWrite;" + + " a connector must pick at most one partition-distribution arm"); + } } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java index f313cefb0dbe4c..5de4cc93713b21 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePlanProvider.java @@ -195,6 +195,22 @@ default boolean requiresPartitionLocalSort() { return false; } + /** + * Whether dynamic-partition writes must be hash-distributed by partition columns but not locally + * sorted by them. A hive-style file writer buffers a separate partition writer per partition value, so — + * unlike {@link #requiresPartitionLocalSort()} (MaxCompute's streaming Storage-API writer, which closes a + * partition writer as soon as a different partition value appears and therefore needs the rows grouped by a + * local sort) — the hash distribution alone keeps each partition's rows on one instance and the output file + * count at ~one-per-partition, with no sort cost. The engine ({@code PhysicalConnectorTableSink}) picks the + * matching distribution: {@code requiresPartitionLocalSort} ⇒ hash + {@code MustLocalSortOrderSpec}; + * {@code requiresPartitionHashWrite} ⇒ hash only. A connector sets at most one of the two hash arms; a + * connector declaring this should also declare {@link #requiresParallelWrite()} + + * {@link #requiresFullSchemaWriteOrder()}. Default: no. + */ + default boolean requiresPartitionHashWrite() { + return false; + } + /** * Whether the connector's data files physically retain partition columns, so a static-partition write * must materialize the PARTITION-clause literal into the data column instead of NULL-filling it (e.g. diff --git a/fe/fe-connector/fe-connector-hive/pom.xml b/fe/fe-connector/fe-connector-hive/pom.xml index 055331c269c4de..30d66dbcadf21c 100644 --- a/fe/fe-connector/fe-connector-hive/pom.xml +++ b/fe/fe-connector/fe-connector-hive/pom.xml @@ -62,6 +62,17 @@ under the License. provided + + + ${project.groupId} + fe-filesystem-spi + ${project.version} + provided + + ${project.groupId} @@ -79,6 +90,17 @@ under the License. junit-jupiter test + + + + commons-lang + commons-lang + test + diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java new file mode 100644 index 00000000000000..ed0afd6cb6440c --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java @@ -0,0 +1,465 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.hms.HmsAcidConstants; + +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hive.common.ValidReadTxnList; +import org.apache.hadoop.hive.common.ValidReaderWriteIdList; +import org.apache.hadoop.hive.common.ValidTxnList; +import org.apache.hadoop.hive.common.ValidWriteIdList; +import org.apache.hadoop.hive.common.ValidWriteIdList.RangeResponse; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * Pure directory-name ACID state resolution for transactional Hive tables. + * + *

Plugin-side port of fe-core {@code AcidUtil.getAcidState}. It resolves, for one partition + * directory, the "best" base plus the working set of delta / delete-delta directories under a snapshot + * ({@code ValidTxnList} + {@code ValidWriteIdList}) and returns the surviving data files (base + + * non-delete deltas) plus the delete-delta descriptors. The BE later applies row deletes from the + * delete deltas.

+ * + *

The fe-core version drags in {@code FileCacheValue}/{@code LocationPath}/{@code AcidInfo}/ + * {@code HivePartition}; those all drop out at the plugin boundary because the plugin lists files with a + * raw Hadoop {@link FileSystem} and emits {@link HiveScanRange} directly. Only the pure name-parsing plus + * the {@code hive-common} {@code Valid*} algorithm ports.

+ * + *

Ref: hive/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidState (the fe-core copy + * exists because hive3 cannot read hive4 transaction tables and using hive4 directly is problematic).

+ */ +public final class HiveAcidUtil { + private static final Logger LOG = LogManager.getLogger(HiveAcidUtil.class); + + private static final String HIVE_TRANSACTIONAL_ORC_BUCKET_PREFIX = "bucket_"; + private static final String DELTA_SIDE_FILE_SUFFIX = "_flush_length"; + + private HiveAcidUtil() { + } + + /** Resolved ACID state for one partition: surviving data files + delete-delta descriptors. */ + public static final class AcidState { + private final List dataFiles; + private final List deleteDeltas; + + AcidState(List dataFiles, List deleteDeltas) { + this.dataFiles = dataFiles; + this.deleteDeltas = deleteDeltas; + } + + /** Base + non-delete delta bucket files that survive the snapshot; each becomes a scan split. */ + public List getDataFiles() { + return dataFiles; + } + + /** Delete-delta directories (with the bucket file names inside) the BE must subtract. */ + public List getDeleteDeltas() { + return deleteDeltas; + } + } + + /** A single delete-delta directory plus the (filtered) delete file names inside it. */ + public static final class DeleteDelta { + private final String directoryLocation; + private final List fileNames; + + DeleteDelta(String directoryLocation, List fileNames) { + this.directoryLocation = directoryLocation; + this.fileNames = fileNames; + } + + public String getDirectoryLocation() { + return directoryLocation; + } + + public List getFileNames() { + return fileNames; + } + } + + private static final class ParsedBase { + private final long writeId; + private final long visibilityId; + + ParsedBase(long writeId, long visibilityId) { + this.writeId = writeId; + this.visibilityId = visibilityId; + } + } + + private static ParsedBase parseBase(String name) { + //format1 : base_writeId + //format2 : base_writeId_visibilityId detail: https://issues.apache.org/jira/browse/HIVE-20823 + name = name.substring("base_".length()); + int index = name.indexOf("_v"); + if (index == -1) { + return new ParsedBase(Long.parseLong(name), 0); + } + return new ParsedBase( + Long.parseLong(name.substring(0, index)), + Long.parseLong(name.substring(index + 2))); + } + + private static final class ParsedDelta implements Comparable { + private final long min; + private final long max; + private final String path; + private final int statementId; + private final boolean deleteDelta; + private final long visibilityId; + + ParsedDelta(long min, long max, String path, int statementId, + boolean deleteDelta, long visibilityId) { + this.min = min; + this.max = max; + this.path = path; + this.statementId = statementId; + this.deleteDelta = deleteDelta; + this.visibilityId = visibilityId; + } + + /* + * Smaller minWID orders first; + * If minWID is the same, larger maxWID orders first; + * Otherwise, sort by stmtID; files w/o stmtID orders first. + * + * Compactions (Major/Minor) merge deltas/bases but delete of old files + * happens in a different process; thus it's possible to have bases/deltas with + * overlapping writeId boundaries. The sort order helps figure out the "best" set of files + * to use to get data. + * This sorts "wider" delta before "narrower" i.e. delta_5_20 sorts before delta_5_10 (and delta_11_20) + */ + @Override + public int compareTo(ParsedDelta other) { + return min != other.min ? Long.compare(min, other.min) : + other.max != max ? Long.compare(other.max, max) : + statementId != other.statementId + ? Integer.compare(statementId, other.statementId) : + path.compareTo(other.path); + } + } + + private static boolean isValidMetaDataFile(FileSystem fileSystem, String baseDir) + throws IOException { + String fileLocation = baseDir + "_metadata_acid"; + try { + return fileSystem.exists(new Path(fileLocation)); + } catch (IOException e) { + return false; + } + } + + private static boolean isValidBase(FileSystem fileSystem, String baseDir, + ParsedBase base, ValidWriteIdList writeIdList) throws IOException { + if (base.writeId == Long.MIN_VALUE) { + //Ref: https://issues.apache.org/jira/browse/HIVE-13369 + //such base is created by 1st compaction in case of non-acid to acid table conversion.(you + //will get dir: `base_-9223372036854775808`) + //By definition there are no open txns with id < 1. + //After this: https://issues.apache.org/jira/browse/HIVE-18192, txns(global transaction ID) => writeId. + return true; + } + + // hive 4 : just check "_v" suffix, before hive 4 : check `_metadata_acid` file in baseDir. + if ((base.visibilityId > 0) || isValidMetaDataFile(fileSystem, baseDir)) { + return writeIdList.isValidBase(base.writeId); + } + + // if here, it's a result of IOW + return writeIdList.isWriteIdValid(base.writeId); + } + + private static ParsedDelta parseDelta(String fileName, String deltaPrefix, String path) { + // format1: delta_min_max_statementId_visibilityId, delete_delta_min_max_statementId_visibilityId + // _visibilityId maybe not exists. + // detail: https://issues.apache.org/jira/browse/HIVE-20823 + // format2: delta_min_max_visibilityId, delete_delta_min_visibilityId + // when minor compaction runs, we collapse per statement delta files inside a single + // transaction so we no longer need a statementId in the file name + + long visibilityId = 0; + int visibilityIdx = fileName.indexOf("_v"); + if (visibilityIdx != -1) { + visibilityId = Long.parseLong(fileName.substring(visibilityIdx + 2)); + fileName = fileName.substring(0, visibilityIdx); + } + + boolean deleteDelta = deltaPrefix.equals("delete_delta_"); + + String rest = fileName.substring(deltaPrefix.length()); + int split = rest.indexOf('_'); + int split2 = rest.indexOf('_', split + 1); + long min = Long.parseLong(rest.substring(0, split)); + + if (split2 == -1) { + long max = Long.parseLong(rest.substring(split + 1)); + return new ParsedDelta(min, max, path, -1, deleteDelta, visibilityId); + } + + long max = Long.parseLong(rest.substring(split + 1, split2)); + int statementId = Integer.parseInt(rest.substring(split2 + 1)); + return new ParsedDelta(min, max, path, statementId, deleteDelta, visibilityId); + } + + private interface FileFilter { + boolean accept(String fileName); + } + + private static final class FullAcidFileFilter implements FileFilter { + @Override + public boolean accept(String fileName) { + return fileName.startsWith(HIVE_TRANSACTIONAL_ORC_BUCKET_PREFIX) + && !fileName.endsWith(DELTA_SIDE_FILE_SUFFIX); + } + } + + private static final class InsertOnlyFileFilter implements FileFilter { + @Override + public boolean accept(String fileName) { + return true; + } + } + + /** + * Lists the immediate file children of {@code dir} (directories excluded). + * + *

Mirrors fe-core {@code globList(fs, dir, false)} on a bare directory path: with no wildcard the + * glob has a {@code null} pattern, so it returns files only and skips any nested directory. A raw + * {@link FileSystem#listStatus} returns both, so the directory entries are filtered out here.

+ */ + private static List listFiles(FileSystem fs, String dir) throws IOException { + FileStatus[] statuses = fs.listStatus(new Path(dir)); + List files = new ArrayList<>(statuses.length); + for (FileStatus status : statuses) { + if (!status.isDirectory()) { + files.add(status); + } + } + return files; + } + + /** + * Resolves the ACID state of one partition directory under the given snapshot. + * + * @param fs raw Hadoop file system for the partition location + * @param partitionPath the partition directory (e.g. {@code hdfs://.../data_id=200103}) + * @param txnValidIds the two serialized {@code Valid*} lists keyed by {@link HmsAcidConstants} + * @param isFullAcid full-ACID (bucket_ filter + delete deltas) vs insert-only (accept-all) + * @return surviving data files + delete-delta descriptors for the partition + */ + public static AcidState getAcidState(FileSystem fs, String partitionPath, + Map txnValidIds, boolean isFullAcid) throws IOException { + // Ref: https://issues.apache.org/jira/browse/HIVE-18192 + // Readers should use the combination of ValidTxnList and ValidWriteIdList(Table) for snapshot isolation. + // ValidReadTxnList implements ValidTxnList + // ValidReaderWriteIdList implements ValidWriteIdList + ValidTxnList validTxnList; + if (txnValidIds.containsKey(HmsAcidConstants.VALID_TXNS_KEY)) { + validTxnList = new ValidReadTxnList(); + validTxnList.readFromString(txnValidIds.get(HmsAcidConstants.VALID_TXNS_KEY)); + } else { + throw new RuntimeException("Miss ValidTxnList"); + } + + ValidWriteIdList validWriteIdList; + if (txnValidIds.containsKey(HmsAcidConstants.VALID_WRITEIDS_KEY)) { + validWriteIdList = new ValidReaderWriteIdList(); + validWriteIdList.readFromString(txnValidIds.get(HmsAcidConstants.VALID_WRITEIDS_KEY)); + } else { + throw new RuntimeException("Miss ValidWriteIdList"); + } + + //hdfs://xxxxx/user/hive/warehouse/username/data_id=200103 + // List all files and folders, without recursion. + FileStatus[] partitionEntries = fs.listStatus(new Path(partitionPath)); + + String oldestBase = null; + long oldestBaseWriteId = Long.MAX_VALUE; + String bestBasePath = null; + long bestBaseWriteId = 0; + boolean haveOriginalFiles = false; + List workingDeltas = new ArrayList<>(); + + for (FileStatus entry : partitionEntries) { + if (entry.isDirectory()) { + String dirName = entry.getPath().getName(); //dirName: base_xxx,delta_xxx,... + String dirPath = partitionPath + "/" + dirName; + + if (dirName.startsWith("base_")) { + ParsedBase base = parseBase(dirName); + if (!validTxnList.isTxnValid(base.visibilityId)) { + //checks visibilityTxnId to see if it is committed in current snapshot. + continue; + } + + long writeId = base.writeId; + if (oldestBaseWriteId > writeId) { + oldestBase = dirPath; + oldestBaseWriteId = writeId; + } + + if (((bestBasePath == null) || (bestBaseWriteId < writeId)) + && isValidBase(fs, dirPath, base, validWriteIdList)) { + //IOW will generator a base_N/ directory: https://issues.apache.org/jira/browse/HIVE-14988 + //So maybe need consider: https://issues.apache.org/jira/browse/HIVE-25777 + + bestBasePath = dirPath; + bestBaseWriteId = writeId; + } + } else if (dirName.startsWith("delta_") || dirName.startsWith("delete_delta_")) { + String deltaPrefix = dirName.startsWith("delta_") ? "delta_" : "delete_delta_"; + ParsedDelta delta = parseDelta(dirName, deltaPrefix, dirPath); + + if (!validTxnList.isTxnValid(delta.visibilityId)) { + continue; + } + + // No need check (validWriteIdList.isWriteIdRangeAborted(min,max) != RangeResponse.ALL) + // It is a subset of (validWriteIdList.isWriteIdRangeValid(min, max) != RangeResponse.NONE) + if (validWriteIdList.isWriteIdRangeValid(delta.min, delta.max) != RangeResponse.NONE) { + workingDeltas.add(delta); + } + } else { + //Sometimes hive will generate temporary directories(`.hive-staging_hive_xxx` ), + // which do not need to be read. + LOG.warn("Read Hive Acid Table ignore the contents of this folder:" + dirName); + } + } else { + haveOriginalFiles = true; + } + } + + if (bestBasePath == null && haveOriginalFiles) { + // ALTER TABLE nonAcidTbl SET TBLPROPERTIES ('transactional'='true'); + throw new UnsupportedOperationException("For no acid table convert to acid, please COMPACT 'major'."); + } + + if ((oldestBase != null) && (bestBasePath == null)) { + /* + * If here, it means there was a base_x (> 1 perhaps) but none were suitable for given + * {@link writeIdList}. Note that 'original' files are logically a base_Long.MIN_VALUE and thus + * cannot have any data for an open txn. We could check {@link deltas} has files to cover + * [1,n] w/o gaps but this would almost never happen... + * + * We only throw for base_x produced by Compactor since that base erases all history and + * cannot be used for a client that has a snapshot in which something inside this base is + * open. (Nor can we ignore this base of course) But base_x which is a result of IOW, + * contains all history so we treat it just like delta wrt visibility. Imagine, IOW which + * aborts. It creates a base_x, which can and should just be ignored.*/ + long[] exceptions = validWriteIdList.getInvalidWriteIds(); + String minOpenWriteId = ((exceptions != null) + && (exceptions.length > 0)) ? String.valueOf(exceptions[0]) : "x"; + throw new IOException( + String.format("Not enough history available for ({},{}). Oldest available base: {}", + validWriteIdList.getHighWatermark(), minOpenWriteId, oldestBase)); + } + + workingDeltas.sort(null); + + List deltas = new ArrayList<>(); + long current = bestBaseWriteId; + int lastStatementId = -1; + ParsedDelta prev = null; + // find need read delta/delete_delta file. + for (ParsedDelta next : workingDeltas) { + if (next.max > current) { + if (validWriteIdList.isWriteIdRangeValid(current + 1, next.max) != RangeResponse.NONE) { + deltas.add(next); + current = next.max; + lastStatementId = next.statementId; + prev = next; + } + } else if ((next.max == current) && (lastStatementId >= 0)) { + //make sure to get all deltas within a single transaction; multi-statement txn + //generate multiple delta files with the same txnId range + //of course, if maxWriteId has already been minor compacted, + // all per statement deltas are obsolete + + deltas.add(next); + prev = next; + } else if ((prev != null) + && (next.max == prev.max) + && (next.min == prev.min) + && (next.statementId == prev.statementId)) { + // The 'next' parsedDelta may have everything equal to the 'prev' parsedDelta, except + // the path. This may happen when we have split update and we have two types of delta + // directories- 'delta_x_y' and 'delete_delta_x_y' for the SAME txn range. + + // Also note that any delete_deltas in between a given delta_x_y range would be made + // obsolete. For example, a delta_30_50 would make delete_delta_40_40 obsolete. + // This is valid because minor compaction always compacts the normal deltas and the delete + // deltas for the same range. That is, if we had 3 directories, delta_30_30, + // delete_delta_40_40 and delta_50_50, then running minor compaction would produce + // delta_30_50 and delete_delta_30_50. + deltas.add(next); + prev = next; + } + } + + List dataFiles = new ArrayList<>(); + List deleteDeltas = new ArrayList<>(); + + FileFilter fileFilter = isFullAcid ? new FullAcidFileFilter() : new InsertOnlyFileFilter(); + + // delta directories + for (ParsedDelta delta : deltas) { + String location = delta.path; + List entries = listFiles(fs, location); + if (delta.deleteDelta) { + List deleteDeltaFileNames = new ArrayList<>(); + for (FileStatus entry : entries) { + String name = entry.getPath().getName(); + if (fileFilter.accept(name)) { + deleteDeltaFileNames.add(name); + } + } + deleteDeltas.add(new DeleteDelta(location, deleteDeltaFileNames)); + continue; + } + for (FileStatus entry : entries) { + if (fileFilter.accept(entry.getPath().getName())) { + dataFiles.add(entry); + } + } + } + + // base + if (bestBasePath != null) { + List entries = listFiles(fs, bestBasePath); + for (FileStatus entry : entries) { + if (fileFilter.accept(entry.getPath().getName())) { + dataFiles.add(entry); + } + } + } + + if (!isFullAcid && !deleteDeltas.isEmpty()) { + throw new RuntimeException("No Hive Full Acid Table have delete_delta_* Dir."); + } + return new AcidState(dataFiles, deleteDeltas); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index d489f974c2af68..56b7f08689926a 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.ThriftHmsClient; @@ -46,6 +47,10 @@ public class HiveConnector implements Connector { private final ConnectorContext context; private volatile HmsClient hmsClient; + // Read-transaction manager for transactional (ACID) Hive scans. One per connector, keyed by query. + // Plugin-owned and dormant until the read cutover wires its query-finish commit (see the manager). + private final HiveReadTransactionManager readTxnManager = new HiveReadTransactionManager(); + public HiveConnector(Map properties, ConnectorContext context) { this.properties = Collections.unmodifiableMap(properties); this.context = context; @@ -58,7 +63,12 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { @Override public ConnectorScanPlanProvider getScanPlanProvider() { - return new HiveScanPlanProvider(getOrCreateClient(), properties); + return new HiveScanPlanProvider(getOrCreateClient(), properties, readTxnManager); + } + + @Override + public ConnectorWritePlanProvider getWritePlanProvider() { + return new HiveWritePlanProvider(getOrCreateClient(), properties, context); } private HmsClient getOrCreateClient() { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 4f2116dec864ac..c83cbda935b7dd 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -28,6 +28,7 @@ import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.connector.api.pushdown.ConnectorAnd; import org.apache.doris.connector.api.pushdown.ConnectorComparison; import org.apache.doris.connector.api.pushdown.ConnectorExpression; @@ -451,6 +452,20 @@ public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, // HMSCachedClient has no rename), so the SPI default throw ("RENAME TABLE not supported") preserves the // pre-flip behavior. + // ========== ConnectorWriteOps: transactions ========== + + /** + * Opens a {@link HiveConnectorTransaction} for a hive non-ACID INSERT / INSERT OVERWRITE, mirroring the + * iceberg one-liner (design D1: {@code planWrite} lives in {@code HiveWritePlanProvider}, the metadata + * carries only the begin factory). The transaction id is the engine-allocated Doris global id (it is + * registered in the engine transaction registry and stamped into the sink), so it must come from the + * session, not be minted here. Dormant until the P7.4/P7.5 cutover. + */ + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session) { + return new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient, context); + } + /** * Drops {@code dbName.tableName} after rejecting a transactional table, mirroring legacy * {@code HiveMetadataOps.dropTableImpl}. Shared by the direct DROP TABLE and the force DROP DATABASE diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java new file mode 100644 index 00000000000000..d496d9facaef74 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java @@ -0,0 +1,1707 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. +// This file ports the write/commit lifecycle of the legacy fe-core +// org.apache.doris.datasource.hive.HMSTransaction (itself derived from Trino's +// SemiTransactionalHiveMetastore) onto the connector SPI, breaking the fe-core couplings per the +// P7.3 design decisions (D1-D12). Control flow is preserved; only the fe-core seams are replaced. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsCommonStatistics; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsPartitionStatistics; +import org.apache.doris.connector.hms.HmsPartitionWithStatistics; +import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.connector.hms.HmsTypeMapping; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.FileEntry; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.FileSystemUtil; +import org.apache.doris.filesystem.Location; +import org.apache.doris.filesystem.properties.StorageKind; +import org.apache.doris.filesystem.properties.StorageProperties; +import org.apache.doris.filesystem.spi.FileSystemProvider; +import org.apache.doris.filesystem.spi.ObjFileSystem; +import org.apache.doris.thrift.TFileType; +import org.apache.doris.thrift.THiveLocationParams; +import org.apache.doris.thrift.THivePartitionUpdate; +import org.apache.doris.thrift.TS3MPUPendingUpload; +import org.apache.doris.thrift.TUpdateMode; + +import com.google.common.base.Preconditions; +import com.google.common.base.Verify; +import com.google.common.collect.Iterables; +import org.apache.hadoop.fs.Path; +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TException; +import org.apache.thrift.protocol.TBinaryProtocol; + +import java.io.IOException; +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Objects; +import java.util.Optional; +import java.util.Queue; +import java.util.ServiceLoader; +import java.util.Set; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ConcurrentLinkedQueue; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Executor; +import java.util.concurrent.ExecutorService; +import java.util.concurrent.Executors; +import java.util.concurrent.ThreadFactory; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.stream.Collectors; + +/** + * Hive non-ACID write transaction ported from the legacy fe-core {@code HMSTransaction} onto the + * connector {@link ConnectorTransaction} SPI (design P7.3). It accumulates the BE-reported commit + * fragments ({@link THivePartitionUpdate}, fed via {@link #addCommitData}), classifies each partition + * update into a table/partition action (APPEND / NEW / OVERWRITE, with a NEW→APPEND downgrade on an + * HMS existence probe), and drives the {@link HmsCommitter}: staging→target renames, object-store + * multipart-upload complete/abort, {@code addPartitions}, and statistics updates. + * + *

fe-core couplings broken per design: query profiling dropped (D4); metastore access via the plugin + * {@link HmsClient} SPI instead of {@code HiveMetadataOps} (D10); object-store multipart uploads via a + * plugin-side {@code fe-filesystem-spi} {@link ObjFileSystem} built from {@code context.getStorageProperties()} + * instead of {@code SpiSwitchingFileSystem} (D6); plugin-owned async pool threads each auth-wrapped + * (D5); full-ACID writes hard-rejected at begin (D7); {@code rollback()} deletes staging + aborts MPUs (D9). + * + *

Gate-closed / dormant: hive is not in {@code SPI_READY_TYPES}, so nothing routes plugin-driven hive + * writes through this class until the P7.4/P7.5 cutover. {@link #beginWrite} is wired by INC-4's + * {@code HiveWritePlanProvider.planWrite}. + */ +public class HiveConnectorTransaction implements ConnectorTransaction { + + private static final Logger LOG = LogManager.getLogger(HiveConnectorTransaction.class); + + private final long transactionId; + private final HmsClient hmsClient; + private final ConnectorContext context; + + // Plugin-owned async pool for staging renames + MPU complete/abort (the legacy class had this injected + // by fe-core). Shut down in close(). authWrappingExecutor wraps each submitted task in the catalog auth + // context (D5: plugin-owned pool threads do not inherit the caller pin). + private final ExecutorService fileSystemExecutor; + private final Executor authWrappingExecutor; + + // Built lazily from the catalog's object-store StorageProperties (D6). Held so staging renames/deletes + // and MPU complete/abort share one FileSystem; closed in close(). + private volatile FileSystem fs; + + private NameMapping nameMapping; + private volatile HmsTableInfo hmsTableInfo; + private String queryId; + private boolean isOverwrite; + private TFileType fileType; + private Optional stagingDirectory = Optional.empty(); + private boolean isMockedPartitionUpdate = false; + + private List hivePartitionUpdates = new ArrayList<>(); + private final Map> tableActions = new HashMap<>(); + private final Map, Action>> partitionActions = new HashMap<>(); + private final Set uncompletedMpuPendingUploads = new HashSet<>(); + + private HmsCommitter hmsCommitter; + + public HiveConnectorTransaction(long transactionId, HmsClient hmsClient, ConnectorContext context) { + this.transactionId = transactionId; + this.hmsClient = hmsClient; + this.context = context; + this.fileSystemExecutor = Executors.newFixedThreadPool(16, namedDaemonThreadFactory("hive-write-fs-%d")); + this.authWrappingExecutor = command -> fileSystemExecutor.execute(() -> { + try { + context.executeAuthenticated(() -> { + command.run(); + return null; + }); + } catch (Exception e) { + throw new RuntimeException(e); + } + }); + } + + // ─────────────────────────────── SPI surface ─────────────────────────────── + + @Override + public long getTransactionId() { + return transactionId; + } + + @Override + public String profileLabel() { + // D2: maps to the existing fe-core TransactionType.HMS; any other string would fall back to UNKNOWN. + return "HMS"; + } + + @Override + public void addCommitData(byte[] commitFragment) { + THivePartitionUpdate pu = new THivePartitionUpdate(); + try { + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(pu, commitFragment); + } catch (TException e) { + throw new DorisConnectorException("failed to deserialize Hive partition update", e); + } + synchronized (this) { + hivePartitionUpdates.add(pu); + } + } + + @Override + public long getUpdateCnt() { + // D3: preserve legacy behavior — the affected-row count is the sum of the fragment row counts. + return hivePartitionUpdates.stream().mapToLong(THivePartitionUpdate::getRowCount).sum(); + } + + @Override + public void close() { + shutdownExecutorService(fileSystemExecutor); + FileSystem local = fs; + if (local != null) { + try { + local.close(); + } catch (Exception e) { + LOG.warn("Failed to close filesystem for hive write transaction {}: {}", + transactionId, e.getMessage()); + } + } + } + + /** + * Opens the write for {@code db.tableName} (analogue of iceberg {@code beginWrite}; folds the legacy + * {@code beginInsertTable} plus the table load and the full-ACID-write guard). Called by INC-4's + * {@code HiveWritePlanProvider.planWrite}. The table is loaded under the catalog auth context (D5) — the + * only pre-commit point that has the table — so the full-ACID reject (D7) can run here. + */ + public void beginWrite(ConnectorSession session, String db, String tableName, HiveWriteContext ctx) { + this.queryId = ctx.getQueryId(); + this.isOverwrite = ctx.isOverwrite(); + this.fileType = ctx.getFileType(); + this.stagingDirectory = (fileType == TFileType.FILE_S3) + ? Optional.empty() : Optional.of(ctx.getWritePath()); + this.nameMapping = new NameMapping(context.getCatalogId(), db, tableName, db, tableName); + try { + context.executeAuthenticated(() -> { + HmsTableInfo table = hmsClient.getTable(db, tableName); + rejectTransactionalWrite(table.getParameters()); + this.hmsTableInfo = table; + return null; + }); + } catch (Exception e) { + throw new DorisConnectorException( + "Failed to begin write for hive table " + tableName + ": " + e.getMessage(), e); + } + } + + @Override + public void commit() { + // The classification (finishInsertTable) ran from the executor in the legacy class; the unified SPI + // exposes only commit(), so it runs here (before the committer) to populate the action maps. If it + // throws, the committer was never created and the engine's subsequent rollback() cleans up. + finishInsertTable(nameMapping); + hmsCommitter = new HmsCommitter(); + try { + for (Map.Entry> entry : tableActions.entrySet()) { + NameMapping nm = entry.getKey(); + Action action = entry.getValue(); + switch (action.getType()) { + case INSERT_EXISTING: + hmsCommitter.prepareInsertExistingTable(nm, action.getData()); + break; + case ALTER: + hmsCommitter.prepareAlterTable(nm, action.getData()); + break; + default: + throw new UnsupportedOperationException( + "Unsupported table action type: " + action.getType()); + } + } + + for (Map.Entry, Action>> tableEntry + : partitionActions.entrySet()) { + NameMapping nm = tableEntry.getKey(); + for (Map.Entry, Action> partitionEntry + : tableEntry.getValue().entrySet()) { + Action action = partitionEntry.getValue(); + switch (action.getType()) { + case INSERT_EXISTING: + hmsCommitter.prepareInsertExistPartition(nm, action.getData()); + break; + case ADD: + hmsCommitter.prepareAddPartition(nm, action.getData()); + break; + case ALTER: + hmsCommitter.prepareAlterPartition(nm, action.getData()); + break; + default: + throw new UnsupportedOperationException( + "Unsupported partition action type: " + action.getType()); + } + } + } + + hmsCommitter.doCommit(); + } catch (Throwable t) { + LOG.warn("Failed to commit for {}, abort it.", queryId); + try { + hmsCommitter.abort(); + hmsCommitter.rollback(); + } catch (RuntimeException e) { + t.addSuppressed(new Exception("Failed to roll back after commit failure", e)); + } + throw t; + } finally { + hmsCommitter.runClearPathsForFinish(); + hmsCommitter.shutdownExecutorService(); + } + } + + @Override + public void rollback() { + if (hmsCommitter == null) { + collectUncompletedMpuPendingUploads(hivePartitionUpdates); + if (uncompletedMpuPendingUploads.isEmpty()) { + return; + } + hmsCommitter = new HmsCommitter(); + try { + hmsCommitter.rollback(); + } finally { + hmsCommitter.shutdownExecutorService(); + } + return; + } + try { + hmsCommitter.abort(); + hmsCommitter.rollback(); + } finally { + hmsCommitter.shutdownExecutorService(); + } + } + + // ─────────────────────────────── begin-guard (D7) ─────────────────────────────── + + // DEC-2: reject ANY transactional Hive table (full-ACID AND insert-only), matching legacy + // InsertIntoTableCommand's AcidUtils.isTransactionalTable gate. A narrower full-ACID-only reject would let + // an insert-only ACID table slip through and get plain files (corrupt/invisible data). + private static void rejectTransactionalWrite(Map tableParameters) { + if (isTransactionalTable(tableParameters)) { + throw new DorisConnectorException( + "Cannot write to a transactional Hive table (only non-ACID INSERT/OVERWRITE is supported)"); + } + } + + // Mirrors hive AcidUtils.isTablePropertyTransactional: "transactional" (or its upper-cased key) parsed as + // a boolean. D8: derived plugin-side from the raw HMS parameters (fe-core parses no properties). + private static boolean isTransactionalTable(Map params) { + if (params == null) { + return false; + } + String value = params.get("transactional"); + if (value == null) { + value = params.get("transactional".toUpperCase(Locale.ROOT)); + } + return Boolean.parseBoolean(value); + } + + // Mirrors hive AcidUtils.isInsertOnlyTable: transactional_properties == "insert_only" (case-insensitive). + private static boolean isInsertOnlyTable(Map params) { + if (params == null) { + return false; + } + return "insert_only".equalsIgnoreCase(params.get("transactional_properties")); + } + + // Mirrors hive AcidUtils.isFullAcidTable: transactional AND NOT insert-only. + private static boolean isFullAcidTable(Map params) { + return isTransactionalTable(params) && !isInsertOnlyTable(params); + } + + // ─────────────────────────────── classification (legacy finishInsertTable) ─────────────────────────────── + + void finishInsertTable(NameMapping nameMapping) { + HmsTableInfo table = getTable(nameMapping); + if (hivePartitionUpdates.isEmpty() && isOverwrite && table.getPartitionKeys().isEmpty()) { + // INSERT OVERWRITE from an empty source: fabricate one empty OVERWRITE update to clean the table. + isMockedPartitionUpdate = true; + THivePartitionUpdate emptyUpdate = new THivePartitionUpdate(); + emptyUpdate.setUpdateMode(TUpdateMode.OVERWRITE); + emptyUpdate.setFileSize(0); + emptyUpdate.setRowCount(0); + emptyUpdate.setFileNames(Collections.emptyList()); + if (fileType == TFileType.FILE_S3) { + emptyUpdate.setS3MpuPendingUploads(new ArrayList<>(Collections.singletonList( + new TS3MPUPendingUpload()))); + THiveLocationParams location = new THiveLocationParams(); + location.setWritePath(table.getLocation()); + emptyUpdate.setLocation(location); + } else if (stagingDirectory.isPresent()) { + String v = stagingDirectory.get(); + try { + getFileSystem().mkdirs(Location.of(v)); + } catch (IOException e) { + throw new RuntimeException("Failed to create staging directory: " + v, e); + } + THiveLocationParams location = new THiveLocationParams(); + location.setWritePath(v); + emptyUpdate.setLocation(location); + } + hivePartitionUpdates = new ArrayList<>(Collections.singletonList(emptyUpdate)); + } + + List mergedPUs = HiveWriteUtils.mergePartitions(hivePartitionUpdates); + collectUncompletedMpuPendingUploads(mergedPUs); + List> insertExistsPartitions = new ArrayList<>(); + for (THivePartitionUpdate pu : mergedPUs) { + TUpdateMode updateMode = pu.getUpdateMode(); + HmsPartitionStatistics stats = HmsPartitionStatistics.fromCommonStatistics( + pu.getRowCount(), pu.getFileNamesSize(), pu.getFileSize()); + String writePath = pu.getLocation().getWritePath(); + if (table.getPartitionKeys().isEmpty()) { + Preconditions.checkArgument(mergedPUs.size() == 1, + "When updating a non-partitioned table, multiple partitions should not be written"); + switch (updateMode) { + case APPEND: + finishChangingExistingTable(ActionType.INSERT_EXISTING, nameMapping, writePath, + pu.getFileNames(), stats, pu); + break; + case OVERWRITE: + dropTable(nameMapping); + createTable(nameMapping, table, writePath, pu.getFileNames(), stats, pu); + break; + default: + throw new RuntimeException("Not support mode:[" + updateMode + "] in unPartitioned table"); + } + } else { + switch (updateMode) { + case APPEND: + insertExistsPartitions.add(new AbstractMap.SimpleImmutableEntry<>(pu, stats)); + break; + case NEW: + String partitionName = pu.getName(); + if (partitionName == null || partitionName.isEmpty()) { + LOG.warn("Partition name is null/empty for NEW mode in partitioned table, skipping"); + break; + } + List partitionValues = HiveWriteUtils.toPartitionValues(partitionName); + boolean existsInHms; + try { + existsInHms = hmsClient.partitionExists(nameMapping.getRemoteDbName(), + nameMapping.getRemoteTblName(), partitionValues); + } catch (Exception e) { + // Not found (or the probe failed) -> treat as truly new, mirroring the legacy + // getPartition()-in-try-catch. + existsInHms = false; + if (LOG.isDebugEnabled()) { + LOG.debug("Partition {} existence probe failed, will create it", partitionName); + } + } + if (existsInHms) { + LOG.info("Partition {} already exists in HMS (Doris cache miss), treating as APPEND", + partitionName); + insertExistsPartitions.add(new AbstractMap.SimpleImmutableEntry<>(pu, stats)); + } else { + createAndAddPartition(nameMapping, table, partitionValues, writePath, pu, stats, false); + } + break; + case OVERWRITE: + String overwritePartitionName = pu.getName(); + if (overwritePartitionName == null || overwritePartitionName.isEmpty()) { + LOG.warn("Partition name is null/empty for OVERWRITE mode in partitioned table, " + + "skipping"); + break; + } + createAndAddPartition(nameMapping, table, + HiveWriteUtils.toPartitionValues(overwritePartitionName), + writePath, pu, stats, true); + break; + default: + throw new RuntimeException("Not support mode:[" + updateMode + "] in partitioned table"); + } + } + } + + if (!insertExistsPartitions.isEmpty()) { + convertToInsertExistingPartitionAction(nameMapping, insertExistsPartitions); + } + } + + private void collectUncompletedMpuPendingUploads(List hivePartitionUpdates) { + for (THivePartitionUpdate pu : hivePartitionUpdates) { + if (pu.getS3MpuPendingUploads() != null) { + for (TS3MPUPendingUpload s3MpuPendingUpload : pu.getS3MpuPendingUploads()) { + uncompletedMpuPendingUploads.add( + new UncompletedMpuPendingUpload(s3MpuPendingUpload, pu.getLocation().getWritePath())); + } + } + } + } + + private void convertToInsertExistingPartitionAction( + NameMapping nameMapping, + List> partitions) { + Map, Action> partitionActionsForTable = + partitionActions.computeIfAbsent(nameMapping, k -> new HashMap<>()); + + for (List> partitionBatch + : Iterables.partition(partitions, 100)) { + + List partitionNames = partitionBatch.stream() + .map(pair -> pair.getKey().getName()) + .collect(Collectors.toList()); + + // check in partitionAction + Action oldPartitionAction = partitionActionsForTable.get(partitionNames); + if (oldPartitionAction != null) { + switch (oldPartitionAction.getType()) { + case DROP: + case DROP_PRESERVE_DATA: + throw new RuntimeException("Not found partition from partition actions" + + "for " + nameMapping.getFullLocalName() + ", partitions: " + partitionNames); + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + throw new UnsupportedOperationException("Inserting into a partition that were added, altered," + + "or inserted into in the same transaction is not supported"); + default: + throw new IllegalStateException("Unknown action type: " + oldPartitionAction.getType()); + } + } + + List hmsPartitions = hmsClient.getPartitions( + nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), partitionNames); + // Mirror HiveUtil.convertToNamePartitionMap's Collectors.toMap: fail loud on a duplicate + // key (two HMS partitions with identical values, or a repeated requested partition name) + // instead of silently overwriting. A silent overwrite would shrink partitionsByNamesMap + // below partitionNames.size(), and the size()-bounded loop below would then drop a + // partition's INSERT_EXISTING action (silent write loss) rather than abort the txn. + Map, HmsPartitionInfo> partitionsByValues = new HashMap<>(); + for (HmsPartitionInfo p : hmsPartitions) { + if (partitionsByValues.put(p.getValues(), p) != null) { + throw new IllegalStateException("Duplicate key " + p.getValues()); + } + } + Map partitionsByNamesMap = new HashMap<>(); + for (String name : partitionNames) { + HmsPartitionInfo p = partitionsByValues.get(HiveWriteUtils.toPartitionValues(name)); + if (p != null && partitionsByNamesMap.put(name, p) != null) { + throw new IllegalStateException("Duplicate key " + name); + } + } + + for (int i = 0; i < partitionsByNamesMap.size(); i++) { + String partitionName = partitionNames.get(i); + HmsPartitionInfo partition = partitionsByNamesMap.get(partitionName); + if (partition == null) { + // Prevent this partition from being deleted by other engines. + throw new RuntimeException("Not found partition from hms for " + nameMapping.getFullLocalName() + + ", partitions: " + partitionNames); + } + THivePartitionUpdate pu = partitionBatch.get(i).getKey(); + HmsPartitionStatistics updateStats = partitionBatch.get(i).getValue(); + List partitionValues = HiveWriteUtils.toPartitionValues(pu.getName()); + + partitionActionsForTable.put( + partitionValues, + new Action<>(ActionType.INSERT_EXISTING, + new PartitionAndMore( + partitionValues, + partition.getLocation(), + pu.getLocation().getWritePath(), + pu.getName(), + pu.getFileNames(), + updateStats, + pu))); + } + } + } + + // ─────────────────────────────── state-transition helpers (legacy, synchronized) ─────────────────────────────── + + private synchronized HmsTableInfo getTable(NameMapping nameMapping) { + Action tableAction = tableActions.get(nameMapping); + if (tableAction == null) { + // Reuse the begin-time table snapshot (loaded in beginWrite for the D7 reject); the transaction + // targets exactly this one table, so no re-fetch is needed. + return hmsTableInfo; + } + switch (tableAction.getType()) { + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + return tableAction.getData().getTable(); + case DROP: + case DROP_PRESERVE_DATA: + break; + default: + throw new IllegalStateException("Unknown action type: " + tableAction.getType()); + } + throw new RuntimeException("Not Found table: " + nameMapping); + } + + private synchronized void finishChangingExistingTable( + ActionType actionType, + NameMapping nameMapping, + String location, + List fileNames, + HmsPartitionStatistics statisticsUpdate, + THivePartitionUpdate hivePartitionUpdate) { + Action oldTableAction = tableActions.get(nameMapping); + if (oldTableAction == null) { + tableActions.put(nameMapping, + new Action<>(actionType, + new TableAndMore(hmsTableInfo, location, fileNames, statisticsUpdate, + hivePartitionUpdate))); + return; + } + switch (oldTableAction.getType()) { + case DROP: + throw new RuntimeException("Not found table: " + nameMapping.getFullLocalName()); + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + throw new UnsupportedOperationException("Inserting into an unpartitioned table that were added, " + + "altered,or inserted into in the same transaction is not supported"); + case DROP_PRESERVE_DATA: + break; + default: + throw new IllegalStateException("Unknown action type: " + oldTableAction.getType()); + } + } + + private synchronized void createTable( + NameMapping nameMapping, + HmsTableInfo table, String location, List fileNames, + HmsPartitionStatistics statistics, + THivePartitionUpdate hivePartitionUpdate) { + // When creating a table, it should never have partition actions. This is just a sanity check. + checkNoPartitionAction(nameMapping); + Action oldTableAction = tableActions.get(nameMapping); + TableAndMore tableAndMore = new TableAndMore(table, location, fileNames, statistics, hivePartitionUpdate); + if (oldTableAction == null) { + tableActions.put(nameMapping, new Action<>(ActionType.ADD, tableAndMore)); + return; + } + switch (oldTableAction.getType()) { + case DROP: + tableActions.put(nameMapping, new Action<>(ActionType.ALTER, tableAndMore)); + return; + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + throw new RuntimeException("Table already exists: " + nameMapping.getFullLocalName()); + case DROP_PRESERVE_DATA: + break; + default: + throw new IllegalStateException("Unknown action type: " + oldTableAction.getType()); + } + } + + private synchronized void dropTable(NameMapping nameMapping) { + // Dropping table with partition actions requires cleaning up staging data, which is not implemented yet. + checkNoPartitionAction(nameMapping); + Action oldTableAction = tableActions.get(nameMapping); + if (oldTableAction == null || oldTableAction.getType() == ActionType.ALTER) { + tableActions.put(nameMapping, new Action<>(ActionType.DROP, null)); + return; + } + switch (oldTableAction.getType()) { + case DROP: + throw new RuntimeException("Not found table: " + nameMapping.getFullLocalName()); + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + throw new RuntimeException("Dropping a table added/modified in the same transaction is not supported"); + case DROP_PRESERVE_DATA: + break; + default: + throw new IllegalStateException("Unknown action type: " + oldTableAction.getType()); + } + } + + private void checkNoPartitionAction(NameMapping nameMapping) { + Map, Action> partitionActionsForTable = partitionActions.get(nameMapping); + if (partitionActionsForTable != null && !partitionActionsForTable.isEmpty()) { + throw new RuntimeException( + "Cannot make schema changes to a table with modified partitions in the same transaction"); + } + } + + private void createAndAddPartition( + NameMapping nameMapping, + HmsTableInfo table, + List partitionValues, + String writePath, + THivePartitionUpdate pu, + HmsPartitionStatistics statistics, + boolean dropFirst) { + String pathForHms = this.fileType == TFileType.FILE_S3 + ? writePath + : pu.getLocation().getTargetPath(); + if (dropFirst) { + dropPartition(nameMapping, partitionValues, true); + } + addPartition(nameMapping, partitionValues, pathForHms, writePath, pu.getName(), pu.getFileNames(), + statistics, pu); + } + + private synchronized void addPartition( + NameMapping nameMapping, + List partitionValues, + String targetPath, + String currentLocation, + String partitionName, + List files, + HmsPartitionStatistics statistics, + THivePartitionUpdate hivePartitionUpdate) { + Map, Action> partitionActionsForTable = + partitionActions.computeIfAbsent(nameMapping, k -> new HashMap<>()); + Action oldPartitionAction = partitionActionsForTable.get(partitionValues); + if (oldPartitionAction == null) { + partitionActionsForTable.put(partitionValues, + new Action<>(ActionType.ADD, + new PartitionAndMore(partitionValues, targetPath, currentLocation, partitionName, files, + statistics, hivePartitionUpdate))); + return; + } + switch (oldPartitionAction.getType()) { + case DROP: + case DROP_PRESERVE_DATA: + partitionActionsForTable.put(partitionValues, + new Action<>(ActionType.ALTER, + new PartitionAndMore(partitionValues, targetPath, currentLocation, partitionName, + files, statistics, hivePartitionUpdate))); + return; + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + throw new RuntimeException("Partition already exists for table: " + + nameMapping.getFullLocalName() + ", partition values: " + partitionValues); + default: + throw new IllegalStateException("Unknown action type: " + oldPartitionAction.getType()); + } + } + + private synchronized void dropPartition( + NameMapping nameMapping, + List partitionValues, + boolean deleteData) { + Map, Action> partitionActionsForTable = + partitionActions.computeIfAbsent(nameMapping, k -> new HashMap<>()); + Action oldPartitionAction = partitionActionsForTable.get(partitionValues); + if (oldPartitionAction == null) { + if (deleteData) { + partitionActionsForTable.put(partitionValues, new Action<>(ActionType.DROP, null)); + } else { + partitionActionsForTable.put(partitionValues, new Action<>(ActionType.DROP_PRESERVE_DATA, null)); + } + return; + } + switch (oldPartitionAction.getType()) { + case DROP: + case DROP_PRESERVE_DATA: + throw new RuntimeException("Not found partition from partition actions for " + + nameMapping.getFullLocalName() + ", partitions: " + partitionValues); + case ADD: + case ALTER: + case INSERT_EXISTING: + case MERGE: + throw new RuntimeException("Dropping a partition added in the same transaction is not supported: " + + nameMapping.getFullLocalName() + ", partition values: " + partitionValues); + default: + throw new IllegalStateException("Unknown action type: " + oldPartitionAction.getType()); + } + } + + // ─────────────────────────────── filesystem (D6) ─────────────────────────────── + + private FileSystem getFileSystem() { + FileSystem local = fs; + if (local == null) { + synchronized (this) { + local = fs; + if (local == null) { + StorageProperties objSp = context.getStorageProperties().stream() + .filter(sp -> sp.kind() == StorageKind.OBJECT_STORAGE) + .findFirst() + .orElse(null); + try { + local = resolveObjectStoreFileSystem(objSp); + } catch (IOException e) { + throw new DorisConnectorException( + "Failed to build filesystem for hive write: " + e.getMessage(), e); + } + fs = local; + } + } + } + return local; + } + + /** + * Builds the plugin-side object-store {@link FileSystem} used for staging renames/deletes and MPU + * complete/abort (D6). Isolated behind this single method so production ServiceLoader discovery — a + * cutover-time concern (the object-store providers may be directory-loaded in a separate classloader) — + * is swappable and unit tests can inject a fake FileSystem by overriding it. + */ + protected FileSystem resolveObjectStoreFileSystem(StorageProperties objSp) throws IOException { + if (objSp == null) { + throw new DorisConnectorException("No object-store StorageProperties available for hive write"); + } + for (FileSystemProvider provider : ServiceLoader.load(FileSystemProvider.class)) { + if (provider.supports(objSp.rawProperties())) { + return provider.create(objSp.rawProperties()); + } + } + throw new DorisConnectorException("No FileSystemProvider supports the object-store properties for hive write"); + } + + // ─────────────────────────────── commit-time object-store MPU (D6) ─────────────────────────────── + + /** + * Completes the BE-initiated multipart uploads on the object store (FE finalizes what BE staged). Ported + * from the legacy {@code objCommit}; only the FileSystem source (plugin-built vs SpiSwitchingFileSystem) + * and the per-task auth wrap (D5) differ. Skipped for a mocked empty overwrite. + */ + private void objCommit(List> asyncFileSystemTaskFutures, + AtomicBoolean fileSystemTaskCancelled, THivePartitionUpdate hivePartitionUpdate, String path) { + if (isMockedPartitionUpdate) { + return; + } + FileSystem resolved = getFileSystem(); + if (!(resolved instanceof ObjFileSystem)) { + throw new RuntimeException("Expected ObjFileSystem for MPU commit at path '" + path + "', got: " + + resolved.getClass().getSimpleName() + ". This path does not point to an object-storage " + + "backend."); + } + ObjFileSystem objFs = (ObjFileSystem) resolved; + for (TS3MPUPendingUpload s3MpuPendingUpload : hivePartitionUpdate.getS3MpuPendingUploads()) { + asyncFileSystemTaskFutures.add(CompletableFuture.runAsync(() -> { + if (fileSystemTaskCancelled.get()) { + return; + } + String remotePath = "s3://" + s3MpuPendingUpload.getBucket() + "/" + s3MpuPendingUpload.getKey(); + try { + context.executeAuthenticated(() -> { + objFs.completeMultipartUpload(remotePath, s3MpuPendingUpload.getUploadId(), + s3MpuPendingUpload.getEtags()); + return null; + }); + } catch (Exception e) { + throw new RuntimeException("Failed to complete MPU for " + remotePath, e); + } + uncompletedMpuPendingUploads.remove(new UncompletedMpuPendingUpload(s3MpuPendingUpload, path)); + }, fileSystemExecutor)); + } + } + + // ─────────────────────────────── filesystem walk helpers (legacy) ─────────────────────────────── + + private void recursiveDeleteItems(Path directory, boolean deleteEmptyDir, boolean reverse) { + DeleteRecursivelyResult deleteResult = recursiveDeleteFiles(directory, deleteEmptyDir, reverse); + if (!deleteResult.getNotDeletedEligibleItems().isEmpty()) { + LOG.warn("Failed to delete directory {}. Some eligible items can't be deleted: {}.", + directory.toString(), deleteResult.getNotDeletedEligibleItems()); + throw new RuntimeException( + "Failed to delete directory for files: " + deleteResult.getNotDeletedEligibleItems()); + } else if (deleteEmptyDir && !deleteResult.dirNotExists()) { + LOG.warn("Failed to delete directory {} due to dir isn't empty", directory.toString()); + throw new RuntimeException("Failed to delete directory for empty dir: " + directory.toString()); + } + } + + private DeleteRecursivelyResult recursiveDeleteFiles(Path directory, boolean deleteEmptyDir, boolean reverse) { + try { + boolean dirExists = getFileSystem().exists(Location.of(directory.toString())); + if (!dirExists) { + return new DeleteRecursivelyResult(true, Collections.emptyList()); + } + } catch (IOException e) { + return new DeleteRecursivelyResult(false, + new ArrayList<>(Collections.singletonList(directory.toString() + "/*"))); + } + return doRecursiveDeleteFiles(directory, deleteEmptyDir, queryId, reverse); + } + + private DeleteRecursivelyResult doRecursiveDeleteFiles(Path directory, boolean deleteEmptyDir, + String queryId, boolean reverse) { + List allFiles; + Set allDirs; + try { + allFiles = getFileSystem().listFilesRecursive(Location.of(directory.toString())); + allDirs = getFileSystem().listDirectories(Location.of(directory.toString())); + } catch (IOException e) { + return new DeleteRecursivelyResult(false, + new ArrayList<>(Collections.singletonList(directory + "/*"))); + } + + boolean allDescendentsDeleted = true; + List notDeletedEligibleItems = new ArrayList<>(); + for (FileEntry file : allFiles) { + String fileName = new Path(file.location().uri()).getName(); + String filePath = file.location().uri(); + if (reverse ^ fileName.startsWith(queryId)) { + if (!deleteIfExists(new Path(filePath))) { + allDescendentsDeleted = false; + notDeletedEligibleItems.add(filePath); + } + } else { + allDescendentsDeleted = false; + } + } + + for (String dir : allDirs) { + DeleteRecursivelyResult subResult = doRecursiveDeleteFiles(new Path(dir), deleteEmptyDir, queryId, reverse); + if (!subResult.dirNotExists()) { + allDescendentsDeleted = false; + } + if (!subResult.getNotDeletedEligibleItems().isEmpty()) { + notDeletedEligibleItems.addAll(subResult.getNotDeletedEligibleItems()); + } + } + + if (allDescendentsDeleted && deleteEmptyDir) { + Verify.verify(notDeletedEligibleItems.isEmpty()); + if (!deleteDirectoryIfExists(directory)) { + return new DeleteRecursivelyResult(false, + new ArrayList<>(Collections.singletonList(directory + "/"))); + } + // all items of the location have been deleted. + return new DeleteRecursivelyResult(true, Collections.emptyList()); + } + return new DeleteRecursivelyResult(false, notDeletedEligibleItems); + } + + private boolean deleteIfExists(Path path) { + deleteFile(path.toString()); + try { + return !getFileSystem().exists(Location.of(path.toString())); + } catch (IOException e) { + return false; + } + } + + private boolean deleteDirectoryIfExists(Path path) { + deleteDir(path.toString()); + try { + return !getFileSystem().exists(Location.of(path.toString())); + } catch (IOException e) { + return false; + } + } + + private void deleteFile(String remotePath) { + try { + getFileSystem().delete(Location.of(remotePath), false); + } catch (IOException e) { + LOG.warn("Failed to delete {}: {}", remotePath, e.getMessage()); + } + } + + private void deleteDir(String remotePath) { + try { + getFileSystem().delete(Location.of(remotePath), true); + } catch (IOException e) { + LOG.warn("Failed to delete directory {}: {}", remotePath, e.getMessage()); + } + } + + private void renameDirectory(String origFilePath, String destFilePath, Runnable runWhenPathNotExist) { + try { + getFileSystem().renameDirectory(Location.of(origFilePath), Location.of(destFilePath), + runWhenPathNotExist); + } catch (IOException e) { + throw new RuntimeException("Failed to rename directory from " + origFilePath + + " to " + destFilePath + ": " + e.getMessage(), e); + } + } + + private void deleteTargetPathContents(String targetPath, String excludedChildPath) { + try { + Set dirs = getFileSystem().listDirectories(Location.of(targetPath)); + for (String dir : dirs) { + if (excludedChildPath != null && HiveWriteUtils.pathsEqual(dir, excludedChildPath)) { + continue; + } + deleteDir(dir); + } + List files = getFileSystem().listFiles(Location.of(targetPath)); + for (FileEntry file : files) { + deleteFile(file.location().uri()); + } + } catch (IOException e) { + throw new RuntimeException("Failed to list/delete contents under " + targetPath, e); + } + } + + private void ensureDirectory(String path) { + try { + getFileSystem().mkdirs(Location.of(path)); + } catch (IOException e) { + throw new RuntimeException("Failed to create directory " + path + ": " + e.getMessage(), e); + } + } + + // ─────────────────────────────── column conversion (GAP-4) ─────────────────────────────── + + private static List toFieldSchemas(List columns) { + List result = new ArrayList<>(columns.size()); + for (ConnectorColumn c : columns) { + result.add(new FieldSchema(c.getName(), HmsTypeMapping.toHiveTypeString(c.getType()), c.getComment())); + } + return result; + } + + // ─────────────────────────────── test seams (package-private) ─────────────────────────────── + + NameMapping getNameMapping() { + return nameMapping; + } + + Map> getTableActions() { + return tableActions; + } + + Map, Action>> getPartitionActions() { + return partitionActions; + } + + // ─────────────────────────────── static helpers ─────────────────────────────── + + private static ThreadFactory namedDaemonThreadFactory(String nameFormat) { + AtomicInteger counter = new AtomicInteger(0); + return runnable -> { + Thread thread = new Thread(runnable); + thread.setName(String.format(nameFormat, counter.getAndIncrement())); + thread.setDaemon(true); + return thread; + }; + } + + private static Object getFutureValue(CompletableFuture future) { + try { + return future.get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw new RuntimeException(e); + } catch (ExecutionException e) { + Throwable cause = e.getCause(); + if (cause instanceof RuntimeException) { + throw (RuntimeException) cause; + } + if (cause instanceof Error) { + throw (Error) cause; + } + throw new RuntimeException(cause); + } + } + + private static void addSuppressedExceptions(List suppressedExceptions, Throwable t, + List descriptions, String description) { + descriptions.add(description); + // A limit is needed to avoid having a huge exception object. 5 was chosen arbitrarily. + if (suppressedExceptions.size() < 5) { + suppressedExceptions.add(t); + } + } + + private static void shutdownExecutorService(ExecutorService executor) { + // Disable new tasks from being submitted. + executor.shutdown(); + try { + // Wait a while for existing tasks to terminate. + if (!executor.awaitTermination(60, TimeUnit.SECONDS)) { + // Cancel currently executing tasks. + executor.shutdownNow(); + // Wait a while for tasks to respond to being cancelled. + if (!executor.awaitTermination(60, TimeUnit.SECONDS)) { + LOG.warn("Pool did not terminate"); + } + } + } catch (InterruptedException e) { + // (Re-)Cancel if current thread also interrupted. + executor.shutdownNow(); + // Preserve interrupt status. + Thread.currentThread().interrupt(); + } + } + + // ─────────────────────────────── inner: committer ─────────────────────────────── + + class HmsCommitter { + + // update statistics for unPartitioned table or existed partition + private final List updateStatisticsTasks = new ArrayList<>(); + private final ExecutorService updateStatisticsExecutor = Executors.newFixedThreadPool(16); + + // add new partition + private final AddPartitionsTask addPartitionsTask = new AddPartitionsTask(); + + // for file system rename operation: whether to cancel the file system tasks + private final AtomicBoolean fileSystemTaskCancelled = new AtomicBoolean(false); + // file system tasks that are executed asynchronously, including rename_file, rename_dir + private final List> asyncFileSystemTaskFutures = new ArrayList<>(); + // when aborted, we need to delete all files under this path, even the current directory + private final Queue directoryCleanUpTasksForAbort = new ConcurrentLinkedQueue<>(); + // when aborted, we need restore directory + private final List renameDirectoryTasksForAbort = new ArrayList<>(); + // when finished, we need clear some directories + private final List clearDirsForFinish = new ArrayList<>(); + private final List s3cleanWhenSuccess = new ArrayList<>(); + + void cancelUnStartedAsyncFileSystemTask() { + fileSystemTaskCancelled.set(true); + } + + private void undoUpdateStatisticsTasks() { + List> undoUpdateFutures = new ArrayList<>(); + for (UpdateStatisticsTask task : updateStatisticsTasks) { + undoUpdateFutures.add(CompletableFuture.runAsync(() -> { + try { + task.undo(hmsClient); + } catch (Throwable throwable) { + LOG.warn("Failed to rollback: {}", task.getDescription(), throwable); + } + }, updateStatisticsExecutor)); + } + for (CompletableFuture undoUpdateFuture : undoUpdateFutures) { + getFutureValue(undoUpdateFuture); + } + updateStatisticsTasks.clear(); + } + + private void undoAddPartitionsTask() { + if (addPartitionsTask.isEmpty()) { + return; + } + List> rollbackFailedPartitions = addPartitionsTask.rollback(hmsClient); + if (!rollbackFailedPartitions.isEmpty()) { + LOG.warn("Failed to rollback: add_partition for partition values {}", rollbackFailedPartitions); + } + addPartitionsTask.clear(); + } + + private void waitForAsyncFileSystemTaskSuppressThrowable() { + for (CompletableFuture future : asyncFileSystemTaskFutures) { + try { + future.get(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } catch (Throwable t) { + // ignore + } + } + asyncFileSystemTaskFutures.clear(); + } + + void prepareInsertExistingTable(NameMapping nameMapping, TableAndMore tableAndMore) { + HmsTableInfo table = tableAndMore.getTable(); + String targetPath = table.getLocation(); + String writePath = tableAndMore.getCurrentLocation(); + // In the BE, all object stores are unified under the "s3" URI scheme, so a rename is only needed + // when target and write paths differ AFTER ignoring an s3-vs-native scheme difference (GAP-11: + // NOT HiveWriteUtils.pathsEqual, which treats a scheme mismatch as a different filesystem). + boolean needRename = !HiveWriteUtils.equalsIgnoreSchemeIfOneIsS3(targetPath, writePath); + if (needRename) { + FileSystemUtil.asyncRenameFiles(getFileSystem(), authWrappingExecutor, asyncFileSystemTaskFutures, + fileSystemTaskCancelled, writePath, targetPath, tableAndMore.getFileNames()); + } else if (hasPendingUploads(tableAndMore.getHivePartitionUpdate())) { + objCommit(asyncFileSystemTaskFutures, fileSystemTaskCancelled, + tableAndMore.getHivePartitionUpdate(), targetPath); + } + directoryCleanUpTasksForAbort.add(new DirectoryCleanUpTask(targetPath, false)); + updateStatisticsTasks.add(new UpdateStatisticsTask(nameMapping, Optional.empty(), + tableAndMore.getStatisticsUpdate(), true)); + } + + void prepareAlterTable(NameMapping nameMapping, TableAndMore tableAndMore) { + HmsTableInfo table = tableAndMore.getTable(); + String targetPath = table.getLocation(); + String writePath = tableAndMore.getCurrentLocation(); + if (!targetPath.equals(writePath)) { + if (HiveWriteUtils.isSubDirectory(targetPath, writePath)) { + String stagingRoot = HiveWriteUtils.getImmediateChildPath(targetPath, writePath); + deleteTargetPathContents(targetPath, stagingRoot); + ensureDirectory(targetPath); + FileSystemUtil.asyncRenameFiles(getFileSystem(), authWrappingExecutor, asyncFileSystemTaskFutures, + fileSystemTaskCancelled, writePath, targetPath, tableAndMore.getFileNames()); + } else { + Path path = new Path(targetPath); + String oldTablePath = new Path( + path.getParent(), "_temp_" + queryId + "_" + path.getName()).toString(); + renameDirectoryTasksForAbort.add(new RenameDirectoryTask(oldTablePath, targetPath)); + renameDirectory(targetPath, oldTablePath, () -> { }); + clearDirsForFinish.add(oldTablePath); + directoryCleanUpTasksForAbort.add(new DirectoryCleanUpTask(targetPath, true)); + renameDirectory(writePath, targetPath, () -> { }); + } + } else if (hasPendingUploads(tableAndMore.getHivePartitionUpdate())) { + s3cleanWhenSuccess.add(targetPath); + objCommit(asyncFileSystemTaskFutures, fileSystemTaskCancelled, + tableAndMore.getHivePartitionUpdate(), targetPath); + } + updateStatisticsTasks.add(new UpdateStatisticsTask(nameMapping, Optional.empty(), + tableAndMore.getStatisticsUpdate(), false)); + } + + void prepareAddPartition(NameMapping nameMapping, PartitionAndMore partitionAndMore) { + String targetPath = partitionAndMore.getTargetPath(); + String writePath = partitionAndMore.getCurrentLocation(); + if (!targetPath.equals(writePath)) { + directoryCleanUpTasksForAbort.add(new DirectoryCleanUpTask(targetPath, true)); + FileSystemUtil.asyncRenameDir(getFileSystem(), authWrappingExecutor, asyncFileSystemTaskFutures, + fileSystemTaskCancelled, writePath, targetPath, () -> { }); + } else if (hasPendingUploads(partitionAndMore.getHivePartitionUpdate())) { + objCommit(asyncFileSystemTaskFutures, fileSystemTaskCancelled, + partitionAndMore.getHivePartitionUpdate(), targetPath); + } + + // Rebuild the partition storage descriptor from the table at commit time (mirrors the legacy + // "build partition SD from table SD"): the ADD case is the only one that needs columns/formats. + HmsTableInfo table = getTable(nameMapping); + HmsPartitionWithStatistics partitionWithStats = HmsPartitionWithStatistics.builder() + .name(partitionAndMore.getPartitionName()) + .partitionValues(partitionAndMore.getPartitionValues()) + .location(targetPath) + .columns(toFieldSchemas(table.getColumns())) + .inputFormat(table.getInputFormat()) + .outputFormat(table.getOutputFormat()) + .serde(table.getSerializationLib()) + .parameters(new HashMap<>()) + .statistics(partitionAndMore.getStatisticsUpdate()) + .build(); + addPartitionsTask.addPartition(nameMapping, partitionWithStats); + } + + void prepareInsertExistPartition(NameMapping nameMapping, PartitionAndMore partitionAndMore) { + String targetPath = partitionAndMore.getTargetPath(); + String writePath = partitionAndMore.getCurrentLocation(); + directoryCleanUpTasksForAbort.add(new DirectoryCleanUpTask(targetPath, false)); + if (!targetPath.equals(writePath)) { + FileSystemUtil.asyncRenameFiles(getFileSystem(), authWrappingExecutor, asyncFileSystemTaskFutures, + fileSystemTaskCancelled, writePath, targetPath, partitionAndMore.getFileNames()); + } else if (hasPendingUploads(partitionAndMore.getHivePartitionUpdate())) { + objCommit(asyncFileSystemTaskFutures, fileSystemTaskCancelled, + partitionAndMore.getHivePartitionUpdate(), targetPath); + } + updateStatisticsTasks.add(new UpdateStatisticsTask(nameMapping, + Optional.of(partitionAndMore.getPartitionName()), partitionAndMore.getStatisticsUpdate(), true)); + } + + void prepareAlterPartition(NameMapping nameMapping, PartitionAndMore partitionAndMore) { + String targetPath = partitionAndMore.getTargetPath(); + String writePath = partitionAndMore.getCurrentLocation(); + if (!targetPath.equals(writePath)) { + Path path = new Path(targetPath); + String oldPartitionPath = new Path( + path.getParent(), "_temp_" + queryId + "_" + path.getName()).toString(); + renameDirectoryTasksForAbort.add(new RenameDirectoryTask(oldPartitionPath, targetPath)); + renameDirectory(targetPath, oldPartitionPath, () -> { }); + clearDirsForFinish.add(oldPartitionPath); + directoryCleanUpTasksForAbort.add(new DirectoryCleanUpTask(targetPath, true)); + renameDirectory(writePath, targetPath, () -> { }); + } else if (hasPendingUploads(partitionAndMore.getHivePartitionUpdate())) { + s3cleanWhenSuccess.add(targetPath); + objCommit(asyncFileSystemTaskFutures, fileSystemTaskCancelled, + partitionAndMore.getHivePartitionUpdate(), targetPath); + } + updateStatisticsTasks.add(new UpdateStatisticsTask(nameMapping, + Optional.of(partitionAndMore.getPartitionName()), partitionAndMore.getStatisticsUpdate(), false)); + } + + private void runDirectoryClearUpTasksForAbort() { + for (DirectoryCleanUpTask cleanUpTask : directoryCleanUpTasksForAbort) { + recursiveDeleteItems(cleanUpTask.getPath(), cleanUpTask.isDeleteEmptyDir(), false); + } + directoryCleanUpTasksForAbort.clear(); + } + + private void runRenameDirTasksForAbort() { + for (RenameDirectoryTask task : renameDirectoryTasksForAbort) { + try { + boolean srcExists = getFileSystem().exists(Location.of(task.getRenameFrom())); + if (srcExists) { + renameDirectory(task.getRenameFrom(), task.getRenameTo(), () -> { }); + } + } catch (IOException e) { + LOG.warn("Failed to abort rename dir from {} to {}: {}", + task.getRenameFrom(), task.getRenameTo(), e.getMessage()); + } + } + renameDirectoryTasksForAbort.clear(); + } + + void runClearPathsForFinish() { + for (String path : clearDirsForFinish) { + deleteDir(path); + } + } + + private void runS3cleanWhenSuccess() { + for (String path : s3cleanWhenSuccess) { + recursiveDeleteItems(new Path(path), false, true); + } + } + + private void waitForAsyncFileSystemTasks() { + for (CompletableFuture future : asyncFileSystemTaskFutures) { + getFutureValue(future); + } + } + + private void doAddPartitionsTask() { + // No committer-side batching: ThriftHmsClient.addPartitions batches internally (GAP-7), so the + // whole list is added in one call. + if (!addPartitionsTask.isEmpty()) { + addPartitionsTask.run(hmsClient); + } + } + + private void doUpdateStatisticsTasks() { + List> updateStatsFutures = new ArrayList<>(); + List failedTaskDescriptions = new ArrayList<>(); + List suppressedExceptions = new ArrayList<>(); + for (UpdateStatisticsTask task : updateStatisticsTasks) { + updateStatsFutures.add(CompletableFuture.runAsync(() -> { + try { + task.run(hmsClient); + } catch (Throwable t) { + synchronized (suppressedExceptions) { + addSuppressedExceptions(suppressedExceptions, t, failedTaskDescriptions, + task.getDescription()); + } + } + }, updateStatisticsExecutor)); + } + for (CompletableFuture executeUpdateFuture : updateStatsFutures) { + getFutureValue(executeUpdateFuture); + } + if (!suppressedExceptions.isEmpty()) { + StringBuilder message = new StringBuilder(); + message.append("Failed to execute some updating statistics tasks: "); + message.append(String.join("; ", failedTaskDescriptions)); + RuntimeException exception = new RuntimeException(message.toString()); + suppressedExceptions.forEach(exception::addSuppressed); + throw exception; + } + } + + private void pruneAndDeleteStagingDirectories() { + stagingDirectory.ifPresent((v) -> recursiveDeleteItems(new Path(v), true, false)); + } + + private void abortMultiUploads() { + if (uncompletedMpuPendingUploads.isEmpty()) { + return; + } + FileSystem resolved = getFileSystem(); + for (UncompletedMpuPendingUpload uncompletedMpuPendingUpload : uncompletedMpuPendingUploads) { + if (!(resolved instanceof ObjFileSystem)) { + LOG.warn("FileSystem {} is not object-storage, skipping MPU abort for {}", + resolved.getClass().getSimpleName(), uncompletedMpuPendingUpload.path); + continue; + } + ObjFileSystem objFs = (ObjFileSystem) resolved; + TS3MPUPendingUpload mpu = uncompletedMpuPendingUpload.s3MPUPendingUpload; + String remotePath = "s3://" + mpu.getBucket() + "/" + mpu.getKey(); + asyncFileSystemTaskFutures.add(CompletableFuture.runAsync(() -> { + try { + context.executeAuthenticated(() -> { + objFs.getObjStorage().abortMultipartUpload(remotePath, mpu.getUploadId()); + return null; + }); + } catch (Exception e) { + LOG.warn("Failed to abort MPU for {}: {}", remotePath, e.getMessage()); + } + }, fileSystemExecutor)); + } + uncompletedMpuPendingUploads.clear(); + } + + void doCommit() { + waitForAsyncFileSystemTasks(); + runS3cleanWhenSuccess(); + doAddPartitionsTask(); + doUpdateStatisticsTasks(); + // delete write path + pruneAndDeleteStagingDirectories(); + } + + void abort() { + cancelUnStartedAsyncFileSystemTask(); + undoUpdateStatisticsTasks(); + undoAddPartitionsTask(); + waitForAsyncFileSystemTaskSuppressThrowable(); + runDirectoryClearUpTasksForAbort(); + runRenameDirTasksForAbort(); + } + + void rollback() { + // delete write path + pruneAndDeleteStagingDirectories(); + // abort the in-progress multipart uploads + abortMultiUploads(); + for (CompletableFuture future : asyncFileSystemTaskFutures) { + getFutureValue(future); + } + asyncFileSystemTaskFutures.clear(); + } + + void shutdownExecutorService() { + HiveConnectorTransaction.shutdownExecutorService(updateStatisticsExecutor); + } + + private boolean hasPendingUploads(THivePartitionUpdate pu) { + List uploads = pu.getS3MpuPendingUploads(); + return uploads != null && !uploads.isEmpty(); + } + } + + // ─────────────────────────────── inner: tasks ─────────────────────────────── + + private static class UpdateStatisticsTask { + private final NameMapping nameMapping; + private final Optional partitionName; + private final HmsPartitionStatistics updatePartitionStat; + private final boolean merge; + private boolean done; + + UpdateStatisticsTask(NameMapping nameMapping, Optional partitionName, + HmsPartitionStatistics statistics, boolean merge) { + this.nameMapping = Objects.requireNonNull(nameMapping, "nameMapping is null"); + this.partitionName = Objects.requireNonNull(partitionName, "partitionName is null"); + this.updatePartitionStat = Objects.requireNonNull(statistics, "statistics is null"); + this.merge = merge; + } + + void run(HmsClient client) { + if (partitionName.isPresent()) { + client.updatePartitionStatistics(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), + partitionName.get(), this::updateStatistics); + } else { + client.updateTableStatistics(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), + this::updateStatistics); + } + done = true; + } + + void undo(HmsClient client) { + if (!done) { + return; + } + if (partitionName.isPresent()) { + client.updatePartitionStatistics(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), + partitionName.get(), this::resetStatistics); + } else { + client.updateTableStatistics(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), + this::resetStatistics); + } + } + + String getDescription() { + if (partitionName.isPresent()) { + return "alter partition parameters " + nameMapping.getFullLocalName() + " " + partitionName.get(); + } else { + return "alter table parameters " + nameMapping.getFullRemoteName(); + } + } + + private HmsPartitionStatistics updateStatistics(HmsPartitionStatistics currentStats) { + return merge ? HmsPartitionStatistics.merge(currentStats, updatePartitionStat) : updatePartitionStat; + } + + private HmsPartitionStatistics resetStatistics(HmsPartitionStatistics currentStatistics) { + // GAP-2: ReduceOperator lives on HmsCommonStatistics. + return HmsPartitionStatistics.reduce(currentStatistics, updatePartitionStat, + HmsCommonStatistics.ReduceOperator.SUBTRACT); + } + } + + private static class AddPartitionsTask { + private final List partitions = new ArrayList<>(); + private final List> createdPartitionValues = new ArrayList<>(); + private NameMapping nameMapping; + + boolean isEmpty() { + return partitions.isEmpty(); + } + + void clear() { + partitions.clear(); + createdPartitionValues.clear(); + } + + void addPartition(NameMapping nameMapping, HmsPartitionWithStatistics partition) { + this.nameMapping = nameMapping; + partitions.add(partition); + } + + void run(HmsClient client) { + client.addPartitions(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), partitions); + for (HmsPartitionWithStatistics partition : partitions) { + createdPartitionValues.add(partition.getPartitionValues()); + } + } + + List> rollback(HmsClient client) { + List> rollbackFailedPartitions = new ArrayList<>(); + for (List createdPartitionValue : createdPartitionValues) { + try { + client.dropPartition(nameMapping.getRemoteDbName(), nameMapping.getRemoteTblName(), + createdPartitionValue, false); + } catch (Throwable t) { + LOG.warn("Failed to drop partition on {} when rollback: {}", + nameMapping.getFullLocalName(), createdPartitionValue); + rollbackFailedPartitions.add(createdPartitionValue); + } + } + return rollbackFailedPartitions; + } + } + + private static class DirectoryCleanUpTask { + private final Path path; + private final boolean deleteEmptyDir; + + DirectoryCleanUpTask(String path, boolean deleteEmptyDir) { + this.path = new Path(path); + this.deleteEmptyDir = deleteEmptyDir; + } + + Path getPath() { + return path; + } + + boolean isDeleteEmptyDir() { + return deleteEmptyDir; + } + } + + private static class DeleteRecursivelyResult { + private final boolean dirNoLongerExists; + private final List notDeletedEligibleItems; + + DeleteRecursivelyResult(boolean dirNoLongerExists, List notDeletedEligibleItems) { + this.dirNoLongerExists = dirNoLongerExists; + this.notDeletedEligibleItems = notDeletedEligibleItems; + } + + boolean dirNotExists() { + return dirNoLongerExists; + } + + List getNotDeletedEligibleItems() { + return notDeletedEligibleItems; + } + } + + private static class RenameDirectoryTask { + private final String renameFrom; + private final String renameTo; + + RenameDirectoryTask(String renameFrom, String renameTo) { + this.renameFrom = renameFrom; + this.renameTo = renameTo; + } + + String getRenameFrom() { + return renameFrom; + } + + String getRenameTo() { + return renameTo; + } + } + + private static class UncompletedMpuPendingUpload { + private final TS3MPUPendingUpload s3MPUPendingUpload; + private final String path; + + UncompletedMpuPendingUpload(TS3MPUPendingUpload s3MPUPendingUpload, String path) { + this.s3MPUPendingUpload = s3MPUPendingUpload; + this.path = path; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (o == null || getClass() != o.getClass()) { + return false; + } + UncompletedMpuPendingUpload that = (UncompletedMpuPendingUpload) o; + return Objects.equals(s3MPUPendingUpload, that.s3MPUPendingUpload) && Objects.equals(path, that.path); + } + + @Override + public int hashCode() { + return Objects.hash(s3MPUPendingUpload, path); + } + } + + // ─────────────────────────────── inner: action model ─────────────────────────────── + + enum ActionType { + // drop a table/partition + DROP, + // drop a table/partition but will preserve data + DROP_PRESERVE_DATA, + // add a table/partition + ADD, + // drop then add a table/partition, like overwrite + ALTER, + // insert into an existing table/partition + INSERT_EXISTING, + // merge into an existing table/partition + MERGE + } + + static class Action { + private final ActionType type; + private final T data; + + Action(ActionType type, T data) { + this.type = Objects.requireNonNull(type, "type is null"); + if (type == ActionType.DROP || type == ActionType.DROP_PRESERVE_DATA) { + Preconditions.checkArgument(data == null, "data is not null"); + } else { + Objects.requireNonNull(data, "data is null"); + } + this.data = data; + } + + ActionType getType() { + return type; + } + + T getData() { + Preconditions.checkState(type != ActionType.DROP); + return data; + } + } + + private static class TableAndMore { + private final HmsTableInfo table; + private final String currentLocation; + private final List fileNames; + private final HmsPartitionStatistics statisticsUpdate; + private final THivePartitionUpdate hivePartitionUpdate; + + TableAndMore(HmsTableInfo table, String currentLocation, List fileNames, + HmsPartitionStatistics statisticsUpdate, THivePartitionUpdate hivePartitionUpdate) { + this.table = Objects.requireNonNull(table, "table is null"); + this.currentLocation = Objects.requireNonNull(currentLocation, "currentLocation is null"); + this.fileNames = Objects.requireNonNull(fileNames, "fileNames is null"); + this.statisticsUpdate = Objects.requireNonNull(statisticsUpdate, "statisticsUpdate is null"); + this.hivePartitionUpdate = Objects.requireNonNull(hivePartitionUpdate, "hivePartitionUpdate is null"); + } + + HmsTableInfo getTable() { + return table; + } + + String getCurrentLocation() { + return currentLocation; + } + + List getFileNames() { + return fileNames; + } + + HmsPartitionStatistics getStatisticsUpdate() { + return statisticsUpdate; + } + + THivePartitionUpdate getHivePartitionUpdate() { + return hivePartitionUpdate; + } + } + + private static class PartitionAndMore { + private final List partitionValues; + private final String targetPath; + private final String currentLocation; + private final String partitionName; + private final List fileNames; + private final HmsPartitionStatistics statisticsUpdate; + private final THivePartitionUpdate hivePartitionUpdate; + + PartitionAndMore(List partitionValues, String targetPath, String currentLocation, + String partitionName, List fileNames, HmsPartitionStatistics statisticsUpdate, + THivePartitionUpdate hivePartitionUpdate) { + this.partitionValues = Objects.requireNonNull(partitionValues, "partitionValues is null"); + this.targetPath = Objects.requireNonNull(targetPath, "targetPath is null"); + this.currentLocation = Objects.requireNonNull(currentLocation, "currentLocation is null"); + this.partitionName = Objects.requireNonNull(partitionName, "partitionName is null"); + this.fileNames = Objects.requireNonNull(fileNames, "fileNames is null"); + this.statisticsUpdate = Objects.requireNonNull(statisticsUpdate, "statisticsUpdate is null"); + this.hivePartitionUpdate = Objects.requireNonNull(hivePartitionUpdate, "hivePartitionUpdate is null"); + } + + List getPartitionValues() { + return partitionValues; + } + + String getTargetPath() { + return targetPath; + } + + String getCurrentLocation() { + return currentLocation; + } + + String getPartitionName() { + return partitionName; + } + + List getFileNames() { + return fileNames; + } + + HmsPartitionStatistics getStatisticsUpdate() { + return statisticsUpdate; + } + + THivePartitionUpdate getHivePartitionUpdate() { + return hivePartitionUpdate; + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransaction.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransaction.java new file mode 100644 index 00000000000000..3f6247402356e6 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransaction.java @@ -0,0 +1,100 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.hms.HmsClient; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * Holds the state of one Hive read transaction, used when reading a transactional (ACID) Hive table. + * Each instance is bound to a single query. + * + *

Plugin-side port of fe-core {@code HiveTransaction}. It drives the four read-side metastore + * primitives through the shared {@link HmsClient} ({@code openTxn} / {@code acquireSharedLock} / + * {@code getValidWriteIds} / {@code commitTxn}) instead of the fe-core {@code HMSExternalCatalog}/ + * {@code HMSCachedClient}/{@code TableNameInfo} chain — the table identity is carried as plain + * {@code (dbName, tableName)} strings.

+ */ +public class HiveReadTransaction { + private final String queryId; + private final String user; + private final String dbName; + private final String tableName; + private final boolean isFullAcid; + private final HmsClient hmsClient; + + private long txnId; + private final List partitionNames = new ArrayList<>(); + + private Map txnValidIds = null; + + public HiveReadTransaction(String queryId, String user, String dbName, String tableName, + boolean isFullAcid, HmsClient hmsClient) { + this.queryId = queryId; + this.user = user; + this.dbName = dbName; + this.tableName = tableName; + this.isFullAcid = isFullAcid; + this.hmsClient = hmsClient; + } + + public String getQueryId() { + return queryId; + } + + public void addPartition(String partitionName) { + this.partitionNames.add(partitionName); + } + + public boolean isFullAcid() { + return isFullAcid; + } + + /** + * Acquires a shared read lock and fetches the snapshot ({@code ValidTxnList} + {@code + * ValidWriteIdList}) for the current transaction, memoizing so the lock is taken at most once. The + * lock is released when the transaction is committed at query finish. + */ + public Map getValidWriteIds() { + if (txnValidIds == null) { + hmsClient.acquireSharedLock(queryId, txnId, user, dbName, tableName, partitionNames, 5000); + txnValidIds = hmsClient.getValidWriteIds(dbName + "." + tableName, txnId); + } + return txnValidIds; + } + + public void begin() { + try { + this.txnId = hmsClient.openTxn(user); + } catch (RuntimeException e) { + throw new DorisConnectorException(e.getMessage(), e); + } + } + + public void commit() { + try { + hmsClient.commitTxn(txnId); + } catch (RuntimeException e) { + throw new DorisConnectorException(e.getMessage(), e); + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransactionManager.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransactionManager.java new file mode 100644 index 00000000000000..b5b36ff171b091 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveReadTransactionManager.java @@ -0,0 +1,62 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + +/** + * Manages the read transaction of each query against transactional (ACID) Hive tables. For each query a + * {@link HiveReadTransaction} is registered (which opens the metastore transaction); when the query + * finishes it is deregistered (which commits the transaction, releasing the shared read lock). + * + *

Plugin-side port of fe-core {@code HiveTransactionMgr}. It is plugin-owned and dormant until + * the read cutover: today transactional-hive reads still flow through fe-core + * {@code HiveScanNode}/{@code Env.getCurrentHiveTransactionMgr()}. At cutover the query-finish trigger + * (fe-core {@code QueryFinishCallbackRegistry}) is wired to {@link #deregister} and the legacy + * {@code Env} manager is removed. Until then nothing invokes this manager on a live path.

+ */ +public class HiveReadTransactionManager { + private static final Logger LOG = LogManager.getLogger(HiveReadTransactionManager.class); + + private final Map txnMap = new ConcurrentHashMap<>(); + + /** + * Opens the transaction (see {@link HiveReadTransaction#begin}) and registers it under its query id, so + * the query-finish path can find and commit it. Mirrors fe-core: one {@code HiveReadTransaction} is + * created per transactional-table scan (each pins its own table's write-id snapshot), keyed by query id. + */ + public void register(HiveReadTransaction txn) { + txn.begin(); + txnMap.put(txn.getQueryId(), txn); + } + + public void deregister(String queryId) { + HiveReadTransaction txn = txnMap.remove(queryId); + if (txn != null) { + try { + txn.commit(); + } catch (RuntimeException e) { + LOG.warn("failed to commit hive read txn: " + queryId, e); + } + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 2baae9a79926d3..f567c393e4a566 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -78,12 +78,18 @@ public class HiveScanPlanProvider implements ConnectorScanPlanProvider { public static final String PROP_PATH_PARTITION_KEYS = "path_partition_keys"; public static final String PROP_LOCATION_PREFIX = "location."; + /** Input format of a full-ACID (ORC) transactional Hive table; other formats are rejected. */ + private static final String ORC_ACID_INPUT_FORMAT = "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + private final HmsClient hmsClient; private final Map catalogProperties; + private final HiveReadTransactionManager readTxnManager; - public HiveScanPlanProvider(HmsClient hmsClient, Map catalogProperties) { + public HiveScanPlanProvider(HmsClient hmsClient, Map catalogProperties, + HiveReadTransactionManager readTxnManager) { this.hmsClient = hmsClient; this.catalogProperties = catalogProperties; + this.readTxnManager = readTxnManager; } @Override @@ -114,21 +120,113 @@ public List planScan( List ranges = new ArrayList<>(); Configuration hadoopConf = buildHadoopConf(); + if (hiveHandle.isTransactional()) { + // Transactional (ACID) table: descend into base/delta directories under the query's + // write-id snapshot and emit ACID-annotated ranges. (Dormant — see planAcidScan.) + planAcidScan(session, hiveHandle, partitions, hadoopConf, fileFormat, + splittable, targetSplitSize, ranges); + } else { + for (PartitionScanInfo partition : partitions) { + HiveFileFormat partFormat = partition.fileFormat != null + ? partition.fileFormat : fileFormat; + try { + listAndSplitFiles(hadoopConf, partition, partFormat, + splittable, targetSplitSize, ranges); + } catch (IOException e) { + throw new DorisConnectorException( + "Failed to list files for partition: " + partition.location, e); + } + } + } + + LOG.info("Hive scan plan: table={}.{}, partitions={}, splits={}", + dbName, tableName, partitions.size(), ranges.size()); + return ranges; + } + + /** + * Plans a scan of a transactional (ACID) Hive table. + * + *

Opens (or reuses) the query's read transaction — which acquires a shared read lock and pins a + * write-id snapshot — then, per partition, runs {@link HiveAcidUtil#getAcidState} to resolve the + * surviving base/delta data files and delete-delta directories, and emits one ACID-annotated + * {@link HiveScanRange} per data-file split. The BE subtracts the delete deltas on read.

+ * + *

Dormant: hive is not yet in {@code SPI_READY_TYPES}, so this path is never reached on a + * live query — transactional reads still flow through fe-core {@code HiveScanNode}. It opens a real + * metastore transaction/lock; the matching commit (lock release) is driven by + * {@link HiveReadTransactionManager#deregister} at query finish, wired only at the read cutover + * (P7.4/P7.5) together with routing hive through the plugin. Do not flip one without the other, or a + * shared read lock would leak.

+ */ + private void planAcidScan(ConnectorSession session, HiveTableHandle handle, + List partitions, Configuration hadoopConf, HiveFileFormat fileFormat, + boolean splittable, long targetSplitSize, List ranges) { + boolean isFullAcid = handle.isFullAcid(); + if (isFullAcid && !ORC_ACID_INPUT_FORMAT.equals(handle.getInputFormat())) { + // Full-ACID data is only stored in the ORC ACID layout; reject other formats loudly + // (legacy parity: HMSExternalTable.isFullAcidTable throws "no Orc Format"). + throw new DorisConnectorException("This table is full Acid Table, but no Orc Format."); + } + + // One transaction per transactional-table scan, pinning this table's write-id snapshot (mirrors + // fe-core, which opens a HiveTransaction per scan node). The manager holds it for commit at finish. + HiveReadTransaction txn = new HiveReadTransaction(session.getQueryId(), session.getUser(), + handle.getDbName(), handle.getTableName(), isFullAcid, hmsClient); + readTxnManager.register(txn); + for (PartitionScanInfo partition : partitions) { + if (!partition.partitionValues.isEmpty()) { + txn.addPartition(buildPartitionName(partition.partitionValues)); + } + } + Map txnValidIds = txn.getValidWriteIds(); + for (PartitionScanInfo partition : partitions) { HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; try { - listAndSplitFiles(hadoopConf, partition, partFormat, - splittable, targetSplitSize, ranges); + Path partPath = new Path(partition.location); + FileSystem fs = FileSystem.get(partPath.toUri(), hadoopConf); + HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( + fs, partition.location, txnValidIds, isFullAcid); + // Only full-ACID reads are marked transactional_hive (delete deltas applied by the BE + // merge-on-read reader). Insert-only tables use the write-id snapshot to pick files but + // store plain data files, so their splits stay plain hive — matching legacy AcidUtil, + // which sets acidInfo only when isFullAcid. + String acidLocation = isFullAcid ? partition.location : null; + List encodedDeltas = isFullAcid + ? encodeDeleteDeltas(state.getDeleteDeltas()) : null; + for (FileStatus dataFile : state.getDataFiles()) { + splitFile(dataFile, partition, partFormat, splittable, targetSplitSize, + acidLocation, encodedDeltas, ranges); + } } catch (IOException e) { throw new DorisConnectorException( - "Failed to list files for partition: " + partition.location, e); + "Failed to list ACID files for partition: " + partition.location, e); } } + } - LOG.info("Hive scan plan: table={}.{}, partitions={}, splits={}", - dbName, tableName, partitions.size(), ranges.size()); - return ranges; + /** Encodes each delete-delta as {@code "dir|file1,file2"} for {@link HiveScanRange.Builder#acidInfo}. */ + private static List encodeDeleteDeltas(List deltas) { + List encoded = new ArrayList<>(deltas.size()); + for (HiveAcidUtil.DeleteDelta delta : deltas) { + encoded.add(delta.getDirectoryLocation() + "|" + + String.join(",", delta.getFileNames())); + } + return encoded; + } + + /** Builds the Hive partition name {@code k1=v1/k2=v2} used for the shared read lock components. */ + private static String buildPartitionName(Map partitionValues) { + StringBuilder sb = new StringBuilder(); + for (Map.Entry entry : partitionValues.entrySet()) { + if (sb.length() > 0) { + sb.append('/'); + } + sb.append(entry.getKey()).append('=').append(entry.getValue()); + } + return sb.toString(); } @Override @@ -245,16 +343,20 @@ private void listAndSplitFiles(Configuration conf, continue; } splitFile(status, partition, fileFormat, splittable, - targetSplitSize, ranges); + targetSplitSize, null, null, ranges); } } /** * Splits a file into scan ranges based on target split size. + * + *

When {@code acidPartitionLocation} is non-null the ranges carry ACID delete-delta info + * (marking them {@code transactional_hive}); otherwise they are plain Hive ranges.

*/ private void splitFile(FileStatus fileStatus, PartitionScanInfo partition, - HiveFileFormat fileFormat, boolean splittable, - long targetSplitSize, List ranges) { + HiveFileFormat fileFormat, boolean splittable, long targetSplitSize, + String acidPartitionLocation, List acidDeltas, + List ranges) { long fileSize = fileStatus.getLen(); String filePath = fileStatus.getPath().toString(); long modTime = fileStatus.getModificationTime(); @@ -265,16 +367,8 @@ private void splitFile(FileStatus fileStatus, PartitionScanInfo partition, if (!splittable || targetSplitSize <= 0 || fileSize <= targetSplitSize) { // Single range for the whole file - ranges.add(HiveScanRange.builder() - .path(filePath) - .start(0) - .length(fileSize) - .fileSize(fileSize) - .modificationTime(modTime) - .fileFormat(fileFormat.getFormatName()) - .tableFormatType("hive") - .partitionValues(partition.partitionValues) - .build()); + ranges.add(newRangeBuilder(filePath, 0, fileSize, fileSize, modTime, fileFormat, + partition, acidPartitionLocation, acidDeltas).build()); return; } @@ -282,20 +376,31 @@ private void splitFile(FileStatus fileStatus, PartitionScanInfo partition, long offset = 0; while (offset < fileSize) { long splitSize = Math.min(targetSplitSize, fileSize - offset); - ranges.add(HiveScanRange.builder() - .path(filePath) - .start(offset) - .length(splitSize) - .fileSize(fileSize) - .modificationTime(modTime) - .fileFormat(fileFormat.getFormatName()) - .tableFormatType("hive") - .partitionValues(partition.partitionValues) - .build()); + ranges.add(newRangeBuilder(filePath, offset, splitSize, fileSize, modTime, fileFormat, + partition, acidPartitionLocation, acidDeltas).build()); offset += splitSize; } } + private static HiveScanRange.Builder newRangeBuilder(String filePath, long start, long length, + long fileSize, long modTime, HiveFileFormat fileFormat, PartitionScanInfo partition, + String acidPartitionLocation, List acidDeltas) { + HiveScanRange.Builder builder = HiveScanRange.builder() + .path(filePath) + .start(start) + .length(length) + .fileSize(fileSize) + .modificationTime(modTime) + .fileFormat(fileFormat.getFormatName()) + .tableFormatType("hive") + .partitionValues(partition.partitionValues); + if (acidPartitionLocation != null) { + // Sets tableFormatType="transactional_hive" and attaches the delete-delta descriptors. + builder.acidInfo(acidPartitionLocation, acidDeltas); + } + return builder; + } + private boolean shouldSkipFile(String fileName) { return fileName.startsWith("_") || fileName.startsWith("."); } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveSinkHelper.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveSinkHelper.java new file mode 100644 index 00000000000000..37a6f2d2b51a8e --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveSinkHelper.java @@ -0,0 +1,246 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.thrift.TFileCompressType; +import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.THiveSerDeProperties; + +import java.util.EnumSet; +import java.util.Locale; +import java.util.Map; +import java.util.Optional; +import java.util.Set; + +/** + * Pure, metastore-parameter-only helpers ported verbatim from the legacy fe-core write planner + * ({@code planner.BaseExternalTableDataSink} format/compress resolution and + * {@code datasource.hive.HiveProperties} + {@code HiveMetaStoreClientHelper} SerDe-delimiter resolution), + * homed here so {@link HiveWritePlanProvider} stays readable. Operates only on the SPI + * {@link HmsTableInfo} (table params + SD params + serialization lib) and Thrift types — no fe-core. + */ +final class HiveSinkHelper { + + private HiveSinkHelper() { + } + + // The write file-format types the hive sink supports (legacy HiveTableSink.supportedTypes). FORMAT_UNKNOWN + // is intentionally absent so an unrecognized input format is rejected loudly. + private static final Set SUPPORTED_FORMAT_TYPES = EnumSet.of( + TFileFormatType.FORMAT_CSV_PLAIN, + TFileFormatType.FORMAT_ORC, + TFileFormatType.FORMAT_PARQUET, + TFileFormatType.FORMAT_TEXT); + + // Serialization lib that triggers multi-char field delimiters (legacy + // HiveMetaStoreClientHelper.HIVE_MULTI_DELIMIT_SERDE — NOT the contrib MultiDelimitSerDe variant). + private static final String HIVE_MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; + + // SerDe / SD property keys (legacy HiveProperties). + private static final String PROP_FIELD_DELIMITER = "field.delim"; + private static final String PROP_SERIALIZATION_FORMAT = "serialization.format"; + private static final String PROP_LINE_DELIMITER = "line.delim"; + // "colelction.delim" is Hive2's intentional typo; "collection.delim" is Hive3. Both are checked. + private static final String PROP_COLLECTION_DELIMITER_HIVE2 = "colelction.delim"; + private static final String PROP_COLLECTION_DELIMITER_HIVE3 = "collection.delim"; + private static final String PROP_MAP_KV_DELIMITER = "mapkey.delim"; + private static final String PROP_ESCAPE_DELIMITER = "escape.delim"; + private static final String PROP_NULL_FORMAT = "serialization.null.format"; + + // Per-delimiter defaults (legacy HiveProperties). + private static final String DEFAULT_FIELD_DELIMITER = "\1"; + private static final String DEFAULT_LINE_DELIMITER = "\n"; + private static final String DEFAULT_COLLECTION_DELIMITER = "\2"; + private static final String DEFAULT_MAP_KV_DELIMITER = "\003"; + private static final String DEFAULT_ESCAPE_DELIMITER = "\\"; + private static final String DEFAULT_NULL_FORMAT = "\\N"; + + /** + * Port of legacy {@code BaseExternalTableDataSink.getTFileFormatType}. LZO input formats are rejected + * FIRST — their class names also contain "text" (e.g. {@code LzoTextInputFormat}) and would otherwise + * match {@code FORMAT_CSV_PLAIN}, but the BE hive sink has no LZO codec so a Doris-written LZO partition + * becomes permanently invisible. Then {@code orc}/{@code parquet}/{@code text} substring-match (text maps + * to {@code FORMAT_CSV_PLAIN}); an unsupported result is rejected loudly. Called for the table SD and for + * every existing partition SD (per-partition LZO guard). + */ + static TFileFormatType getTFileFormatType(String format) { + if (format != null && format.toLowerCase(Locale.ROOT).contains("lzo")) { + throw new DorisConnectorException("INSERT INTO is not supported for LZO Hive tables " + + "(input format: " + format + "). LZO tables are read-only in Doris."); + } + TFileFormatType fileFormatType = TFileFormatType.FORMAT_UNKNOWN; + String lowerCase = format.toLowerCase(Locale.ROOT); + if (lowerCase.contains("orc")) { + fileFormatType = TFileFormatType.FORMAT_ORC; + } else if (lowerCase.contains("parquet")) { + fileFormatType = TFileFormatType.FORMAT_PARQUET; + } else if (lowerCase.contains("text")) { + fileFormatType = TFileFormatType.FORMAT_CSV_PLAIN; + } + if (!SUPPORTED_FORMAT_TYPES.contains(fileFormatType)) { + throw new DorisConnectorException("Unsupported input format type: " + format); + } + return fileFormatType; + } + + /** Port of legacy {@code BaseExternalTableDataSink.getTFileCompressType} (hive codecs; null -> PLAIN). */ + static TFileCompressType getTFileCompressType(String compressType) { + if ("snappy".equalsIgnoreCase(compressType)) { + return TFileCompressType.SNAPPYBLOCK; + } else if ("lz4".equalsIgnoreCase(compressType)) { + return TFileCompressType.LZ4BLOCK; + } else if ("lzo".equalsIgnoreCase(compressType)) { + return TFileCompressType.LZO; + } else if ("zlib".equalsIgnoreCase(compressType)) { + return TFileCompressType.ZLIB; + } else if ("zstd".equalsIgnoreCase(compressType)) { + return TFileCompressType.ZSTD; + } else if ("gzip".equalsIgnoreCase(compressType)) { + return TFileCompressType.GZ; + } else if ("bzip2".equalsIgnoreCase(compressType)) { + return TFileCompressType.BZ2; + } else if ("uncompressed".equalsIgnoreCase(compressType)) { + return TFileCompressType.PLAIN; + } else { + // try to use plain type to decompress parquet or orc file + return TFileCompressType.PLAIN; + } + } + + /** + * Port of legacy {@code HiveTableSink.setSerDeProperties} + {@code HiveProperties}: builds the six BE + * SerDe delimiters from the table's SD/table parameters. Field delimiter allows multi-char only for the + * {@code MultiDelimitSerDe} lib; the escape char is emitted only when present. + */ + static THiveSerDeProperties buildSerDeProperties(HmsTableInfo table) { + Map tableParams = table.getParameters(); + Map sdParams = table.getSdParameters(); + String serDeLib = table.getSerializationLib(); + + THiveSerDeProperties serDeProperties = new THiveSerDeProperties(); + // 1. field delimiter + if (HIVE_MULTI_DELIMIT_SERDE.equals(serDeLib)) { + serDeProperties.setFieldDelim(getFieldDelimiter(tableParams, sdParams, true)); + } else { + serDeProperties.setFieldDelim(getFieldDelimiter(tableParams, sdParams, false)); + } + // 2. line delimiter + serDeProperties.setLineDelim(getLineDelimiter(tableParams, sdParams)); + // 3. collection delimiter + serDeProperties.setCollectionDelim(getCollectionDelimiter(tableParams, sdParams)); + // 4. mapkv delimiter + serDeProperties.setMapkvDelim(getMapKvDelimiter(tableParams, sdParams)); + // 5. escape delimiter (only when present) + getEscapeDelimiter(tableParams, sdParams).ifPresent(serDeProperties::setEscapeChar); + // 6. null format + serDeProperties.setNullFormat(getNullFormat(tableParams, sdParams)); + return serDeProperties; + } + + private static String getFieldDelimiter(Map tableParams, Map sdParams, + boolean supportMultiChar) { + Optional fieldDelim = getSerdeProperty(tableParams, sdParams, PROP_FIELD_DELIMITER); + Optional serFormat = getSerdeProperty(tableParams, sdParams, PROP_SERIALIZATION_FORMAT); + String delimiter = firstPresentOrDefault("", fieldDelim, serFormat); + return supportMultiChar ? delimiter : getByte(delimiter, DEFAULT_FIELD_DELIMITER); + } + + private static String getLineDelimiter(Map tableParams, Map sdParams) { + Optional lineDelim = getSerdeProperty(tableParams, sdParams, PROP_LINE_DELIMITER); + return getByte(firstPresentOrDefault("", lineDelim), DEFAULT_LINE_DELIMITER); + } + + private static String getMapKvDelimiter(Map tableParams, Map sdParams) { + Optional mapkvDelim = getSerdeProperty(tableParams, sdParams, PROP_MAP_KV_DELIMITER); + return getByte(firstPresentOrDefault("", mapkvDelim), DEFAULT_MAP_KV_DELIMITER); + } + + private static String getCollectionDelimiter(Map tableParams, Map sdParams) { + Optional collectionDelimHive2 = getSerdeProperty(tableParams, sdParams, + PROP_COLLECTION_DELIMITER_HIVE2); + Optional collectionDelimHive3 = getSerdeProperty(tableParams, sdParams, + PROP_COLLECTION_DELIMITER_HIVE3); + return getByte(firstPresentOrDefault("", collectionDelimHive2, collectionDelimHive3), + DEFAULT_COLLECTION_DELIMITER); + } + + private static Optional getEscapeDelimiter(Map tableParams, + Map sdParams) { + Optional escapeDelim = getSerdeProperty(tableParams, sdParams, PROP_ESCAPE_DELIMITER); + if (escapeDelim.isPresent()) { + return Optional.of(getByte(escapeDelim.get(), DEFAULT_ESCAPE_DELIMITER)); + } + return Optional.empty(); + } + + private static String getNullFormat(Map tableParams, Map sdParams) { + Optional nullFormat = getSerdeProperty(tableParams, sdParams, PROP_NULL_FORMAT); + return firstPresentOrDefault(DEFAULT_NULL_FORMAT, nullFormat); + } + + // Port of legacy HiveMetaStoreClientHelper.getSerdeProperty: table parameters win over SD SerDe parameters. + private static Optional getSerdeProperty(Map tableParams, + Map sdParams, String key) { + String valueFromTbl = tableParams == null ? null : tableParams.get(key); + String valueFromSd = sdParams == null ? null : sdParams.get(key); + return firstNonNullable(valueFromTbl, valueFromSd); + } + + private static Optional firstNonNullable(String first, String second) { + if (first != null) { + return Optional.of(first); + } + if (second != null) { + return Optional.of(second); + } + return Optional.empty(); + } + + private static String firstPresentOrDefault(String defaultValue, Optional first) { + return first.orElse(defaultValue); + } + + private static String firstPresentOrDefault(String defaultValue, Optional first, + Optional second) { + if (first.isPresent()) { + return first.get(); + } + if (second.isPresent()) { + return second.get(); + } + return defaultValue; + } + + /** + * Port of legacy {@code HiveMetaStoreClientHelper.getByte}: a numeric delimiter value like {@code "9"} + * decodes to the byte char {@code (byte + 256) % 256}; a non-numeric value falls back to its first raw + * char; a null/empty value falls back to {@code defValue}. + */ + static String getByte(String altValue, String defValue) { + if (altValue != null && altValue.length() > 0) { + try { + return Character.toString((char) ((Byte.parseByte(altValue) + 256) % 256)); + } catch (NumberFormatException e) { + return altValue.substring(0, 1); + } + } + return defValue; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java index b6272ee2409c90..fc9344ea52b9b9 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java @@ -22,6 +22,7 @@ import java.util.Collections; import java.util.List; +import java.util.Locale; import java.util.Map; /** @@ -36,6 +37,10 @@ public class HiveTableHandle implements ConnectorTableHandle { private static final long serialVersionUID = 1L; + /** Metastore parameter key marking an ACID table insert-only (vs full-ACID). */ + private static final String TRANSACTIONAL_PROPERTIES = "transactional_properties"; + private static final String INSERT_ONLY = "insert_only"; + private final String dbName; private final String tableName; private final HiveTableType tableType; @@ -111,6 +116,41 @@ public Map getTableParameters() { return tableParameters; } + /** + * Whether the metastore parameters mark this table transactional (ACID), replicating Hive's + * {@code AcidUtils.isTransactionalTable} (case-insensitive {@code "true"} under the + * {@code transactional} key, with the upper-cased key as a fallback). + */ + public boolean isTransactional() { + return isTransactionalTable(tableParameters); + } + + /** + * Whether this table is full-ACID (transactional and not insert-only), i.e. its reads must + * apply row-level deletes from delete-delta directories. Mirrors Hive's + * {@code AcidUtils.isFullAcidTable}: transactional and {@code transactional_properties} is not + * {@code insert_only}. + */ + public boolean isFullAcid() { + if (!isTransactional()) { + return false; + } + String props = tableParameters.get(TRANSACTIONAL_PROPERTIES); + return !INSERT_ONLY.equalsIgnoreCase(props); + } + + private static boolean isTransactionalTable(Map tableParameters) { + if (tableParameters == null) { + return false; + } + String value = tableParameters.get(HiveConnectorProperties.CREATE_TRANSACTIONAL); + if (value == null) { + value = tableParameters.get( + HiveConnectorProperties.CREATE_TRANSACTIONAL.toUpperCase(Locale.ROOT)); + } + return "true".equalsIgnoreCase(value); + } + public List getPrunedPartitions() { return prunedPartitions; } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteContext.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteContext.java new file mode 100644 index 00000000000000..9d0f9e31070f4a --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteContext.java @@ -0,0 +1,89 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.thrift.TFileType; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +/** + * Immutable op context for a single hive write, threaded into + * {@link HiveConnectorTransaction#beginWrite}. The connector-internal equivalent of the fe-core + * {@code HiveInsertCommandContext} (which the connector cannot import): it carries the write operation, + * the overwrite mode, the static partition spec (for {@code INSERT OVERWRITE ... PARTITION}), the query + * id (replaces {@code ConnectContext}, per design D4), and the BE-facing file type / staging write path + * (which decides an in-place object-store write from a staged HDFS/local write). + * + *

Peer of {@code IcebergWriteContext}; drops iceberg's {@code branchName}/{@code readSnapshotId} and + * adds hive's {@code queryId}/{@code fileType}/{@code writePath}. INC-4's {@code HiveWritePlanProvider} + * constructs it in {@code buildWriteContext}; INC-3 only consumes it via {@link + * HiveConnectorTransaction#beginWrite}.

+ */ +final class HiveWriteContext { + + private final WriteOperation writeOperation; + private final boolean overwrite; + private final Map staticPartitionValues; + private final String queryId; + private final TFileType fileType; + private final String writePath; + + HiveWriteContext(WriteOperation writeOperation, boolean overwrite, + Map staticPartitionValues, String queryId, + TFileType fileType, String writePath) { + this.writeOperation = writeOperation; + this.overwrite = overwrite; + this.staticPartitionValues = staticPartitionValues == null + ? Collections.emptyMap() : new HashMap<>(staticPartitionValues); + this.queryId = queryId; + this.fileType = fileType; + this.writePath = writePath; + } + + WriteOperation getWriteOperation() { + return writeOperation; + } + + boolean isOverwrite() { + return overwrite; + } + + Map getStaticPartitionValues() { + return staticPartitionValues; + } + + /** An {@code INSERT OVERWRITE ... PARTITION(col=val, ...)} (a non-empty static partition spec). */ + boolean isStaticPartitionOverwrite() { + return overwrite && !staticPartitionValues.isEmpty(); + } + + String getQueryId() { + return queryId; + } + + TFileType getFileType() { + return fileType; + } + + String getWritePath() { + return writePath; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWritePlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWritePlanProvider.java new file mode 100644 index 00000000000000..9a603bdd0ebbfb --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWritePlanProvider.java @@ -0,0 +1,362 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.connector.spi.ConnectorBrokerAddress; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.properties.StorageProperties; +import org.apache.doris.thrift.TDataSink; +import org.apache.doris.thrift.TDataSinkType; +import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.TFileType; +import org.apache.doris.thrift.THiveBucket; +import org.apache.doris.thrift.THiveColumn; +import org.apache.doris.thrift.THiveColumnType; +import org.apache.doris.thrift.THiveLocationParams; +import org.apache.doris.thrift.THivePartition; +import org.apache.doris.thrift.THiveTableSink; +import org.apache.doris.thrift.TNetworkAddress; + +import com.google.common.base.Strings; +import org.apache.hadoop.fs.Path; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.EnumSet; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; + +/** + * Write plan provider for hive non-ACID INSERT / INSERT OVERWRITE. + * + *

Builds the opaque {@link THiveTableSink} for a bound write and binds the write to the current + * {@link HiveConnectorTransaction}: it loads the table under the catalog auth context, opens the write via + * {@link HiveConnectorTransaction#beginWrite} (which re-loads the table and applies the begin-guards, incl. + * the transactional-table reject), then assembles the sink Thrift from the loaded table. The Thrift is + * byte-identical to the legacy fe-core {@code planner.HiveTableSink.bindDataSink} (zero BE change), so the + * BE writer is unaffected by the migration.

+ * + *

Scope. INSERT / OVERWRITE only ({@code THiveTableSink}); an overwriting INSERT is promoted to the + * OVERWRITE operation in {@link #buildWriteContext}. The write distribution capability + * {@link #requiresPartitionHashWrite()} (hash-by-partition, no local sort) matches the legacy + * {@code PhysicalHiveTableSink}.

+ * + *

Gate-closed / dormant. Hive is not in {@code SPI_READY_TYPES} until the P7.4/P7.5 cutover, so + * nothing routes hive writes through this provider yet; {@link #planWrite} requires the executor-bound + * connector transaction and fails loud if absent.

+ */ +public class HiveWritePlanProvider implements ConnectorWritePlanProvider { + + // Staging-directory keys (connector-local copies of HMSExternalCatalog.HIVE_STAGING_DIR / + // DEFAULT_STAGING_BASE_DIR — connectors must not import fe-core). + private static final String HIVE_STAGING_DIR = "hive.staging_dir"; + private static final String DEFAULT_STAGING_BASE_DIR = "/tmp/.doris_staging"; + + private final HmsClient hmsClient; + private final Map properties; + private final ConnectorContext context; + + public HiveWritePlanProvider(HmsClient hmsClient, Map properties, ConnectorContext context) { + this.hmsClient = hmsClient; + this.properties = properties; + this.context = context; + } + + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + HiveTableHandle tableHandle = (HiveTableHandle) handle.getTableHandle(); + HiveConnectorTransaction transaction = currentTransaction(session); + + // Load the table under the catalog auth context; it drives both the location resolution + // (buildWriteContext) and the sink assembly (buildSink). beginWrite re-loads it for its own + // begin-guard — the double-load is accepted (mirrors iceberg), keeping the flow simple. + HmsTableInfo table = loadTable(tableHandle); + HiveWriteContext writeContext = buildWriteContext(session, tableHandle, table, handle); + transaction.beginWrite(session, tableHandle.getDbName(), tableHandle.getTableName(), writeContext); + + THiveTableSink sink = buildSink(session, tableHandle, table, handle, writeContext); + TDataSink dataSink = new TDataSink(TDataSinkType.HIVE_TABLE_SINK); + dataSink.setHiveTableSink(sink); + return new ConnectorSinkPlan(dataSink); + } + + @Override + public void appendExplainInfo(StringBuilder output, String prefix, + ConnectorSession session, ConnectorWriteHandle handle) { + // Surface the connector-specific write detail the generic plugin-driven sink line cannot (mirrors the + // legacy HiveTableSink.getExplainString "HIVE TABLE SINK" block). + HiveTableHandle tableHandle = (HiveTableHandle) handle.getTableHandle(); + output.append(prefix).append(" HIVE TABLE: ") + .append(tableHandle.getDbName()).append(".").append(tableHandle.getTableName()).append("\n"); + } + + @Override + public Set supportedOperations() { + return EnumSet.of(WriteOperation.INSERT, WriteOperation.OVERWRITE); + } + + @Override + public boolean requiresParallelWrite() { + return true; + } + + @Override + public boolean requiresFullSchemaWriteOrder() { + return true; + } + + @Override + public boolean requiresPartitionHashWrite() { + return true; + } + + /** + * Builds the op-context threaded into {@link HiveConnectorTransaction#beginWrite}. Port of the legacy + * {@code HiveTableSink.bindDataSink} location block: resolves the BE file type from the raw table + * location and the write path (an in-place normalized path for an object store, or a staging temp path + * for HDFS/local/broker). An overwriting INSERT is promoted to the OVERWRITE operation. Package-private so + * the promotion is directly assertable. + */ + HiveWriteContext buildWriteContext(ConnectorSession session, HiveTableHandle tableHandle, + HmsTableInfo table, ConnectorWriteHandle handle) { + String rawLocation = table.getLocation(); + TFileType fileType = TFileType.valueOf(context.getBackendFileType(rawLocation, Collections.emptyMap())); + String writePath = fileType == TFileType.FILE_S3 + ? context.normalizeStorageUri(rawLocation, Collections.emptyMap()) + : createTempPath(session, rawLocation); + WriteOperation op = handle.getWriteOperation(); + if (op == WriteOperation.INSERT && handle.isOverwrite()) { + op = WriteOperation.OVERWRITE; + } + return new HiveWriteContext(op, handle.isOverwrite(), handle.getWriteContext(), + session.getQueryId(), fileType, writePath); + } + + // Re-impl of legacy HiveTableSink.createTempPath + LocationPath.getTempWritePath (pure hadoop Path + UUID): + // ///. A relative stagingBaseDir resolves under rawLoc; an absolute one + // keeps rawLoc's scheme/authority. + private String createTempPath(ConnectorSession session, String rawLocation) { + String user = session.getUser(); + String stagingBaseDir = properties.getOrDefault(HIVE_STAGING_DIR, DEFAULT_STAGING_BASE_DIR); + Path prefix = new Path(stagingBaseDir, user); + Path temp = new Path(new Path(rawLocation, prefix), UUID.randomUUID().toString().replace("-", "")); + return temp.toString(); + } + + private THiveTableSink buildSink(ConnectorSession session, HiveTableHandle tableHandle, + HmsTableInfo table, ConnectorWriteHandle handle, HiveWriteContext writeContext) { + THiveTableSink tSink = new THiveTableSink(); + tSink.setDbName(tableHandle.getDbName()); + tSink.setTableName(tableHandle.getTableName()); + + // Columns: data columns tagged REGULAR first, then partition keys tagged PARTITION_KEY (HMS order). + tSink.setColumns(buildColumns(table)); + + // Existing partitions (partitioned table only; empty otherwise). + tSink.setPartitions(buildExistingPartitions(table)); + + // Bucket info. + THiveBucket bucketInfo = new THiveBucket(); + bucketInfo.setBucketedBy(table.getBucketCols()); + bucketInfo.setBucketCount(table.getNumBuckets()); + tSink.setBucketInfo(bucketInfo); + + // File format (ports the LZO reject + supported-set validation) + compression. + TFileFormatType formatType = HiveSinkHelper.getTFileFormatType(table.getInputFormat()); + tSink.setFileFormat(formatType); + setCompressType(tSink, formatType, table, session); + + // SerDe delimiters. + tSink.setSerdeProperties(HiveSinkHelper.buildSerDeProperties(table)); + + // Location: an object-store write goes in-place (write == target == normalized, original == raw); an + // HDFS/local/broker write goes to a staging temp path (write == original == staging, target == raw). + THiveLocationParams locationParams = new THiveLocationParams(); + String rawLocation = table.getLocation(); + if (writeContext.getFileType() == TFileType.FILE_S3) { + locationParams.setWritePath(writeContext.getWritePath()); + locationParams.setOriginalWritePath(rawLocation); + locationParams.setTargetPath(writeContext.getWritePath()); + } else { + locationParams.setWritePath(writeContext.getWritePath()); + locationParams.setOriginalWritePath(writeContext.getWritePath()); + locationParams.setTargetPath(rawLocation); + } + locationParams.setFileType(writeContext.getFileType()); + tSink.setLocation(locationParams); + + // Broker addresses only for a broker backend (fails loud when empty, mirroring legacy). + if (writeContext.getFileType() == TFileType.FILE_BROKER) { + tSink.setBrokerAddresses(resolveBrokerAddresses()); + } + + // Hadoop config (BE-canonical static creds; hive has no vended overlay). + tSink.setHadoopConfig(buildHadoopConfig()); + + tSink.setOverwrite(handle.isOverwrite()); + return tSink; + } + + private static List buildColumns(HmsTableInfo table) { + List columns = new ArrayList<>(); + for (ConnectorColumn col : table.getColumns()) { + THiveColumn tHiveColumn = new THiveColumn(); + tHiveColumn.setName(col.getName()); + tHiveColumn.setColumnType(THiveColumnType.REGULAR); + columns.add(tHiveColumn); + } + for (ConnectorColumn col : table.getPartitionKeys()) { + THiveColumn tHiveColumn = new THiveColumn(); + tHiveColumn.setName(col.getName()); + tHiveColumn.setColumnType(THiveColumnType.PARTITION_KEY); + columns.add(tHiveColumn); + } + return columns; + } + + // Port of legacy HiveTableSink.setPartitionValues: for a partitioned table, list live partitions and + // convert each to a THivePartition (values + per-partition file format + in-place location). Live/uncached + // (matches the scan path; registered deviation DV-INC4-livepart). HmsClient calls self-authenticate. + private List buildExistingPartitions(HmsTableInfo table) { + List partitions = new ArrayList<>(); + if (table.getPartitionKeys().isEmpty()) { + return partitions; + } + List partitionNames = hmsClient.listPartitionNames( + table.getDbName(), table.getTableName(), -1); + List hmsPartitions = hmsClient.getPartitions( + table.getDbName(), table.getTableName(), partitionNames); + for (HmsPartitionInfo partition : hmsPartitions) { + THivePartition hivePartition = new THivePartition(); + hivePartition.setValues(partition.getValues()); + hivePartition.setFileFormat(HiveSinkHelper.getTFileFormatType(partition.getInputFormat())); + THiveLocationParams locationParams = new THiveLocationParams(); + String location = partition.getLocation(); + // The write and target path of an existing partition are the same (BE reads it in place). + locationParams.setWritePath(location); + locationParams.setTargetPath(location); + locationParams.setFileType(TFileType.valueOf( + context.getBackendFileType(location, Collections.emptyMap()))); + hivePartition.setLocation(locationParams); + partitions.add(hivePartition); + } + return partitions; + } + + // Port of legacy HiveTableSink.setCompressType: the compression codec is read from the table parameters by + // format (orc.compress / parquet.compression / text.compression, the text one falling back to the session + // default), then mapped to the BE compress type. + private void setCompressType(THiveTableSink tSink, TFileFormatType formatType, + HmsTableInfo table, ConnectorSession session) { + Map params = table.getParameters(); + String compressType; + switch (formatType) { + case FORMAT_ORC: + compressType = params.get("orc.compress"); + break; + case FORMAT_PARQUET: + compressType = params.get("parquet.compression"); + break; + case FORMAT_CSV_PLAIN: + case FORMAT_TEXT: + compressType = params.get("text.compression"); + if (Strings.isNullOrEmpty(compressType)) { + compressType = resolveTextCompressionDefault(session); + } + break; + default: + compressType = "plain"; + break; + } + tSink.setCompressionType(HiveSinkHelper.getTFileCompressType(compressType)); + } + + // Re-impl of legacy SessionVariable.hiveTextCompression() (the "uncompressed" alias maps to "plain"); the + // value rides on the session properties threaded from the engine. + private static String resolveTextCompressionDefault(ConnectorSession session) { + String textCompression = session.getSessionProperties() + .get(HiveConnectorProperties.SESSION_HIVE_TEXT_COMPRESSION); + if (HiveConnectorProperties.TEXT_COMPRESSION_UNCOMPRESSED.equals(textCompression)) { + return HiveConnectorProperties.TEXT_COMPRESSION_PLAIN; + } + return textCompression; + } + + // Mirror iceberg buildHadoopConfig: BE-canonical static catalog creds from the typed fe-filesystem + // StorageProperties (AWS_* for object stores, dfs/hadoop for HDFS). Hive has no vended overlay. + private Map buildHadoopConfig() { + Map merged = new HashMap<>(); + if (context != null) { + for (StorageProperties sp : context.getStorageProperties()) { + sp.toBackendProperties().ifPresent(b -> merged.putAll(b.toMap())); + } + } + return merged; + } + + // Resolve the broker backend addresses through the neutral SPI; fail loud when none is resolved (the same + // message legacy BaseExternalTableDataSink.getBrokerAddresses threw), so a broker write never ships BE an + // empty broker list. + private List resolveBrokerAddresses() { + List addresses = context.getBrokerAddresses(); + if (addresses.isEmpty()) { + throw new DorisConnectorException("No alive broker."); + } + List result = new ArrayList<>(addresses.size()); + for (ConnectorBrokerAddress address : addresses) { + result.add(new TNetworkAddress(address.getHost(), address.getPort())); + } + return result; + } + + private HmsTableInfo loadTable(HiveTableHandle tableHandle) { + try { + return context.executeAuthenticated( + () -> hmsClient.getTable(tableHandle.getDbName(), tableHandle.getTableName())); + } catch (Exception e) { + throw new DorisConnectorException("Failed to load hive table " + + tableHandle.getDbName() + "." + tableHandle.getTableName() + ": " + e.getMessage(), e); + } + } + + private HiveConnectorTransaction currentTransaction(ConnectorSession session) { + Optional transaction = session.getCurrentTransaction(); + if (!transaction.isPresent()) { + throw new DorisConnectorException( + "Hive write requires an active connector transaction bound to the session; none is " + + "present. The executor must open it via beginTransaction and bind it to the " + + "session (wired at the hive cutover)."); + } + return (HiveConnectorTransaction) transaction.get(); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java new file mode 100644 index 00000000000000..b3f7ea70be0bba --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java @@ -0,0 +1,268 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.thrift.THivePartitionUpdate; + +import org.apache.hadoop.fs.Path; + +import java.net.URI; +import java.net.URISyntaxException; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * Pure, side-effect-free helpers for the Hive connector write path, ported verbatim from the legacy + * fe-core {@code HMSTransaction}. Deliberately fe-core-free: the only dependencies are the Hadoop + * {@code Path}, {@code java.net.URI}, and the shared thrift {@code THivePartitionUpdate}. Consumed + * by the (in-progress) connector write transaction and its staging-cleanup logic. + */ +final class HiveWriteUtils { + private HiveWriteUtils() { + } + + /** + * Merges partition updates that target the same partition name. For collisions the file sizes + * and row counts are summed and the pending MPU-upload and file-name lists are concatenated onto + * the first-seen update; distinct names are kept as-is. Mirrors the legacy + * {@code HMSTransaction.mergePartitions}. + * + *

The first-seen update's {@code fileNames} / {@code s3MpuPendingUploads} lists are mutated in + * place, so they must be mutable (thrift deserialization produces mutable {@code ArrayList}s). + */ + static List mergePartitions(List hivePartitionUpdates) { + Map merged = new HashMap<>(); + for (THivePartitionUpdate pu : hivePartitionUpdates) { + if (merged.containsKey(pu.getName())) { + THivePartitionUpdate old = merged.get(pu.getName()); + old.setFileSize(old.getFileSize() + pu.getFileSize()); + old.setRowCount(old.getRowCount() + pu.getRowCount()); + if (old.getS3MpuPendingUploads() != null && pu.getS3MpuPendingUploads() != null) { + old.getS3MpuPendingUploads().addAll(pu.getS3MpuPendingUploads()); + } + old.getFileNames().addAll(pu.getFileNames()); + } else { + merged.put(pu.getName(), pu); + } + } + return new ArrayList<>(merged.values()); + } + + /** + * Returns true when {@code child} is a strict subdirectory of {@code parent} on the same file + * system (matching scheme + authority, path-prefix under a {@code /} boundary). + */ + static boolean isSubDirectory(String parent, String child) { + if (parent == null || child == null) { + return false; + } + Path parentPath = new Path(parent); + Path childPath = new Path(child); + URI parentUri = parentPath.toUri(); + URI childUri = childPath.toUri(); + if (!sameFileSystem(parentUri, childUri)) { + return false; + } + String parentPathValue = normalizePath(parentUri.getPath()); + String childPathValue = normalizePath(childUri.getPath()); + if (parentPathValue.isEmpty() || childPathValue.isEmpty()) { + return false; + } + return !parentPathValue.equals(childPathValue) + && childPathValue.startsWith(parentPathValue + "/"); + } + + /** + * Returns the first-level child path of {@code parent} that contains {@code child}, + * or null if {@code child} is not a subdirectory of {@code parent}. + * Example: parent=/warehouse/table, child=/warehouse/table/.doris_staging/user/uuid + * returns /warehouse/table/.doris_staging. + */ + static String getImmediateChildPath(String parent, String child) { + if (!isSubDirectory(parent, child)) { + return null; + } + Path parentPath = new Path(parent); + URI parentUri = parentPath.toUri(); + URI childUri = new Path(child).toUri(); + String parentPathValue = normalizePath(parentUri.getPath()); + String childPathValue = normalizePath(childUri.getPath()); + String relative = childPathValue.substring(parentPathValue.length() + 1); + int slashIndex = relative.indexOf("/"); + String firstComponent = slashIndex == -1 ? relative : relative.substring(0, slashIndex); + return new Path(parentPath, firstComponent).toString(); + } + + /** + * Returns true when {@code left} and {@code right} resolve to the same normalized path on the + * same file system. Null-safe: two nulls are equal, one null is not. + */ + static boolean pathsEqual(String left, String right) { + if (left == null || right == null) { + return left == null && right == null; + } + URI leftUri = new Path(left).toUri(); + URI rightUri = new Path(right).toUri(); + if (!sameFileSystem(leftUri, rightUri)) { + return false; + } + return normalizePath(leftUri.getPath()).equals(normalizePath(rightUri.getPath())); + } + + /** + * Compares two URI strings for equality with special handling for the "s3" scheme. Byte-faithful port + * of fe-core {@code PathUtils.equalsIgnoreSchemeIfOneIsS3} (NOT the same as {@link #pathsEqual}, which + * treats a scheme mismatch as a different file system): in the BE all object stores are unified under + * the "s3" URI scheme, so a path written with a different underlying scheme (e.g. "oss://") is the SAME + * physical location as the "s3://" form. Used by the committer's {@code needRename} decision so an + * in-place object-store write is not needlessly renamed. + * + *

Rules: same scheme -> case-insensitive full-string compare; different schemes but one is "s3" + * -> compare only authority + path (trailing slashes stripped); otherwise not equal. + */ + static boolean equalsIgnoreSchemeIfOneIsS3(String p1, String p2) { + if (p1 == null || p2 == null) { + return p1 == null && p2 == null; + } + try { + URI uri1 = new URI(p1); + URI uri2 = new URI(p2); + + String scheme1 = uri1.getScheme(); + String scheme2 = uri2.getScheme(); + + // If schemes are equal, compare the full URI strings ignoring case + if (scheme1 != null && scheme1.equalsIgnoreCase(scheme2)) { + return p1.equalsIgnoreCase(p2); + } + + // If schemes differ but one is "s3", compare only authority and path ignoring scheme + if ("s3".equalsIgnoreCase(scheme1) || "s3".equalsIgnoreCase(scheme2)) { + String auth1 = stripTrailingSlashes(uri1.getAuthority()); + String auth2 = stripTrailingSlashes(uri2.getAuthority()); + String path1 = stripTrailingSlashes(uri1.getPath()); + String path2 = stripTrailingSlashes(uri2.getPath()); + return Objects.equals(auth1, auth2) && Objects.equals(path1, path2); + } + + // Otherwise, URIs are not equal + return false; + } catch (URISyntaxException e) { + // If URI parsing fails, fallback to simple case-insensitive string comparison + return p1.equalsIgnoreCase(p2); + } + } + + /** + * Splits a Hive partition name ("c1=a/c2=b/c3=c") into its ordered values ("a", "b", "c"), URL-decoding + * each value with {@link #unescapePathName}. Byte-faithful port of fe-core {@code HiveUtil.toPartitionValues} + * (which delegated to Hive's {@code FileUtils.unescapePathName}). Ported inline so the connector needs no + * hive-common dependency. Used to key the write transaction's per-partition action map and to build the + * partition-value argument passed to the metastore write primitives. + */ + static List toPartitionValues(String partitionName) { + List result = new ArrayList<>(); + int start = 0; + while (true) { + while (start < partitionName.length() && partitionName.charAt(start) != '=') { + start++; + } + start++; + int end = start; + while (end < partitionName.length() && partitionName.charAt(end) != '/') { + end++; + } + if (start > partitionName.length()) { + break; + } + result.add(unescapePathName(partitionName.substring(start, end))); + start = end + 1; + } + return result; + } + + /** + * URL-decodes a Hive-escaped path component (e.g. "a%2Fb" -> "a/b"). Byte-faithful port of Hive's + * {@code org.apache.hadoop.hive.common.FileUtils.unescapePathName}, inlined to avoid a hive-common + * dependency. + */ + private static String unescapePathName(String path) { + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < path.length(); i++) { + char c = path.charAt(i); + if (c == '%' && i + 2 < path.length()) { + int code = -1; + try { + code = Integer.parseInt(path.substring(i + 1, i + 3), 16); + } catch (Exception e) { + code = -1; + } + if (code >= 0) { + sb.append((char) code); + i += 2; + continue; + } + } + sb.append(c); + } + return sb.toString(); + } + + /** Strip trailing slashes for {@link #equalsIgnoreSchemeIfOneIsS3} (mirrors PathUtils.normalize). */ + private static String stripTrailingSlashes(String s) { + if (s == null) { + return ""; + } + String trimmed = s.replaceAll("/+$", ""); + return trimmed.isEmpty() ? "" : trimmed; + } + + private static boolean sameFileSystem(URI left, URI right) { + String leftScheme = normalizeUriPart(left.getScheme()); + String rightScheme = normalizeUriPart(right.getScheme()); + if (!leftScheme.isEmpty() && !rightScheme.isEmpty() + && !leftScheme.equalsIgnoreCase(rightScheme)) { + return false; + } + String leftAuthority = normalizeUriPart(left.getAuthority()); + String rightAuthority = normalizeUriPart(right.getAuthority()); + if (!leftAuthority.isEmpty() && !rightAuthority.isEmpty() + && !leftAuthority.equalsIgnoreCase(rightAuthority)) { + return false; + } + return true; + } + + private static String normalizeUriPart(String value) { + return value == null ? "" : value; + } + + private static String normalizePath(String path) { + if (path == null || path.isEmpty()) { + return ""; + } + int end = path.length(); + while (end > 1 && path.charAt(end - 1) == '/') { + end--; + } + return path.substring(0, end); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/NameMapping.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/NameMapping.java new file mode 100644 index 00000000000000..bc715a6207d848 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/NameMapping.java @@ -0,0 +1,104 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import java.util.Objects; + +/** + * Mapping between the local (Doris) and remote (metastore) database/table names of a Hive + * write/read transaction. This is a plugin-local, fe-core-free copy of {@code + * org.apache.doris.datasource.NameMapping} (JDK-only, no lombok/guava), used as the identity key of + * the connector transaction's per-table / per-partition action maps: the remote names drive the + * actual metastore calls, and the local names surface in diagnostics. + */ +public final class NameMapping { + private final long ctlId; + private final String localDbName; + private final String localTblName; + private final String remoteDbName; + private final String remoteTblName; + + public NameMapping(long ctlId, String localDbName, String localTblName, + String remoteDbName, String remoteTblName) { + this.ctlId = ctlId; + this.localDbName = localDbName; + this.localTblName = localTblName; + this.remoteDbName = remoteDbName; + this.remoteTblName = remoteTblName; + } + + public long getCtlId() { + return ctlId; + } + + public String getLocalDbName() { + return localDbName; + } + + public String getLocalTblName() { + return localTblName; + } + + public String getRemoteDbName() { + return remoteDbName; + } + + public String getRemoteTblName() { + return remoteTblName; + } + + public String getFullLocalName() { + return String.format("%s.%s", localDbName, localTblName); + } + + public String getFullRemoteName() { + return String.format("%s.%s", remoteDbName, remoteTblName); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof NameMapping)) { + return false; + } + NameMapping that = (NameMapping) o; + return ctlId == that.ctlId + && localDbName.equals(that.localDbName) + && localTblName.equals(that.localTblName) + && remoteDbName.equals(that.remoteDbName) + && remoteTblName.equals(that.remoteTblName); + } + + @Override + public int hashCode() { + return Objects.hash(ctlId, localDbName, localTblName, remoteDbName, remoteTblName); + } + + @Override + public String toString() { + return "NameMapping{" + + "ctlId=" + ctlId + + ", localDbName='" + localDbName + '\'' + + ", localTblName='" + localTblName + '\'' + + ", remoteDbName='" + remoteDbName + '\'' + + ", remoteTblName='" + remoteTblName + '\'' + + '}'; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java new file mode 100644 index 00000000000000..53e8a89c154b26 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java @@ -0,0 +1,226 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.hms.HmsAcidConstants; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.hive.common.ValidReadTxnList; +import org.apache.hadoop.hive.common.ValidReaderWriteIdList; +import org.apache.hadoop.hive.common.ValidTxnList; +import org.apache.hadoop.hive.common.ValidWriteIdList; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.file.Files; +import java.util.BitSet; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Tests the pure ACID directory descent {@link HiveAcidUtil#getAcidState} against a real Hadoop + * {@link FileSystem} over a local temp tree. + * + *

WHY: for a transactional Hive table the reader must reconstruct the correct snapshot from the + * base/delta/delete-delta directory layout — pick the best valid base, layer on the deltas whose + * write-id range is still valid, drop obsolete/out-of-snapshot directories, and hand the BE the + * delete-delta file names so it can subtract deleted rows. A wrong base or a dropped delta silently + * returns stale or over-/under-deleted data. These tests pin that selection algorithm.

+ */ +public class HiveAcidUtilTest { + + @TempDir + java.nio.file.Path tempDir; + + private FileSystem localFs() throws IOException { + return FileSystem.getLocal(new Configuration()); + } + + /** Creates {@code //} with 1 byte of content. */ + private void createBucketFile(String dir, String fileName) throws IOException { + java.nio.file.Path d = tempDir.resolve(dir); + Files.createDirectories(d); + Files.write(d.resolve(fileName), new byte[] {1}); + } + + /** Snapshot with all visibility txns valid and write-ids valid up to {@code highWatermark}. */ + private Map snapshot(long highWatermark) { + ValidTxnList validTxnList = + new ValidReadTxnList(new long[0], new BitSet(), Long.MAX_VALUE, Long.MAX_VALUE); + ValidWriteIdList validWriteIdList = + new ValidReaderWriteIdList("db.tbl", new long[0], new BitSet(), highWatermark); + Map ids = new HashMap<>(); + ids.put(HmsAcidConstants.VALID_TXNS_KEY, validTxnList.writeToString()); + ids.put(HmsAcidConstants.VALID_WRITEIDS_KEY, validWriteIdList.writeToString()); + return ids; + } + + @Test + public void testBestBaseWithDeltaAndDeleteDelta() throws IOException { + // Best base = base_5 (base_2 superseded); delta_6 layered on; delete_delta_6 survives via the + // split-update pairing with delta_6; delta_9 is out of the write-id=6 snapshot; the staging dir + // and the _flush_length side file are ignored. + createBucketFile("base_0000005", "bucket_00000"); + createBucketFile("base_0000002", "bucket_00000"); + createBucketFile("delta_0000006_0000006", "bucket_00000"); + createBucketFile("delta_0000006_0000006", "bucket_00000_flush_length"); + createBucketFile("delete_delta_0000006_0000006", "bucket_00000"); + createBucketFile("delta_0000009_0000009", "bucket_00000"); + createBucketFile(".hive-staging_hive_2026-01-01", "junk"); + + HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( + localFs(), tempDir.toString(), snapshot(6L), true); + + List dataFiles = state.getDataFiles(); + Assertions.assertEquals(2, dataFiles.size(), + "surviving data files must be exactly base_5 + delta_6 bucket files"); + boolean hasBase5 = false; + boolean hasDelta6 = false; + for (FileStatus f : dataFiles) { + String p = f.getPath().toString(); + Assertions.assertFalse(p.contains("base_0000002"), "superseded base must be dropped: " + p); + Assertions.assertFalse(p.contains("delta_0000009"), "out-of-snapshot delta dropped: " + p); + Assertions.assertFalse(p.endsWith("_flush_length"), "side file excluded: " + p); + hasBase5 |= p.contains("/base_0000005/bucket_00000"); + hasDelta6 |= p.contains("/delta_0000006_0000006/bucket_00000"); + } + Assertions.assertTrue(hasBase5, "best base_5 bucket file must survive"); + Assertions.assertTrue(hasDelta6, "delta_6 bucket file must survive"); + + List deletes = state.getDeleteDeltas(); + Assertions.assertEquals(1, deletes.size(), "the paired delete_delta_6 must survive"); + HiveAcidUtil.DeleteDelta d = deletes.get(0); + Assertions.assertTrue(d.getDirectoryLocation().endsWith("/delete_delta_0000006_0000006"), + "delete-delta dir: " + d.getDirectoryLocation()); + Assertions.assertEquals(List.of("bucket_00000"), d.getFileNames(), + "delete-delta file names must be captured (BE under-deletes without them)"); + } + + @Test + public void testInsertOnlyRejectsDeleteDelta() throws IOException { + // An insert-only ACID table must never carry delete deltas; a stray one is a corruption signal. + createBucketFile("base_0000005", "000000_0"); + createBucketFile("delta_0000006_0000006", "000000_0"); + createBucketFile("delete_delta_0000006_0000006", "000000_0"); + + RuntimeException ex = Assertions.assertThrows(RuntimeException.class, + () -> HiveAcidUtil.getAcidState(localFs(), tempDir.toString(), snapshot(6L), false)); + Assertions.assertTrue(ex.getMessage().contains("delete_delta"), + "insert-only table with a delete delta must fail loud: " + ex.getMessage()); + } + + @Test + public void testInsertOnlyAcceptsAllFileNames() throws IOException { + // Insert-only uses the accept-all filter: files that are not bucket_* still count as data. + createBucketFile("base_0000005", "000000_0"); + createBucketFile("delta_0000006_0000006", "000001_0"); + + HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( + localFs(), tempDir.toString(), snapshot(6L), false); + Assertions.assertEquals(2, state.getDataFiles().size(), + "insert-only accepts non-bucket_ data file names"); + Assertions.assertTrue(state.getDeleteDeltas().isEmpty()); + } + + @Test + public void testMissingValidWriteIdsThrows() throws IOException { + createBucketFile("base_0000005", "bucket_00000"); + Map ids = snapshot(6L); + ids.remove(HmsAcidConstants.VALID_WRITEIDS_KEY); + + RuntimeException ex = Assertions.assertThrows(RuntimeException.class, + () -> HiveAcidUtil.getAcidState(localFs(), tempDir.toString(), ids, true)); + Assertions.assertTrue(ex.getMessage().contains("ValidWriteIdList"), ex.getMessage()); + } + + @Test + public void testOriginalFilesWithoutBaseThrows() throws IOException { + // A bare file directly under the partition (no base_) means an unconverted non-ACID table. + Files.write(tempDir.resolve("000000_0"), new byte[] {1}); + + Assertions.assertThrows(UnsupportedOperationException.class, + () -> HiveAcidUtil.getAcidState(localFs(), tempDir.toString(), snapshot(6L), true)); + } + + @Test + public void testBaseWithUncommittedVisibilityTxnIsSkipped() throws IOException { + // base_5 was produced by visibility txn 100, which is NOT committed in this snapshot -> it must + // be skipped, falling back to the older committed base_3. This pins the visibility-txn filter. + createBucketFile("base_0000005_v0000100", "bucket_00000"); + createBucketFile("base_0000003", "bucket_00000"); + + // Txn high-watermark 50: visibility txn 100 is not yet valid; base_3's visibility (0) is valid. + ValidTxnList txns = new ValidReadTxnList(new long[0], new BitSet(), 50L, Long.MAX_VALUE); + ValidWriteIdList writeIds = new ValidReaderWriteIdList("db.tbl", new long[0], new BitSet(), 5L); + Map ids = new HashMap<>(); + ids.put(HmsAcidConstants.VALID_TXNS_KEY, txns.writeToString()); + ids.put(HmsAcidConstants.VALID_WRITEIDS_KEY, writeIds.writeToString()); + + HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( + localFs(), tempDir.toString(), ids, true); + Assertions.assertEquals(1, state.getDataFiles().size()); + Assertions.assertTrue(state.getDataFiles().get(0).getPath().toString().contains("/base_0000003/"), + "a base whose visibility txn is uncommitted must be skipped for the committed base"); + } + + @Test + public void testHighestBaseInvalidFallsBackToLowerValidBaseAndDropsObsoleteDelta() throws IOException { + // base_8 is beyond the write-id watermark (invalid) and must be rejected despite its higher + // write id, falling back to base_4; delta_2 predates base_4 so it is obsolete and dropped. This + // pins isValidBase discrimination + the selection loop's "delta below the base" rejection. + createBucketFile("base_0000008", "bucket_00000"); + createBucketFile("base_0000004", "bucket_00000"); + createBucketFile("delta_0000002_0000002", "bucket_00000"); + + HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( + localFs(), tempDir.toString(), snapshot(5L), true); + + Assertions.assertEquals(1, state.getDataFiles().size(), + "only the best VALID base (base_4) survives; the higher but invalid base_8 is rejected"); + String p = state.getDataFiles().get(0).getPath().toString(); + Assertions.assertTrue(p.contains("/base_0000004/"), p); + Assertions.assertFalse(p.contains("base_0000008"), "base above the write-id watermark is invalid"); + } + + @Test + public void testNoValidBaseThrowsNotEnoughHistory() throws IOException { + // Only base_8 exists but it is beyond the write-id watermark -> no usable base and no original + // files -> the reader cannot reconstruct the snapshot and must fail loud. + createBucketFile("base_0000008", "bucket_00000"); + + IOException ex = Assertions.assertThrows(IOException.class, + () -> HiveAcidUtil.getAcidState(localFs(), tempDir.toString(), snapshot(5L), true)); + Assertions.assertTrue(ex.getMessage().contains("Not enough history"), ex.getMessage()); + } + + @Test + public void testMissingValidTxnListThrows() throws IOException { + createBucketFile("base_0000005", "bucket_00000"); + Map ids = snapshot(6L); + ids.remove(HmsAcidConstants.VALID_TXNS_KEY); + + RuntimeException ex = Assertions.assertThrows(RuntimeException.class, + () -> HiveAcidUtil.getAcidState(localFs(), tempDir.toString(), ids, true)); + Assertions.assertTrue(ex.getMessage().contains("ValidTxnList"), ex.getMessage()); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java new file mode 100644 index 00000000000000..817d548d80c99b --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java @@ -0,0 +1,658 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsPartitionStatistics; +import org.apache.doris.connector.hms.HmsPartitionWithStatistics; +import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.filesystem.DorisInputFile; +import org.apache.doris.filesystem.DorisOutputFile; +import org.apache.doris.filesystem.FileIterator; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.Location; +import org.apache.doris.filesystem.UploadPartResult; +import org.apache.doris.filesystem.properties.StorageProperties; +import org.apache.doris.filesystem.spi.ObjFileSystem; +import org.apache.doris.filesystem.spi.ObjStorage; +import org.apache.doris.filesystem.spi.RemoteObject; +import org.apache.doris.filesystem.spi.RemoteObjects; +import org.apache.doris.filesystem.spi.RequestBody; +import org.apache.doris.thrift.TFileType; +import org.apache.doris.thrift.THiveLocationParams; +import org.apache.doris.thrift.THivePartitionUpdate; +import org.apache.doris.thrift.TS3MPUPendingUpload; +import org.apache.doris.thrift.TUpdateMode; + +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.thrift.TException; +import org.apache.thrift.TSerializer; +import org.apache.thrift.protocol.TBinaryProtocol; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Function; +import java.util.stream.Collectors; + +/** + * Unit tests for {@link HiveConnectorTransaction}, exercised offline against a recording {@link HmsClient} + * fake (no metastore, no Mockito — the connector test convention). These cover the parts of the ported + * {@code HMSTransaction} whose behaviour a compile cannot verify and where a silent regression corrupts + * data or drops writes: + * + *
    + *
  • Classification / NEW→APPEND downgrade — the {@code finishInsertTable} state machine and + * the {@code partitionExists} probe that saves a partitioned {@code INSERT} from failing when Doris' + * cache missed a partition that already exists in HMS (a wrong branch here either double-adds a + * partition or loses the rows).
    • + *
  • transactional write reject (DEC-2) — writing to ANY transactional table (full-ACID AND + * insert-only) must be refused; the transactional flag is derived plugin-side from the raw HMS + * parameters (an insert-only ACID table would otherwise get plain files — corrupt/invisible data).
    • + *
  • {@code addCommitData}/{@code getUpdateCnt} — the affected-row count reported back to the + * engine is the sum of the BE fragment row counts (D3); an off-by-one here misreports load results.
    • + *
  • SPI identity — {@code profileLabel()} must stay {@code "HMS"} (D2: maps to the existing + * fe-core {@code TransactionType.HMS}; any other value silently degrades to UNKNOWN).
    • + *
+ * + *

The commit-time object-store work is driven through a fake {@code ObjFileSystem}/{@code ObjStorage} + * injected via the {@code resolveObjectStoreFileSystem} seam: multipart-upload complete on commit + * and abort on rollback (D6/D9), the idempotency of a repeated rollback, and the single + * batched {@code addPartitions} call (GAP-7 — the 20-at-a-time batching lives in the client, so the + * committer adds the whole list once). The staging-directory rename walk and the full multi-partition + * commit remain for the P7.4/P7.5 cutover integration gate (the live write path).

+ */ +public class HiveConnectorTransactionTest { + + private static final long CATALOG_ID = 0L; + private static final String DB = "db"; + private static final String TBL = "t"; + + private HiveConnectorTransaction newTxn(HmsClient client) { + return new HiveConnectorTransaction(42L, client, new FakeConnectorContext( + "test_catalog", CATALOG_ID, Collections.emptyMap())); + } + + private NameMapping nameMapping() { + return new NameMapping(CATALOG_ID, DB, TBL, DB, TBL); + } + + private HiveWriteContext ctx(boolean overwrite) { + // FILE_S3 keeps the classification path from needing a real staging filesystem (createAndAddPartition + // then uses the write path directly rather than a target path). + return new HiveWriteContext(overwrite ? WriteOperation.OVERWRITE : WriteOperation.INSERT, overwrite, + Collections.emptyMap(), "query-1", TFileType.FILE_S3, "s3://bucket/db/t/_staging"); + } + + private static ConnectorColumn col(String name, String type) { + return new ConnectorColumn(name, ConnectorType.of(type), "", true, null); + } + + private static HmsTableInfo table(boolean partitioned, Map params) { + return HmsTableInfo.builder() + .dbName(DB).tableName(TBL).tableType("MANAGED_TABLE") + .location("s3://bucket/db/t") + .inputFormat("org.apache.hadoop.mapred.TextInputFormat") + .outputFormat("org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat") + .serializationLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe") + .columns(Collections.singletonList(col("c1", "int"))) + .partitionKeys(partitioned + ? Collections.singletonList(col("dt", "string")) + : Collections.emptyList()) + .parameters(params == null ? Collections.emptyMap() : params) + .build(); + } + + private static THivePartitionUpdate pu(String name, TUpdateMode mode, String writePath, + List fileNames, long fileSize, long rowCount) { + THivePartitionUpdate u = new THivePartitionUpdate(); + u.setName(name); + u.setUpdateMode(mode); + THiveLocationParams loc = new THiveLocationParams(); + loc.setWritePath(writePath); + loc.setTargetPath(writePath); + u.setLocation(loc); + u.setFileNames(fileNames); + u.setFileSize(fileSize); + u.setRowCount(rowCount); + return u; + } + + private static byte[] serialize(THivePartitionUpdate u) throws TException { + return new TSerializer(new TBinaryProtocol.Factory()).serialize(u); + } + + private static THivePartitionUpdate puWithMpu(String name, TUpdateMode mode, String writePath, + String bucket, String key, String uploadId, Map etags) { + THivePartitionUpdate u = pu(name, mode, writePath, Collections.singletonList("f1"), 100, 4); + TS3MPUPendingUpload mpu = new TS3MPUPendingUpload(); + mpu.setBucket(bucket); + mpu.setKey(key); + mpu.setUploadId(uploadId); + mpu.setEtags(etags); + u.setS3MpuPendingUploads(new ArrayList<>(Collections.singletonList(mpu))); + return u; + } + + // Builds a transaction whose object-store FileSystem is the injected fake, bypassing the production + // ServiceLoader discovery (a cutover-time concern) — the seam the design carved out for exactly this. + private HiveConnectorTransaction newTxnWithFs(HmsClient client, FileSystem fakeFs) { + return new HiveConnectorTransaction(42L, client, + new FakeConnectorContext("test_catalog", CATALOG_ID, Collections.emptyMap())) { + @Override + protected FileSystem resolveObjectStoreFileSystem(StorageProperties objSp) { + return fakeFs; + } + }; + } + + // ─────────────────────────────── SPI identity ─────────────────────────────── + + @Test + public void testProfileLabelAndTransactionId() { + // D2: the profile label must resolve to the existing fe-core TransactionType.HMS; a drift to "HIVE" + // (or anything else) would silently map to UNKNOWN and mislabel the transaction in the engine. + HiveConnectorTransaction txn = newTxn(new RecordingHmsClient()); + Assertions.assertEquals("HMS", txn.profileLabel()); + Assertions.assertEquals(42L, txn.getTransactionId()); + } + + // ─────────────────────────────── addCommitData / getUpdateCnt ─────────────────────────────── + + @Test + public void testGetUpdateCntSumsFragmentRowCounts() throws TException { + // D3: the affected-row count is the SUM of the BE fragment row counts. Feed three serialized + // fragments and assert the running total — proves both the TBinaryProtocol round-trip in + // addCommitData and the accumulation (a wrong reducer here misreports load results to the client). + HiveConnectorTransaction txn = newTxn(new RecordingHmsClient()); + txn.addCommitData(serialize(pu("dt=a", TUpdateMode.NEW, "s3://b/a", Arrays.asList("f1"), 10, 3))); + txn.addCommitData(serialize(pu("dt=b", TUpdateMode.NEW, "s3://b/b", Arrays.asList("f2"), 20, 5))); + txn.addCommitData(serialize(pu("dt=c", TUpdateMode.NEW, "s3://b/c", Arrays.asList("f3"), 30, 7))); + Assertions.assertEquals(15L, txn.getUpdateCnt()); + } + + @Test + public void testAddCommitDataRejectsGarbageBytes() { + // A corrupt commit fragment must surface as a DorisConnectorException (not a raw TException leaking + // through the SPI), so the engine can fail the load cleanly. + HiveConnectorTransaction txn = newTxn(new RecordingHmsClient()); + Assertions.assertThrows(DorisConnectorException.class, + () -> txn.addCommitData(new byte[] {1, 2, 3, 4, 5})); + } + + // ─────────────────────────────── transactional write reject (DEC-2) ─────────────────────────────── + + @Test + public void testBeginWriteRejectsFullAcidTable() { + // DEC-2: writing to a full-ACID table (transactional=true, not insert-only) is not supported and must + // be refused at begin — otherwise the write would silently corrupt the ACID base/delta layout. + RecordingHmsClient client = new RecordingHmsClient(); + Map params = new HashMap<>(); + params.put("transactional", "true"); + client.table = table(false, params); + HiveConnectorTransaction txn = newTxn(client); + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> txn.beginWrite(null, DB, TBL, ctx(false))); + Assertions.assertTrue(ex.getMessage().contains("begin write"), + "expected the begin-write wrapper message, got: " + ex.getMessage()); + } + + @Test + public void testBeginWriteRejectsInsertOnlyTransactionalTable() { + // DEC-2: the write guard now rejects ANY transactional table — full-ACID AND insert-only. An + // insert-only ACID table would otherwise slip through the narrower full-ACID-only guard and receive + // plain files (corrupt/invisible data), matching legacy InsertIntoTableCommand's isTransactionalTable + // reject. + RecordingHmsClient client = new RecordingHmsClient(); + Map params = new HashMap<>(); + params.put("transactional", "true"); + params.put("transactional_properties", "insert_only"); + client.table = table(false, params); + HiveConnectorTransaction txn = newTxn(client); + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> txn.beginWrite(null, DB, TBL, ctx(false))); + Assertions.assertTrue(ex.getMessage().contains("transactional"), + "expected the transactional-table reject, got: " + ex.getMessage()); + } + + @Test + public void testBeginWriteAllowsPlainTable() { + // A plain non-transactional table is the common INSERT case — begin must succeed. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(false, Collections.emptyMap()); + HiveConnectorTransaction txn = newTxn(client); + txn.beginWrite(null, DB, TBL, ctx(false)); // must not throw + } + + // ─────────────────────────────── classification / NEW→APPEND downgrade ─────────────────────────────── + + @Test + public void testPartitionedNewWhenAbsentCreatesPartition() throws TException { + // A NEW partition that HMS does NOT already have must be created: the existence probe is consulted, + // and because it is absent we take the create branch — which needs no getPartitions() lookup. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(true, Collections.emptyMap()); + client.partitionExistsResult = false; + HiveConnectorTransaction txn = newTxn(client); + txn.beginWrite(null, DB, TBL, ctx(false)); + txn.addCommitData(serialize(pu("dt=2024-01-01", TUpdateMode.NEW, "s3://bucket/db/t/dt=2024-01-01", + Arrays.asList("f1"), 100, 4))); + + txn.finishInsertTable(nameMapping()); + + Assertions.assertTrue(client.calls.contains("partitionExists:[2024-01-01]"), + "the NEW-mode branch must probe partition existence; calls=" + client.calls); + Assertions.assertFalse(client.calls.stream().anyMatch(c -> c.startsWith("getPartitions")), + "a genuinely-new partition takes the create path, which does not fetch existing partitions; " + + "calls=" + client.calls); + } + + @Test + public void testPartitionedNewWhenPresentDowngradesToAppend() throws TException { + // NEW→APPEND downgrade: when Doris' cache missed a partition that HMS actually has, the probe returns + // true and we must treat the write as an APPEND into the existing partition (getPartitions() is then + // consulted to build the INSERT_EXISTING action) — instead of trying to ADD a duplicate partition. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(true, Collections.emptyMap()); + client.partitionExistsResult = true; + client.partitions = Collections.singletonList(new HmsPartitionInfo( + Collections.singletonList("2024-01-01"), "s3://bucket/db/t/dt=2024-01-01", + "org.apache.hadoop.mapred.TextInputFormat", + "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", + "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe", Collections.emptyMap())); + HiveConnectorTransaction txn = newTxn(client); + txn.beginWrite(null, DB, TBL, ctx(false)); + txn.addCommitData(serialize(pu("dt=2024-01-01", TUpdateMode.NEW, "s3://bucket/db/t/dt=2024-01-01", + Arrays.asList("f1"), 100, 4))); + + txn.finishInsertTable(nameMapping()); + + Assertions.assertTrue(client.calls.contains("partitionExists:[2024-01-01]"), + "the downgrade must first probe existence; calls=" + client.calls); + Assertions.assertTrue(client.calls.stream().anyMatch(c -> c.startsWith("getPartitions")), + "an existing partition downgrades to APPEND, which fetches the real partition metadata; " + + "calls=" + client.calls); + } + + @Test + public void testPartitionedAppendFetchesExistingPartition() throws TException { + // A declared APPEND into a partitioned table goes straight to the INSERT_EXISTING path, which fetches + // the existing partition (no existence probe needed) — a wrong branch here would try to add it. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(true, Collections.emptyMap()); + client.partitions = Collections.singletonList(new HmsPartitionInfo( + Collections.singletonList("2024-01-01"), "s3://bucket/db/t/dt=2024-01-01", + "org.apache.hadoop.mapred.TextInputFormat", + "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", + "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe", Collections.emptyMap())); + HiveConnectorTransaction txn = newTxn(client); + txn.beginWrite(null, DB, TBL, ctx(false)); + txn.addCommitData(serialize(pu("dt=2024-01-01", TUpdateMode.APPEND, "s3://bucket/db/t/dt=2024-01-01", + Arrays.asList("f1"), 100, 4))); + + txn.finishInsertTable(nameMapping()); + + Assertions.assertTrue(client.calls.stream().anyMatch(c -> c.startsWith("getPartitions")), + "APPEND into a partitioned table must fetch the existing partition; calls=" + client.calls); + Assertions.assertFalse(client.calls.contains("partitionExists:[2024-01-01]"), + "a declared APPEND does not need the NEW-mode existence probe; calls=" + client.calls); + } + + @Test + public void testDuplicatePartitionValuesFromHmsFailsLoud() throws TException { + // Fidelity guard, ported from HiveUtil.convertToNamePartitionMap's Collectors.toMap: if HMS returns + // two partitions with identical values for a batch, the txn must abort LOUDLY rather than silently + // overwrite one. A silent HashMap overwrite would shrink the name→partition map below the requested + // count, and the size()-bounded loop would then drop a trailing partition's INSERT_EXISTING action — + // a silently lost write instead of a failed load. (HEAD's toMap throws IllegalStateException here.) + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(true, Collections.emptyMap()); + HmsPartitionInfo dup = new HmsPartitionInfo( + Collections.singletonList("2024-01-01"), "s3://bucket/db/t/dt=2024-01-01", + "org.apache.hadoop.mapred.TextInputFormat", + "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat", + "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe", Collections.emptyMap()); + client.partitions = Arrays.asList(dup, dup); // HMS returns the same partition values twice + HiveConnectorTransaction txn = newTxn(client); + txn.beginWrite(null, DB, TBL, ctx(false)); + txn.addCommitData(serialize(pu("dt=2024-01-01", TUpdateMode.APPEND, "s3://bucket/db/t/dt=2024-01-01", + Arrays.asList("f1"), 100, 4))); + + IllegalStateException ex = Assertions.assertThrows(IllegalStateException.class, + () -> txn.finishInsertTable(nameMapping())); + Assertions.assertTrue(ex.getMessage().contains("Duplicate key"), + "expected a fail-loud abort on duplicate partition values, got: " + ex.getMessage()); + } + + // ─────────────────────────────── commit-time object-store MPU (D6) ─────────────────────────────── + + @Test + public void testCommitCompletesMultipartUploads() throws TException { + // On commit the FE finalizes the multipart uploads that BE staged on the object store (D6). For an + // unpartitioned APPEND whose write path already equals the table location (no rename needed), the + // committer must complete the MPU with the s3://bucket/key URI, the upload id, and the part ETags + // sorted by part number — a missing/mis-ordered completion would leave the written data invisible. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(false, Collections.emptyMap()); + RecordingObjStorage objStorage = new RecordingObjStorage(); + HiveConnectorTransaction txn = newTxnWithFs(client, new RecordingObjFileSystem(objStorage)); + txn.beginWrite(null, DB, TBL, ctx(false)); + Map etags = new HashMap<>(); + etags.put(2, "etag-2"); + etags.put(1, "etag-1"); + // writePath == table location "s3://bucket/db/t" -> needRename=false -> objCommit (MPU complete) path. + txn.addCommitData(serialize(puWithMpu("", TUpdateMode.APPEND, "s3://bucket/db/t", + "bucket", "db/t/data", "upload-1", etags))); + + txn.commit(); + txn.close(); + + Assertions.assertEquals( + Collections.singletonList("complete:s3://bucket/db/t/data:upload-1:[1=etag-1, 2=etag-2]"), + objStorage.calls, + "commit must complete the MPU once with the ETags sorted by part number; calls=" + objStorage.calls); + Assertions.assertTrue(client.calls.contains("updateTableStatistics:" + DB + "." + TBL), + "an unpartitioned INSERT_EXISTING must also update the table statistics; calls=" + client.calls); + } + + @Test + public void testRollbackAbortsPendingMultipartUploads() throws TException { + // rollback() is NOT a no-op for hive (D9): data files are staged before commit, so a rollback must + // abort the in-flight object-store multipart uploads — otherwise they linger server-side (leaked / + // billed). Here the committer was never built (rollback before commit), so the pending uploads are + // collected straight from the accumulated commit fragments. + RecordingObjStorage objStorage = new RecordingObjStorage(); + HiveConnectorTransaction txn = newTxnWithFs(new RecordingHmsClient(), + new RecordingObjFileSystem(objStorage)); + txn.addCommitData(serialize(puWithMpu("", TUpdateMode.APPEND, "s3://bucket/db/t", + "bucket", "db/t/data", "upload-9", Collections.singletonMap(1, "etag-1")))); + + txn.rollback(); + txn.close(); + + Assertions.assertEquals(Collections.singletonList("abort:s3://bucket/db/t/data:upload-9"), + objStorage.calls, + "rollback must abort the pending MPU exactly once; calls=" + objStorage.calls); + } + + @Test + public void testSecondRollbackIsIdempotent() throws TException { + // Rollback must be safe to call more than once (the engine may retry cleanup): the second call must + // neither throw nor re-abort an already-aborted upload (a double abort would error against the store). + RecordingObjStorage objStorage = new RecordingObjStorage(); + HiveConnectorTransaction txn = newTxnWithFs(new RecordingHmsClient(), + new RecordingObjFileSystem(objStorage)); + txn.addCommitData(serialize(puWithMpu("", TUpdateMode.APPEND, "s3://bucket/db/t", + "bucket", "db/t/data", "upload-9", Collections.singletonMap(1, "etag-1")))); + + txn.rollback(); + Assertions.assertDoesNotThrow(txn::rollback, "a second rollback must not throw"); + txn.close(); + + Assertions.assertEquals(Collections.singletonList("abort:s3://bucket/db/t/data:upload-9"), + objStorage.calls, + "the pending MPU must be aborted exactly once across repeated rollbacks; calls=" + objStorage.calls); + } + + @Test + public void testCommitAddsNewPartitionOnce() throws TException { + // GAP-7: the 20-at-a-time batching moved INTO ThriftHmsClient.addPartitions, so the committer must + // call addPartitions ONCE with the whole list (not re-batch it). GAP-4: the new partition's storage + // descriptor (values/location/columns) is rebuilt from the table at commit time. A genuinely-new + // partition takes the ADD path; on FILE_S3 the write path == target path, so no rename/MPU runs and + // the object-store FileSystem is never resolved (hence newTxn, not newTxnWithFs). + RecordingHmsClient client = new RecordingHmsClient(); + client.table = table(true, Collections.emptyMap()); + client.partitionExistsResult = false; + HiveConnectorTransaction txn = newTxn(client); + txn.beginWrite(null, DB, TBL, ctx(false)); + txn.addCommitData(serialize(pu("dt=2024-01-01", TUpdateMode.NEW, "s3://bucket/db/t/dt=2024-01-01", + Collections.singletonList("f1"), 100, 4))); + + txn.commit(); + txn.close(); + + Assertions.assertEquals(1, client.addedPartitions.size(), + "exactly one partition must be added; calls=" + client.calls); + Assertions.assertEquals(1L, client.calls.stream().filter(c -> c.startsWith("addPartitions:")).count(), + "addPartitions must be invoked exactly once (batching lives in the client); calls=" + client.calls); + HmsPartitionWithStatistics added = client.addedPartitions.get(0); + Assertions.assertEquals(Collections.singletonList("2024-01-01"), added.getPartitionValues()); + Assertions.assertEquals("s3://bucket/db/t/dt=2024-01-01", added.getLocation()); + Assertions.assertEquals(Collections.singletonList("c1"), + added.getColumns().stream().map(FieldSchema::getName).collect(Collectors.toList()), + "the new partition's SD columns must be rebuilt from the table schema"); + } + + /** + * Recording {@link HmsClient} fake: implements the abstract read surface, records the calls the + * transaction makes, and returns canned metadata. The Phase-3+ write primitives ({@code addPartitions} / + * statistics / {@code dropPartition}) are recorded too — the committer reaches them once the FS tests + * drive a full commit. + */ + private static final class RecordingHmsClient implements HmsClient { + private final List calls = new ArrayList<>(); + private final List addedPartitions = new ArrayList<>(); + private HmsTableInfo table; + private boolean partitionExistsResult; + private List partitions = Collections.emptyList(); + + @Override + public List listDatabases() { + return Collections.emptyList(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException("getDatabase"); + } + + @Override + public List listTables(String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return table != null; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + calls.add("getTable:" + dbName + "." + tableName); + if (table == null) { + throw new UnsupportedOperationException("no canned table"); + } + return table; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return Collections.emptyList(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + calls.add("getPartitions:" + partNames); + return partitions; + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException("getPartition"); + } + + @Override + public boolean partitionExists(String dbName, String tableName, List partitionValues) { + calls.add("partitionExists:" + partitionValues); + return partitionExistsResult; + } + + @Override + public void addPartitions(String dbName, String tableName, List partitions) { + calls.add("addPartitions:" + dbName + "." + tableName + ":" + partitions.size()); + addedPartitions.addAll(partitions); + } + + @Override + public void updateTableStatistics(String dbName, String tableName, + Function update) { + calls.add("updateTableStatistics:" + dbName + "." + tableName); + } + + @Override + public void updatePartitionStatistics(String dbName, String tableName, String partitionName, + Function update) { + calls.add("updatePartitionStatistics:" + dbName + "." + tableName + ":" + partitionName); + } + + @Override + public boolean dropPartition(String dbName, String tableName, List partitionValues, + boolean deleteData) { + calls.add("dropPartition:" + partitionValues + ":" + deleteData); + return true; + } + + @Override + public void close() { + } + } + + /** + * Recording {@link ObjStorage} fake capturing both multipart-upload completions and aborts (the two + * object-store operations the commit/rollback paths reach). Every other operation fails loud so an + * unexpected code path surfaces immediately rather than silently no-op'ing. + */ + private static final class RecordingObjStorage implements ObjStorage { + private final List calls = new ArrayList<>(); + + @Override + public void completeMultipartUpload(String remotePath, String uploadId, List parts) { + String partStr = parts.stream().map(p -> p.partNumber() + "=" + p.etag()) + .collect(Collectors.joining(", ", "[", "]")); + calls.add("complete:" + remotePath + ":" + uploadId + ":" + partStr); + } + + @Override + public void abortMultipartUpload(String remotePath, String uploadId) { + calls.add("abort:" + remotePath + ":" + uploadId); + } + + @Override + public Object getClient() { + throw new UnsupportedOperationException("getClient"); + } + + @Override + public RemoteObjects listObjects(String remotePath, String continuationToken) { + throw new UnsupportedOperationException("listObjects"); + } + + @Override + public RemoteObject headObject(String remotePath) { + throw new UnsupportedOperationException("headObject"); + } + + @Override + public void putObject(String remotePath, RequestBody requestBody) { + throw new UnsupportedOperationException("putObject"); + } + + @Override + public void deleteObject(String remotePath) { + throw new UnsupportedOperationException("deleteObject"); + } + + @Override + public void copyObject(String srcPath, String dstPath) { + throw new UnsupportedOperationException("copyObject"); + } + + @Override + public String initiateMultipartUpload(String remotePath) { + throw new UnsupportedOperationException("initiateMultipartUpload"); + } + + @Override + public UploadPartResult uploadPart(String remotePath, String uploadId, int partNum, RequestBody body) { + throw new UnsupportedOperationException("uploadPart"); + } + + @Override + public void close() { + } + } + + /** + * Fake {@link ObjFileSystem} over a {@link RecordingObjStorage}. {@code ObjFileSystem} already provides + * {@code exists()}/{@code close()} and the {@code completeMultipartUpload(Map)} overload that converts the + * ETag map to a sorted part list before delegating to the storage; the remaining core FileSystem methods + * are not exercised by the MPU tests and fail loud if unexpectedly reached. + */ + private static final class RecordingObjFileSystem extends ObjFileSystem { + RecordingObjFileSystem(ObjStorage objStorage) { + super(objStorage); + } + + @Override + public void mkdirs(Location location) { + throw new UnsupportedOperationException("mkdirs"); + } + + @Override + public void delete(Location location, boolean recursive) { + throw new UnsupportedOperationException("delete"); + } + + @Override + public void rename(Location src, Location dst) { + throw new UnsupportedOperationException("rename"); + } + + @Override + public FileIterator list(Location location) { + throw new UnsupportedOperationException("list"); + } + + @Override + public DorisInputFile newInputFile(Location location) { + throw new UnsupportedOperationException("newInputFile"); + } + + @Override + public DorisOutputFile newOutputFile(Location location) { + throw new UnsupportedOperationException("newOutputFile"); + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java new file mode 100644 index 00000000000000..6f596702a44378 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java @@ -0,0 +1,204 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Tests the plugin read-side transaction lifecycle ({@link HiveReadTransaction} / + * {@link HiveReadTransactionManager}) against a recording-fake {@link HmsClient}. + * + *

WHY: reading a transactional Hive table must open a metastore transaction, take a shared read lock + * exactly once (holding it for the whole scan pins a consistent write-id snapshot), and release it by + * committing when the query finishes. Acquiring the lock more than once, or forgetting to commit, either + * corrupts snapshot isolation or leaks the lock. These tests pin that once-per-query lock + commit + * contract.

+ */ +public class HiveReadTransactionTest { + + @Test + public void testBeginOpensTxnAndValidWriteIdsAcquiresLockOnce() { + RecordingHmsClient client = new RecordingHmsClient(); + client.openTxnReturn = 777L; + client.validIds.put("hive.txn.valid.writeids", "db.tbl:8:x"); + + HiveReadTransaction txn = new HiveReadTransaction( + "q1", "alice", "db", "tbl", true, client); + txn.begin(); + Assertions.assertEquals(List.of("openTxn:alice"), client.calls); + + txn.addPartition("dt=2026-01-01"); + Map ids = txn.getValidWriteIds(); + Assertions.assertEquals("db.tbl:8:x", ids.get("hive.txn.valid.writeids")); + // The lock must be scoped to the transaction, the (db, table), and the added partitions. + Assertions.assertEquals(777L, client.lockTxnId); + Assertions.assertEquals("db.tbl", client.getValidWriteIdsTable); + Assertions.assertEquals(777L, client.getValidWriteIdsTxnId); + Assertions.assertEquals(List.of("dt=2026-01-01"), client.lockPartitions); + + // Second call must be memoized: no extra lock / getValidWriteIds round-trips. + Map ids2 = txn.getValidWriteIds(); + Assertions.assertSame(ids, ids2); + Assertions.assertEquals(1, count(client.calls, "acquireSharedLock:q1")); + Assertions.assertEquals(1, count(client.calls, "getValidWriteIds:db.tbl")); + } + + @Test + public void testCommitCommitsTheTxn() { + RecordingHmsClient client = new RecordingHmsClient(); + client.openTxnReturn = 42L; + HiveReadTransaction txn = new HiveReadTransaction("q1", "bob", "db", "t", false, client); + txn.begin(); + txn.commit(); + Assertions.assertTrue(client.calls.contains("commitTxn:42"), client.calls.toString()); + } + + @Test + public void testManagerRegisterBeginsAndDeregisterCommits() { + RecordingHmsClient client = new RecordingHmsClient(); + client.openTxnReturn = 5L; + HiveReadTransactionManager mgr = new HiveReadTransactionManager(); + + HiveReadTransaction txn = new HiveReadTransaction("q1", "u", "db", "t", true, client); + mgr.register(txn); + Assertions.assertEquals(1, count(client.calls, "openTxn:u"), "register opens the txn"); + + mgr.deregister("q1"); + Assertions.assertEquals(1, count(client.calls, "commitTxn:5"), "deregister commits the txn"); + + // Idempotent: a second deregister and an unknown query must be no-ops. + mgr.deregister("q1"); + mgr.deregister("unknown"); + Assertions.assertEquals(1, count(client.calls, "commitTxn:5"), "no double commit"); + } + + private static int count(List calls, String prefix) { + int n = 0; + for (String c : calls) { + if (c.startsWith(prefix)) { + n++; + } + } + return n; + } + + /** + * Recording {@link HmsClient} fake: stubs the abstract read surface and records the four read-ACID + * primitives the read transaction drives. All other primitives keep their default {@code throw}. + */ + private static final class RecordingHmsClient implements HmsClient { + final List calls = new ArrayList<>(); + long openTxnReturn; + long lockTxnId; + List lockPartitions; + long getValidWriteIdsTxnId; + String getValidWriteIdsTable; + final Map validIds = new HashMap<>(); + + @Override + public long openTxn(String user) { + calls.add("openTxn:" + user); + return openTxnReturn; + } + + @Override + public void acquireSharedLock(String queryId, long txnId, String user, String dbName, + String tableName, List partitionNames, long timeoutMs) { + calls.add("acquireSharedLock:" + queryId + ":" + txnId); + this.lockTxnId = txnId; + this.lockPartitions = new ArrayList<>(partitionNames); + } + + @Override + public Map getValidWriteIds(String fullTableName, long currentTransactionId) { + calls.add("getValidWriteIds:" + fullTableName + ":" + currentTransactionId); + this.getValidWriteIdsTxnId = currentTransactionId; + this.getValidWriteIdsTable = fullTableName; + return validIds; + } + + @Override + public void commitTxn(long txnId) { + calls.add("commitTxn:" + txnId); + } + + // ---- unused abstract read surface ---- + + @Override + public List listDatabases() { + return Collections.emptyList(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException("getDatabase"); + } + + @Override + public List listTables(String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return false; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException("getTable"); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return Collections.emptyList(); + } + + @Override + public List getPartitions(String dbName, String tableName, + List partNames) { + return Collections.emptyList(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException("getPartition"); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableHandleAcidTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableHandleAcidTest.java new file mode 100644 index 00000000000000..12adca92e4b655 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableHandleAcidTest.java @@ -0,0 +1,90 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests the ACID classification derived from a {@link HiveTableHandle}'s metastore parameters. + * + *

WHY: the scan path branches on these flags. {@code isTransactional} decides whether the ACID + * descent runs at all; {@code isFullAcid} (transactional AND not insert-only) decides whether delete + * deltas and the bucket_ file filter apply. Getting insert-only vs full-ACID wrong either skips real + * row deletes or mis-filters data files.

+ */ +public class HiveTableHandleAcidTest { + + private HiveTableHandle handleWithParams(Map params) { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat") + .tableParameters(params) + .build(); + } + + @Test + public void testNonTransactionalByDefault() { + HiveTableHandle handle = handleWithParams(new HashMap<>()); + Assertions.assertFalse(handle.isTransactional()); + Assertions.assertFalse(handle.isFullAcid()); + } + + @Test + public void testFullAcidWhenTransactionalAndNotInsertOnly() { + Map params = new HashMap<>(); + params.put("transactional", "true"); + HiveTableHandle handle = handleWithParams(params); + Assertions.assertTrue(handle.isTransactional()); + Assertions.assertTrue(handle.isFullAcid()); + } + + @Test + public void testInsertOnlyIsTransactionalButNotFullAcid() { + Map params = new HashMap<>(); + params.put("transactional", "true"); + params.put("transactional_properties", "insert_only"); + HiveTableHandle handle = handleWithParams(params); + Assertions.assertTrue(handle.isTransactional()); + Assertions.assertFalse(handle.isFullAcid(), + "insert_only ACID tables have no row-level deletes"); + } + + @Test + public void testInsertOnlyDetectionIsCaseInsensitive() { + Map params = new HashMap<>(); + params.put("transactional", "true"); + params.put("transactional_properties", "INSERT_ONLY"); + Assertions.assertFalse(handleWithParams(params).isFullAcid(), + "insert_only detection must be case-insensitive"); + } + + @Test + public void testTransactionalTrueIsCaseInsensitive() { + Map params = new HashMap<>(); + params.put("transactional", "TRUE"); + Assertions.assertTrue(handleWithParams(params).isTransactional()); + + Map upperKey = new HashMap<>(); + upperKey.put("TRANSACTIONAL", "true"); + Assertions.assertTrue(handleWithParams(upperKey).isTransactional(), + "the upper-cased parameter key is accepted as a fallback"); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWritePlanProviderTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWritePlanProviderTest.java new file mode 100644 index 00000000000000..737d2f669c3832 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWritePlanProviderTest.java @@ -0,0 +1,716 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.connector.spi.ConnectorBrokerAddress; +import org.apache.doris.filesystem.FileSystemType; +import org.apache.doris.filesystem.properties.BackendStorageKind; +import org.apache.doris.filesystem.properties.BackendStorageProperties; +import org.apache.doris.filesystem.properties.StorageKind; +import org.apache.doris.filesystem.properties.StorageProperties; +import org.apache.doris.thrift.TDataSinkType; +import org.apache.doris.thrift.TFileCompressType; +import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.TFileType; +import org.apache.doris.thrift.THiveColumn; +import org.apache.doris.thrift.THiveColumnType; +import org.apache.doris.thrift.THiveLocationParams; +import org.apache.doris.thrift.THivePartition; +import org.apache.doris.thrift.THiveTableSink; +import org.apache.doris.thrift.TNetworkAddress; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Pins {@link HiveWritePlanProvider#planWrite} for INSERT / INSERT OVERWRITE against legacy + * {@code planner.HiveTableSink.bindDataSink} expected values (recording {@link HmsClient} fake, no + * Mockito). + * + *

WHY this matters: the {@code THiveTableSink} assembly moved out of the fe-core planner into the + * connector. The sink Thrift goes to BE unchanged (zero BE change), so every field must be byte-identical to + * the legacy sink: the tagged column list, the existing-partition list, bucket info, file format/compression, + * the staging-vs-in-place location, the SerDe delimiters, and the overwrite flag. A parity-by-omission (a + * dropped/mis-tagged field) silently corrupts hive writes once hive cuts over. Each assertion pins the WHY the + * field is load-bearing for BE.

+ */ +public class HiveWritePlanProviderTest { + + private static final String DB = "db"; + private static final String TBL = "t"; + + private static final String TEXT_INPUT_FORMAT = "org.apache.hadoop.mapred.TextInputFormat"; + private static final String ORC_INPUT_FORMAT = "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + private static final String PARQUET_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String LZO_INPUT_FORMAT = "com.hadoop.mapred.DeprecatedLzoTextInputFormat"; + private static final String TEXT_OUTPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat"; + private static final String LAZY_SIMPLE_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; + private static final String MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; + + // ───────────────────────────── helpers ───────────────────────────── + + private static ConnectorColumn col(String name, String type) { + return new ConnectorColumn(name, ConnectorType.of(type), "", true, null); + } + + private static HmsTableInfo.Builder tableBuilder() { + return HmsTableInfo.builder() + .dbName(DB).tableName(TBL).tableType("MANAGED_TABLE") + .location("oss://bucket/db/t") + .inputFormat(TEXT_INPUT_FORMAT) + .outputFormat(TEXT_OUTPUT_FORMAT) + .serializationLib(LAZY_SIMPLE_SERDE) + .columns(Collections.singletonList(col("c1", "int"))) + .partitionKeys(Collections.emptyList()) + .parameters(Collections.emptyMap()); + } + + private HiveWritePlanProvider providerFor(RecordingHmsClient client, RecordingConnectorContext ctx) { + return new HiveWritePlanProvider(client, Collections.emptyMap(), ctx); + } + + private WriteSession sessionFor(RecordingHmsClient client, RecordingConnectorContext ctx, + Map sessionProperties) { + return new WriteSession(new HiveConnectorTransaction(42L, client, ctx), sessionProperties); + } + + private THiveTableSink planSink(RecordingHmsClient client, RecordingConnectorContext ctx, + WriteHandle handle) { + return planSink(client, ctx, handle, Collections.emptyMap()); + } + + private THiveTableSink planSink(RecordingHmsClient client, RecordingConnectorContext ctx, + WriteHandle handle, Map sessionProperties) { + ConnectorSinkPlan plan = providerFor(client, ctx) + .planWrite(sessionFor(client, ctx, sessionProperties), handle); + Assertions.assertEquals(TDataSinkType.HIVE_TABLE_SINK, plan.getDataSink().getType()); + return plan.getDataSink().getHiveTableSink(); + } + + private static WriteHandle handle() { + return new WriteHandle(new HiveTableHandle(DB, TBL, HiveTableType.HIVE)); + } + + // ───────────────────────────── columns ───────────────────────────── + + @Test + public void planWriteEmitsRegularColumnsThenPartitionKeys() { + // BE writes the data columns then maps the trailing partition keys; a wrong tag or order would write + // rows into the wrong file layout. Data cols are REGULAR (in schema order), partition keys are + // PARTITION_KEY, appended last (HMS/HMSExternalTable order). + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder() + .columns(Arrays.asList(col("c1", "int"), col("c2", "string"))) + .partitionKeys(Collections.singletonList(col("dt", "string"))) + .build(); + + List columns = planSink(client, new RecordingConnectorContext(), handle()).getColumns(); + + Assertions.assertEquals(3, columns.size()); + Assertions.assertEquals("c1", columns.get(0).getName()); + Assertions.assertEquals(THiveColumnType.REGULAR, columns.get(0).getColumnType()); + Assertions.assertEquals("c2", columns.get(1).getName()); + Assertions.assertEquals(THiveColumnType.REGULAR, columns.get(1).getColumnType()); + Assertions.assertEquals("dt", columns.get(2).getName()); + Assertions.assertEquals(THiveColumnType.PARTITION_KEY, columns.get(2).getColumnType(), + "partition keys must be tagged PARTITION_KEY and appended after the data columns"); + } + + // ───────────────────────────── bucket ───────────────────────────── + + @Test + public void planWriteSetsBucketInfo() { + // The BE hive writer buckets rows by these columns into this many files; a dropped bucket spec would + // write an un-bucketed layout that a bucketed reader then mis-reads. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder() + .bucketCols(Collections.singletonList("c1")) + .numBuckets(8) + .build(); + + THiveTableSink sink = planSink(client, new RecordingConnectorContext(), handle()); + + Assertions.assertEquals(Collections.singletonList("c1"), sink.getBucketInfo().getBucketedBy()); + Assertions.assertEquals(8, sink.getBucketInfo().getBucketCount()); + } + + // ───────────────────────────── file format ───────────────────────────── + + @Test + public void planWriteFileFormatPerInputFormat() { + // The input-format class name decides the BE writer dialect: orc -> ORC, parquet -> PARQUET, and + // (parity trap) text -> FORMAT_CSV_PLAIN. Writing the wrong container makes every row unreadable. + Assertions.assertEquals(TFileFormatType.FORMAT_ORC, fileFormatFor(ORC_INPUT_FORMAT)); + Assertions.assertEquals(TFileFormatType.FORMAT_PARQUET, fileFormatFor(PARQUET_INPUT_FORMAT)); + Assertions.assertEquals(TFileFormatType.FORMAT_CSV_PLAIN, fileFormatFor(TEXT_INPUT_FORMAT)); + } + + private TFileFormatType fileFormatFor(String inputFormat) { + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().inputFormat(inputFormat).build(); + return planSink(client, new RecordingConnectorContext(), handle()).getFileFormat(); + } + + // ───────────────────────────── compression ───────────────────────────── + + @Test + public void planWriteCompressionPerFormat() { + // The compression codec is read from the table parameters per format (orc.compress / + // parquet.compression / text.compression). BE writes the file with this codec; a wrong codec yields + // files a reader cannot decompress. + Assertions.assertEquals(TFileCompressType.ZLIB, + compressionFor(ORC_INPUT_FORMAT, Collections.singletonMap("orc.compress", "ZLIB"), + Collections.emptyMap())); + Assertions.assertEquals(TFileCompressType.SNAPPYBLOCK, + compressionFor(PARQUET_INPUT_FORMAT, Collections.singletonMap("parquet.compression", "snappy"), + Collections.emptyMap())); + Assertions.assertEquals(TFileCompressType.GZ, + compressionFor(TEXT_INPUT_FORMAT, Collections.singletonMap("text.compression", "gzip"), + Collections.emptyMap())); + } + + @Test + public void planWriteTextCompressionFallsBackToSessionDefault() { + // A text table with no text.compression property falls back to the session hive_text_compression + // default (legacy SessionVariable.hiveTextCompression); "uncompressed" is aliased to "plain". + Assertions.assertEquals(TFileCompressType.ZSTD, + compressionFor(TEXT_INPUT_FORMAT, Collections.emptyMap(), + Collections.singletonMap("hive_text_compression", "zstd"))); + Assertions.assertEquals(TFileCompressType.PLAIN, + compressionFor(TEXT_INPUT_FORMAT, Collections.emptyMap(), + Collections.singletonMap("hive_text_compression", "uncompressed"))); + } + + private TFileCompressType compressionFor(String inputFormat, Map tableParams, + Map sessionProperties) { + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().inputFormat(inputFormat).parameters(tableParams).build(); + return planSink(client, new RecordingConnectorContext(), handle(), sessionProperties).getCompressionType(); + } + + // ───────────────────────────── location ───────────────────────────── + + @Test + public void planWriteLocationInPlaceForObjectStore() { + // An object-store write goes in place: the write and target paths are the normalized (s3://) URI and + // the original raw URI is preserved so BE can resolve credentials for the native scheme. No staging + // dir — BE writes straight to the table location. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().location("oss://bucket/db/t").build(); + RecordingConnectorContext ctx = new RecordingConnectorContext(); + ctx.backendFileType = TFileType.FILE_S3; + + THiveLocationParams loc = planSink(client, ctx, handle()).getLocation(); + + Assertions.assertEquals(TFileType.FILE_S3, loc.getFileType()); + Assertions.assertEquals("s3://bucket/db/t", loc.getWritePath(), + "an object-store write path must be the normalized (s3://) URI BE opens"); + Assertions.assertEquals("s3://bucket/db/t", loc.getTargetPath()); + Assertions.assertEquals("oss://bucket/db/t", loc.getOriginalWritePath(), + "the original raw URI must be preserved so BE can resolve creds for the native scheme"); + } + + @Test + public void planWriteLocationStagingForHdfs() { + // A non-object-store write goes to a staging temp dir under the table location; BE writes there and the + // commit renames it to the target. write == original == staging; target == the raw table location. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().location("hdfs://ns/wh/db/t").build(); + RecordingConnectorContext ctx = new RecordingConnectorContext(); + ctx.backendFileType = TFileType.FILE_HDFS; + + THiveLocationParams loc = planSink(client, ctx, handle()).getLocation(); + + Assertions.assertEquals(TFileType.FILE_HDFS, loc.getFileType()); + Assertions.assertEquals("hdfs://ns/wh/db/t", loc.getTargetPath(), + "the target path must remain the live table location the commit renames into"); + Assertions.assertEquals(loc.getWritePath(), loc.getOriginalWritePath(), + "a staged write has no scheme rewrite, so write and original paths are identical"); + Assertions.assertNotEquals("hdfs://ns/wh/db/t", loc.getWritePath(), + "the staged write path must differ from the live table location"); + Assertions.assertTrue(loc.getWritePath().contains(".doris_staging"), + "the staged write path must sit under the .doris_staging base dir; got: " + loc.getWritePath()); + } + + // ───────────────────────────── serde delimiters ───────────────────────────── + + @Test + public void planWriteSerdeDefaultDelimiters() { + // With no SerDe params BE must fall back to the hive text defaults; a wrong default silently mis-splits + // every text row. escape is unset (no escape.delim), matching legacy's Optional-only emission. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().build(); + + THiveTableSink sink = planSink(client, new RecordingConnectorContext(), handle()); + + Assertions.assertEquals("\1", sink.getSerdeProperties().getFieldDelim()); + Assertions.assertEquals("\n", sink.getSerdeProperties().getLineDelim()); + Assertions.assertEquals("\2", sink.getSerdeProperties().getCollectionDelim()); + Assertions.assertEquals("\003", sink.getSerdeProperties().getMapkvDelim()); + Assertions.assertEquals("\\N", sink.getSerdeProperties().getNullFormat()); + Assertions.assertFalse(sink.getSerdeProperties().isSetEscapeChar(), + "with no escape.delim the escape char must stay unset (legacy sets it only when present)"); + } + + @Test + public void planWriteSerdeMultiDelimitKeepsMultiCharFieldDelim() { + // The MultiDelimitSerDe lib is the ONLY case where a multi-char field delimiter is passed through + // verbatim (not byte-decoded); collapsing "||" to one char would corrupt every row boundary. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder() + .serializationLib(MULTI_DELIMIT_SERDE) + .sdParameters(Collections.singletonMap("field.delim", "||")) + .build(); + + Assertions.assertEquals("||", + planSink(client, new RecordingConnectorContext(), handle()).getSerdeProperties().getFieldDelim()); + } + + @Test + public void planWriteSerdeNumericFieldDelimDecodesByte() { + // A numeric field.delim like "9" is a byte value, decoded to that char ((9 + 256) % 256 -> char 9 = + // TAB) — not the literal digit '9'. A missed decode would split on the wrong byte. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder() + .sdParameters(Collections.singletonMap("field.delim", "9")) + .build(); + + Assertions.assertEquals("\t", + planSink(client, new RecordingConnectorContext(), handle()).getSerdeProperties().getFieldDelim()); + } + + @Test + public void planWriteSerdeTableParamWinsOverSdParam() { + // getSerdeProperty is table-param-FIRST: a field.delim in the table parameters overrides the SD SerDe + // parameters. A reversed precedence would pick the wrong delimiter for a table that carries both. + RecordingHmsClient client = new RecordingHmsClient(); + Map tableParams = new HashMap<>(); + tableParams.put("field.delim", "a"); + client.table = tableBuilder() + .parameters(tableParams) + .sdParameters(Collections.singletonMap("field.delim", "b")) + .build(); + + // "a" is non-numeric, so getByte returns its first raw char "a" (proves the table value, not "b", won). + Assertions.assertEquals("a", + planSink(client, new RecordingConnectorContext(), handle()).getSerdeProperties().getFieldDelim()); + } + + // ───────────────────────────── overwrite + promotion ───────────────────────────── + + @Test + public void planWriteOverwriteFlag() { + // The overwrite flag drives BE's truncate-and-insert vs append; a dropped flag turns an INSERT + // OVERWRITE into a silent append (duplicated data). + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().build(); + Assertions.assertTrue(planSink(client, new RecordingConnectorContext(), handle().overwrite(true)) + .isOverwrite()); + + RecordingHmsClient client2 = new RecordingHmsClient(); + client2.table = tableBuilder().build(); + Assertions.assertFalse(planSink(client2, new RecordingConnectorContext(), handle().overwrite(false)) + .isOverwrite()); + } + + @Test + public void buildWriteContextPromotesOverwritingInsertToOverwrite() { + // An overwriting INSERT is promoted to the OVERWRITE operation on the op-context (the transaction + // classifies APPEND vs OVERWRITE off this); a plain INSERT stays INSERT. + RecordingHmsClient client = new RecordingHmsClient(); + HmsTableInfo table = tableBuilder().build(); + client.table = table; + RecordingConnectorContext ctx = new RecordingConnectorContext(); + HiveWritePlanProvider provider = providerFor(client, ctx); + ConnectorSession session = new WriteSession(null, Collections.emptyMap()); + HiveTableHandle tableHandle = new HiveTableHandle(DB, TBL, HiveTableType.HIVE); + + HiveWriteContext promoted = provider.buildWriteContext(session, tableHandle, table, + new WriteHandle(tableHandle).overwrite(true)); + Assertions.assertEquals(WriteOperation.OVERWRITE, promoted.getWriteOperation(), + "an overwriting INSERT must be promoted to OVERWRITE"); + + HiveWriteContext plain = provider.buildWriteContext(session, tableHandle, table, + new WriteHandle(tableHandle).overwrite(false)); + Assertions.assertEquals(WriteOperation.INSERT, plain.getWriteOperation(), + "a non-overwriting INSERT must stay INSERT"); + } + + // ───────────────────────────── broker backend ───────────────────────────── + + @Test + public void planWriteBrokerBackendResolvesAddresses() { + // A broker-backed write must carry the resolved broker addresses, or BE gets a broker sink with an + // empty broker list and the write fails. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().location("ofs://bucket/db/t").build(); + RecordingConnectorContext ctx = new RecordingConnectorContext(); + ctx.backendFileType = TFileType.FILE_BROKER; + ctx.brokerAddresses = Collections.singletonList(new ConnectorBrokerAddress("h1", 8000)); + + List brokers = planSink(client, ctx, handle()).getBrokerAddresses(); + + Assertions.assertEquals(1, brokers.size()); + Assertions.assertEquals("h1", brokers.get(0).getHostname()); + Assertions.assertEquals(8000, brokers.get(0).getPort()); + } + + @Test + public void planWriteBrokerBackendFailsLoudWhenNoBroker() { + // No alive broker for a broker-backed write must fail loud rather than ship BE an empty broker list. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().location("ofs://bucket/db/t").build(); + RecordingConnectorContext ctx = new RecordingConnectorContext(); + ctx.backendFileType = TFileType.FILE_BROKER; + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> planSink(client, ctx, handle())); + Assertions.assertTrue(ex.getMessage().contains("No alive broker"), + "the broker resolution must fail loud with 'No alive broker.', got: " + ex.getMessage()); + } + + // ───────────────────────────── LZO reject ───────────────────────────── + + @Test + public void planWriteRejectsLzoInputFormat() { + // A Doris-written LZO file has no .lzo suffix while the LZO read path filters to *.lzo only, so every + // written row would be permanently invisible. The write must be rejected loudly at plan time. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().inputFormat(LZO_INPUT_FORMAT).build(); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> planSink(client, new RecordingConnectorContext(), handle())); + Assertions.assertTrue(ex.getMessage().contains("LZO"), + "the LZO reject must name LZO, got: " + ex.getMessage()); + } + + // ───────────────────────────── existing partitions ───────────────────────────── + + @Test + public void planWritePartitionedTableEmitsExistingPartitions() { + // For a partitioned table BE needs the existing partition dirs (to distinguish APPEND from NEW); the + // list is fetched live. Each THivePartition carries its values, its own file format, and its in-place + // location (write == target). + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder() + .partitionKeys(Collections.singletonList(col("dt", "string"))) + .build(); + client.partitionNames = Collections.singletonList("dt=2024-01-01"); + client.partitions = Collections.singletonList(new HmsPartitionInfo( + Collections.singletonList("2024-01-01"), "oss://bucket/db/t/dt=2024-01-01", + PARQUET_INPUT_FORMAT, TEXT_OUTPUT_FORMAT, LAZY_SIMPLE_SERDE, Collections.emptyMap())); + RecordingConnectorContext ctx = new RecordingConnectorContext(); + ctx.backendFileType = TFileType.FILE_S3; + + List partitions = planSink(client, ctx, handle()).getPartitions(); + + Assertions.assertEquals(1, partitions.size()); + THivePartition part = partitions.get(0); + Assertions.assertEquals(Collections.singletonList("2024-01-01"), part.getValues()); + Assertions.assertEquals(TFileFormatType.FORMAT_PARQUET, part.getFileFormat(), + "each partition's file format is resolved from its own input format"); + Assertions.assertEquals("oss://bucket/db/t/dt=2024-01-01", part.getLocation().getWritePath()); + Assertions.assertEquals("oss://bucket/db/t/dt=2024-01-01", part.getLocation().getTargetPath(), + "an existing partition is read in place, so its write and target paths are the same"); + Assertions.assertEquals(TFileType.FILE_S3, part.getLocation().getFileType()); + Assertions.assertTrue(client.calls.contains("listPartitionNames"), + "the existing-partition list must be fetched live; calls=" + client.calls); + Assertions.assertTrue(client.calls.stream().anyMatch(c -> c.startsWith("getPartitions")), + "the existing-partition metadata must be fetched live; calls=" + client.calls); + } + + @Test + public void planWriteUnpartitionedTableEmitsNoPartitions() { + // An unpartitioned table must emit an empty partition list (and never touch the partition-listing + // client calls) — a spurious partition would confuse the BE commit classification. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().build(); + + THiveTableSink sink = planSink(client, new RecordingConnectorContext(), handle()); + + Assertions.assertTrue(sink.getPartitions().isEmpty()); + Assertions.assertFalse(client.calls.contains("listPartitionNames"), + "an unpartitioned table must not list partitions; calls=" + client.calls); + } + + // ───────────────────────────── hadoop config ───────────────────────────── + + @Test + public void planWriteHadoopConfigFromStorageProperties() { + // BE's S3 sink reads ONLY the AWS_* canonical creds; they are sourced from the typed fe-filesystem + // StorageProperties (the same source the scan path uses). A dropped cred yields a 403 on a private + // bucket. + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().build(); + RecordingConnectorContext ctx = new RecordingConnectorContext(); + ctx.storageProperties = Collections.singletonList( + fakeBackendStorage(Collections.singletonMap("AWS_ACCESS_KEY", "AK123"))); + + Assertions.assertEquals("AK123", + planSink(client, ctx, handle()).getHadoopConfig().get("AWS_ACCESS_KEY")); + } + + // ───────────────────────────── no transaction ───────────────────────────── + + @Test + public void planWriteFailsLoudWithoutTransaction() { + // planWrite requires the executor-bound connector transaction; without it the write must fail loud + // (the cutover wires the transaction onto the session). + RecordingHmsClient client = new RecordingHmsClient(); + client.table = tableBuilder().build(); + HiveWritePlanProvider provider = providerFor(client, new RecordingConnectorContext()); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> provider.planWrite(new WriteSession(null, Collections.emptyMap()), handle())); + Assertions.assertTrue(ex.getMessage().contains("active connector transaction"), + "expected the missing-transaction message, got: " + ex.getMessage()); + } + + // ───────────────────────────── test doubles ───────────────────────────── + + /** A fe-filesystem {@link StorageProperties} whose {@code toBackendProperties().toMap()} returns the given + * BE-canonical map — mirrors how a real object-store binding hands BE creds to the connector. Adapted from + * the iceberg write test. */ + private static StorageProperties fakeBackendStorage(Map beMap) { + BackendStorageProperties backend = new BackendStorageProperties() { + @Override + public BackendStorageKind backendKind() { + return BackendStorageKind.S3_COMPATIBLE; + } + + @Override + public Map toMap() { + return beMap; + } + }; + return new StorageProperties() { + @Override + public String providerName() { + return "fake"; + } + + @Override + public StorageKind kind() { + return StorageKind.OBJECT_STORAGE; + } + + @Override + public FileSystemType type() { + return FileSystemType.S3; + } + + @Override + public Map rawProperties() { + return Collections.emptyMap(); + } + + @Override + public Map matchedProperties() { + return Collections.emptyMap(); + } + + @Override + public Optional toBackendProperties() { + return Optional.of(backend); + } + }; + } + + /** A bound write request wrapping a {@link HiveTableHandle}; mirrors the engine's PluginDrivenWriteHandle. */ + private static final class WriteHandle implements ConnectorWriteHandle { + private final ConnectorTableHandle tableHandle; + private boolean overwrite; + + WriteHandle(ConnectorTableHandle tableHandle) { + this.tableHandle = tableHandle; + } + + WriteHandle overwrite(boolean v) { + this.overwrite = v; + return this; + } + + @Override + public ConnectorTableHandle getTableHandle() { + return tableHandle; + } + + @Override + public List getColumns() { + return Collections.emptyList(); + } + + @Override + public boolean isOverwrite() { + return overwrite; + } + + @Override + public Map getWriteContext() { + return Collections.emptyMap(); + } + } + + /** A session carrying the bound connector transaction and the session-variable overrides. */ + private static final class WriteSession implements ConnectorSession { + private final ConnectorTransaction txn; + private final Map sessionProperties; + + WriteSession(ConnectorTransaction txn, Map sessionProperties) { + this.txn = txn; + this.sessionProperties = sessionProperties; + } + + @Override + public Optional getCurrentTransaction() { + return Optional.ofNullable(txn); + } + + @Override + public Map getSessionProperties() { + return sessionProperties; + } + + @Override + public String getQueryId() { + return "q"; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return 0; + } + + @Override + public String getCatalogName() { + return "test"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + } + + /** Recording {@link HmsClient} fake returning canned table/partition metadata and recording the calls the + * provider (and the transaction's begin-guard) make. */ + private static final class RecordingHmsClient implements HmsClient { + private final List calls = new ArrayList<>(); + private HmsTableInfo table; + private List partitionNames = Collections.emptyList(); + private List partitions = Collections.emptyList(); + + @Override + public List listDatabases() { + return Collections.emptyList(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException("getDatabase"); + } + + @Override + public List listTables(String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return table != null; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + calls.add("getTable:" + dbName + "." + tableName); + if (table == null) { + throw new UnsupportedOperationException("no canned table"); + } + return table; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + calls.add("listPartitionNames"); + return partitionNames; + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + calls.add("getPartitions:" + partNames); + return partitions; + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException("getPartition"); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWriteUtilsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWriteUtilsTest.java new file mode 100644 index 00000000000000..e4633e6e032fbe --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveWriteUtilsTest.java @@ -0,0 +1,165 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.thrift.THivePartitionUpdate; +import org.apache.doris.thrift.TS3MPUPendingUpload; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +/** + * Unit tests for {@link HiveWriteUtils}. These pin the pure write-path helpers that the connector + * write transaction relies on: staging-directory containment/equality checks and the partition + * update accumulator merge. A behavior change in any of these silently corrupts commit accounting + * or staging cleanup, so the tests encode the exact contract, not just smoke coverage. + */ +public class HiveWriteUtilsTest { + + private static THivePartitionUpdate update(String name, long fileSize, long rowCount, String... fileNames) { + return new THivePartitionUpdate() + .setName(name) + .setFileSize(fileSize) + .setRowCount(rowCount) + .setFileNames(new ArrayList<>(Arrays.asList(fileNames))); + } + + @Test + public void mergePartitionsSumsAndConcatsSameName() { + THivePartitionUpdate first = update("p=1", 100L, 10L, "f1", "f2"); + THivePartitionUpdate second = update("p=1", 55L, 5L, "f3"); + + List merged = HiveWriteUtils.mergePartitions(Arrays.asList(first, second)); + + Assertions.assertEquals(1, merged.size()); + THivePartitionUpdate m = merged.get(0); + Assertions.assertEquals("p=1", m.getName()); + Assertions.assertEquals(155L, m.getFileSize(), "file sizes must be summed"); + Assertions.assertEquals(15L, m.getRowCount(), "row counts must be summed"); + Assertions.assertEquals(Arrays.asList("f1", "f2", "f3"), m.getFileNames(), "file names must be concatenated"); + } + + @Test + public void mergePartitionsKeepsDistinctNames() { + List merged = HiveWriteUtils.mergePartitions(Arrays.asList( + update("p=1", 10L, 1L, "a"), + update("p=2", 20L, 2L, "b"), + update("p=1", 30L, 3L, "c"))); + + Map byName = merged.stream() + .collect(Collectors.toMap(THivePartitionUpdate::getName, u -> u)); + Assertions.assertEquals(2, byName.size()); + Assertions.assertEquals(40L, byName.get("p=1").getFileSize()); + Assertions.assertEquals(4L, byName.get("p=1").getRowCount()); + Assertions.assertEquals(20L, byName.get("p=2").getFileSize()); + } + + @Test + public void mergePartitionsConcatsPendingUploads() { + THivePartitionUpdate first = update("p=1", 0L, 0L, "a"); + first.setS3MpuPendingUploads(new ArrayList<>(Arrays.asList(new TS3MPUPendingUpload()))); + THivePartitionUpdate second = update("p=1", 0L, 0L, "b"); + second.setS3MpuPendingUploads(new ArrayList<>(Arrays.asList( + new TS3MPUPendingUpload(), new TS3MPUPendingUpload()))); + + List merged = HiveWriteUtils.mergePartitions(Arrays.asList(first, second)); + + Assertions.assertEquals(1, merged.size()); + Assertions.assertEquals(3, merged.get(0).getS3MpuPendingUploads().size(), + "pending MPU uploads across BE reports for one partition must be aggregated"); + } + + @Test + public void mergePartitionsToleratesNullPendingUploads() { + // Neither update carries pending uploads; the merge must not NPE and just sums files. + List merged = HiveWriteUtils.mergePartitions(Arrays.asList( + update("p=1", 1L, 1L, "a"), + update("p=1", 2L, 2L, "b"))); + Assertions.assertEquals(1, merged.size()); + Assertions.assertEquals(3L, merged.get(0).getFileSize()); + } + + @Test + public void isSubDirectoryHappyPath() { + Assertions.assertTrue(HiveWriteUtils.isSubDirectory("/warehouse/table", "/warehouse/table/p=1")); + Assertions.assertTrue(HiveWriteUtils.isSubDirectory( + "/warehouse/table", "/warehouse/table/.doris_staging/user/uuid")); + } + + @Test + public void isSubDirectoryRejectsEqualAndSiblingAndPrefix() { + // Equal paths are not a strict subdirectory. + Assertions.assertFalse(HiveWriteUtils.isSubDirectory("/warehouse/table", "/warehouse/table")); + // A shared textual prefix without a path boundary is not containment. + Assertions.assertFalse(HiveWriteUtils.isSubDirectory("/warehouse/table", "/warehouse/table2")); + // Unrelated path. + Assertions.assertFalse(HiveWriteUtils.isSubDirectory("/warehouse/table", "/other/x")); + } + + @Test + public void isSubDirectoryRejectsDifferentFileSystem() { + // Same path suffix but different authority (namenode) => different file system. + Assertions.assertFalse(HiveWriteUtils.isSubDirectory("hdfs://nn1/w/t", "hdfs://nn2/w/t/p=1")); + // Different scheme. + Assertions.assertFalse(HiveWriteUtils.isSubDirectory("s3://bucket/w/t", "hdfs://nn/w/t/p=1")); + } + + @Test + public void isSubDirectoryNullSafe() { + Assertions.assertFalse(HiveWriteUtils.isSubDirectory(null, "/a/b")); + Assertions.assertFalse(HiveWriteUtils.isSubDirectory("/a", null)); + } + + @Test + public void getImmediateChildPathReturnsFirstLevel() { + Assertions.assertEquals("/warehouse/table/.doris_staging", + HiveWriteUtils.getImmediateChildPath( + "/warehouse/table", "/warehouse/table/.doris_staging/user/uuid")); + // Direct child returns the child itself. + Assertions.assertEquals("/warehouse/table/p=1", + HiveWriteUtils.getImmediateChildPath("/warehouse/table", "/warehouse/table/p=1")); + } + + @Test + public void getImmediateChildPathReturnsNullWhenNotChild() { + Assertions.assertNull(HiveWriteUtils.getImmediateChildPath("/warehouse/table", "/other/x")); + Assertions.assertNull(HiveWriteUtils.getImmediateChildPath("/warehouse/table", "/warehouse/table")); + } + + @Test + public void pathsEqualNormalizesTrailingSlashAndFileSystem() { + Assertions.assertTrue(HiveWriteUtils.pathsEqual("/a/b", "/a/b/")); + Assertions.assertTrue(HiveWriteUtils.pathsEqual("hdfs://nn/a/b", "hdfs://nn/a/b")); + Assertions.assertFalse(HiveWriteUtils.pathsEqual("/a/b", "/a/c")); + // Different namenode => not equal even with identical path. + Assertions.assertFalse(HiveWriteUtils.pathsEqual("hdfs://nn1/a/b", "hdfs://nn2/a/b")); + } + + @Test + public void pathsEqualNullSafe() { + Assertions.assertTrue(HiveWriteUtils.pathsEqual(null, null)); + Assertions.assertFalse(HiveWriteUtils.pathsEqual(null, "/a")); + Assertions.assertFalse(HiveWriteUtils.pathsEqual("/a", null)); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/NameMappingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/NameMappingTest.java new file mode 100644 index 00000000000000..026c1bbb761b8a --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/NameMappingTest.java @@ -0,0 +1,67 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Unit tests for {@link NameMapping}. The connector write transaction keys its per-table and + * per-partition action maps by {@code NameMapping}, so value-based {@code equals}/{@code hashCode} + * is load-bearing: a broken key would split one table's actions across map buckets and drop writes. + * These tests pin that contract, not just field storage. + */ +public class NameMappingTest { + + @Test + public void equalValuesAreEqualAndHashAlike() { + NameMapping a = new NameMapping(1L, "localDb", "localTbl", "remoteDb", "remoteTbl"); + NameMapping b = new NameMapping(1L, "localDb", "localTbl", "remoteDb", "remoteTbl"); + Assertions.assertEquals(a, b); + Assertions.assertEquals(a.hashCode(), b.hashCode()); + } + + @Test + public void anyFieldDifferenceBreaksEquality() { + NameMapping base = new NameMapping(1L, "ld", "lt", "rd", "rt"); + Assertions.assertNotEquals(base, new NameMapping(2L, "ld", "lt", "rd", "rt")); + Assertions.assertNotEquals(base, new NameMapping(1L, "LD", "lt", "rd", "rt")); + Assertions.assertNotEquals(base, new NameMapping(1L, "ld", "LT", "rd", "rt")); + Assertions.assertNotEquals(base, new NameMapping(1L, "ld", "lt", "RD", "rt")); + Assertions.assertNotEquals(base, new NameMapping(1L, "ld", "lt", "rd", "RT")); + } + + @Test + public void usableAsMapKeyByValue() { + Map actions = new HashMap<>(); + actions.put(new NameMapping(7L, "ld", "lt", "rd", "rt"), "action"); + // A distinct instance with identical values must resolve to the same bucket. + Assertions.assertEquals("action", actions.get(new NameMapping(7L, "ld", "lt", "rd", "rt"))); + Assertions.assertNull(actions.get(new NameMapping(7L, "ld", "lt", "rd", "other"))); + } + + @Test + public void fullNamesJoinDbAndTable() { + NameMapping m = new NameMapping(1L, "localDb", "localTbl", "remoteDb", "remoteTbl"); + Assertions.assertEquals("localDb.localTbl", m.getFullLocalName()); + Assertions.assertEquals("remoteDb.remoteTbl", m.getFullRemoteName()); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/RecordingConnectorContext.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/RecordingConnectorContext.java new file mode 100644 index 00000000000000..2c25049a2a9d69 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/RecordingConnectorContext.java @@ -0,0 +1,98 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.spi.ConnectorBrokerAddress; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.properties.StorageProperties; +import org.apache.doris.thrift.TFileType; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.Callable; + +/** + * Hand-written {@link ConnectorContext} test double (no Mockito) for the hive write-plan tests, adapted from + * the iceberg connector's {@code RecordingConnectorContext}. Resolves the BE file type / normalized write + * path / broker addresses / static storage creds that {@link HiveWritePlanProvider} consults, and passes + * {@link #executeAuthenticated} through so the table load runs inside the (fake) auth context. + */ +final class RecordingConnectorContext implements ConnectorContext { + + int authCount; + + /** BE file type the fake returns from {@link #getBackendFileType} (drives in-place vs staging write path). */ + TFileType backendFileType = TFileType.FILE_S3; + + /** Raw URIs the connector routed through {@link #normalizeStorageUri}. */ + final List normalizedUris = new ArrayList<>(); + + /** Static storage properties the fake returns from {@link #getStorageProperties()} (BE-canonical creds via + * {@code sp.toBackendProperties().toMap()}); default none. */ + List storageProperties = Collections.emptyList(); + + /** Broker addresses the fake returns from {@link #getBrokerAddresses()}; default none, so a FILE_BROKER + * write fails loud ("No alive broker.") unless a test populates it. */ + List brokerAddresses = Collections.emptyList(); + + @Override + public String getCatalogName() { + return "test"; + } + + @Override + public long getCatalogId() { + return 0; + } + + @Override + public String getBackendFileType(String rawUri, Map rawVendedCredentials) { + return backendFileType.name(); + } + + @Override + public String normalizeStorageUri(String rawUri) { + return normalizeStorageUri(rawUri, null); + } + + @Override + public String normalizeStorageUri(String rawUri, Map rawVendedCredentials) { + normalizedUris.add(rawUri); + // Canonicalize the scheme the way DefaultConnectorContext does for native object-store paths + // (oss/cos/obs/s3a -> s3), so a test can prove the connector routes the location through this seam. + return rawUri == null ? null : rawUri.replaceFirst("^(oss|cos|obs|s3a)://", "s3://"); + } + + @Override + public List getStorageProperties() { + return storageProperties; + } + + @Override + public List getBrokerAddresses() { + return brokerAddresses; + } + + @Override + public T executeAuthenticated(Callable task) throws Exception { + authCount++; + return task.call(); + } +} diff --git a/fe/fe-connector/fe-connector-hms/pom.xml b/fe/fe-connector/fe-connector-hms/pom.xml index 553891aab14476..013a6945cef52c 100644 --- a/fe/fe-connector/fe-connector-hms/pom.xml +++ b/fe/fe-connector/fe-connector-hms/pom.xml @@ -93,6 +93,29 @@ under the License. junit-jupiter test + + + + org.apache.hadoop + hadoop-mapreduce-client-core + test + + + + + commons-lang + commons-lang + test + diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsAcidConstants.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsAcidConstants.java new file mode 100644 index 00000000000000..eac80de20042a3 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsAcidConstants.java @@ -0,0 +1,39 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +/** + * Hadoop-configuration keys that carry the ACID snapshot (valid transaction list + valid write-id + * list) from FE to BE for transactional Hive reads. + * + *

These are a fixed Hive/BE contract: the same literal strings fe-core's {@code AcidUtil} + * produces (via {@link HmsClient#getValidWriteIds}) and consumes. Defined once here so the write-id + * producer ({@link ThriftHmsClient}) and the future read-side ACID descent share one source of + * truth.

+ */ +public final class HmsAcidConstants { + + /** Serialized {@code ValidTxnList}. */ + public static final String VALID_TXNS_KEY = "hive.txn.valid.txns"; + + /** Serialized {@code ValidWriteIdList}. */ + public static final String VALID_WRITEIDS_KEY = "hive.txn.valid.writeids"; + + private HmsAcidConstants() { + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java index af728e9c433eeb..f3e16e10acae62 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java @@ -20,6 +20,7 @@ import java.io.Closeable; import java.util.List; import java.util.Map; +import java.util.function.Function; /** * Clean interface for Hive MetaStore client operations. @@ -33,8 +34,8 @@ *
    *
  • Phase 1: Read-only metadata (list, get, exists)
    • *
  • Phase 2: Partition operations
    • - *
  • Phase 3: DDL / write operations (future)
    • - *
  • Phase 4: ACID + events (future)
    • + *
  • Phase 3: DDL / write operations
    • + *
  • Phase 4: ACID transactions
    • *
*/ public interface HmsClient extends Closeable { @@ -195,4 +196,138 @@ default void dropTable(String dbName, String tableName) { default void truncateTable(String dbName, String tableName, List partitions) { throw new UnsupportedOperationException("truncateTable is not supported by this client"); } + + /** + * Add partitions to a table, stamping each partition's basic statistics onto its parameters. + * Ports the legacy {@code HMSTransaction} add-partition commit; the implementation batches large + * lists internally. + * + * @param dbName database name + * @param tableName table name + * @param partitions partitions to create, each with its data-layout and statistics + * @throws HmsClientException if the operation fails + */ + default void addPartitions(String dbName, String tableName, + List partitions) { + throw new UnsupportedOperationException("addPartitions is not supported by this client"); + } + + /** + * Read-modify-write a table's basic statistics parameters (numRows / totalSize / numFiles). + * + * @param dbName database name + * @param tableName table name + * @param update receives the table's current statistics and returns the new statistics + * @throws HmsClientException if the operation fails + */ + default void updateTableStatistics(String dbName, String tableName, + Function update) { + throw new UnsupportedOperationException( + "updateTableStatistics is not supported by this client"); + } + + /** + * Read-modify-write a single partition's basic statistics parameters. + * + * @param dbName database name + * @param tableName table name + * @param partitionName partition name (e.g. "dt=2024-01-01") + * @param update receives the partition's current statistics and returns the new statistics + * @throws HmsClientException if the operation fails + */ + default void updatePartitionStatistics(String dbName, String tableName, String partitionName, + Function update) { + throw new UnsupportedOperationException( + "updatePartitionStatistics is not supported by this client"); + } + + /** + * Drop a partition by its values. + * + * @param dbName database name + * @param tableName table name + * @param partitionValues partition column values in declaration order + * @param deleteData whether to also delete the partition's data files + * @return true if a partition was dropped + * @throws HmsClientException if the operation fails + */ + default boolean dropPartition(String dbName, String tableName, + List partitionValues, boolean deleteData) { + throw new UnsupportedOperationException("dropPartition is not supported by this client"); + } + + /** + * Not-found-tolerant probe for whether a partition exists (used to downgrade a NEW-partition + * write to an APPEND when the Doris cache missed a partition that already exists in HMS). + * + * @param dbName database name + * @param tableName table name + * @param partitionValues partition column values in declaration order + * @return true if the partition exists in the metastore + * @throws HmsClientException if the operation fails for a reason other than not-found + */ + default boolean partitionExists(String dbName, String tableName, + List partitionValues) { + throw new UnsupportedOperationException("partitionExists is not supported by this client"); + } + + // ========== Phase 4: ACID transactions ========== + // + // Read-side ACID primitives for transactional Hive tables. Like the Phase 3 block, they default + // to throwing so read-only / non-transactional clients need not implement them. + + /** + * Open a Hive ACID transaction. + * + * @param user the user opening the transaction + * @return the new transaction id + * @throws HmsClientException if the operation fails + */ + default long openTxn(String user) { + throw new UnsupportedOperationException("openTxn is not supported by this client"); + } + + /** + * Commit a Hive ACID transaction (also releases the transaction's locks). + * + * @param txnId the transaction id + * @throws HmsClientException if the operation fails + */ + default void commitTxn(long txnId) { + throw new UnsupportedOperationException("commitTxn is not supported by this client"); + } + + /** + * Get the valid-transaction / valid-write-id snapshot for a transactional table, as the two + * Hadoop-configuration entries ({@link HmsAcidConstants}) that BE consumes for ACID reads. On a + * metastore-incompatibility error the implementation degrades to a max watermark rather than + * failing the read. + * + * @param fullTableName fully-qualified "db.table" + * @param currentTransactionId the reader's open transaction id + * @return map with {@link HmsAcidConstants#VALID_TXNS_KEY} and + * {@link HmsAcidConstants#VALID_WRITEIDS_KEY} + * @throws HmsClientException if the operation fails + */ + default Map getValidWriteIds(String fullTableName, long currentTransactionId) { + throw new UnsupportedOperationException("getValidWriteIds is not supported by this client"); + } + + /** + * Acquire a shared (read) lock over a table and, if given, specific partitions, polling until the + * lock is granted or the timeout elapses. + * + * @param queryId the query id (lock owner) + * @param txnId the transaction id + * @param user the requesting user + * @param dbName database name + * @param tableName table name + * @param partitionNames partitions to lock; empty locks the whole table + * @param timeoutMs maximum time to wait for the lock, in milliseconds + * @throws HmsClientException if the lock cannot be acquired within the timeout + */ + default void acquireSharedLock(String queryId, long txnId, String user, String dbName, + String tableName, List partitionNames, long timeoutMs) { + throw new UnsupportedOperationException("acquireSharedLock is not supported by this client"); + } } diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCommonStatistics.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCommonStatistics.java new file mode 100644 index 00000000000000..6e2a53de3c261d --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsCommonStatistics.java @@ -0,0 +1,86 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +/** + * Basic ("common") statistics for a Hive table or partition: row count, file count, total file bytes. + * + *

SPI-clean port of fe-core's {@code datasource.statistics.CommonStatistics} (which is JDK-only in + * the original). Carried by {@link HmsPartitionStatistics} and consumed by the metastore + * statistics-update primitives on {@link HmsClient}.

+ */ +public final class HmsCommonStatistics { + + /** Operator for combining two statistic values. */ + public enum ReduceOperator { + ADD, + SUBTRACT, + MIN, + MAX + } + + public static final HmsCommonStatistics EMPTY = new HmsCommonStatistics(0L, 0L, 0L); + + private final long rowCount; + private final long fileCount; + private final long totalFileBytes; + + public HmsCommonStatistics(long rowCount, long fileCount, long totalFileBytes) { + this.rowCount = rowCount; + this.fileCount = fileCount; + this.totalFileBytes = totalFileBytes; + } + + public long getRowCount() { + return rowCount; + } + + public long getFileCount() { + return fileCount; + } + + public long getTotalFileBytes() { + return totalFileBytes; + } + + public static HmsCommonStatistics reduce(HmsCommonStatistics current, HmsCommonStatistics update, + ReduceOperator operator) { + return new HmsCommonStatistics( + reduce(current.getRowCount(), update.getRowCount(), operator), + reduce(current.getFileCount(), update.getFileCount(), operator), + reduce(current.getTotalFileBytes(), update.getTotalFileBytes(), operator)); + } + + public static long reduce(long current, long update, ReduceOperator operator) { + if (current >= 0 && update >= 0) { + switch (operator) { + case ADD: + return current + update; + case SUBTRACT: + return current - update; + case MAX: + return Math.max(current, update); + case MIN: + return Math.min(current, update); + default: + throw new IllegalArgumentException("Unexpected operator: " + operator); + } + } + return 0; + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionStatistics.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionStatistics.java new file mode 100644 index 00000000000000..a06a3eabbd2131 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionStatistics.java @@ -0,0 +1,71 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +/** + * Statistics for a Hive partition/table write. SPI-clean port of fe-core's + * {@code datasource.hive.HivePartitionStatistics}, reduced to the basic (common) statistics. + * + *

The legacy type also carried a {@code Map} column-statistics map, + * but the INSERT/commit write path never populates it (it is always empty, and legacy {@code merge} + * does not actually merge column stats), and {@code HiveColumnStatistics} is a fe-core type. It is + * therefore dropped from the plugin DTO — only the row/file/byte common statistics travel.

+ */ +public final class HmsPartitionStatistics { + + public static final HmsPartitionStatistics EMPTY = + new HmsPartitionStatistics(HmsCommonStatistics.EMPTY); + + private final HmsCommonStatistics commonStatistics; + + public HmsPartitionStatistics(HmsCommonStatistics commonStatistics) { + this.commonStatistics = commonStatistics; + } + + public HmsCommonStatistics getCommonStatistics() { + return commonStatistics; + } + + public static HmsPartitionStatistics fromCommonStatistics(long rowCount, long fileCount, + long totalFileBytes) { + return new HmsPartitionStatistics( + new HmsCommonStatistics(rowCount, fileCount, totalFileBytes)); + } + + public static HmsPartitionStatistics merge(HmsPartitionStatistics current, + HmsPartitionStatistics update) { + if (current.getCommonStatistics().getRowCount() <= 0) { + return update; + } else if (update.getCommonStatistics().getRowCount() <= 0) { + return current; + } + return new HmsPartitionStatistics(HmsCommonStatistics.reduce( + current.getCommonStatistics(), update.getCommonStatistics(), + HmsCommonStatistics.ReduceOperator.ADD)); + } + + public static HmsPartitionStatistics reduce(HmsPartitionStatistics first, + HmsPartitionStatistics second, HmsCommonStatistics.ReduceOperator operator) { + HmsCommonStatistics left = first.getCommonStatistics(); + HmsCommonStatistics right = second.getCommonStatistics(); + return HmsPartitionStatistics.fromCommonStatistics( + HmsCommonStatistics.reduce(left.getRowCount(), right.getRowCount(), operator), + HmsCommonStatistics.reduce(left.getFileCount(), right.getFileCount(), operator), + HmsCommonStatistics.reduce(left.getTotalFileBytes(), right.getTotalFileBytes(), operator)); + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionWithStatistics.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionWithStatistics.java new file mode 100644 index 00000000000000..fcce1fea9d34ab --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsPartitionWithStatistics.java @@ -0,0 +1,175 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.hadoop.hive.metastore.api.FieldSchema; + +import java.util.Collections; +import java.util.List; +import java.util.Map; + +/** + * A Hive partition to be created, bundled with the statistics to stamp on it. + * + *

SPI-clean port of fe-core's {@code datasource.hive.HivePartitionWithStatistics} plus the + * storage-descriptor fields of {@code HivePartition} (a fe-core type), flattened so the DTO depends + * only on connector-api / JDK / hive-metastore-api types. {@link HmsWriteConverter#toHivePartitions} + * turns a list of these into Hive metastore {@code Partition} objects for + * {@link HmsClient#addPartitions}.

+ */ +public final class HmsPartitionWithStatistics { + + private final String name; + private final List partitionValues; + private final String location; + private final List columns; + private final String inputFormat; + private final String outputFormat; + private final String serde; + private final Map parameters; + private final HmsPartitionStatistics statistics; + + private HmsPartitionWithStatistics(Builder builder) { + this.name = builder.name; + this.partitionValues = builder.partitionValues == null + ? Collections.emptyList() + : Collections.unmodifiableList(builder.partitionValues); + this.location = builder.location; + this.columns = builder.columns == null + ? Collections.emptyList() + : Collections.unmodifiableList(builder.columns); + this.inputFormat = builder.inputFormat; + this.outputFormat = builder.outputFormat; + this.serde = builder.serde; + this.parameters = builder.parameters == null + ? Collections.emptyMap() + : Collections.unmodifiableMap(builder.parameters); + this.statistics = builder.statistics == null + ? HmsPartitionStatistics.EMPTY + : builder.statistics; + } + + /** Partition name (e.g. "dt=2024-01-01"). Used by the committer for tracking/logging. */ + public String getName() { + return name; + } + + /** Partition column values in declaration order. */ + public List getPartitionValues() { + return partitionValues; + } + + public String getLocation() { + return location; + } + + public List getColumns() { + return columns; + } + + public String getInputFormat() { + return inputFormat; + } + + public String getOutputFormat() { + return outputFormat; + } + + public String getSerde() { + return serde; + } + + public Map getParameters() { + return parameters; + } + + public HmsPartitionStatistics getStatistics() { + return statistics; + } + + public static Builder builder() { + return new Builder(); + } + + /** + * Builder for HmsPartitionWithStatistics. + */ + public static final class Builder { + private String name; + private List partitionValues; + private String location; + private List columns; + private String inputFormat; + private String outputFormat; + private String serde; + private Map parameters; + private HmsPartitionStatistics statistics; + + private Builder() { + } + + public Builder name(String val) { + this.name = val; + return this; + } + + public Builder partitionValues(List val) { + this.partitionValues = val; + return this; + } + + public Builder location(String val) { + this.location = val; + return this; + } + + public Builder columns(List val) { + this.columns = val; + return this; + } + + public Builder inputFormat(String val) { + this.inputFormat = val; + return this; + } + + public Builder outputFormat(String val) { + this.outputFormat = val; + return this; + } + + public Builder serde(String val) { + this.serde = val; + return this; + } + + public Builder parameters(Map val) { + this.parameters = val; + return this; + } + + public Builder statistics(HmsPartitionStatistics val) { + this.statistics = val; + return this; + } + + public HmsPartitionWithStatistics build() { + return new HmsPartitionWithStatistics(this); + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java index e103b3c3ebae68..7989dda3e0dba5 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java @@ -41,6 +41,8 @@ public final class HmsTableInfo { private final String serializationLib; private final List columns; private final List partitionKeys; + private final List bucketCols; + private final int numBuckets; private final Map parameters; private final Map sdParameters; @@ -58,6 +60,10 @@ private HmsTableInfo(Builder builder) { this.partitionKeys = builder.partitionKeys == null ? Collections.emptyList() : Collections.unmodifiableList(builder.partitionKeys); + this.bucketCols = builder.bucketCols == null + ? Collections.emptyList() + : Collections.unmodifiableList(builder.bucketCols); + this.numBuckets = builder.numBuckets; this.parameters = builder.parameters == null ? Collections.emptyMap() : Collections.unmodifiableMap(builder.parameters); @@ -105,6 +111,16 @@ public List getPartitionKeys() { return partitionKeys; } + /** Bucketing columns (empty when the table is not bucketed). */ + public List getBucketCols() { + return bucketCols; + } + + /** Bucket count (0 when the table is not bucketed). */ + public int getNumBuckets() { + return numBuckets; + } + public Map getParameters() { return parameters; } @@ -137,6 +153,8 @@ public static final class Builder { private String serializationLib; private List columns; private List partitionKeys; + private List bucketCols; + private int numBuckets; private Map parameters; private Map sdParameters; @@ -188,6 +206,16 @@ public Builder partitionKeys(List val) { return this; } + public Builder bucketCols(List val) { + this.bucketCols = val; + return this; + } + + public Builder numBuckets(int val) { + this.numBuckets = val; + return this; + } + public Builder parameters(Map val) { this.parameters = val; return this; diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java index 87e924c442b6d2..f9ef32db40b07b 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsWriteConverter.java @@ -19,8 +19,10 @@ import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.hadoop.hive.common.StatsSetupConst; import org.apache.hadoop.hive.metastore.api.Database; import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Partition; import org.apache.hadoop.hive.metastore.api.PrincipalType; import org.apache.hadoop.hive.metastore.api.SerDeInfo; import org.apache.hadoop.hive.metastore.api.StorageDescriptor; @@ -142,6 +144,80 @@ public static Database toHiveDatabase(HmsCreateDatabaseRequest req) { return database; } + /** + * Build Hive metastore {@link Partition} objects for a set of partitions to add, stamping each + * partition's basic statistics onto its parameters. Faithful port of legacy + * {@code HiveUtil.toMetastoreApiPartition} + {@code makeStorageDescriptorFromHivePartition}, with + * the fe-core {@code HivePartition} fields carried directly on {@link HmsPartitionWithStatistics}. + */ + public static List toHivePartitions(String dbName, String tableName, + List partitions) { + List result = new ArrayList<>(partitions.size()); + for (HmsPartitionWithStatistics partition : partitions) { + result.add(toHivePartition(dbName, tableName, partition)); + } + return result; + } + + private static Partition toHivePartition(String dbName, String tableName, + HmsPartitionWithStatistics partitionWithStatistics) { + Partition partition = new Partition(); + partition.setDbName(dbName); + partition.setTableName(tableName); + partition.setValues(partitionWithStatistics.getPartitionValues()); + partition.setSd(toHivePartitionStorageDesc(tableName, partitionWithStatistics)); + partition.setParameters(toStatisticsParameters( + partitionWithStatistics.getParameters(), + partitionWithStatistics.getStatistics().getCommonStatistics())); + return partition; + } + + private static StorageDescriptor toHivePartitionStorageDesc(String tableName, + HmsPartitionWithStatistics partition) { + SerDeInfo serdeInfo = new SerDeInfo(); + serdeInfo.setName(tableName); + serdeInfo.setSerializationLib(partition.getSerde()); + + StorageDescriptor sd = new StorageDescriptor(); + sd.setLocation(emptyToNull(partition.getLocation())); + sd.setCols(partition.getColumns()); + sd.setSerdeInfo(serdeInfo); + sd.setInputFormat(partition.getInputFormat()); + sd.setOutputFormat(partition.getOutputFormat()); + sd.setParameters(new HashMap<>()); + return sd; + } + + /** + * Merge basic statistics into a copy of the given parameters. Faithful port of legacy + * {@code HiveUtil.updateStatisticsParameters} (numRows / totalSize / numFiles, plus the CDH + * stats-task workaround flag). + */ + public static Map toStatisticsParameters(Map parameters, + HmsCommonStatistics statistics) { + Map result = new HashMap<>(parameters); + result.put(StatsSetupConst.NUM_FILES, String.valueOf(statistics.getFileCount())); + result.put(StatsSetupConst.ROW_COUNT, String.valueOf(statistics.getRowCount())); + result.put(StatsSetupConst.TOTAL_SIZE, String.valueOf(statistics.getTotalFileBytes())); + // CDH 5.16 metastore ignores stats unless STATS_GENERATED_VIA_STATS_TASK is set. + if (!parameters.containsKey("STATS_GENERATED_VIA_STATS_TASK")) { + result.put("STATS_GENERATED_VIA_STATS_TASK", + "workaround for potential lack of HIVE-12730"); + } + return result; + } + + /** + * Read basic statistics out of table/partition parameters. Faithful port of legacy + * {@code HiveUtil.toHivePartitionStatistics}. + */ + public static HmsPartitionStatistics toPartitionStatistics(Map params) { + long rowCount = Long.parseLong(params.getOrDefault(StatsSetupConst.ROW_COUNT, "-1")); + long totalSize = Long.parseLong(params.getOrDefault(StatsSetupConst.TOTAL_SIZE, "-1")); + long numFiles = Long.parseLong(params.getOrDefault(StatsSetupConst.NUM_FILES, "-1")); + return HmsPartitionStatistics.fromCommonStatistics(rowCount, numFiles, totalSize); + } + private static void setCompressType(HmsCreateTableRequest req, Map props) { String fileFormat = req.getFileFormat(); String compression = props.get(COMPRESSION_KEY); @@ -257,6 +333,10 @@ private static boolean isEmpty(String s) { return s == null || s.isEmpty(); } + private static String emptyToNull(String s) { + return isEmpty(s) ? null : s; + } + private static boolean isNotEmpty(String s) { return !isEmpty(s); } diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 00dd40d65b98e2..55e0eaa031de5d 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -25,29 +25,46 @@ import org.apache.commons.pool2.impl.DefaultPooledObject; import org.apache.commons.pool2.impl.GenericObjectPool; import org.apache.commons.pool2.impl.GenericObjectPoolConfig; +import org.apache.hadoop.hive.common.ValidReadTxnList; +import org.apache.hadoop.hive.common.ValidReaderWriteIdList; +import org.apache.hadoop.hive.common.ValidTxnList; +import org.apache.hadoop.hive.common.ValidTxnWriteIdList; +import org.apache.hadoop.hive.common.ValidWriteIdList; import org.apache.hadoop.hive.conf.HiveConf; import org.apache.hadoop.hive.metastore.HiveMetaHookLoader; import org.apache.hadoop.hive.metastore.HiveMetaStoreClient; import org.apache.hadoop.hive.metastore.IMetaStoreClient; +import org.apache.hadoop.hive.metastore.LockComponentBuilder; +import org.apache.hadoop.hive.metastore.LockRequestBuilder; import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient; +import org.apache.hadoop.hive.metastore.api.DataOperationType; import org.apache.hadoop.hive.metastore.api.Database; import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.LockComponent; +import org.apache.hadoop.hive.metastore.api.LockRequest; +import org.apache.hadoop.hive.metastore.api.LockResponse; +import org.apache.hadoop.hive.metastore.api.LockState; import org.apache.hadoop.hive.metastore.api.MetaException; +import org.apache.hadoop.hive.metastore.api.NoSuchObjectException; import org.apache.hadoop.hive.metastore.api.Partition; import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint; import org.apache.hadoop.hive.metastore.api.StorageDescriptor; import org.apache.hadoop.hive.metastore.api.Table; +import org.apache.hadoop.hive.metastore.api.TableValidWriteIds; +import org.apache.hadoop.hive.metastore.txn.TxnUtils; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import shade.doris.hive.org.apache.thrift.TApplicationException; import java.io.IOException; import java.util.ArrayList; +import java.util.BitSet; import java.util.Collections; import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.concurrent.Callable; +import java.util.function.Function; import java.util.stream.Collectors; /** @@ -72,6 +89,8 @@ public class ThriftHmsClient implements HmsClient { private static final HiveMetaHookLoader DUMMY_HOOK_LOADER = tbl -> null; private static final int MAX_LIST_PARTITION_NUM = Short.MAX_VALUE; private static final long POOL_BORROW_TIMEOUT_MS = 60_000L; + private static final int ADD_PARTITIONS_BATCH_SIZE = 20; + private static final String TRANSIENT_LAST_DDL_TIME = "transient_lastDdlTime"; private final HiveConf hiveConf; private final GenericObjectPool clientPool; @@ -283,6 +302,194 @@ private static List buildDefaultConstraints(HmsCreateTable return defaults; } + // ========== Phase 3: partition / statistics writes ========== + + @Override + public void addPartitions(String dbName, String tableName, + List partitions) { + execute(client -> { + List hivePartitions = + HmsWriteConverter.toHivePartitions(dbName, tableName, partitions); + for (int i = 0; i < hivePartitions.size(); i += ADD_PARTITIONS_BATCH_SIZE) { + int end = Math.min(i + ADD_PARTITIONS_BATCH_SIZE, hivePartitions.size()); + client.add_partitions(new ArrayList<>(hivePartitions.subList(i, end))); + } + return null; + }); + } + + @Override + public void updateTableStatistics(String dbName, String tableName, + Function update) { + execute(client -> { + Table originTable = client.getTable(dbName, tableName); + Map originParams = originTable.getParameters(); + HmsPartitionStatistics updatedStats = + update.apply(HmsWriteConverter.toPartitionStatistics(originParams)); + Table newTable = originTable.deepCopy(); + Map newParams = HmsWriteConverter.toStatisticsParameters( + originParams, updatedStats.getCommonStatistics()); + newParams.put(TRANSIENT_LAST_DDL_TIME, + String.valueOf(System.currentTimeMillis() / 1000)); + newTable.setParameters(newParams); + client.alter_table(dbName, tableName, newTable); + return null; + }); + } + + @Override + public void updatePartitionStatistics(String dbName, String tableName, String partitionName, + Function update) { + execute(client -> { + List partitions = client.getPartitionsByNames( + dbName, tableName, Collections.singletonList(partitionName)); + if (partitions.size() != 1) { + throw new HmsClientException( + "Metastore returned multiple partitions for name: " + partitionName); + } + Partition originPartition = partitions.get(0); + Map originParams = originPartition.getParameters(); + HmsPartitionStatistics updatedStats = + update.apply(HmsWriteConverter.toPartitionStatistics(originParams)); + Partition modifiedPartition = originPartition.deepCopy(); + Map newParams = HmsWriteConverter.toStatisticsParameters( + originParams, updatedStats.getCommonStatistics()); + newParams.put(TRANSIENT_LAST_DDL_TIME, + String.valueOf(System.currentTimeMillis() / 1000)); + modifiedPartition.setParameters(newParams); + client.alter_partition(dbName, tableName, modifiedPartition); + return null; + }); + } + + @Override + public boolean dropPartition(String dbName, String tableName, + List partitionValues, boolean deleteData) { + return execute(client -> + client.dropPartition(dbName, tableName, partitionValues, deleteData)); + } + + @Override + public boolean partitionExists(String dbName, String tableName, + List partitionValues) { + return execute(client -> { + try { + client.getPartition(dbName, tableName, partitionValues); + return Boolean.TRUE; + } catch (NoSuchObjectException e) { + // Not found: a normal answer, not a client fault — do not taint the pooled client. + return Boolean.FALSE; + } + }); + } + + // ========== Phase 4: ACID transactions ========== + + @Override + public long openTxn(String user) { + return execute(client -> client.openTxn(user)); + } + + @Override + public void commitTxn(long txnId) { + execute(client -> { + client.commitTxn(txnId); + return null; + }); + } + + @Override + public Map getValidWriteIds(String fullTableName, long currentTransactionId) { + Map conf = new HashMap<>(); + try { + return execute(client -> { + // Use the recent snapshot of valid transactions (no currentTxn arg): passing + // currentTransactionId here would break Hive's delta-directory listing if a major + // compaction removed deltas for transactions that were valid when this txn opened. + ValidTxnList validTransactions = client.getValidTxns(); + List tableValidWriteIdsList = client.getValidWriteIds( + Collections.singletonList(fullTableName), validTransactions.toString()); + if (tableValidWriteIdsList.size() != 1) { + throw new HmsClientException("tableValidWriteIdsList's size should be 1"); + } + ValidTxnWriteIdList validTxnWriteIdList = TxnUtils.createValidTxnWriteIdList( + currentTransactionId, tableValidWriteIdsList); + ValidWriteIdList writeIdList = + validTxnWriteIdList.getTableValidWriteIdList(fullTableName); + conf.put(HmsAcidConstants.VALID_TXNS_KEY, validTransactions.writeToString()); + conf.put(HmsAcidConstants.VALID_WRITEIDS_KEY, writeIdList.writeToString()); + return conf; + }); + } catch (Exception e) { + // Older / incompatible metastores may lack these APIs; degrade to a max watermark + // instead of failing the scan (mirrors legacy ThriftHMSCachedClient). + LOG.warn("failed to get valid write ids for {}, transaction {}", + fullTableName, currentTransactionId, e); + ValidTxnList validTransactions = new ValidReadTxnList( + new long[0], new BitSet(), Long.MAX_VALUE, Long.MAX_VALUE); + ValidWriteIdList writeIdList = new ValidReaderWriteIdList( + fullTableName, new long[0], new BitSet(), Long.MAX_VALUE); + conf.put(HmsAcidConstants.VALID_TXNS_KEY, validTransactions.writeToString()); + conf.put(HmsAcidConstants.VALID_WRITEIDS_KEY, writeIdList.writeToString()); + return conf; + } + } + + @Override + public void acquireSharedLock(String queryId, long txnId, String user, String dbName, + String tableName, List partitionNames, long timeoutMs) { + LockRequestBuilder requestBuilder = new LockRequestBuilder(queryId) + .setTransactionId(txnId) + .setUser(user); + for (LockComponent component + : createLockComponentsForRead(dbName, tableName, partitionNames)) { + requestBuilder.addLockComponent(component); + } + LockRequest lockRequest = requestBuilder.build(); + LockResponse response = execute(client -> client.lock(lockRequest)); + long start = System.currentTimeMillis(); + while (response.getState() == LockState.WAITING) { + if (System.currentTimeMillis() - start > timeoutMs) { + throw new HmsClientException("acquire lock timeout for txn " + txnId + + " of query " + queryId + ", timeout(ms): " + timeoutMs); + } + long lockId = response.getLockid(); + response = execute(client -> client.checkLock(lockId)); + } + if (response.getState() != LockState.ACQUIRED) { + throw new HmsClientException( + "failed to acquire lock, lock in state " + response.getState()); + } + } + + private static List createLockComponentsForRead(String dbName, String tableName, + List partitionNames) { + List components = new ArrayList<>( + partitionNames.isEmpty() ? 1 : partitionNames.size()); + if (partitionNames.isEmpty()) { + components.add(createLockComponentForRead(dbName, tableName, null)); + } else { + for (String partitionName : partitionNames) { + components.add(createLockComponentForRead(dbName, tableName, partitionName)); + } + } + return components; + } + + private static LockComponent createLockComponentForRead(String dbName, String tableName, + String partitionName) { + LockComponentBuilder builder = new LockComponentBuilder(); + builder.setShared(); + builder.setOperationType(DataOperationType.SELECT); + builder.setDbName(dbName); + builder.setTableName(tableName); + if (partitionName != null) { + builder.setPartitionName(partitionName); + } + builder.setIsTransactional(true); + return builder.build(); + } + @Override public synchronized void close() throws IOException { if (closed) { @@ -350,7 +557,9 @@ private HmsTableInfo convertTable(Table table) { if (sd != null) { builder.location(sd.getLocation()) .inputFormat(sd.getInputFormat()) - .outputFormat(sd.getOutputFormat()); + .outputFormat(sd.getOutputFormat()) + .bucketCols(sd.getBucketCols()) + .numBuckets(sd.getNumBuckets()); if (sd.getSerdeInfo() != null) { builder.serializationLib( sd.getSerdeInfo().getSerializationLib()); diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java index 65abb48be337b2..f74950af21c3a2 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsWriteConverterTest.java @@ -20,8 +20,10 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorType; +import org.apache.hadoop.hive.common.StatsSetupConst; import org.apache.hadoop.hive.metastore.api.Database; import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Partition; import org.apache.hadoop.hive.metastore.api.PrincipalType; import org.apache.hadoop.hive.metastore.api.StorageDescriptor; import org.apache.hadoop.hive.metastore.api.Table; @@ -209,6 +211,114 @@ public void testToHiveDatabaseNoLocationNoOwner() { Assertions.assertEquals("", db.getDescription()); } + @Test + public void testToHivePartitionsBuildsPartitionWithStatsAndSd() { + // WHY: the add-partition commit path depends on the storage descriptor (location/serde/format) + // and the basic-stats parameters being stamped exactly as HMS expects, or the created + // partition is unreadable or its row/byte counts are wrong. + HmsPartitionWithStatistics part = HmsPartitionWithStatistics.builder() + .name("dt=2024-01-01") + .partitionValues(Collections.singletonList("2024-01-01")) + .location("hdfs://ns/db/t/dt=2024-01-01") + .columns(Collections.singletonList(fieldSchema("id", "int"))) + .inputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat") + .outputFormat("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat") + .serde("org.apache.hadoop.hive.ql.io.orc.OrcSerde") + .parameters(new HashMap<>()) + .statistics(HmsPartitionStatistics.fromCommonStatistics(10, 2, 100)) + .build(); + + List partitions = HmsWriteConverter.toHivePartitions("db", "t", + Collections.singletonList(part)); + + Assertions.assertEquals(1, partitions.size()); + Partition p = partitions.get(0); + Assertions.assertEquals("db", p.getDbName()); + Assertions.assertEquals("t", p.getTableName()); + Assertions.assertEquals(Collections.singletonList("2024-01-01"), p.getValues()); + + StorageDescriptor sd = p.getSd(); + Assertions.assertEquals("hdfs://ns/db/t/dt=2024-01-01", sd.getLocation()); + Assertions.assertEquals(Collections.singletonList("id"), names(sd.getCols())); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat", sd.getInputFormat()); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat", sd.getOutputFormat()); + Assertions.assertEquals("org.apache.hadoop.hive.ql.io.orc.OrcSerde", + sd.getSerdeInfo().getSerializationLib()); + + // Basic statistics land on the partition parameters (numRows / numFiles / totalSize). + Assertions.assertEquals("10", p.getParameters().get(StatsSetupConst.ROW_COUNT)); + Assertions.assertEquals("2", p.getParameters().get(StatsSetupConst.NUM_FILES)); + Assertions.assertEquals("100", p.getParameters().get(StatsSetupConst.TOTAL_SIZE)); + } + + @Test + public void testToHivePartitionsEmptyLocationBecomesNull() { + // Strings.emptyToNull parity: an empty location must not be written as "" (HMS treats "" and + // null differently for partition location resolution). + HmsPartitionWithStatistics part = HmsPartitionWithStatistics.builder() + .partitionValues(Collections.singletonList("2024")) + .location("") + .columns(Collections.emptyList()) + .parameters(new HashMap<>()) + .statistics(HmsPartitionStatistics.EMPTY) + .build(); + Partition p = HmsWriteConverter.toHivePartitions("db", "t", + Collections.singletonList(part)).get(0); + Assertions.assertNull(p.getSd().getLocation()); + } + + @Test + public void testToStatisticsParametersStampsBasicStatsAndWorkaroundFlag() { + Map origin = mutableMap("existing", "kept"); + Map result = HmsWriteConverter.toStatisticsParameters( + origin, new HmsCommonStatistics(7, 3, 70)); + Assertions.assertEquals("7", result.get(StatsSetupConst.ROW_COUNT)); + Assertions.assertEquals("3", result.get(StatsSetupConst.NUM_FILES)); + Assertions.assertEquals("70", result.get(StatsSetupConst.TOTAL_SIZE)); + // Pre-existing params survive. + Assertions.assertEquals("kept", result.get("existing")); + // CDH workaround flag added when absent. + Assertions.assertEquals("workaround for potential lack of HIVE-12730", + result.get("STATS_GENERATED_VIA_STATS_TASK")); + // The source map is not mutated (defensive copy). + Assertions.assertFalse(origin.containsKey(StatsSetupConst.ROW_COUNT)); + } + + @Test + public void testToStatisticsParametersKeepsExistingWorkaroundFlag() { + Map origin = mutableMap("STATS_GENERATED_VIA_STATS_TASK", "already-set"); + Map result = HmsWriteConverter.toStatisticsParameters( + origin, HmsCommonStatistics.EMPTY); + Assertions.assertEquals("already-set", result.get("STATS_GENERATED_VIA_STATS_TASK")); + } + + @Test + public void testToPartitionStatisticsReadsBackParamsAndDefaults() { + Map params = new HashMap<>(); + params.put(StatsSetupConst.ROW_COUNT, "42"); + params.put(StatsSetupConst.NUM_FILES, "4"); + params.put(StatsSetupConst.TOTAL_SIZE, "420"); + HmsCommonStatistics stats = + HmsWriteConverter.toPartitionStatistics(params).getCommonStatistics(); + Assertions.assertEquals(42, stats.getRowCount()); + Assertions.assertEquals(4, stats.getFileCount()); + Assertions.assertEquals(420, stats.getTotalFileBytes()); + + // Missing params default to -1 (legacy sentinel for "unknown"). + HmsCommonStatistics missing = + HmsWriteConverter.toPartitionStatistics(new HashMap<>()).getCommonStatistics(); + Assertions.assertEquals(-1, missing.getRowCount()); + Assertions.assertEquals(-1, missing.getFileCount()); + Assertions.assertEquals(-1, missing.getTotalFileBytes()); + } + + private static FieldSchema fieldSchema(String name, String type) { + FieldSchema fs = new FieldSchema(); + fs.setName(name); + fs.setType(type); + return fs; + } + private static List names(List schemas) { return schemas.stream().map(FieldSchema::getName).collect(java.util.stream.Collectors.toList()); } diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientWriteAcidTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientWriteAcidTest.java new file mode 100644 index 00000000000000..2d495293a5e013 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientWriteAcidTest.java @@ -0,0 +1,455 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.hadoop.hive.common.StatsSetupConst; +import org.apache.hadoop.hive.common.ValidReadTxnList; +import org.apache.hadoop.hive.metastore.IMetaStoreClient; +import org.apache.hadoop.hive.metastore.api.DataOperationType; +import org.apache.hadoop.hive.metastore.api.LockComponent; +import org.apache.hadoop.hive.metastore.api.LockRequest; +import org.apache.hadoop.hive.metastore.api.LockResponse; +import org.apache.hadoop.hive.metastore.api.LockState; +import org.apache.hadoop.hive.metastore.api.LockType; +import org.apache.hadoop.hive.metastore.api.NoSuchObjectException; +import org.apache.hadoop.hive.metastore.api.Partition; +import org.apache.hadoop.hive.metastore.api.Table; +import org.apache.hadoop.hive.metastore.api.TableValidWriteIds; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.Method; +import java.lang.reflect.Proxy; +import java.nio.ByteBuffer; +import java.util.ArrayList; +import java.util.BitSet; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; + +/** + * Tests the write / read-ACID primitives added to {@link ThriftHmsClient} — the SPI-clean port of + * fe-core {@code ThriftHMSCachedClient}'s add-partition / statistics / txn / lock / valid-write-id + * calls. + * + *

WHY: these primitives are the metastore mutations behind a Hive INSERT commit and a + * transactional-read snapshot. A dropped argument, a wrong forward, or a lost graceful-degradation + * fallback silently corrupts a commit or fails a scan. The tests inject a recording fake + * {@link IMetaStoreClient} (a JDK {@link Proxy}; no Mockito in the connector modules) via the + * package-private client-provider seam with pooling disabled, then assert exactly what each primitive + * forwards to the metastore and what it returns — the behavior contract the connector transaction + * depends on (Rule 9).

+ */ +public class ThriftHmsClientWriteAcidTest { + + // ---- openTxn / commitTxn -------------------------------------------------------------------- + + @Test + public void testOpenTxnForwardsUserAndReturnsId() { + RecordingClient fake = new RecordingClient().stub("openTxn", 42L); + ThriftHmsClient client = newClient(fake); + + long txnId = client.openTxn("alice"); + + Assertions.assertEquals(42L, txnId); + Assertions.assertEquals("alice", argsOf(fake, "openTxn")[0]); + } + + @Test + public void testCommitTxnForwardsId() { + RecordingClient fake = new RecordingClient(); + ThriftHmsClient client = newClient(fake); + + client.commitTxn(7L); + + Assertions.assertEquals(7L, argsOf(fake, "commitTxn")[0]); + } + + // ---- dropPartition / partitionExists -------------------------------------------------------- + + @Test + public void testDropPartitionForwardsArgsAndReturnsResult() { + RecordingClient fake = new RecordingClient().stub("dropPartition", Boolean.TRUE); + ThriftHmsClient client = newClient(fake); + + boolean dropped = client.dropPartition("db", "t", + Collections.singletonList("2024"), false); + + Assertions.assertTrue(dropped); + Object[] args = argsOf(fake, "dropPartition"); + Assertions.assertEquals("db", args[0]); + Assertions.assertEquals("t", args[1]); + Assertions.assertEquals(Collections.singletonList("2024"), args[2]); + Assertions.assertEquals(false, args[3]); + } + + @Test + public void testPartitionExistsTrueWhenFound() { + RecordingClient fake = new RecordingClient().stub("getPartition", new Partition()); + ThriftHmsClient client = newClient(fake); + + Assertions.assertTrue(client.partitionExists("db", "t", + Collections.singletonList("2024"))); + } + + @Test + public void testPartitionExistsFalseOnNoSuchObject() { + // A not-found probe must swallow NoSuchObjectException and return false (drives the + // NEW->APPEND downgrade), not propagate it as a client failure. + RecordingClient fake = new RecordingClient() + .stub("getPartition", new NoSuchObjectException("no such partition")); + ThriftHmsClient client = newClient(fake); + + Assertions.assertFalse(client.partitionExists("db", "t", + Collections.singletonList("2024"))); + } + + // ---- addPartitions -------------------------------------------------------------------------- + + @Test + public void testAddPartitionsBuildsMetastorePartitions() { + RecordingClient fake = new RecordingClient().stub("add_partitions", 1); + ThriftHmsClient client = newClient(fake); + + HmsPartitionWithStatistics part = HmsPartitionWithStatistics.builder() + .partitionValues(Collections.singletonList("2024")) + .location("hdfs://ns/db/t/dt=2024") + .columns(Collections.emptyList()) + .inputFormat("in") + .outputFormat("out") + .serde("serde") + .parameters(new HashMap<>()) + .statistics(HmsPartitionStatistics.fromCommonStatistics(10, 2, 100)) + .build(); + + client.addPartitions("db", "t", Collections.singletonList(part)); + + Object[] args = argsOf(fake, "add_partitions"); + @SuppressWarnings("unchecked") + List sent = (List) args[0]; + Assertions.assertEquals(1, sent.size()); + Partition p = sent.get(0); + Assertions.assertEquals("db", p.getDbName()); + Assertions.assertEquals("t", p.getTableName()); + Assertions.assertEquals(Collections.singletonList("2024"), p.getValues()); + Assertions.assertEquals("hdfs://ns/db/t/dt=2024", p.getSd().getLocation()); + Assertions.assertEquals("10", p.getParameters().get(StatsSetupConst.ROW_COUNT)); + } + + @Test + public void testAddPartitionsBatchesInChunksOfTwenty() { + // 45 partitions -> ceil(45/20) = 3 add_partitions calls; a single giant call can exceed the + // metastore's thrift message limit. + RecordingClient fake = new RecordingClient().stub("add_partitions", 0); + ThriftHmsClient client = newClient(fake); + + List many = new ArrayList<>(); + for (int i = 0; i < 45; i++) { + many.add(HmsPartitionWithStatistics.builder() + .partitionValues(Collections.singletonList("p" + i)) + .columns(Collections.emptyList()) + .parameters(new HashMap<>()) + .statistics(HmsPartitionStatistics.EMPTY) + .build()); + } + + client.addPartitions("db", "t", many); + + // 3 chunks (20 + 20 + 5), and — the load-bearing contract — no partition is lost or + // duplicated across the chunk boundaries: the union of forwarded partitions is exactly p0..p44. + List forwarded = new ArrayList<>(); + long calls = 0; + for (int idx = 0; idx < fake.methodNames.size(); idx++) { + if (!"add_partitions".equals(fake.methodNames.get(idx))) { + continue; + } + calls++; + @SuppressWarnings("unchecked") + List batch = (List) fake.argsList.get(idx)[0]; + Assertions.assertTrue(batch.size() <= 20, "a batch exceeded ADD_PARTITIONS_BATCH_SIZE"); + for (Partition p : batch) { + forwarded.add(p.getValues().get(0)); + } + } + Assertions.assertEquals(3, calls); + Assertions.assertEquals(45, forwarded.size()); + Set expected = new HashSet<>(); + for (int i = 0; i < 45; i++) { + expected.add("p" + i); + } + Assertions.assertEquals(expected, new HashSet<>(forwarded)); + } + + // ---- table / partition statistics ----------------------------------------------------------- + + @Test + public void testUpdateTableStatisticsRebuildsParamsAndAlters() { + RecordingClient fake = new RecordingClient(); + Table origin = new Table(); + origin.setParameters(new HashMap<>()); + fake.stub("getTable", origin); + ThriftHmsClient client = newClient(fake); + + client.updateTableStatistics("db", "t", + current -> HmsPartitionStatistics.fromCommonStatistics(5, 1, 50)); + + Object[] args = argsOf(fake, "alter_table"); + Assertions.assertEquals("db", args[0]); + Assertions.assertEquals("t", args[1]); + Table altered = (Table) args[2]; + Assertions.assertEquals("5", altered.getParameters().get(StatsSetupConst.ROW_COUNT)); + Assertions.assertEquals("1", altered.getParameters().get(StatsSetupConst.NUM_FILES)); + Assertions.assertEquals("50", altered.getParameters().get(StatsSetupConst.TOTAL_SIZE)); + Assertions.assertTrue(altered.getParameters().containsKey("transient_lastDdlTime")); + } + + @Test + public void testUpdatePartitionStatisticsRebuildsParamsAndAlters() { + RecordingClient fake = new RecordingClient(); + Partition origin = new Partition(); + origin.setParameters(new HashMap<>()); + fake.stub("getPartitionsByNames", Collections.singletonList(origin)); + ThriftHmsClient client = newClient(fake); + + client.updatePartitionStatistics("db", "t", "dt=2024", + current -> HmsPartitionStatistics.fromCommonStatistics(3, 1, 30)); + + Object[] args = argsOf(fake, "alter_partition"); + Partition altered = (Partition) args[2]; + Assertions.assertEquals("3", altered.getParameters().get(StatsSetupConst.ROW_COUNT)); + Assertions.assertEquals("30", altered.getParameters().get(StatsSetupConst.TOTAL_SIZE)); + } + + @Test + public void testUpdatePartitionStatisticsRejectsAmbiguousName() { + RecordingClient fake = new RecordingClient(); + fake.stub("getPartitionsByNames", new ArrayList()); // size 0 != 1 + ThriftHmsClient client = newClient(fake); + + Assertions.assertThrows(HmsClientException.class, () -> + client.updatePartitionStatistics("db", "t", "dt=2024", + current -> HmsPartitionStatistics.EMPTY)); + } + + // ---- getValidWriteIds ----------------------------------------------------------------------- + + @Test + public void testGetValidWriteIdsSuccessPathEmitsSnapshotThenWriteIds() { + // Primary read-ACID path: a compatible metastore returns exactly one write-id list. A + // distinctive snapshot (high-watermark 100, NOT the fallback's Long.MAX_VALUE) makes this test + // fail if the code silently degrades to the fallback branch. + ValidReadTxnList snapshot = new ValidReadTxnList(new long[0], new BitSet(), 100L, 5L); + TableValidWriteIds writeIds = new TableValidWriteIds( + "db.t", 100L, Collections.emptyList(), ByteBuffer.allocate(0)); + RecordingClient fake = new RecordingClient() + .stub("getValidTxns", snapshot) + .stub("getValidWriteIds", Collections.singletonList(writeIds)); + ThriftHmsClient client = newClient(fake); + + Map conf = client.getValidWriteIds("db.t", 42L); + + // (a) The recent snapshot string — NOT currentTransactionId (42) — drives the write-id lookup. + Object[] gvwiArgs = argsOf(fake, "getValidWriteIds"); + Assertions.assertEquals(Collections.singletonList("db.t"), gvwiArgs[0]); + Assertions.assertEquals(snapshot.toString(), gvwiArgs[1]); + // (b) VALID_TXNS_KEY carries the txn snapshot; VALID_WRITEIDS_KEY carries the write-id list — + // no key swap. Both are the SUCCESS values (snapshot watermark 100), not the fallback watermark. + Assertions.assertEquals(snapshot.writeToString(), conf.get(HmsAcidConstants.VALID_TXNS_KEY)); + Assertions.assertTrue(conf.get(HmsAcidConstants.VALID_WRITEIDS_KEY).contains("db.t")); + Assertions.assertNotEquals(conf.get(HmsAcidConstants.VALID_TXNS_KEY), + conf.get(HmsAcidConstants.VALID_WRITEIDS_KEY)); + } + + @Test + public void testGetValidWriteIdsFallsBackToMaxWatermark() { + // An incompatible metastore returns an unexpected write-id list; rather than fail the scan, + // getValidWriteIds must degrade to a max watermark and still emit both config keys. + ValidReadTxnList snapshot = new ValidReadTxnList(); + RecordingClient fake = new RecordingClient() + .stub("getValidTxns", snapshot) + .stub("getValidWriteIds", new ArrayList<>()); // size 0 -> triggers fallback + ThriftHmsClient client = newClient(fake); + + Map conf = client.getValidWriteIds("db.t", 5L); + + // The recent snapshot (not currentTransactionId) was still forwarded before the size guard threw. + Object[] gvwiArgs = argsOf(fake, "getValidWriteIds"); + Assertions.assertEquals(Collections.singletonList("db.t"), gvwiArgs[0]); + Assertions.assertEquals(snapshot.toString(), gvwiArgs[1]); + // Both BE-contract keys are present, and the fallback write-id list is scoped to the table. + Assertions.assertNotNull(conf.get(HmsAcidConstants.VALID_TXNS_KEY)); + Assertions.assertNotNull(conf.get(HmsAcidConstants.VALID_WRITEIDS_KEY)); + Assertions.assertTrue(conf.get(HmsAcidConstants.VALID_WRITEIDS_KEY).contains("db.t")); + } + + // ---- acquireSharedLock ---------------------------------------------------------------------- + + @Test + public void testAcquireSharedLockBuildsSharedReadComponentsAndReturnsWhenAcquired() { + RecordingClient fake = new RecordingClient() + .stub("lock", new LockResponse(1L, LockState.ACQUIRED)); + ThriftHmsClient client = newClient(fake); + + client.acquireSharedLock("q1", 5L, "alice", "db", "t", + Collections.emptyList(), 1000L); + + LockRequest request = (LockRequest) argsOf(fake, "lock")[0]; + Assertions.assertEquals(1, request.getComponent().size()); + LockComponent component = request.getComponent().get(0); + Assertions.assertEquals("db", component.getDbname()); + Assertions.assertEquals("t", component.getTablename()); + Assertions.assertEquals(LockType.SHARED_READ, component.getType()); + Assertions.assertEquals(DataOperationType.SELECT, component.getOperationType()); + } + + @Test + public void testAcquireSharedLockOneComponentPerPartition() { + RecordingClient fake = new RecordingClient() + .stub("lock", new LockResponse(1L, LockState.ACQUIRED)); + ThriftHmsClient client = newClient(fake); + + client.acquireSharedLock("q1", 5L, "alice", "db", "t", + java.util.Arrays.asList("dt=1", "dt=2"), 1000L); + + LockRequest request = (LockRequest) argsOf(fake, "lock")[0]; + Assertions.assertEquals(2, request.getComponent().size()); + Assertions.assertEquals("dt=1", request.getComponent().get(0).getPartitionname()); + Assertions.assertEquals("dt=2", request.getComponent().get(1).getPartitionname()); + } + + @Test + public void testAcquireSharedLockPollsUntilAcquired() { + RecordingClient fake = new RecordingClient() + .stub("lock", new LockResponse(2L, LockState.WAITING)) + .stub("checkLock", new LockResponse(2L, LockState.ACQUIRED)); + ThriftHmsClient client = newClient(fake); + + client.acquireSharedLock("q", 5L, "u", "db", "t", Collections.emptyList(), 5000L); + + Assertions.assertTrue(fake.methodNames.contains("checkLock")); + } + + @Test + public void testAcquireSharedLockTimesOut() { + RecordingClient fake = new RecordingClient() + .stub("lock", new LockResponse(3L, LockState.WAITING)) + .stub("checkLock", new LockResponse(3L, LockState.WAITING)); + ThriftHmsClient client = newClient(fake); + + // timeoutMs = -1 => the elapsed guard trips on the first poll iteration, deterministically. + Assertions.assertThrows(HmsClientException.class, () -> + client.acquireSharedLock("q", 5L, "u", "db", "t", Collections.emptyList(), -1L)); + } + + // ---- harness -------------------------------------------------------------------------------- + + private static ThriftHmsClient newClient(RecordingClient handler) { + IMetaStoreClient fake = (IMetaStoreClient) Proxy.newProxyInstance( + IMetaStoreClient.class.getClassLoader(), + new Class[] {IMetaStoreClient.class}, + handler); + // poolSize 0 -> no pool: every call creates a fresh client via the provider (our fake). + HmsClientConfig config = new HmsClientConfig(new HashMap<>(), 0); + return new ThriftHmsClient(config, null, hiveConf -> fake, HmsTypeMapping.Options.DEFAULT); + } + + private static Object[] argsOf(RecordingClient handler, String method) { + int idx = handler.methodNames.indexOf(method); + Assertions.assertTrue(idx >= 0, "expected a call to " + method); + return handler.argsList.get(idx); + } + + /** + * A recording fake {@link IMetaStoreClient}: records every method name + args in call order and + * returns per-method canned values (a {@link Throwable} value is thrown from the call). + * Implemented as an {@link InvocationHandler} because {@code IMetaStoreClient} has hundreds of + * methods — an anonymous class is impractical and Mockito is not on the connector test path. + */ + private static final class RecordingClient implements InvocationHandler { + private final List methodNames = new ArrayList<>(); + private final List argsList = new ArrayList<>(); + private final Map responses = new HashMap<>(); + + RecordingClient stub(String method, Object value) { + responses.put(method, value); + return this; + } + + @Override + public Object invoke(Object proxy, Method method, Object[] args) throws Throwable { + String name = method.getName(); + if (method.getDeclaringClass() == Object.class) { + switch (name) { + case "toString": + return "RecordingClient"; + case "hashCode": + return System.identityHashCode(proxy); + case "equals": + return proxy == args[0]; + default: + return null; + } + } + if ("close".equals(name)) { + return null; + } + methodNames.add(name); + argsList.add(args == null ? new Object[0] : args); + if (responses.containsKey(name)) { + Object value = responses.get(name); + if (value instanceof Throwable) { + throw (Throwable) value; + } + return value; + } + return defaultValue(method.getReturnType()); + } + + private static Object defaultValue(Class type) { + if (!type.isPrimitive() || type == void.class) { + return null; + } + if (type == boolean.class) { + return false; + } + if (type == long.class) { + return 0L; + } + if (type == int.class) { + return 0; + } + if (type == short.class) { + return (short) 0; + } + if (type == byte.class) { + return (byte) 0; + } + if (type == char.class) { + return (char) 0; + } + if (type == float.class) { + return 0f; + } + return 0d; + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index ea2b0b75c26a92..cfc50b952e93f8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -201,6 +201,22 @@ public boolean requirePartitionLocalSortOnWrite() { return connector != null && connector.requiresPartitionLocalSort(); } + /** + * Returns whether the underlying connector requires dynamic-partition writes to be hash-distributed by + * partition columns but not locally sorted (e.g. Hive: the file writer buffers a per-partition + * writer, so the hash alone keeps each partition on one instance without a sort). Used by + * {@code PhysicalConnectorTableSink} to require that hash distribution (no {@code MustLocalSortOrderSpec}) for + * dynamic-partition writes; a connector sets at most one of this and {@link #requirePartitionLocalSortOnWrite()}. + * Defaults to false so non-partitioned connectors are unaffected. + */ + public boolean requirePartitionHashOnWrite() { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return false; + } + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + return connector != null && connector.requiresPartitionHashWrite(); + } + /** * Returns whether the underlying connector maps write data columns positionally against the full * table schema (e.g. MaxCompute), requiring the sink to project rows to full-schema order with diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java index 3457daab9a894d..6f61c92becbc1c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSink.java @@ -218,6 +218,34 @@ public PhysicalProperties getRequirePhysicalProperties() { } } + if (table.requirePartitionHashOnWrite()) { + Set partitionNames = table.getPartitionColumns().stream() + .map(Column::getName) + .collect(Collectors.toSet()); + if (!partitionNames.isEmpty()) { + // Hash-distribute by partition columns with NO local sort (byte-exact to legacy + // PhysicalHiveTableSink.getRequirePhysicalProperties): same partition value -> same writer + // instance keeps the output file count at ~one-per-partition, and the hive file writer buffers a + // per-partition writer so — unlike the MaxCompute arm above — no MustLocalSortOrderSpec is added. + // Index by full-schema position, which is aligned 1:1 with child output because a connector + // declaring requiresPartitionHashWrite also declares requiresFullSchemaWriteOrder. + List columnIdx = new ArrayList<>(); + List fullSchema = targetTable.getFullSchema(); + for (int i = 0; i < fullSchema.size(); i++) { + if (partitionNames.contains(fullSchema.get(i).getName())) { + columnIdx.add(i); + } + } + List exprIds = columnIdx.stream() + .map(idx -> child().getOutput().get(idx).getExprId()) + .collect(Collectors.toList()); + DistributionSpecHiveTableSinkHashPartitioned shuffleInfo + = new DistributionSpecHiveTableSinkHashPartitioned(); + shuffleInfo.setOutputColExprIds(exprIds); + return new PhysicalProperties(shuffleInfo); + } + } + if (table.supportsParallelWrite()) { return PhysicalProperties.SINK_RANDOM_PARTITIONED; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorContractValidatorTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorContractValidatorTest.java index 79c995b9e52cd3..5e47ae7b7f6c85 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorContractValidatorTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/ConnectorContractValidatorTest.java @@ -90,6 +90,57 @@ void validatorRejectsLocalSortWithoutFullSchema() { Assertions.assertTrue(ex.getMessage().contains("fake_localsort_no_fullschema"), "got: " + ex.getMessage()); } + @Test + void validatorRejectsHashWriteWithoutParallelAndFullSchema() { + // Invariant #4: requiresPartitionHashWrite() (hash-by-partition without a local sort) likewise + // implies BOTH requiresParallelWrite() AND requiresFullSchemaWriteOrder() — the hash arm in + // PhysicalConnectorTableSink indexes partition columns by full-schema position and distributes in + // parallel, so declaring hash-write without the other two must fail loud, not silently mis-plan. + Connector fake = Mockito.mock(Connector.class); + Mockito.when(fake.requiresPartitionHashWrite()).thenReturn(true); + Mockito.when(fake.requiresParallelWrite()).thenReturn(false); + Mockito.when(fake.requiresFullSchemaWriteOrder()).thenReturn(true); + + IllegalStateException ex = Assertions.assertThrows(IllegalStateException.class, + () -> ConnectorContractValidator.validate(fake, "fake_hash_no_parallel")); + Assertions.assertTrue(ex.getMessage().contains("requiresPartitionHashWrite"), "got: " + ex.getMessage()); + Assertions.assertTrue(ex.getMessage().contains("fake_hash_no_parallel"), "got: " + ex.getMessage()); + } + + @Test + void validatorRejectsBothPartitionDistributionArms() { + // Invariant #5: the two hash arms are mutually exclusive. PhysicalConnectorTableSink checks + // requirePartitionLocalSortOnWrite() BEFORE requirePartitionHashOnWrite(), so a connector declaring + // both would silently get the local-sort arm and never the hash-without-sort it asked for. That is a + // misconfiguration, so it must fail loud at registration. Both are otherwise internally consistent + // (parallel + full-schema) to isolate the mutual-exclusion check as the sole reason for the throw. + Connector fake = Mockito.mock(Connector.class); + Mockito.when(fake.requiresParallelWrite()).thenReturn(true); + Mockito.when(fake.requiresFullSchemaWriteOrder()).thenReturn(true); + Mockito.when(fake.requiresPartitionLocalSort()).thenReturn(true); + Mockito.when(fake.requiresPartitionHashWrite()).thenReturn(true); + + IllegalStateException ex = Assertions.assertThrows(IllegalStateException.class, + () -> ConnectorContractValidator.validate(fake, "fake_both_arms")); + Assertions.assertTrue(ex.getMessage().contains("requiresPartitionHashWrite"), "got: " + ex.getMessage()); + Assertions.assertTrue(ex.getMessage().contains("fake_both_arms"), "got: " + ex.getMessage()); + } + + @Test + void validatorPassesForAHashWriteConnector() { + // Positive control (Rule 9) for the hive-shaped connector: parallel write + full-schema write order + + // hash-write (no local sort), INSERT/OVERWRITE, no branch — internally consistent, must NOT throw. + Connector fake = Mockito.mock(Connector.class); + Mockito.when(fake.supportedWriteOperations()) + .thenReturn(EnumSet.of(WriteOperation.INSERT, WriteOperation.OVERWRITE)); + Mockito.when(fake.supportsWriteBranch()).thenReturn(false); + Mockito.when(fake.requiresParallelWrite()).thenReturn(true); + Mockito.when(fake.requiresFullSchemaWriteOrder()).thenReturn(true); + Mockito.when(fake.requiresPartitionHashWrite()).thenReturn(true); + + Assertions.assertDoesNotThrow(() -> ConnectorContractValidator.validate(fake, "fake_hash_consistent")); + } + @Test void validatorPassesForAnInternallyConsistentConnector() { // Positive control (Rule 9): a maxcompute-shaped fake (parallel write + full-schema write order + diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java index 03f70ef59e7f80..3d5fa29ec103f5 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalConnectorTableSinkTest.java @@ -49,6 +49,9 @@ * provider returning {@code true} from {@code requiresPartitionLocalSort()} → hash-by-partition + * mandatory local sort, so the MaxCompute Storage API streaming partition writer does not hit * "writer has been closed" on un-grouped rows; + *
  • partitioned write + write provider returning {@code true} from + * {@code requiresPartitionHashWrite()} (Hive-style) → hash-by-partition with no local sort + * (byte-exact to legacy {@code PhysicalHiveTableSink});
    • *
  • non-partition / all-static write + write provider returning {@code true} from * {@code requiresParallelWrite()} → {@code SINK_RANDOM_PARTITIONED} (parallel writers, NG-4 * parity);
    • @@ -250,6 +253,55 @@ public void rewriteModeGathersEvenOnPartitionedTable() { + "overriding the partition-shuffle distribution"); } + /** + * Hash-write connector (Hive-style, {@code requiresPartitionHashWrite=true} but + * {@code requiresPartitionLocalSort=false}): a partitioned write hash-distributes by the partition + * columns with NO mandatory local sort — byte-exact to legacy {@code PhysicalHiveTableSink}, which + * hashed by the partition columns and never attached an order spec (the hive file writer buffers a + * per-partition writer, so grouping the rows by a sort is unnecessary). The MaxCompute arm is skipped + * because {@code requirePartitionLocalSortOnWrite()} is false, so this reaches the new no-sort arm. + */ + @Test + public void partitionHashWriteHashesByPartitionWithoutLocalSort() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + SlotReference partSlot = new SlotReference("part", IntegerType.INSTANCE); + PhysicalConnectorTableSink sink = sink( + table(true, false, true, ImmutableList.of(PART), ImmutableList.of(DATA, PART)), + Arrays.asList(DATA, PART), + ImmutableList.of(dataSlot, partSlot)); + + PhysicalProperties props = sink.getRequirePhysicalProperties(); + + Assertions.assertTrue(props.getDistributionSpec() instanceof DistributionSpecHiveTableSinkHashPartitioned, + "a hash-write connector must hash-distribute a partitioned write by its partition columns"); + DistributionSpecHiveTableSinkHashPartitioned dist = + (DistributionSpecHiveTableSinkHashPartitioned) props.getDistributionSpec(); + Assertions.assertEquals(ImmutableList.of(partSlot.getExprId()), dist.getOutputColExprIds(), + "hash key must be the partition-column slot taken at its full-schema position"); + Assertions.assertFalse(props.getOrderSpec() instanceof MustLocalSortOrderSpec, + "a no-sort hash-write connector must NOT add a mandatory local sort (byte-exact to legacy " + + "PhysicalHiveTableSink, which hash-distributed without a sort) — else a hive write " + + "would pay an unnecessary sort the legacy path never had"); + } + + /** + * Non-partitioned write on a hash-write connector: the hash arm's {@code !partitionNames.isEmpty()} + * gate falls through to the parallel arm, matching legacy {@code PhysicalHiveTableSink}'s + * non-partitioned {@code SINK_RANDOM_PARTITIONED}. Guards that the hash arm never fires without + * partition columns (which would build an empty-key distribution). + */ + @Test + public void nonPartitionedHashWriteConnectorUsesRandomWhenParallel() { + SlotReference dataSlot = new SlotReference("data", IntegerType.INSTANCE); + PhysicalConnectorTableSink sink = sink( + table(true, false, true, ImmutableList.of(), ImmutableList.of(DATA)), + Arrays.asList(DATA), + ImmutableList.of(dataSlot)); + + Assertions.assertSame(PhysicalProperties.SINK_RANDOM_PARTITIONED, sink.getRequirePhysicalProperties(), + "a non-partitioned hash-write connector falls through to parallel writers, not the hash arm"); + } + // ==================== helpers ==================== private static PluginDrivenExternalTable table(boolean parallelWrite, boolean requirePartitionSort, @@ -262,6 +314,14 @@ private static PluginDrivenExternalTable table(boolean parallelWrite, boolean re return table; } + /** As {@link #table(boolean, boolean, List, List)} but also stubs the hash-write capability. */ + private static PluginDrivenExternalTable table(boolean parallelWrite, boolean requirePartitionSort, + boolean requirePartitionHash, List partitionColumns, List fullSchema) { + PluginDrivenExternalTable table = table(parallelWrite, requirePartitionSort, partitionColumns, fullSchema); + Mockito.when(table.requirePartitionHashOnWrite()).thenReturn(requirePartitionHash); + return table; + } + /** * Builds a {@link PhysicalConnectorTableSink} exercising only {@code getRequirePhysicalProperties()}. * Uses CALLS_REAL_METHODS to skip the heavyweight ctor and injects the three fields the method From d13d2597bca20306c47801783f56c2cff38f143d Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 6 Jul 2026 23:35:24 +0800 Subject: [PATCH 026/334] =?UTF-8?q?[doc](catalog)=20P7.3=20hive:=20FINAL?= =?UTF-8?q?=20atomic=20feature=20commit=20landed=20(0b19506acfe)=20?= =?UTF-8?q?=E2=86=92=20refresh=20HANDOFF,=20next=20=3D=20cutover?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INC-1..INC-5 (hive plugin-side write transaction + committer + write plan provider + ACID read producer, dormant) are now committed as one atomic feature commit 0b19506acfe (40 files, 22 new + 18 modified). Working tree no longer carries the batch as uncommitted WIP. Refresh HANDOFF: current-state + this-session log now record the FINAL commit and its fresh verification (hive 104 / hms 49 / api 55 / fe-core 16 tests, checkstyle 0, import-gate clean); next-session task retargeted from FINAL to cutover (P7.4/P7.5): flip SPI_READY_TYPES, retype fe-core write chain, wire the plugin read-txn manager + query-finish deregister, drop legacy HiveTransactionMgr + datasource/hive (guarding cross-connector delete order), and run the ACID integration suite (R-002). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 45 ++++++++++++++++++++------------------------- 1 file changed, 20 insertions(+), 25 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c7c841b644ebff..705a87a9b1433d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,47 +5,42 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1..INC-5 全部完成 + 全绿（写链 + 读链都齐了）。下一步 = FINAL：INC-1..INC-5 一次原子 feature commit（path-whitelist，待用户点头）。全批仍 UNCOMMITTED 在工作树里（跨 session WIP 载体，勿 commit / 勿 revert / 勿 `git checkout`）。本轮已 commit 的只有 HANDOFF + 设计文档 + INC-5 port-map（文档，不含功能码）。** +# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1..INC-5 的 FINAL 原子 feature commit 已落地 = `0b19506acfe`（40 文件：22 新 + 18 改，+7456/−34）。写链 + 读链全部完成、全绿、且已提交。工作树不再持有本批未提交功能码（原子批 WIP 阶段结束）。下一步 = cutover（P7.4/P7.5，另起原子批）：翻闸 `SPI_READY_TYPES` + fe-core 写链 retype + 读侧放锁接线 + 摘 legacy `HiveTransactionMgr` + 删 `datasource/hive`（守跨连接器删序）+ 跑 ACID 集成测试套件（R-002 项目最大风险）。** -> **本轮做了什么（INC-5：读侧 ACID 生产端 + 插件读事务生命周期，dormant）**：新建 `HiveAcidUtil`（把 HEAD `AcidUtil.getAcidState` 的**纯目录名解析 + 快照挑选算法**逐条移到插件：`parseBase`/`parseDelta`/`isValidBase`/`isValidMetaDataFile` + FullAcid/InsertOnly 文件过滤 + best-base/working-delta 选择环；fe-core `FileCacheValue`/`LocationPath`/`AcidInfo`/`HivePartition` 全落地为原生 Hadoop `FileSystem`/`FileStatus`；`globList(...,"/*")`→`fs.listStatus`[文件+目录]、`globList(bareDir)`→`fs.listStatus`+仅留文件；读 `HmsAcidConstants` 两键）+ `HiveReadTransaction`/`HiveReadTransactionManager`（移植 `HiveTransaction`/`HiveTransactionMgr`，用 INC-1 的 `openTxn`/`acquireSharedLock`/`getValidWriteIds`/`commitTxn`；`TableNameInfo`→(db,table) 串）+ `HiveScanPlanProvider.planAcidScan` 下潜 + `.acidInfo("dir|file1,file2")` 生产端接线（含 full-acid 非 ORC fail-loud）+ `HiveTableHandle.isTransactional/isFullAcid` 从 `getTableParameters()` 派生（D8）+ `HiveConnector` 持有（dormant）读事务管理器。 +> **本轮做了什么（FINAL：一次原子 feature commit，已落地）**：按白名单 path-whitelist `git add`（api 3 + hms 11 + hive 22 + fe-core 4 = 40；`git add /src` 仅收该模块 src 下 wanted 文件，`target/` 已 gitignore）→ 暂存后 `git diff --cached` 自检 = 正好 **40/40（22 A + 18 M）**，无一文件落在 4 白名单区域外、无 `.bak`/`regression-conf` 明文 key/`.audit-scratch`/`conf.cmy`/`META-INF`/`docker`/`.claude`/`plan-doc` 混入 → commit `0b19506acfe`（message 沿用 `[feat](catalog) P7.3 hive …` 范式 + 三段模块摘要 + 验证数据 + `Refs: apache/doris#65185` + `Co-Authored-By`/`Claude-Session` 尾）。**核实**：`FakeConnectorContext`/`HmsPartitionInfo` 是既有已跟踪共享类（被既有 hudi/hive 测试引用），非本批新增（HANDOFF 旧叙述误记为"新"）→ 本批自洽、单独 checkout 白名单集即可编译。 -> **本轮验证（独立重跑，不信自述）**：`fe-connector-hive` 全模块 `clean test` **104 测试 0 失败 0 错误 BUILD SUCCESS（原 87 + 新 17：`HiveAcidUtilTest`=9、`HiveReadTransactionTest`=3、`HiveTableHandleAcidTest`=5，既有 `HiveScanRangeAcidTest`=3 保绿）**；`checkstyle:check` 0 违规；`check-connector-imports.sh` 净（新码只碰 Hadoop/hive-common/thrift/`HmsAcidConstants`，无 fe-core）。**净室对抗复审（`wf-inc5-review.js`，6 维 × find→对抗 verify，13 agent 0 error）：1 条 CONFIRMED（已修）+ 1 条自查先修 + 测试补强**：(a) **CONFIRMED** insert-only 事务表被错标 `transactional_hive`（HEAD 只在 `isFullAcid` 才挂 `acidInfo`；insert-only 存的是平文件 → 错走 BE merge-on-read）→ 修为 `planAcidScan` 仅 full-acid 才挂 `acidInfo`（insert-only 仍按快照选文件、但标平 `hive`）；(b) 复审前**自查**发现读事务管理器按 queryId 跨不同表复用一个事务 → 第二张表拿到第一张的快照，改为**每表各开各事务**（`register` 非复用，对齐 HEAD）；(c) 按复审 test-quality 发现补了 4 个 descent 测试（可见性 txn 过滤 / `isValidBase` 甄别 / not-enough-history）+ insert_only 大小写。其余 5 条 refuted 为 benign/有意偏差。**遗留（交割 cutover 集成门）**：`planScan` 级 insert-only 标记门禁的回归测试（需 live session/txn/FS 管线，设计 §7 把集成测试推到 cutover）。 +> **本轮验证（独立重跑、不信自述、提交前 fresh 编译杜绝 stale `.class`）**：`fe-connector-hive` `clean test` **104 测试 0 失败 0 错误 0 跳过 BUILD SUCCESS**；`fe-connector-hms` 49、`fe-connector-api` 55 全绿；`fe-core` 从零编译 4560 源 + 受影响 2 测试类（`ConnectorContractValidatorTest` 7 + `PhysicalConnectorTableSinkTest` 9 = **16** 0 失败 0 跳过）+ checkstyle 0；连接器 3 模块 checkstyle 0；`check-connector-imports.sh` 净（仅命中 `HiveVersionUtil` 已知误报）。 -> **⚠️ 工作树未提交的 INC-1..INC-4 代码**（`git status` 会显示；原子批模型，设计文档 §8——**若工作树被清就照各 port-map 重建**）： -> - INC-1（`fe-connector-hms`）：`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics`/`HmsPartitionInfo` 新 + 2 test。 -> - INC-2（`fe-connector-hive`）：`HiveWriteUtils.java` + `NameMapping.java`（+ 2 test）。 -> - **INC-3（`fe-connector-hive`）**：**新** `HiveWriteContext.java`(89) + `HiveConnectorTransaction.java`(1704，含 dup-key 修复；本轮 INC-4 又改了 guard→`rejectTransactionalWrite`) + `HiveConnectorTransactionTest.java`(14 例) + `FakeConnectorContext.java`(64)；**改** `HiveConnectorMetadata.java`(`beginTransaction` override) + `pom.xml` + `HiveWriteUtils.java`。 -> - **INC-4（`fe-connector-hive` + 通用层）**：**新** `HiveWritePlanProvider.java`(362) + `HiveSinkHelper.java`(246) + `HiveWritePlanProviderTest.java`(716,20 例) + `RecordingConnectorContext.java`(98)；**改** `HiveConnector.java`(+getWritePlanProvider) + `HiveConnectorTransaction.java`(guard 放宽) + `HiveConnectorTransactionTest.java`(+insert-only 拒绝)。**通用层（DEC-1，本批唯一 fe-core 改动）**：`fe-connector-api` `ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator` + `fe-core` `PluginDrivenExternalTable`/`PhysicalConnectorTableSink` + 2 个 fe-core test 文件各加用例 + `fe-connector-hms` `HmsTableInfo`(+bucketCols/numBuckets)/`ThriftHmsClient`(convertTable 填桶)。 -> - **INC-5（`fe-connector-hive`）**：**新** `HiveAcidUtil.java` + `HiveReadTransaction.java` + `HiveReadTransactionManager.java` + 3 test（`HiveAcidUtilTest`/`HiveReadTransactionTest`/`HiveTableHandleAcidTest`）；**改** `HiveScanPlanProvider.java`(+`planAcidScan`/`splitFile`+acid 参/`newRangeBuilder`/`encodeDeleteDeltas`/`buildPartitionName`) + `HiveTableHandle.java`(+`isTransactional`/`isFullAcid`) + `HiveConnector.java`(+读事务管理器) + `pom.xml`(+`commons-lang` test dep)。 +> **✅ INC-1..INC-5 已全部提交进 `0b19506acfe`**（不再是 uncommitted 工作树 WIP；原子批载体阶段结束）。逐增量移植记录留档备查：INC-3=`tasks/P7.3-INC-3-portmap.md`、INC-4=`tasks/P7.3-INC-4-portmap.md`、INC-5=`tasks/P7.3-INC-5-portmap.md`；各 INC 的文件清单见 `git show --stat 0b19506acfe`。 > **权威实现依据**（信 HEAD 控制流，不信行号）： > - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 -> - **`tasks/P7.3-INC-3-portmap.md`（本轮新落地，595 行）**：逐行移植地图 + §7 12 处 GAP + **§9 已核实的实现前检查**（GAP-11/12 结论、D6 MPU map 重载确认、FS 构建/鉴权/iceberg 模板签名全确认）。INC-4/INC-5 也可续用其读侧 §5.4 + 测试计划 §6。 +> - **`tasks/P7.3-INC-3-portmap.md`（595 行）**：逐行移植地图 + §7 12 处 GAP + **§9 已核实的实现前检查**（GAP-11/12 结论、D6 MPU map 重载确认、FS 构建/鉴权/iceberg 模板签名全确认）。INC-4/INC-5 也可续用其读侧 §5.4 + 测试计划 §6。 > - 移植源 = HEAD `fe/fe-core/.../datasource/hive/HMSTransaction.java`；模板 = `fe-connector-iceberg/.../IcebergConnectorTransaction.java`+`IcebergWriteContext.java`。 --- -# 🚀 下个 session 任务 = **FINAL：INC-1..INC-5 一次原子 feature commit（待用户点头）** +# 🚀 下个 session 任务 = **cutover（P7.4/P7.5，另起原子批）：把 dormant 的 hive 插件真正接上线** -> **① INC-1..INC-5 —— ✅ 全部完成 + 全绿**（写链 INC-1..4 + 读链 INC-5，见 🎯）。`fe-connector-hive` **104 测试** + `fe-core` 16 测试全绿；checkstyle 0；import-gate 净；INC-5 已过净室对抗复审（1 CONFIRMED 已修 + 1 自查已修 + 测试补强）。**不再回炒任何增量。** port-map：INC-3=`tasks/P7.3-INC-3-portmap.md`、INC-4=`tasks/P7.3-INC-4-portmap.md`、INC-5=`tasks/P7.3-INC-5-portmap.md`。 +> **① FINAL —— ✅ 已落地**（commit `0b19506acfe`）。INC-1..INC-5 全部提交 + 全绿 + 已过净室对抗复审（INC-5：1 CONFIRMED 已修 + 1 自查已修 + 测试补强）。**不再回炒任何增量、勿重写/勿再复审已提交代码。** -> **② FINAL —— 一次原子 feature commit**（§8；**path-whitelist `git add`，严禁 `-A`**）。白名单 = INC-1..INC-5 全部功能码 + 通用层（DEC-1）： -> - `fe/fe-connector/fe-connector-hms/`（`HmsClient`/`ThriftHmsClient`/`HmsWriteConverter`/`HmsTableInfo`/`pom.xml` 改 + `HmsAcidConstants`/`HmsCommonStatistics`/`HmsPartitionStatistics`/`HmsPartitionWithStatistics` 新 + 2 test） -> - `fe/fe-connector/fe-connector-hive/`（INC-2/3/4/5 全部新+改文件 + `pom.xml`，见 🎯 各段清单） -> - `fe/fe-connector/fe-connector-api/`（`ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator`） -> - `fe/fe-core/`（`datasource/PluginDrivenExternalTable.java`、`nereids/.../physical/PhysicalConnectorTableSink.java` + 两个对应 test） -> - **切勿混入**：`regression-test/conf/regression-conf.groovy*`、`*.bak`、`.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`、`.claude/*.js`（workflow 脚本，非仓内）、`plan-doc/reviews/...`。commit message 见 `git log` 范式 + `Co-Authored-By`/`Claude-Session`。HANDOFF/doc/port-map **另起单独 commit**（本轮已 commit 文档）。 -> - **提交前**务必 `git status` 核对暂存清单，且再跑一次 `fe-connector-hive` fresh `clean test` 确认无 stale `.class`。 +> **② cutover（P7.4/P7.5，最硬风险 R-002）** —— 参照 P5 paimon(#64446/#64653) + P6 iceberg(#64688) 翻闸+删 legacy 范式： +> - **翻闸**：把 `HIVE` 加入 `SPI_READY_TYPES`（当前 hive 天然 dormant——编译+单测但零线上路由，翻闸后才真正走插件路径）。 +> - **写链 retype**：fe-core 写链改走通用 `ConnectorWritePlanProvider` 路径（DEC-1 seam 已就位、默认关）。 +> - **读侧接线**：把 `Env.getCurrentHiveTransactionMgr()` 换成插件 `HiveReadTransactionManager`；把 `QueryFinishCallbackRegistry` 接到 `deregister`（放读锁 / 提交读事务）。 +> - **摘 legacy + 删除**：摘 legacy `HiveTransactionMgr`；删 `datasource/hive`（+hudi + 23 个 HMS-iceberg 支撑类，**守跨连接器删序**：`datasource/hive/` 删不掉直到 `HudiUtils`/`HudiScanNode`/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 等全 retype 到 generic——见文末背景段）。 +> - **硬门 = 跑 ACID 集成测试套件**（R-002 项目最大风险，含 INC-5 注留为 cutover 门的 insert-only 标记门禁的 `planScan` 级回归——需 live 读/写路径 + txn + FS 管线，**勿静默跳过**，Rule 12）。 +> - **范围内**：full-ACID **读** + insert-only **读**（INC-5 已落地）；full-ACID **写**继续硬拒（D7）。 -> **③ cutover（P7.4/P7.5，另起原子批，勿在 FINAL 混入）**：翻闸（加 `HIVE` 到 `SPI_READY_TYPES`）、fe-core 写链 retype、读侧把 `Env.getCurrentHiveTransactionMgr()` 换成插件 `HiveReadTransactionManager` + 把 `QueryFinishCallbackRegistry` 接到 `deregister`（放锁）、摘 legacy `HiveTransactionMgr`、删 `datasource/hive`（+hudi+23 HMS-iceberg 类，守跨连接器删序）、**跑 ACID 集成测试套件**（R-002 最大风险，含 insert-only 标记门禁的 planScan 级回归——INC-5 已注留为 cutover 门）。 +> **③ PR** —— cutover 完成后开 PR（base = `branch-catalog-spi`，squash），引用 tracking issue apache/doris#65185。 ## 开场要点（承接） -1. **起步先读**本文顶部 🎯 段 + 设计文档 §6/§8。**若要做 FINAL commit**：严格按 🚀②的白名单 path-whitelist add，提交前 `git status` 核对 + fresh `clean test`。 -2. **INC-1..INC-5 已全部完成 + 全绿 + 复审，勿重写/勿再复审现有代码**。全批仍 UNCOMMITTED 在工作树（原子批 WIP 载体，勿 `git checkout`/`revert`）。 -3. **范围锁定（勿重议）**：hive **不在** `SPI_READY_TYPES`（整批天然 dormant，编译+单测但零线上路由）；翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 读侧放锁接线 / 删 legacy **均属后续 P7.4/P7.5，另起原子批**（见 🚀③）。full-ACID **写**继续硬拒（D7），full-ACID + insert-only **读**在范围（INC-5，已落地）。 -4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 读/写路径 → P7.4/P7.5 翻闸时跑，勿静默跳过——Rule 12）。 -5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 +1. **起步先读**本文顶部 🎯 段 + 设计文档 §6/§8 + P5/P6 翻闸样板。cutover 是**新原子批**，勿与已提交的 FINAL 混淆。 +2. **INC-1..INC-5 已全部提交（`0b19506acfe`）+ 全绿 + 复审，勿重写/勿再复审已提交代码。** 工作树若还有零散 scratch（`.bak`/`regression-conf`/`.audit-scratch/` 等）均为历史遗留、非本线程产物，勿混入任何 commit。 +3. **范围锁定（勿重议）**：翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 读侧放锁接线 / 删 legacy = cutover（P7.4/P7.5）本体。full-ACID **写**继续硬拒（D7），full-ACID + insert-only **读**在范围（INC-5，已落地）。 +4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 读/写路径 → cutover 翻闸时跑，勿静默跳过——Rule 12）。 +5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）；path-whitelist add，严禁 `-A`。 --- From 823ee07863c24f90e03ab29d3a801f96ea1400ab Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 07:43:03 +0800 Subject: [PATCH 027/334] [feat](catalog) P7.4: per-table scan-provider selection seam (multi-format dispatch keystone, dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add Connector.getScanPlanProvider(ConnectorTableHandle) so one connector can select a different ConnectorScanPlanProvider per table. Selection MUST happen at provider-acquisition time (not inside a single dispatching provider) because ConnectorScanPlanProvider has handle-less methods (e.g. appendExplainInfo) and providers are built fresh/stateless per call — a returned provider must already be bound to the right backing scanner for the handle. fe-core: PluginDrivenScanNode routes all 11 provider look-ups through a single resolveScanProvider() helper keyed on currentHandle. The SPI default ignores the handle and returns the connector-level provider, so every existing connector (es/jdbc/trino/max_compute/ paimon/iceberg/hive) is byte-identical and this change is fully dormant (no connector overrides it yet). This is the keystone that lets a post-cutover heterogeneous hms catalog (plain-hive + iceberg-on-HMS + hudi-on-HMS under one gateway connector) route each table to the right scanner; the first real delegation lands in a later sub-batch. Tests: ConnectorScanProviderSelectionTest (SPI default delegation, null-passthrough, per-handle override); PluginDrivenScanNodeScanProviderSelectionTest (node passes currentHandle). Updated PluginDrivenScanNodeDeleteFilesTest/BatchModeTest to stub the arg overload (bare Mockito mocks do not run the real default method). fe-connector-api 58 tests, fe-core PluginDrivenScanNode* 76 tests, checkstyle 0 (api + core), import-gate clean. Refs: apache/doris#65185 Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../apache/doris/connector/api/Connector.java | 20 +++ .../ConnectorScanProviderSelectionTest.java | 141 ++++++++++++++++++ .../datasource/PluginDrivenScanNode.java | 35 +++-- .../PluginDrivenScanNodeBatchModeTest.java | 5 +- .../PluginDrivenScanNodeDeleteFilesTest.java | 5 +- ...ivenScanNodeScanProviderSelectionTest.java | 69 +++++++++ 6 files changed, 262 insertions(+), 13 deletions(-) create mode 100644 fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProviderSelectionTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index a1a1811dc34f5d..4ecddcd8802309 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.api; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; @@ -47,6 +48,25 @@ default ConnectorScanPlanProvider getScanPlanProvider() { return null; } + /** + * Returns the scan plan provider for the given table, allowing one connector to select a + * different provider per table. + * + *

    The selection MUST happen here, at provider-acquisition time — not inside a single + * dispatching provider — because {@link ConnectorScanPlanProvider} has methods that do not + * carry the handle (e.g. {@code appendExplainInfo}) and providers are built fresh/stateless + * per call, so a provider returned here must already be bound to the correct backing scanner + * for {@code handle}. This is the seam a heterogeneous gateway connector (one catalog serving + * multiple table formats) overrides to delegate to per-format sub-providers by the concrete + * (connector-defined) handle type; the engine never inspects the format.

    + * + *

    The default ignores {@code handle} and returns the connector-level + * {@link #getScanPlanProvider()}, so every single-format connector is unaffected.

    + */ + default ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + return getScanPlanProvider(); + } + /** * Returns the write plan provider for sink ({@code TDataSink}) generation, * or {@code null} if this connector does not support writes. diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java new file mode 100644 index 00000000000000..b03af7599d0ce5 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java @@ -0,0 +1,141 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanRange; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Optional; + +/** + * Pins the per-table scan-provider selection seam + * {@link Connector#getScanPlanProvider(ConnectorTableHandle)}. + * + *

    WHY this matters (Rule 9): after the hive/hms cut-over a single catalog is heterogeneous + * (plain-hive + iceberg-on-HMS + hudi-on-HMS under one gateway connector). The engine must pick a + * DIFFERENT scan provider per table without ever inspecting the format itself. The selection must happen + * here — at provider-acquisition time — rather than inside a single dispatching provider, because + * {@link ConnectorScanPlanProvider} has handle-less methods (e.g. {@code appendExplainInfo}) and providers + * are built fresh/stateless per call, so a returned provider must already be bound to the right backing + * scanner for the handle. The default must stay inert for every single-format connector.

    + */ +public class ConnectorScanProviderSelectionTest { + + /** A scan provider identified by name so tests can assert WHICH provider was selected. */ + private static final class NamedProvider implements ConnectorScanPlanProvider { + private final String name; + + private NamedProvider(String name) { + this.name = name; + } + + @Override + public List planScan(ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter) { + return Collections.emptyList(); + } + + @Override + public String toString() { + return name; + } + } + + /** Bare connector implementing only the abstract methods; nothing scan-related is overridden. */ + private abstract static class FakeConnector implements Connector { + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public void close() { + } + } + + private static ConnectorTableHandle handle() { + return new ConnectorTableHandle() { + }; + } + + @Test + public void defaultDelegatesToNoArgProvider() { + // A single-format connector overrides only the no-arg getter. The per-handle default must delegate + // to it (NOT return null), so every existing connector routes unchanged after the seam is added. + // MUTATION: making the default return null instead of getScanPlanProvider() -> non-null assert red. + NamedProvider only = new NamedProvider("only"); + Connector connector = new FakeConnector() { + @Override + public ConnectorScanPlanProvider getScanPlanProvider() { + return only; + } + }; + + Assertions.assertSame(only, connector.getScanPlanProvider(handle()), + "the per-handle default must delegate to the connector-level no-arg provider"); + } + + @Test + public void defaultReturnsNullWhenConnectorHasNoScanCapability() { + // A connector with no scan capability (no-arg default returns null) must keep returning null through + // the per-handle seam, preserving the null-tolerant scan path in PluginDrivenScanNode. + Connector connector = new FakeConnector() { + }; + + Assertions.assertNull(connector.getScanPlanProvider(handle()), + "with no scan provider at all the per-handle seam stays null"); + } + + @Test + public void overrideSelectsProviderPerHandle() { + // A heterogeneous gateway overrides the per-handle seam and returns a DIFFERENT provider per table, + // and must NOT fall back to the no-arg provider once it has an answer for the handle. + // MUTATION: keying the override on the no-arg getter (ignoring the handle) -> per-handle assert red. + ConnectorTableHandle icebergHandle = handle(); + ConnectorTableHandle hiveHandle = handle(); + NamedProvider icebergProvider = new NamedProvider("iceberg"); + NamedProvider hiveProvider = new NamedProvider("hive"); + NamedProvider fallback = new NamedProvider("fallback"); + Connector gateway = new FakeConnector() { + @Override + public ConnectorScanPlanProvider getScanPlanProvider() { + return fallback; + } + + @Override + public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + return handle == icebergHandle ? icebergProvider : hiveProvider; + } + }; + + Assertions.assertSame(icebergProvider, gateway.getScanPlanProvider(icebergHandle), + "gateway routes the iceberg-on-HMS handle to the iceberg provider"); + Assertions.assertSame(hiveProvider, gateway.getScanPlanProvider(hiveHandle), + "gateway routes the plain-hive handle to the hive provider"); + Assertions.assertNotSame(fallback, gateway.getScanPlanProvider(icebergHandle), + "an overriding gateway does not fall back to the connector-level no-arg provider"); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 14361b85e407d3..dd4ddad8c721e8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -196,6 +196,19 @@ public static PluginDrivenScanNode create(PlanNodeId id, TupleDescriptor desc, scanContext, connector, session, handle); } + /** + * Resolves the scan plan provider for the table being scanned, allowing a heterogeneous gateway + * connector to select a per-table (per-format) provider. Keyed on {@link #currentHandle} — the + * same handle the rest of the scan uses (pushdown may refine it, but a table's format, and thus + * the selected provider, does not change across a scan). For every single-format connector the + * SPI default ignores the handle and returns the connector-level provider, so this stays + * byte-identical to the former {@code connector.getScanPlanProvider()} and may still be + * {@code null} for connectors without scan capability. + */ + private ConnectorScanPlanProvider resolveScanProvider() { + return connector.getScanPlanProvider(currentHandle); + } + /** * Injects the Nereids partition-pruning result. Called by the translator so the pruned * partition set can be pushed down to the connector's scan plan (see {@link #getSplits}). @@ -372,7 +385,7 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { // keys, so a connector that distinguishes native/JNI reads (paimon) can emit its // "paimonNativeReadSplits=/" line without an SPI signature change. The copy keeps // the cached scanNodeProperties unpolluted; non-paimon providers ignore the extra keys. - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider != null) { Map explainProps = new HashMap<>(props); explainProps.put(NATIVE_READ_SPLITS_KEY, String.valueOf(nativeReadSplitNum)); @@ -472,7 +485,7 @@ protected TColumnCategory classifyColumn(SlotDescriptor slot, List parti * A connector with no scan provider (no scan capability) contributes no special columns ({@code DEFAULT}). */ ConnectorColumnCategory classifyColumnByConnector(String columnName) { - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider == null) { return ConnectorColumnCategory.DEFAULT; } @@ -521,7 +534,7 @@ private static T onPluginClassLoader(ConnectorScanPlanProvider provider, Sup */ @Override protected List getDeleteFiles(TFileRangeDesc rangeDesc) { - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider == null || rangeDesc == null || !rangeDesc.isSetTableFormatParams()) { return Collections.emptyList(); } @@ -882,7 +895,7 @@ void checkSysTableScanConstraints() throws UserException { * stays unit-testable on a Mockito mock without a live connector (mirrors the guard's own visibility). */ boolean sysTableSupportsTimeTravel() { - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider == null) { return false; } @@ -895,7 +908,7 @@ public List getSplits(int numBackends) throws UserException { // Attempt limit and projection pushdown via SPI protocol tryPushDownLimit(); - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider == null) { LOG.warn("Connector does not provide a scan plan provider, returning empty splits"); return Collections.emptyList(); @@ -1039,7 +1052,7 @@ public boolean isBatchMode() { private boolean computeBatchMode() { // getScanPlanProvider() may be null for connectors without scan capability; mirror the // null-guard in getSplits() so isBatchMode (run on the dispatch + explain paths) never NPEs. - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider == null) { return false; } @@ -1171,7 +1184,7 @@ public void startSplit(int numBackends) { // Scope the scan to a distributed rewrite group's files (no-op for every non-rewrite read). pinRewriteFileScope(); final ConnectorTableHandle handle = currentHandle; - final ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + final ConnectorScanPlanProvider scanProvider = resolveScanProvider(); final List allPartitions = new ArrayList<>(selectedPartitions.selectedPartitions.keySet()); final int batchSize = sessionVariable.getNumPartitionsInBatchMode(); @@ -1257,7 +1270,7 @@ private void startStreamingSplit() { } pinRewriteFileScope(); final ConnectorTableHandle handle = currentHandle; - final ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + final ConnectorScanPlanProvider scanProvider = resolveScanProvider(); Executor scheduleExecutor = Env.getCurrentEnv().getExtMetaCacheMgr().getScheduleExecutor(); CompletableFuture.runAsync(() -> { ConnectorSplitSource source = null; @@ -1310,7 +1323,7 @@ protected void setScanParams(TFileRangeDesc rangeDesc, Split split) { @Override protected Optional getSerializedTable() { - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider != null) { Map props = getOrLoadScanNodeProperties(); String serialized = onPluginClassLoader(scanProvider, () -> scanProvider.getSerializedTable(props)); @@ -1325,7 +1338,7 @@ protected Optional getSerializedTable() { public void createScanRangeLocations() throws UserException { super.createScanRangeLocations(); // Delegate scan-level Thrift params to the connector SPI - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider != null) { Map props = getOrLoadScanNodeProperties(); onPluginClassLoader(scanProvider, () -> { @@ -1407,7 +1420,7 @@ private void pruneConjunctsFromNodeProperties() { */ private ScanNodePropertiesResult getOrLoadPropertiesResult() { if (cachedPropertiesResult == null) { - ConnectorScanPlanProvider scanProvider = connector.getScanPlanProvider(); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); if (scanProvider != null) { List columns = buildColumnHandles(); Optional filter = buildRemainingFilter(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java index 59e7de941a3c10..9adc6e2315774f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java @@ -144,7 +144,10 @@ public void testAboveThresholdBatches() { private static PluginDrivenScanNode streamingNode(ConnectorScanPlanProvider provider) { PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS); Connector connector = Mockito.mock(Connector.class); - Mockito.when(connector.getScanPlanProvider()).thenReturn(provider); + // The node resolves the provider PER TABLE via getScanPlanProvider(currentHandle); a real connector + // delegates that overload to the no-arg getter, so the mock answers the arg form (currentHandle is + // null in this partial node, hence the null-tolerant any() matcher). + Mockito.when(connector.getScanPlanProvider(Mockito.any())).thenReturn(provider); Deencapsulation.setField(node, "connector", connector); // hasSlots = true (the streaming gate requires output slots). getSlots() returns ArrayList (concrete). TupleDescriptor desc = Mockito.mock(TupleDescriptor.class); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeDeleteFilesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeDeleteFilesTest.java index 634c1ec6e5fa59..5683b3ec2c4864 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeDeleteFilesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeDeleteFilesTest.java @@ -51,7 +51,10 @@ private static PluginDrivenScanNode nodeWithProvider(ConnectorScanPlanProvider p PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS); Connector connector = Mockito.mock(Connector.class); - Mockito.when(connector.getScanPlanProvider()).thenReturn(provider); + // The node resolves the provider PER TABLE via getScanPlanProvider(currentHandle); a real connector + // delegates that overload to the no-arg getter, so the mock answers the arg form (currentHandle is + // null in this partial node, hence the null-tolerant any() matcher). + Mockito.when(connector.getScanPlanProvider(Mockito.any())).thenReturn(provider); Deencapsulation.setField(node, "connector", connector); return node; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProviderSelectionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProviderSelectionTest.java new file mode 100644 index 00000000000000..be844025582122 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProviderSelectionTest.java @@ -0,0 +1,69 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +/** + * Guards that {@link PluginDrivenScanNode} resolves its scan plan provider PER TABLE — passing the table's + * {@code currentHandle} to {@link Connector#getScanPlanProvider(ConnectorTableHandle)} — so a heterogeneous + * gateway connector can route each table to the right backing scanner. + * + *

    WHY this matters (Rule 9): before this seam the node called the no-arg + * {@code connector.getScanPlanProvider()} for every table, so one catalog could expose exactly ONE scan + * provider. All provider look-ups now route through {@code resolveScanProvider()} keyed on the handle; a + * mutant that drops the handle (reverts to the no-arg getter) would send an iceberg-on-HMS table to the hive + * scanner and return wrong/empty rows. Driven on a partial ({@code CALLS_REAL_METHODS}) node with only + * {@code connector}/{@code currentHandle} injected — the same technique as + * {@code PluginDrivenScanNodeVerboseExplainTest}.

    + */ +public class PluginDrivenScanNodeScanProviderSelectionTest { + + @Test + public void resolvesProviderForCurrentHandle() { + PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class, Mockito.CALLS_REAL_METHODS); + + ConnectorTableHandle icebergHandle = Mockito.mock(ConnectorTableHandle.class); + ConnectorTableHandle hiveHandle = Mockito.mock(ConnectorTableHandle.class); + ConnectorScanPlanProvider icebergProvider = Mockito.mock(ConnectorScanPlanProvider.class); + ConnectorScanPlanProvider hiveProvider = Mockito.mock(ConnectorScanPlanProvider.class); + + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getScanPlanProvider(icebergHandle)).thenReturn(icebergProvider); + Mockito.when(connector.getScanPlanProvider(hiveHandle)).thenReturn(hiveProvider); + Deencapsulation.setField(node, "connector", connector); + + // The provider is selected by whichever handle the scan currently holds (pushdown may refine it). + Deencapsulation.setField(node, "currentHandle", icebergHandle); + Assertions.assertSame(icebergProvider, Deencapsulation.invoke(node, "resolveScanProvider"), + "the node must resolve the provider for the iceberg-on-HMS handle it is scanning"); + + // After the handle changes the node must re-resolve to the matching provider (per-table routing), + // not cache the first one. + Deencapsulation.setField(node, "currentHandle", hiveHandle); + Assertions.assertSame(hiveProvider, Deencapsulation.invoke(node, "resolveScanProvider"), + "after the handle changes the node must resolve the matching provider (per-table routing)"); + } +} From c811170431cad97882d866b8cf460921051728bf Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 07:45:38 +0800 Subject: [PATCH 028/334] [doc](catalog) P7.4 sub-batch A DONE (per-table scan-provider seam) + decompose cutover into 6 sub-batches + refresh HANDOFF Land the cutover scope map (P7-cutover-scope-map-2026-07-06.md, from the 6-agent recon wf_536a2968-2c8): the "cutover" is not a compact flip but the entire P7 tail decomposed into sub-batches A-F (multi-format dispatch seam / iceberg-on-HMS delegation / hudi live cutover / event pipeline / DLA+write retype / flip+delete). Land sub-batch A design doc (P7.4-scan-provider-per-table-seam-design.md), marked DONE for commit 0923077fe67. Refresh HANDOFF: current state = A done + dormant, next = sub-batch B (iceberg-on-HMS delegation), with the delete-order constraint and each sub-batch's deferred user decisions recorded. Refs: apache/doris#65185 Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 79 ++++++++----------- .../tasks/P7-cutover-scope-map-2026-07-06.md | 42 ++++++++++ ...7.4-scan-provider-per-table-seam-design.md | 58 ++++++++++++++ 3 files changed, 135 insertions(+), 44 deletions(-) create mode 100644 plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md create mode 100644 plan-doc/tasks/P7.4-scan-provider-per-table-seam-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 705a87a9b1433d..76c493c576dc18 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,72 +5,63 @@ --- -# 🎯 当前状态（2026-07-06）= **P7.3 原子批 INC-1..INC-5 的 FINAL 原子 feature commit 已落地 = `0b19506acfe`（40 文件：22 新 + 18 改，+7456/−34）。写链 + 读链全部完成、全绿、且已提交。工作树不再持有本批未提交功能码（原子批 WIP 阶段结束）。下一步 = cutover（P7.4/P7.5，另起原子批）：翻闸 `SPI_READY_TYPES` + fe-core 写链 retype + 读侧放锁接线 + 摘 legacy `HiveTransactionMgr` + 删 `datasource/hive`（守跨连接器删序）+ 跑 ACID 集成测试套件（R-002 项目最大风险）。** +# 🎯 当前状态（2026-07-07） -> **本轮做了什么（FINAL：一次原子 feature commit，已落地）**：按白名单 path-whitelist `git add`（api 3 + hms 11 + hive 22 + fe-core 4 = 40；`git add /src` 仅收该模块 src 下 wanted 文件，`target/` 已 gitignore）→ 暂存后 `git diff --cached` 自检 = 正好 **40/40（22 A + 18 M）**，无一文件落在 4 白名单区域外、无 `.bak`/`regression-conf` 明文 key/`.audit-scratch`/`conf.cmy`/`META-INF`/`docker`/`.claude`/`plan-doc` 混入 → commit `0b19506acfe`（message 沿用 `[feat](catalog) P7.3 hive …` 范式 + 三段模块摘要 + 验证数据 + `Refs: apache/doris#65185` + `Co-Authored-By`/`Claude-Session` 尾）。**核实**：`FakeConnectorContext`/`HmsPartitionInfo` 是既有已跟踪共享类（被既有 hudi/hive 测试引用），非本批新增（HANDOFF 旧叙述误记为"新"）→ 本批自洽、单独 checkout 白名单集即可编译。 +**插件侧写/读能力已全部提交（休眠）= `0b19506acfe`**（hive/hms 写事务 + 提交器 + 写计划 + 读侧 ACID 生产者 + 读事务管理器；hive 尚未进 `SPI_READY_TYPES`，线上零路由）。P7.1（DDL 元数据）也已提交。 -> **本轮验证（独立重跑、不信自述、提交前 fresh 编译杜绝 stale `.class`）**：`fe-connector-hive` `clean test` **104 测试 0 失败 0 错误 0 跳过 BUILD SUCCESS**；`fe-connector-hms` 49、`fe-connector-api` 55 全绿；`fe-core` 从零编译 4560 源 + 受影响 2 测试类（`ConnectorContractValidatorTest` 7 + `PhysicalConnectorTableSinkTest` 9 = **16** 0 失败 0 跳过）+ checkstyle 0；连接器 3 模块 checkstyle 0；`check-connector-imports.sh` 净（仅命中 `HiveVersionUtil` 已知误报）。 +**关键校正（勿再误读）**：交接旧叙述把"下一步 = cutover"写成一次轻量翻闸，**实为错**。对照 HEAD 的 6-agent 侦察（`wf_536a2968-2c8`）证实：剩余"cutover"是**整个 P7 尾段、全项目最大最险的一块**，已拆成 **6 个子批 A–F**，完整地图 + 每子批开放决策见 **`plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md`**（起步必读）。 -> **✅ INC-1..INC-5 已全部提交进 `0b19506acfe`**（不再是 uncommitted 工作树 WIP；原子批载体阶段结束）。逐增量移植记录留档备查：INC-3=`tasks/P7.3-INC-3-portmap.md`、INC-4=`tasks/P7.3-INC-4-portmap.md`、INC-5=`tasks/P7.3-INC-5-portmap.md`；各 INC 的文件清单见 `git show --stat 0b19506acfe`。 - -> **权威实现依据**（信 HEAD 控制流，不信行号）： -> - `tasks/P7.3-hive-write-txn-implementation-design.md`（§2 决策 D1–D12、§4 签名、§5 移植细节、§6 构建顺序）。 -> - **`tasks/P7.3-INC-3-portmap.md`（595 行）**：逐行移植地图 + §7 12 处 GAP + **§9 已核实的实现前检查**（GAP-11/12 结论、D6 MPU map 重载确认、FS 构建/鉴权/iceberg 模板签名全确认）。INC-4/INC-5 也可续用其读侧 §5.4 + 测试计划 §6。 -> - 移植源 = HEAD `fe/fe-core/.../datasource/hive/HMSTransaction.java`；模板 = `fe-connector-iceberg/.../IcebergConnectorTransaction.java`+`IcebergWriteContext.java`。 +**子批清单 + 进度**（依赖序）： +- **A（多格式分发地基 / 按表选 scan-provider 缝）—— ✅ DONE `0923077fe67`（2026-07-07）**。加 `Connector.getScanPlanProvider(ConnectorTableHandle)` 默认委派无参版；`PluginDrivenScanNode` 11 个取 provider 点收敛到 `resolveScanProvider()`（按 `currentHandle`）。**休眠**、对现有 7 连接器逐字不变。设计 = `plan-doc/tasks/P7.4-scan-provider-per-table-seam-design.md`。验证：api 58 测 + core `PluginDrivenScanNode*` 76 测 + checkstyle 0 + import-gate 净。用户签字：纯地基、放 `Connector`（非 `ConnectorMetadata`）、tableFormatType threading 延后。 +- **B（iceberg-on-HMS 委派）—— ⏭ 下一步**。把"Hive 元存储上的 Iceberg 表"经 hms 网关按表委派给 iceberg 连接器（用 A 的缝）；补按列统计 SPI（保 `enableFetchIcebergStats`）；抽 `IcebergUtils` 纯 helper 使 fe-core 去 iceberg-SDK。骑在 A 上。 +- **C（hudi 上线切换）**：搬 `datasource/hudi`(15 文件)入 `fe-connector-hudi`；拆 `HudiDlaTable`/`HudiScanNode extends HiveScanNode`；延后特性（增量/schema-evo/time-travel）移植或保持 fail-loud。骑在 A 上。 +- **D（事件管道搬迁）**：21 事件类 + processor 搬 `fe-connector-hms`；中立 `ConnectorMetaInvalidator`；fe-core 留薄 role-aware driver；R-010 线程生命周期 + TCCL。**独立、可并行**。 +- **E（表类改造 + 写链改造 + 读事务接线）**：中立化 42 `instanceof HMSExternal*` + 22 `dlaType`；`HMSExternalTable`→generic；删旧 hive sink（6-8 文件）；`ConnectorSession` 加 query-finish 缝；`Env` 摘 `HiveTransactionMgr`；`HMSAnalysisTask` 中立；搬 `BIND_BROKER_NAME`。骑在 A + 部分 D。 +- **F（翻闸 + GSON 兼容 + 删旧 + 硬门）**：`SPI_READY_TYPES` 加 `"hms"`；`GsonUtils` 三工厂 `registerCompatibleSubtype`；分批跨连接器删 `datasource/hive`+`hudi`+23 iceberg 类；净室对抗复审 + 能力孪生审计 + GSON replay 测 + **docker ACID/事件/异构 e2e 硬门（R-002）** + 用户二签。**严格最后**。 --- -# 🚀 下个 session 任务 = **cutover（P7.4/P7.5，另起原子批）：把 dormant 的 hive 插件真正接上线** - -> **① FINAL —— ✅ 已落地**（commit `0b19506acfe`）。INC-1..INC-5 全部提交 + 全绿 + 已过净室对抗复审（INC-5：1 CONFIRMED 已修 + 1 自查已修 + 测试补强）。**不再回炒任何增量、勿重写/勿再复审已提交代码。** +# 🚀 下个 session 任务 = **子批 B（iceberg-on-HMS 委派）** -> **② cutover（P7.4/P7.5，最硬风险 R-002）** —— 参照 P5 paimon(#64446/#64653) + P6 iceberg(#64688) 翻闸+删 legacy 范式： -> - **翻闸**：把 `HIVE` 加入 `SPI_READY_TYPES`（当前 hive 天然 dormant——编译+单测但零线上路由，翻闸后才真正走插件路径）。 -> - **写链 retype**：fe-core 写链改走通用 `ConnectorWritePlanProvider` 路径（DEC-1 seam 已就位、默认关）。 -> - **读侧接线**：把 `Env.getCurrentHiveTransactionMgr()` 换成插件 `HiveReadTransactionManager`；把 `QueryFinishCallbackRegistry` 接到 `deregister`（放读锁 / 提交读事务）。 -> - **摘 legacy + 删除**：摘 legacy `HiveTransactionMgr`；删 `datasource/hive`（+hudi + 23 个 HMS-iceberg 支撑类，**守跨连接器删序**：`datasource/hive/` 删不掉直到 `HudiUtils`/`HudiScanNode`/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 等全 retype 到 generic——见文末背景段）。 -> - **硬门 = 跑 ACID 集成测试套件**（R-002 项目最大风险，含 INC-5 注留为 cutover 门的 insert-only 标记门禁的 `planScan` 级回归——需 live 读/写路径 + txn + FS 管线，**勿静默跳过**，Rule 12）。 -> - **范围内**：full-ACID **读** + insert-only **读**（INC-5 已落地）；full-ACID **写**继续硬拒（D7）。 +> 起步先读 `P7-cutover-scope-map-2026-07-06.md`（§B 段 + 删序约束）+ A 的设计文档（缝的契约）。B 是**首个真实委派**，验证 A 的地基。 -> **③ PR** —— cutover 完成后开 PR（base = `branch-catalog-spi`，squash），引用 tracking issue apache/doris#65185。 +- **做什么**：hms 网关连接器（`fe-connector-hive`）**依赖 `fe-connector-iceberg`**；其 `getScanPlanProvider(handle)` 覆写，对 iceberg-on-HMS 句柄委派给 iceberg 连接器的 scan provider（普通 hive 表仍走 hive provider）。网关需在 `getTableHandle` 探测格式并把它编进**自己的**句柄子类（判别标识是**连接器内部**事，fe-core 不看格式）。 +- **连带（B 范围）**：补 `ConnectorStatisticsOps` 按列统计方法 + iceberg 实现（保 `enableFetchIcebergStats` 快路径）；抽 `IcebergUtils` 纯 helper（`isIcebergRowLineageColumn`/`getEffectiveIcebergFormatVersion`/行血缘常量）到 SDK-free util，使删 23 类后 fe-core 去 iceberg-SDK。 +- **删序**：`IcebergHMSSource`/`IcebergScanNode`/`IcebergSysTable`/`StatisticsUtil.getIcebergColumnStats`/`HMSAnalysisTask` 等**retype 到 generic 后**才能删 `datasource/iceberg` 23 类；`datasource/hive` 要等 B + C 都完成（HudiScanNode/HudiUtils 也解绑）。 +- **B 的开放决策（recon 后用户签字，先中文讲背景+示例+推荐，不引任务代号）**：格式判别标识形态（枚举 vs 串 vs capability，连接器内部）；按列统计 SPI vs 放弃 per-column HMS-iceberg 统计；`IcebergUtils` 抽纯 helper vs 留 SDK-linked shrunk 版。 ## 开场要点（承接） - -1. **起步先读**本文顶部 🎯 段 + 设计文档 §6/§8 + P5/P6 翻闸样板。cutover 是**新原子批**，勿与已提交的 FINAL 混淆。 -2. **INC-1..INC-5 已全部提交（`0b19506acfe`）+ 全绿 + 复审，勿重写/勿再复审已提交代码。** 工作树若还有零散 scratch（`.bak`/`regression-conf`/`.audit-scratch/` 等）均为历史遗留、非本线程产物，勿混入任何 commit。 -3. **范围锁定（勿重议）**：翻闸 / fe-core 写链 retype / 摘 `HiveTransactionMgr` / 读侧放锁接线 / 删 legacy = cutover（P7.4/P7.5）本体。full-ACID **写**继续硬拒（D7），full-ACID + insert-only **读**在范围（INC-5，已落地）。 -4. **硬门 = ACID 集成测试套件**（R-002 项目最大风险，需 live 读/写路径 → cutover 翻闸时跑，勿静默跳过——Rule 12）。 -5. **纪律**：每轮完成即更新本 HANDOFF + commit（memory `handoff-discipline-per-phase`）；上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）；path-whitelist add，严禁 `-A`。 +1. **起步先读** `P7-cutover-scope-map-2026-07-06.md`（6 子批全图 + 删序约束 + 每子批开放决策）+ 本文顶部 🎯 段。cutover ≠ 一次翻闸，是 A–F 六批。 +2. **A 已提交（`0923077fe67`）+ 全绿 + 休眠，勿重写/勿再复审**。P7.1 + P7.3（`0b19506acfe`）同样已提交、勿回炒。 +3. **纪律**：每子批 = 先 code-grounded recon → 设计文档 → 用户签字该批专属决策 → 实现 → 独立 commit → 更新本 HANDOFF。上下文超 30% 找干净节点交接。**path-whitelist `git add`，严禁 `-A`**。 +4. **硬门 = ACID/事件/异构集成测试**在 F（R-002 最大风险，需 live 路径，勿静默跳过——Rule 12）。full-ACID **写**继续硬拒；full-ACID + insert-only **读**在范围（已落地插件侧）。 --- # 📦 分支 / Commit 须知 -- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**（复用 P5-T29 #64653 / P6 #64688 范式）。 -- **公开 tracking issue = apache/doris#65185**（catalog-SPI 迁移 umbrella）；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 -- **⚠️ path-whitelist `git add`，严禁 `git add -A`**（scrub `regression-test/conf/regression-conf.groovy` 明文 key + `*.bak` + scratch `.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...` + `plan-doc/reviews/P5-paimon-rereview3-*`；`.claude/` 是 memory、非仓内）。 -- **⚠️ 本批 FINAL 原子 commit 的白名单现已跨通用层**（DEC-1，用户 2026-07-06 授权）：除 `fe-connector-hms`/`fe-connector-hive` 外，还含 `fe-connector-api`（`ConnectorWritePlanProvider`/`Connector`/`ConnectorContractValidator`）+ `fe-core`（`datasource/PluginDrivenExternalTable.java`、`nereids/.../physical/PhysicalConnectorTableSink.java` + 两个对应 test）。设计文档 §4.5"fe-core ZERO changes"就此**作废**（改为一处通用、connector-agnostic、默认关的新能力）。 -- commit message：见 `git log` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 任务清单 + 设计文档 + port-map 单独 commit。 +- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**。**打包/复审策略（flip 前/后、单 PR vs 分 PR）= F 的开放决策**（paimon 分 PR vs iceberg 合并 squash 两先例）。 +- **公开 tracking issue = apache/doris#65185**；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 +- **⚠️ path-whitelist `git add`，严禁 `git add -A`**（工作树有历史遗留 scratch：`*.bak`·`regression-test/conf/regression-conf.groovy` 明文 key·`.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`·`plan-doc/reviews/P5-*`·`.claude/`——均**非本线程产物，勿混入任何 commit**）。 +- commit message：`[feat](catalog) P7.x …` / `[doc](catalog) …` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每子批/每条 fix = 独立 commit**；HANDOFF + 设计文档单独 commit（与 code 分开）。A 已是普通独立 commit（非 P7.3 那种原子批）。 # ⚙️ 操作须知（复用） -- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**checkstyle 别加 `-am`**：`mvn -pl : checkstyle:check`。 -- **⚠️ bash 工具默认 timeout 120s**：fe-connector-hive 全模块 build/test ~2min → 调 `timeout` ~580000ms。**后台 task 通知的 "exit code" 是末尾 echo 的、非 maven 的**——读 LOG 的 `BUILD SUCCESS` 行或 surefire XML（`Tests run=/Failures=/Errors=`），别信通知 exit。maven 经管道 `$?` 是管道尾的 → grep `BUILD SUCCESS`。**改代码后 commit 前务必 fresh recompile**（stale `.class` 假红）。 -- **连接器测试无 Mockito**（真 recording fakes；本轮 `HiveConnectorTransactionTest` 即用手写 recording `HmsClient` fake + `FakeConnectorContext`，`HmsClient` 多数 Phase-3+ 方法 default-throw、fake 只覆盖用到的）。checkstyle **禁 static import**（用 `Assertions.assertX`）、**扫 test 源**。 -- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 -- **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 紧**（2.0T ~82% used，360G free）；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够；本轮实现 agent 即在主树工作）。 +- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**checkstyle 别加 `-am`**：`mvn -pl : checkstyle:check`。artifactId：`fe-core` / `fe-connector-api` / `fe-connector-hive` / `fe-connector-hms` / `fe-connector-iceberg` / `fe-connector-hudi`。 +- **⚠️ bash 工具默认 timeout 120s**：fe-core 全量编译 > 2min → **务必**把 `timeout` 调到 ~580000ms（曾因忘调而被 SIGTERM 打断）。**后台/管道 exit 不可信**——读 LOG 的 `BUILD SUCCESS` 行 + surefire `Tests run=/Failures=/Errors=`。改代码后 commit 前 fresh recompile 杜绝 stale `.class`。 +- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**。⚠️ `Mockito.mock(接口)` **不跑 default 方法**（会返 null）——给接口加 default 方法后，凡 mock 该接口并只 stub 旧方法的测试都要补 stub 新方法（A 即遇此：`getScanPlanProvider(handle)` 在 bare mock 上返 null，须 stub arg 重载）。checkstyle **禁 static import**、**扫 test 源**。 +- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`（**repo 根**，非 `fe/`）。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory）。 +- **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 紧**（~82% used）；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 # 🧠 起步必读 -1. 本文档顶部 🎯/🚀 段 + `tasks/P7.3-INC-3-portmap.md` + 设计文档 `tasks/P7.3-hive-write-txn-implementation-design.md` §6。 -2. **样板**：`tasks/P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`tasks/P6-iceberg-migration.md`（阶段拆分范式）。模板事务 = `IcebergConnectorTransaction`/`IcebergWriteContext`。 -3. **铁律**：fe-core 不得新增 `if(hive)`/`instanceof HMSExternal*`/引擎名判别；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；插件跨边界须 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 -4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`memory-keep-only-general-or-requested`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-plugin-tccl-classloader-gotcha`、`catalog-spi-hms-hiveversionutil-gate-false-positive`、`catalog-spi-tracking-issue`。 +1. **`plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md`**（6 子批全图 + keystone + 删序 + 每子批开放决策 + Trino 对比）+ 本文顶部 🎯/🚀。 +2. **样板**：`P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`（净室复审 + 能力孪生审计 + GSON replay 范式）。委派/缝模板 = 本轮 A 的设计文档。 +3. **铁律**：fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/`switch(dlaType)`/引擎名判别（新缝走中立 SPI / ConnectorCapability）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；插件跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`）；history_schema_info nested 名 lowercase（memory）。 +4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-tracking-issue`。 --- -## 背景：#64688（P6 iceberg 收官，已合入 `branch-catalog-spi`）+ P7 关键遗留 - -整条 catalog-SPI 主线阶段链均已合入 upstream `branch-catalog-spi`：P0 #63582 · P1 #63641 · P2 trino #64096 · P3 hudi #64143 · P4 maxcompute #64300 · P5 paimon #64446+#64653 · P3b kerberos #64655 · **P6 iceberg #64688**。#64688 把原生 iceberg 整体迁到自包含 `fe-connector-iceberg` + 翻闸 + 删 fe-core 原生 iceberg 子系统 + 属性/鉴权全归插件（用户 2026-07-05 架构裁定，memory `catalog-spi-no-property-parsing-in-fecore`）。 +## 背景：跨连接器删除排序（**F 最硬约束**，B/C 为其解绑） -**⚠️ P7 必须接手的遗留**：`#64688` 删的是原生 iceberg；但 **iceberg-on-HMS**（`type=hms` 下 `DlaType.ICEBERG`）仍走 fe-core，故 fe-core `datasource/iceberg/` 还**故意保活 23 个 HMS-iceberg 支撑类**（`IcebergUtils`/`IcebergMetadataOps`/`source/IcebergScanNode`+…/`cache/`/`IcebergMvccSnapshot`/… ）。→ P7 hive 迁移把 HMS-iceberg 挪到连接器路径后，这 23 文件才能删（P7.4/P7.5 阶段四）。同理 fe-core `datasource/hudi/`、`datasource/hive/` 也在 P7 范围。**删除排序（最硬约束）**：`datasource/hive/` 删不掉，直到 `HudiUtils`/`HudiScanNode`/`IcebergHMSSource`/`HMSAnalysisTask`/`StatisticsUtil.getIcebergColumnStats` 等全 retype 到 generic。 +`datasource/hive/` **删不掉**，直到以下非-hive 消费者全 retype 到 generic（否则编译断）：`datasource/hudi/HudiUtils`/`HudiScanNode`（extends `HiveScanNode`）/`HudiExternalMetaCache`（C 解绑）；`datasource/iceberg/source/IcebergHMSSource`、`statistics/HMSAnalysisTask`（`setTable(HMSExternalTable)`）、`statistics/util/StatisticsUtil.getIcebergColumnStats`(iceberg SDK)、`datasource/systable/IcebergSysTable`（B 解绑）。P6 #64688 删的是原生 iceberg，但 iceberg-on-HMS 仍走 fe-core，故 `datasource/iceberg/` 还保活 23 个 HMS-iceberg 支撑类——B 把它们切到连接器路径后，F 才能删。同理 `datasource/hudi/`、`datasource/hive/` 在 C/E/F 范围。整条 catalog-SPI 阶段链已合入 upstream `branch-catalog-spi`：P0 #63582 · P1 #63641 · P2 #64096 · P3 #64143 · P4 #64300 · P5 #64446+#64653 · P3b #64655 · P6 #64688。 diff --git a/plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md b/plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md new file mode 100644 index 00000000000000..6124b8af23a7c5 --- /dev/null +++ b/plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md @@ -0,0 +1,42 @@ +# P7 cutover — scope map & sub-batch decomposition (recon 2026-07-06) + +> Produced by a 6-agent code-grounded recon (`wf_536a2968-2c8`) after the FINAL plugin-side write/read commit `0b19506acfe`. Raw findings: workflow journal + `tool-results/blgola3k6.txt` (this session). **Purpose**: correct the HANDOFF framing ("next = flip + delete") — the remaining "cutover" is the entire rest of P7 (event pipeline + heterogeneous routing split + hudi/iceberg-on-HMS delegation + write-chain retype + flip + delete + ACID gate), the largest & riskiest chunk of the whole catalog-SPI migration. **Trust HEAD, not doc line numbers.** + +## What is DONE (committed, dormant) +- **P7.1** DDL/partition metadata ops (`HiveConnectorMetadata` create/drop db+table, truncate). +- **P7.3** plugin-side write transaction + committer + `HiveWritePlanProvider` + read-side ACID producer + read-txn manager (all in `fe-connector-hive`/`-hms`, `0b19506acfe`). **Dormant** — hive not in `SPI_READY_TYPES`. +- Neutral read-txn seam `QueryFinishCallbackRegistry` (fe-core `qe`) — **live**, used by legacy `HiveScanNode`. +- Generic connector write path (`UnboundConnectorTableSink → Logical/PhysicalConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor`) — **live** for iceberg/paimon/maxcompute. Paimon legacy sink **fully deleted** = cleanest precedent. + +## What REMAINS (all still fe-core, blocks the flip) — 6 sub-batches + +| # | Sub-batch | Size | Independent? | Gates | +|---|-----------|------|--------------|-------| +| **A** | **Per-table multi-format dispatch seam (D-020) + `tableFormatType` threading** — the KEYSTONE | XL-core | critical path | unblocks B, C, and the DLA retype in E | +| **B** | iceberg-on-HMS delegation (route ICEBERG tables to `-iceberg` via hms gateway; per-column stats SPI; extract `IcebergUtils` pure helpers) | L | rides A | delete 23 datasource/iceberg + 4 blockers | +| **C** | hudi live cutover (migrate `datasource/hudi` 15 files into `fe-connector-hudi`; dismantle `HudiDlaTable`/`HudiScanNode extends HiveScanNode`; port or fail-loud deferred incremental/schema-evo/time-travel) | L | rides A | delete `datasource/hudi` + frees `HiveScanNode`/`HivePartition`/`HMSSchemaCacheValue` | +| **D** | Event pipeline → `fe-connector-hms` (21 event classes + processor; neutral `ConnectorMetaInvalidator`; thin fe-core role-aware driver; R-010 thread lifecycle + TCCL) | L | **independent** (parallel to A/B/C) | removes `instanceof HMSExternalCatalog` poller gate + edit-log replay CCE hazard | +| **E** | DLA retype + write-chain retype + read-txn rewire (neutralize 42 `instanceof HMSExternal*` + 22 `dlaType` across 8 areas; retype `HMSExternalTable`→generic; delete legacy hive sink 6-8 files; add `ConnectorSession` query-finish seam; remove `HiveTransactionMgr` from `Env`; neutralize `HMSAnalysisTask`; relocate `BIND_BROKER_NAME`) | XL | rides A + partly D | last blocker before delete | +| **F** | Flip + GSON compat + delete legacy + gates (add `"hms"` to `SPI_READY_TYPES`; `GsonUtils` 3-factory `registerCompatibleSubtype`; staged cross-connector delete of `datasource/hive`+`hudi`+23 iceberg classes; clean-room adversarial review; ENG-1 capability-twin audit over 85 sites; GSON replay test; **ACID/event/heterogeneous docker e2e hard gate R-002**; user sign-off) | XL | strictly last | — | + +## The keystone (A) — why it's first +A real `type=hms` catalog is **heterogeneous**: one catalog holds plain-hive (non-MVCC) + iceberg-on-HMS (MVCC) + hudi-on-HMS, today carried by ONE class `HMSExternalTable` + lazy `dlaType` probe + `switch(dlaType)` everywhere. After the flip, a hms catalog becomes a `PluginDrivenExternalCatalog` that wraps **ONE** connector (`PluginDrivenExternalCatalog.connector` is single; `PluginDrivenScanNode` always calls `connector.getScanPlanProvider()` no-arg). **There is no per-table connector/provider selection seam anywhere** (`Connector.getScanPlanProvider()` no-arg; `ConnectorTableHandle` empty marker, no `getTableType`; `tableFormatType` computed by connectors but DROPPED at `PluginDrivenExternalTable.toSchemaCacheValue`). So without A, iceberg-on-HMS/hudi tables cannot be routed to the right scanner and B/C/E cannot proceed. A is shared by all three formats and is a public-SPI change → settle it first. + +**Trino comparison**: Trino uses **one connector per format** (separate `hive`/`iceberg`/`delta`/`hudi` catalogs) plus **table redirection** — when a hive catalog encounters an iceberg table it redirects to a configured iceberg catalog (`hive.iceberg-catalog-name`). Doris's locked decision **D-020** is the analogous idea kept inside one gateway: the `"hms"` connector (`HiveConnectorProvider`) overrides `getScanPlanProvider(handle)` and **delegates** to the `-iceberg`/`-hudi` providers by table type. So `fe-connector-hive` becomes the "hms gateway" depending on `-iceberg`/`-hudi`. Approach is locked; **open implementation decisions** (defer to A's design): discriminator shape (enum `getTableType` vs `tableFormatType` string vs per-table capability); whether `tableFormatType` is persisted in the schema-cache/gson image or recomputed. + +## Highest-risk flip decision (surfaces in E/F, coupled to A) +**GSON single-row MVCC conflict**: historically hive + hudi-on-HMS + iceberg-on-HMS all persist as one `clazz=HMSExternalTable` tag. `registerCompatibleSubtype` maps one tag → ONE class, but MVCC is **subclass-gated** (`PluginDrivenMvccExternalTable extends PluginDrivenExternalTable`): plain-hive/hudi need the base class, iceberg-on-HMS needs the Mvcc subclass. Resolutions: (a) re-persist the three types under distinct discriminators during the routing split, or (b) **move MVCC-ness off the subclass to a runtime `ConnectorCapability` check** so one row targets the base class. Also: plain-hive is non-MVCC with **timestamp/lastDdlTime freshness** (`MTMVMaxTimestampSnapshot`), but `PluginDrivenMvccExternalTable.getTableSnapshot` is snapshot-id-only → needs a timestamp-freshness mode or a separate non-Mvcc MTMV-capable table. + +## Recommended sequencing +`A (keystone)` → then `B` + `C` (both ride A; can overlap) and `D` (independent, parallel anytime) → `E` (retype + write-chain, rides A) → `F` (flip + delete + gates, strictly last). F carries the clean-room review + ENG-1 twin audit + GSON replay test + docker ACID e2e as the terminal hard gate (per iceberg #64688 / paimon #64446+#64653 precedent). + +## Per-sub-batch open decisions (defer sign-off to each sub-batch's recon/design) +- **A**: discriminator shape (enum vs string vs capability); persist `tableFormatType` vs recompute. +- **D (OQ-EVT)**: structured register/unregister SPI seam vs degrade-to-invalidate + lazy re-list; poller location (full move vs thin fe-core role driver); keep `ExternalMetaIdMgr` for HMS or drop (ids derivable via `Util.genIdByName`); partition-NAME invalidation SPI extension vs table-level over-invalidate. +- **C (OQ-SHARE)**: delete `HiveScanNode`/`HiveSplit`/`HivePartition`/`HMSSchemaCacheValue` (paimon precedent: plugin has its own `ConnectorScanRange`) vs co-move a shared base into `fe-connector-hms`; port deferred hudi features now vs fail-loud. +- **B (OQ-COLSTATS)**: add per-column `ConnectorStatisticsOps` method to preserve `enableFetchIcebergStats` vs drop per-column HMS-iceberg stats; keep shrunk `IcebergUtils` (SDK-linked) vs extract pure helpers to make fe-core iceberg-SDK-free. +- **E**: how the connector reaches the query-finish seam (add `registerQueryFinishCallback` to `ConnectorSession` vs fe-core-driven registration); `BIND_BROKER_NAME` relocation home; clean the vestigial iceberg leftovers (`UnboundIcebergTableSink`, dead `InsertOverwrite` branch) or leave. +- **F (packaging)**: separate flip PR then delete PR (paimon model) vs combined squash (iceberg model); clean-room review before flip (iceberg) vs after flip before delete (paimon); make fe-core fully hive/hudi-SDK-free (P4 odps model) vs leave STILL-CONSUMED deps. + +## Reusable (do NOT delete) +`PluginDrivenExternalCatalog/Database`, `PluginDrivenExternalTable`, `PluginDrivenMvccExternalTable`, `PluginDrivenSysExternalTable`, `ConnectorTimeTravelSpec`, `ConnectorProcedureOps`, whole write-path SPI, `DistributionSpecHiveTableSinkHash/UnPartitioned` (misleadingly hive-named but shared by the connector path), `QueryFinishCallbackRegistry`. diff --git a/plan-doc/tasks/P7.4-scan-provider-per-table-seam-design.md b/plan-doc/tasks/P7.4-scan-provider-per-table-seam-design.md new file mode 100644 index 00000000000000..71f7149d2b5ddd --- /dev/null +++ b/plan-doc/tasks/P7.4-scan-provider-per-table-seam-design.md @@ -0,0 +1,58 @@ +# P7.4 sub-batch A — per-table scan-provider selection seam (the multi-format dispatch keystone) + +> Design doc (research-design-workflow, design-doc-first). Grounded on HEAD reads this session (recon `wf_536a2968-2c8` + direct file reads). **Scope = the SEAM only**, dormant. First real use (iceberg-on-HMS delegation) is the next sub-batch. +> +> **✅ DONE — committed `0923077fe67`** (2026-07-07). User-signed: pure seam on `Connector` (not `ConnectorMetadata`), tableFormatType threading deferred. All §7 TODO items landed; verified: fe-connector-api 58 tests, fe-core `PluginDrivenScanNode*` 76 tests (incl. new `PluginDrivenScanNodeScanProviderSelectionTest`), checkstyle 0 (api + core), import-gate clean. Fixed 2 existing tests (`DeleteFilesTest`/`BatchModeTest`) to stub the arg overload (bare Mockito mocks don't run the real default). + +## 1. Goal / Non-goals +**Goal**: give fe-core a connector-agnostic way to pick a *different* `ConnectorScanPlanProvider` **per table** within one catalog, so that after the flip a single heterogeneous `hms` catalog (plain-hive + iceberg-on-HMS + hudi-on-HMS, all under one `PluginDrivenExternalCatalog` wrapping one gateway connector) can route each table to the right scanner. Locked decision D-020 ("per-table SPI provider; the hms gateway connector delegates to `-iceberg`/`-hudi`"). + +**Non-goals (deferred, explicitly out of this sub-batch)**: +- The hive gateway's actual delegation to iceberg/hudi providers (module-dependency restructure, format probe) → sub-batches B (iceberg-on-HMS) / C (hudi). +- Threading `tableFormatType` from `ConnectorTableSchema` into `PluginDrivenSchemaCacheValue` → deferred to the consumer that needs it (BE `TTableFormatFileDesc`, capability derivation, MTMV — sub-batches B/E). The seam does **not** need it (see §3). +- Any change to write/DDL/stats/MVCC routing. +- Any GSON / flip / delete work. + +## 2. Why a selection seam is *necessary* (not just internal dispatch) — the decisive finding +`ConnectorScanPlanProvider.planScan(session, handle, …)` **does** receive the `ConnectorTableHandle`, so one could imagine the gateway returning a single "dispatching" provider from the existing no-arg `getScanPlanProvider()` that routes inside `planScan` by inspecting the handle. **This does not work**, because: +- Providers are built **fresh per call** and are **stateless** (see the "fresh provider per call" comments in `IcebergScanPlanProvider:159`, `PaimonScanPlanProvider:172`). +- Several `ConnectorScanPlanProvider` methods **do not carry the handle** — notably `appendExplainInfo(output, prefix, props)` (called at `PluginDrivenScanNode:375`) and the node-properties/streaming-estimate paths. A fresh, stateless dispatching provider has no handle for those methods and cannot know which sub-provider to use. + +Therefore the selection **must happen at provider-acquisition time**: fe-core asks the connector for the provider **for this handle**, and the connector returns a provider already bound to the correct sub-provider (so even the handle-less methods are routed correctly). This is exactly D-020. + +**Trino parallel**: Trino runs one connector per format (`hive`/`iceberg`/`hudi`) and uses *table redirection* — the hive connector reports "this table lives in catalog X"; the engine **resolves the redirect once at analysis time** and re-binds all subsequent operations to the iceberg connector. We resolve once at provider-acquisition time and bind the provider — same "resolve once, then all ops follow" shape, kept inside one gateway connector instead of across catalogs (Doris exposes one `hms` catalog historically; a cross-catalog redirect would be a visible, breaking change). + +## 3. SPI shape — the one real decision +Add a **per-handle overload** that defaults to today's behavior. Two candidate homes: + +- **(Rec) `Connector.getScanPlanProvider(ConnectorTableHandle handle)`** — sits next to the existing no-arg `Connector.getScanPlanProvider()` (`Connector.java:46`, returns `null` by default). Default: `return getScanPlanProvider();`. The gateway connector owns its sub-connectors (iceberg/hudi) as fields and returns the right one's provider. **Smallest fe-core change**: `PluginDrivenScanNode` already holds `connector` + `currentHandle`; no metadata plumbing. +- (Alt) `ConnectorMetadata.getScanPlanProvider(ConnectorSession, ConnectorTableHandle)` — groups with the other per-handle methods (`getTableStatistics(handle)`, `applySnapshot(handle)`). But provider-selection is a connector-structural concern (which sub-connector), not a metadata-transaction op; the node would fetch metadata per call. + +**The discriminator (how a connector maps a handle → sub-provider) is intentionally NOT in the SPI**: the handle is the connector's own opaque subclass, so the gateway carries whatever it needs (a DLA/format enum) and casts it internally. fe-core never branches on format (iron rule preserved). → the enum-vs-string-vs-capability discriminator question is a **B/C connector-internal** decision, not an A decision. + +## 4. fe-core change (localized to `PluginDrivenScanNode`) +- Replace the **11** uniform call sites `connector.getScanPlanProvider()` (lines 375, 475, 524, 885, 898, 1042, 1174, 1260, 1313, 1328, 1410) with a single private helper `resolveScanProvider()` → `connector.getScanPlanProvider(currentHandle)`. +- Null-tolerance preserved (default overload → no-arg → may be `null`; existing null checks unchanged, e.g. the `:1042` "may be null for connectors without scan capability" path). +- `currentHandle` is the field already set in the ctor (from `create()` `resolveConnectorTableHandle`, `:191`) and updated by filter/projection pushdown — so per-handle selection uses the same handle the rest of the scan already uses. No new state. +- **Zero behavior change for every existing connector** (es/jdbc/trino/maxcompute/paimon/iceberg/hive): none overrides the overload → all fall through to no-arg → byte-identical. + +## 5. Test plan (unit, no Mockito on connector side; fe-core may use its infra) +- `PluginDrivenScanNode` contract test with a **fake multi-provider connector**: `getScanPlanProvider(handle)` returns provider-A for handle-type-1 and provider-B for handle-type-2; assert the node's `planScan`/explain path uses the handle-matched provider. Assert a connector that does **not** override the overload falls back to the no-arg provider (byte-identical), and that a null no-arg provider still yields the existing null-safe behavior. +- Regression: existing `PluginDrivenScanNode*Test` (verbose-explain etc.) stays green (proves inertness). + +## 6. Risks +- **R1 (low)**: missing one of the 11 call sites → that path stays connector-level (single provider) while others are per-handle → a heterogeneous table could mis-route on that path only. Mitigation: the single-helper refactor + a grep assertion that no `connector.getScanPlanProvider()` (no-arg) remains in `PluginDrivenScanNode`. +- **R2 (low)**: `currentHandle` is mutated by pushdown (`applyFilter`/`applyProjection`); confirm the provider identity does not need to be stable across a mutation within one scan (it does not — the gateway keys on table format, which pushdown does not change). Documented in the helper. +- No SPI compatibility risk: additive default method; no existing connector or test implements it. + +## 7. Ordered TODO +1. Add `Connector.getScanPlanProvider(ConnectorTableHandle handle)` default (`return getScanPlanProvider();`) + javadoc stating the acquisition-time-binding contract (§2) and that the default keeps single-provider behavior. +2. `PluginDrivenScanNode`: private `resolveScanProvider()` helper; swap all 11 sites; keep null-tolerance. +3. Fake multi-provider connector + contract test; run existing PluginDrivenScanNode tests. +4. Build `fe-connector-api` (`-am`) + `fe-core`; checkstyle 0 (api + core, no `-am`); `check-connector-imports.sh` clean. +5. Update HANDOFF + this doc; commit (this seam is a standalone, dormant, additive commit — safe to land independently, unlike the P7.3 atomic batch). + +## 8. Open decision for user sign-off (this sub-batch) +- **Scope**: pure seam + fake-connector test (dormant), vs. seam + first real iceberg-on-HMS delegation folded in (bigger, pulls module-dep restructure forward). +- **SPI home**: `Connector` overload (rec) vs `ConnectorMetadata`. +(Discriminator shape, tableFormatType threading, GSON MVCC conflict, per-column stats → later sub-batches.) From 970a106a0689fc418c2a14dbb7477e2fad42249f Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 09:22:13 +0800 Subject: [PATCH 029/334] [fix](catalog) key HMS-event cursor replay by catalogId, never cast to HMSExternalCatalog ExternalMetaIdMgr.replayMetaIdMappingsLog propagates the master's synced-event-id cursor on every FE by casting the live catalog to HMSExternalCatalog. The cast is gratuitous: updateMasterLastSyncedEventId only needs the catalog id, which the log already carries. Once an HMS catalog is served by a generic PluginDrivenExternalCatalog, that cast throws ClassCastException during edit-log replay and wedges FE startup. Key the cursor update by catalogId (long) instead. Behavior is byte-identical today (the live catalog is still HMSExternalCatalog); this removes a latent replay crash that would otherwise block routing an HMS catalog through the plugin connector SPI. - ExternalMetaIdMgr: drop the (HMSExternalCatalog) cast + now-unused import; pass catalogId. - MetastoreEventsProcessor.updateMasterLastSyncedEventId(HMSExternalCatalog, long) -> (long catalogId, long eventId); its sole caller is the replay path above. - Test: replaying a fromHmsEvent cursor log against a non-HMSExternalCatalog live catalog must not throw and must propagate the cursor keyed by catalogId. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../doris/datasource/ExternalMetaIdMgr.java | 10 +-- .../hive/event/MetastoreEventsProcessor.java | 4 +- .../datasource/ExternalMetaIdMgrTest.java | 61 +++++++++++++++++++ 3 files changed, 69 insertions(+), 6 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java index 151bc93f982e2b..5de639566fe701 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java @@ -18,7 +18,6 @@ package org.apache.doris.datasource; import org.apache.doris.catalog.Env; -import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.event.MetastoreEventsProcessor; import com.google.common.base.Preconditions; @@ -115,11 +114,14 @@ public void replayMetaIdMappingsLog(@NotNull MetaIdMappingsLog log) { handleMetaIdMapping(mapping, ctlMetaIdMgr); } if (log.isFromHmsEvent()) { - CatalogIf catalogIf = Env.getCurrentEnv().getCatalogMgr().getCatalog(log.getCatalogId()); + // Propagate the master's synced-event-id cursor to this FE, keyed by catalogId only + // (the log already carries it). Never cast the live catalog to HMSExternalCatalog: + // once an HMS catalog is served by a generic PluginDrivenExternalCatalog, that cast + // would throw ClassCastException and abort edit-log replay, wedging FE startup. + CatalogIf catalogIf = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogId); if (catalogIf != null) { MetastoreEventsProcessor metastoreEventsProcessor = Env.getCurrentEnv().getMetastoreEventsProcessor(); - metastoreEventsProcessor.updateMasterLastSyncedEventId( - (HMSExternalCatalog) catalogIf, log.getLastSyncedEventId()); + metastoreEventsProcessor.updateMasterLastSyncedEventId(catalogId, log.getLastSyncedEventId()); } } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventsProcessor.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventsProcessor.java index 3d3a94eb4e19cf..aa1a8adad92f9e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventsProcessor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/event/MetastoreEventsProcessor.java @@ -322,8 +322,8 @@ private long getMasterLastSyncedEventId(HMSExternalCatalog hmsExternalCatalog) { return masterLastSyncedEventIdMap.getOrDefault(hmsExternalCatalog.getId(), -1L); } - public void updateMasterLastSyncedEventId(HMSExternalCatalog hmsExternalCatalog, long eventId) { - masterLastSyncedEventIdMap.put(hmsExternalCatalog.getId(), eventId); + public void updateMasterLastSyncedEventId(long catalogId, long eventId) { + masterLastSyncedEventIdMap.put(catalogId, eventId); } private void refreshCatalogForMaster(HMSExternalCatalog hmsExternalCatalog) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaIdMgrTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaIdMgrTest.java index 12e018a4cffaaf..54bd621ff11178 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaIdMgrTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalMetaIdMgrTest.java @@ -17,8 +17,13 @@ package org.apache.doris.datasource; +import org.apache.doris.catalog.Env; +import org.apache.doris.datasource.hive.event.MetastoreEventsProcessor; + import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; public class ExternalMetaIdMgrTest { @@ -73,4 +78,60 @@ public void testReplayMetaIdMappingsLog() { Assertions.assertNotEquals(-1L, mgr.getPartitionId(1L, "db1", "tbl1", "p1")); } + /** + * An HMS-event id-mapping log carries the master's synced-event-id cursor and is replayed on + * every FE. Once an HMS catalog is served by a generic (non-{@code HMSExternalCatalog}) catalog + * class, replay must still propagate that cursor keyed by {@code catalogId} without casting the + * live catalog to {@code HMSExternalCatalog} — that cast would throw {@link ClassCastException} + * and abort edit-log replay, wedging FE startup. + */ + @Test + public void testReplayHmsEventCursorDoesNotRequireHmsCatalogType() { + final long catalogId = 7L; + final long lastSyncedEventId = 42L; + + CatalogMgr catalogMgr = Mockito.mock(CatalogMgr.class); + // A live catalog that is NOT an HMSExternalCatalog (mirrors the post-cutover generic catalog); + // doReturn avoids stubbing the wildcard-generic return type of getCatalog(long). + Mockito.doReturn(Mockito.mock(CatalogIf.class)).when(catalogMgr).getCatalog(catalogId); + MetastoreEventsProcessor processor = Mockito.mock(MetastoreEventsProcessor.class); + Env env = new TestingEnv(catalogMgr, processor); + + MetaIdMappingsLog log = new MetaIdMappingsLog(); + log.setCatalogId(catalogId); + log.setFromHmsEvent(true); + log.setLastSyncedEventId(lastSyncedEventId); + + try (MockedStatic envMockedStatic = Mockito.mockStatic(Env.class)) { + envMockedStatic.when(Env::getCurrentEnv).thenReturn(env); + + // Before the fix this threw ClassCastException on the (HMSExternalCatalog) cast. + Assertions.assertDoesNotThrow(() -> new ExternalMetaIdMgr().replayMetaIdMappingsLog(log)); + + // The cursor is propagated keyed by catalogId, not by casting the live catalog. + Mockito.verify(processor).updateMasterLastSyncedEventId(catalogId, lastSyncedEventId); + } + } + + private static final class TestingEnv extends Env { + private final CatalogMgr catalogMgr; + private final MetastoreEventsProcessor metastoreEventsProcessor; + + private TestingEnv(CatalogMgr catalogMgr, MetastoreEventsProcessor metastoreEventsProcessor) { + super(true); + this.catalogMgr = catalogMgr; + this.metastoreEventsProcessor = metastoreEventsProcessor; + } + + @Override + public CatalogMgr getCatalogMgr() { + return catalogMgr; + } + + @Override + public MetastoreEventsProcessor getMetastoreEventsProcessor() { + return metastoreEventsProcessor; + } + } + } From 7b6c1d291dee4c26d29c7b65b394e41f1672cffa Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 09:25:25 +0800 Subject: [PATCH 030/334] [doc](catalog) reframe remaining HMS work as one flip; land two recon findings + refresh HANDOFF MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two code-grounded recons this session (iceberg-on-HMS delegation; HMS metastore-event pipeline) plus direct HEAD verification establish that both are FLIP-GATED: the per-table scan-provider seam only fires inside PluginDrivenScanNode (instanceof PluginDrivenExternalTable), and only a PluginDrivenExternalCatalog holds a plugin Connector. A legacy HMSExternalCatalog has neither, so neither delegation nor the SPI-driven event poller can be wired until the HMS catalog/table are retyped to the generic plugin-driven classes. The prior "6 independent pre-flip sub-batches" framing is corrected: the remaining HMS migration is essentially one flip, and format-delegation / events / hudi are facets of it. - iceberg-on-hms-delegation-findings: why flip-gated; cross-plugin sibling-connector handoff (co-packaging rejected — 2nd AWS SDK poisons S3 JVM-wide); design sketch; deferred deletions / per-column stats (recommend DROP) / IcebergUtils 6-member extraction. - hms-event-pipeline-findings: why flip-gated; primary blocker (events do structural Env object-graph mutation, not just cache invalidation, which the neutral ConnectorMetaInvalidator cannot express); recommended Model B (thin fe-core role driver + plugin poll-once SPI); ExternalMetaIdMgr droppable for HMS but opcode 470 must stay replayable. - HANDOFF: next = execute the flip (retype HMS catalog/table to generic), starting with a recon+design of the retype; the replay-CCE fix (a6dc782d816) is the one pre-flip de-risking item already landed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 62 ++++++++++--------- .../hms-event-pipeline-findings-2026-07-07.md | 37 +++++++++++ ...g-on-hms-delegation-findings-2026-07-07.md | 48 ++++++++++++++ 3 files changed, 117 insertions(+), 30 deletions(-) create mode 100644 plan-doc/tasks/hms-event-pipeline-findings-2026-07-07.md create mode 100644 plan-doc/tasks/iceberg-on-hms-delegation-findings-2026-07-07.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 76c493c576dc18..abaf82ddacd55b 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,61 +7,63 @@ # 🎯 当前状态（2026-07-07） -**插件侧写/读能力已全部提交（休眠）= `0b19506acfe`**（hive/hms 写事务 + 提交器 + 写计划 + 读侧 ACID 生产者 + 读事务管理器；hive 尚未进 `SPI_READY_TYPES`，线上零路由）。P7.1（DDL 元数据）也已提交。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 -**关键校正（勿再误读）**：交接旧叙述把"下一步 = cutover"写成一次轻量翻闸，**实为错**。对照 HEAD 的 6-agent 侦察（`wf_536a2968-2c8`）证实：剩余"cutover"是**整个 P7 尾段、全项目最大最险的一块**，已拆成 **6 个子批 A–F**，完整地图 + 每子批开放决策见 **`plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md`**（起步必读）。 +**本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： -**子批清单 + 进度**（依赖序）： -- **A（多格式分发地基 / 按表选 scan-provider 缝）—— ✅ DONE `0923077fe67`（2026-07-07）**。加 `Connector.getScanPlanProvider(ConnectorTableHandle)` 默认委派无参版；`PluginDrivenScanNode` 11 个取 provider 点收敛到 `resolveScanProvider()`（按 `currentHandle`）。**休眠**、对现有 7 连接器逐字不变。设计 = `plan-doc/tasks/P7.4-scan-provider-per-table-seam-design.md`。验证：api 58 测 + core `PluginDrivenScanNode*` 76 测 + checkstyle 0 + import-gate 净。用户签字：纯地基、放 `Connector`（非 `ConnectorMetadata`）、tableFormatType threading 延后。 -- **B（iceberg-on-HMS 委派）—— ⏭ 下一步**。把"Hive 元存储上的 Iceberg 表"经 hms 网关按表委派给 iceberg 连接器（用 A 的缝）；补按列统计 SPI（保 `enableFetchIcebergStats`）；抽 `IcebergUtils` 纯 helper 使 fe-core 去 iceberg-SDK。骑在 A 上。 -- **C（hudi 上线切换）**：搬 `datasource/hudi`(15 文件)入 `fe-connector-hudi`；拆 `HudiDlaTable`/`HudiScanNode extends HiveScanNode`；延后特性（增量/schema-evo/time-travel）移植或保持 fail-loud。骑在 A 上。 -- **D（事件管道搬迁）**：21 事件类 + processor 搬 `fe-connector-hms`；中立 `ConnectorMetaInvalidator`；fe-core 留薄 role-aware driver；R-010 线程生命周期 + TCCL。**独立、可并行**。 -- **E（表类改造 + 写链改造 + 读事务接线）**：中立化 42 `instanceof HMSExternal*` + 22 `dlaType`；`HMSExternalTable`→generic；删旧 hive sink（6-8 文件）；`ConnectorSession` 加 query-finish 缝；`Env` 摘 `HiveTransactionMgr`；`HMSAnalysisTask` 中立；搬 `BIND_BROKER_NAME`。骑在 A + 部分 D。 -- **F（翻闸 + GSON 兼容 + 删旧 + 硬门）**：`SPI_READY_TYPES` 加 `"hms"`；`GsonUtils` 三工厂 `registerCompatibleSubtype`；分批跨连接器删 `datasource/hive`+`hudi`+23 iceberg 类；净室对抗复审 + 能力孪生审计 + GSON replay 测 + **docker ACID/事件/异构 e2e 硬门（R-002）** + 用户二签。**严格最后**。 +- **`type=hms` 目录今天 100% 是旧类**：`CatalogFactory` case `"hms"` → `new HMSExternalCatalog`（`CatalogFactory.java:133-134`）；`HMSExternalTable extends ExternalTable`（非 `PluginDrivenExternalTable`）；`HMSExternalCatalog extends ExternalCatalog`（非 `PluginDrivenExternalCatalog`）。 +- **按表选扫描器接缝只在 `PluginDrivenScanNode` 内被调用**，而它只对 `instanceof PluginDrivenExternalTable` 的表生效（`PhysicalPlanTranslator.java:808`）；HMS 表走旧 `switch(dlaType)`→`IcebergScanNode` 分支（`:818-847`）。 +- **只有 `PluginDrivenExternalCatalog` 持有插件连接器**（`getConnector()`）；旧 `HMSExternalCatalog` 无连接器，用旧 `HMSCachedClient`。fe-core 只 maven 依赖 `fe-connector-api/-spi`，插件类经**独立 child-first 类加载器**加载。 +- **⇒ 结论**：iceberg/hudi 格式委派、事件管道搬迁，**都需要"HMS 目录先变成通用目录类（持有插件连接器）"才可达/可测**。它们不是翻闸前的独立步骤，而是**同一次翻闸的不同侧面**。翻闸前只能预置休眠脚手架或落"翻闸前减负"小修复。跨插件委派**不能靠加 maven 依赖/共包**（会出现第二份 AWS SDK，JVM 级毒化 S3，Paimon RC-3 有记录），须新增"网关拿到兄弟连接器"的中立 SPI。 + +**本 session 已落地（翻闸前唯一干净独立的减负项）**：`a6dc782d816` `[fix](catalog)` 消除编辑日志重放的 CCE 隐患——`ExternalMetaIdMgr.replayMetaIdMappingsLog` 原来强转 `(HMSExternalCatalog)` 只为取 id（日志本就带 catalogId），翻闸后会 CCE 使 FE 重放崩溃起不来；改为按 `catalogId(long)` 传，行为今天逐字不变，拆掉翻闸时必爆的雷。全绿（fe-core 编译 BUILD SUCCESS + `ExternalMetaIdMgrTest` 2 测过 + checkstyle 0）。 + +**两份侦察存档（起步必读，别重炒）**： +- `plan-doc/tasks/iceberg-on-hms-delegation-findings-2026-07-07.md` — 委派为何 flip-gated + 跨插件 handoff 机制（对照 Trino）+ 具体设计草图 + 删 23 类/按列统计(建议 DROP)/`IcebergUtils` 抽 6 个纯 helper 的推迟清单。 +- `plan-doc/tasks/hms-event-pipeline-findings-2026-07-07.md` — 事件管道 flip-gated + 主障碍（事件做**结构性变更**非仅失效，中立 `ConnectorMetaInvalidator` 缺 register/rename 动词 + 分区 NAME 粒度缺口）+ 推荐 Model B（薄 fe-core 角色驱动 + 插件 pollOnce SPI）+ `ExternalMetaIdMgr` 可对 HMS 弃用（但 opcode 470 须保重放）。 --- -# 🚀 下个 session 任务 = **子批 B（iceberg-on-HMS 委派）** +# 🚀 下个 session 任务 = **执行翻闸（把 HMS 目录/表改造成通用类）** + +翻闸是全项目最大最险的一块，是解锁一切（格式委派/事件/删旧类）的**真正关口**。样板 = P5 paimon（`P5-paimon-migration.md`）、P6 iceberg（`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`），机制相同、但 HMS 因**异构（三格式）+ 事件管道 + 写链 + DLA** 而更大。 -> 起步先读 `P7-cutover-scope-map-2026-07-06.md`（§B 段 + 删序约束）+ A 的设计文档（缝的契约）。B 是**首个真实委派**，验证 A 的地基。 +**开工方式（守纪律，别一上来写代码）**：先做一轮**针对"HMS 目录/表 retype 到通用类"的 code-grounded 侦察 + 设计文档**——把旧 `HMSExternalCatalog`/`HMSExternalTable` 提供的每项能力（元数据 init、schema、分区、统计、DLA 分派、MVCC、系统表、事件、写/DDL）逐项映射到连接器 SPI，标出缺口（跨插件兄弟连接器 SPI、事件 Model-B 驱动 + 结构性变更 SPI、GSON `registerCompatibleSubtype` 兼容、按表 MVCC 能力），并把翻闸拆成**内部有序阶段**（连接器补齐→加 `"hms"` 进 `SPI_READY_TYPES`→GSON 兼容→删旧类→ACID/事件/异构 e2e 硬门 R-002）。**每个开放决策先中文讲背景+示例+推荐、不引任务代号，请用户签字**再实现。 -- **做什么**：hms 网关连接器（`fe-connector-hive`）**依赖 `fe-connector-iceberg`**；其 `getScanPlanProvider(handle)` 覆写，对 iceberg-on-HMS 句柄委派给 iceberg 连接器的 scan provider（普通 hive 表仍走 hive provider）。网关需在 `getTableHandle` 探测格式并把它编进**自己的**句柄子类（判别标识是**连接器内部**事，fe-core 不看格式）。 -- **连带（B 范围）**：补 `ConnectorStatisticsOps` 按列统计方法 + iceberg 实现（保 `enableFetchIcebergStats` 快路径）；抽 `IcebergUtils` 纯 helper（`isIcebergRowLineageColumn`/`getEffectiveIcebergFormatVersion`/行血缘常量）到 SDK-free util，使删 23 类后 fe-core 去 iceberg-SDK。 -- **删序**：`IcebergHMSSource`/`IcebergScanNode`/`IcebergSysTable`/`StatisticsUtil.getIcebergColumnStats`/`HMSAnalysisTask` 等**retype 到 generic 后**才能删 `datasource/iceberg` 23 类；`datasource/hive` 要等 B + C 都完成（HudiScanNode/HudiUtils 也解绑）。 -- **B 的开放决策（recon 后用户签字，先中文讲背景+示例+推荐，不引任务代号）**：格式判别标识形态（枚举 vs 串 vs capability，连接器内部）；按列统计 SPI vs 放弃 per-column HMS-iceberg 统计；`IcebergUtils` 抽纯 helper vs 留 SDK-linked shrunk 版。 +翻闸内部要覆盖的特性清单（把旧"6 子批"当**特性 inventory** 用，但**不再当作翻闸前的独立步骤**）见 `P7-cutover-scope-map-2026-07-06.md`（其 A–F 分解仍是有用的能力地图；**其"翻闸前可独立落地"的排序前提已被本 session 校正**）。 ## 开场要点（承接） -1. **起步先读** `P7-cutover-scope-map-2026-07-06.md`（6 子批全图 + 删序约束 + 每子批开放决策）+ 本文顶部 🎯 段。cutover ≠ 一次翻闸，是 A–F 六批。 -2. **A 已提交（`0923077fe67`）+ 全绿 + 休眠，勿重写/勿再复审**。P7.1 + P7.3（`0b19506acfe`）同样已提交、勿回炒。 -3. **纪律**：每子批 = 先 code-grounded recon → 设计文档 → 用户签字该批专属决策 → 实现 → 独立 commit → 更新本 HANDOFF。上下文超 30% 找干净节点交接。**path-whitelist `git add`，严禁 `-A`**。 -4. **硬门 = ACID/事件/异构集成测试**在 F（R-002 最大风险，需 live 路径，勿静默跳过——Rule 12）。full-ACID **写**继续硬拒；full-ACID + insert-only **读**在范围（已落地插件侧）。 +1. **先读两份 findings 文档 + 本文顶部 🎯 段**。剩余 HMS 迁移 ≈ 一次翻闸；B/C/D 是它的侧面，非独立前置步。 +2. **已提交勿回炒**：P7.1 / P7.3(`0b19506acfe`) / A(`0923077fe67`) 全休眠；replay-CCE fix(`a6dc782d816`) 已合。 +3. **纪律**：先 code-grounded recon → 设计文档 → 用户签字该阶段专属决策 → 实现 → 独立 commit → 更新本 HANDOFF。上下文超 30% 找干净节点交接。**path-whitelist `git add`，严禁 `-A`**。 +4. **硬门 = ACID/事件/异构集成测试**（R-002 最大风险，需 live 路径，勿静默跳过——Rule 12）。full-ACID **写**继续硬拒；full-ACID + insert-only **读**在范围（已落地插件侧）。 --- # 📦 分支 / Commit 须知 -- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**。**打包/复审策略（flip 前/后、单 PR vs 分 PR）= F 的开放决策**（paimon 分 PR vs iceberg 合并 squash 两先例）。 +- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**。**打包/复审策略（翻闸前/后、单 PR vs 分 PR）= 翻闸阶段的开放决策**（paimon 分 PR vs iceberg 合并 squash 两先例）。 - **公开 tracking issue = apache/doris#65185**；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 -- **⚠️ path-whitelist `git add`，严禁 `git add -A`**（工作树有历史遗留 scratch：`*.bak`·`regression-test/conf/regression-conf.groovy` 明文 key·`.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`·`plan-doc/reviews/P5-*`·`.claude/`——均**非本线程产物，勿混入任何 commit**）。 -- commit message：`[feat](catalog) P7.x …` / `[doc](catalog) …` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每子批/每条 fix = 独立 commit**；HANDOFF + 设计文档单独 commit（与 code 分开）。A 已是普通独立 commit（非 P7.3 那种原子批）。 +- **⚠️ path-whitelist `git add`，严禁 `git add -A`**（工作树有历史遗留 scratch：`*.bak`·`regression-test/conf/regression-conf.groovy` 明文 key·`.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`·`plan-doc/reviews/P5-*`·`.claude/`·`failed-cases.out`——均**非本线程产物，勿混入任何 commit**）。 +- commit message：`[feat](catalog) …` / `[fix](catalog) …` / `[doc](catalog) …` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 设计/findings 文档单独 commit（与 code 分开）。 # ⚙️ 操作须知（复用） -- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。**checkstyle 别加 `-am`**：`mvn -pl : checkstyle:check`。artifactId：`fe-core` / `fe-connector-api` / `fe-connector-hive` / `fe-connector-hms` / `fe-connector-iceberg` / `fe-connector-hudi`。 -- **⚠️ bash 工具默认 timeout 120s**：fe-core 全量编译 > 2min → **务必**把 `timeout` 调到 ~580000ms（曾因忘调而被 SIGTERM 打断）。**后台/管道 exit 不可信**——读 LOG 的 `BUILD SUCCESS` 行 + surefire `Tests run=/Failures=/Errors=`。改代码后 commit 前 fresh recompile 杜绝 stale `.class`。 -- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**。⚠️ `Mockito.mock(接口)` **不跑 default 方法**（会返 null）——给接口加 default 方法后，凡 mock 该接口并只 stub 旧方法的测试都要补 stub 新方法（A 即遇此：`getScanPlanProvider(handle)` 在 bare mock 上返 null，须 stub arg 重载）。checkstyle **禁 static import**、**扫 test 源**。 +- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。跑单测加 `-Dtest=`。**checkstyle 别加 `-am`**：`mvn -pl : checkstyle:check`。artifactId：`fe-core` / `fe-connector-api` / `fe-connector-hive` / `fe-connector-hms` / `fe-connector-iceberg` / `fe-connector-hudi`。 +- **⚠️ bash 工具默认 timeout 120s**：fe-core 全量编译 ~1.5–2min → **务必**把 `timeout` 调到 ~580000ms。**后台/管道 exit 不可信**——读 LOG 的 `BUILD SUCCESS` 行 + surefire `Tests run=/Failures=/Errors=`。改代码后 commit 前 fresh recompile 杜绝 stale `.class`。 +- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`mockStatic(Env.class)` 是本仓惯用法，191 处；stub `Env.getCurrentEnv()`→自定义 `TestingEnv extends Env{super(true)}` 覆写 getter，见 `ExternalMetaCacheRouteResolverTest` / 本轮 `ExternalMetaIdMgrTest`）。⚠️ `Mockito.mock(接口)` **不跑 default 方法**（返 null）。checkstyle **禁 static import**、**扫 test 源**。 - 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`（**repo 根**，非 `fe/`）。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory）。 - **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 紧**（~82% used）；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 # 🧠 起步必读 -1. **`plan-doc/tasks/P7-cutover-scope-map-2026-07-06.md`**（6 子批全图 + keystone + 删序 + 每子批开放决策 + Trino 对比）+ 本文顶部 🎯/🚀。 -2. **样板**：`P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`（净室复审 + 能力孪生审计 + GSON replay 范式）。委派/缝模板 = 本轮 A 的设计文档。 -3. **铁律**：fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/`switch(dlaType)`/引擎名判别（新缝走中立 SPI / ConnectorCapability）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；插件跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`）；history_schema_info nested 名 lowercase（memory）。 +1. **两份 findings 文档**（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）+ 本文顶部 🎯/🚀。 +2. **样板**：`P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`（净室复审 + 能力孪生审计 + GSON replay 范式）。委派/缝模板 = A 的设计文档 `P7.4-scan-provider-per-table-seam-design.md`。特性地图 = `P7-cutover-scope-map-2026-07-06.md`（排序前提已校正）。 +3. **铁律**：fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/`switch(dlaType)`/引擎名判别（翻闸靠"表类=通用类 + 网关按句柄委派"，不靠在 `PhysicalPlanTranslator` 加分支）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；跨插件/跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`，事件轮询后台线程 R-010 亦需）；history_schema_info nested 名 lowercase（memory）。 4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-tracking-issue`。 --- -## 背景：跨连接器删除排序（**F 最硬约束**，B/C 为其解绑） +## 背景：跨连接器删除排序（翻闸最硬约束） -`datasource/hive/` **删不掉**，直到以下非-hive 消费者全 retype 到 generic（否则编译断）：`datasource/hudi/HudiUtils`/`HudiScanNode`（extends `HiveScanNode`）/`HudiExternalMetaCache`（C 解绑）；`datasource/iceberg/source/IcebergHMSSource`、`statistics/HMSAnalysisTask`（`setTable(HMSExternalTable)`）、`statistics/util/StatisticsUtil.getIcebergColumnStats`(iceberg SDK)、`datasource/systable/IcebergSysTable`（B 解绑）。P6 #64688 删的是原生 iceberg，但 iceberg-on-HMS 仍走 fe-core，故 `datasource/iceberg/` 还保活 23 个 HMS-iceberg 支撑类——B 把它们切到连接器路径后，F 才能删。同理 `datasource/hudi/`、`datasource/hive/` 在 C/E/F 范围。整条 catalog-SPI 阶段链已合入 upstream `branch-catalog-spi`：P0 #63582 · P1 #63641 · P2 #64096 · P3 #64143 · P4 #64300 · P5 #64446+#64653 · P3b #64655 · P6 #64688。 +`datasource/hive/` **删不掉**，直到非-hive 消费者全 retype 到 generic：`datasource/hudi/HudiUtils`/`HudiScanNode`(extends `HiveScanNode`)/`HudiExternalMetaCache`；`datasource/iceberg/source/IcebergHMSSource`、`statistics/HMSAnalysisTask`、`statistics/util/StatisticsUtil.getIcebergColumnStats`、`datasource/systable/IcebergSysTable`。P6 #64688 删的是原生 iceberg，但 iceberg-on-HMS 仍走 fe-core，故 `datasource/iceberg/` 还保活 ~23 个 HMS-iceberg 支撑类（两 tier，见委派 findings 文档）——翻闸把它们切到连接器路径后才能删。同理 `datasource/hudi/`、`datasource/hive/`。整条 catalog-SPI 阶段链已合入 upstream `branch-catalog-spi`：P0 #63582 · P1 #63641 · P2 #64096 · P3 #64143 · P4 #64300 · P5 #64446+#64653 · P3b #64655 · P6 #64688。 diff --git a/plan-doc/tasks/hms-event-pipeline-findings-2026-07-07.md b/plan-doc/tasks/hms-event-pipeline-findings-2026-07-07.md new file mode 100644 index 00000000000000..015ce5b4f48c27 --- /dev/null +++ b/plan-doc/tasks/hms-event-pipeline-findings-2026-07-07.md @@ -0,0 +1,37 @@ +# HMS metastore-event pipeline relocation — recon findings (2026-07-07) + +> Produced by a 6-dim code-grounded recon (`wf_46c0c020-08f`) + lead-engineer verification. **Purpose**: capture the analysis of "move the HMS notification-event pipeline out of fe-core into the hms plugin" for the flip step. **User decision (2026-07-07)**: the full relocation is **flip-gated** (needs a plugin connector, which only exists once the HMS catalog is a `PluginDrivenExternalCatalog`), so it is **folded into the flip**, same as the iceberg-on-HMS delegation. The one genuinely-independent, pre-flip-safe piece — neutralizing the edit-log replay CCE — **is DONE** (`a6dc782d816`). + +## Why the relocation is flip-gated (verified) +- fe-core maven-depends only on `fe-connector-api` + `fe-connector-spi` (`fe-core/pom.xml`); it does NOT depend on `fe-connector-hms`. So event classes moved into `fe-connector-hms` are reachable only via the plugin classloader through the SPI. +- Only `PluginDrivenExternalCatalog` holds a plugin `Connector` (`getConnector()`, `PluginDrivenExternalCatalog.java:315`). A legacy `HMSExternalCatalog` has NO connector — it uses fe-core's `HMSCachedClient` (`HMSExternalCatalog.java:201-203`). +- The neutral push seam `ConnectorMetaInvalidator` is obtained via `ConnectorContext.getMetaInvalidator()` (`DefaultConnectorContext:150`) — only wired for plugin connectors. It exists end-to-end but is **dead code today** (no production caller; pre-built for exactly this relocation). +→ A relocated, SPI-driven poller needs a plugin connector to live in and push through; that only exists after the catalog flips to the generic class. Pre-flip HMS catalogs cannot host it. + +## DONE pre-flip (committed a6dc782d816): replay-CCE neutralization +`ExternalMetaIdMgr.replayMetaIdMappingsLog` propagated the master's synced-event-id cursor by casting the live catalog to `(HMSExternalCatalog)` — gratuitous, since `updateMasterLastSyncedEventId` only needs the catalog id (already in the log). Post-flip that cast would `ClassCastException` on every FE during edit-log replay → wedged startup. Fixed: key the cursor by `catalogId (long)`; dropped the cast + unused import; changed `updateMasterLastSyncedEventId(HMSExternalCatalog,long)` → `(long,long)` (sole caller was the replay path). Byte-identical behavior today; latent flip-crash removed. Regression test added (replay a `fromHmsEvent` cursor log against a non-HMS catalog → no throw + cursor keyed by id). + +## The pipeline (what the flip must relocate) +- **One poller** `MetastoreEventsProcessor extends MasterDaemon`, constructed + started unconditionally from `Env` (`Env.java:764,2081`); iterates ALL catalogs, gates on `instanceof HMSExternalCatalog` (`MetastoreEventsProcessor.java:116` — the single in-scope gate to remove). Role-aware: master fetches HMS `NotificationEvent`s directly + writes `MetaIdMappingsLog` to edit-log; followers only advance to `masterLastSyncedEventId` (learned via replay) and fall back to forwarding `REFRESH CATALOG` to master via `MasterOpExecutor` (`:336-349`). Cursor state = two `Map` on the singleton (NOT persisted, rebuilt as -1 on restart). +- **~21 event classes** under `datasource/hive/event/`. Each `event.process()` reaches `Env.getCurrentEnv().getCatalogMgr()/getRefreshManager()` and HMS-thrift/Hive-messaging deserializers (`NotificationEvent`, `AddPartitionMessage`, `GzipJSONMessageDeserializer`). Fe-core imports saturate them → cannot compile plugin-side unchanged. + +## The primary blocker: events do STRUCTURAL mutation, not just cache invalidation +`event.process()` calls `CatalogMgr`/`RefreshManager` mutators that downcast to HMS types and **rebuild the Env catalog→db→table object graph** (register/unregister table+db, add/drop partition cache), NOT merely drop caches (`CatalogMgr.java:742,751,769,782,822,861`; `RefreshManager:217-291`). The neutral `ConnectorMetaInvalidator` has only 5 **cache-drop** verbs (`invalidateAll/Database/Table/Partition/Statistics`) — no register/unregister/rename. Per-event breakdown: +- Pure cache-drop (already expressible): `Insert`, plain (non-rename) `AlterTable`, plain `AlterPartition` → `invalidateTable`. +- Structural (NOT expressible today): `Create/Drop Table`, `Create/Drop/Alter(rename) Database`, `AlterTable` rename/view-recreate, `Add/Drop/Alter(rename) Partition`. +- **Partition granularity gap**: SPI `invalidatePartition` carries partition VALUES, but the cache is keyed by partition NAME → the bridge (`ExternalMetaCacheInvalidator:60-69`) degrades to whole-table invalidation. Events already have the NAME. + +## Recommended target architecture (Model B — thin fe-core driver + plugin poll-once SPI) +- fe-core keeps ONE connector-agnostic, role-aware `MasterDaemon` driver that iterates catalogs and **probes each connector for an optional event-source capability via SPI** (a capability probe, NOT `instanceof`), then calls e.g. `connector.pollOnce(lastSyncedEventId, isMaster)` returning a **neutral list of change descriptors** (db/table/partition + op). fe-core applies the descriptors to cache/table-list/edit-log (its rightful concern — HA/replication/role stay fe-core) and retains the follower→master `REFRESH CATALOG` forward. +- The plugin (`fe-connector-hms`) severs only **fetch + parse** (add `getNextNotification`/`getCurrentNotificationEventId` to `HmsClient`; move the deserializers + event→descriptor mapping). No fe-core imports remain plugin-side. +- The change-descriptor vocabulary must cover register/unregister/rename + partition-NAME granularity (closing the `invalidatePartition` values-vs-names gap), OR the driver keeps structural table-list mutation in fe-core and the descriptors just name what changed. +- **TCCL/R-010**: fe-core wraps the `pollOnce` call in an `onPluginClassLoader`-style pin (mirrors `PluginDrivenScanNode`), covering both the notification RPC and the JSON/GZIP deserialization (today there is NO pin in the poller path). `ThriftHmsClient.doAs` pins to the SYSTEM loader for the RPC only — insufficient for deserialization. +- Alternative (Model A — full plugin-owned daemon) rejected: re-implements MasterDaemon lifecycle, HA/edit-log replication, and follower-forward plugin-side; HA/edit-log belong in fe-core. + +## ExternalMetaIdMgr fate (for the flip) +- Its id getters (`getDbId/getTblId/getPartitionId`) are **dead (test-only)**; event-driven ids are name-derived via `Util.genIdByName` (`CatalogMgr:743,783`); `nextMetaId()`/`Env.getNextId()` allocation is vestigial (allocated id is stored in the unread map, live objects use `genIdByName`). So `ExternalMetaIdMgr` can be **dropped for HMS**. +- BUT: persisted `MetaIdMappingsLog` (opcode `OP_ADD_META_ID_MAPPINGS = 470`) is connector-neutral GSON (no HMS-typed blob → no deserialize CCE) and lives in old journals → **the opcode + a neutral replay handler must survive** for on-disk compat, even if the class is retired. The replay-CCE fix (done) already makes the `fromHmsEvent` cursor branch catalogId-keyed. + +## Scope boundary (for the flip) +- The event-poller gate is exactly ONE site (`MetastoreEventsProcessor:116`). All other `instanceof HMSExternalCatalog` (~17) + `instanceof HMSExternalTable` (~24) sites are unrelated DLA/TVF/sink/stats/DDL coupling handled by the broader catalog/table retype (also part of the flip). `RefreshManager:195-201` is event-adjacent (partition invalidation via the edit-log) — flag, handle with the retype. +- One poller covers ALL formats on an HMS catalog (hive + iceberg-on-HMS + hudi-on-HMS; name-keyed, no `DLAType` reference) → the event relocation is per-HMS-connector, independent of the (also-flip-folded) iceberg/hudi format delegation, but coupled to the catalog/table retype. diff --git a/plan-doc/tasks/iceberg-on-hms-delegation-findings-2026-07-07.md b/plan-doc/tasks/iceberg-on-hms-delegation-findings-2026-07-07.md new file mode 100644 index 00000000000000..09e6d0ff297ec2 --- /dev/null +++ b/plan-doc/tasks/iceberg-on-hms-delegation-findings-2026-07-07.md @@ -0,0 +1,48 @@ +# Iceberg-on-HMS delegation — recon findings (2026-07-07) + +> Produced by a code-grounded recon (`wf_24c2052f-198`, 5 structured dims + 2 re-run dims) plus lead-engineer direct verification of the load-bearing facts. **Purpose**: capture the analysis of "route Iceberg-on-HMS tables to the iceberg connector via the hms gateway" so the work can resume when it becomes reachable. **User decision (2026-07-07)**: this delegation is **folded into the catalog-switch (flip) step**, NOT built as a standalone dormant increment now — because the recon proved it is unreachable and untestable before the flip, and entangled with it. Next work pivoted to the HMS event-pipeline relocation (independent, unblocked). Deletions / per-column stats SPI / `IcebergUtils` extraction are ALL deferred to the flip step regardless. + +## The three verified facts that reframed this work + +**Fact 1 — the delegation seam is UNREACHABLE until the hms catalog/table class flips.** +Scan-node selection is by table **class**, not by connector/handle: +- `PhysicalPlanTranslator.visitPhysicalFileScan` picks `PluginDrivenScanNode` **only** for `table instanceof PluginDrivenExternalTable` (`PhysicalPlanTranslator.java:808`); an HMS table falls into the `else if (instanceof HMSExternalTable)` arm and runs the legacy `switch(getDlaType())` → `new IcebergScanNode(...)` (`:818-847`). +- The per-table scan-provider seam `Connector.getScanPlanProvider(handle)` (landed `0923077fe67`) is consulted **only inside `PluginDrivenScanNode`** (`PluginDrivenScanNode.java:209`). +- HMS is still 100% legacy: `HMSExternalTable extends ExternalTable` (`HMSExternalTable.java:125`, NOT `PluginDrivenExternalTable`); `HMSExternalCatalog extends ExternalCatalog` (`HMSExternalCatalog.java:52`); `CatalogFactory` case `"hms"` builds `new HMSExternalCatalog(...)` (`CatalogFactory.java:133-134`); `HMSExternalDatabase.buildTableInternal` unconditionally builds `HMSExternalTable`. +→ Until the hms catalog is added to `SPI_READY_TYPES` and built as a `PluginDrivenExternalCatalog` (the flip), the gateway's `getScanPlanProvider(handle)` **can never fire** for an iceberg-on-HMS table. Building the delegation before the flip = dead, un-e2e-testable code. + +**Fact 2 — it is NOT "scan-only" delegation; the gateway must route the WHOLE per-table lifecycle to iceberg.** +`IcebergScanPlanProvider` unconditionally casts the handle to `IcebergTableHandle` (`IcebergScanPlanProvider.java:319,364,476`). A `HiveTableHandle` handed to it → `ClassCastException`. So the gateway must produce **iceberg-typed handles** for iceberg tables, i.e. delegate `getTableHandle` / schema / stats / scan / sys-tables / time-travel to an embedded iceberg connector — not just the scan provider. This is Trino's "resolve once at analysis, rebind all ops" shape, kept inside one gateway (D-020) instead of across catalogs. + +**Fact 3 — cross-plugin classloader isolation forbids the naive "add a maven dep" approach.** +Each connector is a **separate plugin zip in its own child-first classloader** (`ConnectorPluginManager` parent-first only for `org.apache.doris.connector.*`/`.filesystem.*`; `DirectoryPluginRuntimeManager` one loader per plugin dir). Corrections to earlier assumptions: +- `fe-connector-hive` deps = spi/api/hms only; it does **NOT** see iceberg classes. `fe-connector-hms` does **NOT** depend on iceberg (the only "iceberg" hit in its pom is a **comment**, line 102). The real edge is reversed: `fe-connector-iceberg → fe-connector-hms`. +- **Co-packaging iceberg into the hive plugin zip is REJECTED**: a second copy of the AWS SDK in a distinct classloader "permanently poisons S3 for the whole JVM" (documented in `fe-connector-iceberg/pom.xml` as paimon RC-3). Iceberg-core/AWS/Caffeine are bundled child-first specifically to keep one copy per loader. + +## The mechanism (locked by the above): cross-plugin sibling-connector handoff + +The gateway (`HiveConnector`) must obtain a **genuinely iceberg-plugin-loaded** `IcebergConnector` and delegate to it. This works because: +- The SPI interfaces (`Connector`, `ConnectorScanPlanProvider`, `ConnectorMetadata`, `ConnectorTableHandle`) are parent-first (single fe-core copy) → a hive-loader object can hold/invoke an iceberg-loader provider type-safely. +- The scan-thread TCCL pin (`PluginDrivenScanNode.onPluginClassLoader`, keys off `provider.getClass().getClassLoader()`) **auto-pins** to the iceberg loader once the returned provider is a real iceberg-loader class — **zero fe-core change, no `if(iceberg)` branch**. + +**New SPI required**: `ConnectorContext` today exposes NO sibling-connector lookup, and `ConnectorPluginManager.createConnector` is fe-core-only. So the flip must add a neutral seam so a gateway connector can obtain a sibling connector **of a named type, built in that type's plugin loader, sharing properties** — e.g. `ConnectorContext.getOrCreateSiblingConnector(type, props)` implemented by fe-core over `ConnectorPluginManager`. The gateway names `"iceberg"` (plugin-side, allowed); fe-core stays connector-agnostic (generic "give me a connector of type X" factory). + +**Trino parallel**: Trino runs one connector per format + *table redirection* — the hive connector reports the table lives in catalog X; the **engine** resolves the redirect once and rebinds ops to the iceberg catalog's connector (in its own classloader). Same "engine mediates the cross-format handoff" shape; Doris keeps one visible `hms` catalog (D-020) so the gateway holds the sibling internally instead of exposing a second catalog. + +## Concrete design sketch (for the flip step) + +1. **Format probe already exists, already runs**: `HiveConnectorMetadata.getTableHandle` calls `HiveTableFormatDetector.detect(HmsTableInfo)` → `HiveTableType` (ICEBERG when table param `table_type=ICEBERG`, case-insensitive; iceberg-before-hudi-before-hive order), baked into `HiveTableHandle.tableType`. Matches legacy `HMSExternalTable.supportedIcebergTable()` exactly. No new probe needed. +2. **Gateway embeds ONE shared `IcebergConnector` per hms catalog** (shared HMS + snapshot/manifest caches + worker-pool pin + TCCL-pinning context + lazy plugin authenticator). Build it via the sibling seam; forward `close()`/`invalidateTable`/`invalidateAll` to it. +3. **Property synthesis (plugin-side)**: an hms catalog's props lack `iceberg.catalog.type`; the gateway must inject `iceberg.catalog.type=hms` (`IcebergConnectorProperties.TYPE_HMS`) before constructing the iceberg connector — else `IcebergCatalogFactory.resolveFlavor` throws "Missing 'iceberg.catalog.type'". Shared `hive.metastore.uris`/`uri` pass through unchanged; `warehouse` is NOT required for the hms flavor (`IcebergHmsMetaStoreProperties.validate` runs only `validateConnection`). Reuses `MetaStoreProviders.bindForType("hms", ...)` → `HmsMetaStoreProperties.toHiveConfOverrides` → `IcebergCatalogFactory.assembleHiveConf` → `CatalogUtil.buildIcebergCatalog`. +4. **Route iceberg-typed tables**: for an ICEBERG handle, the gateway's metadata delegates `getTableHandle`→`IcebergTableHandle`, and `getScanPlanProvider(handle)` returns the embedded iceberg connector's provider (5-arg: properties, catalogOps-over-live-hms-iceberg-catalog, iceberg's TcclPinningContext, shared manifestCache, shared rewritableDeleteStash). Sys tables ($snapshots/$history/$manifests) + time-travel come free once handles are iceberg-typed (`IcebergConnectorMetadata.getSysTableHandle`, `IcebergScanPlanProvider.supportsSystemTableTimeTravel()==true`). +5. **MVCC / table-class**: the base-vs-`PluginDrivenMvccExternalTable` selection (`PluginDrivenExternalDatabase.buildTableInternal`, gated on `SUPPORTS_MVCC_SNAPSHOT`) is unreachable until the catalog flips; it lands WITH the flip. NB open Q: if the hms gateway advertises `SUPPORTS_MVCC_SNAPSHOT`, plain-hive tables would also be built as the Mvcc subclass (plain-hive is `EmptyMvccSnapshot` today) — decide per-table vs table-uniform MVCC capability. + +## Deferred to the flip (NOT this-step work — all blocked) + +- **Delete the ~23 `datasource/iceberg` classes** — two tiers. TIER-1 (scan cluster, deletable once the `case ICEBERG` branch is gone at flip): `IcebergScanNode`, `IcebergHMSSource`, `IcebergSource`, `IcebergSplit`, `IcebergDeleteFileFilter`, `IcebergTableQueryInfo`, `cache/IcebergManifestCacheLoader`, `cache/ManifestCacheValue`, `profile/IcebergMetricsReporter`. TIER-2 (blocked until `HMSExternalTable`/`IcebergDlaTable` metadata cutover): `IcebergUtils`, `IcebergExternalMetaCache`, `IcebergMetadataOps`, `IcebergSchemaCacheKey/Value`, `IcebergSnapshot(CacheValue)`, `IcebergPartition(Info)`, `IcebergTableCacheValue`, `IcebergManifestEntryKey`, `IcebergMvccSnapshot`, `DorisTypeToIcebergType`, `IcebergCatalogConstants`. +- **4 blockers**: `IcebergHMSSource` (dies with `IcebergScanNode`); `StatisticsUtil.getIcebergColumnStats` + `HMSExternalTable` ICEBERG stats branch (single-consumer, deletable with the retype); `datasource/systable/IcebergSysTable` (already a throwing dead-end — `createSysExternalTable` unconditionally throws; querying `$snapshots` on iceberg-on-HMS already fails today → no regression to defer). +- **Per-column stats SPI (OQ-COLSTATS)**: the fast-path is `HMSExternalTable.getColumnStatistic` case ICEBERG, gated by `enable_fetch_iceberg_stats` (**default OFF**); it feeds query-planning cache-miss, NOT ANALYZE (ANALYZE already degrades to SQL sampling with no iceberg branch). **Native iceberg already dropped this fast-path** (`PluginDrivenExternalTable` has no `getColumnStatistic` override → inherits `Optional.empty()`). Recommendation: **DROP** for iceberg-on-HMS (parity with native iceberg, default-off, coarse estimate anyway — min/max forced to ±INF, no NDV) rather than add a new per-column SPI method + neutral DTO. Table-level rowCount MUST still be provided via `getTableStatistics` (iceberg connector already computes it). Revisit at the retype. +- **`IcebergUtils` extraction (OQ-ICEBERGUTILS)**: the ONLY members the connector-agnostic engine needs to survive the 23-class deletion are **6**: constants `ICEBERG_ROW_ID_COL`, `ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL`, `ICEBERG_ROW_LINEAGE_MIN_VERSION`; methods `isIcebergRowLineageColumn(String)`, `isIcebergRowLineageColumn(Column)`; and `getEffectiveIcebergFormatVersion(Map,Map)`. Callers: `BindExpression:358`, `CreateTableInfo:1140,1144,1164,1166`, `IcebergMergeCommand:176,201,260,342-343,369`, `IcebergUpdateCommand:106` (+ tests `LogicalFileScanTest`, `PluginDrivenScanNodeClassifyColumnTest`). Extract these to a small **fe-core** SDK-free util (must be fe-core, not fe-common — the `Column` overload needs `org.apache.doris.catalog.Column`); `getEffectiveIcebergFormatVersion` needs its 3 iceberg string constants inlined as literals (`"table-override."`/`"table-default."`/`"format-version"`) to go SDK-free. NB: extraction only matters WHEN the 23 classes are deleted (flip). Also: even after the split, `StatisticsUtil` + `IcebergSysTable` remain independent SDK-leak sites → "fe-core fully iceberg-SDK-free" is a flip-scoped success criterion, not this step's. + +## Iron-rule guardrail (do not cheat reachability) +Do NOT add an `if(iceberg)`/`switch(dlaType)`/`instanceof HMSExternalTable` arm in `PhysicalPlanTranslator` to divert HMS-iceberg into `PluginDrivenScanNode`. The only correct lever is the table class (`instanceof PluginDrivenExternalTable`) + the gateway's `getScanPlanProvider(handle)` returning the iceberg provider for ICEBERG-format handles — which is exactly what the flip delivers. From 905ff08e646c3f63501d85a061d0dc427094bf4c Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 11:55:03 +0800 Subject: [PATCH 031/334] [doc](catalog) HMS cutover retype design: atomic-flip model + dormant-precondition build-out + capability-twin map (recon wf_e0586006-60f) Code-grounded recon (8 dimension readers + completeness/ordering critics, HEAD- verified) supersedes the "6 independent sub-batches" framing: the HMS catalog/ table retype is ONE atomic, catalog-wide flip fronted by a large dormant connector-SPI build-out, then a mechanical cyclic delete. Key established facts (all HEAD-verified): - HiveConnectorProvider.getType()=="hms" already exists; the flip = add "hms" to SPI_READY_TYPES + 3 GSON tag remaps + case-hms removal, one atomic commit (GSON remap cannot precede the flip: labels-must-be-unique vs write-registration). - Heterogeneity + non-persisted dlaType force mapping "HMSExternalTable" -> PluginDrivenMvccExternalTable + HiveConnector declaring SUPPORTS_MVCC_SNAPSHOT; hive/hudi/iceberg resolved per-handle inside the connector. - Dormant blockers: Kerberos authenticator, connector read-SPI (partition_columns /listPartitions/stats/views/caps), freshness-aware getTableSnapshot, cross-plugin createSiblingConnector SPI + gateway wrapper handle, read-ACID query-finish wiring. - datasource/hive+hudi+iceberg are a cyclic delete unit (delete strictly last). - 4 open decisions surfaced for sign-off (stats fast-path, cache ownership, iceberg-on-hms timing, PR shape). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../hms-cutover-retype-design-2026-07-07.md | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md diff --git a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md new file mode 100644 index 00000000000000..f410c06ee3fce9 --- /dev/null +++ b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md @@ -0,0 +1,128 @@ +# HMS cutover — catalog/table retype design (2026-07-07) + +> Produced by a code-grounded recon (`wf_e0586006-60f`: 8 dimension readers + 2 critics — completeness + ordering/risk) with lead-engineer HEAD verification of the load-bearing facts. Supersedes the "6 independent sub-batches" framing: the recon proves the retype is **one atomic flip** with a large **dormant-precondition** build-out in front of it. **Trust HEAD, not doc line numbers.** Full recon detail: `tool-results/w0bg9i509.output` (per-dimension `capabilities`/`couplingSites`/`gaps`). + +--- + +## 1. Goal / Non-goals + +**Goal.** Make a `type=hms` catalog a `PluginDrivenExternalCatalog` holding a plugin `Connector` (`HiveConnectorProvider.getType()=="hms"` already exists), its databases `PluginDrivenExternalDatabase`, and its tables `PluginDrivenMvccExternalTable` — so the heterogeneous HMS catalog (plain-hive + iceberg-on-HMS + hudi-on-HMS) is served by the connector SPI and fe-core sheds every `instanceof HMSExternal*` / `switch(dlaType)`. This unblocks deleting `datasource/hive` + `datasource/hudi` + the ~23 `datasource/iceberg` HMS-support classes. + +**Non-goals.** Full-ACID **write** stays hard-rejected (unchanged). Full-ACID + insert-only **read** stays in scope (plugin producer landed dormant, `0b19506acfe`). No new visible catalog type (D-020: one `hms` gateway, iceberg/hudi delegated internally, not a second catalog). No `if(format)`/`instanceof format`/`switch(dlaType)` in fe-core (iron rule); no property parsing in fe-core. + +--- + +## 2. The shape of the work: dormant preconditions → one atomic flip → mechanical delete + +The **ordering critic verdict** (confirmed against HEAD + paimon/iceberg precedent): the retype is **necessarily one atomic, catalog-wide switch**. It cannot be staged per-table or per-dimension, because dispatch is by **class identity** (`instanceof PluginDrivenExternalCatalog/Table`), not a runtime toggle. The moment `"hms"` enters `SPI_READY_TYPES` and the GSON tags remap, **every** hms catalog/db/table retypes at once across scan/write/DDL/stats/MVCC/GSON/event-poller/cache-routing. + +Three regimes: + +### (A) DORMANT — build ahead of the flip, zero user-visible effect +`CatalogFactory.java:103` never routes `hms` through the plugin path until `"hms"` is in `SPI_READY_TYPES`, so all connector-side scaffolding lands dormant (proven by P7.1 DDL / P7.3 write+ACID / P7.4 scan-seam). Everything in §4 is built here. + +### (B) THE FLIP — one atomic commit (the "must-be-atomic set") +1. **3 GSON tag remaps** — delete `registerSubtype` + add `registerCompatibleSubtype` for catalog (`GsonUtils.java:366`), db (`:447`), table (`:471`). *These are mutually locked:* you cannot add the compat mapping without first deleting the `registerSubtype` (else `RuntimeTypeAdapterFactory:279` throws "labels must be unique" at static-init → FE won't boot); but deleting `registerSubtype` also kills the **write** registration, which crashes image-write for any live `HMSExternalCatalog` — and `CatalogFactory:134` keeps building those until `"hms"` is in `SPI_READY_TYPES`. **⇒ GSON remap and the SPI_READY flip must ship in the same commit** (a standalone "GSON-first" PR is impossible; paimon `a26eaeff84c` did exactly this in one commit). +2. `SPI_READY_TYPES += "hms"` (`CatalogFactory.java:49-50`) + delete the dead `case "hms"` fallback (`:133-135`). +3. `buildDbForInit` `case HMS` → `new PluginDrivenExternalDatabase` (`ExternalCatalog.java:966-968`, mirror the ICEBERG case). +4. Remove the now-dead `PhysicalPlanTranslator` HMS arms (`:818-846` scan, `:925-928` hudi) + the `DLAType` import (`:65`) — they become unreachable (the `:808` PluginDriven arm wins) but still compile-reference the legacy scan nodes, blocking their deletion. +5. Neutralize the **event-poller gate** (`MetastoreEventsProcessor:116`) and **cache-routing gates** (`ExternalMetaCacheRouteResolver:66`, `HiveExternalMetaCache:203/274`, `IcebergExternalMetaCache:234`) — all four go `false` at the flip **simultaneously and silently** (no crash, no log). Event sync would halt and cache-routing collapse to `ENGINE_DEFAULT` unless handled in the atomic set (or the flip is gated until §4's event/cache work lands). *This is the strongest proof the flip is atomic.* +6. Land `HmsGsonCompatReplayTest` (template: `IcebergGsonCompatReplayTest` / `PaimonGsonCompatReplayTest`). + +### (C) DELETE — mechanical, one PR, one cyclic unit +`datasource/hive`, `datasource/hudi`, `datasource/iceberg` (legacy) **import each other cyclically** (`HudiScanNode extends HiveScanNode` :94; `HudiUtils`→`HMSExternalTable`; `IcebergHMSSource`/`IcebergScanNode`/`IcebergExternalMetaCache`→hive; `HMSExternalTable`→`IcebergDlaTable`/`HudiDlaTable`). **Hive cannot be deleted alone.** The deletion unit = `{datasource/hive legacy, datasource/hudi legacy, datasource/iceberg ~23 classes, statistics/HMSAnalysisTask, StatisticsUtil.getIcebergColumnStats + getHiveColumnStats, systable/IcebergSysTable}`, deleted together. (`fe-connector-hive` has **zero** imports of `datasource.hive` — the connector build-out is not entangled; only the fe-core legacy graph is cyclic.) + +**Recommended PR shape:** paimon-style **two-PR split** — a revertable **flip-PR** (the atomic set in B) then a mechanical **delete-PR** (C). GSON single-row MVCC does **not** force a squash: `registerCompatibleSubtype` takes a label **string**, which survives class deletion. + +--- + +## 3. The heterogeneity crux + the GSON single-row conflict (the central correctness problem) + +One HMS catalog mixes non-MVCC plain-hive + MVCC iceberg/hudi-on-HMS, today carried by **one class** `HMSExternalTable` that implements `MvccTable`/`MTMVRelatedTableIf` for **every** `dlaType`, branching at runtime on a **lazily-probed, non-persisted** `dlaType`. Two independent facts force the **same** answer: + +- **GSON:** the persisted `clazz` tag is uniformly `"HMSExternalTable"` (dlaType is transient — excluded by `HiddenAnnotationExclusionStrategy`, never in the JSON), so `registerCompatibleSubtype` can only map that **one label → one class**. Content-based dispatch is impossible in one release (no persisted discriminator). +- **buildTableInternal:** selects base-vs-Mvcc from the **catalog-level** `SUPPORTS_MVCC_SNAPSHOT` capability (`PluginDrivenExternalDatabase.java:53`), uniform for all tables in the catalog. + +**⇒ Decision (evidence-forced, matches paimon/iceberg precedent at `GsonUtils:490/494`):** map `"HMSExternalTable"` → **`PluginDrivenMvccExternalTable`** AND have the hive gateway declare `SUPPORTS_MVCC_SNAPSHOT`; resolve hive/hudi/iceberg heterogeneity **at runtime inside the connector by table handle** (hive-handle path degrades MVCC to an empty pin; iceberg-handle delegates to the sibling). The GSON read-target and what `buildTableInternal` builds live **must agree** for round-trip consistency. The safety of the single base-tag mapping rests on the verified invariant that **deserialized external-table objects never survive a replay** (metaCache transient, `initialized` reset to false, tables rebuilt live via `buildTableInternal`, MTMV/BaseTableInfo use id-refs) — pinned by `HmsGsonCompatReplayTest`. + +**Consequence to handle (MAJOR):** plain-hive now uses the Mvcc subclass, whose `loadSnapshot(empty,empty)` calls `materializeLatest()` (eager partition listing) vs the legacy `EmptyMvccSnapshot` (lazy). Mitigation: the hive gateway's `beginQuerySnapshot` returns empty for hive handles → empty pin → hive partition reads stay on the normal `listPartitions` path. And `getTableSnapshot` must become **freshness-kind-aware** (see §4 blocker) or MTMV over a hive base table never detects changes. + +--- + +## 4. Dormant precondition build-out (regime A) — the real work + +Ordered by the phase plan in §5. Blockers marked ⛔. + +### 4.1 Connector auth + property parity (fe-connector-hive) +- ⛔ **Kerberos authenticator.** Today `HiveMetadataOps:73` builds its thrift client with `catalog.getExecutionAuthenticator()` — a **real** authenticator set by `HMSExternalCatalog.initPreExecutionAuthenticator:129`. `PluginDrivenExternalCatalog` deliberately does **not** override it (base no-op), and `HiveConnector:105` uses the raw `context::executeAuthenticated` — unlike `IcebergConnector` it neither builds a plugin-side authenticator nor wraps the context in `TcclPinningConnectorContext` (contrast `IcebergConnector:162-177,783-821`). **Secured HMS would silently degrade to SIMPLE auth.** Fix: give `HiveConnector` the iceberg treatment (lazy plugin-side `HadoopAuthenticator` + `TcclPinningConnectorContext`). **Gate: a Kerberos-HMS smoke test.** +- **Property parity moves connector-side** (fe-core stays property-agnostic): TTL range checks (`file.meta.cache.ttl-second` / `partition.cache.ttl-second`, `checkProperties:108`) → hms `ConnectorFactory.validateProperties`; the `ipc.client.fallback-to-simple-auth-allowed` default (`setDefaultPropsIfMissing:234`) → `HiveConnector.deriveStorageProperties`. Regression tests for both TTL error messages + mixed kerberos/simple auth. + +### 4.2 Connector read-side SPI (fe-connector-hive is still a read-mostly skeleton) +- ⛔ **`partition_columns` emission.** `HiveConnectorMetadata.getTableSchema` does not emit partition columns → `toSchemaCacheValue` derives **empty** partition columns → every partitioned hive/hudi table reads as **unpartitioned** (wrong pruning/count, MTMV breakage). Fix: add a first-class `List partitionColumnNames` to `ConnectorTableSchema` (preferred over the MaxCompute string-property hack); connector applies the STRING→`varchar(65535)` coercion when it emits them. +- ⛔ **Missing SPI methods:** `listPartitions` (real per-partition `lastModifiedMillis`), `getTableStatistics` (HMS-param rowCount + file-list estimate fallback + dataSize), `buildTableDescriptor` (`THiveTable`/`TTableType.HIVE_TABLE`), `listSupportedSysTables`, `viewExists`/`getViewDefinition` (Presto-view base64 parsing), and `getCapabilities`. Without these, `getNameToPartitionItems`/`fetchRowCount`/`toThrift`/`$partitions`/hive-views all degrade to empty on a flipped table. +- **`getCapabilities`** must declare `SUPPORTS_MVCC_SNAPSHOT` + the per-table caps the legacy `dlaType` gates imply. **⚠ Over-eligibility risk (completeness critic):** `SUPPORTS_TOPN_LAZY_MATERIALIZE` and `SUPPORTS_NESTED_COLUMN_PRUNE` are per-table **file-format** gated in legacy (orc/parquet only, views/hudi excluded); as catalog-wide flags they'd make text/json/csv/view/hudi tables over-eligible (nested prune on non-parquet can read NULL leaves). **⇒ these two need a per-handle format gate, not a blanket capability.** +- **Fail-loud parity:** `HiveTableFormatDetector.detect` returns `UNKNOWN` silently where legacy `supportedHiveTable()` throws `NotSupportedException` ("Unsupported hive input format"). Connector must fail loud in `getTableHandle`/`getTableSchema` (message aligned to connector wording). +- **Read file-format serde nuance (completeness critic):** legacy `getFileFormatType(SessionVariable)` has a `read_hive_json_in_one_column` session-var branch (+ a first-column-string `UserException`) and a `FORMAT_TEXT` vs `FORMAT_CSV_PLAIN` distinction the connector's `HiveFileFormat.detect`+`mapFileFormatType` collapses. Decide: reproduce (needs a session-var surface on `ConnectorSession`) or accept the behavior change (document). + +### 4.3 Connector MVCC / MTMV / sys-table SPI +- ⛔ **`getTableSnapshot` freshness-kind-aware.** `PluginDrivenMvccExternalTable.getTableSnapshot` hardcodes `MTMVSnapshotIdSnapshot(snapshotId)`; for a plain-hive empty pin (`-1`) this is a **constant** → MTMV over a hive base table never detects change (**stale MV**). Fix: branch like `getPartitionSnapshot` — when the connector's freshness is `LAST_MODIFIED` (hive), emit a timestamp snapshot (`MTMVTimestampSnapshot` partitioned; `MTMVMaxTimestampSnapshot`-equivalent from a connector-supplied table-level lastDdlTime unpartitioned). Requires the connector to surface table last-DDL/modified time. +- **Sys tables:** iceberg-on-HMS `$snapshots` already throws today (`IcebergSysTable.createSysExternalTable`) → the connector-driven native path is a net improvement. Hive `$partitions` is a legacy **TVF** (`PartitionsSysTable`) — the fe-core resolver supports both native + TVF; decide keep-TVF-routed vs expose-native. + +### 4.4 Cross-plugin sibling-connector SPI (iceberg-on-HMS delegation) +- ⛔ **`ConnectorContext.createSiblingConnector(String type, Map props)`** (0 hits today). Default: `throw UnsupportedOperationException` (non-gateway connectors unaffected). Implement in `DefaultConnectorContext` (same fe-core pkg as `ConnectorFactory`) over `ConnectorFactory.createConnector(type, props, this)` → builds a real `IcebergConnector` in the **iceberg plugin's own child-first classloader** (co-packaging iceberg into the hive zip is rejected — 2nd AWS SDK poisons S3). All three TCCL pin loci are then covered **for free** because the sibling is a real `IcebergConnector` (scan-thread auto-pin via provider classloader; write/DDL via the sibling's own `TcclPinningConnectorContext`; worker pool via `pinIcebergWorkerPoolToPluginClassLoader`). +- **Property synthesis (plugin-side):** gateway injects `iceberg.catalog.type=hms` (`IcebergConnectorProperties.TYPE_HMS`) + shares hms metastore/storage/kerberos props before building the sibling. +- **Gateway-owned wrapper handle** (`HmsGatewayTableHandle{ HiveTableType format; ConnectorTableHandle delegate }`): the gateway (hive loader) must **never** `instanceof`/cast the foreign `IcebergTableHandle` (iceberg loader). All dispatch branches on the gateway-owned format enum (same loader) and passes the unwrapped delegate to the sibling. +- **Per-table metadata + write/procedure delegation** (seam A is scan-only): the gateway's `ConnectorMetadata` delegates `getTableHandle`/schema/MVCC internally keyed on `HiveTableFormatDetector` (no new engine seam). For write/procedure (which carry no handle today): add default handle-aware overloads `getWritePlanProvider(handle)`/`getProcedureOps(handle)` mirroring seam A (recommended for symmetry), or a gateway dispatching provider. + +### 4.5 Write-chain + read-txn rewire +- The plain-hive **write** path is already class-forked (`UnboundTableSinkCreator`/`BindSink`/`PhysicalPlanTranslator`), so the retype makes it fall through to the live `UnboundConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` driven by the dormant `HiveWritePlanProvider`/`HiveConnectorTransaction` — no new fe-core branches. The whole legacy sink chain becomes deletable. +- ⛔ **Read-ACID query-finish commit is unwired plugin-side.** `HiveScanPlanProvider.planAcidScan` registers a `HiveReadTransaction` (read lock + write-id snapshot) but nothing deregisters/commits it. Fix (fe-core-driven, keeps `ConnectorSession` free of query lifecycle): when the plan provider opens a read txn, `PluginDrivenScanNode` registers a `QueryFinishCallbackRegistry` callback → new `ConnectorScanPlanProvider.releaseReadTransaction(queryId)` → `HiveReadTransactionManager.deregister` (mirrors how `PluginDrivenInsertExecutor` drives the write commit). +- **Two hive write pre-checks** (`BindSink`: reject explicit-partition-spec INSERT `:668`; reject LZO read-only `:678`) → push into the hive connector write path (fail-loud), keeping fe-core connector-agnostic. +- **Env-held read txn mgr** (`Env.hiveTransactionMgr` + accessors `:568/865/1097/1101`) → delete at the read cutover once legacy `HiveScanNode` (its only caller) is deleted; the plugin `HiveReadTransactionManager` owns it per connector. +- **`BIND_BROKER_NAME`** constant (`HMSExternalCatalog:60`, referenced by `ExternalCatalog.bindBrokerName():1320`) → relocate to `BrokerProperties.BIND_BROKER_NAME_KEY` (already the identical `broker.name` key); drop the HMS copy. +- **`pluginCatalogTypeToEngine`** (`CreateTableInfo`) has no `"hms"` entry → returns null → no-ENGINE CREATE TABLE throws. Add `case "hms": return ENGINE_HIVE;`. + +### 4.6 Coupling-site neutralization (65 sites; NONE need an `if(format)` to survive) +Every `instanceof HMSExternal*` goes false at the flip and the PluginDriven arm already beside most sites takes over. Residual sites needing a **connector capability/SPI** (not a branch): `canSample`/`isSamplingPartition` (add `SUPPORTS_SAMPLE_ANALYZE`); `partition_values()` TVF (add PluginDriven to the gate + `MetadataGenerator` case); SQL-cache version tracking (generalize the type gate to plugin tables with `getUpdateTime()`); partition-level cache invalidation (add `Connector.invalidatePartition(s)` — pairs with the event pipeline); hive-view config gate (`enable_query_iceberg_views` is iceberg-specific — carry the view-enable/dialect per-connector); SHOW PARTITIONS / SHOW CREATE (connector partition-list / show-create SPI or accept generic rendering). Full list + file:line in the recon output `couplingSites`. + +### 4.7 Event pipeline relocation (Model B — flip-gated) +Covered by `hms-event-pipeline-findings-2026-07-07.md`. Thin fe-core role-aware `MasterDaemon` driver + plugin `pollOnce` SPI returning neutral change-descriptors (must cover register/unregister/rename + partition-NAME granularity). fe-core wraps `pollOnce` in an `onPluginClassLoader` pin (R-010). The one poller gate (`MetastoreEventsProcessor:116`) is part of the atomic flip set. `ExternalMetaIdMgr` can be dropped for HMS (ids name-derived) but opcode `470` + a neutral replay handler must survive on-disk (replay-CCE fix already landed `a6dc782d816`). + +--- + +## 5. Internal phase ordering (hard-sequenced) + +1. **Connector SPI additions (dormant)** — 4.1 auth+props, 4.2 read-SPI, 4.3 MVCC/sys, 4.4 sibling-SPI + gateway delegation, 4.5 read-ACID wiring + write pre-checks + BIND_BROKER + engine-map. Each lands as an independent dormant commit. *Constraints:* sibling-SPI before iceberg-on-hms routing; MVCC capability declaration before the flip or iceberg/hudi lose MVCC. +2. **Coupling-site capability seams (dormant)** — 4.6 add the connector capabilities/SPI methods + the PluginDriven arms beside each HMS arm (still dormant — HMS tables aren't PluginDriven yet). +3. **Event pipeline Model B (dormant driver + plugin SPI)** — 4.7, ready so the poller keeps flowing at the flip. +4. **THE FLIP (atomic commit)** — §2.B, including neutralizing the event/cache gates + dead translator arms + `HmsGsonCompatReplayTest`. +5. **DELETE (mechanical PR)** — §2.C cyclic unit. +6. **Hard e2e gate (R-002)** — §7. + +*Global constraints:* GSON remap atomic-with the flip (not earlier); the flip is one commit; delete strictly last; the event + cache-routing work must be **in** the flip set or the flip is gated until it lands. + +--- + +## 6. Open decisions (require sign-off before regime-A implementation) + +See the companion Chinese write-up. Summary + recommendation: + +- **D1 — Hive metadata statistics fast-path.** Legacy has a no-scan HMS/spark column-stats fast path (`getColumnStatistic` for query planning, default-gated) + `HMSAnalysisTask` metadata ANALYZE. The generic path is SQL-only. **Options:** (a) **drop** for parity with the iceberg-native flip (which already dropped it), rely on ANALYZE-collected stats; (b) add a `ConnectorStatisticsOps.getColumnStatistics` SPI to preserve it. **Recommend (a)** unless the no-ANALYZE fast path is deemed high-value for hive users. (Table-level rowCount is preserved regardless via `getTableStatistics`.) +- **D2 — Cache ownership.** Legacy fe-core owns `HiveExternalMetaCache`/hudi/iceberg-on-hms meta caches. **Options:** (a) **connector-owned** scan-side caching, retire the fe-core caches + delete the route/loader `instanceof` sites (architecture-consistent; paimon/iceberg-native precedent); (b) keep fe-core caches with engine/capability-keyed routing + a connector metastore SPI for loaders. **Recommend (a).** +- **D3 — Iceberg-on-HMS delivery vs flip timing.** The sibling-connector SPI (4.4) is substantial and is a hard precondition. **Options:** (a) **build it before the flip** → no regression; (b) at the flip, **fail-loud-reject** iceberg-on-HMS tables and deliver delegation later → a **functional regression** for existing iceberg-on-HMS tables. **Recommend (a)** (no regression). +- **D4 — PR shape (decide at flip time).** two-PR split (paimon: revertable flip + mechanical delete) vs one squash (iceberg). **Recommend two-PR.** + +--- + +## 7. Success criteria / hard gate (R-002 — do not silently skip, Rule 12) + +Docker e2e over a **live** path: (1) transactional-hive read (full-ACID+insert-only in scope; full-ACID write hard-rejected); (2) HMS event replay keeps metadata in sync post-flip; (3) **one heterogeneous catalog** holding plain-hive + iceberg-on-HMS + hudi-on-HMS, each queried through the right provider; (4) MTMV over iceberg-on-HMS + hive base tables (freshness), time-travel; (5) Kerberos-HMS smoke (§4.1). Plus: clean-room adversarial review + ENG-1 capability-twin audit over the coupling sites + `HmsGsonCompatReplayTest`. Precedent: iceberg #64688 / paimon #64446+#64653. + +--- + +## 8. References +- Recon output (full per-dimension detail): `tool-results/w0bg9i509.output`; workflow `wf_e0586006-60f`. +- `iceberg-on-hms-delegation-findings-2026-07-07.md` (sibling handoff, deferred deletions). +- `hms-event-pipeline-findings-2026-07-07.md` (Model B, event descriptors). +- Precedents: `P5-paimon-migration.md` (two-PR flip+delete, GSON), `P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md` (one-squash, GSON replay, clean-room). Seam template: `P7.4-scan-provider-per-table-seam-design.md`. +- Tracking: apache/doris#65185. From bd58bf483cfe7150dec952fbce5a0cf129e4343d Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 12:03:49 +0800 Subject: [PATCH 032/334] [doc](catalog) HMS cutover: lock 4 flip decisions (sign-off) + refresh HANDOFF to phase-1 dormant SPI build-out MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit User sign-off 2026-07-07 on the 4 open decisions in the cutover design: - D1 statistics: PRESERVE the no-scan column-stats fast path via a new ConnectorStatisticsOps.getColumnStatistics SPI (not drop-for-parity); pulls a per-column stats SPI into the regime-A build-out (design §4.2a). - D2 cache ownership: CONNECTOR-OWNED (retire fe-core hive/hudi/iceberg caches). - D3 iceberg-on-HMS: build the cross-plugin sibling SPI BEFORE the flip (no regression). - D4 PR shape: leaning two-PR (paimon), confirm at flip. HANDOFF next step = implement the dormant connector-SPI build-out (design §5 phase 1), recommended first commit = §4.1 Hive connector Kerberos authenticator + property parity. Recon/design phase closed; design doc is the authoritative plan. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 22 +++++++++++-------- .../hms-cutover-retype-design-2026-07-07.md | 15 +++++++------ 2 files changed, 21 insertions(+), 16 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index abaf82ddacd55b..c4f245e4ad765d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -24,19 +24,23 @@ --- -# 🚀 下个 session 任务 = **执行翻闸（把 HMS 目录/表改造成通用类）** +# 🚀 下个 session 任务 = **实现翻闸前的"休眠期"连接器 SPI 补齐（设计文档第 4/5 章 阶段 1）** -翻闸是全项目最大最险的一块，是解锁一切（格式委派/事件/删旧类）的**真正关口**。样板 = P5 paimon（`P5-paimon-migration.md`）、P6 iceberg（`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`），机制相同、但 HMS 因**异构（三格式）+ 事件管道 + 写链 + DLA** 而更大。 +**本 session 已完成（recon + 设计 + 签字）**：针对"HMS 目录/表 retype 到通用类"的聚焦 code-grounded 侦察（`wf_e0586006-60f`：8 维读者 + 完整性/排序两个 critic，全 HEAD 核对）→ 落地权威设计文档 **`hms-cutover-retype-design-2026-07-07.md`（起步必读 #1）**。核心结论：翻闸是**一次原子、整库同时生效**的切换（靠类身份 `instanceof PluginDriven*` 分派，非运行时开关），前置一大批**休眠期**连接器补齐，末尾一次**循环依赖单元**的机械删除。**已 HEAD 证实**：`HiveConnectorProvider.getType()=="hms"` 已存在；`createSiblingConnector` 全树 0 处（跨插件兄弟 SPI 真缺）；GSON `registerSubtype` HMS 三处（catalog:366/db:447/table:471）；`buildTableInternal` 按**目录级** `SUPPORTS_MVCC_SNAPSHOT` 选基类/Mvcc 子类；`PhysicalPlanTranslator` `:808` PluginDriven 臂先于 `:818` HMS 臂命中。 -**开工方式（守纪律，别一上来写代码）**：先做一轮**针对"HMS 目录/表 retype 到通用类"的 code-grounded 侦察 + 设计文档**——把旧 `HMSExternalCatalog`/`HMSExternalTable` 提供的每项能力（元数据 init、schema、分区、统计、DLA 分派、MVCC、系统表、事件、写/DDL）逐项映射到连接器 SPI，标出缺口（跨插件兄弟连接器 SPI、事件 Model-B 驱动 + 结构性变更 SPI、GSON `registerCompatibleSubtype` 兼容、按表 MVCC 能力），并把翻闸拆成**内部有序阶段**（连接器补齐→加 `"hms"` 进 `SPI_READY_TYPES`→GSON 兼容→删旧类→ACID/事件/异构 e2e 硬门 R-002）。**每个开放决策先中文讲背景+示例+推荐、不引任务代号，请用户签字**再实现。 +**已签字决策（2026-07-07，设计文档 §6）**： +- **D1 统计 = 保留免扫描列统计**（新增 `ConnectorStatisticsOps.getColumnStatistics` SPI + Hive/Iceberg 各实现；表级行数照旧）——**列统计 SPI 进入休眠期补齐清单（§4.2a）**。 +- **D2 缓存 = 连接器自持**（退休 fe-core 的 Hive/hudi/iceberg 元数据缓存，删 route/loader 的 `instanceof` 站点；随翻闸集落地）。 +- **D3 混合 Iceberg 表 = 翻闸前先做好跨插件委派，无倒退**（兄弟连接器 SPI §4.4 是硬前置，滑期则翻闸顺延，不走 fail-loud 拒绝）。 +- **D4 打包**（翻闸时再定）：倾向两 PR（可回滚翻闸 + 机械删除）。 -翻闸内部要覆盖的特性清单（把旧"6 子批"当**特性 inventory** 用，但**不再当作翻闸前的独立步骤**）见 `P7-cutover-scope-map-2026-07-06.md`（其 A–F 分解仍是有用的能力地图；**其"翻闸前可独立落地"的排序前提已被本 session 校正**）。 +**下一步 = 开始休眠期补齐（设计文档 §5 阶段 1，每子步一个独立休眠 commit，线上零影响）**。推荐**首个 commit = §4.1 Hive 连接器 Kerberos 认证器 + 属性校验对齐**（自包含 BLOCKER，有 `IcebergConnector` 现成模板 `:162-177,783-821`，测试门清晰）。其余子步：§4.2 读侧 SPI（`partition_columns`/`listPartitions`/`getTableStatistics`/视图/capabilities）、§4.2a 列统计 SPI、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） -1. **先读两份 findings 文档 + 本文顶部 🎯 段**。剩余 HMS 迁移 ≈ 一次翻闸；B/C/D 是它的侧面，非独立前置步。 -2. **已提交勿回炒**：P7.1 / P7.3(`0b19506acfe`) / A(`0923077fe67`) 全休眠；replay-CCE fix(`a6dc782d816`) 已合。 -3. **纪律**：先 code-grounded recon → 设计文档 → 用户签字该阶段专属决策 → 实现 → 独立 commit → 更新本 HANDOFF。上下文超 30% 找干净节点交接。**path-whitelist `git add`，严禁 `-A`**。 -4. **硬门 = ACID/事件/异构集成测试**（R-002 最大风险，需 live 路径，勿静默跳过——Rule 12）。full-ACID **写**继续硬拒；full-ACID + insert-only **读**在范围（已落地插件侧）。 +1. **先读设计文档 `hms-cutover-retype-design-2026-07-07.md`（权威计划）+ 两份 findings + 本文顶部 🎯 段**。剩余 HMS 迁移 = 一次原子翻闸 + 前置休眠补齐 + 末尾循环删除。 +2. **已提交勿回炒**：P7.1 / P7.3(`0b19506acfe`) / 按表 scan seam(`0923077fe67`) 全休眠；replay-CCE fix(`a6dc782d816`) 已合；设计文档 commit `5bfc55f6d59`。 +3. **纪律**：设计已签字 → **现在进入实现**：每子步独立休眠 commit（fresh recompile 杜绝 stale `.class`）→ 更新本 HANDOFF。上下文超 30% 找干净节点交接。**path-whitelist `git add`，严禁 `-A`**。铁律见 🧠 起步必读 #3（fe-core 不加 `if(format)`/`instanceof HMSExternal*`/`switch(dlaType)`；不解析属性；跨插件 pin TCCL）。 +4. **硬门 = ACID/事件/异构集成测试 + Kerberos-HMS 冒烟**（R-002 最大风险，需 live 路径，勿静默跳过——Rule 12）。full-ACID **写**继续硬拒；full-ACID + insert-only **读**在范围（已落地插件侧）。 --- @@ -57,7 +61,7 @@ # 🧠 起步必读 -1. **两份 findings 文档**（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）+ 本文顶部 🎯/🚀。 +1. **权威计划 = `hms-cutover-retype-design-2026-07-07.md`**（原子翻闸模型 + 休眠补齐清单 §4 + 能力孪生图 + 阶段序 §5 + 已签字决策 §6 + 硬门 §7；全 recon 明细在 `tool-results/w0bg9i509.output`）。再读**两份 findings 文档**（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）+ 本文顶部 🎯/🚀。 2. **样板**：`P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`（净室复审 + 能力孪生审计 + GSON replay 范式）。委派/缝模板 = A 的设计文档 `P7.4-scan-provider-per-table-seam-design.md`。特性地图 = `P7-cutover-scope-map-2026-07-06.md`（排序前提已校正）。 3. **铁律**：fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/`switch(dlaType)`/引擎名判别（翻闸靠"表类=通用类 + 网关按句柄委派"，不靠在 `PhysicalPlanTranslator` 加分支）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；跨插件/跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`，事件轮询后台线程 R-010 亦需）；history_schema_info nested 名 lowercase（memory）。 4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-tracking-issue`。 diff --git a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md index f410c06ee3fce9..a747a3a8b20387 100644 --- a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md +++ b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md @@ -92,7 +92,7 @@ Covered by `hms-event-pipeline-findings-2026-07-07.md`. Thin fe-core role-aware ## 5. Internal phase ordering (hard-sequenced) -1. **Connector SPI additions (dormant)** — 4.1 auth+props, 4.2 read-SPI, 4.3 MVCC/sys, 4.4 sibling-SPI + gateway delegation, 4.5 read-ACID wiring + write pre-checks + BIND_BROKER + engine-map. Each lands as an independent dormant commit. *Constraints:* sibling-SPI before iceberg-on-hms routing; MVCC capability declaration before the flip or iceberg/hudi lose MVCC. +1. **Connector SPI additions (dormant)** — 4.1 auth+props, 4.2 read-SPI, **4.2a per-column stats SPI (D1=preserve)**, 4.3 MVCC/sys, 4.4 sibling-SPI + gateway delegation, 4.5 read-ACID wiring + write pre-checks + BIND_BROKER + engine-map. Each lands as an independent dormant commit. *Constraints:* sibling-SPI before iceberg-on-hms routing; MVCC capability declaration before the flip or iceberg/hudi lose MVCC. 2. **Coupling-site capability seams (dormant)** — 4.6 add the connector capabilities/SPI methods + the PluginDriven arms beside each HMS arm (still dormant — HMS tables aren't PluginDriven yet). 3. **Event pipeline Model B (dormant driver + plugin SPI)** — 4.7, ready so the poller keeps flowing at the flip. 4. **THE FLIP (atomic commit)** — §2.B, including neutralizing the event/cache gates + dead translator arms + `HmsGsonCompatReplayTest`. @@ -103,14 +103,15 @@ Covered by `hms-event-pipeline-findings-2026-07-07.md`. Thin fe-core role-aware --- -## 6. Open decisions (require sign-off before regime-A implementation) +## 6. Decisions — LOCKED (user sign-off 2026-07-07) -See the companion Chinese write-up. Summary + recommendation: +- **D1 — Hive metadata statistics fast-path → PRESERVE via new SPI (option b).** Add a `ConnectorStatisticsOps.getColumnStatistics(session, handle, colName)` SPI returning a neutral column-stat DTO (count/ndv/numNulls/dataSize/avgSize) + a `PluginDrivenExternalTable.getColumnStatistic` override. The hive connector ports `getHiveColumnStats`/`setStatData` (no-scan HMS/spark col-stats, query-planning fast path, `enable_fetch_iceberg_stats`-gated for the iceberg sibling); the iceberg connector ports `getIcebergColumnStats`. Table-level rowCount stays via `getTableStatistics` regardless. *(Chosen over dropping-for-parity: the no-ANALYZE fast path is kept high-value for hive.)* **⇒ this pulls the per-column stats SPI into the regime-A build-out (§4.2a).** +- **D2 — Cache ownership → CONNECTOR-OWNED (option a).** Retire the fe-core `HiveExternalMetaCache`/hudi/iceberg-on-hms meta caches; the connector owns scan-side caching (paimon/iceberg-native precedent). Delete the route/loader `instanceof HMSExternalCatalog` sites (`ExternalMetaCacheRouteResolver:66`, `HiveExternalMetaCache:203/274`, `IcebergExternalMetaCache:234`) + the partition-cache-inconsistency self-heal moves connector-side (or is dropped — decide at implementation). This is the largest catalog-adjacent structural item; it lands with the flip set. +- **D3 — Iceberg-on-HMS delivery → BUILD SIBLING SPI BEFORE THE FLIP (option a), no regression.** The cross-plugin sibling-connector SPI (§4.4) is a hard precondition and must land dormant before the flip so existing iceberg-on-HMS tables keep working. If it slips, the flip slips (no fail-loud-reject regression). +- **D4 — PR shape (decide at flip time).** two-PR split (paimon: revertable flip + mechanical delete) vs one squash (iceberg). **Leaning two-PR;** confirm at the flip. -- **D1 — Hive metadata statistics fast-path.** Legacy has a no-scan HMS/spark column-stats fast path (`getColumnStatistic` for query planning, default-gated) + `HMSAnalysisTask` metadata ANALYZE. The generic path is SQL-only. **Options:** (a) **drop** for parity with the iceberg-native flip (which already dropped it), rely on ANALYZE-collected stats; (b) add a `ConnectorStatisticsOps.getColumnStatistics` SPI to preserve it. **Recommend (a)** unless the no-ANALYZE fast path is deemed high-value for hive users. (Table-level rowCount is preserved regardless via `getTableStatistics`.) -- **D2 — Cache ownership.** Legacy fe-core owns `HiveExternalMetaCache`/hudi/iceberg-on-hms meta caches. **Options:** (a) **connector-owned** scan-side caching, retire the fe-core caches + delete the route/loader `instanceof` sites (architecture-consistent; paimon/iceberg-native precedent); (b) keep fe-core caches with engine/capability-keyed routing + a connector metastore SPI for loaders. **Recommend (a).** -- **D3 — Iceberg-on-HMS delivery vs flip timing.** The sibling-connector SPI (4.4) is substantial and is a hard precondition. **Options:** (a) **build it before the flip** → no regression; (b) at the flip, **fail-loud-reject** iceberg-on-HMS tables and deliver delegation later → a **functional regression** for existing iceberg-on-HMS tables. **Recommend (a)** (no regression). -- **D4 — PR shape (decide at flip time).** two-PR split (paimon: revertable flip + mechanical delete) vs one squash (iceberg). **Recommend two-PR.** +### §4.2a — Per-column statistics SPI (from D1=preserve) +- Add `ConnectorStatisticsOps.getColumnStatistics(session, handle, colName)` → neutral column-stat DTO. `PluginDrivenExternalTable.getColumnStatistic` override consults it (query-planning cache-miss fast path). Hive connector ports the HMS/spark col-stats read (incl. partition-level col stats `getHivePartitionColumnStats` for the ANALYZE metadata path); iceberg connector ports `getIcebergColumnStats` behind `enable_fetch_iceberg_stats`. This also gives `HMSAnalysisTask`'s metadata fast path a home (revisit whether `ExternalAnalysisTask` consults the new SPI vs SQL-only FULL analyze — the DTO makes preserving the metadata path feasible). --- From 6c59c528d14da925c4d8fd731d9efaa8b1511a77 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 12:36:27 +0800 Subject: [PATCH 033/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.1=20Kerberos=20authenticator:=20drafted+verified=20impl=20not?= =?UTF-8?q?es;=20blocked=20on=20metastore-parser=20module=20home?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Deep code-grounded design of the first dormant commit (Hive connector plugin-side Kerberos authenticator). Verified the fix is NOT a clean iceberg mirror: - ThriftHmsClient.doAs pins the RPC TCCL to the SYSTEM classloader, so we substitute ONLY the AuthAction (plugin UGI doAs), not the whole context (iceberg's wrapper would fight that pin). hadoop+fe-kerberos are child-first so plugin UGI == RPC UGI. - AuthAction.execute is generic -> anonymous class (no lambda); doAs takes PrivilegedExceptionAction -> callable::call. Reproduces legacy HMSBaseProperties.initHadoopAuthenticator selection via metastore-spi HmsMetaStoreProperties.kerberos(). OPEN DECISION blocking apply: no neutral fe-connector-metastore-hms exists; the HMS MetaStoreProvider is registered only in fe-connector-metastore-iceberg/-paimon. Options (a) reuse metastore-iceberg (smell), (b) metastore-paimon, (c) new neutral hms module, (d) build in fe-connector-hms. Reused by all hive read-SPI -> needs a deliberate call. Full drafted diffs + test captured for verbatim apply once decided. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- ...ver-kerberos-auth-impl-notes-2026-07-07.md | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md diff --git a/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md b/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md new file mode 100644 index 00000000000000..f93161f1824b6c --- /dev/null +++ b/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md @@ -0,0 +1,45 @@ +# HMS cutover §4.1 — Hive connector Kerberos authenticator: implementation notes (2026-07-07) + +> First dormant commit of the cutover build-out (design doc `hms-cutover-retype-design-2026-07-07.md` §4.1). Design fully specified + drafted + verified against HEAD; **BLOCKED on one architecture decision** (metastore-parser module home, §3 below). Apply verbatim once decided. Provenance: lead recon + drafting subagent `a419cb99c60a3537d`. + +## 1. The blocker (verified) +After the flip, `HiveConnector` gets a `DefaultConnectorContext` whose `executeAuthenticated` = `authSupplier.get().execute(task)` with `authSupplier = () -> NOOP_AUTH` (2-arg ctor, `DefaultConnectorContext.java:100-101,164-166`; `NOOP_AUTH.execute` runs `task.call()` directly). The HMS metastore RPC runs via `ThriftHmsClient` whose `authAction = context::executeAuthenticated` (`HiveConnector.java:105`) → **SIMPLE auth for a Kerberos HMS = silent security downgrade.** + +## 2. The fix (settled design — do NOT use iceberg's full context wrapper) +Substitute **only the `AuthAction`** given to `ThriftHmsClient` with a plugin-side Kerberos `doAs`; do **not** wrap the whole `ConnectorContext` in a plugin-loader-pinning `TcclPinningConnectorContext` (iceberg's approach). Reason: `ThriftHmsClient.doAs` (`ThriftHmsClient.java:529-538`) deliberately pins the RPC's TCCL to `ClassLoader.getSystemClassLoader()`; iceberg's wrapper would re-pin to the plugin loader inside `executeAuthenticated` and override it. hadoop + fe-kerberos are bundled **child-first** in the hive plugin (parent-first prefixes are only `org.apache.doris.connector.`/`.filesystem.`), so the plugin's `HadoopAuthenticator` and the plugin's `ThriftHmsClient` RPC share the SAME (plugin) UGI copy — the `doAs` correctly wraps the RPC with **zero** TCCL change. + +Two verified corrections to the naive draft: +- `AuthAction.execute` is a **generic** method (` T execute(Callable)`) → a lambda cannot implement it (JLS 15.27.3, which is why the existing code uses a method reference). Use an **anonymous class**. +- `HadoopAuthenticator.doAs` takes `PrivilegedExceptionAction`, not `Callable` → adapt with `callable::call` (mirrors `TcclPinningConnectorContext:108` `auth.doAs(task::call)`). + +Authenticator selection must reproduce legacy `HMSBaseProperties.initHadoopAuthenticator` (`HMSBaseProperties.java:152-185`): storage-kerberos (`hadoop.security.authentication=kerberos`) first, else HMS-metastore kerberos via the metastore-spi parser `HmsMetaStoreProperties.kerberos()` (which itself covers the HDFS-kerberos backward-compat fallback). `buildPluginAuthenticator` is `static` + package-visible for KDC-free unit testing (mirrors iceberg). + +## 3. ⛔ OPEN DECISION (blocks apply): metastore-parser module home +`MetaStoreProviders.bindForType("hms", …)` + `HmsMetaStoreProperties` need a **registered `MetaStoreProvider`** on the hive plugin classpath. Verified: the HMS provider is registered **only** in `fe-connector-metastore-iceberg` (`IcebergHmsMetaStoreProvider`) and `fe-connector-metastore-paimon` (`PaimonHmsMetaStoreProvider`) — there is **no neutral `fe-connector-metastore-hms`**. Options: +- **(a) depend on `fe-connector-metastore-iceberg`** (subagent default) — zero new code, reuses the canonical parser, but bundles an **iceberg-named** module (and its rest/glue/dlf/jdbc providers) into the **hive** plugin: a dependency-inversion smell at the foundation of the hive migration. +- **(b) depend on `fe-connector-metastore-paimon`** — same smell, different flavor. +- **(c) create a neutral `fe-connector-metastore-hms`** module (extract the HMS provider + `HmsMetaStoreProperties`; iceberg/paimon can later dedup onto it) — cleanest long-term, reused by ALL hive read-SPI, but new-module scope now. +- **(d) build the authenticator inside `fe-connector-hms`** (reuse its existing `HmsClientConfig`/`HmsConfHelper` HiveConf + add `fe-kerberos`), hand-reproducing the small legacy key logic — HMS-native, minimal deps, but bypasses the centralized metastore-spi parser (a minor inconsistency with the established "meta parsing via metastore-spi" pattern). + +This choice is reused by the entire hive read-SPI (design §4.2), so it must be settled deliberately, not defaulted. + +## 4. Drafted implementation (apply once §3 decided; the metastore import/dep lines follow the chosen option) + +### `HiveConnector.java` +- Imports: `HmsMetaStoreProperties`, `MetaStoreProviders` (from the chosen module), `kerberos.HadoopAuthenticator`, `kerberos.KerberosAuthSpec`, `kerberos.KerberosAuthenticationConfig`, `org.apache.hadoop.conf.Configuration`, `java.util.Optional`, `java.util.concurrent.Callable`. +- New const `HADOOP_SECURITY_AUTHENTICATION = "hadoop.security.authentication"`; new fields `volatile HadoopAuthenticator pluginAuth; volatile boolean pluginAuthComputed;`. +- `createClient()` tail: build `HadoopAuthenticator auth = pluginAuthenticator()`; if non-null, `authAction = new ThriftHmsClient.AuthAction(){ public T execute(Callable c) throws Exception { return auth.doAs(c::call);} }`; else `authAction = context::executeAuthenticated` (byte-for-byte unchanged). +- `pluginAuthenticator()` lazy double-checked memo → `buildPluginAuthenticator(properties)`. +- `static HadoopAuthenticator buildPluginAuthenticator(Map properties)`: storage-kerberos branch → `getHadoopAuthenticator(buildHadoopConf(properties))`; else `bindForType("hms", properties, emptyMap).kerberos()` present+`hasCredentials()` → `getHadoopAuthenticator(new KerberosAuthenticationConfig(principal, keytab, conf))` with `conf` set `hadoop.security.authentication=kerberos` + `hive.metastore.sasl.enabled=true`; else null. +- `private static Configuration buildHadoopConf(Map)`: plain `new Configuration()` (NOT HiveConf — HiveConf static-init drags hadoop-mapreduce onto the unit-test classpath), `setClassLoader(HiveConnector.class.getClassLoader())`, `properties.forEach(conf::set)`. *(Consider filtering to `hadoop.*`/auth keys rather than dumping all props — minor.)* + +### `fe-connector-hive/pom.xml` +Add the chosen metastore module (+ `fe-kerberos` if option (d)). No assembly change for options (a)/(b)/(c): `plugin-zip.xml`'s runtime `dependencySet` bundles the new module + transitives (metastore-api/-spi, fe-kerberos, fe-foundation) into `lib/`; they are hadoop-free (no version clash with the plugin's `hadoop-common`). Verify with a redeploy smoke test. + +### Test: `HiveConnectorPluginAuthenticatorTest` (JUnit 5, no Mockito; login is lazy so no KDC) +Mirror `IcebergConnectorPluginAuthenticatorTest`. Assert `buildPluginAuthenticator` returns non-null for storage-kerberos props and for HMS-metastore-kerberos-with-simple-storage props (the blocker case), and null for simple-auth / plain / kerberos-without-creds. + +## 5. Residual risks (Rule 12) +- Cannot fully verify without a live KDC — unit tests assert built/not-built only; the real `loginUserFromKeytab` + doAs-wraps-RPC (and that the child-first plugin UGI copy is the one the RPC uses) needs a Kerberized-HMS redeploy smoke (the design's R-002 gate). +- Write-path HDFS Kerberos (`HiveWritePlanProvider`/committer use `context.executeAuthenticated` directly) is NOT covered by this commit — it still resolves to NOOP. It is dormant (P7.3) + full-ACID write is hard-rejected; insert-only write to Kerberos HDFS auth is a follow-up at the write cutover (wrap the write-path context there, where a plugin-loader TCCL pin is appropriate — no ThriftHmsClient SYSTEM-pin conflict on that path). +- Case-sensitivity of the storage-auth value + hive-site.xml-only krb settings not folded into the login conf — both mirror iceberg's existing behavior. From e3c5f20cdaa5a8a80e0013cace27d8faa85baa57 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 13:10:15 +0800 Subject: [PATCH 034/334] [feat](catalog) hive connector plugin-side Kerberos authenticator + neutral fe-connector-metastore-hms module (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First dormant precondition of the HMS cutover (design §4.1). After the future catalog flip, HiveConnector receives a DefaultConnectorContext whose executeAuthenticated resolves to NOOP (SIMPLE) auth (2-arg ctor), so a Kerberos HMS metastore RPC would silently downgrade to SIMPLE. Fix: build a lazy plugin-side HadoopAuthenticator from the catalog properties (storage-kerberos, else HMS-metastore kerberos via the shared metastore-spi parser) and substitute ONLY the AuthAction given to ThriftHmsClient (an anonymous class, since AuthAction.execute is generic; auth.doAs(callable::call)). Deliberately NOT iceberg's TcclPinningConnectorContext: ThriftHmsClient.doAs pins the RPC's TCCL to the system classloader, and hadoop + fe-kerberos are bundled child-first, so the plugin's authenticator and the plugin's RPC share the same UGI copy — the doAs wraps the RPC with zero TCCL change. Non-Kerberos behavior is byte-for-byte unchanged. The plugin-side HMS metastore/Kerberos parser has no neutral home (registered only in fe-connector-metastore-iceberg/-paimon), so add a neutral fe-connector-metastore-hms module (DefaultHmsMetaStoreProperties + HmsMetaStoreProvider via ServiceLoader) that fe-connector-hive depends on; iceberg/paimon can dedup onto it later. The neutral API/SPI (HmsMetaStoreProperties, AbstractHmsMetaStoreProperties) already existed. Dormant: hive is not in SPI_READY_TYPES. Full Kerberos correctness needs the R-002 docker smoke gate (live KDC); this commit's unit test asserts authenticator built/not-built for the storage-kerberos, HMS-kerberos-over-simple-storage (the blocker), simple, plain, and blank-creds cases. Verified: fe-connector-metastore-hms + fe-connector-hive BUILD SUCCESS, Tests run: 5 Failures: 0 Errors: 0, 0 checkstyle violations (both modules). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- fe/fe-connector/fe-connector-hive/pom.xml | 10 ++ .../doris/connector/hive/HiveConnector.java | 101 ++++++++++++++++- .../HiveConnectorPluginAuthenticatorTest.java | 106 ++++++++++++++++++ .../fe-connector-metastore-hms/pom.xml | 105 +++++++++++++++++ .../hms/DefaultHmsMetaStoreProperties.java | 47 ++++++++ .../metastore/hms/HmsMetaStoreProvider.java | 49 ++++++++ ....connector.metastore.spi.MetaStoreProvider | 17 +++ fe/fe-connector/pom.xml | 1 + 8 files changed, 435 insertions(+), 1 deletion(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTest.java create mode 100644 fe/fe-connector/fe-connector-metastore-hms/pom.xml create mode 100644 fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/DefaultHmsMetaStoreProperties.java create mode 100644 fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/HmsMetaStoreProvider.java create mode 100644 fe/fe-connector/fe-connector-metastore-hms/src/main/resources/META-INF/services/org.apache.doris.connector.metastore.spi.MetaStoreProvider diff --git a/fe/fe-connector/fe-connector-hive/pom.xml b/fe/fe-connector/fe-connector-hive/pom.xml index 30d66dbcadf21c..3bc4646ca89de1 100644 --- a/fe/fe-connector/fe-connector-hive/pom.xml +++ b/fe/fe-connector/fe-connector-hive/pom.xml @@ -80,6 +80,16 @@ under the License. ${project.version} + + + ${project.groupId} + fe-connector-metastore-hms + ${project.version} + + org.apache.logging.log4j log4j-api diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 56b7f08689926a..c5f3f24828c8c2 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -26,14 +26,22 @@ import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.ThriftHmsClient; +import org.apache.doris.connector.metastore.HmsMetaStoreProperties; +import org.apache.doris.connector.metastore.spi.MetaStoreProviders; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.kerberos.HadoopAuthenticator; +import org.apache.doris.kerberos.KerberosAuthSpec; +import org.apache.doris.kerberos.KerberosAuthenticationConfig; +import org.apache.hadoop.conf.Configuration; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import java.io.IOException; import java.util.Collections; import java.util.Map; +import java.util.Optional; +import java.util.concurrent.Callable; /** * Hive connector implementation. Manages the lifecycle of @@ -43,10 +51,19 @@ public class HiveConnector implements Connector { private static final Logger LOG = LogManager.getLogger(HiveConnector.class); + // Catalog property key gating the plugin-side Kerberos authenticator (value matches AuthType.KERBEROS). + private static final String HADOOP_SECURITY_AUTHENTICATION = "hadoop.security.authentication"; + private final Map properties; private final ConnectorContext context; private volatile HmsClient hmsClient; + // Lazily-built plugin-side Kerberos authenticator (single-owner auth), null for a non-Kerberos catalog. + // Its doAs acts on the PLUGIN's UserGroupInformation copy — the one the plugin's ThriftHmsClient RPC reads + // (hadoop + fe-kerberos bundled child-first) — not the app-loader copy the FE-injected context would use. + private volatile HadoopAuthenticator pluginAuth; + private volatile boolean pluginAuthComputed; + // Read-transaction manager for transactional (ACID) Hive scans. One per connector, keyed by query. // Plugin-owned and dormant until the read cutover wires its query-finish commit (see the manager). private final HiveReadTransactionManager readTxnManager = new HiveReadTransactionManager(); @@ -102,10 +119,92 @@ private HmsClient createClient() { context.getCatalogName(), config.getMetastoreUri(), config.getMetastoreType(), poolSize); - ThriftHmsClient.AuthAction authAction = context::executeAuthenticated; + // For a Kerberos catalog run the metastore RPC under the PLUGIN's UGI doAs (buildPluginAuthenticator), + // NOT the FE-injected context: after the catalog flip that context resolves to NOOP (SIMPLE) auth, which + // would silently downgrade a Kerberos HMS. AuthAction.execute is a generic method ( T execute(...)), + // so it cannot be a lambda — use an anonymous class. ThriftHmsClient.doAs already pins the RPC's TCCL to + // the system classloader; the plugin's HadoopAuthenticator only wraps it in a UGI doAs (no TCCL change). + HadoopAuthenticator auth = pluginAuthenticator(); + ThriftHmsClient.AuthAction authAction; + if (auth != null) { + authAction = new ThriftHmsClient.AuthAction() { + @Override + public T execute(Callable callable) throws Exception { + return auth.doAs(callable::call); + } + }; + } else { + authAction = context::executeAuthenticated; + } return new ThriftHmsClient(config, authAction); } + /** + * Lazily builds and memoizes the plugin-side Kerberos authenticator that {@link #createClient()} wraps the + * metastore RPC under, so the RPC uses the PLUGIN's own {@code UserGroupInformation} copy (hadoop + + * fe-kerberos are bundled child-first in the hive plugin). Returns {@code null} for a non-Kerberos catalog + * so the FE-injected auth path is preserved unchanged. Construction is cheap — the keytab login is lazy in + * {@code getUGI()} on the first {@code doAs}. + */ + private HadoopAuthenticator pluginAuthenticator() { + if (!pluginAuthComputed) { + synchronized (this) { + if (!pluginAuthComputed) { + pluginAuth = buildPluginAuthenticator(properties); + pluginAuthComputed = true; + } + } + } + return pluginAuth; + } + + /** + * Resolves the plugin-side Kerberos authenticator for the catalog, or {@code null} for a non-Kerberos + * catalog. Two Kerberos sources are covered, in precedence order (mirroring the legacy + * {@code HMSBaseProperties.initHadoopAuthenticator}): + *
      + *
    1. Storage Kerberos — the raw {@code hadoop.security.authentication=kerberos} passthrough + * (HDFS login), built from the catalog Hadoop configuration. When storage is Kerberos this single + * login also carries the HMS metastore RPC (same UGI).
      2. + *
    2. HMS-metastore Kerberos with non-Kerberos storage — a secured Hive Metastore whose data + * storage is simple (e.g. a Kerberized HMS over S3). The HMS client principal/keytab facts + * ({@link HmsMetaStoreProperties#kerberos()}, resolved through the shared metastore-spi parser) feed a + * {@link KerberosAuthenticationConfig}, so the {@code doAs} logs in the same client identity fe-core + * used. The HMS service principal / SASL settings ride the catalog's own HiveConf, not the + * login.
      3. + *
    + * Package-visible + static for KDC-free unit testing. + */ + static HadoopAuthenticator buildPluginAuthenticator(Map properties) { + if ("kerberos".equalsIgnoreCase(properties.get(HADOOP_SECURITY_AUTHENTICATION))) { + return HadoopAuthenticator.getHadoopAuthenticator(buildHadoopConf(properties)); + } + HmsMetaStoreProperties hms = (HmsMetaStoreProperties) MetaStoreProviders.bindForType( + HmsClientConfig.METASTORE_TYPE_HMS, properties, Collections.emptyMap()); + Optional spec = hms.kerberos(); + if (spec.isPresent() && spec.get().hasCredentials()) { + Configuration conf = buildHadoopConf(properties); + conf.set("hadoop.security.authentication", "kerberos"); + conf.set("hive.metastore.sasl.enabled", "true"); + return HadoopAuthenticator.getHadoopAuthenticator( + new KerberosAuthenticationConfig(spec.get().getPrincipal(), spec.get().getKeytab(), conf)); + } + return null; + } + + /** + * Builds a plain Hadoop {@link Configuration} from the catalog properties for the authenticator. A plain + * {@code new Configuration()} (NOT {@code HiveConf}) is used deliberately: HiveConf static-init would drag + * hadoop-mapreduce onto the unit-test classpath. The classloader is pinned to the plugin loader so the + * child-first (plugin) copy of the auth classes is resolved. + */ + private static Configuration buildHadoopConf(Map properties) { + Configuration conf = new Configuration(); + conf.setClassLoader(HiveConnector.class.getClassLoader()); + properties.forEach(conf::set); + return conf; + } + @Override public void close() throws IOException { HmsClient c = hmsClient; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTest.java new file mode 100644 index 00000000000000..2a3fdd46c511cb --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTest.java @@ -0,0 +1,106 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.kerberos.HadoopAuthenticator; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Unit tests for {@link HiveConnector#buildPluginAuthenticator(Map)} — the connector-owned plugin-side + * Kerberos authenticator resolution. + * + *

    The load-bearing case is HMS-metastore Kerberos with simple (non-Kerberos) storage + * (e.g. a Kerberized Hive Metastore over S3). After the catalog flip the FE-injected + * {@code ConnectorContext.executeAuthenticated} resolves to NOOP (SIMPLE) auth, so a Kerberos HMS would be + * silently downgraded unless the connector owns the login itself. These tests pin that the connector builds a + * plugin authenticator from the HMS client principal/keytab facts, and does NOT build one when the metastore + * is simple-auth (which would force needless SIMPLE-vs-Kerberos churn). + * + *

    The actual keytab login is lazy (on first {@code doAs}), so these assertions never touch a KDC. + */ +public class HiveConnectorPluginAuthenticatorTest { + + private static Map props(String... kv) { + Map m = new HashMap<>(); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + /** Storage-level Kerberos (raw hadoop.security.authentication) — unchanged prior behavior. */ + @Test + public void storageKerberosBuildsAuthenticator() { + HadoopAuthenticator auth = HiveConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hadoop.security.authentication", "kerberos", + "hadoop.kerberos.principal", "doris@EXAMPLE.COM", + "hadoop.kerberos.keytab", "/etc/security/doris.keytab")); + Assertions.assertNotNull(auth, "storage kerberos must yield a plugin authenticator"); + } + + /** + * THE BLOCKER CASE: a Kerberized HMS whose data storage is simple. Storage auth is unset, so the storage + * gate is off; the connector must fall back to the HMS client-principal/keytab facts and still build a + * plugin authenticator (mirroring the fe-core HMS authenticator it replaces). + */ + @Test + public void hmsMetastoreKerberosWithSimpleStorageBuildsAuthenticator() { + HadoopAuthenticator auth = HiveConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hive.metastore.authentication.type", "kerberos", + "hive.metastore.client.principal", "doris@EXAMPLE.COM", + "hive.metastore.client.keytab", "/etc/security/doris.keytab")); + Assertions.assertNotNull(auth, + "HMS-metastore kerberos with simple storage must yield a plugin authenticator"); + } + + /** A simple-auth HMS builds no authenticator (a spurious one would force needless SIMPLE-vs-Kerberos churn). */ + @Test + public void hmsSimpleAuthReturnsNull() { + HadoopAuthenticator auth = HiveConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hive.metastore.authentication.type", "simple")); + Assertions.assertNull(auth, "simple-auth HMS must not build a plugin authenticator"); + } + + /** A plain HMS with no auth configured builds no authenticator. */ + @Test + public void plainHmsWithoutKerberosReturnsNull() { + HadoopAuthenticator auth = HiveConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083")); + Assertions.assertNull(auth, "plain HMS without kerberos must not build an authenticator"); + } + + /** + * HMS declares kerberos auth-type but the client principal/keytab are blank — the {@code hasCredentials} + * guard must reject it (an authenticator with no login pair would fail obscurely at first doAs). + */ + @Test + public void hmsKerberosWithBlankCredsReturnsNull() { + HadoopAuthenticator auth = HiveConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hive.metastore.authentication.type", "kerberos")); + Assertions.assertNull(auth, "kerberos HMS without a client principal/keytab pair must not build one"); + } +} diff --git a/fe/fe-connector/fe-connector-metastore-hms/pom.xml b/fe/fe-connector/fe-connector-metastore-hms/pom.xml new file mode 100644 index 00000000000000..51ebaf0b9d6f27 --- /dev/null +++ b/fe/fe-connector/fe-connector-metastore-hms/pom.xml @@ -0,0 +1,105 @@ + + + + 4.0.0 + + + org.apache.doris + fe-connector + ${revision} + ../pom.xml + + + fe-connector-metastore-hms + jar + Doris FE Connector Metastore HMS + + The neutral, engine-agnostic Hive Metastore (HMS) metastore backend: the concrete + DefaultHmsMetaStoreProperties impl + its MetaStoreProvider entry (META-INF/services), discovered by + ServiceLoader at CREATE-CATALOG time. The shared extension point, framework, and engine-neutral HMS + conf base (AbstractHmsMetaStoreProperties) live in fe-connector-metastore-spi; this module supplies + only the neutral flavor of validate() — the shared HMS connection check, with no warehouse + requirement (HMS is a metastore, not a storage). Bundled child-first into the consuming connector + plugin-zip (NOT compiled into fe-core); only that plugin classpath sees this provider, so + bindForType("hms") resolves within-engine. + + + + + + ${project.groupId} + fe-connector-metastore-spi + ${project.version} + + + + ${project.groupId} + fe-connector-metastore-api + ${project.version} + + + + ${project.groupId} + fe-foundation + ${project.version} + + + + ${project.groupId} + fe-kerberos + ${project.version} + + + + org.apache.commons + commons-lang3 + + + org.junit.jupiter + junit-jupiter + test + + + + + doris-fe-connector-metastore-hms + + + org.apache.maven.plugins + maven-dependency-plugin + + + + copy-plugin-deps + none + + + + + + diff --git a/fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/DefaultHmsMetaStoreProperties.java b/fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/DefaultHmsMetaStoreProperties.java new file mode 100644 index 00000000000000..270c8de97697ce --- /dev/null +++ b/fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/DefaultHmsMetaStoreProperties.java @@ -0,0 +1,47 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.metastore.hms; + +import org.apache.doris.connector.metastore.spi.AbstractHmsMetaStoreProperties; +import org.apache.doris.foundation.property.ConnectorPropertiesUtils; + +import java.util.Map; + +/** + * The shared, engine-neutral Hive Metastore (HMS) metastore backend. All of the conf + * ({@code toHiveConfOverrides}), the neutral {@code hive.*}/{@code hadoop.security.*} keys, and the + * connection rules live in the shared {@link AbstractHmsMetaStoreProperties}. {@link #validate()} runs + * ONLY the connection check — an HMS is a metastore, not a storage, so it imposes no warehouse requirement. + */ +public final class DefaultHmsMetaStoreProperties extends AbstractHmsMetaStoreProperties { + + private DefaultHmsMetaStoreProperties(Map raw, Map storageHadoopConfig) { + super(raw, storageHadoopConfig); + } + + public static DefaultHmsMetaStoreProperties of(Map raw, Map storageHadoopConfig) { + DefaultHmsMetaStoreProperties props = new DefaultHmsMetaStoreProperties(raw, storageHadoopConfig); + ConnectorPropertiesUtils.bindConnectorProperties(props, raw); + return props; + } + + @Override + public void validate() { + validateConnection(); + } +} diff --git a/fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/HmsMetaStoreProvider.java b/fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/HmsMetaStoreProvider.java new file mode 100644 index 00000000000000..fcb429eb9dc703 --- /dev/null +++ b/fe/fe-connector/fe-connector-metastore-hms/src/main/java/org/apache/doris/connector/metastore/hms/HmsMetaStoreProvider.java @@ -0,0 +1,49 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.metastore.hms; + +import org.apache.doris.connector.metastore.HmsMetaStoreProperties; +import org.apache.doris.connector.metastore.spi.MetaStoreProvider; +import org.apache.doris.foundation.property.ConnectorPropertiesUtils; + +import java.util.Map; +import java.util.Set; + +/** Selects the shared, engine-neutral Hive Metastore backend ({@code catalog.type == hms}). */ +public final class HmsMetaStoreProvider implements MetaStoreProvider { + + @Override + public boolean supportsType(String catalogType) { + return "hms".equalsIgnoreCase(catalogType); + } + + @Override + public HmsMetaStoreProperties bind(Map properties, Map storageHadoopConfig) { + return DefaultHmsMetaStoreProperties.of(properties, storageHadoopConfig); + } + + @Override + public Set sensitivePropertyKeys() { + return ConnectorPropertiesUtils.getSensitiveKeys(DefaultHmsMetaStoreProperties.class); + } + + @Override + public String name() { + return "HMS"; + } +} diff --git a/fe/fe-connector/fe-connector-metastore-hms/src/main/resources/META-INF/services/org.apache.doris.connector.metastore.spi.MetaStoreProvider b/fe/fe-connector/fe-connector-metastore-hms/src/main/resources/META-INF/services/org.apache.doris.connector.metastore.spi.MetaStoreProvider new file mode 100644 index 00000000000000..d595306a04ddf2 --- /dev/null +++ b/fe/fe-connector/fe-connector-metastore-hms/src/main/resources/META-INF/services/org.apache.doris.connector.metastore.spi.MetaStoreProvider @@ -0,0 +1,17 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +org.apache.doris.connector.metastore.hms.HmsMetaStoreProvider diff --git a/fe/fe-connector/pom.xml b/fe/fe-connector/pom.xml index d7a2208686505e..1b01ef06f46562 100644 --- a/fe/fe-connector/pom.xml +++ b/fe/fe-connector/pom.xml @@ -47,6 +47,7 @@ under the License. Maven topologically orders the reactor by the dependency graph. --> fe-connector-metastore-paimon fe-connector-metastore-iceberg + fe-connector-metastore-hms fe-connector-es fe-connector-trino fe-connector-maxcompute From 5dc575a6a35f3333faebf86b413a405819bb81a2 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 13:11:11 +0800 Subject: [PATCH 035/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.1=20DONE=20(hive=20Kerberos=20authenticator=20+=20neutral=20H?= =?UTF-8?q?MS=20module);=20HANDOFF=20next=20=3D=20=C2=A74.2=20read-side=20?= =?UTF-8?q?SPI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mark the Kerberos authenticator impl-notes DONE (decision C = new neutral fe-connector-metastore-hms module; commit e63b03fb490, 5/5 tests green). Refresh HANDOFF: §4.1 landed dormant with the SYSTEM-pin rationale + write-path-auth residual noted; next sub-step = §4.2 Hive connector read-side SPI (partition_columns, listPartitions, getTableStatistics, views, capabilities), then §4.2a column stats. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 5 ++++- .../tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md | 2 +- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c4f245e4ad765d..624638c43ccc1e 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -34,7 +34,10 @@ - **D3 混合 Iceberg 表 = 翻闸前先做好跨插件委派，无倒退**（兄弟连接器 SPI §4.4 是硬前置，滑期则翻闸顺延，不走 fail-loud 拒绝）。 - **D4 打包**（翻闸时再定）：倾向两 PR（可回滚翻闸 + 机械删除）。 -**下一步 = 开始休眠期补齐（设计文档 §5 阶段 1，每子步一个独立休眠 commit，线上零影响）**。推荐**首个 commit = §4.1 Hive 连接器 Kerberos 认证器 + 属性校验对齐**（自包含 BLOCKER，有 `IcebergConnector` 现成模板 `:162-177,783-821`，测试门清晰）。其余子步：§4.2 读侧 SPI（`partition_columns`/`listPartitions`/`getTableStatistics`/视图/capabilities）、§4.2a 列统计 SPI、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 +**休眠期补齐进行中（设计文档 §5 阶段 1，每子步一个独立休眠 commit，线上零影响）**： +- **✅ §4.1 DONE（commit `e63b03fb490`）**：Hive 连接器插件侧 Kerberos 认证器（翻闸后注入 context 是 NOOP/SIMPLE，加密 HMS 会静默降级——修法=只替换给 `ThriftHmsClient` 的 `AuthAction` 走插件 UGI `doAs`，**不套 iceberg 的 TcclPinningConnectorContext**，因 `ThriftHmsClient.doAs` 已 pin SYSTEM 类加载器且 hadoop+fe-kerberos 是 child-first→插件认证器与插件 RPC 同一 UGI 副本）。**决策 C 签字**：新建中立 `fe-connector-metastore-hms` 模块（HMS 解析器原只在 iceberg/paimon 元数据模块各一份；中立 API/SPI 已存在）。全绿 5/5 + 0 checkstyle。详见 `hms-cutover-kerberos-auth-impl-notes-2026-07-07.md`。**残留**：写链 HDFS Kerberos（`HiveWritePlanProvider`/committer 直用 `context.executeAuthenticated`，仍 NOOP）留到写翻闸补；真 KDC 冒烟归 R-002 硬门。 +- **下一步 = §4.2 读侧 SPI**（`HiveConnectorMetadata` 目前是 read-mostly 骨架）：⛔`partition_columns` 发射（不发→分区表被当无分区读，建议给 `ConnectorTableSchema` 加一等 `partitionColumnNames` 字段 + 连接器侧做 STRING→varchar(65535) 强转）、⛔`listPartitions`/`getTableStatistics`（HMS 参数行数 + file-list 估算兜底 + dataSize）/`buildTableDescriptor`/`listSupportedSysTables`/`viewExists`+`getViewDefinition`（Presto-view base64）/`getCapabilities`（含 `SUPPORTS_MVCC_SNAPSHOT`；但 TOPN/NESTED_PRUNE 需**按表 format 门**非目录级 flag，见完整性 critic）、UNKNOWN-format fail-loud 对齐、read_hive_json_in_one_column serde 细节（决定复刻或记录变更）。 +- 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） 1. **先读设计文档 `hms-cutover-retype-design-2026-07-07.md`（权威计划）+ 两份 findings + 本文顶部 🎯 段**。剩余 HMS 迁移 = 一次原子翻闸 + 前置休眠补齐 + 末尾循环删除。 diff --git a/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md b/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md index f93161f1824b6c..97c2f3938ff2eb 100644 --- a/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md +++ b/plan-doc/tasks/hms-cutover-kerberos-auth-impl-notes-2026-07-07.md @@ -1,6 +1,6 @@ # HMS cutover §4.1 — Hive connector Kerberos authenticator: implementation notes (2026-07-07) -> First dormant commit of the cutover build-out (design doc `hms-cutover-retype-design-2026-07-07.md` §4.1). Design fully specified + drafted + verified against HEAD; **BLOCKED on one architecture decision** (metastore-parser module home, §3 below). Apply verbatim once decided. Provenance: lead recon + drafting subagent `a419cb99c60a3537d`. +> **✅ DONE — landed dormant in commit `e63b03fb490`.** First dormant commit of the cutover build-out (design doc `hms-cutover-retype-design-2026-07-07.md` §4.1). **Decision on §3 = option (C): new neutral `fe-connector-metastore-hms` module** (user sign-off 2026-07-07). Verified: fe-connector-metastore-hms + fe-connector-hive BUILD SUCCESS, `HiveConnectorPluginAuthenticatorTest` 5/5, 0 checkstyle (both modules). Provenance: lead recon + drafting subagent `a419cb99c60a3537d`, build-out subagent `ae17052f2b14afa31`. The remaining sections are the as-built record. ## 1. The blocker (verified) After the flip, `HiveConnector` gets a `DefaultConnectorContext` whose `executeAuthenticated` = `authSupplier.get().execute(task)` with `authSupplier = () -> NOOP_AUTH` (2-arg ctor, `DefaultConnectorContext.java:100-101,164-166`; `NOOP_AUTH.execute` runs `task.call()` directly). The HMS metastore RPC runs via `ThriftHmsClient` whose `authAction = context::executeAuthenticated` (`HiveConnector.java:105`) → **SIMPLE auth for a Kerberos HMS = silent security downgrade.** From f5eea00665e9128d135857193d908fa33662970e Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 16:01:30 +0800 Subject: [PATCH 036/334] [feat](catalog) hive connector emits partition_columns + widens string partition cols (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HiveConnectorMetadata.getTableSchema now marks which of its emitted columns are partition columns, for the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue), via the cross-connector "partition_columns" CSV-of-raw-names table-property — the identical key the paimon/iceberg/maxcompute connectors already emit. Without it every partitioned hive/hudi table would be read as unpartitioned after the HMS retype (wrong partition pruning / row count, MTMV breakage). String partition columns are widened to varchar(65533) (== fe-type ScalarType.MAX_VARCHAR_LENGTH), replicating legacy HMSExternalTable.initPartitionColumns "to be same as doris managed table"; non-string partition columns and plain string DATA columns are left exactly as HmsTypeMapping produced them. This deliberately reuses the established partition_columns property convention rather than adding a first-class field to the shared ConnectorTableSchema SPI: the property is the unanimous mechanism across all three already-flipped connectors, and hive partition-key names are identifiers (no comma) so the CSV encoding is unambiguous. Dormant: hive is not in SPI_READY_TYPES, so getTableSchema is exercised only by tests today; zero online effect. Tests: HiveConnectorMetadataSchemaTest (6) pins raw-names/declaration order, string partition -> varchar(65533), non-string partition kept, string data column untouched, unpartitioned emits no property. fe-connector-hive metadata suite green (34), checkstyle 0, connector-import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../connector/hive/HiveConnectorMetadata.java | 60 +++++- .../hive/HiveConnectorMetadataSchemaTest.java | 193 ++++++++++++++++++ 2 files changed, 249 insertions(+), 4 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index c83cbda935b7dd..cc3e336f8ddaf7 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; @@ -77,6 +78,17 @@ public class HiveConnectorMetadata implements ConnectorMetadata { private static final Logger LOG = LogManager.getLogger(HiveConnectorMetadata.class); + // FE-internal schema-control property key: a CSV of the RAW remote partition-column names. The generic + // fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) reads it to derive which of the emitted + // columns are partition columns; it is the same key the paimon/iceberg/maxcompute connectors emit and is + // stripped from the user-facing SHOW CREATE properties by fe-core. + private static final String PARTITION_COLUMNS_PROPERTY = "partition_columns"; + + // Connector-side spelling of fe-type ScalarType.MAX_VARCHAR_LENGTH (the connector must not import fe-type); + // a hive `string` partition column is widened to varchar(65533) for legacy parity. Paimon hardcodes the + // identical 65533. + private static final int MAX_VARCHAR_LENGTH = 65533; + private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; @@ -156,16 +168,27 @@ public ConnectorTableSchema getTableSchema( HmsTableInfo tableInfo = hmsClient.getTable(dbName, tableName); List columns = buildColumns(tableInfo); - List partitionKeys = buildPartitionKeys(tableInfo); + List partitionKeys = coercePartitionKeyStringToVarchar(buildPartitionKeys(tableInfo)); - // Merge: regular columns + partition columns + // Merge: regular columns + partition columns (partition columns last, mirroring legacy + // HMSExternalTable full-schema order: data columns then partition keys). List allColumns = new ArrayList<>(columns.size() + partitionKeys.size()); allColumns.addAll(columns); allColumns.addAll(partitionKeys); String formatType = detectFormatType(tableInfo); - Map tableProperties = tableInfo.getParameters() != null - ? tableInfo.getParameters() : Collections.emptyMap(); + // Copy the HMS table parameters so the FE-internal partition_columns marker can be stamped without + // mutating the shared tableInfo map. + Map tableProperties = new HashMap<>( + tableInfo.getParameters() != null ? tableInfo.getParameters() : Collections.emptyMap()); + // Mark which emitted columns are partition columns for the generic fe-core consumer. Without this + // property every partitioned hive/hudi table is read as unpartitioned (wrong pruning/row count, MTMV + // breakage). The value is a CSV of the RAW partition-key names in declaration order; hive partition-key + // names are identifiers (no comma) so the CSV encoding is unambiguous. + if (!partitionKeys.isEmpty()) { + tableProperties.put(PARTITION_COLUMNS_PROPERTY, partitionKeys.stream() + .map(ConnectorColumn::getName).collect(Collectors.joining(","))); + } return new ConnectorTableSchema(tableName, allColumns, formatType, tableProperties); } @@ -544,6 +567,35 @@ private List buildPartitionKeys(HmsTableInfo tableInfo) { return partKeys; } + /** + * Widens a hive {@code string} partition column to {@code varchar(65533)}, replicating legacy + * {@code HMSExternalTable.initPartitionColumns}: a bare-string partition column is coerced to + * {@code varchar(ScalarType.MAX_VARCHAR_LENGTH)} "to be same as doris managed table", while every other + * declared type (int/date/timestamp/decimal/varchar(n)/char(n)/...) is kept exactly as + * {@code HmsTypeMapping} produced it. The gate is the mapped connector type name {@code STRING} (hive + * {@code string}, and {@code binary} when not mapped to varbinary, both land on it), matching legacy's + * {@code PrimitiveType.STRING} check. The widened column keeps the same name/comment/nullability/flags, so + * the full-schema entry and the partition-column view carry the identical type (legacy mutated one shared + * {@code Column} in place). + */ + private static List coercePartitionKeyStringToVarchar(List partitionKeys) { + if (partitionKeys.isEmpty()) { + return partitionKeys; + } + List coerced = new ArrayList<>(partitionKeys.size()); + for (ConnectorColumn col : partitionKeys) { + if ("STRING".equals(col.getType().getTypeName())) { + coerced.add(new ConnectorColumn(col.getName(), + ConnectorType.of("VARCHAR", MAX_VARCHAR_LENGTH, -1), + col.getComment(), col.isNullable(), col.getDefaultValue(), + col.isKey(), col.isAutoInc(), col.isAggregated())); + } else { + coerced.add(col); + } + } + return coerced; + } + private Map getDefaultValues(HmsTableInfo tableInfo) { try { return hmsClient.getDefaultColumnValues( diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java new file mode 100644 index 00000000000000..09f540197c680a --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -0,0 +1,193 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +/** + * Tests {@link HiveConnectorMetadata#getTableSchema} partition-column emission (§4.2 read-side SPI). + * + *

    WHY: the generic fe-core consumer {@code PluginDrivenExternalTable.toSchemaCacheValue} derives a table's + * partition columns SOLELY from a {@code partition_columns} CSV-of-raw-names table-property (the same key + * paimon/iceberg/maxcompute emit). If the hive connector appends partition columns to the schema but omits + * that property, every partitioned hive/hudi table reads as unpartitioned (wrong pruning / row count, MTMV + * breakage). These assertions pin: the property carries the raw partition-key names in declaration order; a + * {@code string} PARTITION column is widened to {@code varchar(65533)} for legacy + * {@code HMSExternalTable.initPartitionColumns} parity while a {@code string} DATA column and a non-string + * partition column are left untouched; partition columns come after data columns; an unpartitioned table + * emits no property.

    + */ +public class HiveConnectorMetadataSchemaTest { + + private static final String PARQUET_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + + private ConnectorTableSchema schemaOf(HmsTableInfo tableInfo) { + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(tableInfo), Collections.emptyMap(), new FakeConnectorContext()); + HiveTableHandle handle = new HiveTableHandle.Builder( + tableInfo.getDbName(), tableInfo.getTableName(), HiveTableType.HIVE).build(); + return metadata.getTableSchema(null, handle); + } + + private static ConnectorColumn col(String name, String typeName) { + return new ConnectorColumn(name, ConnectorType.of(typeName), null, true, null); + } + + private static HmsTableInfo.Builder partitionedTable() { + return HmsTableInfo.builder() + .dbName("db").tableName("t") + .inputFormat(PARQUET_INPUT_FORMAT) + .columns(Arrays.asList(col("id", "INT"), col("name", "STRING"))) + .partitionKeys(Arrays.asList(col("year", "INT"), col("region", "STRING"))) + .parameters(Collections.emptyMap()); + } + + @Test + public void testPartitionColumnsPropertyEmittedWithRawNamesInOrder() { + ConnectorTableSchema schema = schemaOf(partitionedTable().build()); + Assertions.assertEquals("year,region", + schema.getProperties().get("partition_columns")); + } + + @Test + public void testStringPartitionColumnWidenedToVarchar65533() { + ConnectorTableSchema schema = schemaOf(partitionedTable().build()); + ConnectorColumn region = columnByName(schema, "region"); + Assertions.assertEquals("VARCHAR", region.getType().getTypeName()); + Assertions.assertEquals(65533, region.getType().getPrecision()); + } + + @Test + public void testNonStringPartitionColumnKeepsDeclaredType() { + ConnectorTableSchema schema = schemaOf(partitionedTable().build()); + Assertions.assertEquals("INT", columnByName(schema, "year").getType().getTypeName()); + } + + @Test + public void testStringDataColumnIsNotWidened() { + // Only PARTITION string columns are coerced; a plain data column of type string stays STRING. + ConnectorTableSchema schema = schemaOf(partitionedTable().build()); + Assertions.assertEquals("STRING", columnByName(schema, "name").getType().getTypeName()); + } + + @Test + public void testPartitionColumnsComeAfterDataColumns() { + ConnectorTableSchema schema = schemaOf(partitionedTable().build()); + List names = schema.getColumns().stream() + .map(ConnectorColumn::getName).collect(Collectors.toList()); + Assertions.assertEquals(Arrays.asList("id", "name", "year", "region"), names); + } + + @Test + public void testUnpartitionedTableEmitsNoPartitionColumnsProperty() { + HmsTableInfo tableInfo = HmsTableInfo.builder() + .dbName("db").tableName("t") + .inputFormat(PARQUET_INPUT_FORMAT) + .columns(Arrays.asList(col("id", "INT"), col("name", "STRING"))) + .partitionKeys(Collections.emptyList()) + .parameters(Collections.emptyMap()) + .build(); + ConnectorTableSchema schema = schemaOf(tableInfo); + Assertions.assertFalse(schema.getProperties().containsKey("partition_columns")); + } + + private static ConnectorColumn columnByName(ConnectorTableSchema schema, String name) { + return schema.getColumns().stream() + .filter(c -> c.getName().equals(name)) + .findFirst() + .orElseThrow(() -> new AssertionError("column not found: " + name)); + } + + /** + * Minimal {@link HmsClient} double: {@code getTable} echoes the prebuilt {@link HmsTableInfo}; + * {@code getDefaultColumnValues} returns none. The rest fail loud. + */ + private static final class FakeHmsClient implements HmsClient { + private final HmsTableInfo tableInfo; + + FakeHmsClient(HmsTableInfo tableInfo) { + this.tableInfo = tableInfo; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + return tableInfo; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, + List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} From 9cc3de1746b7e2f4410d020edc84c567c63898a3 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 16:03:42 +0800 Subject: [PATCH 037/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20partition=5Fcolumns=20emission=20DONE;=20correct=20varcha?= =?UTF-8?q?r=20width=20(65533)=20+=20property-vs-field=20decision?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mark §4.2 partition-column emission done (32b9526f689). Two doc corrections grounded against HEAD: - The remaining-work note recommended adding a first-class partitionColumnNames field to ConnectorTableSchema "preferred over the MaxCompute string-property hack". That framing is wrong: the partition_columns property is the UNANIMOUS convention across all three already-flipped connectors (paimon:313, iceberg:443, maxcompute:163). Hive reuses it; no SPI type change. - The STRING->varchar coercion width is 65533 (ScalarType.MAX_VARCHAR_LENGTH), not 65535 — the "65535" in both the design doc and the legacy HMSExternalTable.java:835 comment is stale. Also fold the recon (wf_2f4b0381-1e6 D3/D4) scoping into the HANDOFF next-step: per-method sizes, HiveConnector.getCapabilities is unset today, and the TOPN/NESTED_PRUNE per-table-format-gate needs a design decision. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 5 +++-- plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md | 2 +- 2 files changed, 4 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 624638c43ccc1e..6618bbb6eb8f50 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -36,7 +36,8 @@ **休眠期补齐进行中（设计文档 §5 阶段 1，每子步一个独立休眠 commit，线上零影响）**： - **✅ §4.1 DONE（commit `e63b03fb490`）**：Hive 连接器插件侧 Kerberos 认证器（翻闸后注入 context 是 NOOP/SIMPLE，加密 HMS 会静默降级——修法=只替换给 `ThriftHmsClient` 的 `AuthAction` 走插件 UGI `doAs`，**不套 iceberg 的 TcclPinningConnectorContext**，因 `ThriftHmsClient.doAs` 已 pin SYSTEM 类加载器且 hadoop+fe-kerberos 是 child-first→插件认证器与插件 RPC 同一 UGI 副本）。**决策 C 签字**：新建中立 `fe-connector-metastore-hms` 模块（HMS 解析器原只在 iceberg/paimon 元数据模块各一份；中立 API/SPI 已存在）。全绿 5/5 + 0 checkstyle。详见 `hms-cutover-kerberos-auth-impl-notes-2026-07-07.md`。**残留**：写链 HDFS Kerberos（`HiveWritePlanProvider`/committer 直用 `context.executeAuthenticated`，仍 NOOP）留到写翻闸补；真 KDC 冒烟归 R-002 硬门。 -- **下一步 = §4.2 读侧 SPI**（`HiveConnectorMetadata` 目前是 read-mostly 骨架）：⛔`partition_columns` 发射（不发→分区表被当无分区读，建议给 `ConnectorTableSchema` 加一等 `partitionColumnNames` 字段 + 连接器侧做 STRING→varchar(65535) 强转）、⛔`listPartitions`/`getTableStatistics`（HMS 参数行数 + file-list 估算兜底 + dataSize）/`buildTableDescriptor`/`listSupportedSysTables`/`viewExists`+`getViewDefinition`（Presto-view base64）/`getCapabilities`（含 `SUPPORTS_MVCC_SNAPSHOT`；但 TOPN/NESTED_PRUNE 需**按表 format 门**非目录级 flag，见完整性 critic）、UNKNOWN-format fail-loud 对齐、read_hive_json_in_one_column serde 细节（决定复刻或记录变更）。 +- **✅ §4.2 分区列发射 DONE（`32b9526f689`）**：连接器在 `getTableSchema` 发射跨连接器约定的 `partition_columns` 属性（**复用**既有约定、非新增字段——paimon `PaimonConnectorMetadata:313`/iceberg `:443`/maxcompute `:163` 三家均如此发射，已 HEAD 证实；设计文档原"新增一等字段、优于字符串 hack"那句已证伪并在设计文档 §4.2 更正），并把 `string` 分区列 widen 到 **varchar(65533)**（=`ScalarType.MAX_VARCHAR_LENGTH`，**非 65535**——旧代码注释 `HMSExternalTable:835` 与本设计原文都把宽度写错；仅 `PrimitiveType.STRING` 强转，非 string 分区列/普通 string 数据列不动）。全绿（6 新测 + 34 metadata 套 + checkstyle 0 + import gate 净）。 +- **下一步 = §4.2 其余读侧 SPI**（`HiveConnectorMetadata` 仍是 read-mostly 骨架；⚠`HiveConnector` **今天 `getCapabilities` 未 override→空集**，须补）。侦察（`wf_2f4b0381-1e6` D3/D4 + 亲手核对）给出的规模/落点：⛔`listPartitions`（含 per-partition `lastModifiedMillis`，经 `HiveExternalMetaCache`，**M**）、⛔`getTableStatistics`（HMS 参数 numRows + file-list 估算兜底 + dataSize，**S/M**）、`buildTableDescriptor`（`THiveTable`/`TTableType.HIVE_TABLE`，~6 行，**S**；iceberg `IcebergConnectorMetadata:672` 已有 HIVE_TABLE fork 模板）、`listSupportedSysTables`+`viewExists`+`getViewDefinition`（hive view=同表对象带 base64 Presto view-text；`partitions` 系统表旧是 **TVF-routed** 非 scan-routed，**M**）、`getCapabilities`（含 `SUPPORTS_MVCC_SNAPSHOT`；但 **TOPN/NESTED_PRUNE 旧是按表 orc/parquet-format 门**，连接器级 flag 会让 text/json/csv/view/hudi 过度合格→须按句柄 format 门，**需一次设计决策**）、UNKNOWN-format fail-loud 对齐（旧 `NotSupportedException("Unsupported hive input format: "+fmt)`）、`read_hive_json_in_one_column` serde 细节（决定复刻或记录变更；完整 serde→TFileFormatType 映射见 recon 输出 `tasks/wi2yvdkqr.output`）。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） diff --git a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md index a747a3a8b20387..f1f278e535141f 100644 --- a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md +++ b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md @@ -58,7 +58,7 @@ Ordered by the phase plan in §5. Blockers marked ⛔. - **Property parity moves connector-side** (fe-core stays property-agnostic): TTL range checks (`file.meta.cache.ttl-second` / `partition.cache.ttl-second`, `checkProperties:108`) → hms `ConnectorFactory.validateProperties`; the `ipc.client.fallback-to-simple-auth-allowed` default (`setDefaultPropsIfMissing:234`) → `HiveConnector.deriveStorageProperties`. Regression tests for both TTL error messages + mixed kerberos/simple auth. ### 4.2 Connector read-side SPI (fe-connector-hive is still a read-mostly skeleton) -- ⛔ **`partition_columns` emission.** `HiveConnectorMetadata.getTableSchema` does not emit partition columns → `toSchemaCacheValue` derives **empty** partition columns → every partitioned hive/hudi table reads as **unpartitioned** (wrong pruning/count, MTMV breakage). Fix: add a first-class `List partitionColumnNames` to `ConnectorTableSchema` (preferred over the MaxCompute string-property hack); connector applies the STRING→`varchar(65535)` coercion when it emits them. +- ✅ **`partition_columns` emission — DONE (`32b9526f689`, dormant).** `HiveConnectorMetadata.getTableSchema` did not mark which emitted columns are partition columns → `toSchemaCacheValue` derived **empty** partition columns → every partitioned hive/hudi table read as **unpartitioned** (wrong pruning/count, MTMV breakage). Fix landed = emit the cross-connector **`partition_columns` CSV-of-raw-names table-property** (the *unanimous* mechanism — verified emitted by paimon `PaimonConnectorMetadata:313`, iceberg `IcebergConnectorMetadata:443`, maxcompute `MaxComputeConnectorMetadata:163`; the design's earlier "add a first-class `partitionColumnNames` field, preferred over the string-property hack" was **wrong** — the property is the established convention and hive partition-key names are comma-free identifiers, so the field would only fork a working SPI). Connector also widens a `string` partition column to **`varchar(65533)`** — the exact `ScalarType.MAX_VARCHAR_LENGTH`, **not 65535** (the "65535" in this doc AND in the legacy `HMSExternalTable.java:835` comment are both stale; the constant is 65533) — replicating legacy `initPartitionColumns` (only `PrimitiveType.STRING` is coerced; non-string partition cols and plain string DATA cols keep their mapped types). - ⛔ **Missing SPI methods:** `listPartitions` (real per-partition `lastModifiedMillis`), `getTableStatistics` (HMS-param rowCount + file-list estimate fallback + dataSize), `buildTableDescriptor` (`THiveTable`/`TTableType.HIVE_TABLE`), `listSupportedSysTables`, `viewExists`/`getViewDefinition` (Presto-view base64 parsing), and `getCapabilities`. Without these, `getNameToPartitionItems`/`fetchRowCount`/`toThrift`/`$partitions`/hive-views all degrade to empty on a flipped table. - **`getCapabilities`** must declare `SUPPORTS_MVCC_SNAPSHOT` + the per-table caps the legacy `dlaType` gates imply. **⚠ Over-eligibility risk (completeness critic):** `SUPPORTS_TOPN_LAZY_MATERIALIZE` and `SUPPORTS_NESTED_COLUMN_PRUNE` are per-table **file-format** gated in legacy (orc/parquet only, views/hudi excluded); as catalog-wide flags they'd make text/json/csv/view/hudi tables over-eligible (nested prune on non-parquet can read NULL leaves). **⇒ these two need a per-handle format gate, not a blanket capability.** - **Fail-loud parity:** `HiveTableFormatDetector.detect` returns `UNKNOWN` silently where legacy `supportedHiveTable()` throws `NotSupportedException` ("Unsupported hive input format"). Connector must fail loud in `getTableHandle`/`getTableSchema` (message aligned to connector wording). From cb89d612cb038b342dfa5a948e641d5d2dccefaf Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 16:27:46 +0800 Subject: [PATCH 038/334] [feat](catalog) hive connector buildTableDescriptor -> HIVE_TABLE descriptor (dormant) HiveConnectorMetadata.buildTableDescriptor is a direct port of legacy HMSExternalTable.toThrift: emit a TTableType.HIVE_TABLE descriptor carrying a THiveTable(db, table). Without it the SPI default returns null and fe-core (PluginDrivenExternalTable.toThrift) falls back to a generic SCHEMA_TABLE descriptor, so BE would build a SchemaTableDescriptor instead of the HiveTableDescriptor legacy produced. Mirrors the iceberg connector's HIVE_TABLE branch; the SPI carries no handle so this single override serves base and system tables alike (legacy used the identical fork for both). Dormant: hive is not in SPI_READY_TYPES; exercised only by tests today. Tests: HiveConnectorMetadataSchemaTest.testBuildTableDescriptorIsHiveTable pins HIVE_TABLE type + numCols + THiveTable db/table names. fe-connector-hive schema suite green (7), checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../connector/hive/HiveConnectorMetadata.java | 21 +++++++++++++++++++ .../hive/HiveConnectorMetadataSchemaTest.java | 16 ++++++++++++++ 2 files changed, 37 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index cc3e336f8ddaf7..7e17d4f1637f34 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -46,6 +46,9 @@ import org.apache.doris.connector.hms.HmsTableInfo; import org.apache.doris.connector.hms.HmsTypeMapping; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.thrift.THiveTable; +import org.apache.doris.thrift.TTableDescriptor; +import org.apache.doris.thrift.TTableType; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -227,6 +230,24 @@ public Map getColumnHandles( return result; } + /** + * Builds the BE table descriptor for a hive table, a direct port of legacy + * {@code HMSExternalTable.toThrift}: a {@code TTableType.HIVE_TABLE} carrying a {@link THiveTable}. Without + * this override the SPI default returns {@code null} and fe-core ({@code PluginDrivenExternalTable.toThrift}) + * falls back to a generic {@code SCHEMA_TABLE} descriptor. Mirrors the iceberg connector's HIVE_TABLE + * branch; the SPI signature carries no handle, so this single override serves base and system tables alike + * (legacy used the identical fork for both). + */ + @Override + public TTableDescriptor buildTableDescriptor(ConnectorSession session, + long tableId, String tableName, String dbName, String remoteName, int numCols, long catalogId) { + THiveTable tHiveTable = new THiveTable(dbName, tableName, new HashMap<>()); + TTableDescriptor desc = new TTableDescriptor( + tableId, TTableType.HIVE_TABLE, numCols, 0, tableName, dbName); + desc.setHiveTable(tHiveTable); + return desc; + } + // ========== ConnectorPushdownOps: Filter Pushdown ========== @Override diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index 09f540197c680a..1e4c92e9ac0cba 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -24,6 +24,8 @@ import org.apache.doris.connector.hms.HmsDatabaseInfo; import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.thrift.TTableDescriptor; +import org.apache.doris.thrift.TTableType; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -122,6 +124,20 @@ public void testUnpartitionedTableEmitsNoPartitionColumnsProperty() { Assertions.assertFalse(schema.getProperties().containsKey("partition_columns")); } + @Test + public void testBuildTableDescriptorIsHiveTable() { + // Without the override fe-core falls back to a generic SCHEMA_TABLE descriptor; pin that a hive table + // produces a HIVE_TABLE descriptor carrying a THiveTable with the db/table names and column count. + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(partitionedTable().build()), Collections.emptyMap(), new FakeConnectorContext()); + TTableDescriptor desc = metadata.buildTableDescriptor(null, 42L, "t", "db", "t", 4, 7L); + Assertions.assertEquals(TTableType.HIVE_TABLE, desc.getTableType()); + Assertions.assertEquals(4, desc.getNumCols()); + Assertions.assertNotNull(desc.getHiveTable()); + Assertions.assertEquals("db", desc.getHiveTable().getDbName()); + Assertions.assertEquals("t", desc.getHiveTable().getTableName()); + } + private static ConnectorColumn columnByName(ConnectorTableSchema schema, String name) { return schema.getColumns().stream() .filter(c -> c.getName().equals(name)) From 3e7e3e356a7b68eb1cf434cecb16b484698daf7e Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 16:28:56 +0800 Subject: [PATCH 039/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20buildTableDescriptor=20DONE;=20refresh=20HANDOFF=20next-s?= =?UTF-8?q?tep?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Mark §4.2 buildTableDescriptor done (164b25d9c42) and drop it from the remaining read-side SPI list. Remaining: listPartitions, getTableStatistics, sys-table/view group, getCapabilities (needs the per-table TOPN/NESTED_PRUNE format-gate design decision), UNKNOWN-format fail-loud, read_hive_json serde nuance. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 6618bbb6eb8f50..63a9b091373f18 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -37,7 +37,8 @@ **休眠期补齐进行中（设计文档 §5 阶段 1，每子步一个独立休眠 commit，线上零影响）**： - **✅ §4.1 DONE（commit `e63b03fb490`）**：Hive 连接器插件侧 Kerberos 认证器（翻闸后注入 context 是 NOOP/SIMPLE，加密 HMS 会静默降级——修法=只替换给 `ThriftHmsClient` 的 `AuthAction` 走插件 UGI `doAs`，**不套 iceberg 的 TcclPinningConnectorContext**，因 `ThriftHmsClient.doAs` 已 pin SYSTEM 类加载器且 hadoop+fe-kerberos 是 child-first→插件认证器与插件 RPC 同一 UGI 副本）。**决策 C 签字**：新建中立 `fe-connector-metastore-hms` 模块（HMS 解析器原只在 iceberg/paimon 元数据模块各一份；中立 API/SPI 已存在）。全绿 5/5 + 0 checkstyle。详见 `hms-cutover-kerberos-auth-impl-notes-2026-07-07.md`。**残留**：写链 HDFS Kerberos（`HiveWritePlanProvider`/committer 直用 `context.executeAuthenticated`，仍 NOOP）留到写翻闸补；真 KDC 冒烟归 R-002 硬门。 - **✅ §4.2 分区列发射 DONE（`32b9526f689`）**：连接器在 `getTableSchema` 发射跨连接器约定的 `partition_columns` 属性（**复用**既有约定、非新增字段——paimon `PaimonConnectorMetadata:313`/iceberg `:443`/maxcompute `:163` 三家均如此发射，已 HEAD 证实；设计文档原"新增一等字段、优于字符串 hack"那句已证伪并在设计文档 §4.2 更正），并把 `string` 分区列 widen 到 **varchar(65533)**（=`ScalarType.MAX_VARCHAR_LENGTH`，**非 65535**——旧代码注释 `HMSExternalTable:835` 与本设计原文都把宽度写错；仅 `PrimitiveType.STRING` 强转，非 string 分区列/普通 string 数据列不动）。全绿（6 新测 + 34 metadata 套 + checkstyle 0 + import gate 净）。 -- **下一步 = §4.2 其余读侧 SPI**（`HiveConnectorMetadata` 仍是 read-mostly 骨架；⚠`HiveConnector` **今天 `getCapabilities` 未 override→空集**，须补）。侦察（`wf_2f4b0381-1e6` D3/D4 + 亲手核对）给出的规模/落点：⛔`listPartitions`（含 per-partition `lastModifiedMillis`，经 `HiveExternalMetaCache`，**M**）、⛔`getTableStatistics`（HMS 参数 numRows + file-list 估算兜底 + dataSize，**S/M**）、`buildTableDescriptor`（`THiveTable`/`TTableType.HIVE_TABLE`，~6 行，**S**；iceberg `IcebergConnectorMetadata:672` 已有 HIVE_TABLE fork 模板）、`listSupportedSysTables`+`viewExists`+`getViewDefinition`（hive view=同表对象带 base64 Presto view-text；`partitions` 系统表旧是 **TVF-routed** 非 scan-routed，**M**）、`getCapabilities`（含 `SUPPORTS_MVCC_SNAPSHOT`；但 **TOPN/NESTED_PRUNE 旧是按表 orc/parquet-format 门**，连接器级 flag 会让 text/json/csv/view/hudi 过度合格→须按句柄 format 门，**需一次设计决策**）、UNKNOWN-format fail-loud 对齐（旧 `NotSupportedException("Unsupported hive input format: "+fmt)`）、`read_hive_json_in_one_column` serde 细节（决定复刻或记录变更；完整 serde→TFileFormatType 映射见 recon 输出 `tasks/wi2yvdkqr.output`）。 +- **✅ §4.2 表描述符 DONE（`164b25d9c42`）**：`buildTableDescriptor` 端口旧 `toThrift`→`TTableType.HIVE_TABLE`+`THiveTable`（否则 fe-core `PluginDrivenExternalTable.toThrift` 退化到通用 `SCHEMA_TABLE` 描述符→BE 建 SchemaTableDescriptor 而非 HiveTableDescriptor）；照 iceberg HIVE_TABLE 分支，无 handle 故 base+sys 表共用。全绿（schema 套 7 测 + checkstyle 0）。 +- **下一步 = §4.2 其余读侧 SPI**（`HiveConnectorMetadata` 仍是 read-mostly 骨架；⚠`HiveConnector` **今天 `getCapabilities` 未 override→空集**，须补）。侦察（`wf_2f4b0381-1e6` D3/D4 + 亲手核对）给出的规模/落点：⛔`listPartitions`（含 per-partition `lastModifiedMillis`，经 `HiveExternalMetaCache`，**M**）、⛔`getTableStatistics`（HMS 参数 numRows + file-list 估算兜底 + dataSize，**S/M**）、`listSupportedSysTables`+`viewExists`+`getViewDefinition`（hive view=同表对象带 base64 Presto view-text；`partitions` 系统表旧是 **TVF-routed** 非 scan-routed，**M**）、`getCapabilities`（含 `SUPPORTS_MVCC_SNAPSHOT`；但 **TOPN/NESTED_PRUNE 旧是按表 orc/parquet-format 门**，连接器级 flag 会让 text/json/csv/view/hudi 过度合格→须按句柄 format 门，**需一次设计决策**）、UNKNOWN-format fail-loud 对齐（旧 `NotSupportedException("Unsupported hive input format: "+fmt)`）、`read_hive_json_in_one_column` serde 细节（决定复刻或记录变更；完整 serde→TFileFormatType 映射见 recon 输出 `tasks/wi2yvdkqr.output`）。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） From 0f3eda56fc6a79b61babe9f741762aa64844e2a0 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 17:29:10 +0800 Subject: [PATCH 040/334] [feat](catalog) hive connector per-table Top-N lazy-materialize capability seam (dormant) Add a per-table capability seam so a heterogeneous hive catalog can declare Top-N lazy materialization only for its orc/parquet tables. The connector-wide ConnectorCapability set cannot express legacy's per-table file-format gate (HMSExternalTable.supportedHiveTopNLazyTable: parquet/orc only, no views); a blanket connector-wide declare would over-admit text/csv/json/view/hudi tables (a correctness bug for nested-column pruning, which reads NULL leaves without field ids). Mechanism (mirrors the established partition_columns marker: cache-friendly, no extra metastore round-trip, file format stays connector-side per the iron rule): - ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY: FE-internal schema-property marker carrying a CSV of ConnectorCapability.name()s that THIS table supports. - HiveConnectorMetadata.getTableSchema emits SUPPORTS_TOPN_LAZY_MATERIALIZE for a plain-hive orc/parquet non-view base table (exact input-format class match; detect()==HIVE excludes iceberg/hudi-on-HMS). - PluginDrivenExternalTable.supportsTopNLazyMaterialize / supportsNestedColumnPrune resolve via hasScanCapability = connector-wide OR per-table marker (read from the already-cached schema). Iceberg/paimon/jdbc are unchanged (never emit the marker); the marker key is stripped from user-facing SHOW CREATE properties. Nested-column prune routes through the same seam but the hive connector does NOT yet emit its marker: correctness needs per-field ids down the column tree (SlotTypeReplacer rewrites nested access to Column.uniqueId), which the hive connector does not set today -- deferred to a follow-up after end-to-end field-id verification. Dormant: hive is not in SPI_READY_TYPES, zero online effect. Tests: connector marker emission (parquet/orc present; text/view/iceberg-on-HMS absent) + fe-core additive connector-wide-OR-per-table resolution. Full connector suite (178) + fe-core capability tests (32) + checkstyle (0) + import gate green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../connector/api/ConnectorTableSchema.java | 18 +++++ .../connector/hive/HiveConnectorMetadata.java | 51 ++++++++++++++ .../hive/HiveConnectorMetadataSchemaTest.java | 54 +++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 69 ++++++++++++++----- .../PluginDrivenExternalTableTest.java | 49 ++++++++++++- 5 files changed, 221 insertions(+), 20 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java index 8ec6aec6b99823..f637363fa128ea 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java @@ -51,6 +51,24 @@ public final class ConnectorTableSchema { */ public static final String SHOW_SORT_CLAUSE_KEY = "show.sort-clause"; + /** + * Reserved property key carrying a CSV of {@link ConnectorCapability#name()} values that THIS specific + * table supports, refining the connector-wide {@link Connector#getCapabilities()} set per-table. + * + *

    A uniform-format connector (e.g. iceberg — every table orc/parquet) declares a scan capability for all + * its tables connector-wide. A heterogeneous connector (e.g. hive — orc/parquet/text/json/csv/view/hudi in + * one catalog) whose eligibility is per-table file-format gated cannot: Top-N lazy materialization and + * nested-column pruning are orc/parquet-only, and blanket-declaring them connector-wide would over-admit a + * text/json table (a correctness bug for nested-column pruning, which reads NULL leaves without field ids). + * Such a connector instead emits the capability name here, per-table, computed from that table's format.

    + * + *

    fe-core reads it ADDITIVELY (a capability counts as supported if it is in the connector-wide set OR in + * this per-table list) from the already-cached schema — no remote round-trip and no file-format inspection + * in fe-core. Single-format connectors never emit it and are unaffected. Stripped from the user-facing + * SHOW CREATE TABLE PROPERTIES(...) block.

    + */ + public static final String PER_TABLE_CAPABILITIES_KEY = "connector.per-table-capabilities"; + private final String tableName; private final List columns; private final String tableFormatType; diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 7e17d4f1637f34..d6b81a508c7432 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.hive; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; @@ -92,6 +93,19 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // identical 65533. private static final int MAX_VARCHAR_LENGTH = 65533; + // Hive input formats eligible for Top-N lazy materialization, replicating legacy + // HMSExternalTable.SUPPORTED_HIVE_TOPN_LAZY_FILE_FORMATS (parquet/orc only). The match is on the EXACT + // input-format class (not a substring), so a HoodieParquetInputFormatBase hive table — which contains + // "parquet" but is not a Top-N-lazy format in legacy — is correctly excluded. + private static final String MAPRED_PARQUET_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String ORC_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + + // HMS table type for a view, mirroring legacy HMSExternalTable.isView (which keyed off the view text flags); + // a Hive view carries tableType VIRTUAL_VIEW. + private static final String VIRTUAL_VIEW_TABLE_TYPE = "VIRTUAL_VIEW"; + private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; @@ -193,6 +207,20 @@ public ConnectorTableSchema getTableSchema( .map(ConnectorColumn::getName).collect(Collectors.joining(","))); } + // Per-table scan capabilities that the generic fe-core consumer refines the connector-wide capability + // set with. Top-N lazy materialization is orc/parquet-only in hive (legacy + // HMSExternalTable.supportedHiveTopNLazyTable), which the connector-wide SUPPORTS_TOPN_LAZY_MATERIALIZE + // cannot express for a heterogeneous hive catalog; emit it per-table so fe-core enables the optimization + // only for eligible tables and never for text/csv/json/view/hudi. + List perTableCapabilities = new ArrayList<>(); + if (supportsHiveTopNLazyMaterialize(tableInfo)) { + perTableCapabilities.add(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name()); + } + if (!perTableCapabilities.isEmpty()) { + tableProperties.put(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, + String.join(",", perTableCapabilities)); + } + return new ConnectorTableSchema(tableName, allColumns, formatType, tableProperties); } @@ -661,6 +689,29 @@ private static String resolveHiveFileFormat(String inputFormat) { return "HIVE"; } + /** + * Whether {@code tableInfo} is a plain-hive orc/parquet base table eligible for Top-N lazy materialization, + * replicating legacy {@code HMSExternalTable.supportedHiveTopNLazyTable} plus the {@code getDlaType()==HIVE} + * guard the legacy consumer ({@code MaterializeProbeVisitor}) applied: a view is excluded, an + * iceberg/hudi-on-HMS table is excluded (those are served by their own connector, which declares the + * capability connector-wide after the cutover), and only the parquet/orc input formats qualify. + */ + private boolean supportsHiveTopNLazyMaterialize(HmsTableInfo tableInfo) { + if (isView(tableInfo)) { + return false; + } + if (HiveTableFormatDetector.detect(tableInfo) != HiveTableType.HIVE) { + return false; + } + String inputFormat = tableInfo.getInputFormat(); + return MAPRED_PARQUET_INPUT_FORMAT.equals(inputFormat) || ORC_INPUT_FORMAT.equals(inputFormat); + } + + /** Whether the HMS table is a view (tableType VIRTUAL_VIEW), mirroring legacy {@code HMSExternalTable.isView}. */ + private static boolean isView(HmsTableInfo tableInfo) { + return VIRTUAL_VIEW_TABLE_TYPE.equalsIgnoreCase(tableInfo.getTableType()); + } + private static HmsTypeMapping.Options buildTypeMappingOptions(Map props) { boolean binaryAsString = Boolean.parseBoolean( props.getOrDefault(HiveConnectorProperties.ENABLE_MAPPING_BINARY_AS_STRING, "false")); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index 1e4c92e9ac0cba..a05fb3381345cb 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -53,6 +53,10 @@ public class HiveConnectorMetadataSchemaTest { private static final String PARQUET_INPUT_FORMAT = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String ORC_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + private static final String TEXT_INPUT_FORMAT = + "org.apache.hadoop.mapred.TextInputFormat"; private ConnectorTableSchema schemaOf(HmsTableInfo tableInfo) { HiveConnectorMetadata metadata = new HiveConnectorMetadata( @@ -75,6 +79,19 @@ private static HmsTableInfo.Builder partitionedTable() { .parameters(Collections.emptyMap()); } + private static HmsTableInfo.Builder unpartitionedTable(String inputFormat) { + return HmsTableInfo.builder() + .dbName("db").tableName("t") + .inputFormat(inputFormat) + .columns(Arrays.asList(col("id", "INT"), col("name", "STRING"))) + .partitionKeys(Collections.emptyList()) + .parameters(Collections.emptyMap()); + } + + private static String perTableCapabilities(ConnectorTableSchema schema) { + return schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); + } + @Test public void testPartitionColumnsPropertyEmittedWithRawNamesInOrder() { ConnectorTableSchema schema = schemaOf(partitionedTable().build()); @@ -124,6 +141,43 @@ public void testUnpartitionedTableEmitsNoPartitionColumnsProperty() { Assertions.assertFalse(schema.getProperties().containsKey("partition_columns")); } + @Test + public void testTopNLazyCapabilityMarkerEmittedForParquetAndOrc() { + // WHY: Top-N lazy materialize is orc/parquet-only in legacy hive (HMSExternalTable.supportedHiveTopNLazyTable). + // The connector-wide SUPPORTS_TOPN_LAZY_MATERIALIZE cannot express that for a heterogeneous hive catalog, so + // the connector emits it per-table; fe-core (PluginDrivenExternalTable.supportsTopNLazyMaterialize) enables the + // optimization only for tables carrying this marker. MUTATION: not emitting it -> orc/parquet hive tables lose + // Top-N lazy materialization. + Assertions.assertEquals("SUPPORTS_TOPN_LAZY_MATERIALIZE", + perTableCapabilities(schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT).build()))); + Assertions.assertEquals("SUPPORTS_TOPN_LAZY_MATERIALIZE", + perTableCapabilities(schemaOf(unpartitionedTable(ORC_INPUT_FORMAT).build()))); + } + + @Test + public void testTopNLazyCapabilityMarkerAbsentForText() { + // A text hive table is not Top-N-lazy eligible in legacy; emitting the marker would over-enable it. + Assertions.assertNull(perTableCapabilities(schemaOf(unpartitionedTable(TEXT_INPUT_FORMAT).build()))); + } + + @Test + public void testTopNLazyCapabilityMarkerAbsentForView() { + // A view is excluded even when its SD carries a parquet input format, mirroring legacy + // supportedHiveTopNLazyTable which returns false for a view BEFORE the format check. + ConnectorTableSchema schema = schemaOf( + unpartitionedTable(PARQUET_INPUT_FORMAT).tableType("VIRTUAL_VIEW").build()); + Assertions.assertNull(perTableCapabilities(schema)); + } + + @Test + public void testTopNLazyCapabilityMarkerAbsentForIcebergOnHms() { + // An iceberg-on-HMS table (table_type=ICEBERG) is served by the iceberg connector after the cutover; the + // hive connector must NOT claim Top-N for it even though its data files are parquet (detect() != HIVE). + ConnectorTableSchema schema = schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT) + .parameters(Collections.singletonMap("table_type", "ICEBERG")).build()); + Assertions.assertNull(perTableCapabilities(schema)); + } + @Test public void testBuildTableDescriptorIsHiveTable() { // Without the override fe-core falls back to a generic SCHEMA_TABLE descriptor; pin that a hive table diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index cfc50b952e93f8..b363bf0c0cc449 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -127,33 +127,61 @@ public boolean supportsColumnAutoAnalyze() { } /** - * Returns whether the underlying connector's file-scan tables support Top-N lazy materialization. - * The nereids Top-N lazy-materialize probe consults this (in place of the legacy exact-class - * {@code SUPPORT_RELATION_TYPES} membership) to enable lazy materialization for a flipped plugin table. + * Returns whether THIS table supports Top-N lazy materialization. The nereids Top-N lazy-materialize probe + * consults this (in place of the legacy exact-class {@code SUPPORT_RELATION_TYPES} membership) to enable + * lazy materialization for a flipped plugin table. Resolved per-table via {@link #hasScanCapability}: a + * uniform-format connector (iceberg) declares it connector-wide, a heterogeneous connector (hive) emits it + * only for its orc/parquet tables — so a hive text/csv/json/view table is correctly excluded, as it was in + * legacy {@code MaterializeProbeVisitor}. */ public boolean supportsTopNLazyMaterialize() { - if (!(catalog instanceof PluginDrivenExternalCatalog)) { - return false; - } - Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); - return connector != null - && connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE); + return hasScanCapability(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE); } /** - * Returns whether the underlying connector's file-scan tables support nested-column pruning (reading only - * the accessed STRUCT/ARRAY/MAP sub-fields). The nereids nested-column-prune probe - * ({@code LogicalFileScan.supportPruneNestedColumn}) consults this (in place of the legacy exact-class - * {@code IcebergExternalTable} arm) to enable pruning for a flipped plugin table, and the - * {@code SlotTypeReplacer} name-to-field-id rewrite is gated on the same capability. + * Returns whether THIS table supports nested-column pruning (reading only the accessed STRUCT/ARRAY/MAP + * sub-fields). The nereids nested-column-prune probe ({@code LogicalFileScan.supportPruneNestedColumn}) + * consults this (in place of the legacy exact-class {@code IcebergExternalTable} arm) to enable pruning for a + * flipped plugin table, and the {@code SlotTypeReplacer} name-to-field-id rewrite is gated on the same + * answer. Resolved per-table via {@link #hasScanCapability} for the same reason as Top-N: legacy gated it on + * the per-table file format (parquet/orc only), which a connector-wide capability cannot express for a + * heterogeneous hive catalog. */ public boolean supportsNestedColumnPrune() { + return hasScanCapability(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE); + } + + /** + * Whether this table supports a per-table-refinable scan-planning capability, resolved connector-wide OR + * per-table. A uniform-format connector (iceberg — every table orc/parquet) declares the capability for all + * its tables via {@link Connector#getCapabilities()}; a heterogeneous connector (hive) whose eligibility is + * per-table file-format gated instead emits the capability name per-table in the + * {@link ConnectorTableSchema#PER_TABLE_CAPABILITIES_KEY} schema marker, read here from the already-cached + * schema (no remote round-trip). The two sources are additive, so single-format connectors never emit the + * marker and behave exactly as before. fe-core never inspects the file format — the connector decides which + * of its tables qualify by emitting (or not) the capability name. + */ + private boolean hasScanCapability(ConnectorCapability capability) { if (!(catalog instanceof PluginDrivenExternalCatalog)) { return false; } Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); - return connector != null - && connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE); + if (connector == null) { + return false; + } + if (connector.getCapabilities().contains(capability)) { + return true; + } + String csv = rawTableProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); + if (csv == null || csv.isEmpty()) { + return false; + } + for (String name : csv.split(",")) { + if (name.trim().equals(capability.name())) { + return true; + } + } + return false; } /** @@ -517,8 +545,10 @@ private Map rawTableProperties() { * FE-internal schema-control keys ({@code partition_columns} / {@code primary_keys}, emitted by * the connector so {@link #initSchema()} can derive the partition columns) and the SHOW CREATE * render-hint keys ({@code show.location} / {@code show.partition-clause} / {@code show.sort-clause}, - * rendered as the LOCATION / PARTITION BY / ORDER BY clauses via {@link #getShowLocation()} etc.) are - * stripped — they are not user-facing options and must not leak into the rendered PROPERTIES(...). + * rendered as the LOCATION / PARTITION BY / ORDER BY clauses via {@link #getShowLocation()} etc.) and the + * per-table capability marker ({@code connector.per-table-capabilities}, consumed by {@link + * #supportsTopNLazyMaterialize()} / {@link #supportsNestedColumnPrune()}) are stripped — they are not + * user-facing options and must not leak into the rendered PROPERTIES(...). */ public Map getTableProperties() { Map raw = rawTableProperties(); @@ -528,7 +558,8 @@ public Map getTableProperties() { if ("partition_columns".equals(key) || "primary_keys".equals(key) || ConnectorTableSchema.SHOW_LOCATION_KEY.equals(key) || ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY.equals(key) - || ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY.equals(key)) { + || ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY.equals(key) + || ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY.equals(key)) { continue; } result.put(entry.getKey(), entry.getValue()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index d099ca60371770..385ab80cd64549 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -169,16 +169,33 @@ public void getFullSchemaDegradesWhenTableHandleAbsent() { /** * Builds a CALLS_REAL_METHODS PluginDrivenExternalTable whose connector declares exactly - * {@code capabilities}, to exercise the capability-helper methods over the real connector chain. + * {@code capabilities} connector-wide and whose cached schema emits no per-table capability marker, to + * exercise the capability-helper methods over the real connector chain. */ private static PluginDrivenExternalTable pluginTableWithCapabilities(Set capabilities) { + return pluginTableWithCapabilities(capabilities, Collections.emptyMap()); + } + + /** + * Builds a CALLS_REAL_METHODS PluginDrivenExternalTable whose connector declares {@code capabilities} + * connector-wide AND whose cached schema emits {@code perTableProps} as its raw table-property map (carrying + * the {@code connector.per-table-capabilities} marker for heterogeneous connectors like hive). Exercises the + * additive connector-wide-OR-per-table resolution in {@code hasScanCapability}. makeSureInitialized is + * stubbed to a no-op (no Env-backed init in a unit test). + */ + private static PluginDrivenExternalTable pluginTableWithCapabilities( + Set capabilities, Map perTableProps) { Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getCapabilities()).thenReturn(capabilities); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); + PluginDrivenSchemaCacheValue scv = Mockito.mock(PluginDrivenSchemaCacheValue.class); + Mockito.when(scv.getTableProperties()).thenReturn(perTableProps); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); Deencapsulation.setField(table, "catalog", catalog); + Mockito.doNothing().when(table).makeSureInitialized(); + Mockito.doReturn(Optional.of(scv)).when(table).getSchemaCacheValue(); return table; } @@ -220,6 +237,36 @@ public void supportsNestedColumnPruneReflectsConnectorCapability() { EnumSet.noneOf(ConnectorCapability.class)).supportsNestedColumnPrune()); } + @Test + public void scanCapabilityHonorsPerTableMarkerWhenConnectorWideAbsent() { + // WHY: a heterogeneous connector (hive) cannot declare Top-N lazy / nested-column-prune connector-wide + // because eligibility is per-table file-format gated (orc/parquet only) — blanket-declaring would + // over-admit a text/json table (a correctness bug for nested prune). It emits the capability name only + // for eligible tables via the connector.per-table-capabilities schema marker; the helper must honor that + // additively even when the connector-wide set is EMPTY. MUTATION: dropping the per-table marker read -> + // a flipped orc/parquet hive table silently loses the optimization -> red here. + Map topnMarker = Collections.singletonMap( + ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, + ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name()); + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), topnMarker).supportsTopNLazyMaterialize()); + // The marker is capability-specific: a Top-N marker must NOT enable nested-column pruning. + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), topnMarker).supportsNestedColumnPrune()); + // A multi-value marker enables exactly the listed capabilities. + Map bothMarker = Collections.singletonMap( + ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, + ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name() + "," + + ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE.name()); + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), bothMarker).supportsTopNLazyMaterialize()); + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), bothMarker).supportsNestedColumnPrune()); + // An empty / absent marker leaves both off (the plain-hive text-table case). + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), Collections.emptyMap()).supportsTopNLazyMaterialize()); + } + @Test public void supportsExternalMetadataPreloadReflectsConnectorCapability() { // F11: async metadata pre-load is gated on the connector-declared SUPPORTS_METADATA_PRELOAD capability From 93a30f05c5416d45516944491085a06774ac4056 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 17:32:02 +0800 Subject: [PATCH 041/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20per-table=20capability=20seam=20DONE=20+=20refresh=20next?= =?UTF-8?q?-step=20roadmap?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the signed-off decision (per-table capability seam, faithful reproduction, via schema-property marker) and commit 329853bf19c. Capture this session's read-side recon (wf_f904ad07-2d4, 5 dims HEAD-verified) as the ordered next-step roadmap for the remaining §4.2 SPI (getTableStatistics / views+$partitions / file-format serde fix / listPartitions / connector-wide caps / format-detector parity), including the NESTED_COLUMN_PRUNE field-id verification gate that blocks its marker emission. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 63a9b091373f18..d249ca7a99f881 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -38,7 +38,15 @@ - **✅ §4.1 DONE（commit `e63b03fb490`）**：Hive 连接器插件侧 Kerberos 认证器（翻闸后注入 context 是 NOOP/SIMPLE，加密 HMS 会静默降级——修法=只替换给 `ThriftHmsClient` 的 `AuthAction` 走插件 UGI `doAs`，**不套 iceberg 的 TcclPinningConnectorContext**，因 `ThriftHmsClient.doAs` 已 pin SYSTEM 类加载器且 hadoop+fe-kerberos 是 child-first→插件认证器与插件 RPC 同一 UGI 副本）。**决策 C 签字**：新建中立 `fe-connector-metastore-hms` 模块（HMS 解析器原只在 iceberg/paimon 元数据模块各一份；中立 API/SPI 已存在）。全绿 5/5 + 0 checkstyle。详见 `hms-cutover-kerberos-auth-impl-notes-2026-07-07.md`。**残留**：写链 HDFS Kerberos（`HiveWritePlanProvider`/committer 直用 `context.executeAuthenticated`，仍 NOOP）留到写翻闸补；真 KDC 冒烟归 R-002 硬门。 - **✅ §4.2 分区列发射 DONE（`32b9526f689`）**：连接器在 `getTableSchema` 发射跨连接器约定的 `partition_columns` 属性（**复用**既有约定、非新增字段——paimon `PaimonConnectorMetadata:313`/iceberg `:443`/maxcompute `:163` 三家均如此发射，已 HEAD 证实；设计文档原"新增一等字段、优于字符串 hack"那句已证伪并在设计文档 §4.2 更正），并把 `string` 分区列 widen 到 **varchar(65533)**（=`ScalarType.MAX_VARCHAR_LENGTH`，**非 65535**——旧代码注释 `HMSExternalTable:835` 与本设计原文都把宽度写错；仅 `PrimitiveType.STRING` 强转，非 string 分区列/普通 string 数据列不动）。全绿（6 新测 + 34 metadata 套 + checkstyle 0 + import gate 净）。 - **✅ §4.2 表描述符 DONE（`164b25d9c42`）**：`buildTableDescriptor` 端口旧 `toThrift`→`TTableType.HIVE_TABLE`+`THiveTable`（否则 fe-core `PluginDrivenExternalTable.toThrift` 退化到通用 `SCHEMA_TABLE` 描述符→BE 建 SchemaTableDescriptor 而非 HiveTableDescriptor）；照 iceberg HIVE_TABLE 分支，无 handle 故 base+sys 表共用。全绿（schema 套 7 测 + checkstyle 0）。 -- **下一步 = §4.2 其余读侧 SPI**（`HiveConnectorMetadata` 仍是 read-mostly 骨架；⚠`HiveConnector` **今天 `getCapabilities` 未 override→空集**，须补）。侦察（`wf_2f4b0381-1e6` D3/D4 + 亲手核对）给出的规模/落点：⛔`listPartitions`（含 per-partition `lastModifiedMillis`，经 `HiveExternalMetaCache`，**M**）、⛔`getTableStatistics`（HMS 参数 numRows + file-list 估算兜底 + dataSize，**S/M**）、`listSupportedSysTables`+`viewExists`+`getViewDefinition`（hive view=同表对象带 base64 Presto view-text；`partitions` 系统表旧是 **TVF-routed** 非 scan-routed，**M**）、`getCapabilities`（含 `SUPPORTS_MVCC_SNAPSHOT`；但 **TOPN/NESTED_PRUNE 旧是按表 orc/parquet-format 门**，连接器级 flag 会让 text/json/csv/view/hudi 过度合格→须按句柄 format 门，**需一次设计决策**）、UNKNOWN-format fail-loud 对齐（旧 `NotSupportedException("Unsupported hive input format: "+fmt)`）、`read_hive_json_in_one_column` serde 细节（决定复刻或记录变更；完整 serde→TFileFormatType 映射见 recon 输出 `tasks/wi2yvdkqr.output`）。 +- **✅ §4.2 按表能力接缝 DONE（`329853bf19c`）**：**设计决策已签字**——两个"按文件格式启用"的加速能力（Top-N 延迟 / 嵌套列裁剪）走**按表接缝**忠实复刻（不 blanket-declare、不放弃；对齐 Trino 逐表委派）。机制 = 连接器在 `getTableSchema` 发射 FE 内部 schema-property marker `ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY`（CSV of `ConnectorCapability.name()`；缓存友好、无额外 HMS 往返、format 留连接器侧，复用 `partition_columns` 模式），fe-core `PluginDrivenExternalTable.supportsTopNLazyMaterialize/supportsNestedColumnPrune` 经新 `hasScanCapability` = 连接器级 Set **OR** 按表 marker；marker key 从 SHOW CREATE 属性剔除。**Top-N 已发射**（orc/parquet 非 view 非 iceberg/hudi-on-HMS，精确 input-format 类匹配 = 旧 `SUPPORTED_HIVE_TOPN_LAZY_FILE_FORMATS`）。iceberg/paimon/jdbc 零改（永不发 marker）。全绿（连接器 178 测 + fe-core 32 测 + checkstyle 0 + import gate）。 + - ⛔**NESTED_PRUNE 遗留（硬门，下一子步前置验证）**：同接缝，连接器**暂未发射** marker——`SlotTypeReplacer.java:678-680` 把嵌套访问路径重写成 `Column.getUniqueId()`，iceberg 靠自带 field-id（`withUniqueId`）正确，**hive 连接器现在没设 field-id**；对 hive 开启前须端到端验证 hive orc/parquet 的 uniqueId 语义（BE 按 id 匹配嵌套叶子）否则读 NULL。旧 hive nested-prune 走同一 `SlotTypeReplacer` 且能工作→查旧 hive column uniqueId 赋值 vs SPI 路径 `ConnectorColumnConverter.convertColumns`；验证通过则连接器发 `SUPPORTS_NESTED_COLUMN_PRUNE` marker（parquet/orc、非 hudi），否则记差异 fail-loud。 +- **下一步 = §4.2 其余读侧 SPI**（本 session 侦察 `wf_f904ad07-2d4`，5 维全 HEAD 核对；critic 因连接错误未出、5 维已够；`HiveConnectorMetadata` 仍 read-mostly 骨架）。按建议序，每子步独立休眠 commit： + 1. **`getTableStatistics`（rowCount，S）**：镜像 paimon/iceberg 结构。level-1 = HMS 参数（`numRows`→`spark.sql.statistics.numRows`→`totalSize/estimatedRowSize`）；rowCount>0 才 `Optional.of` 否则 empty（=旧 0→UNKNOWN）。⚠`estimatedRowSize` 需 Doris slot size（连接器不能 import fe-type）→ `totalSize/rowSize` 分支要么 fe-core 侧算好 threading 进、要么先推迟；file-list 兜底 `getRowCountFromFileList`（`HMSExternalTable:803-822/1072-1123`）更重，建议推迟并 fail-loud 记差异。 + 2. **views（`viewExists`/`listViewNames`/`getViewDefinition` + 声明 `SUPPORTS_VIEW`，M）**：照 iceberg 模板；`getViewDefinition` 内复刻 Presto/Trino base64 view-text 解码（`HMSExternalTable:598-662` `parseTrinoViewDefinition`：`/* Presto View: */`→Gson→originalSql；dialect 无 caller 读，占位即可）。⚠两处设计决策：(a) `$partitions` 旧是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`，`useNativeTablePath()==false`），SPI sys-table 面（`PluginDrivenSysTable extends NativeSysTable`）**只支持 native**——`$partitions` 留 fe-core vs 建 native；(b) `BindRelation.java:623-653` plugin-view 臂硬编码 iceberg（`enable_query_iceberg_views` + iceberg 文案 + 拒时间旅行）→hive view 需泛化 config/文案 或留 HMS 臂。 + 3. **file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 + 4. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 + 5. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 + 6. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） From dadfd4d5c74c0244fa42c48aea885710f8d04782 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 18:01:03 +0800 Subject: [PATCH 042/334] [feat](catalog) hive connector getTableStatistics: exact rowCount + on-disk dataSize (dormant) Ports the first two of legacy hive's three row-count layers to the connector SPI (dormant until the HMS cutover; hive is not yet in SPI_READY_TYPES, so this has zero online effect): 1. HiveConnectorMetadata.getTableStatistics surfaces two RAW metastore facts, no Doris-type math (the connector must not import fe-type): the exact `numRows` (spark fallback only when the standard key is present-but-non-positive, legacy getRowCountFromParameters parity), and the on-disk `totalSize` (spark fallback when the standard key is absent). A count <= 0 maps to UNKNOWN (legacy "0 -> UNKNOWN" gate; matches paimon/iceberg). Params are read from the handle (loaded live by getTableHandle), so no extra HMS round-trip. 2. PluginDrivenExternalTable.fetchRowCount gains a generic fallback: when the connector reports a data size but no exact row count, estimate the cardinality as dataSize / (row width summed over the FULL schema, mirroring legacy StatisticsUtil.getHiveRowCount). The Doris-type-dependent division lives in fe-core because the connector cannot compute slot sizes. Connector-agnostic: every live connector (iceberg/paimon/jdbc) reports dataSize -1, so the branch is inert for them (verified). The third layer (estimate by listing data files) follows in a separate dormant commit. Tests: HiveConnectorMetadataStatisticsTest (12) pins the metastore-param interpretation incl. the two spark-key asymmetries and defensive parse; PluginDrivenExternalTableRowCountTest (6) pins the exact-count short-circuit, the full-schema-width estimate (partition columns included), and the empty-schema / unresolvable-handle -> UNKNOWN degradation. fe-connector-hive + fe-core: green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../connector/hive/HiveConnectorMetadata.java | 80 +++++++ .../HiveConnectorMetadataStatisticsTest.java | 173 +++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 38 +++- ...PluginDrivenExternalTableRowCountTest.java | 205 ++++++++++++++++++ 4 files changed, 494 insertions(+), 2 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index d6b81a508c7432..1a855332d5a2b7 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; @@ -106,6 +107,15 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // a Hive view carries tableType VIRTUAL_VIEW. private static final String VIRTUAL_VIEW_TABLE_TYPE = "VIRTUAL_VIEW"; + // HMS table-parameter keys for table statistics, replicating legacy StatisticsUtil.getHiveRowCount / + // getRowCountFromParameters / getTotalSizeFromHMS. numRows is the exact row count; totalSize is the on-disk + // data size. Each has a spark-written alternative key (spark writes its own stats keys, not the standard + // hive ones). Read as RAW facts only — the connector must NOT do the Doris-type-dependent estimation. + private static final String PARAM_NUM_ROWS = "numRows"; + private static final String PARAM_SPARK_NUM_ROWS = "spark.sql.statistics.numRows"; + private static final String PARAM_TOTAL_SIZE = "totalSize"; + private static final String PARAM_SPARK_TOTAL_SIZE = "spark.sql.statistics.totalSize"; + private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; @@ -276,6 +286,76 @@ public TTableDescriptor buildTableDescriptor(ConnectorSession session, return desc; } + // ========== ConnectorStatisticsOps ========== + + /** + * Table-level statistics for a hive table, a port of legacy {@code StatisticsUtil.getHiveRowCount} + + * {@code getTotalSizeFromHMS} restricted to the two RAW metastore facts (no Doris-type math — the + * connector must not import fe-type): + *
      + *
    • {@code rowCount} = the exact HMS {@code numRows}, falling back to the spark-written + * {@code spark.sql.statistics.numRows} ONLY when {@code numRows} is present but non-positive + * (legacy {@code getRowCountFromParameters} — a table carrying only the spark key and no plain + * {@code numRows} deliberately does NOT surface a spark count here). A count {@code <= 0} maps to + * -1 (UNKNOWN), matching the legacy "0 -> UNKNOWN" gate and the paimon/iceberg connectors.
      • + *
    • {@code dataSize} = the on-disk {@code totalSize}, falling back to + * {@code spark.sql.statistics.totalSize} when the standard key is ABSENT (legacy size branch — + * note the asymmetry with the row-count fallback).
      • + *
    + * When the exact count is unknown but a size is present, fe-core + * ({@code PluginDrivenExternalTable.fetchRowCount}) estimates the cardinality as + * {@code dataSize / } — the type-dependent division this connector cannot do. Returns + * empty when neither fact is available (fe-core then falls through to the file-list estimate). Params + * are read from the handle (loaded live by {@code getTableHandle}), so this adds no HMS round-trip. + */ + @Override + public Optional getTableStatistics( + ConnectorSession session, ConnectorTableHandle handle) { + Map params = ((HiveTableHandle) handle).getTableParameters(); + if (params == null) { + return Optional.empty(); + } + + long rowCount = -1; + if (params.containsKey(PARAM_NUM_ROWS)) { + rowCount = parseLongOrDefault(params.get(PARAM_NUM_ROWS), -1); + if (rowCount <= 0 && params.containsKey(PARAM_SPARK_NUM_ROWS)) { + rowCount = parseLongOrDefault(params.get(PARAM_SPARK_NUM_ROWS), -1); + } + } + + long dataSize = -1; + if (params.containsKey(PARAM_TOTAL_SIZE)) { + dataSize = parseLongOrDefault(params.get(PARAM_TOTAL_SIZE), -1); + } else if (params.containsKey(PARAM_SPARK_TOTAL_SIZE)) { + dataSize = parseLongOrDefault(params.get(PARAM_SPARK_TOTAL_SIZE), -1); + } + + // Collapse a non-positive count/size to the -1 UNKNOWN sentinel (0 -> UNKNOWN, legacy parity). + long reportedRows = rowCount > 0 ? rowCount : -1; + long reportedSize = dataSize > 0 ? dataSize : -1; + if (reportedRows < 0 && reportedSize < 0) { + return Optional.empty(); + } + return Optional.of(new ConnectorTableStatistics(reportedRows, reportedSize)); + } + + /** + * Parses a metastore numeric parameter defensively. Legacy read these with a bare {@code Long.parseLong} + * under an outer try/catch that logged and returned UNKNOWN; returning {@code defaultValue} on a + * null/blank/malformed value is the same net effect without letting one bad parameter abort the read. + */ + private static long parseLongOrDefault(String value, long defaultValue) { + if (value == null) { + return defaultValue; + } + try { + return Long.parseLong(value.trim()); + } catch (NumberFormatException e) { + return defaultValue; + } + } + // ========== ConnectorPushdownOps: Filter Pushdown ========== @Override diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java new file mode 100644 index 00000000000000..20eb068fcd472c --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java @@ -0,0 +1,173 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorTableStatistics; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HiveConnectorMetadata#getTableStatistics} (§4.2 read-side SPI, layers 1+2). + * + *

    WHY: without this override the connector inherits {@code ConnectorStatisticsOps}'s + * {@code Optional.empty()}, so every flipped hive table reports row count -1 (UNKNOWN) and the Nereids + * cost model collapses cardinality to 1 (join-reorder disabled). The connector surfaces two RAW metastore + * facts and does NO Doris-type math: the exact {@code numRows} row count, and the on-disk {@code totalSize} + * data size (fe-core turns a size-without-count into an estimated row count). These assertions pin the + * legacy {@code StatisticsUtil.getHiveRowCount} / {@code getRowCountFromParameters} / {@code getTotalSizeFromHMS} + * behaviour, including two deliberate asymmetries: the spark {@code numRows} key is consulted ONLY when the + * standard {@code numRows} is present-but-non-positive, whereas the spark {@code totalSize} key is consulted + * when the standard {@code totalSize} is ABSENT.

    + */ +public class HiveConnectorMetadataStatisticsTest { + + // getTableStatistics never touches the HmsClient (it reads the handle's captured parameters), so a null + // client is sufficient and keeps the test focused on the parameter interpretation. + private static Optional statsOf(Map tableParameters) { + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + null, Collections.emptyMap(), new FakeConnectorContext()); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .tableParameters(tableParameters) + .build(); + return metadata.getTableStatistics(null, handle); + } + + private static Map params(String... kv) { + Map m = new HashMap<>(); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + @Test + public void numRowsSurfacedAsExactRowCount() { + // A positive numRows is the exact cardinality; it MUST reach the FE cost model. MUTATION: + // inheriting the default empty -> not present -> red. + Optional stats = statsOf(params("numRows", "1234")); + Assertions.assertTrue(stats.isPresent()); + Assertions.assertEquals(1234L, stats.get().getRowCount()); + } + + @Test + public void zeroNumRowsMapsToUnknown() { + // Legacy gated on rows > 0; a 0 count means UNKNOWN, not a real 0 cardinality (which would corrupt + // cost estimates). With no size either, the whole result is empty. MUTATION: dropping the >0 gate + // (reporting rowCount 0) -> present with rowCount 0 -> red. + Assertions.assertFalse(statsOf(params("numRows", "0")).isPresent(), + "a 0 numRows with no size must map to UNKNOWN (empty)"); + } + + @Test + public void sparkNumRowsUsedOnlyWhenNumRowsPresentButNonPositive() { + // Legacy consults spark.sql.statistics.numRows as a fallback ONLY inside the "numRows key present" + // branch, when numRows <= 0. Here numRows=0 present -> spark 500 is used. + Optional stats = statsOf( + params("numRows", "0", "spark.sql.statistics.numRows", "500")); + Assertions.assertTrue(stats.isPresent()); + Assertions.assertEquals(500L, stats.get().getRowCount()); + } + + @Test + public void sparkNumRowsIgnoredWhenNumRowsKeyAbsent() { + // WHY (legacy quirk, faithfully preserved): getRowCountFromParameters only enters the spark + // fallback if the STANDARD numRows key is present. A table carrying ONLY the spark row-count key + // does not surface it as a row count. MUTATION: checking spark unconditionally would return 500 + // here -> rowCount 500 -> red. + Optional stats = statsOf( + params("spark.sql.statistics.numRows", "500")); + Assertions.assertFalse(stats.isPresent(), + "spark numRows must be ignored when the standard numRows key is absent (legacy parity)"); + } + + @Test + public void totalSizeSurfacedAsDataSizeWithUnknownRowCount() { + // A table with totalSize but no numRows reports rowCount UNKNOWN(-1) + the raw dataSize; fe-core + // performs the totalSize/rowWidth estimation (the connector must not import fe-type). MUTATION: + // estimating in the connector, or dropping dataSize, breaks the fe-core estimate. + Optional stats = statsOf(params("totalSize", "4000000")); + Assertions.assertTrue(stats.isPresent()); + Assertions.assertEquals(-1L, stats.get().getRowCount()); + Assertions.assertEquals(4000000L, stats.get().getDataSize()); + } + + @Test + public void sparkTotalSizeUsedWhenStandardTotalSizeAbsent() { + // The size branch's asymmetry: the spark totalSize IS consulted when the standard totalSize key is + // absent (contrast the numRows fallback). MUTATION: requiring the standard key present -> dataSize + // -1 here -> red. + Optional stats = statsOf( + params("spark.sql.statistics.totalSize", "9999")); + Assertions.assertTrue(stats.isPresent()); + Assertions.assertEquals(9999L, stats.get().getDataSize()); + } + + @Test + public void standardTotalSizePreferredOverSpark() { + // When both size keys exist the standard totalSize wins (legacy: containsKey(TOTAL_SIZE) ? ... : spark). + Optional stats = statsOf( + params("totalSize", "100", "spark.sql.statistics.totalSize", "200")); + Assertions.assertEquals(100L, stats.get().getDataSize()); + } + + @Test + public void exactCountAndSizeBothSurfaced() { + // numRows present AND totalSize present: both facts surface (rowCount wins downstream, dataSize is + // still reported for callers that consume it). Legacy returned only the count, but reporting the + // size too is harmless and lets fe-core keep both. + Optional stats = statsOf( + params("numRows", "10", "totalSize", "4096")); + Assertions.assertEquals(10L, stats.get().getRowCount()); + Assertions.assertEquals(4096L, stats.get().getDataSize()); + } + + @Test + public void bothAbsentReturnsEmpty() { + Assertions.assertFalse(statsOf(params("comment", "hi")).isPresent(), + "a table with neither a row count nor a size must report UNKNOWN (empty)"); + } + + @Test + public void nullParametersReturnsEmpty() { + Assertions.assertFalse(statsOf(null).isPresent()); + } + + @Test + public void malformedNumRowsDoesNotThrowAndFallsBack() { + // A bad numRows must not abort the read. It parses to -1; since the numRows KEY is present, the + // spark fallback then applies. MUTATION: letting Long.parseLong propagate -> exception -> red. + Optional stats = Assertions.assertDoesNotThrow(() -> + statsOf(params("numRows", "not-a-number", "spark.sql.statistics.numRows", "7"))); + Assertions.assertTrue(stats.isPresent()); + Assertions.assertEquals(7L, stats.get().getRowCount()); + } + + @Test + public void malformedTotalSizeDegradesToEmpty() { + // A malformed size parses to -1; with no count either, the result is empty (not an exception). + Optional stats = Assertions.assertDoesNotThrow(() -> + statsOf(params("totalSize", "garbage"))); + Assertions.assertFalse(stats.isPresent()); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index b363bf0c0cc449..cdda549218b57c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -761,12 +761,46 @@ public long fetchRowCount() { } Optional statsOpt = metadata.getTableStatistics(session, handleOpt.get()); - if (statsOpt.isPresent() && statsOpt.get().getRowCount() >= 0) { - return statsOpt.get().getRowCount(); + if (statsOpt.isPresent()) { + ConnectorTableStatistics stats = statsOpt.get(); + if (stats.getRowCount() >= 0) { + return stats.getRowCount(); + } + // The connector surfaced an on-disk data size but no exact row count (e.g. a hive table with + // totalSize set but no numRows). Estimate the cardinality as dataSize / — the Doris-type-dependent division the connector cannot perform (it must not import + // fe-type). Connector-agnostic: every other connector reports dataSize -1, so this branch is + // inert for them. Mirrors legacy StatisticsUtil.getHiveRowCount's totalSize/estimatedRowSize + // path (row width summed over the FULL schema, partition columns included, exactly as legacy). + if (stats.getDataSize() > 0) { + long rowWidth = estimatedRowWidth(); + if (rowWidth > 0) { + return stats.getDataSize() / rowWidth; + } + } } return UNKNOWN_ROW_COUNT; } + /** + * Sum of Doris slot sizes over the full schema — the average uncompressed row width used to turn a + * connector-reported on-disk data size into an estimated row count. Mirrors legacy + * {@code StatisticsUtil.getHiveRowCount}, which summed {@code getDataType().getSlotSize()} over the full + * schema. Returns 0 for an empty/unavailable schema, which {@link #fetchRowCount} treats as "cannot + * estimate" (-> UNKNOWN). + */ + private long estimatedRowWidth() { + List schema = getFullSchema(); + if (schema == null) { + return 0; + } + long rowWidth = 0; + for (Column column : schema) { + rowWidth += column.getDataType().getSlotSize(); + } + return rowWidth; + } + @Override public String getEngine() { // Return the legacy engine name based on the actual catalog type, diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java new file mode 100644 index 00000000000000..b5bf8da3153923 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java @@ -0,0 +1,205 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.PrimitiveType; +import org.apache.doris.catalog.TableIf; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorTableStatistics; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link PluginDrivenExternalTable#fetchRowCount}'s two-layer statistics consumption (§4.2 read-side + * SPI): (1) an exact connector row count is used directly; (2) when the connector reports an on-disk data + * size but no exact count, fe-core estimates the cardinality as {@code dataSize / } — the + * type-dependent division the connector cannot do. The row width is summed over the FULL schema (partition + * columns included), mirroring legacy {@code StatisticsUtil.getHiveRowCount}. This branch is + * connector-agnostic: every non-hive connector reports dataSize -1, leaving it inert. + */ +public class PluginDrivenExternalTableRowCountTest { + + private static Column intCol(String name) { + return new Column(name, PrimitiveType.INT); // slot size 4 + } + + private static Column bigintCol(String name) { + return new Column(name, PrimitiveType.BIGINT); // slot size 8 + } + + @Test + public void exactRowCountUsedDirectlyIgnoringDataSize() { + // A present rowCount >= 0 short-circuits: dataSize is not consulted even when set. MUTATION: + // preferring the dataSize estimate would return 5000/4=1250, not 1234 -> red. + PluginDrivenExternalTable table = tableReturning( + Optional.of(new ConnectorTableStatistics(1234, 5000)), + Collections.singletonList(intCol("v"))); + Assertions.assertEquals(1234L, table.fetchRowCount()); + } + + @Test + public void dataSizeEstimatedOverFullSchemaWidth() { + // rowCount UNKNOWN(-1) + dataSize 4000, schema 10x INT (width 40) -> 4000/40 = 100. MUTATION: + // not estimating (returning UNKNOWN) -> red; wrong width -> wrong number. + List schema = Arrays.asList( + intCol("c0"), intCol("c1"), intCol("c2"), intCol("c3"), intCol("c4"), + intCol("c5"), intCol("c6"), intCol("c7"), intCol("c8"), intCol("c9")); + PluginDrivenExternalTable table = tableReturning( + Optional.of(new ConnectorTableStatistics(-1, 4000)), schema); + Assertions.assertEquals(100L, table.fetchRowCount()); + } + + @Test + public void partitionColumnsCountTowardRowWidth() { + // WHY: legacy getHiveRowCount summed the row width over the FULL schema, INCLUDING partition + // columns. Schema = INT(4) data + BIGINT(8) partition -> width 12; dataSize 1200 -> 100. A + // mutation excluding partition columns would use width 4 -> 300 -> red. + List schema = Arrays.asList(intCol("v"), bigintCol("dt")); + PluginDrivenExternalTable table = tableReturning( + Optional.of(new ConnectorTableStatistics(-1, 1200)), schema); + Assertions.assertEquals(100L, table.fetchRowCount()); + } + + @Test + public void emptyStatisticsYieldUnknown() { + PluginDrivenExternalTable table = tableReturning( + Optional.empty(), Collections.singletonList(intCol("v"))); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + } + + @Test + public void dataSizeWithEmptySchemaYieldsUnknownNotDivideByZero() { + // width 0 -> "cannot estimate" -> UNKNOWN (not an ArithmeticException). MUTATION: dividing by a + // 0 width throws -> red. + PluginDrivenExternalTable table = tableReturning( + Optional.of(new ConnectorTableStatistics(-1, 4000)), Collections.emptyList()); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + } + + @Test + public void unresolvableHandleYieldsUnknown() { + PluginDrivenExternalTable table = tableReturning( + null, Collections.singletonList(intCol("v"))); // null -> getTableHandle returns empty + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + } + + /** + * Builds a table over a mock connector. {@code stats} == null makes {@code getTableHandle} return empty + * (unresolvable handle); otherwise the handle resolves and {@code getTableStatistics} returns {@code stats}. + * The full schema is served from a stubbed schema-cache value. + */ + private static PluginDrivenExternalTable tableReturning( + Optional stats, List schema) { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + if (stats == null) { + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.empty()); + } else { + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.of(handle)); + Mockito.when(metadata.getTableStatistics(session, handle)).thenReturn(stats); + } + TestablePluginCatalog catalog = new TestablePluginCatalog(metadata, session); + + @SuppressWarnings("unchecked") + ExternalDatabase db = Mockito.mock(ExternalDatabase.class); + Mockito.when(db.getRemoteName()).thenReturn("REMOTE_DB"); + + PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( + schema, Collections.emptyList(), Collections.emptyList()); + return new PluginDrivenExternalTable(1L, "tbl", "REMOTE_TBL", catalog, db) { + @Override + protected synchronized void makeSureInitialized() { + // no-op: skip Env-backed catalog/db init + } + + @Override + public Optional getSchemaCacheValue() { + return Optional.of(cacheValue); + } + }; + } + + /** Minimal catalog returning a fixed connector/session without standing up the Doris environment. */ + private static class TestablePluginCatalog extends PluginDrivenExternalCatalog { + private final Connector connector; + private final ConnectorSession session; + + TestablePluginCatalog(ConnectorMetadata metadata, ConnectorSession session) { + this(mockConnector(metadata, session), session); + } + + private TestablePluginCatalog(Connector connector, ConnectorSession session) { + super(1L, "test-catalog", null, props(), "", connector); + this.connector = connector; + this.session = session; + } + + private static Connector mockConnector(ConnectorMetadata metadata, ConnectorSession session) { + Connector c = Mockito.mock(Connector.class); + Mockito.when(c.getMetadata(session)).thenReturn(metadata); + return c; + } + + @Override + public Connector getConnector() { + return connector; + } + + @Override + public ConnectorSession buildConnectorSession() { + return session; + } + + @Override + protected List listDatabaseNames() { + return Collections.emptyList(); + } + + @Override + protected List listTableNamesFromRemote(SessionContext ctx, String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExist(SessionContext ctx, String dbName, String tblName) { + return false; + } + + private static Map props() { + Map props = new HashMap<>(); + props.put("type", "hms"); + return props; + } + } +} From 9abbce0b97e2b90219fc69d174da955dd2046ef5 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 18:16:17 +0800 Subject: [PATCH 043/334] [feat](catalog) hive connector row count layer 3: estimate by listing data files (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the port of legacy hive's three-layer row-count estimate (dormant; hive is not yet in SPI_READY_TYPES). Layer 3 fires only when neither an exact count nor a metastore-recorded size (layers 1/2) exists — a never-analyzed external table. - New SPI ConnectorStatisticsOps.estimateDataSizeByListingFiles (default -1): estimate on-disk data size by listing data files. Only file-based connectors override it. - HiveConnectorMetadata implements it (port of HMSExternalTable.getRowCountFromFileList's file-listing half): plain-hive only (hudi/iceberg-on-HMS -> own connectors, -1); resolve partition locations, sample STATS_PARTITION_SAMPLE_SIZE=30 partitions and scale the summed file bytes back up to the whole table (legacy formula). Best-effort: ANY error -> -1, never throwing. Pins the TCCL to the plugin classloader around the Hadoop FileSystem reflection (the statistics thread is NOT pinned by fe-core, unlike the scan thread). The FileSystem I/O is isolated behind an injected ToLongFunction so the sampling/scale-up/summing math is unit-testable; raw I/O is covered by the docker e2e gate. - PluginDrivenExternalTable.fetchRowCount gains the layer-3 fallback, gated by the global enable_get_row_count_from_file_list, dividing the connector's file-list size by the row width EXCLUDING partition columns (their values live in the path, not the data files) — the deliberate contrast with layer 2's full-schema width. Connector-agnostic (every non-file connector returns -1). Deviations (dormant, documented): (a) runtime tuning of hive_stats_partition_sample_size is not honored on the plugin path (the on/off gate still is) — an internal knob, negligible; (b) precise view-skip lands with the view SPI (a view has no listable data location -> -1 today); (c) the file listing is non-recursive top-level, matching HiveScanPlanProvider. Tests: HiveConnectorMetadataFileListStatsTest (8) pins the sampling/scale-up/summing, the tableType guard, and error/empty -> -1; PluginDrivenExternalTableRowCountTest (+4 = 10) pins the non-partition-width division, the feature gate, and the all-partition-columns -> UNKNOWN guard. fe-connector-api + fe-connector-hive + fe-core: green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../connector/api/ConnectorStatisticsOps.java | 13 ++ .../connector/hive/HiveConnectorMetadata.java | 158 ++++++++++++++ ...iveConnectorMetadataFileListStatsTest.java | 204 ++++++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 37 +++- ...PluginDrivenExternalTableRowCountTest.java | 99 +++++++++ 5 files changed, 504 insertions(+), 7 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java index fb762355e53071..6603c28a2ab2e7 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java @@ -32,4 +32,17 @@ default Optional getTableStatistics( ConnectorTableHandle handle) { return Optional.empty(); } + + /** + * Estimates the table's on-disk data size in bytes by listing its data files, for connectors that can + * cheaply enumerate them (e.g. hive). fe-core uses this to estimate a row count + * ({@code dataSize / }) when neither an exact row count nor a metastore-recorded size (from + * {@link #getTableStatistics}) is available — fe-core only calls it when the + * {@code enable_get_row_count_from_file_list} global is set. A potentially expensive remote listing, so + * connectors that cannot do it cheaply must NOT override it. Returns -1 when the size cannot be + * estimated (not supported, unlistable, empty, or any error) — the default. + */ + default long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTableHandle handle) { + return -1; + } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 1a855332d5a2b7..988377a509d341 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -52,9 +52,14 @@ import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.io.IOException; import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; @@ -64,6 +69,7 @@ import java.util.Map; import java.util.Optional; import java.util.Set; +import java.util.function.ToLongFunction; import java.util.stream.Collectors; /** @@ -116,6 +122,15 @@ public class HiveConnectorMetadata implements ConnectorMetadata { private static final String PARAM_TOTAL_SIZE = "totalSize"; private static final String PARAM_SPARK_TOTAL_SIZE = "spark.sql.statistics.totalSize"; + // Partition-sampling cap for the file-list data-size estimate, matching the default of the legacy + // Config.hive_stats_partition_sample_size (fe-common, unreadable from the plugin). Runtime tuning of that + // specific config no longer applies on the plugin path (negligible — it is an internal estimation knob); + // the on/off feature gate (enable_get_row_count_from_file_list) is still honored, fe-core side. + private static final int STATS_PARTITION_SAMPLE_SIZE = 30; + + // Upper bound on partitions listed from HMS for the file-list estimate, matching HiveScanPlanProvider. + private static final int MAX_PARTITIONS_FOR_STATS = 100000; + private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; @@ -356,6 +371,149 @@ private static long parseLongOrDefault(String value, long defaultValue) { } } + /** + * Estimates the table's on-disk data size (bytes) by listing its data files, a port of the file-listing + * half of legacy {@code HMSExternalTable.getRowCountFromFileList} (fe-core does the + * {@code size / rowWidth} division). Only plain-hive tables are estimated (hudi/iceberg-on-HMS are served + * by their own connectors; a view has no data files) — anything else returns -1. Partitions are sampled + * ({@link #STATS_PARTITION_SAMPLE_SIZE}) and the sampled size scaled back up to the whole table, exactly + * as legacy did. Best-effort: ANY error (unlistable location, remote failure) degrades to -1, never + * throwing — statistics must not fail a query. The Hadoop {@code FileSystem} reflection resolves its + * filesystem impl through the thread context classloader, so this pins the TCCL to the plugin classloader + * for the duration (the statistics thread is not pinned by fe-core, unlike the scan thread). + */ + @Override + public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTableHandle handle) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + if (hiveHandle.getTableType() != HiveTableType.HIVE) { + return -1; + } + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(getClass().getClassLoader()); + Configuration conf = buildHadoopConf(); + return estimateDataSize(hiveHandle, STATS_PARTITION_SAMPLE_SIZE, + location -> sumFileSizesUnder(location, conf)); + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } + + /** + * Sampling + summing + scale-up core of {@link #estimateDataSizeByListingFiles}, isolated from the + * {@code FileSystem} I/O (injected as {@code sizeOf}) so the estimation math is unit-testable. Returns -1 + * when the size cannot be estimated (no listable location, a zero/negative sum, or any error). + */ + long estimateDataSize(HiveTableHandle handle, int sampleSize, ToLongFunction sizeOf) { + try { + List locations = resolvePartitionLocations(handle); + if (locations.isEmpty()) { + return -1; + } + int totalPartitions = locations.size(); + boolean sampled = sampleSize > 0 && sampleSize < totalPartitions; + List chosen = locations; + if (sampled) { + List shuffled = new ArrayList<>(locations); + Collections.shuffle(shuffled); + chosen = shuffled.subList(0, sampleSize); + } + long totalSize = 0; + for (String location : chosen) { + totalSize += Math.max(0, sizeOf.applyAsLong(location)); + } + if (totalSize <= 0) { + return -1; + } + // Scale the sampled size up to the whole table (legacy: totalSize * total / sampled). + if (sampled) { + totalSize = totalSize * totalPartitions / chosen.size(); + } + return totalSize; + } catch (RuntimeException e) { + LOG.warn("Failed to estimate hive data size for {}.{} from file list", + handle.getDbName(), handle.getTableName(), e); + return -1; + } + } + + /** + * Resolves the data locations to list: the table location for an unpartitioned table, else every + * partition's location (bounded by {@link #MAX_PARTITIONS_FOR_STATS}). A partition or table with no + * location contributes nothing. + */ + private List resolvePartitionLocations(HiveTableHandle handle) { + List partKeyNames = handle.getPartitionKeyNames(); + if (partKeyNames == null || partKeyNames.isEmpty()) { + String location = handle.getLocation(); + return (location == null || location.isEmpty()) + ? Collections.emptyList() : Collections.singletonList(location); + } + List partNames = hmsClient.listPartitionNames( + handle.getDbName(), handle.getTableName(), MAX_PARTITIONS_FOR_STATS); + if (partNames.isEmpty()) { + return Collections.emptyList(); + } + List partitions = hmsClient.getPartitions( + handle.getDbName(), handle.getTableName(), partNames); + List locations = new ArrayList<>(partitions.size()); + for (HmsPartitionInfo partition : partitions) { + String location = partition.getLocation(); + if (location != null && !location.isEmpty()) { + locations.add(location); + } + } + return locations; + } + + /** + * Sums the sizes of the data files directly under {@code location} (non-recursive, skipping directories + * and {@code _}/{@code .}-prefixed hidden files — the same filter as {@link HiveScanPlanProvider}). A + * listing failure is surfaced as a {@link RuntimeException} so {@link #estimateDataSize} degrades the + * whole estimate to -1 (legacy's file-list estimate was all-or-nothing best-effort). + */ + private static long sumFileSizesUnder(String location, Configuration conf) { + try { + Path path = new Path(location); + FileSystem fs = FileSystem.get(path.toUri(), conf); + FileStatus[] statuses = fs.listStatus(path); + long sum = 0; + for (FileStatus status : statuses) { + if (status.isDirectory()) { + continue; + } + String fileName = status.getPath().getName(); + if (fileName.startsWith("_") || fileName.startsWith(".")) { + continue; + } + sum += status.getLen(); + } + return sum; + } catch (IOException e) { + throw new DorisConnectorException("Failed to list files under " + location, e); + } + } + + /** + * Builds a Hadoop {@link Configuration} from the catalog properties, mirroring + * {@code HiveScanPlanProvider.buildHadoopConf} (the connector must supply the storage credentials for the + * FileSystem it lists). + */ + private Configuration buildHadoopConf() { + Configuration conf = new Configuration(); + for (Map.Entry entry : properties.entrySet()) { + conf.set(entry.getKey(), entry.getValue()); + } + String defaultFs = properties.get("fs.defaultFS"); + if (defaultFs == null) { + defaultFs = properties.get("hadoop.fs.defaultFS"); + } + if (defaultFs != null) { + conf.set("fs.defaultFS", defaultFs); + } + return conf; + } + // ========== ConnectorPushdownOps: Filter Pushdown ========== @Override diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java new file mode 100644 index 00000000000000..3eae76d0cb8a82 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java @@ -0,0 +1,204 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.function.ToLongFunction; + +/** + * Tests {@link HiveConnectorMetadata#estimateDataSizeByListingFiles} (§4.2 read-side SPI, layer 3 — the + * file-list data-size estimate that fe-core divides by the row width when no exact count or metastore size + * exists). The real {@code FileSystem} listing is injected as a {@code ToLongFunction} so the + * sampling / scale-up / summing math (the tricky part, ported from legacy + * {@code HMSExternalTable.getRowCountFromFileList}) is unit-tested; the raw filesystem I/O is covered by the + * docker e2e gate. + */ +public class HiveConnectorMetadataFileListStatsTest { + + private static HiveConnectorMetadata metadata(HmsClient client) { + return new HiveConnectorMetadata(client, Collections.emptyMap(), new FakeConnectorContext()); + } + + private static HiveTableHandle unpartitioned(String location) { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .location(location) + .build(); + } + + private static HiveTableHandle partitioned() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.singletonList("dt")) + .build(); + } + + @Test + public void unpartitionedTableSumsTableLocation() { + long size = metadata(new PartitionFakeHmsClient(Collections.emptyList())) + .estimateDataSize(unpartitioned("s3://wh/t"), 30, loc -> "s3://wh/t".equals(loc) ? 1000 : 0); + Assertions.assertEquals(1000L, size); + } + + @Test + public void unpartitionedTableWithNoLocationReturnsMinusOne() { + long size = metadata(new PartitionFakeHmsClient(Collections.emptyList())) + .estimateDataSize(unpartitioned(null), 30, loc -> 999); + Assertions.assertEquals(-1L, size); + } + + @Test + public void allPartitionsSummedWhenBelowSampleCap() { + // 3 partitions (< sample cap): the whole table is listed, no scale-up. 100+200+300 = 600. + PartitionFakeHmsClient client = new PartitionFakeHmsClient(Arrays.asList("p0", "p1", "p2")); + ToLongFunction sizes = loc -> loc.endsWith("p0") ? 100 : loc.endsWith("p1") ? 200 : 300; + Assertions.assertEquals(600L, metadata(client).estimateDataSize(partitioned(), 30, sizes)); + } + + @Test + public void sampledPartitionsAreScaledUpToTheWholeTable() { + // 4 partitions, sampleSize 2 -> list 2, scale up by total/sampled = 4/2. With a UNIFORM per-partition + // size the result is deterministic regardless of which 2 are shuffled in: 2*100 * (4/2) = 400. This + // pins the legacy scale-up (totalSize * totalPartitions / samplePartitions). MUTATION: dropping the + // scale-up returns 200 -> red. + PartitionFakeHmsClient client = new PartitionFakeHmsClient(Arrays.asList("p0", "p1", "p2", "p3")); + Assertions.assertEquals(400L, metadata(client).estimateDataSize(partitioned(), 2, loc -> 100)); + } + + @Test + public void zeroTotalSizeReturnsMinusOne() { + PartitionFakeHmsClient client = new PartitionFakeHmsClient(Arrays.asList("p0", "p1")); + Assertions.assertEquals(-1L, metadata(client).estimateDataSize(partitioned(), 30, loc -> 0)); + } + + @Test + public void listingErrorDegradesToMinusOneNotThrow() { + // A per-location listing failure aborts the estimate to -1 (legacy all-or-nothing best-effort), and + // must NOT propagate as a query-killing exception. MUTATION: not catching -> the estimate throws -> red. + PartitionFakeHmsClient client = new PartitionFakeHmsClient(Arrays.asList("p0")); + long size = Assertions.assertDoesNotThrow(() -> metadata(client).estimateDataSize( + partitioned(), 30, loc -> { + throw new RuntimeException("boom"); + })); + Assertions.assertEquals(-1L, size); + } + + @Test + public void partitionWithoutLocationIsSkipped() { + // A partition carrying no location contributes nothing but must not break the estimate. + PartitionFakeHmsClient client = new PartitionFakeHmsClient(Arrays.asList("p0", "p1")); + client.dropLocationFor("p1"); + Assertions.assertEquals(100L, metadata(client).estimateDataSize( + partitioned(), 30, loc -> loc.endsWith("p0") ? 100 : 0)); + } + + @Test + public void nonHiveTableTypeIsNotEstimated() { + // A hudi/iceberg-on-HMS table is served by its own connector; the hive gateway must return -1 BEFORE + // any filesystem I/O (the public entry point's tableType guard). MUTATION: dropping the guard would + // list files for a foreign format. + HiveTableHandle hudiHandle = new HiveTableHandle.Builder("db", "t", HiveTableType.HUDI) + .location("s3://wh/t").build(); + Assertions.assertEquals(-1L, + metadata(new PartitionFakeHmsClient(Collections.emptyList())) + .estimateDataSizeByListingFiles(null, hudiHandle)); + } + + /** + * {@link HmsClient} double serving a fixed set of partition names, each with a synthetic location + * {@code s3://wh/t/}. {@code listPartitionNames} echoes the names; {@code getPartitions} builds an + * {@link HmsPartitionInfo} per requested name. The rest fail loud. + */ + private static final class PartitionFakeHmsClient implements HmsClient { + private final List partitionNames; + private final java.util.Set withoutLocation = new java.util.HashSet<>(); + + PartitionFakeHmsClient(List partitionNames) { + this.partitionNames = partitionNames; + } + + void dropLocationFor(String name) { + withoutLocation.add(name); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return partitionNames; + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + List result = new ArrayList<>(partNames.size()); + for (String name : partNames) { + String location = withoutLocation.contains(name) ? null : "s3://wh/t/" + name; + result.add(new HmsPartitionInfo( + Collections.singletonList(name), location, null, null, null, Collections.emptyMap())); + } + return result; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index cdda549218b57c..79cda0d05c0a15 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -39,6 +39,7 @@ import org.apache.doris.datasource.systable.PluginDrivenSysTable; import org.apache.doris.datasource.systable.SysTable; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.GlobalVariable; import org.apache.doris.statistics.AnalysisInfo; import org.apache.doris.statistics.BaseAnalysisTask; import org.apache.doris.statistics.ExternalAnalysisTask; @@ -773,29 +774,51 @@ public long fetchRowCount() { // inert for them. Mirrors legacy StatisticsUtil.getHiveRowCount's totalSize/estimatedRowSize // path (row width summed over the FULL schema, partition columns included, exactly as legacy). if (stats.getDataSize() > 0) { - long rowWidth = estimatedRowWidth(); + long rowWidth = estimatedRowWidth(false); if (rowWidth > 0) { return stats.getDataSize() / rowWidth; } } } + + // Neither an exact count nor a metastore-recorded size: estimate the on-disk data size by listing + // the table's data files (connector-provided; every non-file connector returns -1) and divide by the + // row width, this time EXCLUDING partition columns because their values live in the directory path, + // not the data files. Mirrors legacy HMSExternalTable.getRowCountFromFileList. Gated by the global + // feature flag because the listing can be a costly remote round-trip; the connector self-samples, + // pins its classloader, and degrades to -1 rather than throwing. + if (GlobalVariable.enable_get_row_count_from_file_list) { + long dataSize = metadata.estimateDataSizeByListingFiles(session, handleOpt.get()); + if (dataSize > 0) { + long rowWidth = estimatedRowWidth(true); + if (rowWidth > 0) { + return dataSize / rowWidth; + } + } + } return UNKNOWN_ROW_COUNT; } /** - * Sum of Doris slot sizes over the full schema — the average uncompressed row width used to turn a - * connector-reported on-disk data size into an estimated row count. Mirrors legacy - * {@code StatisticsUtil.getHiveRowCount}, which summed {@code getDataType().getSlotSize()} over the full - * schema. Returns 0 for an empty/unavailable schema, which {@link #fetchRowCount} treats as "cannot - * estimate" (-> UNKNOWN). + * Sum of Doris slot sizes over the full schema (or over the non-partition columns when + * {@code excludePartitionColumns}) — the average uncompressed row width used to turn a connector-reported + * on-disk data size into an estimated row count. Mirrors the two legacy hive formulas: a metastore + * {@code totalSize} is divided by the FULL-schema width ({@code StatisticsUtil.getHiveRowCount}), whereas + * a file-listed size is divided by the width EXCLUDING partition columns (whose values are not stored in + * the data files, {@code HMSExternalTable.getRowCountFromFileList}). Returns 0 for an empty/unavailable + * schema, which {@link #fetchRowCount} treats as "cannot estimate" (-> UNKNOWN). */ - private long estimatedRowWidth() { + private long estimatedRowWidth(boolean excludePartitionColumns) { List schema = getFullSchema(); if (schema == null) { return 0; } + List partitionColumns = excludePartitionColumns ? getPartitionColumns() : null; long rowWidth = 0; for (Column column : schema) { + if (partitionColumns != null && partitionColumns.contains(column)) { + continue; + } rowWidth += column.getDataType().getSlotSize(); } return rowWidth; diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java index b5bf8da3153923..9d719c29067d89 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.qe.GlobalVariable; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -111,6 +112,104 @@ public void unresolvableHandleYieldsUnknown() { Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); } + // ==================== layer 3: file-list data-size estimate ==================== + + @Test + public void fileListEstimateDividesByNonPartitionWidth() { + // No exact count, no metastore size -> the connector's file-list dataSize (400) is divided by the + // NON-partition row width. Schema = INT(4) data "v" + BIGINT(8) partition "dt"; the BIGINT is + // EXCLUDED (its values live in the path, not the data files) -> width 4 -> 400/4 = 100. This is the + // deliberate contrast with layer 2 (which includes partition columns): a mutation using the full + // width (12) would return 33 -> red. + withFileListGate(true, () -> { + PluginDrivenExternalTable table = tableForFileList(400, + Arrays.asList(intCol("v"), bigintCol("dt")), Collections.singletonList(1)); + Assertions.assertEquals(100L, table.fetchRowCount()); + }); + } + + @Test + public void fileListEstimateSkippedWhenGateDisabled() { + // The global feature flag gates the (potentially costly) file listing. Off -> UNKNOWN even though the + // connector would return a size. MUTATION: ignoring the gate returns 100 here -> red. + withFileListGate(false, () -> { + PluginDrivenExternalTable table = tableForFileList(400, + Arrays.asList(intCol("v"), bigintCol("dt")), Collections.singletonList(1)); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + }); + } + + @Test + public void fileListEstimateUnknownWhenConnectorReturnsMinusOne() { + withFileListGate(true, () -> { + PluginDrivenExternalTable table = tableForFileList(-1, + Collections.singletonList(intCol("v")), Collections.emptyList()); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + }); + } + + @Test + public void fileListEstimateUnknownWhenAllColumnsArePartitions() { + // Every column is a partition column -> non-partition width 0 -> "cannot estimate" -> UNKNOWN (no + // divide-by-zero). MUTATION: dividing by a 0 width throws -> red. + withFileListGate(true, () -> { + PluginDrivenExternalTable table = tableForFileList(400, + Collections.singletonList(bigintCol("dt")), Collections.singletonList(0)); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + }); + } + + private static void withFileListGate(boolean enabled, Runnable body) { + boolean previous = GlobalVariable.enable_get_row_count_from_file_list; + GlobalVariable.enable_get_row_count_from_file_list = enabled; + try { + body.run(); + } finally { + GlobalVariable.enable_get_row_count_from_file_list = previous; + } + } + + /** + * Table whose connector reports no getTableStatistics (empty) but a file-list size of {@code estimateBytes}; + * {@code partitionColumnIndexes} names which schema columns are partition columns (excluded from the + * layer-3 row width). + */ + private static PluginDrivenExternalTable tableForFileList( + long estimateBytes, List schema, List partitionColumnIndexes) { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.of(handle)); + Mockito.when(metadata.getTableStatistics(session, handle)).thenReturn(Optional.empty()); + Mockito.when(metadata.estimateDataSizeByListingFiles(session, handle)).thenReturn(estimateBytes); + TestablePluginCatalog catalog = new TestablePluginCatalog(metadata, session); + + @SuppressWarnings("unchecked") + ExternalDatabase db = Mockito.mock(ExternalDatabase.class); + Mockito.when(db.getRemoteName()).thenReturn("REMOTE_DB"); + + List partitionColumns = new java.util.ArrayList<>(); + List partitionRemoteNames = new java.util.ArrayList<>(); + for (int idx : partitionColumnIndexes) { + partitionColumns.add(schema.get(idx)); + partitionRemoteNames.add(schema.get(idx).getName()); + } + PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( + schema, partitionColumns, partitionRemoteNames); + return new PluginDrivenExternalTable(1L, "tbl", "REMOTE_TBL", catalog, db) { + @Override + protected synchronized void makeSureInitialized() { + // no-op: skip Env-backed catalog/db init + } + + @Override + public Optional getSchemaCacheValue() { + return Optional.of(cacheValue); + } + }; + } + /** * Builds a table over a mock connector. {@code stats} == null makes {@code getTableHandle} return empty * (unresolvable handle); otherwise the handle resolves and {@code getTableStatistics} returns {@code stats}. From f7860dd7ec669c8dd02d5ddc61ad9d41915a24ea Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 18:37:43 +0800 Subject: [PATCH 044/334] [fix](catalog) hive row-count: collapse truncated-to-0 estimates to UNKNOWN (adversarial review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fixes from the clean-room adversarial review of the two row-count commits. - CORRECTNESS (two reviewers, confirmed): both fetchRowCount estimation branches returned the raw integer-division quotient. A small data size over a wide Doris schema (e.g. totalSize=10 over a 12-byte row) truncates to 0, which was reported as a real "empty table" (0 rows) instead of UNKNOWN. Legacy collapses a 0 result to UNKNOWN (`rows > 0 ? rows : UNKNOWN`), and for layer 2 additionally FALLS THROUGH to the file-list estimate (layer 3). Now the layer-2 quotient only returns when > 0 (else it falls through to layer 3), and the layer-3 quotient only returns when > 0 (else UNKNOWN). A 0-row cardinality vs UNKNOWN drives very different optimizer plans. - Extracted HiveConnectorMetadata.scaleSampledSize (multiply-before-divide, documented) so the sampled scale-up formula is pinned by a non-divisible unit test (250*4/3=333, distinguishing it from a divide-first 332) — the prior test could not tell the two apart. Tests added: layer-2/layer-3 truncate-to-0 -> fall-through / UNKNOWN (fe-core); public estimateDataSizeByListingFiles drives real FileSystem I/O on a bogus location and asserts -1 + TCCL restore on the error path; scaleSampledSize multiply-first; a present-but-malformed standard totalSize short-circuits the spark size key. Documented one deliberate deviation: a malformed numRows recovers the spark count defensively instead of legacy's throw-and-fall-to-file-list (strictly more robust on corrupt metadata, never throws). fe-connector-hive (23) + fe-core (13): green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- .../connector/hive/HiveConnectorMetadata.java | 13 ++++- ...iveConnectorMetadataFileListStatsTest.java | 35 +++++++++++++ .../HiveConnectorMetadataStatisticsTest.java | 21 ++++++-- .../datasource/PluginDrivenExternalTable.java | 13 ++++- ...PluginDrivenExternalTableRowCountTest.java | 52 +++++++++++++++++-- 5 files changed, 123 insertions(+), 11 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 988377a509d341..0e994ae88e7387 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -427,7 +427,7 @@ long estimateDataSize(HiveTableHandle handle, int sampleSize, ToLongFunction loc.endsWith("p0") ? 100 : 0)); } + @Test + public void scaleSampledSizeMultipliesBeforeDividing() { + // Non-divisible case pins multiply-first (250*4/3 = 333), distinguishing it from a divide-first + // reordering (250/3*4 = 83*4 = 332) that a mutation might introduce. MUTATION: divide-first -> 332 -> red. + Assertions.assertEquals(333L, HiveConnectorMetadata.scaleSampledSize(250, 4, 3)); + } + + @Test + public void publicEntryDegradesToMinusOneAndRestoresClassLoaderOnError() { + // Drives the PUBLIC entry point: its tableType guard passes for HIVE, then it pins the thread context + // classloader and does real FileSystem I/O. A bogus location makes the listing fail, so the estimate + // must degrade to -1 WITHOUT throwing, and the TCCL must be restored to whatever it was (the pin is in + // a try/finally). MUTATION: dropping the finally leaves the plugin loader set -> the marker assertion + // fails; letting the listing error propagate -> assertDoesNotThrow fails. + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .location("file:///__doris_stats_nonexistent_xyz_998877").build(); + HiveConnectorMetadata metadata = metadata(new PartitionFakeHmsClient(Collections.emptyList())); + + ClassLoader marker = new URLClassLoader(new URL[0], + HiveConnectorMetadataFileListStatsTest.class.getClassLoader()); + ClassLoader original = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(marker); + try { + long size = Assertions.assertDoesNotThrow( + () -> metadata.estimateDataSizeByListingFiles(null, handle)); + Assertions.assertEquals(-1L, size, "an unlistable location must degrade to -1"); + Assertions.assertSame(marker, Thread.currentThread().getContextClassLoader(), + "the TCCL pin must be restored on the error path (try/finally)"); + } finally { + Thread.currentThread().setContextClassLoader(original); + } + } + @Test public void nonHiveTableTypeIsNotEstimated() { // A hudi/iceberg-on-HMS table is served by its own connector; the hive gateway must return -1 BEFORE diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java index 20eb068fcd472c..0d9b105662373d 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataStatisticsTest.java @@ -154,9 +154,12 @@ public void nullParametersReturnsEmpty() { } @Test - public void malformedNumRowsDoesNotThrowAndFallsBack() { - // A bad numRows must not abort the read. It parses to -1; since the numRows KEY is present, the - // spark fallback then applies. MUTATION: letting Long.parseLong propagate -> exception -> red. + public void malformedNumRowsDoesNotThrowAndRecoversSparkCount() { + // DELIBERATE DEVIATION from legacy (documented): legacy's bare Long.parseLong on a malformed numRows + // threw, aborting the whole metastore-stat path so the query fell through to the file-list estimate. + // The connector instead parses defensively (-1) and, because the numRows KEY is present, recovers the + // valid spark count — strictly more useful on corrupt metadata, and it must never throw. MUTATION: + // letting the parse propagate -> exception -> red. Optional stats = Assertions.assertDoesNotThrow(() -> statsOf(params("numRows", "not-a-number", "spark.sql.statistics.numRows", "7"))); Assertions.assertTrue(stats.isPresent()); @@ -170,4 +173,16 @@ public void malformedTotalSizeDegradesToEmpty() { statsOf(params("totalSize", "garbage"))); Assertions.assertFalse(stats.isPresent()); } + + @Test + public void malformedStandardTotalSizeShortCircuitsSparkKey() { + // The size branch is an if/else-if on the STANDARD key: a present-but-malformed totalSize parses to -1 + // and must NOT fall through to the spark key (the standard key's presence wins). With no row count the + // result is empty. MUTATION: restructuring the else-if so a malformed standard key falls to spark would + // surface 9999 -> present -> red. + Optional stats = statsOf( + params("totalSize", "garbage", "spark.sql.statistics.totalSize", "9999")); + Assertions.assertFalse(stats.isPresent(), + "a present (even malformed) standard totalSize must short-circuit the spark size key"); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 79cda0d05c0a15..1800ac3f1dea30 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -773,10 +773,15 @@ public long fetchRowCount() { // fe-type). Connector-agnostic: every other connector reports dataSize -1, so this branch is // inert for them. Mirrors legacy StatisticsUtil.getHiveRowCount's totalSize/estimatedRowSize // path (row width summed over the FULL schema, partition columns included, exactly as legacy). + // A quotient that truncates to 0 is NOT a valid "empty table" answer — legacy collapsed 0 -> + // UNKNOWN and fell through to the file-list estimate below, so only a positive quotient returns. if (stats.getDataSize() > 0) { long rowWidth = estimatedRowWidth(false); if (rowWidth > 0) { - return stats.getDataSize() / rowWidth; + long rows = stats.getDataSize() / rowWidth; + if (rows > 0) { + return rows; + } } } } @@ -792,7 +797,11 @@ public long fetchRowCount() { if (dataSize > 0) { long rowWidth = estimatedRowWidth(true); if (rowWidth > 0) { - return dataSize / rowWidth; + // 0 -> UNKNOWN (legacy getRowCountFromFileList's `rows > 0 ? rows : UNKNOWN` gate). + long rows = dataSize / rowWidth; + if (rows > 0) { + return rows; + } } } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java index 9d719c29067d89..c627fea99f6248 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableRowCountTest.java @@ -159,6 +159,42 @@ public void fileListEstimateUnknownWhenAllColumnsArePartitions() { }); } + @Test + public void layer2ZeroQuotientFallsThroughToFileList() { + // totalSize 10 over a 3x INT schema (full width 12) truncates to 0 at layer 2. Legacy collapsed that + // 0 to UNKNOWN and fell through to the file-list estimate, so the connector's 120-byte file-list size + // / width 12 = 10 must win. MUTATION: returning the layer-2 quotient 0 (a bogus "empty table") or + // failing to fall through -> 0 -> red. + withFileListGate(true, () -> { + PluginDrivenExternalTable table = tableWith( + Optional.of(new ConnectorTableStatistics(-1, 10)), 120, + Arrays.asList(intCol("a"), intCol("b"), intCol("c")), Collections.emptyList()); + Assertions.assertEquals(10L, table.fetchRowCount()); + }); + } + + @Test + public void layer2ZeroQuotientYieldsUnknownWhenFileListDisabled() { + // Same truncate-to-0 layer-2 case, gate off: the answer is UNKNOWN, never the bogus 0. + withFileListGate(false, () -> { + PluginDrivenExternalTable table = tableWith( + Optional.of(new ConnectorTableStatistics(-1, 10)), 120, + Arrays.asList(intCol("a"), intCol("b"), intCol("c")), Collections.emptyList()); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + }); + } + + @Test + public void layer3ZeroQuotientYieldsUnknownNotEmptyTable() { + // A file-list size (10) smaller than the row width (12) truncates to 0; legacy getRowCountFromFileList + // returned UNKNOWN for a 0 result, so this must be UNKNOWN, not 0. MUTATION: returning 0 -> red. + withFileListGate(true, () -> { + PluginDrivenExternalTable table = tableWith(Optional.empty(), 10, + Arrays.asList(intCol("a"), intCol("b"), intCol("c")), Collections.emptyList()); + Assertions.assertEquals(TableIf.UNKNOWN_ROW_COUNT, table.fetchRowCount()); + }); + } + private static void withFileListGate(boolean enabled, Runnable body) { boolean previous = GlobalVariable.enable_get_row_count_from_file_list; GlobalVariable.enable_get_row_count_from_file_list = enabled; @@ -169,19 +205,25 @@ private static void withFileListGate(boolean enabled, Runnable body) { } } + /** Table whose connector reports no getTableStatistics (empty) but a file-list size of {@code estimateBytes}. */ + private static PluginDrivenExternalTable tableForFileList( + long estimateBytes, List schema, List partitionColumnIndexes) { + return tableWith(Optional.empty(), estimateBytes, schema, partitionColumnIndexes); + } + /** - * Table whose connector reports no getTableStatistics (empty) but a file-list size of {@code estimateBytes}; - * {@code partitionColumnIndexes} names which schema columns are partition columns (excluded from the - * layer-3 row width). + * Table whose connector returns {@code stats} from getTableStatistics AND {@code estimateBytes} from the + * file-list estimate; {@code partitionColumnIndexes} names which schema columns are partition columns + * (excluded from the layer-3 row width). */ - private static PluginDrivenExternalTable tableForFileList( + private static PluginDrivenExternalTable tableWith(Optional stats, long estimateBytes, List schema, List partitionColumnIndexes) { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); ConnectorSession session = Mockito.mock(ConnectorSession.class); ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) .thenReturn(Optional.of(handle)); - Mockito.when(metadata.getTableStatistics(session, handle)).thenReturn(Optional.empty()); + Mockito.when(metadata.getTableStatistics(session, handle)).thenReturn(stats); Mockito.when(metadata.estimateDataSizeByListingFiles(session, handle)).thenReturn(estimateBytes); TestablePluginCatalog catalog = new TestablePluginCatalog(metadata, session); From e9c1e4994252d850a81cfac355c48538872097c0 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 18:39:10 +0800 Subject: [PATCH 045/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20getTableStatistics=20three-layer=20row=20count=20DONE;=20?= =?UTF-8?q?next=20=3D=20views?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the three-layer table row-count port (exact numRows / totalSize estimate / file-list estimate) as done across eded05b05e6 + 8c6d247bb34 + 453a1926010, the connector-vs-fe-core split (connector surfaces raw facts, fe-core owns the Doris-type-dependent division), the signed-off decisions (sample size connector-owned, malformed-numRows defensive recovery, non-recursive listing, view-skip deferred to the view SPI), and the adversarial review's truncated-to-0 -> UNKNOWN fix. Advances the read-side roadmap: next sub-step = views. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01SLdLbpKSfgh6AvzaKBFfRT --- plan-doc/HANDOFF.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d249ca7a99f881..56ac31e4bac799 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -40,13 +40,13 @@ - **✅ §4.2 表描述符 DONE（`164b25d9c42`）**：`buildTableDescriptor` 端口旧 `toThrift`→`TTableType.HIVE_TABLE`+`THiveTable`（否则 fe-core `PluginDrivenExternalTable.toThrift` 退化到通用 `SCHEMA_TABLE` 描述符→BE 建 SchemaTableDescriptor 而非 HiveTableDescriptor）；照 iceberg HIVE_TABLE 分支，无 handle 故 base+sys 表共用。全绿（schema 套 7 测 + checkstyle 0）。 - **✅ §4.2 按表能力接缝 DONE（`329853bf19c`）**：**设计决策已签字**——两个"按文件格式启用"的加速能力（Top-N 延迟 / 嵌套列裁剪）走**按表接缝**忠实复刻（不 blanket-declare、不放弃；对齐 Trino 逐表委派）。机制 = 连接器在 `getTableSchema` 发射 FE 内部 schema-property marker `ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY`（CSV of `ConnectorCapability.name()`；缓存友好、无额外 HMS 往返、format 留连接器侧，复用 `partition_columns` 模式），fe-core `PluginDrivenExternalTable.supportsTopNLazyMaterialize/supportsNestedColumnPrune` 经新 `hasScanCapability` = 连接器级 Set **OR** 按表 marker；marker key 从 SHOW CREATE 属性剔除。**Top-N 已发射**（orc/parquet 非 view 非 iceberg/hudi-on-HMS，精确 input-format 类匹配 = 旧 `SUPPORTED_HIVE_TOPN_LAZY_FILE_FORMATS`）。iceberg/paimon/jdbc 零改（永不发 marker）。全绿（连接器 178 测 + fe-core 32 测 + checkstyle 0 + import gate）。 - ⛔**NESTED_PRUNE 遗留（硬门，下一子步前置验证）**：同接缝，连接器**暂未发射** marker——`SlotTypeReplacer.java:678-680` 把嵌套访问路径重写成 `Column.getUniqueId()`，iceberg 靠自带 field-id（`withUniqueId`）正确，**hive 连接器现在没设 field-id**；对 hive 开启前须端到端验证 hive orc/parquet 的 uniqueId 语义（BE 按 id 匹配嵌套叶子）否则读 NULL。旧 hive nested-prune 走同一 `SlotTypeReplacer` 且能工作→查旧 hive column uniqueId 赋值 vs SPI 路径 `ConnectorColumnConverter.convertColumns`；验证通过则连接器发 `SUPPORTS_NESTED_COLUMN_PRUNE` marker（parquet/orc、非 hudi），否则记差异 fail-loud。 -- **下一步 = §4.2 其余读侧 SPI**（本 session 侦察 `wf_f904ad07-2d4`，5 维全 HEAD 核对；critic 因连接错误未出、5 维已够；`HiveConnectorMetadata` 仍 read-mostly 骨架）。按建议序，每子步独立休眠 commit： - 1. **`getTableStatistics`（rowCount，S）**：镜像 paimon/iceberg 结构。level-1 = HMS 参数（`numRows`→`spark.sql.statistics.numRows`→`totalSize/estimatedRowSize`）；rowCount>0 才 `Optional.of` 否则 empty（=旧 0→UNKNOWN）。⚠`estimatedRowSize` 需 Doris slot size（连接器不能 import fe-type）→ `totalSize/rowSize` 分支要么 fe-core 侧算好 threading 进、要么先推迟；file-list 兜底 `getRowCountFromFileList`（`HMSExternalTable:803-822/1072-1123`）更重，建议推迟并 fail-loud 记差异。 - 2. **views（`viewExists`/`listViewNames`/`getViewDefinition` + 声明 `SUPPORTS_VIEW`，M）**：照 iceberg 模板；`getViewDefinition` 内复刻 Presto/Trino base64 view-text 解码（`HMSExternalTable:598-662` `parseTrinoViewDefinition`：`/* Presto View: */`→Gson→originalSql；dialect 无 caller 读，占位即可）。⚠两处设计决策：(a) `$partitions` 旧是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`，`useNativeTablePath()==false`），SPI sys-table 面（`PluginDrivenSysTable extends NativeSysTable`）**只支持 native**——`$partitions` 留 fe-core vs 建 native；(b) `BindRelation.java:623-653` plugin-view 臂硬编码 iceberg（`enable_query_iceberg_views` + iceberg 文案 + 拒时间旅行）→hive view 需泛化 config/文案 或留 HMS 臂。 - 3. **file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 - 4. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 - 5. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 - 6. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 +- **✅ §4.2 `getTableStatistics` 三层全做 DONE**（`eded05b05e6`+`8c6d247bb34`+`453a1926010`；用户拍板"三层全做"，不推迟）。架构裁决：**连接器只回传原始事实、不做 Doris 类型运算**（不 import fe-type），`dataSize/rowWidth` 除法归 fe-core（它有类型信息且此逻辑 connector-agnostic，对现有 iceberg/paimon/jdbc 零影响——它们 dataSize 恒 -1）。三层 = ① 精确 `numRows`（→spark numRows，仅当 numRows 键在但≤0，忠实复刻旧不对称）② `totalSize`（→spark totalSize，标准键缺才回退）作 dataSize，fe-core 除**全列宽** ③ 列文件估算（新 SPI `estimateDataSizeByListingFiles`，只普通 hive、采样 30 分区放大、best-effort→-1、内部 pin TCCL 因统计线程 fe-core 不 pin），fe-core 除**非分区列宽**（旧 getRowCountFromFileList 语义）。**复审修正**（4 维对抗 + 逐条反驳核验 `wf_e0f3a369-77d`）：两层商截 0 须塌成 UNKNOWN（层2→穿透层3、层3→UNKNOWN），非报"空表 0 行"。**已签字裁决（本 session）**：采样数不再随运行时 `hive_stats_partition_sample_size` 变（连接器持默认 30；开关 `enable_get_row_count_from_file_list` 仍 fe-core 侧生效）；malformed `numRows` 防御式恢复 spark 值（比旧 throw 更稳，已文档化 deviation）；文件列举非递归顶层（与扫描侧一致）；精确 view-skip 待 view SPI（今 view 无可列位置→-1）。 +- **下一步 = §4.2 其余读侧 SPI**（本 session 侦察 `wf_f904ad07-2d4`，5 维全 HEAD 核对；`HiveConnectorMetadata` 仍 read-mostly 骨架）。按建议序，每子步独立休眠 commit： + 1. **views（`viewExists`/`listViewNames`/`getViewDefinition` + 声明 `SUPPORTS_VIEW`，M）**：照 iceberg 模板；`getViewDefinition` 内复刻 Presto/Trino base64 view-text 解码（`HMSExternalTable:598-662` `parseTrinoViewDefinition`：`/* Presto View: */`→Gson→originalSql；dialect 无 caller 读，占位即可）。⚠两处设计决策：(a) `$partitions` 旧是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`，`useNativeTablePath()==false`），SPI sys-table 面（`PluginDrivenSysTable extends NativeSysTable`）**只支持 native**——`$partitions` 留 fe-core vs 建 native；(b) `BindRelation.java:623-653` plugin-view 臂硬编码 iceberg（`enable_query_iceberg_views` + iceberg 文案 + 拒时间旅行）→hive view 需泛化 config/文案 或留 HMS 臂。 + 2. **file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 + 3. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 + 4. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 + 5. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） From 0937f73203e47d89fda68efe8bbad85440181801 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 19:22:57 +0800 Subject: [PATCH 046/334] [feat](catalog) hive connector view SPI: viewExists/getViewDefinition/dropView + SUPPORTS_VIEW (dormant) Port the legacy HMS view path to the fe-connector-hive plugin so a hive VIEW is served through the generic plugin-driven path after the eventual hms cutover. Dormant today (hms is not in SPI_READY_TYPES, so the hive connector routes nothing live). - HmsTableInfo + ThriftHmsClient.convertTable: carry viewOriginalText / viewExpandedText (the hive view signal + SQL source), previously dropped. - HiveConnectorMetadata.viewExists: the legacy HMSExternalTable.isView predicate (view text present), NOT tableType -- returns false for a base table so a normal DROP TABLE keeps the table-handle path (dropTable routes a view to dropView via viewExists unconditionally). - HiveConnectorMetadata.getViewDefinition: byte-faithful port of legacy getViewText (expanded-first, skip the "/* Presto View */" sentinel, else base64/Gson-decode the Presto/Trino originalSql, raw-original fallback on any failure) + the view's columns (fe-core builds a flipped view's schema solely from getViewDefinition().getColumns()). dialect is a non-null placeholder fe-core never reads. - HiveConnectorMetadata.dropView: routes through the existing HmsClient.dropTable (hive has no separate drop-view), DorisConnectorException-normalized so PluginDrivenExternalCatalog.dropTable rewraps it as a DdlException. - listViewNames left at the SPI default (empty): hive listTableNames already includes views and the fe-core SHOW TABLES merge is a plain addAll, so a non-empty value would double-list (opposite of iceberg). - HiveConnector.getCapabilities: declare SUPPORTS_VIEW only; other connector-wide caps land in their own substeps, SUPPORTS_MVCC_SNAPSHOT intentionally withheld. Deferred to the flip phase (not exposed while dormant): the fe-core BindRelation plugin-view arm still hardcodes enable_query_iceberg_views + an iceberg-worded time-travel message; a flipped hive view needs a connector-neutral gate. Also the SHOW CREATE VIEW byte change for Presto-authored views. Tests: HiveConnectorMetadataViewTest (7) green; full hive+hms suites green (no regression, incl. the tableType=VIRTUAL_VIEW Top-N test); checkstyle 0; connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 18 ++ .../connector/hive/HiveConnectorMetadata.java | 141 +++++++++ .../hive/HiveConnectorMetadataViewTest.java | 267 ++++++++++++++++++ .../doris/connector/hms/HmsTableInfo.java | 31 ++ .../doris/connector/hms/ThriftHmsClient.java | 3 + 5 files changed, 460 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataViewTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index c5f3f24828c8c2..ca4c42864647ec 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; @@ -39,8 +40,10 @@ import java.io.IOException; import java.util.Collections; +import java.util.EnumSet; import java.util.Map; import java.util.Optional; +import java.util.Set; import java.util.concurrent.Callable; /** @@ -88,6 +91,21 @@ public ConnectorWritePlanProvider getWritePlanProvider() { return new HiveWritePlanProvider(getOrCreateClient(), properties, context); } + @Override + public Set getCapabilities() { + // SUPPORTS_VIEW: legacy HMSExternalTable resolves isView() from the remote table's view text and hive + // views are queryable/droppable/visible in SHOW TABLES. The generic plugin-driven path reproduces this + // ONLY under this capability: PluginDrivenExternalTable.isView() then consults the connector's + // viewExists, PluginDrivenExternalCatalog.dropTable routes a view DROP to dropView, and + // listTableNamesFromRemote merges listViewNames into SHOW TABLES — hive returns an EMPTY listViewNames + // (its listTableNames already includes views), so the merge is a no-op and each view is listed once + // (legacy parity). Inert until hms enters SPI_READY_TYPES. Other connector-wide capabilities + // (auto-analyze, show-create-ddl, metadata-preload) land in their own substeps; SUPPORTS_MVCC_SNAPSHOT + // is intentionally withheld (hive is non-MVCC), and Top-N / nested-prune are per-table markers, not + // connector-wide flags. + return EnumSet.of(ConnectorCapability.SUPPORTS_VIEW); + } + private HmsClient getOrCreateClient() { if (hmsClient == null) { synchronized (this) { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 0e994ae88e7387..87458d8edfedcf 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.ConnectorViewDefinition; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; @@ -52,6 +53,8 @@ import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; +import com.google.gson.Gson; +import com.google.gson.JsonObject; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; import org.apache.hadoop.fs.FileSystem; @@ -60,7 +63,9 @@ import org.apache.logging.log4j.Logger; import java.io.IOException; +import java.nio.charset.StandardCharsets; import java.util.ArrayList; +import java.util.Base64; import java.util.Collections; import java.util.HashMap; import java.util.LinkedHashMap; @@ -113,6 +118,19 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // a Hive view carries tableType VIRTUAL_VIEW. private static final String VIRTUAL_VIEW_TABLE_TYPE = "VIRTUAL_VIEW"; + // Presto/Trino view markers, replicating legacy HMSExternalTable.getViewText / parseTrinoViewDefinition. A + // Presto/Trino-authored hive view stores the bare sentinel as its expanded text (skipped) and the real + // definition as base64-encoded JSON inside the original text ("/* Presto View: */"). + private static final String PRESTO_VIEW_EXPANDED_SENTINEL = "/* Presto View */"; + private static final String PRESTO_VIEW_PREFIX = "/* Presto View: "; + private static final String PRESTO_VIEW_SUFFIX = " */"; + private static final String PRESTO_VIEW_ORIGINAL_SQL_KEY = "originalSql"; + + // Placeholder SQL dialect for a hive view. fe-core never reads ConnectorViewDefinition.getDialect() (the + // view body is converted by the session dialect in BindRelation.parseAndAnalyzeExternalView), but the DTO + // requires a non-null value; legacy getSqlDialect was likewise never consumed by the query path. + private static final String HIVE_VIEW_DIALECT = "hive"; + // HMS table-parameter keys for table statistics, replicating legacy StatisticsUtil.getHiveRowCount / // getRowCountFromParameters / getTotalSizeFromHMS. numRows is the exact row count; totalSize is the on-disk // data size. Each has a spark-written alternative key (spark writes its own stats keys, not the standard @@ -301,6 +319,129 @@ public TTableDescriptor buildTableDescriptor(ConnectorSession session, return desc; } + // ========== ConnectorTableOps: Views ========== + + /** + * Whether {@code dbName.viewName} is a hive VIEW, a connector-side port of legacy + * {@code HMSExternalTable.isView}: the authoritative signal is the PRESENCE OF VIEW TEXT + * ({@code viewOriginalText} or {@code viewExpandedText} set), not the {@code tableType} — a hive view + * always carries view text and a base table never does. Consumed by {@code PluginDrivenExternalTable} + * to resolve {@code isView()} (only when the connector declares {@link ConnectorCapability#SUPPORTS_VIEW}) + * and by {@code PluginDrivenExternalCatalog.dropTable} to route a DROP onto {@link #dropView}; returning + * {@code false} for a base table is exactly what keeps a normal DROP TABLE on the table-handle path. This + * uses the same single {@code getTable} the caller path needs and does NOT wrap in an auth context + * (ThriftHmsClient authenticates internally, unlike the iceberg connector). A missing table is not a view. + * + *

    Distinct from the {@code tableType}-based {@link #isView(HmsTableInfo)} the Top-N gate uses: that gate + * only excludes views from an optimization (a tableType proxy is adequate there and its unit test relies on + * it), whereas this view signal must be the legacy-exact text predicate so {@link #getViewDefinition} is + * only reached when the text needed to build the view SQL exists. + */ + @Override + public boolean viewExists(ConnectorSession session, String dbName, String viewName) { + try { + return isViewTable(hmsClient.getTable(dbName, viewName)); + } catch (HmsClientException e) { + LOG.debug("View existence check: '{}.{}' not found: {}", dbName, viewName, e.getMessage()); + return false; + } + } + + /** + * Loads the stored definition of a hive view, a connector-side port of legacy + * {@code HMSExternalTable.getViewText} plus the view-column half of {@code initHiveSchema}. ONE + * {@code hmsClient.getTable} supplies both the SQL body (via {@link #resolveViewText}) and the view's + * columns — a hive view exposes ordinary columns from its StorageDescriptor, built exactly like a base + * table's data columns. fe-core ({@code PluginDrivenExternalTable.initSchema}) takes a view's columns + * SOLELY from here (it never calls {@code getTableSchema} for a view), so the column list is non-empty for + * a real view. The {@code dialect} is a required-non-null placeholder fe-core never reads. Callers gate on + * {@link #viewExists}, so the view text is present; a defensive fail-loud guards the pathological + * empty-text case rather than letting the DTO constructor NPE. + */ + @Override + public ConnectorViewDefinition getViewDefinition(ConnectorSession session, String dbName, String viewName) { + HmsTableInfo tableInfo = hmsClient.getTable(dbName, viewName); + String sql = resolveViewText(tableInfo); + if (sql == null) { + throw new DorisConnectorException( + "Hive view " + dbName + "." + viewName + " has no view definition text"); + } + List columns = buildColumns(tableInfo); + return new ConnectorViewDefinition(sql, HIVE_VIEW_DIALECT, columns); + } + + /** + * Drops a hive view, a connector-side port of the way legacy {@code HiveMetadataOps.dropTableImpl} dropped a + * view: hive has no separate drop-view, a view is deleted through the same metastore {@code dropTable}. This + * is reached only via {@code PluginDrivenExternalCatalog.dropTable} after {@link #viewExists} confirmed the + * target is a view; a view is never transactional, so the transactional-table guard the table drop applies + * is unnecessary here. Failures are normalized into a {@link DorisConnectorException} (not a bare + * RuntimeException) so {@code PluginDrivenExternalCatalog.dropTable} rewraps them as a {@code DdlException}. + */ + @Override + public void dropView(ConnectorSession session, String dbName, String viewName) { + try { + hmsClient.dropTable(dbName, viewName); + } catch (HmsClientException e) { + throw new DorisConnectorException("Failed to drop Hive view " + + dbName + "." + viewName + ": " + e.getMessage(), e); + } + } + + // listViewNames is intentionally NOT overridden: hive's listTableNames (HMS get_all_tables) already + // includes views, and PluginDrivenExternalCatalog.listTableNamesFromRemote merges listViewNames into + // SHOW TABLES with a plain addAll (no dedup). Returning view names here would DOUBLE-list every hive view; + // the SPI default (empty) keeps SHOW TABLES listing each view exactly once, matching legacy. This is the + // opposite of iceberg, whose listTableNames subtracts views and whose listViewNames re-supplies them. + + /** + * Whether the metastore table carries view text, the exact predicate of legacy + * {@code HMSExternalTable.isView} ({@code isSetViewOriginalText() || isSetViewExpandedText()}). + */ + private static boolean isViewTable(HmsTableInfo tableInfo) { + return tableInfo.getViewOriginalText() != null || tableInfo.getViewExpandedText() != null; + } + + /** + * Resolves a hive view's SQL body, a byte-faithful port of legacy {@code HMSExternalTable.getViewText}: + * prefer {@code viewExpandedText} unless it is empty or the bare {@code "/* Presto View *}{@code /"} + * sentinel, otherwise parse the base64 Presto/Trino definition out of {@code viewOriginalText}. + */ + private static String resolveViewText(HmsTableInfo tableInfo) { + String expanded = tableInfo.getViewExpandedText(); + if (expanded != null && !expanded.isEmpty() && !PRESTO_VIEW_EXPANDED_SENTINEL.equals(expanded)) { + return expanded; + } + return parseTrinoViewDefinition(tableInfo.getViewOriginalText()); + } + + /** + * Extracts the SQL out of a Presto/Trino view definition stored in {@code originalText}, a port of legacy + * {@code HMSExternalTable.parseTrinoViewDefinition}. The format is + * {@code "/* Presto View: *}{@code /"} where the JSON carries an {@code originalSql} field. + * Returns {@code originalText} unchanged when it is not a Presto view, and falls back to the raw + * {@code originalText} on ANY decode/parse failure (legacy parity). + */ + private static String parseTrinoViewDefinition(String originalText) { + if (originalText == null || !originalText.contains(PRESTO_VIEW_PREFIX)) { + return originalText; + } + try { + String base64String = originalText.substring( + originalText.indexOf(PRESTO_VIEW_PREFIX) + PRESTO_VIEW_PREFIX.length(), + originalText.lastIndexOf(PRESTO_VIEW_SUFFIX)).trim(); + byte[] decodedBytes = Base64.getDecoder().decode(base64String); + String decodedString = new String(decodedBytes, StandardCharsets.UTF_8); + JsonObject jsonObject = new Gson().fromJson(decodedString, JsonObject.class); + if (jsonObject.has(PRESTO_VIEW_ORIGINAL_SQL_KEY)) { + return jsonObject.get(PRESTO_VIEW_ORIGINAL_SQL_KEY).getAsString(); + } + } catch (Exception e) { + LOG.warn("Decoding Presto view definition failed", e); + } + return originalText; + } + // ========== ConnectorStatisticsOps ========== /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataViewTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataViewTest.java new file mode 100644 index 00000000000000..487abb2ff6af7f --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataViewTest.java @@ -0,0 +1,267 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorCapability; +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.ConnectorViewDefinition; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsClientException; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Base64; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +/** + * Tests the hive connector VIEW SPI (§4.2 read-side SPI): {@link HiveConnectorMetadata#viewExists}, + * {@link HiveConnectorMetadata#getViewDefinition}, {@link HiveConnectorMetadata#dropView}, the empty + * {@code listViewNames} default, and {@link HiveConnector}'s {@code SUPPORTS_VIEW} declaration. + * + *

    WHY these assertions matter: + *

      + *
    • {@code viewExists} keys off the PRESENCE OF VIEW TEXT (legacy {@code HMSExternalTable.isView}), not the + * {@code tableType}. It drives both {@code PluginDrivenExternalTable.isView()} AND the unconditional + * {@code PluginDrivenExternalCatalog.dropTable -> viewExists -> dropView} routing, so it MUST return + * {@code false} for a base table — otherwise every hive DROP TABLE would misroute into dropView.
      • + *
    • {@code getViewDefinition} reproduces legacy {@code getViewText} bit-for-bit (expanded-first; skip the + * bare {@code /* Presto View *}{@code /} sentinel; else base64-decode the Presto/Trino {@code originalSql}; + * raw-original fallback on any decode failure) and supplies the view's columns — fe-core builds a flipped + * view's schema SOLELY from here, so the column list must be non-empty.
      • + *
    • {@code listViewNames} MUST stay empty: hive {@code listTableNames} already includes views, and the + * fe-core SHOW TABLES merge is a plain {@code addAll} with no dedup, so a non-empty value double-lists.
      • + *
    + */ +public class HiveConnectorMetadataViewTest { + + private static final String DB = "db"; + private static final String VIRTUAL_VIEW = "VIRTUAL_VIEW"; + + private HiveConnectorMetadata metadataOf(FakeHmsClient client) { + return new HiveConnectorMetadata(client, Collections.emptyMap(), new FakeConnectorContext()); + } + + private static ConnectorColumn col(String name, String typeName) { + return new ConnectorColumn(name, ConnectorType.of(typeName), null, true, null); + } + + /** A hive VIEW carrying a plain (non-Presto) expanded text and ordinary columns. */ + private static HmsTableInfo plainView(String name, String expandedText) { + return HmsTableInfo.builder() + .dbName(DB).tableName(name) + .tableType(VIRTUAL_VIEW) + .viewExpandedText(expandedText) + .viewOriginalText(expandedText) + .columns(Arrays.asList(col("id", "INT"), col("name", "STRING"))) + .parameters(Collections.emptyMap()) + .build(); + } + + /** A base table: no view text. */ + private static HmsTableInfo baseTable(String name) { + return HmsTableInfo.builder() + .dbName(DB).tableName(name) + .tableType("MANAGED_TABLE") + .columns(Arrays.asList(col("id", "INT"))) + .parameters(Collections.emptyMap()) + .build(); + } + + /** A Presto/Trino-authored hive view: sentinel expanded text + base64 JSON original text. */ + private static HmsTableInfo prestoView(String name, String originalSql) { + String json = "{\"originalSql\":\"" + originalSql + "\",\"catalog\":\"hive\",\"schema\":\"db\"}"; + String base64 = Base64.getEncoder().encodeToString(json.getBytes(StandardCharsets.UTF_8)); + return HmsTableInfo.builder() + .dbName(DB).tableName(name) + .tableType(VIRTUAL_VIEW) + .viewExpandedText("/* Presto View */") + .viewOriginalText("/* Presto View: " + base64 + " */") + .columns(Arrays.asList(col("id", "INT"))) + .parameters(Collections.emptyMap()) + .build(); + } + + @Test + public void viewExistsTrueForViewFalseForBaseTableAndMissing() { + FakeHmsClient client = new FakeHmsClient(); + client.put(plainView("v", "SELECT 1")); + client.put(baseTable("t")); + HiveConnectorMetadata metadata = metadataOf(client); + + Assertions.assertTrue(metadata.viewExists(null, DB, "v"), "a table with view text is a view"); + Assertions.assertFalse(metadata.viewExists(null, DB, "t"), + "a base table must NOT be a view (else DROP TABLE misroutes to dropView)"); + Assertions.assertFalse(metadata.viewExists(null, DB, "missing"), + "a missing table is not a view (getTable throws HmsClientException -> false)"); + } + + @Test + public void getViewDefinitionReturnsExpandedTextColumnsAndPlaceholderDialect() { + FakeHmsClient client = new FakeHmsClient(); + client.put(plainView("v", "SELECT id, name FROM base")); + HiveConnectorMetadata metadata = metadataOf(client); + + ConnectorViewDefinition def = metadata.getViewDefinition(null, DB, "v"); + Assertions.assertEquals("SELECT id, name FROM base", def.getSql(), + "a plain view returns its expanded text verbatim"); + Assertions.assertEquals("hive", def.getDialect(), "dialect is the placeholder (fe-core never reads it)"); + Assertions.assertEquals(Arrays.asList("id", "name"), + def.getColumns().stream().map(ConnectorColumn::getName).collect(Collectors.toList()), + "view columns come from the metastore table columns (non-empty)"); + } + + @Test + public void getViewDefinitionDecodesPrestoBase64OriginalSql() { + FakeHmsClient client = new FakeHmsClient(); + client.put(prestoView("pv", "SELECT * FROM employees")); + HiveConnectorMetadata metadata = metadataOf(client); + + ConnectorViewDefinition def = metadata.getViewDefinition(null, DB, "pv"); + Assertions.assertEquals("SELECT * FROM employees", def.getSql(), + "the sentinel expanded text is skipped and the base64 originalSql is decoded"); + } + + @Test + public void getViewDefinitionFallsBackToRawOriginalOnMalformedBase64() { + FakeHmsClient client = new FakeHmsClient(); + HmsTableInfo malformed = HmsTableInfo.builder() + .dbName(DB).tableName("bad") + .tableType(VIRTUAL_VIEW) + .viewExpandedText("/* Presto View */") + .viewOriginalText("/* Presto View: @@not-base64@@ */") + .columns(Arrays.asList(col("id", "INT"))) + .parameters(Collections.emptyMap()) + .build(); + client.put(malformed); + HiveConnectorMetadata metadata = metadataOf(client); + + ConnectorViewDefinition def = metadata.getViewDefinition(null, DB, "bad"); + Assertions.assertEquals("/* Presto View: @@not-base64@@ */", def.getSql(), + "a decode failure falls back to the raw original text (legacy parity)"); + } + + @Test + public void dropViewRoutesToMetastoreDropTable() { + FakeHmsClient client = new FakeHmsClient(); + client.put(plainView("v", "SELECT 1")); + HiveConnectorMetadata metadata = metadataOf(client); + + metadata.dropView(null, DB, "v"); + Assertions.assertEquals(Collections.singletonList(DB + ".v"), client.droppedTables, + "a view drop is a metastore dropTable (hive has no separate drop-view)"); + } + + @Test + public void listViewNamesIsEmptyToAvoidDoubleListing() { + HiveConnectorMetadata metadata = metadataOf(new FakeHmsClient()); + Assertions.assertTrue(metadata.listViewNames(null, DB).isEmpty(), + "hive listTableNames already includes views; listViewNames must be empty or SHOW TABLES doubles"); + } + + @Test + public void connectorDeclaresSupportsView() { + HiveConnector connector = new HiveConnector(Collections.emptyMap(), new FakeConnectorContext()); + Assertions.assertTrue(connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_VIEW), + "the hive connector must declare SUPPORTS_VIEW so the generic view path lights up post-flip"); + } + + /** + * Recording fake {@link HmsClient}: serves prebuilt {@link HmsTableInfo}s by name (throwing + * {@link HmsClientException} for an unknown name, like the real client) and records dropTable calls. All + * other operations are unused by the view SPI and throw. + */ + private static final class FakeHmsClient implements HmsClient { + private final Map tables = new HashMap<>(); + private final List droppedTables = new ArrayList<>(); + + void put(HmsTableInfo info) { + tables.put(info.getTableName(), info); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + HmsTableInfo info = tables.get(tableName); + if (info == null) { + throw new HmsClientException("table not found: " + dbName + "." + tableName); + } + return info; + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return tables.containsKey(tableName); + } + + @Override + public void dropTable(String dbName, String tableName) { + droppedTables.add(dbName + "." + tableName); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, + List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java index 7989dda3e0dba5..f16d3bbdd29aaf 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsTableInfo.java @@ -39,6 +39,8 @@ public final class HmsTableInfo { private final String inputFormat; private final String outputFormat; private final String serializationLib; + private final String viewOriginalText; + private final String viewExpandedText; private final List columns; private final List partitionKeys; private final List bucketCols; @@ -54,6 +56,8 @@ private HmsTableInfo(Builder builder) { this.inputFormat = builder.inputFormat; this.outputFormat = builder.outputFormat; this.serializationLib = builder.serializationLib; + this.viewOriginalText = builder.viewOriginalText; + this.viewExpandedText = builder.viewExpandedText; this.columns = builder.columns == null ? Collections.emptyList() : Collections.unmodifiableList(builder.columns); @@ -101,6 +105,21 @@ public String getSerializationLib() { return serializationLib; } + /** + * Raw {@code viewOriginalText} of a view ({@code null} for a base table). For a Presto/Trino-authored hive + * view this carries the {@code "/* Presto View: *}{@code /"} definition; for a native hive view it + * is the original CREATE VIEW SQL. Presence of this (or {@link #getViewExpandedText()}) is the hive + * view signal (legacy {@code HMSExternalTable.isView}). + */ + public String getViewOriginalText() { + return viewOriginalText; + } + + /** Raw {@code viewExpandedText} of a view ({@code null} for a base table); the fully-qualified view SQL. */ + public String getViewExpandedText() { + return viewExpandedText; + } + /** Data columns (excludes partition keys). */ public List getColumns() { return columns; @@ -151,6 +170,8 @@ public static final class Builder { private String inputFormat; private String outputFormat; private String serializationLib; + private String viewOriginalText; + private String viewExpandedText; private List columns; private List partitionKeys; private List bucketCols; @@ -196,6 +217,16 @@ public Builder serializationLib(String val) { return this; } + public Builder viewOriginalText(String val) { + this.viewOriginalText = val; + return this; + } + + public Builder viewExpandedText(String val) { + this.viewExpandedText = val; + return this; + } + public Builder columns(List val) { this.columns = val; return this; diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 55e0eaa031de5d..409a2adb5de925 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -551,6 +551,9 @@ private HmsTableInfo convertTable(Table table) { .tableName(table.getTableName()) .tableType(table.getTableType()) .parameters(table.getParameters()) + // View text (null for a base table) — the hive VIEW SPI's isView signal + view-SQL source. + .viewOriginalText(table.getViewOriginalText()) + .viewExpandedText(table.getViewExpandedText()) .columns(columns) .partitionKeys(partKeys); From 00beb956ab2e9d6423a3db71e63b247b0057b417 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 19:24:16 +0800 Subject: [PATCH 047/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20view=20SPI=20DONE;=20next=20=3D=20file-format=20serde=20f?= =?UTF-8?q?ix?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the hive connector view SPI landing (6676bf365cf) and re-point the next substep to the file-format serde read-correctness fix. Capture the three flip-time view couplings deferred out of the dormant landing (BindRelation iceberg-hardcoded config gate + time-travel message, SHOW CREATE VIEW byte change for Presto-authored views, stale invariant comment) plus the $partitions TVF-vs-native decision, so they are not lost before the flip. Recon: wf_ecb9c379-9bb. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 56ac31e4bac799..0d4e498e8c0829 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -41,9 +41,10 @@ - **✅ §4.2 按表能力接缝 DONE（`329853bf19c`）**：**设计决策已签字**——两个"按文件格式启用"的加速能力（Top-N 延迟 / 嵌套列裁剪）走**按表接缝**忠实复刻（不 blanket-declare、不放弃；对齐 Trino 逐表委派）。机制 = 连接器在 `getTableSchema` 发射 FE 内部 schema-property marker `ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY`（CSV of `ConnectorCapability.name()`；缓存友好、无额外 HMS 往返、format 留连接器侧，复用 `partition_columns` 模式），fe-core `PluginDrivenExternalTable.supportsTopNLazyMaterialize/supportsNestedColumnPrune` 经新 `hasScanCapability` = 连接器级 Set **OR** 按表 marker；marker key 从 SHOW CREATE 属性剔除。**Top-N 已发射**（orc/parquet 非 view 非 iceberg/hudi-on-HMS，精确 input-format 类匹配 = 旧 `SUPPORTED_HIVE_TOPN_LAZY_FILE_FORMATS`）。iceberg/paimon/jdbc 零改（永不发 marker）。全绿（连接器 178 测 + fe-core 32 测 + checkstyle 0 + import gate）。 - ⛔**NESTED_PRUNE 遗留（硬门，下一子步前置验证）**：同接缝，连接器**暂未发射** marker——`SlotTypeReplacer.java:678-680` 把嵌套访问路径重写成 `Column.getUniqueId()`，iceberg 靠自带 field-id（`withUniqueId`）正确，**hive 连接器现在没设 field-id**；对 hive 开启前须端到端验证 hive orc/parquet 的 uniqueId 语义（BE 按 id 匹配嵌套叶子）否则读 NULL。旧 hive nested-prune 走同一 `SlotTypeReplacer` 且能工作→查旧 hive column uniqueId 赋值 vs SPI 路径 `ConnectorColumnConverter.convertColumns`；验证通过则连接器发 `SUPPORTS_NESTED_COLUMN_PRUNE` marker（parquet/orc、非 hudi），否则记差异 fail-loud。 - **✅ §4.2 `getTableStatistics` 三层全做 DONE**（`eded05b05e6`+`8c6d247bb34`+`453a1926010`；用户拍板"三层全做"，不推迟）。架构裁决：**连接器只回传原始事实、不做 Doris 类型运算**（不 import fe-type），`dataSize/rowWidth` 除法归 fe-core（它有类型信息且此逻辑 connector-agnostic，对现有 iceberg/paimon/jdbc 零影响——它们 dataSize 恒 -1）。三层 = ① 精确 `numRows`（→spark numRows，仅当 numRows 键在但≤0，忠实复刻旧不对称）② `totalSize`（→spark totalSize，标准键缺才回退）作 dataSize，fe-core 除**全列宽** ③ 列文件估算（新 SPI `estimateDataSizeByListingFiles`，只普通 hive、采样 30 分区放大、best-effort→-1、内部 pin TCCL 因统计线程 fe-core 不 pin），fe-core 除**非分区列宽**（旧 getRowCountFromFileList 语义）。**复审修正**（4 维对抗 + 逐条反驳核验 `wf_e0f3a369-77d`）：两层商截 0 须塌成 UNKNOWN（层2→穿透层3、层3→UNKNOWN），非报"空表 0 行"。**已签字裁决（本 session）**：采样数不再随运行时 `hive_stats_partition_sample_size` 变（连接器持默认 30；开关 `enable_get_row_count_from_file_list` 仍 fe-core 侧生效）；malformed `numRows` 防御式恢复 spark 值（比旧 throw 更稳，已文档化 deviation）；文件列举非递归顶层（与扫描侧一致）；精确 view-skip 待 view SPI（今 view 无可列位置→-1）。 -- **下一步 = §4.2 其余读侧 SPI**（本 session 侦察 `wf_f904ad07-2d4`，5 维全 HEAD 核对；`HiveConnectorMetadata` 仍 read-mostly 骨架）。按建议序，每子步独立休眠 commit： - 1. **views（`viewExists`/`listViewNames`/`getViewDefinition` + 声明 `SUPPORTS_VIEW`，M）**：照 iceberg 模板；`getViewDefinition` 内复刻 Presto/Trino base64 view-text 解码（`HMSExternalTable:598-662` `parseTrinoViewDefinition`：`/* Presto View: */`→Gson→originalSql；dialect 无 caller 读，占位即可）。⚠两处设计决策：(a) `$partitions` 旧是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`，`useNativeTablePath()==false`），SPI sys-table 面（`PluginDrivenSysTable extends NativeSysTable`）**只支持 native**——`$partitions` 留 fe-core vs 建 native；(b) `BindRelation.java:623-653` plugin-view 臂硬编码 iceberg（`enable_query_iceberg_views` + iceberg 文案 + 拒时间旅行）→hive view 需泛化 config/文案 或留 HMS 臂。 - 2. **file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 +- **✅ §4.2 视图 SPI DONE（`6676bf365cf`）**（本轮侦察 `wf_ecb9c379-9bb`：4 维读者 + 完整性 critic，全 HEAD 核对）。连接器侧四件套：`HmsTableInfo`+`ThriftHmsClient.convertTable` 补 `viewOriginalText`/`viewExpandedText`（原被丢弃）；`viewExists`=**legacy `HMSExternalTable.isView` 谓词=view 文本存在**（非 tableType——base 表须返 false，否则 `PluginDrivenExternalCatalog.dropTable` **无条件** viewExists→dropView 会误路由 base 表 DROP）；`getViewDefinition`=忠实复刻 `getViewText`（expanded-first→跳 `/* Presto View */` sentinel→`viewOriginalText` base64/Gson 解 `originalSql`→任何失败回退 raw；dialect 占位不被读；列走 `buildColumns`，**fe-core 视图列只来自此处**——`initSchema` isView 早返、不调 getTableSchema）；`dropView`=复用 `HmsClient.dropTable`（hive 无独立 drop-view）、`DorisConnectorException` 归一化好让 fe-core 转 `DdlException`；`listViewNames` **保持 SPI 默认 empty**（hive `listTableNames`=getAllTables 已含视图，fe-core SHOW TABLES 合并是无去重 `addAll`，非空会双列；与 iceberg 相反）；`HiveConnector.getCapabilities`=仅 `SUPPORTS_VIEW`（其余连接器级能力各自子步、MVCC 故意不声明）。全绿（7 视图测 + hive 157 + hms 150 套无回归 + checkstyle 0 + import gate 净）。**留到翻闸的耦合（休眠期不暴露，勿现在动）见文末**。 +- **下一步 = §4.2 file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 + - 之后续做：**`listPartitions`（含 `lastModifiedMillis`）** 镜像 paimon `collectPartitions`（`lastModifiedMillis` 唯一影响 MTMV `PluginDrivenMvccExternalTable.listLatestPartitions:269`；旧源 `transient_lastDdlTime*1000` 需 per-partition `getParameters()` 往返——决策=付这往返 vs 接受 -1；**不**声明 `SUPPORTS_PARTITION_STATS`）；**`getCapabilities` 其余连接器级项**（`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；禁 `SUPPORTS_MVCC_SNAPSHOT`/`SUPPORTS_PASSTHROUGH_QUERY`/`SUPPORTS_PARTITION_STATS`）；**`HiveTableFormatDetector` 平价**（补 3 个 LZO text 格式 `HMSExternalTable:170-172`、UNKNOWN 改 throw、view 短路）。 + - ⚠**视图翻闸期遗留（3 项，休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理）**：(1) `BindRelation.java:623-653` plugin-view 臂硬编码 `enable_query_iceberg_views` + iceberg 文案 + 拒时间旅行——翻闸后 hive view 走此臂会用**错的开关**、且 legacy hive view 本无时间旅行拒绝；中和须**连接器中立单开关**（如 `enable_query_external_views`，别 `if(engine)`/别 OR 两开关）。(2) SHOW CREATE VIEW 字节变化：plugin 路径出 Presto-**解码**后的 `getViewText`，legacy HMS 出 raw（Presto-**编码**）——Presto/Trino 视图字节不同，用户可见，翻闸前定。(3) `BindRelation:626-632` 陈旧注释「只有 iceberg 声明 SUPPORTS_VIEW」翻闸时更新。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 4. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 5. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 From cb4478ae9d03cb9fe3ca411b2ac3d92447136075 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 20:08:59 +0800 Subject: [PATCH 048/334] [doc](catalog) HMS cutover: lock view-flip decisions D5 (drop query-time view gate, Trino-aligned) + D6 (accept decoded SHOW CREATE VIEW) User sign-off. At the flip: (D5) the BindRelation plugin-view arm serves any external view whenever isView() with NO enable_query_*_views gate (Trino has no such runtime toggle); both legacy flags become deprecated no-ops. This also ungates iceberg views (currently live behind enable_query_iceberg_views), so it is a visible iceberg change to bundle deliberately with the flip. (D6) SHOW CREATE VIEW keeps the plugin path's Presto/Trino-decoded getViewText output (more readable than the legacy raw-encoded bytes); update any raw-encoded regression baseline at the flip. No fe-core change now (dormant). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md | 2 ++ 2 files changed, 3 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 0d4e498e8c0829..4f978d51146f7a 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -44,7 +44,7 @@ - **✅ §4.2 视图 SPI DONE（`6676bf365cf`）**（本轮侦察 `wf_ecb9c379-9bb`：4 维读者 + 完整性 critic，全 HEAD 核对）。连接器侧四件套：`HmsTableInfo`+`ThriftHmsClient.convertTable` 补 `viewOriginalText`/`viewExpandedText`（原被丢弃）；`viewExists`=**legacy `HMSExternalTable.isView` 谓词=view 文本存在**（非 tableType——base 表须返 false，否则 `PluginDrivenExternalCatalog.dropTable` **无条件** viewExists→dropView 会误路由 base 表 DROP）；`getViewDefinition`=忠实复刻 `getViewText`（expanded-first→跳 `/* Presto View */` sentinel→`viewOriginalText` base64/Gson 解 `originalSql`→任何失败回退 raw；dialect 占位不被读；列走 `buildColumns`，**fe-core 视图列只来自此处**——`initSchema` isView 早返、不调 getTableSchema）；`dropView`=复用 `HmsClient.dropTable`（hive 无独立 drop-view）、`DorisConnectorException` 归一化好让 fe-core 转 `DdlException`；`listViewNames` **保持 SPI 默认 empty**（hive `listTableNames`=getAllTables 已含视图，fe-core SHOW TABLES 合并是无去重 `addAll`，非空会双列；与 iceberg 相反）；`HiveConnector.getCapabilities`=仅 `SUPPORTS_VIEW`（其余连接器级能力各自子步、MVCC 故意不声明）。全绿（7 视图测 + hive 157 + hms 150 套无回归 + checkstyle 0 + import gate 净）。**留到翻闸的耦合（休眠期不暴露，勿现在动）见文末**。 - **下一步 = §4.2 file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 - 之后续做：**`listPartitions`（含 `lastModifiedMillis`）** 镜像 paimon `collectPartitions`（`lastModifiedMillis` 唯一影响 MTMV `PluginDrivenMvccExternalTable.listLatestPartitions:269`；旧源 `transient_lastDdlTime*1000` 需 per-partition `getParameters()` 往返——决策=付这往返 vs 接受 -1；**不**声明 `SUPPORTS_PARTITION_STATS`）；**`getCapabilities` 其余连接器级项**（`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；禁 `SUPPORTS_MVCC_SNAPSHOT`/`SUPPORTS_PASSTHROUGH_QUERY`/`SUPPORTS_PARTITION_STATS`）；**`HiveTableFormatDetector` 平价**（补 3 个 LZO text 格式 `HMSExternalTable:170-172`、UNKNOWN 改 throw、view 短路）。 - - ⚠**视图翻闸期遗留（3 项，休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理）**：(1) `BindRelation.java:623-653` plugin-view 臂硬编码 `enable_query_iceberg_views` + iceberg 文案 + 拒时间旅行——翻闸后 hive view 走此臂会用**错的开关**、且 legacy hive view 本无时间旅行拒绝；中和须**连接器中立单开关**（如 `enable_query_external_views`，别 `if(engine)`/别 OR 两开关）。(2) SHOW CREATE VIEW 字节变化：plugin 路径出 Presto-**解码**后的 `getViewText`，legacy HMS 出 raw（Presto-**编码**）——Presto/Trino 视图字节不同，用户可见，翻闸前定。(3) `BindRelation:626-632` 陈旧注释「只有 iceberg 声明 SUPPORTS_VIEW」翻闸时更新。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 + - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 4. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 5. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 diff --git a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md index f1f278e535141f..99745c16b0dda7 100644 --- a/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md +++ b/plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md @@ -109,6 +109,8 @@ Covered by `hms-event-pipeline-findings-2026-07-07.md`. Thin fe-core role-aware - **D2 — Cache ownership → CONNECTOR-OWNED (option a).** Retire the fe-core `HiveExternalMetaCache`/hudi/iceberg-on-hms meta caches; the connector owns scan-side caching (paimon/iceberg-native precedent). Delete the route/loader `instanceof HMSExternalCatalog` sites (`ExternalMetaCacheRouteResolver:66`, `HiveExternalMetaCache:203/274`, `IcebergExternalMetaCache:234`) + the partition-cache-inconsistency self-heal moves connector-side (or is dropped — decide at implementation). This is the largest catalog-adjacent structural item; it lands with the flip set. - **D3 — Iceberg-on-HMS delivery → BUILD SIBLING SPI BEFORE THE FLIP (option a), no regression.** The cross-plugin sibling-connector SPI (§4.4) is a hard precondition and must land dormant before the flip so existing iceberg-on-HMS tables keep working. If it slips, the flip slips (no fail-loud-reject regression). - **D4 — PR shape (decide at flip time).** two-PR split (paimon: revertable flip + mechanical delete) vs one squash (iceberg). **Leaning two-PR;** confirm at the flip. +- **D5 — External-view query-time enable gate → DROP entirely, Trino-aligned (user sign-off 2026-07-07).** At the flip, the `BindRelation` plugin-view arm (`:634`) stops gating on `Config.enable_query_iceberg_views` (and the dead HMS arm's `enable_query_hive_views` at `:594` goes away with the arm): an external view is served whenever `isView()` — no per-connector query-time toggle (Trino has none; the two Doris flags were legacy safety valves). Both `enable_query_hive_views` / `enable_query_iceberg_views` become deprecated no-ops (keep the `@ConfField` for one release to avoid "unknown config" on upgrade; remove later). Also neutralize the iceberg-worded time-travel rejection message + the stale `:626-632` invariant comment. **Consequence to bundle deliberately:** this ungates *iceberg* views too, which are already live behind `enable_query_iceberg_views` — so D5 is a **visible iceberg behavior change** and must ship with (or as a deliberate part of) the flip, not leak in early. Satisfies the iron rule with **no** `if(engine)`/OR-of-two-flags. +- **D6 — SHOW CREATE VIEW output → accept the plugin path's decoded form (user sign-off 2026-07-07).** Post-flip a hive view's SHOW CREATE renders the generic `CREATE VIEW AS ` where `getViewText()` is the **Presto/Trino-decoded** SQL (legacy HMS `HiveMetaStoreClientHelper.showCreateTable` emitted the **raw encoded** `/* Presto View: */`). The decoded form is more readable and is the accepted output; any hive-view SHOW CREATE regression baseline that pinned the raw-encoded bytes is updated to the decoded text at the flip. ### §4.2a — Per-column statistics SPI (from D1=preserve) - Add `ConnectorStatisticsOps.getColumnStatistics(session, handle, colName)` → neutral column-stat DTO. `PluginDrivenExternalTable.getColumnStatistic` override consults it (query-planning cache-miss fast path). Hive connector ports the HMS/spark col-stats read (incl. partition-level col stats `getHivePartitionColumnStats` for the ANALYZE metadata path); iceberg connector ports `getIcebergColumnStats` behind `enable_fetch_iceberg_stats`. This also gives `HMSAnalysisTask`'s metadata fast path a home (revisit whether `ExternalAnalysisTask` consults the new SPI vs SQL-only FULL analyze — the DTO makes preserving the metadata path feasible). From bca89452e0ba3795b428c6115e6b7857022eaea1 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 21:09:43 +0800 Subject: [PATCH 049/334] [fix](catalog) hive connector file-format detection: serde-authoritative + FORMAT_TEXT vs CSV vs JSON (dormant) Fix two read-correctness bugs in the hive connector's file-format detection by reproducing legacy HMSExternalTable.getFileFormatType faithfully. Dormant today (hms is not in SPI_READY_TYPES; only the hive connector emits the text-family tokens, so the fe-core vocabulary split below is hive-only in effect). Bug 1 (JSON read as CSV): HiveFileFormat.detect was input-format-first, and a standard hive JSON table has inputFormat=TextInputFormat (its JSON-ness lives only in the serde), so it resolved to text and was read by the CSV reader. Bug 2 (default hive-text read as CSV): the connector collapsed LazySimpleSerDe / MultiDelimitSerDe / OpenCSVSerde all to one "text" token, and fe-core mapFileFormatType mapped "text" -> FORMAT_CSV_PLAIN. But BE dispatches FORMAT_TEXT (LazySimpleSerDe/MultiDelimit) to a DISTINCT reader from FORMAT_CSV_PLAIN (OpenCSV) -- the text reader honors hive collection/map delimiters, \N nulls and hive escaping. So the DEFAULT hive text serde was mis-read for nested/complex columns, hive nulls and escapes. - HiveFileFormat: two-stage, serde-authoritative for the text family (inputFormat -> parquet/orc/text-file family by substring; text family -> serde decides json/csv/hive-text by EXACT FQCN). Adds a CSV member. Reproduces the OpenX-JSON read_hive_json_in_one_column branch (one CSV column when the first column is a string; fail-loud otherwise). Fails loud on an unrecognized inputFormat/serde (legacy parity, user sign-off) instead of silently degrading to FORMAT_JNI. - fe-core PluginDrivenScanNode.mapFileFormatType: split the fused case so "text" -> FORMAT_TEXT and "csv" -> FORMAT_CSV_PLAIN. Neutral, pre-existing tokens; no if(engine)/instanceof (iron-rule clean); hive is their sole emitter. Must land atomically with the connector's distinct csv emission or OpenCSV (correct-by-accident today) would regress. - HiveScanPlanProvider: pass the session read_hive_json_in_one_column flag + handle firstColumnIsString into detect; widen the text-props gate to TEXT||CSV||JSON; stop the dead per-partition detect (leave null; the whole node reads one table-level format) so a partition's unusual serde can't fail-loud. - HiveTableHandle + getTableHandle: carry firstColumnIsString (STRING-only, legacy parity), precomputed from the already-loaded metastore table (no extra fetch). - HiveTextProperties: add the legacy hive-2 JsonSerDe + the serde2 MultiDelimit FQCN so the emitted text params match the serdes the detector recognizes. read_hive_json_in_one_column reproduced (user sign-off). Test: HiveFileFormatTest rewritten as the legacy serde truth table (12 cases: text/csv/json split, the JSON-as-CSV regression, OpenX one-column on/off, fail-loud on unknown). Full hive+hms suites green (153/150, no regression); fe-core compiles; checkstyle 0; import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 16 ++ .../hive/HiveConnectorProperties.java | 5 + .../doris/connector/hive/HiveFileFormat.java | 165 ++++++++++++------ .../connector/hive/HiveScanPlanProvider.java | 31 +++- .../doris/connector/hive/HiveTableHandle.java | 20 +++ .../connector/hive/HiveTextProperties.java | 20 ++- .../connector/hive/HiveFileFormatTest.java | 136 ++++++++++----- .../datasource/PluginDrivenScanNode.java | 3 + 8 files changed, 285 insertions(+), 111 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 87458d8edfedcf..7ae7ce0270ece7 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -215,6 +215,7 @@ public Optional getTableHandle( .partitionKeyNames(partKeyNames) .sdParameters(tableInfo.getSdParameters()) .tableParameters(tableInfo.getParameters()) + .firstColumnIsString(firstColumnIsString(tableInfo)) .build(); return Optional.of(handle); } @@ -1102,6 +1103,21 @@ private static boolean isView(HmsTableInfo tableInfo) { return VIRTUAL_VIEW_TABLE_TYPE.equalsIgnoreCase(tableInfo.getTableType()); } + /** + * Whether the table's first (data) column is a {@code STRING}, reproducing legacy + * {@code HMSExternalTable.firstColumnIsString} ({@code isScalarType(PrimitiveType.STRING)} — {@code STRING} + * only, NOT {@code varchar}/{@code char}). Stamped onto the handle so the read-format detector can apply the + * OpenX-JSON {@code read_hive_json_in_one_column} gate without a second metastore fetch. A table with no data + * columns degrades to {@code false} (the OpenX one-column mode is nonsensical there). + */ + private static boolean firstColumnIsString(HmsTableInfo tableInfo) { + List columns = tableInfo.getColumns(); + if (columns == null || columns.isEmpty()) { + return false; + } + return "STRING".equals(columns.get(0).getType().getTypeName()); + } + private static HmsTypeMapping.Options buildTypeMappingOptions(Map props) { boolean binaryAsString = Boolean.parseBoolean( props.getOrDefault(HiveConnectorProperties.ENABLE_MAPPING_BINARY_AS_STRING, "false")); diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java index ed3f99f2aee867..0ab3b22e91f002 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java @@ -82,6 +82,11 @@ private HiveConnectorProperties() { public static final String TEXT_COMPRESSION_UNCOMPRESSED = "uncompressed"; public static final String TEXT_COMPRESSION_PLAIN = "plain"; + // Session variable gating the OpenX-JSON "read the whole JSON row into one CSV column" mode (legacy + // SessionVariable.read_hive_json_in_one_column). Byte-identical to the fe-core session-var name; it is + // surfaced through ConnectorSession.getSessionProperties() (VariableMgr dumps all visible vars). + public static final String SESSION_READ_HIVE_JSON_IN_ONE_COLUMN = "read_hive_json_in_one_column"; + /** * Bucket algorithm string produced by {@code CreateTableInfoToConnectorRequestConverter} for a * random (non-hash) distribution. Hive external tables only support hash bucketing. diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileFormat.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileFormat.java index 034f0d80429b15..dbdf56550b85e0 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileFormat.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileFormat.java @@ -17,16 +17,59 @@ package org.apache.doris.connector.hive; +import org.apache.doris.connector.api.DorisConnectorException; + +import java.util.Locale; + /** - * Maps Hive InputFormat class names to file format strings understood by BE. + * Resolves a Hive table's read file format, a faithful port of legacy + * {@code HMSExternalTable.getFileFormatType} (+ {@code HiveMetaStoreClientHelper.HiveFileFormat.getFormat}). + * + *

    The algorithm is TWO-STAGE and serde-authoritative for the text family: the {@code inputFormat} + * class name picks the coarse family (parquet / orc / text) by substring, and for the text family the + * SerDe picks the fine format. This is why detection must NOT be input-format-first: a standard Hive + * JSON table has {@code inputFormat=org.apache.hadoop.mapred.TextInputFormat} — its JSON-ness lives ONLY in + * the JsonSerDe, so an input-format-first classifier would mis-read it as text/CSV.

    + * + *

    The five values map 1:1 to the BE {@code TFileFormatType} the generic + * {@code PluginDrivenScanNode.mapFileFormatType} resolves the emitted {@link #getFormatName() token} to: + * {@code parquet}->FORMAT_PARQUET, {@code orc}->FORMAT_ORC, {@code text}->FORMAT_TEXT, {@code csv}-> + * FORMAT_CSV_PLAIN, {@code json}->FORMAT_JSON. {@code FORMAT_TEXT} (hive {@code LazySimpleSerDe} / + * {@code MultiDelimitSerDe}) is a DISTINCT BE reader from {@code FORMAT_CSV_PLAIN} (hive {@code OpenCSVSerde}): + * the text reader honors hive collection/map delimiters, {@code \\N} nulls and hive escaping, so the two must + * not be collapsed.

    */ public enum HiveFileFormat { PARQUET("parquet"), ORC("orc"), + // Hive text family (LazySimpleSerDe / MultiDelimitSerDe) -> BE FORMAT_TEXT. TEXT("text"), - JSON("json"), - UNKNOWN("unknown"); + // Hive OpenCSVSerde (and OpenX-JSON read in one column) -> BE FORMAT_CSV_PLAIN. + CSV("csv"), + // Hive JSON serdes -> BE FORMAT_JSON. + JSON("json"); + + // Coarse inputFormat families, mirroring legacy HiveMetaStoreClientHelper.HiveFileFormat descs. Detection + // is a lowercase SUBSTRING match in THIS order (text first), first match wins; the LZO text input formats + // (com.hadoop...LzoTextInputFormat / DeprecatedLzoTextInputFormat) contain "text" and so land on text. + private static final String FAMILY_TEXT = "text"; + private static final String FAMILY_PARQUET = "parquet"; + private static final String FAMILY_ORC = "orc"; + + // SerDe class names, mirroring the legacy constants in HiveMetaStoreClientHelper (the connector cannot + // import fe-core). MultiDelimitSerDe is matched under BOTH its modern (serde2, the legacy Doris constant) + // and historical (contrib.serde2) package names — both are valid Hive classes and both read as FORMAT_TEXT; + // recognizing both is a small, safe superset of legacy's serde2-only match (it only makes more tables + // readable, never breaks one). + private static final String LAZY_SIMPLE_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; + private static final String MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; + private static final String CONTRIB_MULTI_DELIMIT_SERDE = + "org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe"; + private static final String OPEN_CSV_SERDE = "org.apache.hadoop.hive.serde2.OpenCSVSerde"; + private static final String HIVE_JSON_SERDE = "org.apache.hive.hcatalog.data.JsonSerDe"; + private static final String LEGACY_HIVE_JSON_SERDE = "org.apache.hadoop.hive.serde2.JsonSerDe"; + private static final String OPENX_JSON_SERDE = "org.openx.data.jsonserde.JsonSerDe"; private final String formatName; @@ -34,79 +77,97 @@ public enum HiveFileFormat { this.formatName = formatName; } + /** The neutral format token emitted to fe-core (mapped to a BE TFileFormatType there). */ public String getFormatName() { return formatName; } /** - * Detects the file format from the Hive inputFormat class name. + * Resolves the read format of a hive table, reproducing legacy {@code HMSExternalTable.getFileFormatType}. + * The {@code inputFormat} chooses parquet / orc / text-file; for the text-file family the {@code serDeLib} + * chooses json / csv / hive-text. The two extra flags reproduce the legacy OpenX-JSON special case: with + * {@code readHiveJsonInOneColumn} an OpenX-JSON table is read as a single CSV column (only when its first + * column is a string). Fails loud on an unrecognized inputFormat or serde, matching legacy (which threw + * rather than silently degrading). + * + * @param inputFormat the HMS {@code StorageDescriptor.inputFormat} class name + * @param serDeLib the HMS {@code SerDeInfo.serializationLib} class name + * @param readHiveJsonInOneColumn the session {@code read_hive_json_in_one_column} flag + * @param firstColumnIsString whether the table's first column is a {@code STRING} (OpenX one-column gate) + * @throws DorisConnectorException if the inputFormat/serde is unsupported, or the OpenX one-column mode is + * requested on a table whose first column is not a string */ - public static HiveFileFormat fromInputFormat(String inputFormat) { - if (inputFormat == null) { - return UNKNOWN; - } - String lower = inputFormat.toLowerCase(); - if (lower.contains("parquet")) { - return PARQUET; - } - if (lower.contains("orc")) { - return ORC; - } - if (lower.contains("text") || lower.contains("lazySimple") - || lower.contains("lazysimple")) { - return TEXT; - } - if (lower.contains("json")) { - return JSON; + public static HiveFileFormat detect(String inputFormat, String serDeLib, + boolean readHiveJsonInOneColumn, boolean firstColumnIsString) { + switch (detectFamily(inputFormat)) { + case FAMILY_PARQUET: + return PARQUET; + case FAMILY_ORC: + return ORC; + default: + // text family: the serde decides json / csv / hive-text. + return detectTextFamily(serDeLib, readHiveJsonInOneColumn, firstColumnIsString); } - // Many Hive tables use TextInputFormat as the default - if (lower.contains("textinputformat")) { - return TEXT; - } - return UNKNOWN; } /** - * Detects the file format from the SerDe library class name. + * Maps an inputFormat class name to its coarse family token ({@link #FAMILY_TEXT}/{@link #FAMILY_PARQUET}/ + * {@link #FAMILY_ORC}) by lowercase substring, first match wins in text->parquet->orc order (legacy + * {@code HiveMetaStoreClientHelper.HiveFileFormat.getFormat}). Throws on an unrecognized inputFormat, + * matching legacy's {@code "Not supported Hive file format"}. */ - public static HiveFileFormat fromSerDeLib(String serDeLib) { - if (serDeLib == null) { - return UNKNOWN; - } - if (serDeLib.contains("ParquetHiveSerDe") || serDeLib.contains("parquet")) { - return PARQUET; + private static String detectFamily(String inputFormat) { + if (inputFormat == null) { + throw new DorisConnectorException("Not supported Hive file format: null inputFormat"); } - if (serDeLib.contains("OrcSerde") || serDeLib.contains("orc")) { - return ORC; + String lower = inputFormat.toLowerCase(Locale.ROOT); + if (lower.contains(FAMILY_TEXT)) { + return FAMILY_TEXT; } - if (serDeLib.contains("LazySimpleSerDe") || serDeLib.contains("OpenCSVSerde") - || serDeLib.contains("MultiDelimitSerDe")) { - return TEXT; + if (lower.contains(FAMILY_PARQUET)) { + return FAMILY_PARQUET; } - if (serDeLib.contains("JsonSerDe") || serDeLib.contains("json")) { - return JSON; + if (lower.contains(FAMILY_ORC)) { + return FAMILY_ORC; } - return UNKNOWN; + throw new DorisConnectorException("Not supported Hive file format: " + inputFormat); } /** - * Determines the file format using both inputFormat and serDeLib, - * preferring inputFormat when available. + * Chooses the fine text-family format from the serde, reproducing the legacy serde switch verbatim, + * including the OpenX-JSON {@code read_hive_json_in_one_column} branch and the unsupported-serde throw. */ - public static HiveFileFormat detect(String inputFormat, String serDeLib) { - HiveFileFormat fromInput = fromInputFormat(inputFormat); - if (fromInput != UNKNOWN) { - return fromInput; + private static HiveFileFormat detectTextFamily(String serDeLib, + boolean readHiveJsonInOneColumn, boolean firstColumnIsString) { + if (HIVE_JSON_SERDE.equals(serDeLib) || LEGACY_HIVE_JSON_SERDE.equals(serDeLib)) { + return JSON; + } + if (OPENX_JSON_SERDE.equals(serDeLib)) { + if (!readHiveJsonInOneColumn) { + return JSON; + } + if (firstColumnIsString) { + return CSV; + } + throw new DorisConnectorException("read_hive_json_in_one_column = true, but the first column of " + + "the hive table is not a string column."); + } + if (LAZY_SIMPLE_SERDE.equals(serDeLib) + || MULTI_DELIMIT_SERDE.equals(serDeLib) + || CONTRIB_MULTI_DELIMIT_SERDE.equals(serDeLib)) { + return TEXT; + } + if (OPEN_CSV_SERDE.equals(serDeLib)) { + return CSV; } - return fromSerDeLib(serDeLib); + throw new DorisConnectorException("Unsupported hive table serde: " + serDeLib); } /** - * Returns true if this format supports file splitting at byte boundaries. - * Parquet and ORC have internal block structures that support splitting. - * Text files can be split at line boundaries. + * Whether this format supports splitting a file at byte boundaries. Parquet/ORC have internal block + * structure; text/csv are line-splittable. JSON is read whole (not split), matching the legacy behavior. */ public boolean isSplittable() { - return this == PARQUET || this == ORC || this == TEXT; + return this == PARQUET || this == ORC || this == TEXT || this == CSV; } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index f567c393e4a566..7c569fb476a1ad 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -113,7 +113,8 @@ public List planScan( } HiveFileFormat fileFormat = HiveFileFormat.detect( - hiveHandle.getInputFormat(), hiveHandle.getSerializationLib()); + hiveHandle.getInputFormat(), hiveHandle.getSerializationLib(), + readHiveJsonInOneColumn(session), hiveHandle.isFirstColumnString()); long targetSplitSize = getTargetSplitSize(session); boolean splittable = fileFormat.isSplittable(); @@ -240,7 +241,8 @@ public Map getScanNodeProperties( // File format type HiveFileFormat fileFormat = HiveFileFormat.detect( - hiveHandle.getInputFormat(), hiveHandle.getSerializationLib()); + hiveHandle.getInputFormat(), hiveHandle.getSerializationLib(), + readHiveJsonInOneColumn(session), hiveHandle.isFirstColumnString()); props.put(PROP_FILE_FORMAT_TYPE, fileFormat.getFormatName()); // Partition key column names @@ -257,8 +259,11 @@ public Map getScanNodeProperties( } } - // Text format properties (if applicable) - if (fileFormat == HiveFileFormat.TEXT || fileFormat == HiveFileFormat.JSON) { + // Text format properties (delimiters / enclose / escape / null_format / is_json) for every non-columnar + // format — TEXT (hive text serde), CSV (OpenCSV serde) and JSON. BE reads these from the hive.text.* + // props regardless of the resolved TFileFormatType, so all three families must emit them. + if (fileFormat == HiveFileFormat.TEXT || fileFormat == HiveFileFormat.CSV + || fileFormat == HiveFileFormat.JSON) { Map textProps = HiveTextProperties.extract( hiveHandle.getSerializationLib(), hiveHandle.getSdParameters(), @@ -269,6 +274,16 @@ public Map getScanNodeProperties( return props; } + /** + * Reads the {@code read_hive_json_in_one_column} session flag (default false) from the connector session — + * the same channel the write path uses for {@code hive_text_compression}. The engine dumps every visible + * session variable into {@code getSessionProperties()}, so no extra plumbing is needed. + */ + private static boolean readHiveJsonInOneColumn(ConnectorSession session) { + return Boolean.parseBoolean(session.getSessionProperties() + .getOrDefault(HiveConnectorProperties.SESSION_READ_HIVE_JSON_IN_ONE_COLUMN, "false")); + } + /** * Resolves the partitions to scan, using pruned partitions from the handle * if available, or listing all partitions from HMS. @@ -308,10 +323,12 @@ private List convertPartitions( for (int i = 0; i < partKeyNames.size() && i < values.size(); i++) { partValues.put(partKeyNames.get(i), values.get(i)); } - HiveFileFormat partFormat = HiveFileFormat.detect( - part.getInputFormat(), part.getSerializationLib()); + // Per-partition file format is not carried to BE (the whole scan node reads with the single + // table-level format resolved in planScan/getScanNodeProperties); leave it null so planScan falls + // back to the table format, matching the unpartitioned case. Detecting it per partition here would + // also fail-loud spuriously if one partition carried an unusual serde the table itself does not. result.add(new PartitionScanInfo( - part.getLocation(), partValues, partFormat)); + part.getLocation(), partValues, null)); } return result; } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java index fc9344ea52b9b9..f5ee0d1430af71 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableHandle.java @@ -52,6 +52,10 @@ public class HiveTableHandle implements ConnectorTableHandle { private final List partitionKeyNames; private final Map sdParameters; private final Map tableParameters; + // Whether the table's first column is a STRING, precomputed at handle build time (the metastore table is + // already loaded there). Reproduces legacy HMSExternalTable.firstColumnIsString, consulted ONLY for the + // OpenX-JSON read_hive_json_in_one_column read-format branch (see HiveFileFormat.detect). + private final boolean firstColumnIsString; // Set after applyFilter for partition pruning private final List prunedPartitions; @@ -72,6 +76,7 @@ private HiveTableHandle(Builder builder) { this.tableParameters = builder.tableParameters != null ? Collections.unmodifiableMap(builder.tableParameters) : Collections.emptyMap(); + this.firstColumnIsString = builder.firstColumnIsString; this.prunedPartitions = builder.prunedPartitions; } @@ -116,6 +121,14 @@ public Map getTableParameters() { return tableParameters; } + /** + * Whether the table's first column is a {@code STRING}, the gate legacy {@code HMSExternalTable} applies + * before reading an OpenX-JSON table as a single CSV column under {@code read_hive_json_in_one_column}. + */ + public boolean isFirstColumnString() { + return firstColumnIsString; + } + /** * Whether the metastore parameters mark this table transactional (ACID), replicating Hive's * {@code AcidUtils.isTransactionalTable} (case-insensitive {@code "true"} under the @@ -164,6 +177,7 @@ public Builder toBuilder() { b.partitionKeyNames = this.partitionKeyNames; b.sdParameters = this.sdParameters; b.tableParameters = this.tableParameters; + b.firstColumnIsString = this.firstColumnIsString; b.prunedPartitions = this.prunedPartitions; return b; } @@ -186,6 +200,7 @@ public static final class Builder { private List partitionKeyNames; private Map sdParameters; private Map tableParameters; + private boolean firstColumnIsString; private List prunedPartitions; public Builder(String dbName, String tableName, HiveTableType tableType) { @@ -224,6 +239,11 @@ public Builder tableParameters(Map val) { return this; } + public Builder firstColumnIsString(boolean val) { + this.firstColumnIsString = val; + return this; + } + public Builder prunedPartitions(List val) { this.prunedPartitions = val; return this; diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java index e2d56238d3f423..4f69493f7c0ec8 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java @@ -32,15 +32,23 @@ */ public final class HiveTextProperties { - // SerDe library class names + // SerDe library class names. Must stay in sync with HiveFileFormat's detection set: every serde the + // detector classifies as TEXT/CSV/JSON needs a branch here, or its text params (delimiters etc.) are + // dropped and BE reads with defaults. public static final String HIVE_TEXT_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; + // MultiDelimitSerDe exists under two package names across Hive versions; both read as FORMAT_TEXT. public static final String HIVE_MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe"; + public static final String HIVE_MULTI_DELIMIT_SERDE_SERDE2 = + "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; public static final String HIVE_OPEN_CSV_SERDE = "org.apache.hadoop.hive.serde2.OpenCSVSerde"; public static final String HIVE_JSON_SERDE = "org.apache.hive.hcatalog.data.JsonSerDe"; + // Legacy hive-2 JSON serde; legacy fe-core maps it to FORMAT_JSON like the hcatalog one. + public static final String LEGACY_HIVE_JSON_SERDE = + "org.apache.hadoop.hive.serde2.JsonSerDe"; public static final String OPENX_JSON_SERDE = "org.openx.data.jsonserde.JsonSerDe"; @@ -80,12 +88,14 @@ public static Map extract(String serDeLib, if (serDeLib == null) { return result; } - if (HIVE_TEXT_SERDE.equals(serDeLib) || HIVE_MULTI_DELIMIT_SERDE.equals(serDeLib)) { - extractTextSerDeProps(sdParams, result, - HIVE_MULTI_DELIMIT_SERDE.equals(serDeLib)); + boolean multiDelimit = HIVE_MULTI_DELIMIT_SERDE.equals(serDeLib) + || HIVE_MULTI_DELIMIT_SERDE_SERDE2.equals(serDeLib); + if (HIVE_TEXT_SERDE.equals(serDeLib) || multiDelimit) { + extractTextSerDeProps(sdParams, result, multiDelimit); } else if (HIVE_OPEN_CSV_SERDE.equals(serDeLib)) { extractCsvSerDeProps(sdParams, result); - } else if (HIVE_JSON_SERDE.equals(serDeLib) || OPENX_JSON_SERDE.equals(serDeLib)) { + } else if (HIVE_JSON_SERDE.equals(serDeLib) || LEGACY_HIVE_JSON_SERDE.equals(serDeLib) + || OPENX_JSON_SERDE.equals(serDeLib)) { extractJsonSerDeProps(serDeLib, result); } else { return result; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java index d4cfe275cf48a6..30ce7d8ba19693 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileFormatTest.java @@ -17,81 +17,123 @@ package org.apache.doris.connector.hive; +import org.apache.doris.connector.api.DorisConnectorException; + import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; /** - * Tests {@link HiveFileFormat} detection (first test for fe-connector-hive; P3-T07 batch C). + * Tests {@link HiveFileFormat} read-format detection, pinned to the legacy + * {@code HMSExternalTable.getFileFormatType} truth table. * - *

    WHY: the detected format selects which BE file reader runs (parquet/orc/text/json - * scanner). Misdetection causes read failures or silent corruption. Detection is a - * case-insensitive substring match on the InputFormat class name with a SerDe-library - * fallback — these tests pin that contract, the inputFormat-wins precedence, and the - * splittability of each format.

    + *

    WHY these assertions matter: + *

      + *
    • Detection is TWO-STAGE and serde-authoritative for the text family. A standard hive JSON table has + * {@code inputFormat=TextInputFormat} (its JSON-ness is only in the serde); an input-format-first + * classifier mis-read it as text/CSV — the headline read-correctness bug this fixes.
      • + *
    • {@code LazySimpleSerDe}/{@code MultiDelimitSerDe} -> TEXT (BE FORMAT_TEXT), {@code OpenCSVSerde} -> CSV + * (BE FORMAT_CSV_PLAIN): these are DISTINCT BE readers (the text reader honors hive collection/map + * delimiters, {@code \\N} nulls, hive escaping), so they must not collapse.
      • + *
    • Unknown inputFormat/serde fails loud (legacy parity), rather than silently degrading to a JNI read.
      • + *
    */ public class HiveFileFormatTest { + private static final String TEXT_INPUT_FORMAT = "org.apache.hadoop.mapred.TextInputFormat"; + private static final String PARQUET_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String ORC_INPUT_FORMAT = "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + + private static final String LAZY_SIMPLE_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; + private static final String MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; + private static final String CONTRIB_MULTI_DELIMIT_SERDE = + "org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe"; + private static final String OPEN_CSV_SERDE = "org.apache.hadoop.hive.serde2.OpenCSVSerde"; + private static final String HCATALOG_JSON_SERDE = "org.apache.hive.hcatalog.data.JsonSerDe"; + private static final String LEGACY_JSON_SERDE = "org.apache.hadoop.hive.serde2.JsonSerDe"; + private static final String OPENX_JSON_SERDE = "org.openx.data.jsonserde.JsonSerDe"; + private static final String ORC_SERDE = "org.apache.hadoop.hive.ql.io.orc.OrcSerde"; + + private static HiveFileFormat detect(String inputFormat, String serDeLib) { + return HiveFileFormat.detect(inputFormat, serDeLib, false, false); + } + @Test - public void testFromInputFormatDetectsByContent() { - Assertions.assertEquals(HiveFileFormat.PARQUET, - HiveFileFormat.fromInputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat")); - Assertions.assertEquals(HiveFileFormat.ORC, - HiveFileFormat.fromInputFormat("org.apache.hadoop.hive.ql.io.orc.OrcInputFormat")); - Assertions.assertEquals(HiveFileFormat.TEXT, - HiveFileFormat.fromInputFormat("org.apache.hadoop.mapred.TextInputFormat")); - Assertions.assertEquals(HiveFileFormat.JSON, - HiveFileFormat.fromInputFormat("org.apache.hadoop.hive.json.JsonInputFormat")); + public void inputFormatPicksParquetAndOrcRegardlessOfSerde() { + Assertions.assertEquals(HiveFileFormat.PARQUET, detect(PARQUET_INPUT_FORMAT, LAZY_SIMPLE_SERDE)); + Assertions.assertEquals(HiveFileFormat.ORC, detect(ORC_INPUT_FORMAT, ORC_SERDE)); } @Test - public void testFromInputFormatUnknownAndNull() { - Assertions.assertEquals(HiveFileFormat.UNKNOWN, HiveFileFormat.fromInputFormat(null)); - Assertions.assertEquals(HiveFileFormat.UNKNOWN, - HiveFileFormat.fromInputFormat("com.example.CustomInputFormat")); + public void textFamilySerdeDecidesTextVsCsvVsJson() { + // The DEFAULT hive text serde -> FORMAT_TEXT (not CSV): the regression that mis-read hive-text tables. + Assertions.assertEquals(HiveFileFormat.TEXT, detect(TEXT_INPUT_FORMAT, LAZY_SIMPLE_SERDE)); + // MultiDelimit under BOTH package names -> TEXT. + Assertions.assertEquals(HiveFileFormat.TEXT, detect(TEXT_INPUT_FORMAT, MULTI_DELIMIT_SERDE)); + Assertions.assertEquals(HiveFileFormat.TEXT, detect(TEXT_INPUT_FORMAT, CONTRIB_MULTI_DELIMIT_SERDE)); + // OpenCSV -> CSV (FORMAT_CSV_PLAIN). + Assertions.assertEquals(HiveFileFormat.CSV, detect(TEXT_INPUT_FORMAT, OPEN_CSV_SERDE)); } @Test - public void testFromSerDeLib() { - Assertions.assertEquals(HiveFileFormat.PARQUET, - HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe")); - Assertions.assertEquals(HiveFileFormat.ORC, - HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.ql.io.orc.OrcSerde")); - Assertions.assertEquals(HiveFileFormat.TEXT, - HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe")); - Assertions.assertEquals(HiveFileFormat.TEXT, - HiveFileFormat.fromSerDeLib("org.apache.hadoop.hive.serde2.OpenCSVSerde")); + public void jsonTableWithTextInputFormatResolvesToJsonNotText() { + // THE HEADLINE BUG: a JSON table's inputFormat is TextInputFormat; its JSON-ness lives only in the + // serde. Detection must consult the serde and return JSON, not text/CSV. + Assertions.assertEquals(HiveFileFormat.JSON, detect(TEXT_INPUT_FORMAT, HCATALOG_JSON_SERDE)); + Assertions.assertEquals(HiveFileFormat.JSON, detect(TEXT_INPUT_FORMAT, LEGACY_JSON_SERDE)); + // OpenX JSON with the one-column mode OFF (default) -> JSON. + Assertions.assertEquals(HiveFileFormat.JSON, detect(TEXT_INPUT_FORMAT, OPENX_JSON_SERDE)); + } + + @Test + public void openxJsonOneColumnModeReadsAsCsvWhenFirstColumnIsString() { + // read_hive_json_in_one_column=true + first column is string -> the whole row is read as one CSV column. + Assertions.assertEquals(HiveFileFormat.CSV, + HiveFileFormat.detect(TEXT_INPUT_FORMAT, OPENX_JSON_SERDE, true, true)); + // Non-OpenX json serdes ignore the flag (stay JSON). Assertions.assertEquals(HiveFileFormat.JSON, - HiveFileFormat.fromSerDeLib("org.apache.hive.hcatalog.data.JsonSerDe")); - Assertions.assertEquals(HiveFileFormat.UNKNOWN, HiveFileFormat.fromSerDeLib(null)); + HiveFileFormat.detect(TEXT_INPUT_FORMAT, HCATALOG_JSON_SERDE, true, true)); } @Test - public void testDetectPrefersInputFormatThenFallsBackToSerDe() { - // inputFormat wins when recognized (even if the SerDe says otherwise)... - Assertions.assertEquals(HiveFileFormat.PARQUET, - HiveFileFormat.detect( - "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat", - "org.apache.hadoop.hive.ql.io.orc.OrcSerde")); - // ...and the SerDe is the fallback when the inputFormat is unrecognized. - Assertions.assertEquals(HiveFileFormat.TEXT, - HiveFileFormat.detect("com.example.CustomInputFormat", - "org.apache.hadoop.hive.serde2.OpenCSVSerde")); + public void openxJsonOneColumnModeFailsLoudWhenFirstColumnNotString() { + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> HiveFileFormat.detect(TEXT_INPUT_FORMAT, OPENX_JSON_SERDE, true, false)); + Assertions.assertTrue(e.getMessage().contains("read_hive_json_in_one_column"), + "message should name the offending session flag"); } @Test - public void testIsSplittable() { - Assertions.assertTrue(HiveFileFormat.PARQUET.isSplittable()); - Assertions.assertTrue(HiveFileFormat.ORC.isSplittable()); - Assertions.assertTrue(HiveFileFormat.TEXT.isSplittable()); - Assertions.assertFalse(HiveFileFormat.JSON.isSplittable()); - Assertions.assertFalse(HiveFileFormat.UNKNOWN.isSplittable()); + public void unknownSerdeInTextFamilyFailsLoud() { + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> detect(TEXT_INPUT_FORMAT, "com.example.WeirdSerDe")); + Assertions.assertTrue(e.getMessage().contains("Unsupported hive table serde"), + "unknown serde must fail loud (legacy parity), not silently degrade to JNI"); + } + + @Test + public void unknownInputFormatFailsLoud() { + Assertions.assertThrows(DorisConnectorException.class, + () -> detect("com.example.CustomInputFormat", LAZY_SIMPLE_SERDE)); + Assertions.assertThrows(DorisConnectorException.class, + () -> detect(null, LAZY_SIMPLE_SERDE)); } @Test - public void testFormatName() { + public void formatNameMatchesGenericVocabularyToken() { Assertions.assertEquals("parquet", HiveFileFormat.PARQUET.getFormatName()); Assertions.assertEquals("orc", HiveFileFormat.ORC.getFormatName()); Assertions.assertEquals("text", HiveFileFormat.TEXT.getFormatName()); + Assertions.assertEquals("csv", HiveFileFormat.CSV.getFormatName()); Assertions.assertEquals("json", HiveFileFormat.JSON.getFormatName()); } + + @Test + public void splittability() { + Assertions.assertTrue(HiveFileFormat.PARQUET.isSplittable()); + Assertions.assertTrue(HiveFileFormat.ORC.isSplittable()); + Assertions.assertTrue(HiveFileFormat.TEXT.isSplittable()); + Assertions.assertTrue(HiveFileFormat.CSV.isSplittable()); + Assertions.assertFalse(HiveFileFormat.JSON.isSplittable()); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index dd4ddad8c721e8..7e0ad0c36efc83 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -1477,6 +1477,9 @@ private static TFileFormatType mapFileFormatType(String format) { case "orc": return TFileFormatType.FORMAT_ORC; case "text": + // Hive text serde family (LazySimpleSerDe / MultiDelimitSerDe): the BE text reader honors hive + // collection/map delimiters, \N nulls and hive escaping — distinct from the flat CSV reader. + return TFileFormatType.FORMAT_TEXT; case "csv": return TFileFormatType.FORMAT_CSV_PLAIN; case "json": From 1ba830768053d041d440c7b83f00ccf32d4ed77c Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 21:10:58 +0800 Subject: [PATCH 050/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20file-format=20serde=20fix=20DONE;=20next=20=3D=20listPart?= =?UTF-8?q?itions?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the file-format read-correctness fix (c320df27712) — serde-authoritative detection fixing JSON-read-as-CSV + the FORMAT_TEXT/CSV/JSON distinction + OpenX one-column + fail-loud, plus the atomic fe-core mapFileFormatType text/csv split — and re-point the next substep to listPartitions. Recon: wf_470a2e51-373. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4f978d51146f7a..4839a43be854ad 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -42,8 +42,8 @@ - ⛔**NESTED_PRUNE 遗留（硬门，下一子步前置验证）**：同接缝，连接器**暂未发射** marker——`SlotTypeReplacer.java:678-680` 把嵌套访问路径重写成 `Column.getUniqueId()`，iceberg 靠自带 field-id（`withUniqueId`）正确，**hive 连接器现在没设 field-id**；对 hive 开启前须端到端验证 hive orc/parquet 的 uniqueId 语义（BE 按 id 匹配嵌套叶子）否则读 NULL。旧 hive nested-prune 走同一 `SlotTypeReplacer` 且能工作→查旧 hive column uniqueId 赋值 vs SPI 路径 `ConnectorColumnConverter.convertColumns`；验证通过则连接器发 `SUPPORTS_NESTED_COLUMN_PRUNE` marker（parquet/orc、非 hudi），否则记差异 fail-loud。 - **✅ §4.2 `getTableStatistics` 三层全做 DONE**（`eded05b05e6`+`8c6d247bb34`+`453a1926010`；用户拍板"三层全做"，不推迟）。架构裁决：**连接器只回传原始事实、不做 Doris 类型运算**（不 import fe-type），`dataSize/rowWidth` 除法归 fe-core（它有类型信息且此逻辑 connector-agnostic，对现有 iceberg/paimon/jdbc 零影响——它们 dataSize 恒 -1）。三层 = ① 精确 `numRows`（→spark numRows，仅当 numRows 键在但≤0，忠实复刻旧不对称）② `totalSize`（→spark totalSize，标准键缺才回退）作 dataSize，fe-core 除**全列宽** ③ 列文件估算（新 SPI `estimateDataSizeByListingFiles`，只普通 hive、采样 30 分区放大、best-effort→-1、内部 pin TCCL 因统计线程 fe-core 不 pin），fe-core 除**非分区列宽**（旧 getRowCountFromFileList 语义）。**复审修正**（4 维对抗 + 逐条反驳核验 `wf_e0f3a369-77d`）：两层商截 0 须塌成 UNKNOWN（层2→穿透层3、层3→UNKNOWN），非报"空表 0 行"。**已签字裁决（本 session）**：采样数不再随运行时 `hive_stats_partition_sample_size` 变（连接器持默认 30；开关 `enable_get_row_count_from_file_list` 仍 fe-core 侧生效）；malformed `numRows` 防御式恢复 spark 值（比旧 throw 更稳，已文档化 deviation）；文件列举非递归顶层（与扫描侧一致）；精确 view-skip 待 view SPI（今 view 无可列位置→-1）。 - **✅ §4.2 视图 SPI DONE（`6676bf365cf`）**（本轮侦察 `wf_ecb9c379-9bb`：4 维读者 + 完整性 critic，全 HEAD 核对）。连接器侧四件套：`HmsTableInfo`+`ThriftHmsClient.convertTable` 补 `viewOriginalText`/`viewExpandedText`（原被丢弃）；`viewExists`=**legacy `HMSExternalTable.isView` 谓词=view 文本存在**（非 tableType——base 表须返 false，否则 `PluginDrivenExternalCatalog.dropTable` **无条件** viewExists→dropView 会误路由 base 表 DROP）；`getViewDefinition`=忠实复刻 `getViewText`（expanded-first→跳 `/* Presto View */` sentinel→`viewOriginalText` base64/Gson 解 `originalSql`→任何失败回退 raw；dialect 占位不被读；列走 `buildColumns`，**fe-core 视图列只来自此处**——`initSchema` isView 早返、不调 getTableSchema）；`dropView`=复用 `HmsClient.dropTable`（hive 无独立 drop-view）、`DorisConnectorException` 归一化好让 fe-core 转 `DdlException`；`listViewNames` **保持 SPI 默认 empty**（hive `listTableNames`=getAllTables 已含视图，fe-core SHOW TABLES 合并是无去重 `addAll`，非空会双列；与 iceberg 相反）；`HiveConnector.getCapabilities`=仅 `SUPPORTS_VIEW`（其余连接器级能力各自子步、MVCC 故意不声明）。全绿（7 视图测 + hive 157 + hms 150 套无回归 + checkstyle 0 + import gate 净）。**留到翻闸的耦合（休眠期不暴露，勿现在动）见文末**。 -- **下一步 = §4.2 file-format serde 修正（读正确性 bug，H）**：现连接器 `HiveFileFormat.detect` inputFormat-first→**JSON 表当 CSV 读**（标准 hive JSON 表 inputFormat=TextInputFormat，serde 从不被查）；FORMAT_TEXT/FORMAT_CSV_PLAIN 塌缩；UNKNOWN 静默 FORMAT_JNI（旧 throw）。recon 建议 **REPRODUCE**：scan 侧 text 分支改 serde-authoritative（精确 equals）、补 `read_hive_json_in_one_column`（`ConnectorSession.getSessionProperties` 已有 key + 需首列类型 plumb，`HiveTableHandle` 现无列访问器）、UNKNOWN fail-loud（照 `IcebergScanPlanProvider:783-787`）。⚠FORMAT_TEXT 在连接器/通用节点如何表达需一次决策（通用 `PluginDrivenScanNode.mapFileFormatType:1473-1491` 无 FORMAT_TEXT 分支）。 - - 之后续做：**`listPartitions`（含 `lastModifiedMillis`）** 镜像 paimon `collectPartitions`（`lastModifiedMillis` 唯一影响 MTMV `PluginDrivenMvccExternalTable.listLatestPartitions:269`；旧源 `transient_lastDdlTime*1000` 需 per-partition `getParameters()` 往返——决策=付这往返 vs 接受 -1；**不**声明 `SUPPORTS_PARTITION_STATS`）；**`getCapabilities` 其余连接器级项**（`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；禁 `SUPPORTS_MVCC_SNAPSHOT`/`SUPPORTS_PASSTHROUGH_QUERY`/`SUPPORTS_PARTITION_STATS`）；**`HiveTableFormatDetector` 平价**（补 3 个 LZO text 格式 `HMSExternalTable:170-172`、UNKNOWN 改 throw、view 短路）。 +- **✅ §4.2 file-format serde 修正 DONE（`c320df27712`）**（侦察 `wf_470a2e51-373`：4 维 + critic，含 BE C++ 读取器核对）。修**两个**读正确性 bug:(1) `HiveFileFormat.detect` 原 inputFormat-first→标准 JSON 表(inputFormat=TextInputFormat)被当 CSV 读;(2) 默认 hive 文本表(LazySimpleSerDe)本该走 BE `FORMAT_TEXT` 读取器(懂 hive 集合/map 分隔、`\N` 空、转义),却被塌缩成 `FORMAT_CSV_PLAIN`(BE 确有两个不同读取器类)。修法=**两段式、文本类 serde-authoritative**(inputFormat substring 分 parquet/orc/text-file;text 类按**精确 FQCN** serde 分 json/csv/hive-text),新增 CSV 枚举;复刻 OpenX `read_hive_json_in_one_column`(首列 string→单列 CSV,否则 fail-loud;会话开关经 `getSessionProperties` 读,首列类型 plumb 进 `HiveTableHandle`);未知 inputFormat/serde **fail-loud throw**(用户签字,与线上一致)。**fe-core `PluginDrivenScanNode.mapFileFormatType` 拆 `"text"→FORMAT_TEXT`、`"csv"→FORMAT_CSV_PLAIN`**(中立既有 token、无 `if(engine)`、hive 是唯一发射者、休眠;须与连接器 csv 发射**同一 commit**否则 OpenCSV 回退)。`HiveTextProperties` 补 legacy hive-2 JsonSerDe + serde2 MultiDelimit FQCN。全绿(HiveFileFormatTest 真值表 + hive 153/hms 150 无回归 + fe-core 编译 + checkstyle 0 + import gate 净)。 +- **下一步 = §4.2 `listPartitions`(item 3 below)**;其后 `getCapabilities` 其余项(item 4)、`HiveTableFormatDetector` 平价(item 5)。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 4. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 From af371d7b77a64da51f0476576535cbb5c1fac23f Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 22:20:41 +0800 Subject: [PATCH 051/334] [fix](catalog) hms client: -1 maxParts = unbounded partition listing (no silent 32767 cap) ThriftHmsClient.listPartitionNames mapped a non-positive maxParts to Short.MAX_VALUE (32767) and passed it to HMS get_partition_names, hard-capping the result at 32767 partitions. Legacy ThriftHMSCachedClient passed the Config.max_hive_list_partition_num default of -1 straight through, which HMS treats as unbounded ("all"). So a table with >32767 partitions was silently truncated. This already defeated two existing dormant callers that pass -1 intending "all": HiveWritePlanProvider.buildExistingPartitions and HudiConnectorMetadata (the latter carries an explicit "no silent partition truncation" comment). The scan/prune callers pass 100000, which narrows to a negative short (-31072) and is therefore already unbounded -- so the connector was internally inconsistent (full scan universe vs capped listing). Fix: a non-positive maxParts now maps to (short) -1 (unbounded), extracted into a testable static toThriftMaxParts helper. Positive values pass through, narrowing to short -- a value above Short.MAX_VALUE narrows to a negative short (unbounded), which preserves the pre-existing behavior of the 100000-cap callers. No behavior change for any positive caller; the -1 callers now get the "all" listing they always intended. Test: ThriftHmsClientMaxPartsTest pins the mapping (-1/0 -> unbounded, positive pass-through, >Short.MAX_VALUE -> negative/unbounded) without a live metastore. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hms/ThriftHmsClient.java | 23 +++++++- .../hms/ThriftHmsClientMaxPartsTest.java | 59 +++++++++++++++++++ 2 files changed, 79 insertions(+), 3 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientMaxPartsTest.java diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 409a2adb5de925..b6c90fa81a83e0 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -87,7 +87,6 @@ public class ThriftHmsClient implements HmsClient { private static final Logger LOG = LogManager.getLogger(ThriftHmsClient.class); private static final HiveMetaHookLoader DUMMY_HOOK_LOADER = tbl -> null; - private static final int MAX_LIST_PARTITION_NUM = Short.MAX_VALUE; private static final long POOL_BORROW_TIMEOUT_MS = 60_000L; private static final int ADD_PARTITIONS_BATCH_SIZE = 20; private static final String TRANSIENT_LAST_DDL_TIME = "transient_lastDdlTime"; @@ -204,10 +203,28 @@ public Map getDefaultColumnValues(String dbName, @Override public List listPartitionNames(String dbName, String tableName, int maxParts) { - int limit = maxParts <= 0 ? MAX_LIST_PARTITION_NUM : maxParts; return execute( client -> client.listPartitionNames(dbName, tableName, - (short) limit)); + toThriftMaxParts(maxParts))); + } + + /** + * Maps the connector's {@code maxParts} contract onto the {@code short max_parts} that HMS + * {@code get_partition_names} accepts. + * + *

    A non-positive {@code maxParts} means "all partitions" and maps to {@code -1}: HMS treats a + * negative {@code max_parts} as unbounded. It must NOT be clamped to {@code Short.MAX_VALUE} — that + * silently truncates a table with more than 32767 partitions and defeats the {@code -1} "unlimited" + * contract the hive/hudi callers rely on (the legacy {@code ThriftHMSCachedClient} passed the + * {@code Config.max_hive_list_partition_num} default of {@code -1} straight through to HMS, i.e. + * unbounded).

    + * + *

    A positive value is passed through, narrowing to {@code short}; a value above + * {@code Short.MAX_VALUE} narrows to a negative {@code short}, which HMS likewise treats as unbounded — + * the pre-existing behavior of the callers that request up to {@code 100000} partitions.

    + */ + static short toThriftMaxParts(int maxParts) { + return maxParts <= 0 ? (short) -1 : (short) maxParts; } @Override diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientMaxPartsTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientMaxPartsTest.java new file mode 100644 index 00000000000000..2231451a48527e --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientMaxPartsTest.java @@ -0,0 +1,59 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Tests {@link ThriftHmsClient#toThriftMaxParts}: the connector's {@code maxParts} contract mapped onto the + * {@code short max_parts} that HMS {@code get_partition_names} accepts. + * + *

    WHY: a non-positive {@code maxParts} means "all partitions" and MUST map to a negative short (HMS reads + * a negative {@code max_parts} as unbounded). A prior implementation clamped it to {@code Short.MAX_VALUE} + * (32767), which silently truncated any table with more than 32767 partitions — defeating the {@code -1} + * "unlimited" contract the hive/hudi partition-listing callers rely on and diverging from the legacy client, + * which passed {@code Config.max_hive_list_partition_num = -1} straight through. These assertions pin the + * unbounded mapping so the truncation cannot silently return.

    + */ +public class ThriftHmsClientMaxPartsTest { + + @Test + public void testNonPositiveMeansUnbounded() { + // -1 and 0 both mean "all"; HMS reads a negative short as unbounded. + Assertions.assertEquals((short) -1, ThriftHmsClient.toThriftMaxParts(-1)); + Assertions.assertEquals((short) -1, ThriftHmsClient.toThriftMaxParts(0)); + // Must NOT be the old silent cap. + Assertions.assertNotEquals(Short.MAX_VALUE, ThriftHmsClient.toThriftMaxParts(-1)); + } + + @Test + public void testPositiveWithinShortIsPassedThrough() { + Assertions.assertEquals((short) 1, ThriftHmsClient.toThriftMaxParts(1)); + Assertions.assertEquals((short) 100, ThriftHmsClient.toThriftMaxParts(100)); + Assertions.assertEquals(Short.MAX_VALUE, ThriftHmsClient.toThriftMaxParts(Short.MAX_VALUE)); + } + + @Test + public void testPositiveAboveShortNarrowsToUnbounded() { + // The 100000-cap callers rely on this: (short) 100000 is negative, so HMS treats it as unbounded. + short mapped = ThriftHmsClient.toThriftMaxParts(100000); + Assertions.assertEquals((short) 100000, mapped); + Assertions.assertTrue(mapped < 0, "a value above Short.MAX_VALUE must narrow to a negative (unbounded) short"); + } +} From 29b1f82fb9ef81d4c5be7607eadd4bfd0282a440 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 22:21:01 +0800 Subject: [PATCH 052/334] [feat](catalog) hive connector listPartitions/listPartitionNames (dormant) Implements the ConnectorTableOps partition-listing SPI on HiveConnectorMetadata so a flipped hive table's fe-core partition view (PluginDrivenMvccExternalTable. materializeLatest / getNameToPartitionItems), SHOW PARTITIONS names branch and MetadataGenerator resolve real partitions instead of degrading to empty. - listPartitionNames / listPartitions share a collectPartitionNames helper that lists the metastore's rendered partition names (get_partition_names, -1 = all). An unpartitioned table lists nothing without touching the metastore (mirrors paimon's collectPartitions guard). - lastModifiedMillis (and rowCount/sizeBytes/fileCount) are left UNKNOWN(-1) by design (user sign-off): legacy's hot partition-pruning path listed names ONLY -- HiveExternalMetaCache.loadPartitionValues never did the per-partition get_partitions_by_names round-trip; the per-partition transient_lastDdlTime was fetched only at MTMV-refresh time. Filling it in listPartitions (run on every partitioned-hive query) would regress the hot path. Per-partition MTMV freshness is rewired connector-side in the later MVCC/MTMV step. maxcompute precedent also emits -1; hive declares no SUPPORTS_PARTITION_STATS so the -1 stats never surface (SHOW PARTITIONS takes the names-only branch). - The filter is ignored (legacy materialized the full set and pruned FE-side). - Partition values are decoded via HiveWriteUtils.toPartitionValues (byte-faithful port of legacy HiveUtil.toPartitionValues, incl. path unescaping) and keyed by remote partition-column name, which is how PluginDrivenExternalTable.getNameToPartitionItems reads them back. Dormant: hive is not in SPI_READY_TYPES, so this is unreachable online today. Verified: 3-lens adversarial review (legacy-parity / flip-consumer trace / code-edge) converged on a single defect -- the -1 partition cap -- fixed in the preceding ThriftHmsClient commit; value-decoding byte-parity, iron rules, and no-NPE-from-(-1) confirmed. fe-connector-hive 160 tests + 7 new partition tests green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 80 ++++++ ...iveConnectorMetadataPartitionListTest.java | 232 ++++++++++++++++++ 2 files changed, 312 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 7ae7ce0270ece7..31a6380d70b6e2 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; @@ -713,6 +714,85 @@ public Optional> applyFilter( newHandle, constraint.getExpression(), false)); } + // ========== ConnectorTableOps: partition listing ========== + + /** + * Lists a partitioned table's partition display names (e.g. {@code "year=2024/month=01"}), taken + * straight from the metastore's {@code get_partition_names}. Byte-parity with legacy + * {@code HiveExternalMetaCache.loadPartitionValues}, whose hot partition-pruning path listed NAMES ONLY + * (no per-partition metadata round-trip). An unpartitioned table lists nothing (the metastore has no + * partitions; the guard mirrors {@code PaimonConnectorMetadata.collectPartitions}, avoiding a pointless + * RPC). + */ + @Override + public List listPartitionNames(ConnectorSession session, ConnectorTableHandle handle) { + return collectPartitionNames((HiveTableHandle) handle); + } + + /** + * Lists all partitions with metadata. The {@code filter} is intentionally ignored: legacy hive + * materialized its full partition view and pruned FE-side (mirrors {@code PaimonConnectorMetadata} / + * {@code MaxComputeConnectorMetadata}). + * + *

    {@code lastModifiedMillis} is deliberately left {@link ConnectorPartitionInfo#UNKNOWN} (-1): + * reading each partition's {@code transient_lastDdlTime} requires a {@code get_partitions_by_names} + * round-trip that legacy's per-query partition-view path did NOT pay (it read only partition names), + * so filling it here would regress every partitioned-hive query. Legacy fetched per-partition modify + * time only at MTMV-refresh time; that freshness path is rewired connector-side in the MVCC/MTMV step + * (until then a hive MTMV base table's per-partition freshness is UNKNOWN, harmless while hive is + * dormant). {@code rowCount}/{@code sizeBytes}/{@code fileCount} are likewise UNKNOWN — hive does not + * declare {@code SUPPORTS_PARTITION_STATS} (legacy SHOW PARTITIONS lists names only).

    + */ + @Override + public List listPartitions(ConnectorSession session, + ConnectorTableHandle handle, Optional filter) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + List partKeyNames = hiveHandle.getPartitionKeyNames(); + List partitionNames = collectPartitionNames(hiveHandle); + List result = new ArrayList<>(partitionNames.size()); + for (String partitionName : partitionNames) { + result.add(new ConnectorPartitionInfo(partitionName, + toPartitionValueMap(partitionName, partKeyNames), + Collections.emptyMap())); + } + return result; + } + + /** + * Shared partition-name lister backing {@link #listPartitionNames} and {@link #listPartitions}. Returns + * the metastore's rendered partition names ({@code key=value/...}); an unpartitioned table (no partition + * keys) lists nothing without touching the metastore. + */ + private List collectPartitionNames(HiveTableHandle handle) { + List partKeyNames = handle.getPartitionKeyNames(); + if (partKeyNames == null || partKeyNames.isEmpty()) { + return Collections.emptyList(); + } + // -1 = "all partitions": ThriftHmsClient maps it to an unbounded HMS listing (no silent cap), + // matching legacy's default (Config.max_hive_list_partition_num = -1). + return hmsClient.listPartitionNames(handle.getDbName(), handle.getTableName(), -1); + } + + /** + * Parses a rendered partition name ({@code key1=v1/key2=v2}) into a remote-key -> value map, unescaping + * each value via {@link HiveWriteUtils#toPartitionValues} (the byte-faithful port of legacy + * {@code HiveUtil.toPartitionValues}). Keyed by the handle's remote partition-column names in schema + * order, which is how {@code PluginDrivenExternalTable.getNameToPartitionItems} reads the values back. + * Returns an empty map when the parsed value arity does not match the partition-key arity (defensive; a + * malformed name is logged-and-skipped by the fe-core partition-item builder). + */ + private static Map toPartitionValueMap(String partitionName, List partKeyNames) { + List values = HiveWriteUtils.toPartitionValues(partitionName); + if (partKeyNames == null || values.size() != partKeyNames.size()) { + return Collections.emptyMap(); + } + Map valueMap = new LinkedHashMap<>(); + for (int i = 0; i < partKeyNames.size(); i++) { + valueMap.put(partKeyNames.get(i), values.get(i)); + } + return valueMap; + } + // ========== ConnectorSchemaOps: DDL writes (create/drop database) ========== /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java new file mode 100644 index 00000000000000..bd890732be956e --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java @@ -0,0 +1,232 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorPartitionInfo; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HiveConnectorMetadata#listPartitions} / {@link HiveConnectorMetadata#listPartitionNames} + * (HMS cutover §4.2, dormant). + * + *

    WHY these assertions matter:

    + *
      + *
    • Names-only, no per-partition round-trip. The single most important invariant: listing + * partitions must call {@code get_partition_names} ONLY and never {@code get_partitions_by_names}. + * Legacy's hot partition-pruning path ({@code HiveExternalMetaCache.loadPartitionValues}) listed names + * only; paying the heavier per-partition fetch here would regress every partitioned-hive query. The + * fake fails loud if {@code getPartitions} is touched.
      • + *
    • {@code lastModifiedMillis} and the stat fields are UNKNOWN(-1). This is the deliberate + * §4.2 decision (freshness deferred to the MVCC/MTMV step); a regression that silently started filling + * them would re-introduce the round-trip this method exists to avoid.
      • + *
    • Value maps are keyed by remote partition-column name and unescaped. + * {@code PluginDrivenExternalTable.getNameToPartitionItems} reads values back by remote name, and + * legacy decoded Hive path-escaping ({@code %2F} -> {@code /}); getting either wrong corrupts the + * partition-value view.
      • + *
    • Unpartitioned tables list nothing without any metastore call (parity guard).
      • + *
    • The filter is ignored (legacy materialized the full set and pruned FE-side).
      • + *
    + */ +public class HiveConnectorMetadataPartitionListTest { + + private static final List PARTITIONS = Arrays.asList( + "year=2023/month=12", + "year=2024/month=01", + "year=2024/month=02"); + + private static final List PART_KEYS = Arrays.asList("year", "month"); + + private HiveConnectorMetadata metadata(FakeHmsClient client) { + return new HiveConnectorMetadata(client, Collections.emptyMap(), new FakeConnectorContext()); + } + + private HiveTableHandle partitionedHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(PART_KEYS) + .build(); + } + + @Test + public void testListPartitionNames() { + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + List names = metadata(client).listPartitionNames(null, partitionedHandle()); + Assertions.assertEquals(PARTITIONS, names); + // The connector must request ALL partitions via the -1 "unbounded" sentinel; requesting a finite + // cap would silently truncate a >32767-partition table. That -1 maps to an unbounded HMS listing + // (not Short.MAX_VALUE) is pinned separately by ThriftHmsClientMaxPartsTest. + Assertions.assertEquals(-1, client.lastMaxParts); + Assertions.assertFalse(client.getPartitionsCalled, + "listPartitionNames must not touch get_partitions_by_names"); + } + + @Test + public void testListPartitionsNamesAndValues() { + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + List parts = + metadata(client).listPartitions(null, partitionedHandle(), Optional.empty()); + + Assertions.assertEquals(3, parts.size()); + ConnectorPartitionInfo first = parts.get(0); + Assertions.assertEquals("year=2023/month=12", first.getPartitionName()); + Map values = first.getPartitionValues(); + Assertions.assertEquals(2, values.size()); + Assertions.assertEquals("2023", values.get("year")); + Assertions.assertEquals("12", values.get("month")); + } + + @Test + public void testListPartitionsLeavesStatsUnknown() { + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + List parts = + metadata(client).listPartitions(null, partitionedHandle(), Optional.empty()); + for (ConnectorPartitionInfo part : parts) { + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, part.getLastModifiedMillis(), + "lastModifiedMillis must stay UNKNOWN (freshness deferred to the MVCC/MTMV step)"); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, part.getRowCount()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, part.getSizeBytes()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, part.getFileCount()); + } + } + + @Test + public void testListPartitionsNeverFetchesPerPartitionMetadata() { + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + metadata(client).listPartitions(null, partitionedHandle(), Optional.empty()); + Assertions.assertFalse(client.getPartitionsCalled, + "listPartitions must list names only, not get_partitions_by_names (hot-path parity)"); + } + + @Test + public void testEscapedPartitionValueIsUnescaped() { + // A Hive partition value containing '/' is path-escaped as %2F in the partition name; the value + // map must decode it (byte-parity with legacy HiveUtil.toPartitionValues). + FakeHmsClient client = new FakeHmsClient(Collections.singletonList("year=2024/month=a%2Fb")); + List parts = + metadata(client).listPartitions(null, partitionedHandle(), Optional.empty()); + Assertions.assertEquals("a/b", parts.get(0).getPartitionValues().get("month")); + } + + @Test + public void testFilterIsIgnored() { + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + ConnectorExpression filter = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("year", org.apache.doris.connector.api.ConnectorType.of("STRING")), + new ConnectorLiteral(org.apache.doris.connector.api.ConnectorType.of("STRING"), "2024")); + List parts = + metadata(client).listPartitions(null, partitionedHandle(), Optional.of(filter)); + // Full set returned despite the predicate: pruning is a separate applyFilter concern. + Assertions.assertEquals(3, parts.size()); + } + + @Test + public void testUnpartitionedTableListsNothingWithoutMetastoreCall() { + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.emptyList()) + .build(); + HiveConnectorMetadata metadata = metadata(client); + Assertions.assertTrue(metadata.listPartitionNames(null, handle).isEmpty()); + Assertions.assertTrue(metadata.listPartitions(null, handle, Optional.empty()).isEmpty()); + Assertions.assertFalse(client.listPartitionNamesCalled, + "an unpartitioned table must not call the metastore"); + } + + /** + * Minimal {@link HmsClient} double: {@code listPartitionNames} returns a fixed list and records the + * requested {@code maxParts}; {@code getPartitions} fails loud (the per-partition round-trip this path + * must never make). The rest are unsupported. + */ + private static final class FakeHmsClient implements HmsClient { + private final List partitionNames; + private boolean listPartitionNamesCalled; + private boolean getPartitionsCalled; + private int lastMaxParts; + + FakeHmsClient(List partitionNames) { + this.partitionNames = partitionNames; + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + listPartitionNamesCalled = true; + lastMaxParts = maxParts; + return partitionNames; + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + getPartitionsCalled = true; + throw new AssertionError("get_partitions_by_names must not be called by partition listing"); + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} From cb9a12f705b648562f5f7ae729d6a900793e5578 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 22:22:55 +0800 Subject: [PATCH 053/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20listPartitions=20DONE=20+=20hms=20-1=3Dunbounded=20fix;?= =?UTF-8?q?=20next=20=3D=20getCapabilities?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4839a43be854ad..c61a9095b402da 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -43,9 +43,11 @@ - **✅ §4.2 `getTableStatistics` 三层全做 DONE**（`eded05b05e6`+`8c6d247bb34`+`453a1926010`；用户拍板"三层全做"，不推迟）。架构裁决：**连接器只回传原始事实、不做 Doris 类型运算**（不 import fe-type），`dataSize/rowWidth` 除法归 fe-core（它有类型信息且此逻辑 connector-agnostic，对现有 iceberg/paimon/jdbc 零影响——它们 dataSize 恒 -1）。三层 = ① 精确 `numRows`（→spark numRows，仅当 numRows 键在但≤0，忠实复刻旧不对称）② `totalSize`（→spark totalSize，标准键缺才回退）作 dataSize，fe-core 除**全列宽** ③ 列文件估算（新 SPI `estimateDataSizeByListingFiles`，只普通 hive、采样 30 分区放大、best-effort→-1、内部 pin TCCL 因统计线程 fe-core 不 pin），fe-core 除**非分区列宽**（旧 getRowCountFromFileList 语义）。**复审修正**（4 维对抗 + 逐条反驳核验 `wf_e0f3a369-77d`）：两层商截 0 须塌成 UNKNOWN（层2→穿透层3、层3→UNKNOWN），非报"空表 0 行"。**已签字裁决（本 session）**：采样数不再随运行时 `hive_stats_partition_sample_size` 变（连接器持默认 30；开关 `enable_get_row_count_from_file_list` 仍 fe-core 侧生效）；malformed `numRows` 防御式恢复 spark 值（比旧 throw 更稳，已文档化 deviation）；文件列举非递归顶层（与扫描侧一致）；精确 view-skip 待 view SPI（今 view 无可列位置→-1）。 - **✅ §4.2 视图 SPI DONE（`6676bf365cf`）**（本轮侦察 `wf_ecb9c379-9bb`：4 维读者 + 完整性 critic，全 HEAD 核对）。连接器侧四件套：`HmsTableInfo`+`ThriftHmsClient.convertTable` 补 `viewOriginalText`/`viewExpandedText`（原被丢弃）；`viewExists`=**legacy `HMSExternalTable.isView` 谓词=view 文本存在**（非 tableType——base 表须返 false，否则 `PluginDrivenExternalCatalog.dropTable` **无条件** viewExists→dropView 会误路由 base 表 DROP）；`getViewDefinition`=忠实复刻 `getViewText`（expanded-first→跳 `/* Presto View */` sentinel→`viewOriginalText` base64/Gson 解 `originalSql`→任何失败回退 raw；dialect 占位不被读；列走 `buildColumns`，**fe-core 视图列只来自此处**——`initSchema` isView 早返、不调 getTableSchema）；`dropView`=复用 `HmsClient.dropTable`（hive 无独立 drop-view）、`DorisConnectorException` 归一化好让 fe-core 转 `DdlException`；`listViewNames` **保持 SPI 默认 empty**（hive `listTableNames`=getAllTables 已含视图，fe-core SHOW TABLES 合并是无去重 `addAll`，非空会双列；与 iceberg 相反）；`HiveConnector.getCapabilities`=仅 `SUPPORTS_VIEW`（其余连接器级能力各自子步、MVCC 故意不声明）。全绿（7 视图测 + hive 157 + hms 150 套无回归 + checkstyle 0 + import gate 净）。**留到翻闸的耦合（休眠期不暴露，勿现在动）见文末**。 - **✅ §4.2 file-format serde 修正 DONE（`c320df27712`）**（侦察 `wf_470a2e51-373`：4 维 + critic，含 BE C++ 读取器核对）。修**两个**读正确性 bug:(1) `HiveFileFormat.detect` 原 inputFormat-first→标准 JSON 表(inputFormat=TextInputFormat)被当 CSV 读;(2) 默认 hive 文本表(LazySimpleSerDe)本该走 BE `FORMAT_TEXT` 读取器(懂 hive 集合/map 分隔、`\N` 空、转义),却被塌缩成 `FORMAT_CSV_PLAIN`(BE 确有两个不同读取器类)。修法=**两段式、文本类 serde-authoritative**(inputFormat substring 分 parquet/orc/text-file;text 类按**精确 FQCN** serde 分 json/csv/hive-text),新增 CSV 枚举;复刻 OpenX `read_hive_json_in_one_column`(首列 string→单列 CSV,否则 fail-loud;会话开关经 `getSessionProperties` 读,首列类型 plumb 进 `HiveTableHandle`);未知 inputFormat/serde **fail-loud throw**(用户签字,与线上一致)。**fe-core `PluginDrivenScanNode.mapFileFormatType` 拆 `"text"→FORMAT_TEXT`、`"csv"→FORMAT_CSV_PLAIN`**(中立既有 token、无 `if(engine)`、hive 是唯一发射者、休眠;须与连接器 csv 发射**同一 commit**否则 OpenCSV 回退)。`HiveTextProperties` 补 legacy hive-2 JsonSerDe + serde2 MultiDelimit FQCN。全绿(HiveFileFormatTest 真值表 + hive 153/hms 150 无回归 + fe-core 编译 + checkstyle 0 + import gate 净)。 -- **下一步 = §4.2 `listPartitions`(item 3 below)**;其后 `getCapabilities` 其余项(item 4)、`HiveTableFormatDetector` 平价(item 5)。 +- **✅ §4.2 `listPartitions`/`listPartitionNames` DONE（`68e9fc8835b` + 前置修复 `3540d1327b6`）**（本轮侦察+3维对抗 review `wf_c25cc43a-43a`：legacy-parity/flip-consumer/code-edge 三镜头一致收敛，全 HEAD 核对）。**已签字决策（用户 2026-07-07）= `lastModifiedMillis` 填 -1，不付 per-partition 往返**：legacy 热分区裁剪路径（`HiveExternalMetaCache.loadPartitionValues`）只列**分区名**（无 `get_partitions_by_names` 往返），`transient_lastDdlTime` 只在 MTMV 刷新时取；`listPartitions` 跑在每次分区 hive 查询的 materialize 路径上，填 lastModified 会热路径倒退。故连接器 `listPartitions`/`listPartitionNames` 共用 `collectPartitionNames`（names-only，-1=全量），`lastModifiedMillis`/rowCount/size/fileCount 全留 UNKNOWN(-1)（对齐 maxcompute；hive 不声明 `SUPPORTS_PARTITION_STATS`→SHOW PARTITIONS 走 names-only 臂，-1 统计不外露）；filter 忽略（legacy 全量物化+FE 侧裁剪）；分区值经 `HiveWriteUtils.toPartitionValues`（byte-faithful port，含 unescape）解、按 remote 名 keying（`PluginDrivenExternalTable.getNameToPartitionItems` 按 remote 名读回）。**per-partition MTMV 新鲜度留到 §4.3 MVCC/MTMV 步**（届时像 legacy 只在刷新时取完整分区）。**前置修复（独立 commit `3540d1327b6`）**：`ThriftHmsClient.listPartitionNames` 原把 `maxParts<=0` 塌成 `Short.MAX_VALUE(32767)`→>32767 分区表**静默截断**（3 镜头一致标记的唯一真缺陷；且已默默毁掉 hudi/hive-write 传 -1 的"无截断"意图），改真 `(short)-1` 无界（正数透传、>32767 窄化成负 short 亦无界=保留 100000-callers 现状）。全绿（fe-connector-hive 160 + 7 新分区测 + fe-connector-hms 52 + 3 新 maxParts 测 + checkstyle 0 + import gate 净）。 + - ⚠**遗留（休眠期不动，翻闸/§4.3 处理）**：`getPartitionSnapshot` 对 hive 空 pin 返 `MTMVTimestampSnapshot(-1)`（两次 -1 相等→不会误判变更，但也不检测真实分区改动，须 §4.3 供真 millis）；`listPartitionValues`（`partition_values()` TVF）连接器**未实现**（无 fe-core 消费者，归 §4.6 TVF 门）。 +- **下一步 = §4.2 `getCapabilities` 连接器级其余项(item 4)**；其后 `HiveTableFormatDetector` 平价(item 5)。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 - 3. **`listPartitions`（含 `lastModifiedMillis`，M）**：镜像 paimon `collectPartitions`。`lastModifiedMillis` 唯一有 MTMV 影响（`PluginDrivenMvccExternalTable.listLatestPartitions:269` 唯一读者）；旧源 `transient_lastDdlTime*1000`（`HivePartition.getLastModifiedTime`）需 per-partition `getParameters()` 往返——**决策**：付这往返 vs 接受 -1。**不**声明 `SUPPORTS_PARTITION_STATS`（旧 hive SHOW PARTITIONS 只列名）。 + 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 5. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 From 5766e50e937893009ab35cb88077a39b861eeb36 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 22:45:19 +0800 Subject: [PATCH 054/334] [feat](catalog) hive connector getCapabilities: auto-analyze + metadata-preload (dormant) Declares the connector-wide capabilities the flipped hms catalog needs, each a faithful port of a legacy HMSExternalTable/HMS admission, alongside the already-declared SUPPORTS_VIEW: - SUPPORTS_COLUMN_AUTO_ANALYZE: legacy StatisticsUtil.supportAutoAnalyze admitted HMS tables of dlaType HIVE (and ICEBERG) into the background per-column FULL auto-analyze; the capability replaces that instanceof-HMSExternalTable arm. FULL is forced (sample analyze throws for external SQL-driven tables), and admitting hive needs no new SPI (FULL runs a SQL scan over the already-built read path). - SUPPORTS_METADATA_PRELOAD: legacy HMSExternalTable.supportsExternalMetadataPreload() returned true; replaces the legacy engine-name "jdbc" gate. Opt-in via enable_preload_external_metadata (default off), pure lock-latency optimization, no correctness effect. Gates only supportsExternalMetadataPreload -- plugin tables keep the false default for supportsLatestSnapshotPreload, so hive does not snapshot-preload. Deliberately withheld (verified sound): - SUPPORTS_MVCC_SNAPSHOT: the mixed hms catalog DOES need it (iceberg/hudi-on-HMS are MvccTable; GSON single-row maps HMSExternalTable -> PluginDrivenMvccExternalTable and buildTableInternal selects the Mvcc subclass from this catalog-level capability), but it lands WITH the MVCC/MTMV machinery (freshness-aware getTableSnapshot + hive empty-pin + iceberg sibling delegation), not as a bare flag. This corrects the earlier "hive is non-MVCC" note -- the deferral is timing, not permanent. - SUPPORTS_SHOW_CREATE_DDL: needs the connector to emit the table location + a generic-vs-hive-specific rendering decision (own substep). - SUPPORTS_PASSTHROUGH_QUERY / SUPPORTS_PARTITION_STATS: no query() TVF; names-only SHOW PARTITIONS. - SUPPORTS_TOPN_LAZY_MATERIALIZE: per-table marker (already emitted, orc/parquet). - SUPPORTS_NESTED_COLUMN_PRUNE: deferred like MVCC -- legacy parquet/orc pruned nested columns, but the connector must first carry nested field-ids or BE reads NULL leaves. Dormant: hms not in SPI_READY_TYPES, so getCapabilities is never consulted for hive tables today. Verified: 3-lens adversarial review confirmed the declared caps are faithful legacy ports with preconditions met and every deferral sound; residual = connector-wide auto-analyze over-admits hudi-on-HMS at the flip (legacy excluded hudi), to be gated per-handle or accepted at the delegation substep. fe-connector-hive 164 tests (+4 new capability tests) green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 50 ++++++++--- .../hive/HiveConnectorCapabilitiesTest.java | 90 +++++++++++++++++++ 2 files changed, 129 insertions(+), 11 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index ca4c42864647ec..31d25d9200e06b 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -93,17 +93,45 @@ public ConnectorWritePlanProvider getWritePlanProvider() { @Override public Set getCapabilities() { - // SUPPORTS_VIEW: legacy HMSExternalTable resolves isView() from the remote table's view text and hive - // views are queryable/droppable/visible in SHOW TABLES. The generic plugin-driven path reproduces this - // ONLY under this capability: PluginDrivenExternalTable.isView() then consults the connector's - // viewExists, PluginDrivenExternalCatalog.dropTable routes a view DROP to dropView, and - // listTableNamesFromRemote merges listViewNames into SHOW TABLES — hive returns an EMPTY listViewNames - // (its listTableNames already includes views), so the merge is a no-op and each view is listed once - // (legacy parity). Inert until hms enters SPI_READY_TYPES. Other connector-wide capabilities - // (auto-analyze, show-create-ddl, metadata-preload) land in their own substeps; SUPPORTS_MVCC_SNAPSHOT - // is intentionally withheld (hive is non-MVCC), and Top-N / nested-prune are per-table markers, not - // connector-wide flags. - return EnumSet.of(ConnectorCapability.SUPPORTS_VIEW); + // Connector-wide capabilities for the flipped hms catalog, each a faithful port of a legacy + // HMSExternalTable/HMS admission. Inert until hms enters SPI_READY_TYPES. + // - SUPPORTS_VIEW: legacy resolves isView() from the remote table's view text; the plugin path then + // consults viewExists, routes a view DROP to dropView, and merges listViewNames into SHOW TABLES — + // hive returns an EMPTY listViewNames (its listTableNames already includes views), so the merge is a + // no-op and each view is listed once (legacy parity). + // - SUPPORTS_COLUMN_AUTO_ANALYZE: legacy StatisticsUtil.supportAutoAnalyze admitted HMS tables of + // dlaType HIVE and ICEBERG (NOT hudi) into the background per-column FULL auto-analyze (sample is + // unimplemented for external SQL-driven tables); this capability replaces that instanceof- + // HMSExternalTable arm. It is connector-wide, so at the flip it also admits hudi-on-HMS tables that + // legacy excluded — a residual for the iceberg/hudi delegation substep to gate per-handle or + // explicitly accept (there is no per-table escape for it today, unlike Top-N). + // - SUPPORTS_METADATA_PRELOAD: legacy HMSExternalTable.supportsExternalMetadataPreload() returned true; + // the capability replaces the legacy engine-name "jdbc" gate. Opt-in via enable_preload_external_ + // metadata (default off), a pure lock-latency optimization with no correctness effect. + // + // Deliberately NOT declared here: + // - SUPPORTS_MVCC_SNAPSHOT: the heterogeneous hms catalog DOES need it (its iceberg/hudi-on-HMS tables + // are MvccTable, and the GSON single-row maps "HMSExternalTable" -> PluginDrivenMvccExternalTable, so + // buildTableInternal selects the Mvcc subclass from this catalog-level capability). It is declared + // together with the MVCC/MTMV machinery (freshness-aware getTableSnapshot + hive empty-pin + + // iceberg sibling delegation) in that substep, NOT as a bare flag here — declaring it now without the + // machinery would give plain-hive tables an eager empty snapshot with no freshness. (This is the fix + // to the earlier "hive is non-MVCC" note: hive is non-MVCC per table, but the catalog-level flag is + // required for the mixed catalog.) + // - SUPPORTS_SHOW_CREATE_DDL: the connector must first emit the table location (show.location) and a + // generic-vs-hive-specific SHOW CREATE rendering must be decided — its own substep. + // - SUPPORTS_PASSTHROUGH_QUERY / SUPPORTS_PARTITION_STATS: hive exposes no query() TVF, and legacy SHOW + // PARTITIONS lists names only. + // - SUPPORTS_TOPN_LAZY_MATERIALIZE: a per-table marker emitted in getTableSchema (orc/parquet only), + // never a connector-wide flag. + // - SUPPORTS_NESTED_COLUMN_PRUNE: NOT emitted yet — a genuine deferral like MVCC. Legacy parquet/orc + // pruned nested columns, but the connector must first carry stable nested field-ids down its column + // tree (SlotTypeReplacer rewrites nested access to field-ids; without them BE reads NULL leaves). + // Restore it as a per-table marker once hive field-ids are verified. + return EnumSet.of( + ConnectorCapability.SUPPORTS_VIEW, + ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE, + ConnectorCapability.SUPPORTS_METADATA_PRELOAD); } private HmsClient getOrCreateClient() { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java new file mode 100644 index 00000000000000..7e8428c2d4929d --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java @@ -0,0 +1,90 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorCapability; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Set; + +/** + * Pins the exact connector-wide capability set the hive connector declares (HMS cutover §4.2, dormant). + * + *

    Each capability is either a faithful port of a legacy HMSExternalTable/HMS admission (declared) or a + * deliberate deferral (withheld). The set is inert until hms enters SPI_READY_TYPES, so these assertions are + * a Rule-9 guard: flipping any capability without the supporting machinery (or dropping a legacy-parity one) + * would silently change post-flip behavior, and this test fails loud instead.

    + */ +public class HiveConnectorCapabilitiesTest { + + private Set capabilities() { + return new HiveConnector(Collections.emptyMap(), new FakeConnectorContext()).getCapabilities(); + } + + @Test + public void declaresLegacyParityCapabilities() { + Set caps = capabilities(); + // Legacy resolved isView() from the remote view text; the plugin view path is gated on this. + Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_VIEW), + "SUPPORTS_VIEW: legacy hive views are queryable/droppable/listed"); + // Legacy StatisticsUtil.supportAutoAnalyze admitted HMS dlaType HIVE into background FULL auto-analyze. + Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE), + "SUPPORTS_COLUMN_AUTO_ANALYZE: legacy admitted HMS HIVE tables into auto-analyze"); + // Legacy HMSExternalTable.supportsExternalMetadataPreload() returned true. + Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_METADATA_PRELOAD), + "SUPPORTS_METADATA_PRELOAD: legacy HMS tables were preload-eligible"); + } + + @Test + public void withholdsMvccUntilMvccMachineryLands() { + // The mixed hms catalog ultimately NEEDS this (iceberg/hudi-on-HMS are MvccTable), but it is declared + // together with the MVCC/MTMV machinery (freshness-aware getTableSnapshot + empty-pin + sibling + // delegation), not as a bare flag. Declaring it now would give plain-hive tables an eager empty + // snapshot with no freshness. + Assertions.assertFalse(capabilities().contains(ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT), + "SUPPORTS_MVCC_SNAPSHOT must land with the MVCC machinery, not here"); + } + + @Test + public void withholdsCapabilitiesWithoutSupportingSpi() { + Set caps = capabilities(); + // Needs the connector to emit the table location (show.location) + a rendering-parity decision first. + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_SHOW_CREATE_DDL), + "SUPPORTS_SHOW_CREATE_DDL needs location emission + a SHOW CREATE rendering decision"); + // Hive exposes no query() TVF (no getColumnsFromQuery). + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_PASSTHROUGH_QUERY), + "hive has no passthrough query()"); + // Legacy SHOW PARTITIONS lists names only; listPartitions emits UNKNOWN stats. + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_PARTITION_STATS), + "hive SHOW PARTITIONS is names-only"); + } + + @Test + public void topNAndNestedPruneAreNotConnectorWide() { + Set caps = capabilities(); + // Both are per-table markers emitted in getTableSchema (orc/parquet only), never connector-wide flags, + // otherwise a text/json/csv/view/hudi table would be wrongly eligible. + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE), + "Top-N lazy is a per-table marker, not connector-wide"); + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE), + "nested-column-prune is a per-table marker, not connector-wide"); + } +} From 4909bb2e807b36e837b7d9cde9946ee7b0e19ba3 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 22:46:35 +0800 Subject: [PATCH 055/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20getCapabilities=20DONE=20(auto-analyze=20+=20preload);=20?= =?UTF-8?q?MVCC/SHOW=5FCREATE=20deferred=20not=20forbidden;=20next=20=3D?= =?UTF-8?q?=20HiveTableFormatDetector?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c61a9095b402da..48095d66884053 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -45,10 +45,12 @@ - **✅ §4.2 file-format serde 修正 DONE（`c320df27712`）**（侦察 `wf_470a2e51-373`：4 维 + critic，含 BE C++ 读取器核对）。修**两个**读正确性 bug:(1) `HiveFileFormat.detect` 原 inputFormat-first→标准 JSON 表(inputFormat=TextInputFormat)被当 CSV 读;(2) 默认 hive 文本表(LazySimpleSerDe)本该走 BE `FORMAT_TEXT` 读取器(懂 hive 集合/map 分隔、`\N` 空、转义),却被塌缩成 `FORMAT_CSV_PLAIN`(BE 确有两个不同读取器类)。修法=**两段式、文本类 serde-authoritative**(inputFormat substring 分 parquet/orc/text-file;text 类按**精确 FQCN** serde 分 json/csv/hive-text),新增 CSV 枚举;复刻 OpenX `read_hive_json_in_one_column`(首列 string→单列 CSV,否则 fail-loud;会话开关经 `getSessionProperties` 读,首列类型 plumb 进 `HiveTableHandle`);未知 inputFormat/serde **fail-loud throw**(用户签字,与线上一致)。**fe-core `PluginDrivenScanNode.mapFileFormatType` 拆 `"text"→FORMAT_TEXT`、`"csv"→FORMAT_CSV_PLAIN`**(中立既有 token、无 `if(engine)`、hive 是唯一发射者、休眠;须与连接器 csv 发射**同一 commit**否则 OpenCSV 回退)。`HiveTextProperties` 补 legacy hive-2 JsonSerDe + serde2 MultiDelimit FQCN。全绿(HiveFileFormatTest 真值表 + hive 153/hms 150 无回归 + fe-core 编译 + checkstyle 0 + import gate 净)。 - **✅ §4.2 `listPartitions`/`listPartitionNames` DONE（`68e9fc8835b` + 前置修复 `3540d1327b6`）**（本轮侦察+3维对抗 review `wf_c25cc43a-43a`：legacy-parity/flip-consumer/code-edge 三镜头一致收敛，全 HEAD 核对）。**已签字决策（用户 2026-07-07）= `lastModifiedMillis` 填 -1，不付 per-partition 往返**：legacy 热分区裁剪路径（`HiveExternalMetaCache.loadPartitionValues`）只列**分区名**（无 `get_partitions_by_names` 往返），`transient_lastDdlTime` 只在 MTMV 刷新时取；`listPartitions` 跑在每次分区 hive 查询的 materialize 路径上，填 lastModified 会热路径倒退。故连接器 `listPartitions`/`listPartitionNames` 共用 `collectPartitionNames`（names-only，-1=全量），`lastModifiedMillis`/rowCount/size/fileCount 全留 UNKNOWN(-1)（对齐 maxcompute；hive 不声明 `SUPPORTS_PARTITION_STATS`→SHOW PARTITIONS 走 names-only 臂，-1 统计不外露）；filter 忽略（legacy 全量物化+FE 侧裁剪）；分区值经 `HiveWriteUtils.toPartitionValues`（byte-faithful port，含 unescape）解、按 remote 名 keying（`PluginDrivenExternalTable.getNameToPartitionItems` 按 remote 名读回）。**per-partition MTMV 新鲜度留到 §4.3 MVCC/MTMV 步**（届时像 legacy 只在刷新时取完整分区）。**前置修复（独立 commit `3540d1327b6`）**：`ThriftHmsClient.listPartitionNames` 原把 `maxParts<=0` 塌成 `Short.MAX_VALUE(32767)`→>32767 分区表**静默截断**（3 镜头一致标记的唯一真缺陷；且已默默毁掉 hudi/hive-write 传 -1 的"无截断"意图），改真 `(short)-1` 无界（正数透传、>32767 窄化成负 short 亦无界=保留 100000-callers 现状）。全绿（fe-connector-hive 160 + 7 新分区测 + fe-connector-hms 52 + 3 新 maxParts 测 + checkstyle 0 + import gate 净）。 - ⚠**遗留（休眠期不动，翻闸/§4.3 处理）**：`getPartitionSnapshot` 对 hive 空 pin 返 `MTMVTimestampSnapshot(-1)`（两次 -1 相等→不会误判变更，但也不检测真实分区改动，须 §4.3 供真 millis）；`listPartitionValues`（`partition_values()` TVF）连接器**未实现**（无 fe-core 消费者，归 §4.6 TVF 门）。 -- **下一步 = §4.2 `getCapabilities` 连接器级其余项(item 4)**；其后 `HiveTableFormatDetector` 平价(item 5)。 +- **✅ §4.2 `getCapabilities` 连接器级 DONE（`96953cd8eec`）**（3 维对抗 review `wf_8813de68-7f9`：declared-preconditions/completeness-omissions/mvcc-heterogeneity，`fix-then-ship`，两 minor 已按建议改注释）。声明 = `SUPPORTS_VIEW` + `SUPPORTS_COLUMN_AUTO_ANALYZE`（legacy `StatisticsUtil.supportAutoAnalyze` 收 HMS dlaType HIVE/ICEBERG 进后台 FULL 自动分析）+ `SUPPORTS_METADATA_PRELOAD`（legacy `HMSExternalTable.supportsExternalMetadataPreload()`=true；只管元数据预热、不触发 `supportsLatestSnapshotPreload`——插件用 false 默认，hive 不做 snapshot 预热）。三者休眠期零消费（hive 非 PluginDriven）。**⚠ 纠正交接旧错**：`SUPPORTS_MVCC_SNAPSHOT` **不是"禁"而是"推迟到 §4.3 MVCC 步"**——设计文档 §3 明确混合 HMS 目录**必须**声明它（iceberg/hudi-on-HMS 是 MvccTable；GSON 单行把 `HMSExternalTable`→`PluginDrivenMvccExternalTable`，`buildTableInternal` 按**目录级** MVCC 能力选 Mvcc 子类，`PluginDrivenExternalDatabase:56` 已 HEAD 证实）；现在裸声明会让 plain-hive 拿到无新鲜度的空快照，故与 MVCC 机件（freshness-aware `getTableSnapshot`+空 pin+兄弟委派）同步落。同理 `SUPPORTS_SHOW_CREATE_DDL` 推迟（须连接器先发 `show.location` + 定通用 vs hive-specific 渲染）；`SUPPORTS_NESTED_COLUMN_PRUNE` 推迟（须 hive 嵌套 field-id，否则 BE 读 NULL 叶——与前"按表能力接缝"步的 ⛔ 硬门同源）。全绿（fe-connector-hive 164 + 4 新能力测 + checkstyle 0 + import gate 净）。 + - ⚠**翻闸期残留（记账，勿现在动）**：`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 → 翻闸时**也**会收 hudi-on-HMS 进自动分析（legacy 排除 HUDL），且**无按表 escape**（不像 TOPN 有 marker）——须在 iceberg/hudi 委派步按 handle 门控或显式接受。`supportsLatestSnapshotPreload` 对 iceberg/hudi-on-HMS legacy=true、插件默认 false → 翻闸时 iceberg-on-HMS 丢 snapshot 预热（纯性能、归 §4.3/§4.4）。 +- **下一步 = §4.2 `HiveTableFormatDetector` 平价(item 5)**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 - 4. **`getCapabilities` 连接器级其余项**：随特性声明 `SUPPORTS_VIEW`/`SUPPORTS_COLUMN_AUTO_ANALYZE`/`SUPPORTS_METADATA_PRELOAD`/`SUPPORTS_SHOW_CREATE_DDL`；**禁** `SUPPORTS_MVCC_SNAPSHOT`（hive 非 MVCC，`EmptyMvccSnapshot`）、`SUPPORTS_PASSTHROUGH_QUERY`、`SUPPORTS_PARTITION_STATS`。TOPN/NESTED 走按表 marker（非此处）。 + 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 5. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 From 61004cf8ef5af5f5004a72be2c94740b57c378a6 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 23:07:19 +0800 Subject: [PATCH 056/334] [feat](catalog) hive HiveTableFormatDetector parity: LZO formats + fail-loud + view short-circuit (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes the last read-side format-detection gaps vs legacy HMSExternalTable: - LZO text InputFormats: HiveTableFormatDetector.detect now classifies the 3 LZO-text InputFormats (com.hadoop.compression.lzo.LzoTextInputFormat, com.hadoop.mapreduce. LzoTextInputFormat, com.hadoop.mapred.DeprecatedLzoTextInputFormat) as HIVE. The connector's SUPPORTED_HIVE_INPUT_FORMATS now exactly matches legacy SUPPORTED_HIVE_FILE_FORMATS (7 entries); without them an LZO-text table would fail the now-fail-loud format check even though legacy read it as hive text (FORMAT_TEXT). - Fail-loud: getTableHandle now throws DorisConnectorException on a null / unrecognized input format (UNKNOWN), replacing the old silent UNKNOWN-type handle. This mirrors legacy HMSExternalTable.supportedHiveTable() throwing NotSupportedException, at the same lazy per-table-access point (getTableHandle is the resolveConnectorTableHandle analogue of makeSureInitialized; SHOW TABLES uses listTableNames and is unaffected). - View short-circuit: the fail-loud is skipped for a view (isViewTable = view text present), matching legacy which returned true for a view before the format check. A view has no data files (usually null input format) and is served by the view SPI, so its handle keeps the UNKNOWN type (never scanned) rather than being rejected. Dormant: hms not in SPI_READY_TYPES. Verified: 3-lens adversarial review (classification- parity / fail-loud blast-radius / exception-catch-parity), verdict ship — format sets match legacy exactly, no bulk-listing abort (throw is lazy per-access), view predicate + detect ordering equivalent. Two residuals owned by the flip (not this step, pre-existing cross- connector SPI properties already exhibited by iceberg): existence-via-getTableHandle now fails loud for an unsupported non-view table (CREATE-IF-NOT-EXISTS/DROP), and a query on an unsupported table surfaces DorisConnectorException which fe-core ConnectProcessor classifies as a generic error rather than ERR_NOT_SUPPORTED_YET -- both fixable only in fe-core read-path handling at the flip. fe-connector-hive 171 tests (+7 new) green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 11 + .../hive/HiveTableFormatDetector.java | 10 + .../hive/HiveTableFormatDetectionTest.java | 212 ++++++++++++++++++ 3 files changed, 233 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableFormatDetectionTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 31a6380d70b6e2..e2eeca9402c2db 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -199,6 +199,17 @@ public Optional getTableHandle( } HmsTableInfo tableInfo = hmsClient.getTable(dbName, tableName); HiveTableType tableType = HiveTableFormatDetector.detect(tableInfo); + // Fail-loud parity with legacy HMSExternalTable.supportedHiveTable(), which threw on a null or + // unrecognized input format instead of silently degrading (the old detector returned UNKNOWN). A view + // short-circuits: legacy returns true for a view before the format check — a view has no data files so + // its (usually null) input format is irrelevant, and it is served through the view SPI, not the scan + // path, so its handle keeps the UNKNOWN type (never scanned) rather than being rejected here. + if (tableType == HiveTableType.UNKNOWN && !isViewTable(tableInfo)) { + String inputFormat = tableInfo.getInputFormat(); + throw new DorisConnectorException(inputFormat == null + ? "remote table's storage input format is null" + : "Unsupported hive input format: " + inputFormat); + } // Build partition key column names List partKeyNames = Collections.emptyList(); diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableFormatDetector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableFormatDetector.java index 13c1c046467728..02b08656d0cb91 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableFormatDetector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTableFormatDetector.java @@ -46,6 +46,16 @@ public final class HiveTableFormatDetector { // Some Hudi tables use HoodieParquetInputFormatBase as input format // but cannot be treated as Hudi — read parquet files directly as Hive. hiveFormats.add("org.apache.hudi.hadoop.HoodieParquetInputFormatBase"); + // LZO-compressed text InputFormats (hadoop-lzo / lzo-hadoop). Read-only: all three class names + // contain "text", so HiveFileFormat resolves them to TEXT_FILE, which with LazySimpleSerDe yields + // BE FORMAT_TEXT. Parity with legacy HMSExternalTable.SUPPORTED_HIVE_FILE_FORMATS; without these an + // LZO-text table would fail the (now fail-loud) format check. + // com.hadoop.compression.lzo.LzoTextInputFormat - twitter hadoop-lzo (GPL) + // com.hadoop.mapreduce.LzoTextInputFormat - lzo-hadoop mapreduce API (org.anarres) + // com.hadoop.mapred.DeprecatedLzoTextInputFormat - lzo-hadoop legacy mapred API (org.anarres) + hiveFormats.add("com.hadoop.compression.lzo.LzoTextInputFormat"); + hiveFormats.add("com.hadoop.mapreduce.LzoTextInputFormat"); + hiveFormats.add("com.hadoop.mapred.DeprecatedLzoTextInputFormat"); SUPPORTED_HIVE_INPUT_FORMATS = Collections.unmodifiableSet(hiveFormats); SUPPORTED_HUDI_INPUT_FORMATS = Collections.unmodifiableSet(new HashSet<>(Arrays.asList( diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableFormatDetectionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableFormatDetectionTest.java new file mode 100644 index 00000000000000..9e1c9c8b639c33 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTableFormatDetectionTest.java @@ -0,0 +1,212 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HiveTableFormatDetector#detect} format classification and the fail-loud + view short-circuit + * that {@link HiveConnectorMetadata#getTableHandle} layers on top (HMS cutover §4.2, dormant). + * + *

    WHY:

    + *
      + *
    • LZO parity. Legacy {@code HMSExternalTable.SUPPORTED_HIVE_FILE_FORMATS} includes three + * LZO-text InputFormats; omitting them would make an LZO-text table fail the (now fail-loud) format + * check even though legacy read it as hive text.
      • + *
    • Fail-loud parity. Legacy {@code supportedHiveTable()} threw on a null/unrecognized input + * format; the old connector silently returned UNKNOWN, which would let an unreadable table degrade + * instead of surfacing the error.
      • + *
    • View short-circuit. A view has no data files (null input format) but is valid; legacy + * returned true for it before the format check. The fail-loud guard must skip views or every hive + * view would be rejected at handle resolution.
      • + *
    + */ +public class HiveTableFormatDetectionTest { + + private static final String PARQUET = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String ORC = "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; + private static final String TEXT = "org.apache.hadoop.mapred.TextInputFormat"; + private static final String HUDI = "org.apache.hudi.hadoop.HoodieParquetInputFormat"; + + // ===== detect(): pure format classification ===== + + @Test + public void lzoTextFormatsClassifyAsHive() { + // All three LZO-text InputFormats legacy supported (com.hadoop.compression.lzo / mapreduce / mapred). + Assertions.assertEquals(HiveTableType.HIVE, + HiveTableFormatDetector.detect(hiveTable("com.hadoop.compression.lzo.LzoTextInputFormat"))); + Assertions.assertEquals(HiveTableType.HIVE, + HiveTableFormatDetector.detect(hiveTable("com.hadoop.mapreduce.LzoTextInputFormat"))); + Assertions.assertEquals(HiveTableType.HIVE, + HiveTableFormatDetector.detect(hiveTable("com.hadoop.mapred.DeprecatedLzoTextInputFormat"))); + } + + @Test + public void standardFormatsClassifyCorrectly() { + Assertions.assertEquals(HiveTableType.HIVE, HiveTableFormatDetector.detect(hiveTable(PARQUET))); + Assertions.assertEquals(HiveTableType.HIVE, HiveTableFormatDetector.detect(hiveTable(ORC))); + Assertions.assertEquals(HiveTableType.HIVE, HiveTableFormatDetector.detect(hiveTable(TEXT))); + Assertions.assertEquals(HiveTableType.HUDI, HiveTableFormatDetector.detect(hiveTable(HUDI))); + Assertions.assertEquals(HiveTableType.ICEBERG, HiveTableFormatDetector.detect(icebergTable())); + } + + @Test + public void unrecognizedOrNullFormatIsUnknown() { + Assertions.assertEquals(HiveTableType.UNKNOWN, + HiveTableFormatDetector.detect(hiveTable("com.example.MyCustomInputFormat"))); + Assertions.assertEquals(HiveTableType.UNKNOWN, HiveTableFormatDetector.detect(hiveTable(null))); + // detect() is format-only: a view (no data-file format) also classifies UNKNOWN here; the + // short-circuit that keeps it from being rejected lives in getTableHandle. + Assertions.assertEquals(HiveTableType.UNKNOWN, HiveTableFormatDetector.detect(viewTable())); + } + + // ===== getTableHandle(): fail-loud + view short-circuit ===== + + @Test + public void unsupportedFormatFailsLoud() { + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> handleFor(hiveTable("com.example.MyCustomInputFormat"))); + Assertions.assertTrue(ex.getMessage().contains("Unsupported hive input format"), ex.getMessage()); + } + + @Test + public void nullFormatNonViewFailsLoud() { + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> handleFor(hiveTable(null))); + Assertions.assertTrue(ex.getMessage().contains("input format is null"), ex.getMessage()); + } + + @Test + public void viewIsNotRejected() { + // A view has a null input format but must NOT fail loud; its handle keeps the UNKNOWN type (a view is + // served by the view SPI, never scanned). + Optional handle = handleFor(viewTable()); + Assertions.assertTrue(handle.isPresent(), "a hive view must resolve a handle, not be rejected"); + Assertions.assertEquals(HiveTableType.UNKNOWN, ((HiveTableHandle) handle.get()).getTableType()); + } + + @Test + public void supportedFormatsResolveHandle() { + Assertions.assertEquals(HiveTableType.HIVE, + ((HiveTableHandle) handleFor(hiveTable(PARQUET)).get()).getTableType()); + // An LZO-text table now resolves instead of failing loud. + Assertions.assertEquals(HiveTableType.HIVE, + ((HiveTableHandle) handleFor(hiveTable("com.hadoop.compression.lzo.LzoTextInputFormat")) + .get()).getTableType()); + } + + // ===== helpers ===== + + private Optional handleFor(HmsTableInfo tableInfo) { + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(tableInfo), Collections.emptyMap(), new FakeConnectorContext()); + return metadata.getTableHandle(null, "db", "t"); + } + + private static HmsTableInfo hiveTable(String inputFormat) { + return HmsTableInfo.builder() + .dbName("db").tableName("t").tableType("MANAGED_TABLE") + .inputFormat(inputFormat) + .build(); + } + + private static HmsTableInfo icebergTable() { + return HmsTableInfo.builder() + .dbName("db").tableName("t").tableType("EXTERNAL_TABLE") + .parameters(Collections.singletonMap("table_type", "ICEBERG")) + .build(); + } + + private static HmsTableInfo viewTable() { + return HmsTableInfo.builder() + .dbName("db").tableName("t").tableType("VIRTUAL_VIEW") + .viewOriginalText("SELECT 1") + .build(); + } + + /** Minimal {@link HmsClient} double serving one prebuilt table; the rest fail loud. */ + private static final class FakeHmsClient implements HmsClient { + private final HmsTableInfo table; + + FakeHmsClient(HmsTableInfo table) { + this.table = table; + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return true; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + return table; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} From 27b75d5872ed6f39d4759219379c997d7bf832c7 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 23:08:24 +0800 Subject: [PATCH 057/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2=20HiveTableFormatDetector=20DONE=20(read-side=20=C2=A74.2?= =?UTF-8?q?=20complete);=20next=20=3D=20column-stats=20SPI?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 48095d66884053..41131d4f41812f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -47,11 +47,13 @@ - ⚠**遗留（休眠期不动，翻闸/§4.3 处理）**：`getPartitionSnapshot` 对 hive 空 pin 返 `MTMVTimestampSnapshot(-1)`（两次 -1 相等→不会误判变更，但也不检测真实分区改动，须 §4.3 供真 millis）；`listPartitionValues`（`partition_values()` TVF）连接器**未实现**（无 fe-core 消费者，归 §4.6 TVF 门）。 - **✅ §4.2 `getCapabilities` 连接器级 DONE（`96953cd8eec`）**（3 维对抗 review `wf_8813de68-7f9`：declared-preconditions/completeness-omissions/mvcc-heterogeneity，`fix-then-ship`，两 minor 已按建议改注释）。声明 = `SUPPORTS_VIEW` + `SUPPORTS_COLUMN_AUTO_ANALYZE`（legacy `StatisticsUtil.supportAutoAnalyze` 收 HMS dlaType HIVE/ICEBERG 进后台 FULL 自动分析）+ `SUPPORTS_METADATA_PRELOAD`（legacy `HMSExternalTable.supportsExternalMetadataPreload()`=true；只管元数据预热、不触发 `supportsLatestSnapshotPreload`——插件用 false 默认，hive 不做 snapshot 预热）。三者休眠期零消费（hive 非 PluginDriven）。**⚠ 纠正交接旧错**：`SUPPORTS_MVCC_SNAPSHOT` **不是"禁"而是"推迟到 §4.3 MVCC 步"**——设计文档 §3 明确混合 HMS 目录**必须**声明它（iceberg/hudi-on-HMS 是 MvccTable；GSON 单行把 `HMSExternalTable`→`PluginDrivenMvccExternalTable`，`buildTableInternal` 按**目录级** MVCC 能力选 Mvcc 子类，`PluginDrivenExternalDatabase:56` 已 HEAD 证实）；现在裸声明会让 plain-hive 拿到无新鲜度的空快照，故与 MVCC 机件（freshness-aware `getTableSnapshot`+空 pin+兄弟委派）同步落。同理 `SUPPORTS_SHOW_CREATE_DDL` 推迟（须连接器先发 `show.location` + 定通用 vs hive-specific 渲染）；`SUPPORTS_NESTED_COLUMN_PRUNE` 推迟（须 hive 嵌套 field-id，否则 BE 读 NULL 叶——与前"按表能力接缝"步的 ⛔ 硬门同源）。全绿（fe-connector-hive 164 + 4 新能力测 + checkstyle 0 + import gate 净）。 - ⚠**翻闸期残留（记账，勿现在动）**：`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 → 翻闸时**也**会收 hudi-on-HMS 进自动分析（legacy 排除 HUDL），且**无按表 escape**（不像 TOPN 有 marker）——须在 iceberg/hudi 委派步按 handle 门控或显式接受。`supportsLatestSnapshotPreload` 对 iceberg/hudi-on-HMS legacy=true、插件默认 false → 翻闸时 iceberg-on-HMS 丢 snapshot 预热（纯性能、归 §4.3/§4.4）。 -- **下一步 = §4.2 `HiveTableFormatDetector` 平价(item 5)**。 +- **✅ §4.2 `HiveTableFormatDetector` 平价 DONE（`191449c7bd6`）**（3 维对抗 review `wf_9a5db88a-956`：classification-parity/fail-loud-blast-radius/exception-catch-parity，verdict `ship`，全 HEAD 核对）。①补 3 个 LZO text 格式→连接器 `SUPPORTED_HIVE_INPUT_FORMATS` 与 legacy `SUPPORTED_HIVE_FILE_FORMATS` **逐条相等（7 格式）**；②`getTableHandle` 检出 UNKNOWN 且非视图→ **fail-loud** `DorisConnectorException`（"Unsupported hive input format"/"input format is null"，对齐 legacy `supportedHiveTable` throw，且 throw 时机=懒惰按表访问，SHOW TABLES 走 `listTableNames` 不受影响）；③视图短路（`isViewTable`=view 文本在=legacy 谓词，视图 handle 保 UNKNOWN 类型不被拒、由视图 SPI 服务）。detect() 仍是纯分类器（不 throw），供 TopN 门/detectFormatType 复用。全绿（fe-connector-hive 171 + 7 新测 + checkstyle 0 + import gate 净）。 + - ⚠**翻闸期残留（两处，均**非本步引入、iceberg 已同款**，记账留翻闸的 fe-core 读路径处理，勿现在改）**：(1) **异常分类**——查询不支持格式的 hive 表抛 `DorisConnectorException`(RuntimeException)，fe-core `ConnectProcessor.handleQueryException` 无对应臂→归类 `ERR_UNKNOWN_ERROR`("maybe palo bug") 而非 legacy 的 `ERR_NOT_SUPPORTED_YET`；须翻闸时在 fe-core 读路径把 `DorisConnectorException` 映射到合适用户错误（跨连接器通用，iceberg `IcebergTypeMapping` 已同款）。(2) **存在性检查 fail-loud**——`PluginDrivenExternalCatalog.tableExist`/`CREATE IF NOT EXISTS` 经 `getTableHandle().isPresent()` 判存在，对"已存在但不支持格式的非视图表"会 throw（legacy `client.tableExists` 只回布尔不查格式）；是 PluginDriven 通用契约（iceberg/paimon 同）、且是有意的 fail-loud 方向，翻闸时定夺是否要免格式检查的存在性路径。 +- **下一步 = §4.2a 列统计 SPI（D1=保留，新增 `ConnectorStatisticsOps.getColumnStatistics`）**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 - 5. **`HiveTableFormatDetector` 平价**：补 3 个 LZO text 格式（`HMSExternalTable:170-172`）、UNKNOWN 改 throw、view 短路。 + 5. ~~`HiveTableFormatDetector` 平价~~ **✅ DONE**（见上 §4.2 DONE 条目；`191449c7bd6`；3 LZO 格式 + getTableHandle fail-loud + view 短路）。 - 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） From ff9b78c5ea22323dbf05c4a84ac1c7a3fd1144b0 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 23:41:29 +0800 Subject: [PATCH 058/334] [feat](catalog) column-statistics fast-path SPI: preserve hive no-scan col stats (dormant) Preserves the legacy HMSExternalTable.getColumnStatistic / getHiveColumnStats / setStatData query-planner fast path (no-scan HMS/spark column stats, consulted by ColumnStatisticsCacheLoader on a stats-cache miss) via a new SPI, following the established "connector returns raw facts, fe-core does the Doris-type math" split (D1=preserve). - fe-connector-api: new ConnectorColumnStatistics DTO (rowCount / ndv / numNulls / avgSizeBytes) + ConnectorStatisticsOps.getColumnStatistics(session, handle, columnName), a default returning empty (other connectors are unaffected and fall back to full ANALYZE). - fe-connector-hms: new neutral HmsColumnStatistics DTO; HmsClient gets a default getTableColumnStatistics (empty) so the test doubles need no change; ThriftHmsClient overrides it and convertColumnStatistics extracts ndv/numNulls/avgColLen per HMS ColumnStatisticsData variant -- a byte-faithful port of setStatData (avgColLen only for string; boolean/binary/timestamp -> ndv=0/numNulls=0, exactly as legacy's unhandled path). - fe-connector-hive: HiveConnectorMetadata.getColumnStatistics serves plain-hive tables only (iceberg-on-HMS goes to the sibling; hudi had no fast path), requires a positive numRows table param as the data-size basis (NO spark fallback here, unlike the table-size branch), fetches the HMS col stats and returns the raw facts. - fe-core: PluginDrivenExternalTable.getColumnStatistic override calls the SPI and toColumnStatistic() does the Doris-type-dependent size math the connector cannot (must not import fe-type): a string column -> round(avgColLen * count), every other type -> count * slot-width; avgSizeByte = dataSize / count; min/max = builder defaults (legacy's NEGATIVE/POSITIVE_INFINITY). Extracted static for direct unit testing. Scope: this is the query-planning column-stat fast path. The partition-level col stats (getHivePartitionColumnStats -> HMSAnalysisTask metadata ANALYZE path) and the iceberg connector's getIcebergColumnStats port are separate follow-ups. Dormant: hms not in SPI_READY_TYPES. Verified: 3-lens adversarial review (math byte-parity / guards+flow / architecture), verdict ship, zero confirmed defects -- field-by-field parity for every HMS variant, guards diverge only strictly-safer toward the ANALYZE fallback, no fe-core imports leak into the connector, SPI defaults keep other connectors inert. New tests: hms 4 (variant extraction) + hive 5 (fast-path gates) + fe-core 4 (Doris math); full fe-connector-hms 56 + fe-connector-hive 176 green, checkstyle + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../api/ConnectorColumnStatistics.java | 101 ++++++++++ .../connector/api/ConnectorStatisticsOps.java | 15 ++ .../connector/hive/HiveConnectorMetadata.java | 39 ++++ .../HiveConnectorMetadataColumnStatsTest.java | 182 ++++++++++++++++++ .../apache/doris/connector/hms/HmsClient.java | 19 ++ .../connector/hms/HmsColumnStatistics.java | 89 +++++++++ .../doris/connector/hms/ThriftHmsClient.java | 57 ++++++ .../hms/ThriftHmsClientColumnStatsTest.java | 108 +++++++++++ .../datasource/PluginDrivenExternalTable.java | 66 +++++++ ...uginDrivenExternalTableColumnStatTest.java | 85 ++++++++ 10 files changed, 761 insertions(+) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataColumnStatsTest.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsColumnStatistics.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientColumnStatsTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableColumnStatTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java new file mode 100644 index 00000000000000..7c413e5d313ab8 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java @@ -0,0 +1,101 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import java.util.Objects; + +/** + * Per-column statistics a connector can serve WITHOUT a table scan (e.g. hive's HMS/spark column stats), + * feeding the query-planner column-statistics fast path. + * + *

    This carries only RAW facts. The connector does NOT compute the Doris-type-dependent + * {@code dataSize}/{@code avgSize} — those depend on the column's fixed slot width, which the connector + * cannot know without importing fe-type. fe-core derives them from {@link #getAvgSizeBytes()} (when the + * source knows the average value size, e.g. a hive string column's {@code avgColLen}) or the column's slot + * width otherwise, mirroring the {@link #getTableStatistics}-style raw/derived split. Use {@link #UNKNOWN} + * when statistics are unavailable.

    + */ +public final class ConnectorColumnStatistics { + + /** Sentinel indicating no per-column statistics are available. */ + public static final ConnectorColumnStatistics UNKNOWN = + new ConnectorColumnStatistics(-1, -1, -1, -1); + + private final long rowCount; + private final long ndv; + private final long numNulls; + private final double avgSizeBytes; + + public ConnectorColumnStatistics(long rowCount, long ndv, long numNulls, double avgSizeBytes) { + this.rowCount = rowCount; + this.ndv = ndv; + this.numNulls = numNulls; + this.avgSizeBytes = avgSizeBytes; + } + + /** The table row count the stats are relative to (source {@code numRows}), or -1 if unknown. */ + public long getRowCount() { + return rowCount; + } + + /** Number of distinct values, or -1 if unknown. */ + public long getNdv() { + return ndv; + } + + /** Number of nulls, or -1 if unknown. */ + public long getNumNulls() { + return numNulls; + } + + /** + * Average per-value size in bytes when the source knows it (e.g. a hive string column's {@code avgColLen}), + * or {@code -1} when it does not — fe-core then falls back to the Doris column's fixed slot width. + */ + public double getAvgSizeBytes() { + return avgSizeBytes; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorColumnStatistics)) { + return false; + } + ConnectorColumnStatistics that = (ConnectorColumnStatistics) o; + return rowCount == that.rowCount + && ndv == that.ndv + && numNulls == that.numNulls + && Double.compare(that.avgSizeBytes, avgSizeBytes) == 0; + } + + @Override + public int hashCode() { + return Objects.hash(rowCount, ndv, numNulls, avgSizeBytes); + } + + @Override + public String toString() { + return "ConnectorColumnStatistics{rowCount=" + rowCount + + ", ndv=" + ndv + + ", numNulls=" + numNulls + + ", avgSizeBytes=" + avgSizeBytes + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java index 6603c28a2ab2e7..8d245ed90da547 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java @@ -33,6 +33,21 @@ default Optional getTableStatistics( return Optional.empty(); } + /** + * Returns per-column statistics the connector can serve WITHOUT a table scan — the query-planner + * column-statistics fast path, consulted on a stats-cache miss (fe-core's + * {@code ColumnStatisticsCacheLoader}). Must be cheap (a metadata read, no scan). Returns empty when + * unavailable, so a connector with no cheap column stats simply does not override it and fe-core falls + * back to a full ANALYZE. fe-core derives the Doris {@code ColumnStatistic} (dataSize / avgSize) from the + * returned raw facts — see {@link ConnectorColumnStatistics}. + */ + default Optional getColumnStatistics( + ConnectorSession session, + ConnectorTableHandle handle, + String columnName) { + return Optional.empty(); + } + /** * Estimates the table's on-disk data size in bytes by listing its data files, for connectors that can * cheaply enumerate them (e.g. hive). fe-core uses this to estimate a row count diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index e2eeca9402c2db..e918f3c9f1eaa8 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -19,6 +19,7 @@ import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorColumnStatistics; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; @@ -44,6 +45,7 @@ import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.HmsClientException; +import org.apache.doris.connector.hms.HmsColumnStatistics; import org.apache.doris.connector.hms.HmsCreateDatabaseRequest; import org.apache.doris.connector.hms.HmsCreateTableRequest; import org.apache.doris.connector.hms.HmsPartitionInfo; @@ -509,6 +511,43 @@ public Optional getTableStatistics( return Optional.of(new ConnectorTableStatistics(reportedRows, reportedSize)); } + /** + * Serves the query-planner column-statistics fast path from HMS-recorded (no-scan) column stats, a port of + * legacy {@code HMSExternalTable.getHiveColumnStats}. Returns RAW facts only (rowCount / ndv / numNulls / + * avgColLen) — fe-core does the Doris-type-dependent {@code dataSize}/{@code avgSize} math in + * {@code PluginDrivenExternalTable.getColumnStatistic} (it must not import fe-type). + * + *

    Empty (fe-core then falls back to a full ANALYZE) when: the table is not a plain-hive table + * (iceberg-on-HMS is served by the iceberg sibling; hudi had no fast path); the table has no positive + * {@code numRows} parameter (legacy required it as the per-column data-size basis, and does NOT fall back + * to the spark count here, unlike the table-level size branch); or HMS holds no stats for the column.

    + */ + @Override + public Optional getColumnStatistics( + ConnectorSession session, ConnectorTableHandle handle, String columnName) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + if (hiveHandle.getTableType() != HiveTableType.HIVE) { + return Optional.empty(); + } + Map params = hiveHandle.getTableParameters(); + if (params == null || !params.containsKey(PARAM_NUM_ROWS)) { + return Optional.empty(); + } + long rowCount = parseLongOrDefault(params.get(PARAM_NUM_ROWS), -1); + if (rowCount <= 0) { + return Optional.empty(); + } + List stats = hmsClient.getTableColumnStatistics( + hiveHandle.getDbName(), hiveHandle.getTableName(), Collections.singletonList(columnName)); + if (stats.isEmpty()) { + return Optional.empty(); + } + // Legacy read at most one stats object per column; take the first. + HmsColumnStatistics stat = stats.get(0); + return Optional.of(new ConnectorColumnStatistics( + rowCount, stat.getNdv(), stat.getNumNulls(), stat.getAvgColLenBytes())); + } + /** * Parses a metastore numeric parameter defensively. Legacy read these with a bare {@code Long.parseLong} * under an outer try/catch that logged and returned UNKNOWN; returning {@code defaultValue} on a diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataColumnStatsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataColumnStatsTest.java new file mode 100644 index 00000000000000..8d47964d0617c7 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataColumnStatsTest.java @@ -0,0 +1,182 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorColumnStatistics; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsColumnStatistics; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HiveConnectorMetadata#getColumnStatistics}, the query-planner column-stat fast path ported from + * legacy {@code HMSExternalTable.getHiveColumnStats} (HMS cutover §4.2a, dormant). + * + *

    WHY: the connector must serve the no-scan HMS column stats as RAW facts (rowCount / ndv / numNulls / + * avgColLen) and gate exactly as legacy did — a positive {@code numRows} is required as the data-size basis + * (and, unlike the table-size branch, there is NO spark-count fallback here), only a plain-hive table is + * served (iceberg-on-HMS goes to the sibling; hudi had no fast path), and a missing basis/stat degrades to + * empty so fe-core falls back to a full ANALYZE — WITHOUT paying the HMS column-stat round-trip.

    + */ +public class HiveConnectorMetadataColumnStatsTest { + + private HiveConnectorMetadata metadata(FakeHmsClient client) { + return new HiveConnectorMetadata(client, Collections.emptyMap(), new FakeConnectorContext()); + } + + private HiveTableHandle hiveHandle(Map params) { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .tableParameters(params) + .build(); + } + + private static Map numRows(String value) { + Map m = new HashMap<>(); + m.put("numRows", value); + return m; + } + + @Test + public void serviceHiveColumnStats() { + FakeHmsClient client = new FakeHmsClient( + Collections.singletonList(new HmsColumnStatistics("c", 10, 2, 5.0))); + Optional stats = + metadata(client).getColumnStatistics(null, hiveHandle(numRows("1000")), "c"); + Assertions.assertTrue(stats.isPresent()); + Assertions.assertEquals(1000, stats.get().getRowCount()); + Assertions.assertEquals(10, stats.get().getNdv()); + Assertions.assertEquals(2, stats.get().getNumNulls()); + Assertions.assertEquals(5.0, stats.get().getAvgSizeBytes(), 0.0); + } + + @Test + public void missingNumRowsReturnsEmptyWithoutHmsCall() { + FakeHmsClient client = new FakeHmsClient( + Collections.singletonList(new HmsColumnStatistics("c", 10, 2, 5.0))); + Assertions.assertFalse( + metadata(client).getColumnStatistics(null, hiveHandle(Collections.emptyMap()), "c").isPresent()); + Assertions.assertFalse(client.columnStatsCalled, + "no numRows basis => must not pay the HMS column-stat round-trip"); + } + + @Test + public void zeroOrMalformedNumRowsReturnsEmpty() { + FakeHmsClient client = new FakeHmsClient( + Collections.singletonList(new HmsColumnStatistics("c", 10, 2, 5.0))); + Assertions.assertFalse( + metadata(client).getColumnStatistics(null, hiveHandle(numRows("0")), "c").isPresent()); + Assertions.assertFalse( + metadata(client).getColumnStatistics(null, hiveHandle(numRows("abc")), "c").isPresent()); + } + + @Test + public void noHmsStatsReturnsEmpty() { + FakeHmsClient client = new FakeHmsClient(Collections.emptyList()); + Assertions.assertFalse( + metadata(client).getColumnStatistics(null, hiveHandle(numRows("1000")), "c").isPresent()); + } + + @Test + public void nonHiveTableReturnsEmptyWithoutHmsCall() { + FakeHmsClient client = new FakeHmsClient( + Collections.singletonList(new HmsColumnStatistics("c", 10, 2, 5.0))); + HiveTableHandle hudiHandle = new HiveTableHandle.Builder("db", "t", HiveTableType.HUDI) + .tableParameters(numRows("1000")) + .build(); + Assertions.assertFalse( + metadata(client).getColumnStatistics(null, hudiHandle, "c").isPresent()); + Assertions.assertFalse(client.columnStatsCalled, + "iceberg/hudi-on-HMS are served by their own connector; the hive fast path must not run"); + } + + /** Minimal {@link HmsClient} double serving preset column stats and recording the fetch. */ + private static final class FakeHmsClient implements HmsClient { + private final List columnStats; + private boolean columnStatsCalled; + + FakeHmsClient(List columnStats) { + this.columnStats = columnStats; + } + + @Override + public List getTableColumnStatistics(String dbName, String tableName, + List columns) { + columnStatsCalled = true; + return columnStats; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java index f3e16e10acae62..c453cf147ef3d8 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hms; import java.io.Closeable; +import java.util.Collections; import java.util.List; import java.util.Map; import java.util.function.Function; @@ -137,6 +138,24 @@ List getPartitions(String dbName, String tableName, HmsPartitionInfo getPartition(String dbName, String tableName, List values); + /** + * Returns HMS-recorded (no-scan) column statistics for the named columns, e.g. for the query-planner + * column-statistics fast path. + * + *

    Optional: defaults to an empty list so read-only test doubles and clients without column-stats + * support need not implement it (the caller then degrades to "no stats"). The production + * {@link ThriftHmsClient} overrides it.

    + * + * @param dbName database name + * @param tableName table name + * @param columns column names to fetch stats for + * @return one {@link HmsColumnStatistics} per column that HAS stats (may be shorter than {@code columns}) + */ + default List getTableColumnStatistics(String dbName, String tableName, + List columns) { + return Collections.emptyList(); + } + // ========== Phase 3: DDL / write operations ========== // // Optional operations: default to throwing so read-only implementations (the partition-pruning diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsColumnStatistics.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsColumnStatistics.java new file mode 100644 index 00000000000000..d6288158b7f276 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsColumnStatistics.java @@ -0,0 +1,89 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import java.util.Objects; + +/** + * Neutral per-column statistics extracted from a hive metastore {@code ColumnStatisticsObj}, hiding the + * hive-thrift type from callers. + * + *

    {@code avgColLenBytes} is set only for string columns (hive's {@code avgColLen}); it is {@code -1} for + * every other type, where the consumer uses the column's fixed slot width instead. {@code ndv}/{@code numNulls} + * are {@code 0} for a stats variant hive records but the legacy reader does not recognize (boolean / binary / + * timestamp), matching legacy {@code HMSExternalTable.setStatData}.

    + */ +public final class HmsColumnStatistics { + + private final String columnName; + private final long ndv; + private final long numNulls; + private final double avgColLenBytes; + + public HmsColumnStatistics(String columnName, long ndv, long numNulls, double avgColLenBytes) { + this.columnName = columnName; + this.ndv = ndv; + this.numNulls = numNulls; + this.avgColLenBytes = avgColLenBytes; + } + + public String getColumnName() { + return columnName; + } + + public long getNdv() { + return ndv; + } + + public long getNumNulls() { + return numNulls; + } + + /** Average string length in bytes for a string column, or {@code -1} for non-string columns. */ + public double getAvgColLenBytes() { + return avgColLenBytes; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof HmsColumnStatistics)) { + return false; + } + HmsColumnStatistics that = (HmsColumnStatistics) o; + return ndv == that.ndv + && numNulls == that.numNulls + && Double.compare(that.avgColLenBytes, avgColLenBytes) == 0 + && Objects.equals(columnName, that.columnName); + } + + @Override + public int hashCode() { + return Objects.hash(columnName, ndv, numNulls, avgColLenBytes); + } + + @Override + public String toString() { + return "HmsColumnStatistics{columnName='" + columnName + + "', ndv=" + ndv + + ", numNulls=" + numNulls + + ", avgColLenBytes=" + avgColLenBytes + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index b6c90fa81a83e0..31660e7acd3fd2 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -37,18 +37,25 @@ import org.apache.hadoop.hive.metastore.LockComponentBuilder; import org.apache.hadoop.hive.metastore.LockRequestBuilder; import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient; +import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData; +import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj; import org.apache.hadoop.hive.metastore.api.DataOperationType; import org.apache.hadoop.hive.metastore.api.Database; +import org.apache.hadoop.hive.metastore.api.DateColumnStatsData; +import org.apache.hadoop.hive.metastore.api.DecimalColumnStatsData; +import org.apache.hadoop.hive.metastore.api.DoubleColumnStatsData; import org.apache.hadoop.hive.metastore.api.FieldSchema; import org.apache.hadoop.hive.metastore.api.LockComponent; import org.apache.hadoop.hive.metastore.api.LockRequest; import org.apache.hadoop.hive.metastore.api.LockResponse; import org.apache.hadoop.hive.metastore.api.LockState; +import org.apache.hadoop.hive.metastore.api.LongColumnStatsData; import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.metastore.api.NoSuchObjectException; import org.apache.hadoop.hive.metastore.api.Partition; import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint; import org.apache.hadoop.hive.metastore.api.StorageDescriptor; +import org.apache.hadoop.hive.metastore.api.StringColumnStatsData; import org.apache.hadoop.hive.metastore.api.Table; import org.apache.hadoop.hive.metastore.api.TableValidWriteIds; import org.apache.hadoop.hive.metastore.txn.TxnUtils; @@ -249,6 +256,56 @@ public HmsPartitionInfo getPartition(String dbName, String tableName, }); } + @Override + public List getTableColumnStatistics(String dbName, String tableName, + List columns) { + List stats = execute( + client -> client.getTableColumnStatistics(dbName, tableName, columns)); + List result = new ArrayList<>(stats.size()); + for (ColumnStatisticsObj obj : stats) { + result.add(convertColumnStatistics(obj)); + } + return result; + } + + /** + * Extracts the neutral ndv / numNulls / avgColLen from a hive {@code ColumnStatisticsObj}, a byte-faithful + * port of legacy {@code HMSExternalTable.setStatData}'s per-variant reads: {@code avgColLen} is captured + * only for a string column (non-string columns leave it -1 so the consumer uses the fixed slot width), and + * a variant the legacy reader does not handle (boolean / binary / timestamp) yields ndv=0 / numNulls=0. + */ + static HmsColumnStatistics convertColumnStatistics(ColumnStatisticsObj obj) { + long ndv = 0; + long numNulls = 0; + double avgColLen = -1; + if (obj.isSetStatsData()) { + ColumnStatisticsData data = obj.getStatsData(); + if (data.isSetStringStats()) { + StringColumnStatsData stringStats = data.getStringStats(); + ndv = stringStats.getNumDVs(); + numNulls = stringStats.getNumNulls(); + avgColLen = stringStats.getAvgColLen(); + } else if (data.isSetLongStats()) { + LongColumnStatsData longStats = data.getLongStats(); + ndv = longStats.getNumDVs(); + numNulls = longStats.getNumNulls(); + } else if (data.isSetDecimalStats()) { + DecimalColumnStatsData decimalStats = data.getDecimalStats(); + ndv = decimalStats.getNumDVs(); + numNulls = decimalStats.getNumNulls(); + } else if (data.isSetDoubleStats()) { + DoubleColumnStatsData doubleStats = data.getDoubleStats(); + ndv = doubleStats.getNumDVs(); + numNulls = doubleStats.getNumNulls(); + } else if (data.isSetDateStats()) { + DateColumnStatsData dateStats = data.getDateStats(); + ndv = dateStats.getNumDVs(); + numNulls = dateStats.getNumNulls(); + } + } + return new HmsColumnStatistics(obj.getColName(), ndv, numNulls, avgColLen); + } + // ========== Phase 3: DDL / write operations ========== @Override diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientColumnStatsTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientColumnStatsTest.java new file mode 100644 index 00000000000000..87b037411903a0 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientColumnStatsTest.java @@ -0,0 +1,108 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.hadoop.hive.metastore.api.BooleanColumnStatsData; +import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData; +import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj; +import org.apache.hadoop.hive.metastore.api.LongColumnStatsData; +import org.apache.hadoop.hive.metastore.api.StringColumnStatsData; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Tests {@link ThriftHmsClient#convertColumnStatistics}: the byte-faithful extraction of ndv / numNulls / + * avgColLen from a hive {@code ColumnStatisticsObj}, ported from legacy {@code HMSExternalTable.setStatData}. + * + *

    WHY: the variant handling must match legacy exactly — a string column captures {@code avgColLen} (used + * for its data-size estimate), a numeric column leaves it {@code -1} (the consumer uses the fixed slot width), + * and a variant the legacy reader does NOT handle (boolean / binary / timestamp) must yield ndv=0 / numNulls=0 + * even though the metastore recorded a null count — otherwise a boolean column would report a spurious ndv.

    + */ +public class ThriftHmsClientColumnStatsTest { + + @Test + public void stringStatsCaptureAvgColLen() { + StringColumnStatsData s = new StringColumnStatsData(); + s.setNumDVs(10); + s.setNumNulls(2); + s.setAvgColLen(5.0); + s.setMaxColLen(20); + HmsColumnStatistics stat = ThriftHmsClient.convertColumnStatistics(objWith("c", "string", s, true)); + Assertions.assertEquals(10, stat.getNdv()); + Assertions.assertEquals(2, stat.getNumNulls()); + Assertions.assertEquals(5.0, stat.getAvgColLenBytes(), 0.0); + } + + @Test + public void longStatsLeaveAvgColLenUnset() { + LongColumnStatsData l = new LongColumnStatsData(); + l.setNumDVs(7); + l.setNumNulls(1); + HmsColumnStatistics stat = ThriftHmsClient.convertColumnStatistics(objWith("c", "bigint", l, false)); + Assertions.assertEquals(7, stat.getNdv()); + Assertions.assertEquals(1, stat.getNumNulls()); + // -1 => non-string; the consumer uses the Doris slot width. + Assertions.assertEquals(-1, stat.getAvgColLenBytes(), 0.0); + } + + @Test + public void unhandledVariantYieldsZeroNdvAndNulls() { + // Boolean is a variant legacy setStatData does not read, so ndv/numNulls stay 0 even though the + // metastore recorded numNulls=3 — pinning legacy parity (no spurious ndv for boolean/binary/timestamp). + BooleanColumnStatsData b = new BooleanColumnStatsData(); + b.setNumTrues(5); + b.setNumFalses(3); + b.setNumNulls(3); + ColumnStatisticsData data = new ColumnStatisticsData(); + data.setBooleanStats(b); + ColumnStatisticsObj obj = new ColumnStatisticsObj(); + obj.setColName("c"); + obj.setColType("boolean"); + obj.setStatsData(data); + HmsColumnStatistics stat = ThriftHmsClient.convertColumnStatistics(obj); + Assertions.assertEquals(0, stat.getNdv()); + Assertions.assertEquals(0, stat.getNumNulls()); + Assertions.assertEquals(-1, stat.getAvgColLenBytes(), 0.0); + } + + @Test + public void missingStatsDataYieldsZeros() { + ColumnStatisticsObj obj = new ColumnStatisticsObj(); + obj.setColName("c"); + HmsColumnStatistics stat = ThriftHmsClient.convertColumnStatistics(obj); + Assertions.assertEquals(0, stat.getNdv()); + Assertions.assertEquals(0, stat.getNumNulls()); + Assertions.assertEquals(-1, stat.getAvgColLenBytes(), 0.0); + Assertions.assertEquals("c", stat.getColumnName()); + } + + private static ColumnStatisticsObj objWith(String name, String type, Object variant, boolean isString) { + ColumnStatisticsData data = new ColumnStatisticsData(); + if (isString) { + data.setStringStats((StringColumnStatsData) variant); + } else { + data.setLongStats((LongColumnStatsData) variant); + } + ColumnStatisticsObj obj = new ColumnStatisticsObj(); + obj.setColName(name); + obj.setColType(type); + obj.setStatsData(data); + return obj; + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 1800ac3f1dea30..7e26f12b1636b9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -27,6 +27,7 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorColumnStatistics; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; @@ -42,6 +43,8 @@ import org.apache.doris.qe.GlobalVariable; import org.apache.doris.statistics.AnalysisInfo; import org.apache.doris.statistics.BaseAnalysisTask; +import org.apache.doris.statistics.ColumnStatistic; +import org.apache.doris.statistics.ColumnStatisticBuilder; import org.apache.doris.statistics.ExternalAnalysisTask; import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; @@ -748,6 +751,69 @@ public BaseAnalysisTask createAnalysisTask(AnalysisInfo info) { return new ExternalAnalysisTask(info); } + /** + * The query-planner column-statistics fast path (consulted by {@code ColumnStatisticsCacheLoader} on a + * stats-cache miss): asks the connector for the no-scan column stats it can serve cheaply and turns the raw + * facts into a Doris {@link ColumnStatistic}. Empty (fe-core falls back to a full ANALYZE) when the + * connector has no such stats. Mirrors legacy {@code HMSExternalTable.getHiveColumnStats} + + * {@code setStatData}: the connector returns raw ndv / numNulls / (string) avgColLen, and THIS side does the + * Doris-type-dependent size math the connector cannot (it must not import fe-type) — a string column's size + * is {@code round(avgColLen * count)}, every other type's is {@code count * }; min/max stay + * unconstrained (the {@link ColumnStatisticBuilder} defaults, i.e. legacy's NEGATIVE/POSITIVE_INFINITY). + */ + @Override + public Optional getColumnStatistic(String colName) { + makeSureInitialized(); + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return Optional.empty(); + } + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + if (connector == null) { + return Optional.empty(); + } + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + if (!handleOpt.isPresent()) { + return Optional.empty(); + } + Optional statsOpt = + metadata.getColumnStatistics(session, handleOpt.get(), colName); + if (!statsOpt.isPresent()) { + return Optional.empty(); + } + return toColumnStatistic(statsOpt.get(), getColumn(colName)); + } + + /** + * Turns the connector's raw {@link ConnectorColumnStatistics} into a Doris {@link ColumnStatistic}, + * doing the Doris-type-dependent size math the connector cannot (legacy {@code setStatData} parity): a + * string column (avgSizeBytes >= 0) sizes to {@code round(avgColLen * count)}, every other type to + * {@code count * }; {@code avgSizeByte = dataSize / count}; min/max stay at the builder's + * unconstrained defaults. Empty when the column is unknown or the row count is non-positive (no size + * basis, legacy returned empty). Package-private + static so the math can be unit-tested directly. + */ + static Optional toColumnStatistic(ConnectorColumnStatistics stats, Column column) { + long count = stats.getRowCount(); + if (column == null || count <= 0) { + return Optional.empty(); + } + double dataSize; + if (stats.getAvgSizeBytes() >= 0) { + dataSize = Math.round(stats.getAvgSizeBytes() * count); + } else { + // Long arithmetic (count * slotSize) exactly like legacy setStatData, then widened. + dataSize = count * column.getType().getSlotSize(); + } + ColumnStatisticBuilder builder = new ColumnStatisticBuilder(count); + builder.setNdv(stats.getNdv()); + builder.setNumNulls(stats.getNumNulls()); + builder.setDataSize(dataSize); + builder.setAvgSizeByte(dataSize / count); + return Optional.of(builder.build()); + } + @Override public long fetchRowCount() { makeSureInitialized(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableColumnStatTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableColumnStatTest.java new file mode 100644 index 00000000000000..0541db18c07692 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableColumnStatTest.java @@ -0,0 +1,85 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.Type; +import org.apache.doris.connector.api.ConnectorColumnStatistics; +import org.apache.doris.statistics.ColumnStatistic; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Optional; + +/** + * Tests {@link PluginDrivenExternalTable#toColumnStatistic}, the Doris-type-dependent column-stat math that + * fe-core does on the connector's raw facts (HMS cutover §4.2a). + * + *

    WHY: this is the byte-parity half of legacy {@code HMSExternalTable.setStatData} the connector cannot do + * (it must not import fe-type). A string column's data size uses the source {@code avgColLen} + * ({@code round(avgColLen * count)}); every other type uses the Doris fixed slot width + * ({@code count * slotSize}); {@code avgSizeByte = dataSize / count}; min/max stay unconstrained. Getting the + * string-vs-fixed-width split or the rounding wrong would skew every cardinality estimate that reads the + * column's stats.

    + */ +public class PluginDrivenExternalTableColumnStatTest { + + @Test + public void stringColumnUsesAvgColLen() { + ConnectorColumnStatistics stats = new ConnectorColumnStatistics(100, 10, 2, 5.0); + ColumnStatistic cs = PluginDrivenExternalTable + .toColumnStatistic(stats, new Column("c", Type.STRING)).get(); + Assertions.assertEquals(100, cs.count, 0.0); + Assertions.assertEquals(10, cs.ndv, 0.0); + Assertions.assertEquals(2, cs.numNulls, 0.0); + // round(5.0 * 100) = 500; avgSizeByte = 500 / 100 = 5.0 + Assertions.assertEquals(500, cs.dataSize, 0.0); + Assertions.assertEquals(5.0, cs.avgSizeByte, 0.0); + Assertions.assertEquals(Double.NEGATIVE_INFINITY, cs.minValue, 0.0); + Assertions.assertEquals(Double.POSITIVE_INFINITY, cs.maxValue, 0.0); + } + + @Test + public void nonStringColumnUsesSlotWidth() { + Column column = new Column("c", Type.INT); + long count = 100; + long slotSize = column.getType().getSlotSize(); + // avgSizeBytes = -1 => non-string => data size from the Doris slot width. + ConnectorColumnStatistics stats = new ConnectorColumnStatistics(count, 7, 1, -1); + ColumnStatistic cs = PluginDrivenExternalTable.toColumnStatistic(stats, column).get(); + Assertions.assertEquals(7, cs.ndv, 0.0); + Assertions.assertEquals(1, cs.numNulls, 0.0); + Assertions.assertEquals((double) count * slotSize, cs.dataSize, 0.0); + Assertions.assertEquals((double) slotSize, cs.avgSizeByte, 0.0); + } + + @Test + public void nonPositiveCountIsEmpty() { + Assertions.assertFalse(PluginDrivenExternalTable + .toColumnStatistic(new ConnectorColumnStatistics(0, 1, 0, -1), new Column("c", Type.INT)) + .isPresent()); + } + + @Test + public void nullColumnIsEmpty() { + Optional result = PluginDrivenExternalTable + .toColumnStatistic(new ConnectorColumnStatistics(100, 1, 0, -1), null); + Assertions.assertFalse(result.isPresent()); + } +} From 4ff2c43a89e0e7ca6fd0f3f9188d983ab4b02bc5 Mon Sep 17 00:00:00 2001 From: morningman Date: Tue, 7 Jul 2026 23:42:30 +0800 Subject: [PATCH 059/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.2a=20column-stats=20fast-path=20DONE;=20next=20=3D=20=C2=A74.?= =?UTF-8?q?3=20MVCC/sys-table=20+=20freshness?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 41131d4f41812f..288029cf2c8661 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-07） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -49,12 +49,14 @@ - ⚠**翻闸期残留（记账，勿现在动）**：`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 → 翻闸时**也**会收 hudi-on-HMS 进自动分析（legacy 排除 HUDL），且**无按表 escape**（不像 TOPN 有 marker）——须在 iceberg/hudi 委派步按 handle 门控或显式接受。`supportsLatestSnapshotPreload` 对 iceberg/hudi-on-HMS legacy=true、插件默认 false → 翻闸时 iceberg-on-HMS 丢 snapshot 预热（纯性能、归 §4.3/§4.4）。 - **✅ §4.2 `HiveTableFormatDetector` 平价 DONE（`191449c7bd6`）**（3 维对抗 review `wf_9a5db88a-956`：classification-parity/fail-loud-blast-radius/exception-catch-parity，verdict `ship`，全 HEAD 核对）。①补 3 个 LZO text 格式→连接器 `SUPPORTED_HIVE_INPUT_FORMATS` 与 legacy `SUPPORTED_HIVE_FILE_FORMATS` **逐条相等（7 格式）**；②`getTableHandle` 检出 UNKNOWN 且非视图→ **fail-loud** `DorisConnectorException`（"Unsupported hive input format"/"input format is null"，对齐 legacy `supportedHiveTable` throw，且 throw 时机=懒惰按表访问，SHOW TABLES 走 `listTableNames` 不受影响）；③视图短路（`isViewTable`=view 文本在=legacy 谓词，视图 handle 保 UNKNOWN 类型不被拒、由视图 SPI 服务）。detect() 仍是纯分类器（不 throw），供 TopN 门/detectFormatType 复用。全绿（fe-connector-hive 171 + 7 新测 + checkstyle 0 + import gate 净）。 - ⚠**翻闸期残留（两处，均**非本步引入、iceberg 已同款**，记账留翻闸的 fe-core 读路径处理，勿现在改）**：(1) **异常分类**——查询不支持格式的 hive 表抛 `DorisConnectorException`(RuntimeException)，fe-core `ConnectProcessor.handleQueryException` 无对应臂→归类 `ERR_UNKNOWN_ERROR`("maybe palo bug") 而非 legacy 的 `ERR_NOT_SUPPORTED_YET`；须翻闸时在 fe-core 读路径把 `DorisConnectorException` 映射到合适用户错误（跨连接器通用，iceberg `IcebergTypeMapping` 已同款）。(2) **存在性检查 fail-loud**——`PluginDrivenExternalCatalog.tableExist`/`CREATE IF NOT EXISTS` 经 `getTableHandle().isPresent()` 判存在，对"已存在但不支持格式的非视图表"会 throw（legacy `client.tableExists` 只回布尔不查格式）；是 PluginDriven 通用契约（iceberg/paimon 同）、且是有意的 fail-loud 方向，翻闸时定夺是否要免格式检查的存在性路径。 -- **下一步 = §4.2a 列统计 SPI（D1=保留，新增 `ConnectorStatisticsOps.getColumnStatistics`）**。 +- **✅ §4.2a 列统计 fast-path SPI DONE（`cbf9526f776`）**（3 维对抗 review `wf_144b6d15-77f`：math-byte-parity/guards+flow/architecture，verdict `ship` 零确认缺陷，全 HEAD 核对）。架构=**连接器回传原始事实、fe-core 做 Doris 类型运算**（沿用 getTableStatistics 先例）：`convertColumnStatistics` 按 HMS variant 抽 ndv/nulls/avgColLen（string 才有 avgColLen，boolean/binary/timestamp→0/0 与 legacy 未处理分支同）；hive `getColumnStatistics` 仅 HIVE 表、需正 `numRows`（**无 spark 回退**，与表级 size 分支不对称、复刻 legacy）；fe-core `toColumnStatistic` 做 size math（string→`round(avgColLen*count)`、否则→`count*slotSize`；`avgSizeByte=dataSize/count`；min/max=builder -inf/+inf 默认）。guard 分歧仅**更安全**地倒向 empty→ANALYZE（负/畸形 numRows）。全绿（hms 4+hive 5+fe-core 4 新测；hms 56/hive 176 套 + checkstyle 0 + import gate 净）。 + - ⚠**本步范围=查询规划列统计 fast-path**；**两处后续**（非本步）：①分区级列统计（`getHivePartitionColumnStats`→`HMSAnalysisTask` 的 ANALYZE 元数据 fast-path，设计 §4.2a 标"revisit `ExternalAnalysisTask` 是否走新 SPI vs SQL-only FULL")；②iceberg 连接器 `getIcebergColumnStats` port（`enable_fetch_iceberg_stats` 门，属 iceberg 连接器/§4.4，非 hive）。 +- **下一步 = §4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`**（此步声明 `SUPPORTS_MVCC_SNAPSHOT`——即 getCapabilities 步推迟的项；连带 hive 空 pin + `getPartitionSnapshot` 真 millis 收 listPartitions 步的遗留）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 5. ~~`HiveTableFormatDetector` 平价~~ **✅ DONE**（见上 §4.2 DONE 条目；`191449c7bd6`；3 LZO 格式 + getTableHandle fail-loud + view 短路）。 -- 后续子步：§4.2a 列统计 SPI（**D1=保留**，新增 `getColumnStatistics`）、§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 +- 后续子步：**§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`（下一步，含声明 `SUPPORTS_MVCC_SNAPSHOT`）**、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。（§4.2a 列统计 fast-path 已 DONE；其分区级 ANALYZE + iceberg col-stats 两处后续见上 §4.2a 条目。）**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） 1. **先读设计文档 `hms-cutover-retype-design-2026-07-07.md`（权威计划）+ 两份 findings + 本文顶部 🎯 段**。剩余 HMS 迁移 = 一次原子翻闸 + 前置休眠补齐 + 末尾循环删除。 From e398d8b2fcd28b481bd501dc94bc44affe935bf4 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 01:43:31 +0800 Subject: [PATCH 060/334] [feat](catalog) hive MVCC/MTMV freshness SPI: last-modified table/partition snapshots (dormant) Fix the stale-MV root cause for hive base tables served by the connector SPI, and declare SUPPORTS_MVCC_SNAPSHOT for the flipped hms catalog. PluginDrivenMvccExternalTable.getTableSnapshot hardcoded MTMVSnapshotIdSnapshot; for a hive empty pin the snapshot id is a constant -1, so an MV over a hive base table would compare equal forever and never refresh. getPartitionSnapshot likewise read the pin's -1 (hive listPartitions is names-only on the scan hot path). Fix = make the table/partition MTMV snapshots freshness-kind-aware, driven by a per-handle signal the connector puts ON THE QUERY-BEGIN PIN (no engine branch in fe-core): - fe-connector-api: ConnectorMvccSnapshot gains lastModifiedFreshness (default false); ConnectorMetadata gains getTableFreshness / getPartitionFreshnessMillis (default empty) + a neutral ConnectorTableFreshness DTO. All consulted only on the MTMV refresh path. - fe-core PluginDrivenMvccExternalTable: getTableSnapshot/getPartitionSnapshot read the pin flag; last-modified (hive) -> MTMVMaxTimestampSnapshot / MTMVTimestampSnapshot from the on-demand SPI (byte-parity legacy HiveDlaTable). A snapshot-id connector (paimon/iceberg) leaves the flag false and takes the EXACT pre-change path off the pin it already holds -- zero extra metadata round-trips (guarded by two verify(never) regression tests). A vanished partition (on-demand empty) raises the legacy "can not find partition". - fe-connector-hive: beginQuerySnapshot returns the empty(-1) pin flagged last-modified; getTableFreshness (unpartitioned = table transient_lastDdlTime, no round-trip; partitioned = max over get_partitions_by_names, named by the owning partition) and getPartitionFreshnessMillis port legacy transient_lastDdlTime*1000 (parse stays connector-side). HiveConnector declares SUPPORTS_MVCC_SNAPSHOT. Scan hot path (listPartitions) stays names-only; freshness is fetched only at MTMV refresh (two-path parity with legacy). Dormant for hive (not yet in SPI_READY_TYPES); paimon/iceberg byte-and-cost unchanged. fe-connector-hive 180 (9 new freshness + capabilities updated) + fe-core PluginDrivenMvccExternalTableTest 53 (7 new) + PluginDriven neighbors green; checkstyle 0; connector-imports gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/api/ConnectorMetadata.java | 39 +++ .../api/mvcc/ConnectorMvccSnapshot.java | 22 ++ .../api/mvcc/ConnectorTableFreshness.java | 86 +++++ .../doris/connector/hive/HiveConnector.java | 22 +- .../connector/hive/HiveConnectorMetadata.java | 116 +++++++ .../hive/HiveConnectorCapabilitiesTest.java | 17 +- .../HiveConnectorMetadataFreshnessTest.java | 301 ++++++++++++++++++ .../PluginDrivenMvccExternalTable.java | 111 ++++++- .../PluginDrivenMvccExternalTableTest.java | 135 ++++++++ 9 files changed, 822 insertions(+), 27 deletions(-) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTableFreshness.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFreshnessTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java index ca7880fec3be85..cdd96016893841 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; import java.io.Closeable; @@ -27,6 +28,7 @@ import java.util.Collections; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; import java.util.Set; /** @@ -86,6 +88,43 @@ default Optional getMvccPartitionView( return Optional.empty(); } + /** + * Whole-table MTMV freshness for a connector whose table-level change signal is a last-modified + * TIMESTAMP rather than a snapshot id (e.g. hive: {@code transient_lastDdlTime} / the max partition + * modify time). The generic model wraps the result in an {@code MTMVMaxTimestampSnapshot} table snapshot. + * + *

    Consulted ONLY when the query-begin pin's {@link ConnectorMvccSnapshot#isLastModifiedFreshness()} + * is set — i.e. a last-modified connector, which the generic model reads off the pin it already holds. + * A snapshot-id connector (paimon/iceberg) leaves the pin flag false, so this is NEVER called for it and it + * pays ZERO extra metadata round-trips. And it is on the MTMV refresh path, never the scan hot path — so a + * partitioned last-modified connector may pay a {@code get_partitions_by_names} round-trip here (mirroring + * legacy, which fetched per-partition modify time only at refresh time) without regressing queries.

    + * + *

    The default returns empty: a connector that sets the pin flag MUST override this.

    + */ + default Optional getTableFreshness( + ConnectorSession session, ConnectorTableHandle handle) { + return Optional.empty(); + } + + /** + * Per-partition last-modified millis for a last-modified connector, wrapped by the generic model in an + * {@code MTMVTimestampSnapshot}. Fetched on the MTMV refresh path only — a last-modified connector's + * {@code listPartitions} is names-only on the scan hot path (its per-partition {@code lastModifiedMillis} + * stays {@code -1}), so the real time is fetched here on demand instead. + * + *

    Consulted ONLY when the query-begin pin's {@link ConnectorMvccSnapshot#isLastModifiedFreshness()} + * is set (a snapshot-id connector never reaches here — zero extra round-trips). fe-core validates the + * partition exists in the materialized set BEFORE calling this; an {@code empty} return therefore means the + * partition VANISHED between the materialize and this fetch (a refresh-time race), and fe-core raises the + * legacy "can not find partition" error. The default returns empty: a last-modified connector MUST + * override it.

    + */ + default OptionalLong getPartitionFreshnessMillis( + ConnectorSession session, ConnectorTableHandle handle, String partitionName) { + return OptionalLong.empty(); + } + /** * Resolves an explicit time-travel spec (extracted from {@code FOR TIME AS OF} / * {@code FOR VERSION AS OF}, or the {@code @tag} / {@code @branch} / {@code @incr} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java index 7a16661ff84099..d89bc7cc976367 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java @@ -38,6 +38,7 @@ public final class ConnectorMvccSnapshot { private final String description; private final long schemaId; private final Map properties; + private final boolean lastModifiedFreshness; private ConnectorMvccSnapshot(Builder b) { this.snapshotId = b.snapshotId; @@ -47,6 +48,7 @@ private ConnectorMvccSnapshot(Builder b) { this.properties = b.properties.isEmpty() ? Collections.emptyMap() : Collections.unmodifiableMap(new HashMap<>(b.properties)); + this.lastModifiedFreshness = b.lastModifiedFreshness; } /** Connector-assigned snapshot identifier (e.g. Iceberg snapshot id). */ @@ -77,6 +79,19 @@ public Map getProperties() { return properties; } + /** + * Whether this table's MTMV freshness is a last-modified TIMESTAMP rather than a snapshot id. When + * {@code true} the generic model ({@code PluginDrivenMvccExternalTable}) serves the table/partition MTMV + * snapshots from {@code ConnectorMetadata.getTableFreshness} / {@code getPartitionFreshnessMillis} (fetched + * on the MTMV refresh path only); when {@code false} (the default, e.g. paimon/iceberg — a snapshot-id + * connector) it keeps the snapshot-id / pin-timestamp freshness with NO extra metadata call. The flag rides + * on the query-begin pin so fe-core reads it off the pin it already holds — a snapshot-id connector pays + * zero extra round-trips. + */ + public boolean isLastModifiedFreshness() { + return lastModifiedFreshness; + } + public static Builder builder() { return new Builder(); } @@ -88,12 +103,19 @@ public static final class Builder { private String description = ""; private long schemaId = -1; private final Map properties = new HashMap<>(); + private boolean lastModifiedFreshness; public Builder snapshotId(long snapshotId) { this.snapshotId = snapshotId; return this; } + /** Marks this table's MTMV freshness as last-modified (see {@link #isLastModifiedFreshness()}). */ + public Builder lastModifiedFreshness(boolean lastModifiedFreshness) { + this.lastModifiedFreshness = lastModifiedFreshness; + return this; + } + public Builder timestampMillis(long timestampMillis) { this.timestampMillis = timestampMillis; return this; diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTableFreshness.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTableFreshness.java new file mode 100644 index 00000000000000..b924e403927a95 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTableFreshness.java @@ -0,0 +1,86 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import java.util.Objects; + +/** + * A connector-supplied, whole-table MTMV freshness marker for a connector whose table-level change + * signal is a last-modified TIMESTAMP rather than a snapshot id. + * + *

    Returned by {@link org.apache.doris.connector.api.ConnectorMetadata#getTableFreshness}. The + * generic table model ({@code PluginDrivenMvccExternalTable}) wraps it into an + * {@code MTMVMaxTimestampSnapshot(name, timestampMillis)} — byte-parity with legacy hive, whose table + * snapshot is {@code MTMVMaxTimestampSnapshot}. A connector that returns {@link java.util.Optional#empty()} + * (the default, e.g. paimon/iceberg) keeps the snapshot-id table snapshot unchanged.

    + * + *

    Semantics (mirror legacy {@code HiveDlaTable.getTableSnapshot}):

    + *
      + *
    • partitioned table ⇒ {@code name} = the partition name carrying the newest modify time, + * {@code timestampMillis} = that max modify time (0 when there are no partitions, with + * {@code name} = the table name);
      • + *
    • unpartitioned table ⇒ {@code name} = the table name, {@code timestampMillis} = the table's + * last-DDL time (0 when absent).
      • + *
    + * + *

    The connector computes the millis (the {@code MTMVMaxTimestampSnapshot} carries BOTH the name and + * the timestamp so that dropping the partition that owns the max time is detected as a change); fe-core + * never parses the underlying source properties.

    + */ +public final class ConnectorTableFreshness { + + private final String name; + private final long timestampMillis; + + public ConnectorTableFreshness(String name, long timestampMillis) { + this.name = Objects.requireNonNull(name, "name"); + this.timestampMillis = timestampMillis; + } + + /** The partition name owning the newest modify time (partitioned) or the table name (unpartitioned). */ + public String getName() { + return name; + } + + /** The newest modify time in epoch millis (max partition modify time, or the table last-DDL time). */ + public long getTimestampMillis() { + return timestampMillis; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorTableFreshness)) { + return false; + } + ConnectorTableFreshness that = (ConnectorTableFreshness) o; + return timestampMillis == that.timestampMillis && name.equals(that.name); + } + + @Override + public int hashCode() { + return Objects.hash(name, timestampMillis); + } + + @Override + public String toString() { + return "ConnectorTableFreshness{name='" + name + "', timestampMillis=" + timestampMillis + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 31d25d9200e06b..784c3f3e9cd15e 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -108,16 +108,19 @@ public Set getCapabilities() { // - SUPPORTS_METADATA_PRELOAD: legacy HMSExternalTable.supportsExternalMetadataPreload() returned true; // the capability replaces the legacy engine-name "jdbc" gate. Opt-in via enable_preload_external_ // metadata (default off), a pure lock-latency optimization with no correctness effect. + // - SUPPORTS_MVCC_SNAPSHOT: the heterogeneous hms catalog needs it (its iceberg/hudi-on-HMS tables are + // MvccTable, and the GSON single-row maps "HMSExternalTable" -> PluginDrivenMvccExternalTable, so + // buildTableInternal selects the Mvcc subclass from this catalog-level capability). Declared HERE + // together with its MTMV freshness machinery: HiveConnectorMetadata.getTableFreshness / + // getPartitionFreshnessMillis surface hive's last-modified freshness (transient_lastDdlTime), which + // PluginDrivenMvccExternalTable wraps into MTMVMaxTimestampSnapshot / MTMVTimestampSnapshot on the + // MTMV refresh path (byte-parity with legacy HiveDlaTable) — so a plain-hive base table's MV detects + // change instead of pinning a constant. Plain-hive stays non-MVCC per table (beginQuerySnapshot + // default-empty -> empty pin -> scan reads current); iceberg/hudi-on-HMS keep their snapshot-id + // freshness through the sibling delegation substep. (Fixes the earlier "hive is non-MVCC" note: hive + // is non-MVCC per table, but the catalog-level flag is required for the mixed catalog.) // // Deliberately NOT declared here: - // - SUPPORTS_MVCC_SNAPSHOT: the heterogeneous hms catalog DOES need it (its iceberg/hudi-on-HMS tables - // are MvccTable, and the GSON single-row maps "HMSExternalTable" -> PluginDrivenMvccExternalTable, so - // buildTableInternal selects the Mvcc subclass from this catalog-level capability). It is declared - // together with the MVCC/MTMV machinery (freshness-aware getTableSnapshot + hive empty-pin + - // iceberg sibling delegation) in that substep, NOT as a bare flag here — declaring it now without the - // machinery would give plain-hive tables an eager empty snapshot with no freshness. (This is the fix - // to the earlier "hive is non-MVCC" note: hive is non-MVCC per table, but the catalog-level flag is - // required for the mixed catalog.) // - SUPPORTS_SHOW_CREATE_DDL: the connector must first emit the table location (show.location) and a // generic-vs-hive-specific SHOW CREATE rendering must be decided — its own substep. // - SUPPORTS_PASSTHROUGH_QUERY / SUPPORTS_PARTITION_STATS: hive exposes no query() TVF, and legacy SHOW @@ -131,7 +134,8 @@ public Set getCapabilities() { return EnumSet.of( ConnectorCapability.SUPPORTS_VIEW, ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE, - ConnectorCapability.SUPPORTS_METADATA_PRELOAD); + ConnectorCapability.SUPPORTS_METADATA_PRELOAD, + ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT); } private HmsClient getOrCreateClient() { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index e918f3c9f1eaa8..1fad270dc9ef93 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -35,6 +35,8 @@ import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.api.pushdown.ConnectorAnd; import org.apache.doris.connector.api.pushdown.ConnectorComparison; import org.apache.doris.connector.api.pushdown.ConnectorExpression; @@ -76,6 +78,7 @@ import java.util.Locale; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; import java.util.Set; import java.util.function.ToLongFunction; import java.util.stream.Collectors; @@ -843,6 +846,119 @@ private static Map toPartitionValueMap(String partitionName, Lis return valueMap; } + // ========== MTMV freshness (last-modified; MTMV refresh path only, NOT the scan hot path) ========== + + /** HMS parameter carrying a table/partition's last-DDL time in SECONDS (byte-parity with legacy hive). */ + private static final String TRANSIENT_LAST_DDL_TIME = "transient_lastDdlTime"; + + /** + * The query-begin pin for a hive table: a non-MVCC EMPTY pin (snapshot id {@code -1}, no scan options — so + * {@code applySnapshot} is a no-op and the scan reads current) but flagged {@code lastModifiedFreshness} so + * the generic model serves this table's MTMV table/partition snapshots from {@link #getTableFreshness} / + * {@link #getPartitionFreshnessMillis} (last-modified) instead of pinning a constant snapshot id. The flag + * rides on the pin so fe-core reads it off the pin it already holds — a snapshot-id connector never fires + * the freshness probe. (Iceberg/hudi-on-HMS delegation, which returns a real snapshot-id pin for those + * handles, lands with the sibling-connector substep; until then every hive-connector handle is last-modified.) + */ + @Override + public Optional beginQuerySnapshot(ConnectorSession session, + ConnectorTableHandle handle) { + return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(-1L).lastModifiedFreshness(true).build()); + } + + /** + * Whole-table MTMV freshness for hive: the table's newest modify time, wrapped by fe-core into an + * {@code MTMVMaxTimestampSnapshot} (byte-parity with legacy {@code HiveDlaTable.getTableSnapshot}). + * Hive's whole-table change signal is a last-modified TIMESTAMP, never a snapshot id. + * + *
      + *
    • Unpartitioned ⇒ the table's {@code transient_lastDdlTime} (already on the handle, no + * round-trip), named by the table.
      • + *
    • Partitioned ⇒ the max {@code transient_lastDdlTime} over all partitions, named by the + * partition owning it (an empty partition set ⇒ {@code (tableName, 0)}). This pays a + * {@code get_partitions_by_names} round-trip, which is why this lives on the MTMV path, NOT + * {@link #listPartitions} (the scan hot path stays names-only).
      • + *
    + */ + @Override + public Optional getTableFreshness(ConnectorSession session, + ConnectorTableHandle handle) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + List partKeyNames = hiveHandle.getPartitionKeyNames(); + if (partKeyNames == null || partKeyNames.isEmpty()) { + // Parity HiveDlaTable.getTableSnapshot UNPARTITIONED branch: MTMVMaxTimestampSnapshot(name, lastDdl). + return Optional.of(new ConnectorTableFreshness(hiveHandle.getTableName(), + lastDdlMillis(hiveHandle.getTableParameters()))); + } + List partitionNames = collectPartitionNames(hiveHandle); + if (partitionNames.isEmpty()) { + // Parity: an empty partition list yields MTMVMaxTimestampSnapshot(tableName, 0). + return Optional.of(new ConnectorTableFreshness(hiveHandle.getTableName(), 0L)); + } + List partitions = + hmsClient.getPartitions(hiveHandle.getDbName(), hiveHandle.getTableName(), partitionNames); + String maxName = hiveHandle.getTableName(); + long maxMillis = 0L; + for (HmsPartitionInfo partition : partitions) { + long millis = lastDdlMillis(partition.getParameters()); + // Strictly-greater keeps the FIRST partition on a tie (parity HiveDlaTable's `> maxVersionTime`). + if (millis > maxMillis) { + maxMillis = millis; + maxName = renderPartitionName(partKeyNames, partition.getValues()); + } + } + return Optional.of(new ConnectorTableFreshness(maxName, maxMillis)); + } + + /** + * Per-partition last-modified millis for hive (parity {@code HiveDlaTable.getPartitionSnapshot} -> + * {@code MTMVTimestampSnapshot(hivePartition.getLastModifiedTime())}). Fetched on demand on the MTMV + * refresh path — {@link #listPartitions} withholds it (names-only) to keep partitioned queries cheap. + * fe-core has already validated the partition exists in the materialized set; an {@code empty} return + * therefore means the partition VANISHED between the materialize and this fetch (a refresh-time race), and + * fe-core raises the legacy "can not find partition" error (parity {@code HiveDlaTable.checkPartitionExists}). + */ + @Override + public OptionalLong getPartitionFreshnessMillis(ConnectorSession session, ConnectorTableHandle handle, + String partitionName) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + List partitions = hmsClient.getPartitions(hiveHandle.getDbName(), + hiveHandle.getTableName(), Collections.singletonList(partitionName)); + if (partitions.isEmpty()) { + return OptionalLong.empty(); + } + return OptionalLong.of(lastDdlMillis(partitions.get(0).getParameters())); + } + + /** + * The last-DDL time in MILLIS from an HMS parameter map, byte-parity with legacy + * {@code HivePartition.getLastModifiedTime} / {@code HMSExternalTable.getLastDdlTime}: the + * {@code transient_lastDdlTime} value (seconds) times 1000, or 0 when the parameter is absent. + */ + private static long lastDdlMillis(Map parameters) { + if (parameters == null || !parameters.containsKey(TRANSIENT_LAST_DDL_TIME)) { + return 0L; + } + return Long.parseLong(parameters.get(TRANSIENT_LAST_DDL_TIME)) * 1000; + } + + /** + * Renders {@code key1=v1/key2=v2} from partition-key names + values, byte-parity with legacy + * {@code HivePartition.getPartitionName} (a raw, unescaped {@code name=value} join). Used only to label + * the {@code MTMVMaxTimestampSnapshot} with the partition owning the max modify time. + */ + private static String renderPartitionName(List partKeyNames, List values) { + StringBuilder sb = new StringBuilder(); + int n = Math.min(partKeyNames.size(), values.size()); + for (int i = 0; i < n; i++) { + if (i != 0) { + sb.append("/"); + } + sb.append(partKeyNames.get(i)).append("=").append(values.get(i)); + } + return sb.toString(); + } + // ========== ConnectorSchemaOps: DDL writes (create/drop database) ========== /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java index 7e8428c2d4929d..d0c16b0a9818f4 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java @@ -51,16 +51,13 @@ public void declaresLegacyParityCapabilities() { // Legacy HMSExternalTable.supportsExternalMetadataPreload() returned true. Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_METADATA_PRELOAD), "SUPPORTS_METADATA_PRELOAD: legacy HMS tables were preload-eligible"); - } - - @Test - public void withholdsMvccUntilMvccMachineryLands() { - // The mixed hms catalog ultimately NEEDS this (iceberg/hudi-on-HMS are MvccTable), but it is declared - // together with the MVCC/MTMV machinery (freshness-aware getTableSnapshot + empty-pin + sibling - // delegation), not as a bare flag. Declaring it now would give plain-hive tables an eager empty - // snapshot with no freshness. - Assertions.assertFalse(capabilities().contains(ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT), - "SUPPORTS_MVCC_SNAPSHOT must land with the MVCC machinery, not here"); + // The mixed hms catalog needs MVCC (iceberg/hudi-on-HMS are MvccTable; GSON maps HMSExternalTable -> + // PluginDrivenMvccExternalTable, and buildTableInternal selects the Mvcc subclass from this + // catalog-level flag). Declared TOGETHER with its MTMV freshness machinery + // (HiveConnectorMetadata.getTableFreshness / getPartitionFreshnessMillis surface hive's last-modified + // freshness), so a plain-hive base table's MV detects change instead of pinning a constant. + Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT), + "SUPPORTS_MVCC_SNAPSHOT: the mixed hms catalog needs it; freshness is served last-modified"); } @Test diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFreshnessTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFreshnessTest.java new file mode 100644 index 00000000000000..b552215ea0f5de --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFreshnessTest.java @@ -0,0 +1,301 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.OptionalLong; + +/** + * Tests {@link HiveConnectorMetadata#getTableFreshness} / {@link HiveConnectorMetadata#getPartitionFreshnessMillis} + * (HMS cutover — MVCC/MTMV substep, dormant). + * + *

    Why these matter: a flipped hive table is a {@code PluginDrivenMvccExternalTable}, and MTMV + * change-detection over it must vary with the source. Hive's whole-table / per-partition change signal is a + * last-modified TIMESTAMP ({@code transient_lastDdlTime}), NOT a snapshot id — so these methods must reproduce + * legacy {@code HiveDlaTable}:

    + *
      + *
    • table freshness = the table's last-DDL time (unpartitioned) or the max partition modify time + * (partitioned), carrying the owning partition name so dropping it is detected as a change;
      • + *
    • the seconds->millis (×1000) conversion and absent->0 policy live connector-side (fe-core must + * not parse the raw HMS property);
      • + *
    • the unpartitioned table pays NO {@code get_partitions_by_names} round-trip (the time is already on the + * handle), and per-partition freshness is fetched on demand here (the MTMV path), never in the names-only + * {@code listPartitions} hot path.
      • + *
    + */ +public class HiveConnectorMetadataFreshnessTest { + + private static final List PART_KEYS = Arrays.asList("year", "month"); + private static final String TRANSIENT_LAST_DDL_TIME = "transient_lastDdlTime"; + + private HiveConnectorMetadata metadata(FakeHmsClient client) { + return new HiveConnectorMetadata(client, Collections.emptyMap(), new FakeConnectorContext()); + } + + private HiveTableHandle partitionedHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(PART_KEYS) + .build(); + } + + private HiveTableHandle unpartitionedHandle(Map tableParams) { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.emptyList()) + .tableParameters(tableParams) + .build(); + } + + // ==================== table freshness: unpartitioned ==================== + + @Test + public void testUnpartitionedUsesTableLastDdlTimeAndNoRoundTrip() { + FakeHmsClient client = new FakeHmsClient(); + // transient_lastDdlTime is in SECONDS; the connector must return millis (x1000), parity + // HMSExternalTable.getLastDdlTime. + ConnectorTableFreshness freshness = metadata(client).getTableFreshness(null, + unpartitionedHandle(Collections.singletonMap(TRANSIENT_LAST_DDL_TIME, "100"))).orElse(null); + + Assertions.assertNotNull(freshness); + Assertions.assertEquals("t", freshness.getName(), + "an unpartitioned table's freshness is named by the table (parity MTMVMaxTimestampSnapshot)"); + Assertions.assertEquals(100_000L, freshness.getTimestampMillis(), + "transient_lastDdlTime seconds must be converted to millis"); + // The time is already on the handle; touching the metastore here would be a needless round-trip. + Assertions.assertFalse(client.listPartitionNamesCalled, + "an unpartitioned table must not list partitions for freshness"); + Assertions.assertFalse(client.getPartitionsCalled, + "an unpartitioned table must not fetch partitions for freshness"); + } + + @Test + public void testUnpartitionedAbsentParamReturnsZero() { + // Absent transient_lastDdlTime -> 0 (parity HMSExternalTable.getLastDdlTime, e.g. hive views). + ConnectorTableFreshness freshness = metadata(new FakeHmsClient()).getTableFreshness(null, + unpartitionedHandle(Collections.emptyMap())).orElse(null); + Assertions.assertNotNull(freshness); + Assertions.assertEquals("t", freshness.getName()); + Assertions.assertEquals(0L, freshness.getTimestampMillis(), + "an absent transient_lastDdlTime must yield 0, not a crash"); + } + + // ==================== table freshness: partitioned ==================== + + @Test + public void testPartitionedReturnsMaxModifyTimeAndOwningPartitionName() { + FakeHmsClient client = new FakeHmsClient() + .partition("year=2023/month=12", 100L) + .partition("year=2024/month=01", 300L) // the max + .partition("year=2024/month=02", 200L); + + ConnectorTableFreshness freshness = + metadata(client).getTableFreshness(null, partitionedHandle()).orElse(null); + + Assertions.assertNotNull(freshness); + // Parity HiveDlaTable.getTableSnapshot: max(partition lastModifiedTime) + the owning partition NAME, + // rendered from values (raw key=value join, parity HivePartition.getPartitionName). + Assertions.assertEquals("year=2024/month=01", freshness.getName(), + "the freshness must be named by the partition owning the max modify time"); + Assertions.assertEquals(300_000L, freshness.getTimestampMillis(), + "the freshness millis must be the max transient_lastDdlTime x 1000"); + } + + @Test + public void testPartitionedEmptyPartitionSetReturnsTableNameZero() { + // No partitions -> MTMVMaxTimestampSnapshot(tableName, 0) parity HiveDlaTable.getTableSnapshot. + ConnectorTableFreshness freshness = + metadata(new FakeHmsClient()).getTableFreshness(null, partitionedHandle()).orElse(null); + Assertions.assertNotNull(freshness); + Assertions.assertEquals("t", freshness.getName()); + Assertions.assertEquals(0L, freshness.getTimestampMillis()); + } + + @Test + public void testPartitionedTieKeepsFirstMax() { + // Two partitions share the max: strictly-greater comparison keeps the FIRST (parity HiveDlaTable's + // `> maxVersionTime`), so the earlier-listed partition name wins. + FakeHmsClient client = new FakeHmsClient() + .partition("year=2024/month=01", 300L) + .partition("year=2024/month=02", 300L); + ConnectorTableFreshness freshness = + metadata(client).getTableFreshness(null, partitionedHandle()).orElse(null); + Assertions.assertNotNull(freshness); + Assertions.assertEquals("year=2024/month=01", freshness.getName(), + "a tie on the max modify time must keep the first partition (strictly-greater)"); + Assertions.assertEquals(300_000L, freshness.getTimestampMillis()); + } + + // ==================== per-partition freshness ==================== + + @Test + public void testPartitionFreshnessMillis() { + FakeHmsClient client = new FakeHmsClient() + .partition("year=2024/month=01", 300L); + OptionalLong millis = metadata(client).getPartitionFreshnessMillis(null, partitionedHandle(), + "year=2024/month=01"); + Assertions.assertTrue(millis.isPresent()); + Assertions.assertEquals(300_000L, millis.getAsLong(), + "per-partition freshness is transient_lastDdlTime x 1000 (parity HivePartition.getLastModifiedTime)"); + } + + @Test + public void testPartitionFreshnessMillisAbsentParamZero() { + FakeHmsClient client = new FakeHmsClient().partitionNoParam("year=2024/month=01"); + OptionalLong millis = metadata(client).getPartitionFreshnessMillis(null, partitionedHandle(), + "year=2024/month=01"); + Assertions.assertTrue(millis.isPresent()); + Assertions.assertEquals(0L, millis.getAsLong(), + "an absent transient_lastDdlTime on a partition must yield 0"); + } + + @Test + public void testPartitionFreshnessMillisVanishedPartitionReturnsEmpty() { + // A partition that vanished between the materialize (existence check) and this fetch (a rare + // refresh-time race): return EMPTY so fe-core raises the legacy "can not find partition" (parity + // HiveDlaTable.checkPartitionExists), rather than emitting a bogus MTMVTimestampSnapshot(0). + OptionalLong millis = metadata(new FakeHmsClient()).getPartitionFreshnessMillis(null, + partitionedHandle(), "year=2024/month=01"); + Assertions.assertFalse(millis.isPresent(), + "a vanished partition must yield empty (fe-core throws 'can not find partition')"); + } + + // ==================== query-begin pin: flags last-modified freshness ==================== + + @Test + public void testBeginQuerySnapshotIsEmptyPinFlaggedLastModified() { + // Hive's query-begin pin is a non-MVCC EMPTY pin (snapshot id -1, no scan options) but FLAGGED + // lastModifiedFreshness, so the generic model serves this table's MTMV snapshots from the last-modified + // freshness SPI instead of pinning a constant snapshot id. The flag rides on the pin so a snapshot-id + // connector (which leaves it false) never fires the freshness probe. + ConnectorMvccSnapshot pin = metadata(new FakeHmsClient()) + .beginQuerySnapshot(null, partitionedHandle()).orElse(null); + Assertions.assertNotNull(pin); + Assertions.assertEquals(-1L, pin.getSnapshotId(), + "hive's pin is the empty (-1) pin: scan reads current (applySnapshot is a no-op)"); + Assertions.assertTrue(pin.isLastModifiedFreshness(), + "hive's pin must flag last-modified freshness so MTMV freshness comes from the on-demand SPI"); + } + + /** + * Minimal {@link HmsClient} double: records the requested partitions by name and returns their + * parameters (with {@code transient_lastDdlTime}) via {@code getPartitions}; {@code listPartitionNames} + * returns the registered names. Records which of the two metastore calls were made. + */ + private static final class FakeHmsClient implements HmsClient { + // Insertion-ordered so getPartitions returns partitions in registration order (drives the tie test). + private final Map partitionDdlSeconds = new LinkedHashMap<>(); + private final List paramlessPartitions = new ArrayList<>(); + private boolean listPartitionNamesCalled; + private boolean getPartitionsCalled; + + FakeHmsClient partition(String name, long ddlSeconds) { + partitionDdlSeconds.put(name, ddlSeconds); + return this; + } + + FakeHmsClient partitionNoParam(String name) { + paramlessPartitions.add(name); + return this; + } + + private HmsPartitionInfo toInfo(String name) { + List values = HiveWriteUtils.toPartitionValues(name); + Map params; + if (paramlessPartitions.contains(name)) { + params = Collections.emptyMap(); + } else { + params = Collections.singletonMap(TRANSIENT_LAST_DDL_TIME, + Long.toString(partitionDdlSeconds.get(name))); + } + return new HmsPartitionInfo(values, "loc", "if", "of", "serde", params); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + listPartitionNamesCalled = true; + List names = new ArrayList<>(partitionDdlSeconds.keySet()); + names.addAll(paramlessPartitions); + return names; + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + getPartitionsCalled = true; + List result = new ArrayList<>(); + for (String name : partNames) { + if (partitionDdlSeconds.containsKey(name) || paramlessPartitions.contains(name)) { + result.add(toInfo(name)); + } + } + return result; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java index 547ac2c68352f4..870878a181b5e9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java @@ -38,12 +38,14 @@ import org.apache.doris.connector.api.mvcc.ConnectorMvccPartition; import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.datasource.hive.HiveUtil; import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.MvccTable; import org.apache.doris.datasource.mvcc.MvccUtil; import org.apache.doris.mtmv.MTMVBaseTableIf; +import org.apache.doris.mtmv.MTMVMaxTimestampSnapshot; import org.apache.doris.mtmv.MTMVRefreshContext; import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.mtmv.MTMVSnapshotIdSnapshot; @@ -580,13 +582,29 @@ public MTMVSnapshotIf getPartitionSnapshot(String partitionName, MTMVRefreshCont if (value == null) { throw new AnalysisException("can not find partition: " + partitionName); } - // Range-view path with snapshot-id freshness pins the per-partition snapshot id (parity master - // IcebergExternalTable.getPartitionSnapshot -> MTMVSnapshotIdSnapshot). The connector pre-resolved the - // `<= 0 -> table snapshot id` fallback, so a non-empty table never carries a non-positive value here. - // The legacy path keeps the last-modified-millis timestamp (paimon parity). - return pin.isSnapshotIdFreshness() - ? new MTMVSnapshotIdSnapshot(value) - : new MTMVTimestampSnapshot(value); + if (pin.isSnapshotIdFreshness()) { + // Range-view path with snapshot-id freshness pins the per-partition snapshot id (parity master + // IcebergExternalTable.getPartitionSnapshot -> MTMVSnapshotIdSnapshot). The connector pre-resolved + // the `<= 0 -> table snapshot id` fallback, so a non-empty table never carries a non-positive value. + return new MTMVSnapshotIdSnapshot(value); + } + if (pin.getConnectorSnapshot().isLastModifiedFreshness()) { + // Last-modified connector (e.g. hive): the REAL per-partition modify time is not in the pin — the + // connector's listPartitions is names-only on the scan hot path (pin value is the -1 sentinel) — so + // fetch it on demand here, on the MTMV refresh path only (parity legacy HiveDlaTable + // .getPartitionSnapshot -> MTMVTimestampSnapshot(partition.getLastModifiedTime())). The pin flag + // gates this, so a snapshot-id connector (paimon/iceberg) NEVER reaches the probe. An empty return + // means the partition vanished after the materialize-time existence check above (a refresh-time + // race): raise the legacy "can not find partition" (parity checkPartitionExists). + OptionalLong onDemand = queryPartitionFreshnessMillis(partitionName); + if (!onDemand.isPresent()) { + throw new AnalysisException("can not find partition: " + partitionName); + } + return new MTMVTimestampSnapshot(onDemand.getAsLong()); + } + // Pin-timestamp connector (paimon): the pin's per-partition last-modified millis is authoritative + // (byte-unchanged — no probe). + return new MTMVTimestampSnapshot(value); } @Override @@ -597,7 +615,84 @@ public MTMVSnapshotIf getTableSnapshot(MTMVRefreshContext context, Optional snapshot) throws AnalysisException { - return new MTMVSnapshotIdSnapshot(getOrMaterialize(snapshot).getConnectorSnapshot().getSnapshotId()); + // Freshness-kind-aware (mirrors getPartitionSnapshot), gated by the pin fe-core already holds so a + // snapshot-id connector pays ZERO extra metadata calls. A last-modified connector (e.g. hive, whose + // whole-table change signal is transient_lastDdlTime / the max partition modify time, NOT a snapshot + // id) flags the query-begin pin; fe-core then wraps getTableFreshness into an MTMVMaxTimestampSnapshot + // (byte-parity with legacy HiveDlaTable.getTableSnapshot). WITHOUT this a plain-hive empty pin's + // snapshot id is a constant -1, so an MV over a hive base table would never detect change. A snapshot-id + // connector (paimon/iceberg) leaves the flag false and keeps the snapshot-id table snapshot, taking the + // EXACT pre-change path (getOrMaterialize was already required for the id — no added round-trip). + PluginDrivenMvccSnapshot pin = getOrMaterialize(snapshot); + if (pin.getConnectorSnapshot().isLastModifiedFreshness()) { + Optional tableFreshness = queryTableFreshness(); + if (tableFreshness.isPresent()) { + return new MTMVMaxTimestampSnapshot(tableFreshness.get().getName(), + tableFreshness.get().getTimestampMillis()); + } + } + return new MTMVSnapshotIdSnapshot(pin.getConnectorSnapshot().getSnapshotId()); + } + + // ──────────────────── on-demand freshness (last-modified connectors) ──────────────────── + + /** + * Whole-table freshness from a last-modified connector (present only for e.g. hive), or empty for a + * snapshot-id connector / a dropped catalog/table. See {@link ConnectorMetadata#getTableFreshness}. + */ + private Optional queryTableFreshness() { + Optional probe = resolveFreshnessProbe(); + if (!probe.isPresent()) { + return Optional.empty(); + } + FreshnessProbe p = probe.get(); + return p.metadata.getTableFreshness(p.session, p.handle); + } + + /** + * Per-partition last-modified millis from a last-modified connector (present only for e.g. hive), or + * empty for a snapshot-id connector / a dropped catalog/table. See + * {@link ConnectorMetadata#getPartitionFreshnessMillis}. + */ + private OptionalLong queryPartitionFreshnessMillis(String partitionName) { + Optional probe = resolveFreshnessProbe(); + if (!probe.isPresent()) { + return OptionalLong.empty(); + } + FreshnessProbe p = probe.get(); + return p.metadata.getPartitionFreshnessMillis(p.session, p.handle, partitionName); + } + + /** + * Resolves (session, metadata, handle) for an on-demand freshness probe, or empty when the catalog/table + * is gone (degrade to the snapshot-id / pin path). Unlike {@link #materializeLatest} this lists NOTHING + * and pins NOTHING — a last-modified connector fetches only the freshness it needs, off the scan hot path. + */ + private Optional resolveFreshnessProbe() { + makeSureInitialized(); + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + if (connector == null) { + return Optional.empty(); + } + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + return handleOpt.map(handle -> new FreshnessProbe(session, metadata, handle)); + } + + /** Bundles the (session, metadata, handle) an on-demand freshness probe needs. */ + private static final class FreshnessProbe { + private final ConnectorSession session; + private final ConnectorMetadata metadata; + private final ConnectorTableHandle handle; + + private FreshnessProbe(ConnectorSession session, ConnectorMetadata metadata, + ConnectorTableHandle handle) { + this.session = session; + this.metadata = metadata; + this.handle = handle; + } } @Override diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java index 0c3346ee0d0310..3195375f4d344e 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java @@ -38,9 +38,11 @@ import org.apache.doris.connector.api.mvcc.ConnectorMvccPartition; import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.MvccTableInfo; +import org.apache.doris.mtmv.MTMVMaxTimestampSnapshot; import org.apache.doris.mtmv.MTMVSnapshotIdSnapshot; import org.apache.doris.mtmv.MTMVTimestampSnapshot; import org.apache.doris.nereids.StatementContext; @@ -121,6 +123,132 @@ public void testGetPartitionSnapshotMissingThrows() { "an unknown partition name must raise AnalysisException, not silently succeed"); } + // ==================== last-modified freshness (e.g. hive): table + partition snapshots ==================== + + /** + * Re-stubs {@code beginQuerySnapshot} so the query-begin pin advertises last-modified freshness (the flag a + * hive connector sets). fe-core reads this off the pin to decide whether to serve MTMV freshness from the + * on-demand SPI (hive) vs the snapshot id (paimon/iceberg). + */ + private static void flagPinLastModified(Fixture f) { + Mockito.when(f.metadata.beginQuerySnapshot(f.session, f.handle)) + .thenReturn(Optional.of(ConnectorMvccSnapshot.builder() + .snapshotId(-1L).lastModifiedFreshness(true).build())); + } + + @Test + public void testGetTableSnapshotLastModifiedEmitsMaxTimestampSnapshot() throws AnalysisException { + // A last-modified connector (hive) flags its pin and reports whole-table freshness via getTableFreshness; + // fe-core must wrap it in MTMVMaxTimestampSnapshot (byte-parity with legacy HiveDlaTable.getTableSnapshot), + // NOT the snapshot-id token. Without this a plain-hive empty pin's snapshot id is a constant -1, so an MV + // over a hive base table would compare equal forever and never refresh. + Fixture f = Fixture.partitioned(); + flagPinLastModified(f); + Mockito.when(f.metadata.getTableFreshness(Mockito.any(), Mockito.any())) + .thenReturn(Optional.of(new ConnectorTableFreshness("dt=2024-02-02", TS_2024_02_02))); + + // MUTATION: keeping the hardcoded MTMVSnapshotIdSnapshot (ignoring getTableFreshness) makes this cast + // throw ClassCastException -> red. + MTMVMaxTimestampSnapshot snap = + (MTMVMaxTimestampSnapshot) f.table.getTableSnapshot(Optional.empty()); + // MTMVMaxTimestampSnapshot.equals compares BOTH the partition name and the timestamp (the name guards + // against dropping the partition that owns the max time). MUTATION: dropping the name or the millis makes + // this red. + Assertions.assertEquals(new MTMVMaxTimestampSnapshot("dt=2024-02-02", TS_2024_02_02), snap, + "a last-modified connector's table snapshot must carry (max-partition-name, max-modify-millis)"); + } + + @Test + public void testGetTableSnapshotContextOverloadAlsoLastModified() throws AnalysisException { + // The MTMVRefreshContext overload (used by MTMVPartitionUtil.getTableSnapshotFromContext) must route to + // the same freshness-aware path, not a separate hardcoded one. + Fixture f = Fixture.partitioned(); + flagPinLastModified(f); + Mockito.when(f.metadata.getTableFreshness(Mockito.any(), Mockito.any())) + .thenReturn(Optional.of(new ConnectorTableFreshness("t", 4242L))); + MTMVMaxTimestampSnapshot snap = + (MTMVMaxTimestampSnapshot) f.table.getTableSnapshot(null, Optional.empty()); + Assertions.assertEquals(new MTMVMaxTimestampSnapshot("t", 4242L), snap); + } + + @Test + public void testGetPartitionSnapshotLastModifiedUsesOnDemandNotPin() throws AnalysisException { + // A last-modified connector withholds per-partition modify time from listPartitions (names-only hot + // path), so the pin carries the -1 UNKNOWN sentinel; getPartitionSnapshot must take the REAL time from + // the on-demand getPartitionFreshnessMillis, not the pin. + Fixture f = Fixture.with(Collections.singletonList( + cpi("dt=2024-01-01", ConnectorPartitionInfo.UNKNOWN))); + flagPinLastModified(f); + Mockito.when(f.metadata.getPartitionFreshnessMillis(Mockito.any(), Mockito.any(), + Mockito.eq("dt=2024-01-01"))).thenReturn(OptionalLong.of(TS_2024_01_01)); + + MTMVTimestampSnapshot ts = (MTMVTimestampSnapshot) f.table.getPartitionSnapshot( + "dt=2024-01-01", null, Optional.empty()); + // MUTATION: reading the pin value (-1) instead of the on-demand fetch makes this red (would be -1), + // which would make every partition compare equal forever (stale MV at partition granularity). + Assertions.assertEquals(TS_2024_01_01, ts.getSnapshotVersion(), + "a last-modified connector's partition snapshot must use the on-demand millis, not the pin's -1"); + } + + @Test + public void testGetPartitionSnapshotLastModifiedMissingStillThrows() { + // Existence is validated against the materialized partition set BEFORE the on-demand fetch, so even a + // last-modified connector raises AnalysisException for an unknown partition (parity legacy + // HiveDlaTable.getPartitionSnapshot -> checkPartitionExists). + Fixture f = Fixture.partitioned(); + flagPinLastModified(f); + Mockito.when(f.metadata.getPartitionFreshnessMillis(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(OptionalLong.of(TS_2024_01_01)); + Assertions.assertThrows(AnalysisException.class, + () -> f.table.getPartitionSnapshot("dt=1999-12-31", null, Optional.empty()), + "an unknown partition must throw even for a last-modified connector (existence checked first)"); + } + + @Test + public void testGetPartitionSnapshotLastModifiedVanishedThrows() { + // The partition IS in the materialized set (existence check passes) but VANISHED before the on-demand + // fetch (a refresh-time race), so getPartitionFreshnessMillis returns empty -> fe-core must raise the + // legacy "can not find partition", NOT emit a bogus MTMVTimestampSnapshot(0). MUTATION: falling back to + // MTMVTimestampSnapshot(0) instead of throwing makes this red. + Fixture f = Fixture.with(Collections.singletonList( + cpi("dt=2024-01-01", ConnectorPartitionInfo.UNKNOWN))); + flagPinLastModified(f); + Mockito.when(f.metadata.getPartitionFreshnessMillis(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(OptionalLong.empty()); + Assertions.assertThrows(AnalysisException.class, + () -> f.table.getPartitionSnapshot("dt=2024-01-01", null, Optional.empty()), + "a vanished partition (on-demand empty) must throw, not return a bogus 0 timestamp"); + } + + // ==================== snapshot-id connectors (paimon/iceberg): NO extra freshness probe ==================== + + @Test + public void testGetTableSnapshotSnapshotIdConnectorSkipsFreshnessProbe() throws AnalysisException { + // A snapshot-id connector (paimon/iceberg) leaves the pin flag false, so getTableSnapshot must take the + // exact pre-change path: read the snapshot id off the pin and NEVER fire the freshness probe (an extra + // getTableHandle round-trip + a new throw surface on the live MTMV path). This guards the regression the + // adversarial review caught. + Fixture f = Fixture.partitioned(); // build() pin has lastModifiedFreshness=false + MTMVSnapshotIdSnapshot snap = (MTMVSnapshotIdSnapshot) f.table.getTableSnapshot(Optional.empty()); + Assertions.assertEquals(PINNED_SNAPSHOT_ID, snap.getSnapshotVersion()); + // MUTATION: dropping the pin-flag gate (probing unconditionally) makes this verify red. + Mockito.verify(f.metadata, Mockito.never()).getTableFreshness(Mockito.any(), Mockito.any()); + } + + @Test + public void testGetPartitionSnapshotSnapshotIdConnectorSkipsFreshnessProbe() throws AnalysisException { + // Same guard at partition granularity: a pin-timestamp connector (paimon) must read the pin value and + // NEVER call getPartitionFreshnessMillis (which, per-partition in the isSyncWithPartitions loop, would be + // an O(partitions) metadata regression). + Fixture f = Fixture.partitioned(); + MTMVTimestampSnapshot ts = (MTMVTimestampSnapshot) f.table.getPartitionSnapshot( + "dt=2024-01-01", null, Optional.empty()); + Assertions.assertEquals(TS_2024_01_01, ts.getSnapshotVersion()); + // MUTATION: probing unconditionally (no pin-flag gate) makes this verify red. + Mockito.verify(f.metadata, Mockito.never()) + .getPartitionFreshnessMillis(Mockito.any(), Mockito.any(), Mockito.any()); + } + // ==================== getNameToPartitionItems: render-from-name parity ==================== @Test @@ -1051,6 +1179,13 @@ private static Fixture build(List partitions, boolean ti ConnectorMvccSnapshot.builder().snapshotId(PINNED_SNAPSHOT_ID).build())); Mockito.when(metadata.listPartitions(Mockito.eq(session), Mockito.eq(handle), Mockito.any())) .thenReturn(partitions); + // A Mockito mock does NOT run interface default methods (returns null for these), so mimic the SPI + // default here: a snapshot-id connector (paimon/iceberg) surfaces no last-modified freshness. The + // last-modified tests below re-stub these to a present value. + Mockito.when(metadata.getTableFreshness(Mockito.any(), Mockito.any())) + .thenReturn(Optional.empty()); + Mockito.when(metadata.getPartitionFreshnessMillis(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(OptionalLong.empty()); // Single partition column "dt" (DATE by default; VARCHAR variant exercises the genuine-null // string-key path) — the LATEST schema. From 03e011c05ee20f2084f803da87c3c30863223b78 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 01:45:15 +0800 Subject: [PATCH 061/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.3=20MVCC/MTMV=20freshness=20DONE;=20sys-tables=20deferred=20t?= =?UTF-8?q?o=20flip;=20next=20=3D=20=C2=A74.4=20sibling=20delegation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.3 landed (3d784673ca4): last-modified table/partition MTMV snapshots gated by a pin flag (so paimon/iceberg — live via P5/P6, NOT dormant — pay zero extra round-trips), + SUPPORTS_MVCC_SNAPSHOT. Records the review-caught regression fix, the deferred getNewestUpdateVersionOrTime dictionary probe + sys-tables ($partitions TVF, user-signed deferral to the flip), and the accepted flip-boundary parity nuances. Next = §4.4 cross-plugin sibling SPI + gateway delegation (D3 hard precondition). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 288029cf2c8661..d19997436adbc9 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,9 +5,9 @@ --- -# 🎯 当前状态（2026-07-07） +# 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -51,12 +51,19 @@ - ⚠**翻闸期残留（两处，均**非本步引入、iceberg 已同款**，记账留翻闸的 fe-core 读路径处理，勿现在改）**：(1) **异常分类**——查询不支持格式的 hive 表抛 `DorisConnectorException`(RuntimeException)，fe-core `ConnectProcessor.handleQueryException` 无对应臂→归类 `ERR_UNKNOWN_ERROR`("maybe palo bug") 而非 legacy 的 `ERR_NOT_SUPPORTED_YET`；须翻闸时在 fe-core 读路径把 `DorisConnectorException` 映射到合适用户错误（跨连接器通用，iceberg `IcebergTypeMapping` 已同款）。(2) **存在性检查 fail-loud**——`PluginDrivenExternalCatalog.tableExist`/`CREATE IF NOT EXISTS` 经 `getTableHandle().isPresent()` 判存在，对"已存在但不支持格式的非视图表"会 throw（legacy `client.tableExists` 只回布尔不查格式）；是 PluginDriven 通用契约（iceberg/paimon 同）、且是有意的 fail-loud 方向，翻闸时定夺是否要免格式检查的存在性路径。 - **✅ §4.2a 列统计 fast-path SPI DONE（`cbf9526f776`）**（3 维对抗 review `wf_144b6d15-77f`：math-byte-parity/guards+flow/architecture，verdict `ship` 零确认缺陷，全 HEAD 核对）。架构=**连接器回传原始事实、fe-core 做 Doris 类型运算**（沿用 getTableStatistics 先例）：`convertColumnStatistics` 按 HMS variant 抽 ndv/nulls/avgColLen（string 才有 avgColLen，boolean/binary/timestamp→0/0 与 legacy 未处理分支同）；hive `getColumnStatistics` 仅 HIVE 表、需正 `numRows`（**无 spark 回退**，与表级 size 分支不对称、复刻 legacy）；fe-core `toColumnStatistic` 做 size math（string→`round(avgColLen*count)`、否则→`count*slotSize`；`avgSizeByte=dataSize/count`；min/max=builder -inf/+inf 默认）。guard 分歧仅**更安全**地倒向 empty→ANALYZE（负/畸形 numRows）。全绿（hms 4+hive 5+fe-core 4 新测；hms 56/hive 176 套 + checkstyle 0 + import gate 净）。 - ⚠**本步范围=查询规划列统计 fast-path**；**两处后续**（非本步）：①分区级列统计（`getHivePartitionColumnStats`→`HMSAnalysisTask` 的 ANALYZE 元数据 fast-path，设计 §4.2a 标"revisit `ExternalAnalysisTask` 是否走新 SPI vs SQL-only FULL")；②iceberg 连接器 `getIcebergColumnStats` port（`enable_fetch_iceberg_stats` 门，属 iceberg 连接器/§4.4，非 hive）。 -- **下一步 = §4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`**（此步声明 `SUPPORTS_MVCC_SNAPSHOT`——即 getCapabilities 步推迟的项；连带 hive 空 pin + `getPartitionSnapshot` 真 millis 收 listPartitions 步的遗留）。 +- **✅ §4.3 MVCC/MTMV 新鲜度 DONE（`3d784673ca4`）**（侦察 `wf_c50495bc-cd9` 6维+critic + 对抗复审 `wf_5ffdf7fb-1dc` 3镜头+verify，全 HEAD 核对）。声明 `SUPPORTS_MVCC_SNAPSHOT`（getCapabilities 步推迟项）+ freshness-kind-aware `getTableSnapshot`/`getPartitionSnapshot` + hive 空 pin + 真 millis（收 listPartitions 步的 -1 遗留）。**关键设计裁决（复审揪出后改正）**：`PluginDrivenMvccExternalTable` **不是休眠**——paimon/iceberg(P5/P6 已上线、声明 MVCC)**实时**走它；故新鲜度探针**必须**用放在 pin 上的 `lastModifiedFreshness` 标志门控（fe-core 读它已持有的 pin，零额外解析），否则给 paimon/iceberg 的 MTMV 每分区加一次 `getTableHandle` 往返+新抛错面（透明改写时可炸）。两个 `verify(never)` 回归测试锁死"snapshot-id 连接器不触发探针"。 + - ⚠**范围=MTMV 表/分区快照新鲜度**；**遗留（翻闸前须补/记账，休眠期不动）**： + 1. **`getNewestUpdateVersionOrTime`（字典自动刷新探针）** 对 hive 仍走 LIST-path max(pin -1 过滤)→0 常量→字典不刷新；须同根因按需取 max 分区时间（未分区语义与 getTableSnapshot 略异=0 非 lastDdlTime），且探针可能高频→须连接器缓存(D2)后再落，故未并入本步。**翻闸前须补**。 + 2. **系统表**（`$partitions` TVF vs native + iceberg-on-HMS `$snapshots`）：用户 2026-07-08 拍板**推迟到整体切换**与跨插件委派一起做（本步只做 MVCC/新鲜度；plugin 系统表只出 native，服务 hive $partitions 须新 fe-core native/TVF 路由缝，无线上消费者故顺延）。 + 3. **flip-boundary 一次性 parity 微差（复审记账，均可接受、勿现在改）**：MTMVMaxTimestampSnapshot 的 name 用**远端**表名/分区键大小写（连接器自然值），legacy 用本地 Doris 名/列——仅 name-mapping/大小写异常的目录在翻闸后首刷一次性误刷新；`renderPartitionName` arity 不匹配 Math.min 静默（legacy checkState throw，HMS 不变式下不可达）；malformed `transient_lastDdlTime` throw=legacy 平价（非回归）；全分区无 modtime→(tableName,0) 比 legacy NPE 更安全；dropped-catalog 降级为 MTMVSnapshotIdSnapshot(-1)=既有降级哲学。 + 4. **vanished-partition** 已按复审改为 on-demand empty→fe-core throw "can not find partition"（parity），非返 0。 + 5. **iceberg/hudi-on-HMS 的 snapshot-id 新鲜度**靠 §4.4 兄弟委派（gateway `beginQuerySnapshot` 对 iceberg handle 委派→非 last-modified pin；per-handle pin 标志设计天然支持混合目录）。 +- **下一步 = §4.4 兄弟连接器 SPI + 网关委派（iceberg-on-HMS，D3 翻闸硬前置）**，或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime` + 连接器缓存）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 5. ~~`HiveTableFormatDetector` 平价~~ **✅ DONE**（见上 §4.2 DONE 条目；`191449c7bd6`；3 LZO 格式 + getTableHandle fail-loud + view 短路）。 -- 后续子步：**§4.3 MVCC/系统表 + freshness-aware `getTableSnapshot`（下一步，含声明 `SUPPORTS_MVCC_SNAPSHOT`）**、§4.4 兄弟 SPI + 网关委派、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。（§4.2a 列统计 fast-path 已 DONE；其分区级 ANALYZE + iceberg col-stats 两处后续见上 §4.2a 条目。）**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 +- 后续子步：**§4.4 兄弟 SPI + 网关委派（下一步，D3 硬前置）**、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。（§4.2a 列统计 fast-path 已 DONE；§4.3 MVCC/MTMV 新鲜度已 DONE，其 `getNewestUpdateVersionOrTime` 字典探针 + 系统表两处遗留见上 §4.3 条目。）**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 ## 开场要点（承接） 1. **先读设计文档 `hms-cutover-retype-design-2026-07-07.md`（权威计划）+ 两份 findings + 本文顶部 🎯 段**。剩余 HMS 迁移 = 一次原子翻闸 + 前置休眠补齐 + 末尾循环删除。 From 040873d4ea5bbef22280e917c7b04509cbb8067b Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 05:51:31 +0800 Subject: [PATCH 062/334] [feat](catalog) cross-plugin sibling-connector SPI seam (dormant) Add ConnectorContext.createSiblingConnector(type, props) so a heterogeneous "gateway" connector (a flipped Hive-metastore catalog) can obtain a sibling connector of another type -- built in that type's OWN child-first plugin classloader and sharing this catalog's context -- to delegate iceberg-on-HMS tables to the iceberg connector without co-packaging (a 2nd AWS SDK in a distinct loader poisons S3 JVM-wide). - SPI default returns null, matching every other ConnectorContext default seam; non-gateway connectors are unaffected. - fe-core DefaultConnectorContext override forwards to ConnectorFactory.createConnector(type, props, this): passes `this` so the sibling reuses the catalog's id/auth/storage, and stays connector-agnostic (no property parsing -- the caller synthesizes the sibling props). - iceberg + paimon TcclPinningConnectorContext decorators delegate the new method to the raw context, keeping their exhaustive-pass-through contract true; the sibling applies its own TCCL/auth pin, so it must receive the unwrapped context (avoids double-pinning). Dormant: nothing calls createSiblingConnector yet (hms is not in SPI_READY_TYPES); the hive gateway substep wires it. First step of the sibling-connector delegation build-out. Tests: SPI default returns null; DefaultConnectorContext builds via the factory and passes props + this-context through (+ null on no-match / uninitialized manager); both decorators delegate the new method to the raw context. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../iceberg/TcclPinningConnectorContext.java | 9 ++ .../iceberg/RecordingConnectorContext.java | 13 ++ .../TcclPinningConnectorContextTest.java | 12 ++ .../paimon/TcclPinningConnectorContext.java | 9 ++ .../paimon/RecordingConnectorContext.java | 14 ++ .../TcclPinningConnectorContextTest.java | 19 +++ .../doris/connector/spi/ConnectorContext.java | 39 ++++++ .../connector/spi/ConnectorContextTest.java | 12 ++ .../connector/DefaultConnectorContext.java | 13 ++ .../DefaultConnectorContextSiblingTest.java | 132 ++++++++++++++++++ 10 files changed, 272 insertions(+) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextSiblingTest.java diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java index fac4ff04baa99d..2e9e013f594de0 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContext.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.iceberg; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; import org.apache.doris.connector.spi.ConnectorBrokerAddress; import org.apache.doris.connector.spi.ConnectorContext; @@ -143,6 +144,14 @@ public ConnectorMetaInvalidator getMetaInvalidator() { return delegate.getMetaInvalidator(); } + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + // Delegate to the raw engine context (not this wrapper): the sibling connector applies its OWN + // TCCL/auth pinning over the context it is handed, so it must receive the unwrapped context to avoid + // double-pinning to this plugin's loader. Keeps this decorator a true exhaustive pass-through. + return delegate.createSiblingConnector(catalogType, properties); + } + @Override public Map loadHiveConfResources(String resources) { return delegate.loadHiveConfResources(resources); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java index 792b9a686cc7e7..4799f993f7b8cd 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/RecordingConnectorContext.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.iceberg; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.spi.ConnectorBrokerAddress; import org.apache.doris.connector.spi.ConnectorContext; import org.apache.doris.connector.spi.ConnectorMetaInvalidator; @@ -146,6 +147,18 @@ public Map loadHiveConfResources(String resources) { return hiveConfResources; } + /** The type the wrapper forwarded to {@link #createSiblingConnector} (proves the decorator delegates it). */ + String lastSiblingType; + /** The properties the wrapper forwarded to {@link #createSiblingConnector}. */ + Map lastSiblingProps; + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + lastSiblingType = catalogType; + lastSiblingProps = properties; + return null; + } + @Override public long getCatalogId() { return 0; diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContextTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContextTest.java index c2ad3bf94bfb29..bbe25c11d5e3ce 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContextTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/TcclPinningConnectorContextTest.java @@ -27,6 +27,8 @@ import java.net.URL; import java.net.URLClassLoader; import java.security.PrivilegedExceptionAction; +import java.util.Collections; +import java.util.Map; /** * Verifies the split-brain guard {@link TcclPinningConnectorContext} adds to the iceberg write/DDL/procedure @@ -119,6 +121,16 @@ public void delegatesNonAuthMethods() { ctx.loadHiveConfResources("a,b"); Assertions.assertTrue(delegate.hiveConfResourcesCalled, "loadHiveConfResources must reach the delegate"); Assertions.assertEquals("a,b", delegate.lastHiveConfResourcesArg); + + // createSiblingConnector is a non-auth engine-service method: the decorator must forward it to the raw + // delegate (else a wrapped gateway context would return the SPI default null, masking a real sibling as + // "provider missing"). Assert the type + props reach the delegate unchanged. + Map siblingProps = Collections.singletonMap("iceberg.catalog.type", "hms"); + ctx.createSiblingConnector("iceberg", siblingProps); + Assertions.assertEquals("iceberg", delegate.lastSiblingType, + "createSiblingConnector type must reach the delegate (decorator is an exhaustive pass-through)"); + Assertions.assertSame(siblingProps, delegate.lastSiblingProps, + "createSiblingConnector properties must reach the delegate unchanged"); } @Test diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/TcclPinningConnectorContext.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/TcclPinningConnectorContext.java index eef676730fe41e..79917cfb91bfe4 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/TcclPinningConnectorContext.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/TcclPinningConnectorContext.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.paimon; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; import org.apache.doris.connector.spi.ConnectorBrokerAddress; import org.apache.doris.connector.spi.ConnectorContext; @@ -122,6 +123,14 @@ public ConnectorMetaInvalidator getMetaInvalidator() { return delegate.getMetaInvalidator(); } + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + // Delegate to the raw engine context (not this wrapper): the sibling connector applies its OWN + // TCCL/auth pinning over the context it is handed, so it must receive the unwrapped context to avoid + // double-pinning to this plugin's loader. Keeps this decorator a true exhaustive pass-through. + return delegate.createSiblingConnector(catalogType, properties); + } + @Override public Map loadHiveConfResources(String resources) { return delegate.loadHiveConfResources(resources); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java index 71069e8fce34ea..8a23b56e880fcd 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/RecordingConnectorContext.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.paimon; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.spi.ConnectorContext; import org.apache.doris.filesystem.properties.StorageProperties; @@ -48,6 +49,19 @@ final class RecordingConnectorContext implements ConnectorContext { /** The {@code resources} string the connector passed to {@link #loadHiveConfResources}. */ String lastHiveConfResourcesArg; + // ---- sibling-connector seam hook (proves the decorator delegates createSiblingConnector) ---- + /** The type the wrapper forwarded to {@link #createSiblingConnector}. */ + String lastSiblingType; + /** The properties the wrapper forwarded to {@link #createSiblingConnector}. */ + Map lastSiblingProps; + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + lastSiblingType = catalogType; + lastSiblingProps = properties; + return null; + } + // ---- C2: getStorageProperties hook (FE-bound fe-filesystem storage props) ---- /** Storage properties the fake returns from {@link #getStorageProperties()} (default: none). */ List storageProperties = Collections.emptyList(); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/TcclPinningConnectorContextTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/TcclPinningConnectorContextTest.java index 729b9ca1f7187d..1eee7e0052ac2c 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/TcclPinningConnectorContextTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/TcclPinningConnectorContextTest.java @@ -27,6 +27,8 @@ import java.net.URL; import java.net.URLClassLoader; import java.security.PrivilegedExceptionAction; +import java.util.Collections; +import java.util.Map; /** * Verifies the {@link TcclPinningConnectorContext} decorator the paimon connector wraps its context in: the op @@ -128,6 +130,23 @@ public void kerberosRunsTaskInPluginDoAsAndBypassesDelegateAuth() throws Excepti } } + @Test + public void delegatesSiblingConnectorToTheRawContext() { + // createSiblingConnector is a non-auth engine-service method: the decorator must forward it to the raw + // delegate (else a wrapped gateway context would return the SPI default null, masking a real sibling as + // "provider missing"). Assert the type + props reach the delegate unchanged. + RecordingConnectorContext delegate = new RecordingConnectorContext(); + TcclPinningConnectorContext ctx = new TcclPinningConnectorContext(delegate, isolatedLoader(), () -> null); + + Map siblingProps = Collections.singletonMap("iceberg.catalog.type", "hms"); + ctx.createSiblingConnector("iceberg", siblingProps); + + Assertions.assertEquals("iceberg", delegate.lastSiblingType, + "createSiblingConnector type must reach the delegate (decorator is an exhaustive pass-through)"); + Assertions.assertSame(siblingProps, delegate.lastSiblingProps, + "createSiblingConnector properties must reach the delegate unchanged"); + } + /** Wiring-only {@link HadoopAuthenticator} double: records doAs calls and runs the action WITHOUT a UGI. */ private static final class RecordingAuthenticator implements HadoopAuthenticator { int doAsCount; diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java index 5c45f131393b37..082a095960b407 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.spi; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; import org.apache.doris.filesystem.properties.StorageProperties; @@ -106,6 +107,44 @@ default ConnectorMetaInvalidator getMetaInvalidator() { return ConnectorMetaInvalidator.NOOP; } + /** + * Builds a sibling connector of another catalog type on top of this same catalog's context, for a + * heterogeneous "gateway" connector that serves more than one table format from a single catalog and must + * delegate some tables to another format's connector (e.g. a Hive-metastore catalog whose Iceberg-registered + * tables are served by the Iceberg connector). + * + *

    The engine builds the sibling through the same connector factory it uses for a top-level catalog, so the + * sibling's concrete class is loaded by that type's own plugin classloader — never co-packaged into + * the caller's plugin (a duplicate native stack, e.g. a second AWS SDK, would poison shared JVM state). The + * returned connector shares THIS context (same catalog id, authentication, and storage), so the sibling reuses + * the caller's metastore/storage/credentials without re-deriving them. + * + *

    fe-core stays connector-agnostic: this is a generic "give me a connector of type {@code catalogType} with + * these {@code properties}" factory. The caller (the gateway connector) is responsible for synthesizing the + * sibling's {@code properties} — the engine does not parse or translate them. + * + *

    Cross-plugin type safety. Because the sibling lives in a different (child-first) classloader, it is + * type-compatible with the caller ONLY through the parent-first SPI interfaces ({@link Connector}, + * {@code ConnectorMetadata}, {@code ConnectorTableHandle}, …). The caller MUST hold the result as the bare + * {@link Connector} interface and MUST NOT cast it — or any object it produces — to a concrete connector type, + * or it will {@code ClassCastException} across the loader split. + * + *

    Lifecycle. The engine tracks and closes only a catalog's primary connector; a sibling built + * here is owned by the caller, which MUST forward {@link Connector#close()} to it from its own {@code close()}. + * + *

    The default returns {@code null} (no sibling support), so every connector that is not a gateway — and the + * no-op default context — is unaffected. + * + * @param catalogType the sibling connector's type (e.g. {@code "iceberg"}); resolved by the same provider set + * the engine uses for top-level catalogs + * @param properties the sibling connector's fully-synthesized catalog properties (caller-owned) + * @return the sibling connector, or {@code null} when no provider matches {@code catalogType} (or the engine has + * no connector factory wired — e.g. the default context) + */ + default Connector createSiblingConnector(String catalogType, Map properties) { + return null; + } + /** * Resolves the catalog's {@code hive.conf.resources} (comma-separated hive-site.xml file names * under the FE's {@code hadoop_config_dir}) into a flat key->value map the connector can diff --git a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorContextTest.java b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorContextTest.java index 980ebbe79367f7..4153f73267678c 100644 --- a/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorContextTest.java +++ b/fe/fe-connector/fe-connector-spi/src/test/java/org/apache/doris/connector/spi/ConnectorContextTest.java @@ -17,11 +17,13 @@ package org.apache.doris.connector.spi; +import org.apache.doris.connector.api.Connector; import org.apache.doris.filesystem.properties.StorageProperties; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.Collections; import java.util.List; public class ConnectorContextTest { @@ -66,4 +68,14 @@ public void getBackendFileType_defaultDerivesFromScheme() { Assertions.assertEquals("FILE_LOCAL", ctx.getBackendFileType("/no/scheme", null)); Assertions.assertEquals("FILE_LOCAL", ctx.getBackendFileType(null, null)); } + + @Test + public void createSiblingConnector_defaultsToNull() { + // The cross-plugin sibling seam: only a gateway connector's context (fe-core's DefaultConnectorContext) + // overrides this to build a real sibling; every other connector keeps the default null, so introducing + // the seam must not change their behavior -- a non-gateway connector that never calls it is unaffected. + Connector sibling = minimalContext().createSiblingConnector("iceberg", Collections.emptyMap()); + Assertions.assertNull(sibling, + "default createSiblingConnector() must return null so non-gateway connectors are unaffected"); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index 41e03da2af1e68..61249a02e9f8b4 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -25,6 +25,7 @@ import org.apache.doris.common.EnvUtils; import org.apache.doris.common.Version; import org.apache.doris.common.util.LocationPath; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; import org.apache.doris.connector.spi.ConnectorBrokerAddress; import org.apache.doris.connector.spi.ConnectorContext; @@ -151,6 +152,18 @@ public ConnectorMetaInvalidator getMetaInvalidator() { return new ExternalMetaCacheInvalidator(catalogId); } + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + // Build the sibling through the SAME factory the engine uses for a top-level catalog, so the sibling's + // concrete class is loaded by that type's own plugin classloader (child-first) — never co-packaged into + // the gateway's plugin. Passing `this` lets the sibling reuse this catalog's id/auth/storage suppliers + // (correct for e.g. iceberg-on-HMS, which shares the HMS catalog's metastore + storage + credentials). + // Returns null when no provider matches the type (or the plugin manager is not initialized); the + // gateway caller null-checks and fails loud with its own (connector-specific) message — fe-core stays + // connector-agnostic and does no property parsing here. + return ConnectorFactory.createConnector(catalogType, properties, this); + } + @Override public String sanitizeJdbcUrl(String jdbcUrl) { try { diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextSiblingTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextSiblingTest.java new file mode 100644 index 00000000000000..41a9318eb92d5b --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextSiblingTest.java @@ -0,0 +1,132 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.connector.spi.ConnectorProvider; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests the fe-core override of the cross-plugin sibling-connector seam + * ({@link DefaultConnectorContext#createSiblingConnector}). The override must build the sibling through the shared + * {@link ConnectorFactory}/{@link ConnectorPluginManager} (so the sibling loads in the requested type's own plugin + * classloader) and pass THIS context through unchanged (so the sibling reuses the caller catalog's id/auth/storage). + * + *

    Dormant: no production code calls {@code createSiblingConnector} yet (the hive gateway substep does), so this + * only exercises the seam in isolation with a recording fake provider registered on the shared manager. + */ +public class DefaultConnectorContextSiblingTest { + + @AfterEach + void tearDown() { + // The plugin manager is a process-wide static; reset it so this test does not leak the recording fake + // provider into any other test that shares the ConnectorFactory singleton. + ConnectorFactory.clearPluginManager(); + } + + @Test + void createSiblingConnector_buildsViaFactory_passesPropsAndThisContext() { + RecordingProvider provider = new RecordingProvider("iceberg"); + ConnectorPluginManager manager = new ConnectorPluginManager(); + manager.registerProvider(provider); + ConnectorFactory.initPluginManager(manager); + + DefaultConnectorContext ctx = new DefaultConnectorContext("hms_catalog", 42L); + Map props = new HashMap<>(); + props.put("iceberg.catalog.type", "hms"); + props.put("hive.metastore.uris", "thrift://host:9083"); + + Connector sibling = ctx.createSiblingConnector("iceberg", props); + + Assertions.assertNotNull(sibling, "sibling must be built when a provider matches the type"); + Assertions.assertSame(provider.lastConnector, sibling, + "the sibling must be exactly the connector the matching provider produced"); + Assertions.assertSame(props, provider.lastProperties, + "the caller-synthesized properties must be forwarded to the sibling provider unchanged"); + Assertions.assertSame(ctx, provider.lastContext, + "the gateway's own context (this) must be passed to the sibling so it shares id/auth/storage"); + } + + @Test + void createSiblingConnector_returnsNull_whenNoProviderMatches() { + ConnectorPluginManager manager = new ConnectorPluginManager(); + manager.registerProvider(new RecordingProvider("iceberg")); + ConnectorFactory.initPluginManager(manager); + + DefaultConnectorContext ctx = new DefaultConnectorContext("hms_catalog", 42L); + // A gateway asking for a type no registered provider serves: fe-core returns null (connector-agnostic); + // the gateway caller is the one that fails loud. Here the only provider serves "iceberg", not "paimon". + Connector sibling = ctx.createSiblingConnector("paimon", new HashMap<>()); + + Assertions.assertNull(sibling, "no matching provider must yield null, not an exception"); + } + + @Test + void createSiblingConnector_returnsNull_whenPluginManagerUninitialized() { + // No initPluginManager -> ConnectorFactory has no manager -> null (never throws). Mirrors the pre-flip + // reality where a connector context may exist before/without the plugin manager being wired. + ConnectorFactory.clearPluginManager(); + + DefaultConnectorContext ctx = new DefaultConnectorContext("hms_catalog", 42L); + Connector sibling = ctx.createSiblingConnector("iceberg", new HashMap<>()); + + Assertions.assertNull(sibling, "uninitialized plugin manager must yield null, not an exception"); + } + + /** A ConnectorProvider that records the exact args of its last {@code create} call and the connector it made. */ + private static final class RecordingProvider implements ConnectorProvider { + private final String type; + private Map lastProperties; + private ConnectorContext lastContext; + private Connector lastConnector; + + RecordingProvider(String type) { + this.type = type; + } + + @Override + public String getType() { + return type; + } + + @Override + public Connector create(Map properties, ConnectorContext context) { + this.lastProperties = properties; + this.lastContext = context; + this.lastConnector = new RecordingConnector(); + return lastConnector; + } + } + + /** A minimal Connector — every method uses its SPI default; identity is all the test asserts on. */ + private static final class RecordingConnector implements Connector { + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + } +} From 5b6bd446afc78199471acd9c5b2ea6a6715fa7e1 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 05:53:29 +0800 Subject: [PATCH 063/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4a=20sibling-connector=20SPI=20seam=20DONE;=20=C2=A74.4=20S0-?= =?UTF-8?q?S6=20decomposition=20+=20wrapper-handle=20correction?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the sibling-connector delegation decomposition (S0-S6) from recon wf_da155752-ebb: the S0 fe-core seam is DONE (45dcca32fa8); S1 props-synthesis, S2 gateway embed (one sibling per gateway + close-forward), S3 metadata instanceof-guard-and-forward, S4 getTableHandle iceberg divert, S5 scan-provider override, S6 name-based residuals remain. Key correction: the design's gateway-owned wrapper handle is dead (the engine threads one handle object from getScanPlanProvider selection straight into IcebergScanPlanProvider's (IcebergTableHandle) cast, no unwrap seam) -> getTableHandle returns the raw foreign iceberg handle and gateway consumers discriminate by instanceof HiveTableHandle (own type), never casting the foreign handle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 7 +-- ...ling-connector-decomposition-2026-07-08.md | 47 +++++++++++++++++++ 2 files changed, 51 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d19997436adbc9..6b86593e19e438 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -58,7 +58,8 @@ 3. **flip-boundary 一次性 parity 微差（复审记账，均可接受、勿现在改）**：MTMVMaxTimestampSnapshot 的 name 用**远端**表名/分区键大小写（连接器自然值），legacy 用本地 Doris 名/列——仅 name-mapping/大小写异常的目录在翻闸后首刷一次性误刷新；`renderPartitionName` arity 不匹配 Math.min 静默（legacy checkState throw，HMS 不变式下不可达）；malformed `transient_lastDdlTime` throw=legacy 平价（非回归）；全分区无 modtime→(tableName,0) 比 legacy NPE 更安全；dropped-catalog 降级为 MTMVSnapshotIdSnapshot(-1)=既有降级哲学。 4. **vanished-partition** 已按复审改为 on-demand empty→fe-core throw "can not find partition"（parity），非返 0。 5. **iceberg/hudi-on-HMS 的 snapshot-id 新鲜度**靠 §4.4 兄弟委派（gateway `beginQuerySnapshot` 对 iceberg handle 委派→非 last-modified pin；per-handle pin 标志设计天然支持混合目录）。 -- **下一步 = §4.4 兄弟连接器 SPI + 网关委派（iceberg-on-HMS，D3 翻闸硬前置）**，或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime` + 连接器缓存）。 +- **✅ §4.4a 兄弟连接器 SPI 接缝 DONE（`45dcca32fa8`）**（recon `wf_da155752-ebb` 6维+2 critic + 对抗复审 `wf_caef8bf7-ec2` 3镜头，全 HEAD 核对；分解权威文档 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`，起步必读）。新 `ConnectorContext.createSiblingConnector(String catalogType, Map props)` 默认 **`return null`**（对齐所有其它 SPI 默认的安全默认约定——**非** throw，更正设计 §4.4 的 UnsupportedOperationException）；fe-core `DefaultConnectorContext` 覆写 = `ConnectorFactory.createConnector(catalogType, props, this)`（传 `this` → 兄弟连接器复用本目录 id/auth/storage，其 hms-flavor HiveCatalog 名默认取 `context.getCatalogName()`，故共享 context 承重；fe-core 保持 connector-agnostic、零属性解析——props 由网关合成）；iceberg+paimon `TcclPinningConnectorContext` 装饰器补委派该新方法到 raw context（保持"穷尽 pass-through"契约为真——兄弟自套 TCCL/auth pin，须收未包裹 context 避免双重 pin）。唯一改引擎的一步、后面所有插件步的地基。全绿（spi 3 + fe-core 3 + iceberg 5 + paimon 4 测 + 4 模块 checkstyle 0 + import gate 净）。**⚠关键设计更正（更正设计 §4.4）**：设计的"网关包装句柄 `{format, delegate}`"方案**已证死**——`PluginDrivenScanNode` 选执行器(:209)与执行扫描(:966)用**同一句柄对象**、中间无拆包缝，`IcebergScanPlanProvider` 无条件强转 `(IcebergTableHandle)`(:319/364/468/1009)→包装句柄必 CCE；**改为** `getTableHandle` 对 iceberg 返**原始外来句柄**（iceberg 强转成功）、网关一律用 `instanceof HiveTableHandle`(自有 hive-loader 类型)→hive、否则转发兄弟，**永不强转外来句柄**，零新 SPI。 +- **下一步 = §4.4 剩余步（休眠，序见分解文档）**：S1 hive 插件侧属性合成纯函数（注入字面量 `iceberg.catalog.type=hms`+携 `hive.metastore.uris`+透传 storage/kerberos）→ S2 `HiveConnector` 持有+懒建兄弟+`close()` 转发（**单例 per-gateway**，非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash）→ S3 元数据每 handle 方法按 `instanceof HiveTableHandle` 守卫转发（**两不相交集**：改写误服务的 hive 实现 + 转发全缺席的 iceberg 子集，后者多为**静默错答**getMvccPartitionView/resolveTimeTravel/applySnapshot/…须一并；`beginQuerySnapshot` 须与 freshness 同步分流，其 last-modified 标志门控 freshness 是否被读；apply* 须转发并原样回传兄弟句柄）→ S4 `getTableHandle` iceberg 分流（返原始外来句柄，点亮 S3）→ S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（返兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法+连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 @@ -90,7 +91,7 @@ # 🧠 起步必读 -1. **权威计划 = `hms-cutover-retype-design-2026-07-07.md`**（原子翻闸模型 + 休眠补齐清单 §4 + 能力孪生图 + 阶段序 §5 + 已签字决策 §6 + 硬门 §7；全 recon 明细在 `tool-results/w0bg9i509.output`）。再读**两份 findings 文档**（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）+ 本文顶部 🎯/🚀。 +1. **权威计划 = `hms-cutover-retype-design-2026-07-07.md`**（原子翻闸模型 + 休眠补齐清单 §4 + 能力孪生图 + 阶段序 §5 + 已签字决策 §6 + 硬门 §7；全 recon 明细在 `tool-results/w0bg9i509.output`）。**§4.4 兄弟连接器/委派的权威分解 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`**（S0–S6 序 + 更正设计 §4.4 包装句柄为死方案 + CCE/TCCL/生命周期硬约束；§4.4a=S0 已 DONE）。再读**两份 findings 文档**（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）+ 本文顶部 🎯/🚀。 2. **样板**：`P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`（净室复审 + 能力孪生审计 + GSON replay 范式）。委派/缝模板 = A 的设计文档 `P7.4-scan-provider-per-table-seam-design.md`。特性地图 = `P7-cutover-scope-map-2026-07-06.md`（排序前提已校正）。 3. **铁律**：fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/`switch(dlaType)`/引擎名判别（翻闸靠"表类=通用类 + 网关按句柄委派"，不靠在 `PhysicalPlanTranslator` 加分支）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；跨插件/跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`，事件轮询后台线程 R-010 亦需）；history_schema_info nested 名 lowercase（memory）。 4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-tracking-issue`。 diff --git a/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md b/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md new file mode 100644 index 00000000000000..232e67be767e03 --- /dev/null +++ b/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md @@ -0,0 +1,47 @@ +# HMS cutover §4.4 — sibling-connector SPI + iceberg-on-HMS gateway delegation: decomposition (2026-07-08) + +> Produced by a code-grounded recon (`wf_da155752-ebb`: 6 dimension readers + completeness critic + decomposition critic, all HEAD-verified at `0aa7812d947`) plus lead-engineer independent read of the load-bearing interfaces. **Supersedes the design doc §4.4 "gateway-owned wrapper handle" sketch** (proven dead — see §2). Authoritative plan for the remaining §4.4 dormant build-out. Trust HEAD over line numbers. + +--- + +## 1. What §4.4 delivers (recap) + +A flipped `hms` catalog is served by the **hive plugin as a GATEWAY**; its iceberg-on-HMS (and, later, hudi-on-HMS) tables are delegated to a **sibling connector built in the iceberg plugin's own child-first classloader**. Co-packaging iceberg into the hive zip is REJECTED (2nd AWS SDK poisons S3 JVM-wide — iceberg pom RC-3 comment). Cross-plugin objects share only the **parent-first SPI interfaces** (`org.apache.doris.connector.*` / `.filesystem.*`); any concrete iceberg type crossing into the hive loader CCEs. + +**Dormancy anchor (hard constraint):** the entire gateway surface is dead until `"hms"` enters `SPI_READY_TYPES` (today `{jdbc,es,trino-connector,max_compute,paimon,iceberg}`, `CatalogFactory.java:49-50`) — that single line is THE flip. **No dormant sub-step may touch `SPI_READY_TYPES`.** + +## 2. The discriminator decision — CORRECTS design §4.4 (evidence-forced) + +Design §4.4 proposed a gateway-owned **wrapper handle** `{HiveTableType format; ConnectorTableHandle delegate}`. **This is dead.** `PluginDrivenScanNode.resolveScanProvider` selects the provider via `connector.getScanPlanProvider(currentHandle)` (`:209`) and then threads the **same** `currentHandle` object into `scanProvider.planScan(...)` (`:966`); `IcebergScanPlanProvider` unconditionally casts `(IcebergTableHandle) handle` (`:319/:364/:468/:1009`). There is **no engine unwrap seam** between selection and invocation — so a wrapper handed in for selection is byte-identically what iceberg then casts → CCE. + +**⇒ Decision (the S3/S4/S5 contract):** `getTableHandle` for an ICEBERG table returns the **raw foreign iceberg handle** (the iceberg connector's own `getTableHandle` output) so iceberg's cast succeeds; every gateway consumer discriminates by **`instanceof HiveTableHandle` (the gateway's OWN hive-loader type) → hive path, else → forward to sibling**. The gateway **never** `instanceof`/casts the foreign iceberg handle. Needs **no new SPI** (no format-tag interface, no handle-identity map). `ConnectorTableHandle` is a bare marker (`interface … extends Serializable {}`, zero methods), consistent with the iron rule (fe-core never reads format). + +## 3. Ordered dormant sub-steps (each an independent休眠 commit) + +- **S0 — fe-core sibling seam `createSiblingConnector` ✅ DONE (this session).** `ConnectorContext.createSiblingConnector(String catalogType, Map properties)` default `return null` (matches every other SPI default's safe-default convention — NOT throw, correcting the design's "UnsupportedOperationException"); `DefaultConnectorContext` override = `return ConnectorFactory.createConnector(catalogType, properties, this)` (passes `this` → sibling shares catalogId/auth/storage suppliers; the sibling's HMS-flavor `HiveCatalog` name defaults to `context.getCatalogName()`, so the shared context is load-bearing). Only fe-core change in the whole feature; hard prerequisite for all plugin work (the hive plugin holds only the `ConnectorContext` interface, so this seam is its ONLY way to build a sibling that inherits the hms catalog's context, and keeps the plugin off `ConnectorFactory` statics). +- **S1 — hive-loader props-synthesis helper (pure).** New `fe-connector-hive` class: `Map synthesize(catalogProps, HmsTableInfo)` injecting **literal** `"iceberg.catalog.type"="hms"` + carrying `"hive.metastore.uris"` + passing through `fs.s3a.*/hadoop.*/dfs.*` + kerberos keys. Must hardcode the literal key/value strings (`IcebergConnectorProperties.TYPE_HMS`/`ICEBERG_CATALOG_TYPE` are child-first, invisible to the hive loader). Minimum working props for hms-flavor: `{iceberg.catalog.type=hms, hive.metastore.uris}`; `warehouse` NOT required. Independent of S0. +- **S2 — embedded-sibling holder + lazy build + close-forward.** `HiveConnector`: memoized `volatile Connector icebergSibling` = `context.createSiblingConnector("iceberg", S1.synthesize(...))`; forward `close()` to it then null. **Must be ONE sibling per gateway connector** (not per-table): `IcebergConnector` holds per-catalog caches (`latestSnapshotCache`, `manifestCache`, scan→write `rewritableDeleteStash`) shared across its tables — a per-op sibling would fragment them. Held ONLY as the parent-first `Connector` type — never cast. Deps: S0, S1. +- **S3 — metadata handle-guard-and-forward.** `HiveConnectorMetadata` every handle-based method: `if (!(handle instanceof HiveTableHandle)) return sibling.getMetadata(session).(…); `. The delegation surface is **two disjoint sets**: (a) override-and-divert the hive-implemented mis-servers (`getTableSchema[+snapshot]`, `getColumnHandles`, `getTableStatistics`, `listPartitions/Names/Values`, `beginQuerySnapshot`, `getTableFreshness`, `getPartitionFreshnessMillis`, `buildTableDescriptor`, `applyFilter`, DDL); (b) forward the wholesale-**absent** iceberg subset that today silently no-ops (`getMvccPartitionView`, `resolveTimeTravel`, `applySnapshot`, `applyRewriteFileScope`, `applyTopnLazyMaterialization`, `applyProjection`, `applyLimit`, `listSupportedSysTables`, `getSysTableHandle`, schema-at-snapshot) — several are **silent wrong answers**, not fail-loud, so a partial gateway ships silent MVCC/time-travel corruption. **`beginQuerySnapshot` must divert together with freshness** (its `isLastModifiedFreshness=true` flag gates whether `getTableFreshness` is even consulted). The handle-in/handle-out `apply*` must forward AND return the sibling's handle **unmodified** (a single un-rewrapped return poisons a downstream `planScan` cast). Deps: S2. +- **S4 — `getTableHandle` ICEBERG divert.** When `HiveTableFormatDetector.detect(...)==ICEBERG`, return the sibling's `getTableHandle` (foreign iceberg handle) instead of a HiveTableHandle stamped ICEBERG. The pivot that lights up S3 + makes iceberg's cast succeed. HUDI stays fail-loud/legacy. Deps: S3 (keep adjacent), S2. +- **S5 — scan-provider gateway override.** `HiveConnector.getScanPlanProvider(ConnectorTableHandle handle)`: `if (handle instanceof HiveTableHandle) return getScanPlanProvider(); return sibling.getScanPlanProvider(handle);`. Loader-safe negative test on the gateway's own type. Returned provider is iceberg-loader-built → scan-side TCCL auto-pins for free (`PluginDrivenScanNode.onPluginClassLoader` keys off `provider.getClass().getClassLoader()`). Deps: S2 (pairs with S4). +- **S6 — name-based + connector-wide residuals divert (thin).** Name-based methods (no handle: view methods, `buildTableDescriptor`) re-run `HiveTableFormatDetector.detect(db,table)` and forward for ICEBERG. Per-handle capability gating is NOT expressible on the connector-wide `getCapabilities` — leave documented residual for the flip (do not over-build). Deps: S2, S4. + +## 4. TCCL / classloader facts (verified) + +- **Scan pin #1** (`PluginDrivenScanNode.onPluginClassLoader`, `:516-524`): connector-agnostic, keys off `provider.getClass().getClassLoader()` → a sibling iceberg scan provider **auto-pins for free**. +- **Write/DDL pin #2** + **worker-pool pin #3**: baked INTO `IcebergConnector` (ctor wraps context in `TcclPinningConnectorContext`; `pinIcebergWorkerPoolToPluginClassLoader` once-per-JVM), keyed off `getClass().getClassLoader()`=iceberg loader → **free** for any sibling built via the factory. +- **Metadata `apply*` are NOT wrapped** in `onPluginClassLoader` (bare on the query thread's app TCCL) — BUT this is **pre-existing, not new**: `PluginDrivenExternalTable` has ZERO pins and the already-live direct iceberg SPI catalog runs the identical `IcebergConnectorMetadata` on app TCCL, relying on iceberg's own `executeAuthenticated` pins. The sibling reuses that identical metadata → inherits identical coverage. Down-graded from "risk" to "unchanged". (S3 may add a cheap gateway-side pin `sibling.getClass().getClassLoader()` around forwarded metadata as belt-and-suspenders; validate at flip.) + +## 5. Not dormant-testable → thin scaffolding + flip-time e2e gate ONLY + +1. **The CCE boundary is invisible to same-loader unit tests** — fe-core/fe-connector test classpaths can't load iceberg's child-first classes; recording fakes are app-loader types, so an illegal foreign-handle cast **passes** in unit tests. Options: build ONE small `URLClassLoader` two-loader test fixture reused across S3/S5, OR accept a flip-time redeploy classloader smoke. Budget an integration check. +2. **Real HMS→iceberg catalog build + parity** (storage/kerberos overlay, manifest cache + profile counters, count-pushdown-from-snapshot, delete-file/deletion-vector thrift, worker-pool pin, AWS-SDK isolation) — live HMS + iceberg table only. Legacy parity target = `IcebergScanNode`/`IcebergHMSSource`→fe-core embedded `HiveCatalog` (entirely bypassed by the sibling; its behaviors — pre-exec-auth wrapping, verbatim static storage map, manifest fast-path/profile lines — must be reproduced by the connector). +3. **Freshness real snapshot-id semantics** + **per-handle capability gating** (connector-wide `getCapabilities` can't express it) — verify at flip. + +## 6. OUT of the §4.4 read-delegation spine (flag, don't build here) + +`getWritePlanProvider(ConnectorTableHandle)` and `getProcedureOps(ConnectorTableHandle)` **do not exist** — write/procedure providers are obtained connector-level with NO handle (`Connector.java:74/:131`; callers resolve the provider BEFORE the handle at `PhysicalPlanTranslator:654/:702`, `ConnectorExecuteAction:116`). Routing per-table writes/EXECUTE to an iceberg sibling needs **new per-handle SPI overloads** — a separate substep AFTER the read spine, not folded in. + +## 7. References +- Recon: `wf_da155752-ebb` (full output `tasks/wuuyinpok.output`). Design: `hms-cutover-retype-design-2026-07-07.md` (§4.4 wrapper-handle bullet corrected here). Findings: `iceberg-on-hms-delegation-findings-2026-07-07.md`. +- Tracking: apache/doris#65185. From 60353e6c4660851e4b4775bea5140bd03d603b72 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 07:35:32 +0800 Subject: [PATCH 064/334] [feat](catalog) hive->iceberg sibling property synthesis (dormant) Add IcebergSiblingProperties.synthesize(catalogProps): the flipped HMS gateway will hand this map to createSiblingConnector("iceberg", ...) to build the embedded iceberg-on-HMS connector. It copies the gateway catalog's whole property map verbatim and injects the literal iceberg.catalog.type=hms. - Copy-all (not a hand-picked subset) is the robust choice: it cannot drop a connectivity key (the `uri` short form, HDFS-HA dfs.*, an S3 endpoint, a kerberos variant). It is also the exact map shape the iceberg connector already handles for a native `type=iceberg, iceberg.catalog.type=hms` catalog (fe-core builds that from the full catalog property map), and its create() path does no property validation, so hive-only keys are ignored, not misused. - The flavor key/value are hardcoded literals: iceberg's IcebergConnectorProperties constants are child-first (invisible to the hive loader). Per-catalog (the sibling is table-agnostic), pure + static (unit-testable with no connector). Dormant: no caller yet (the gateway-embed substep wires it). Tests: injects hms flavor; carries metastore/storage/HA/kerberos/conf keys (incl. the uri short form); does not mutate the input; overrides any pre-existing flavor. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../hive/IcebergSiblingProperties.java | 66 +++++++++++++ .../hive/IcebergSiblingPropertiesTest.java | 97 +++++++++++++++++++ 2 files changed, 163 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/IcebergSiblingProperties.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/IcebergSiblingPropertiesTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/IcebergSiblingProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/IcebergSiblingProperties.java new file mode 100644 index 00000000000000..607b532b79bcbc --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/IcebergSiblingProperties.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import java.util.HashMap; +import java.util.Map; + +/** + * Synthesizes the catalog-property map for the embedded Iceberg sibling connector that a flipped HMS + * gateway delegates its iceberg-on-HMS tables to. + * + *

    The sibling is built once per gateway catalog (not per table) via + * {@code ConnectorContext.createSiblingConnector("iceberg", synthesize(catalogProps))}, sharing the gateway's + * context (metastore auth + storage). The only synthesis needed is to declare the iceberg catalog flavor + * as {@code hms}: the Iceberg connector then reads {@code hive.metastore.uris}/{@code uri}, + * {@code hive.conf.resources}, and the raw {@code hive.*}/{@code fs.*}/{@code dfs.*}/{@code hadoop.*} storage + + * kerberos passthrough straight from this map. That is the SAME map shape a native + * {@code type=iceberg, iceberg.catalog.type=hms} catalog already hands the connector — fe-core builds that + * connector from the full catalog property map ({@code PluginDrivenExternalCatalog + * .createConnectorFromProperties}) — so carrying the gateway catalog's whole property map verbatim and injecting + * the flavor is both sufficient and exactly what the connector expects. Carrying the whole map (rather than a + * hand-picked subset) is also the robust choice: it cannot silently drop a connectivity key (the {@code uri} + * short form, an HDFS-HA {@code dfs.*} set, an S3 endpoint override, a kerberos variant, …). The connector + * ignores keys it does not recognize; its {@code create()} path does no property validation. + * + *

    The flavor key/value are hardcoded literals on purpose: the Iceberg connector's + * {@code IcebergConnectorProperties} constants live in the iceberg plugin's child-first classloader and are not + * visible from the hive loader. + */ +final class IcebergSiblingProperties { + + // Literals of the iceberg-plugin IcebergConnectorProperties.ICEBERG_CATALOG_TYPE / TYPE_HMS: those constants + // live in the iceberg plugin's child-first classloader and are not visible from the hive loader. + static final String ICEBERG_CATALOG_TYPE_KEY = "iceberg.catalog.type"; + static final String ICEBERG_CATALOG_TYPE_HMS = "hms"; + + private IcebergSiblingProperties() { + } + + /** + * Returns a NEW property map = the gateway catalog's properties verbatim with the iceberg catalog flavor + * forced to {@code hms}. The input is never mutated (the gateway holds it unmodifiable and shared). An + * existing {@code iceberg.catalog.type} is overridden unconditionally — an iceberg-on-HMS sibling is always + * the hms flavor. + */ + static Map synthesize(Map gatewayCatalogProperties) { + Map siblingProperties = new HashMap<>(gatewayCatalogProperties); + siblingProperties.put(ICEBERG_CATALOG_TYPE_KEY, ICEBERG_CATALOG_TYPE_HMS); + return siblingProperties; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/IcebergSiblingPropertiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/IcebergSiblingPropertiesTest.java new file mode 100644 index 00000000000000..6895840932b68a --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/IcebergSiblingPropertiesTest.java @@ -0,0 +1,97 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests the pure iceberg-sibling property synthesis. Each case pins WHY the behavior matters for the embedded + * iceberg-on-HMS connector the flipped gateway builds from the returned map. + */ +public class IcebergSiblingPropertiesTest { + + @Test + public void injectsHmsFlavor() { + // Without iceberg.catalog.type the sibling's createCatalog() throws "Missing 'iceberg.catalog.type'"; + // an iceberg-on-HMS table is always served by the hms flavor. + Map in = new HashMap<>(); + in.put("hive.metastore.uris", "thrift://host:9083"); + + Map out = IcebergSiblingProperties.synthesize(in); + + Assertions.assertEquals("hms", out.get("iceberg.catalog.type"), + "the sibling must resolve the hms flavor"); + } + + @Test + public void carriesMetastoreStorageAndKerberosKeys() { + // The sibling connects to the SAME metastore/storage as the gateway; dropping any of these keys would + // leave the embedded HiveCatalog unable to reach HMS or object storage. The uri short form must survive + // too (the iceberg HMS parser binds both hive.metastore.uris and uri). + Map in = new HashMap<>(); + in.put("uri", "thrift://host:9083"); + in.put("fs.s3a.access.key", "AK"); + in.put("dfs.nameservices", "ns1"); + in.put("hadoop.security.authentication", "kerberos"); + in.put("hive.metastore.client.principal", "hive/_HOST@REALM"); + in.put("hive.conf.resources", "hive-site.xml"); + + Map out = IcebergSiblingProperties.synthesize(in); + + Assertions.assertEquals("thrift://host:9083", out.get("uri"), "the uri short form must be carried"); + Assertions.assertEquals("AK", out.get("fs.s3a.access.key"), "object-storage creds must be carried"); + Assertions.assertEquals("ns1", out.get("dfs.nameservices"), "HDFS-HA config must be carried"); + Assertions.assertEquals("kerberos", out.get("hadoop.security.authentication"), + "kerberos auth mode must be carried"); + Assertions.assertEquals("hive/_HOST@REALM", out.get("hive.metastore.client.principal"), + "kerberos principal must be carried"); + Assertions.assertEquals("hive-site.xml", out.get("hive.conf.resources"), + "external hive-site.xml reference must be carried"); + Assertions.assertEquals("hms", out.get("iceberg.catalog.type")); + } + + @Test + public void doesNotMutateInput() { + // The gateway holds its catalog properties unmodifiable and shared; mutating them would corrupt the + // gateway's own hive path. + Map in = new HashMap<>(); + in.put("hive.metastore.uris", "thrift://host:9083"); + + IcebergSiblingProperties.synthesize(in); + + Assertions.assertFalse(in.containsKey("iceberg.catalog.type"), "the input map must not be mutated"); + Assertions.assertEquals(1, in.size()); + } + + @Test + public void overridesAnyPreexistingFlavor() { + // Defensive: even a stray iceberg.catalog.type on the gateway catalog must not select a non-hms flavor + // for the iceberg-on-HMS sibling. + Map in = new HashMap<>(); + in.put("iceberg.catalog.type", "rest"); + + Map out = IcebergSiblingProperties.synthesize(in); + + Assertions.assertEquals("hms", out.get("iceberg.catalog.type"), + "the iceberg-on-HMS sibling is always the hms flavor, overriding any pre-existing value"); + } +} From 701ee38ccf746b92adb8ebdee2846cc876ecd140 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 07:37:01 +0800 Subject: [PATCH 065/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20S1=20props-synthesis=20DONE=20(copy-all=20verified);=20ne?= =?UTF-8?q?xt=20=3D=20S2=20gateway=20embed?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S1 (d971410af62) synthesizes the iceberg sibling's property map by copying the gateway catalog's full property map + injecting iceberg.catalog.type=hms. The copy-all-vs-subset choice was adversarially verified safe: it is the same map shape a native iceberg-on-hms catalog already feeds the connector, and hive-only delta keys are ignored (not misused) since create() does no validation. Next = S2: HiveConnector holds + lazily builds one sibling per gateway (volatile double-check) and forwards close(), held only as the parent-first Connector. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- .../hms-cutover-sibling-connector-decomposition-2026-07-08.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 6b86593e19e438..0b7c910f96922f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -59,7 +59,8 @@ 4. **vanished-partition** 已按复审改为 on-demand empty→fe-core throw "can not find partition"（parity），非返 0。 5. **iceberg/hudi-on-HMS 的 snapshot-id 新鲜度**靠 §4.4 兄弟委派（gateway `beginQuerySnapshot` 对 iceberg handle 委派→非 last-modified pin；per-handle pin 标志设计天然支持混合目录）。 - **✅ §4.4a 兄弟连接器 SPI 接缝 DONE（`45dcca32fa8`）**（recon `wf_da155752-ebb` 6维+2 critic + 对抗复审 `wf_caef8bf7-ec2` 3镜头，全 HEAD 核对；分解权威文档 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`，起步必读）。新 `ConnectorContext.createSiblingConnector(String catalogType, Map props)` 默认 **`return null`**（对齐所有其它 SPI 默认的安全默认约定——**非** throw，更正设计 §4.4 的 UnsupportedOperationException）；fe-core `DefaultConnectorContext` 覆写 = `ConnectorFactory.createConnector(catalogType, props, this)`（传 `this` → 兄弟连接器复用本目录 id/auth/storage，其 hms-flavor HiveCatalog 名默认取 `context.getCatalogName()`，故共享 context 承重；fe-core 保持 connector-agnostic、零属性解析——props 由网关合成）；iceberg+paimon `TcclPinningConnectorContext` 装饰器补委派该新方法到 raw context（保持"穷尽 pass-through"契约为真——兄弟自套 TCCL/auth pin，须收未包裹 context 避免双重 pin）。唯一改引擎的一步、后面所有插件步的地基。全绿（spi 3 + fe-core 3 + iceberg 5 + paimon 4 测 + 4 模块 checkstyle 0 + import gate 净）。**⚠关键设计更正（更正设计 §4.4）**：设计的"网关包装句柄 `{format, delegate}`"方案**已证死**——`PluginDrivenScanNode` 选执行器(:209)与执行扫描(:966)用**同一句柄对象**、中间无拆包缝，`IcebergScanPlanProvider` 无条件强转 `(IcebergTableHandle)`(:319/364/468/1009)→包装句柄必 CCE；**改为** `getTableHandle` 对 iceberg 返**原始外来句柄**（iceberg 强转成功）、网关一律用 `instanceof HiveTableHandle`(自有 hive-loader 类型)→hive、否则转发兄弟，**永不强转外来句柄**，零新 SPI。 -- **下一步 = §4.4 剩余步（休眠，序见分解文档）**：S1 hive 插件侧属性合成纯函数（注入字面量 `iceberg.catalog.type=hms`+携 `hive.metastore.uris`+透传 storage/kerberos）→ S2 `HiveConnector` 持有+懒建兄弟+`close()` 转发（**单例 per-gateway**，非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash）→ S3 元数据每 handle 方法按 `instanceof HiveTableHandle` 守卫转发（**两不相交集**：改写误服务的 hive 实现 + 转发全缺席的 iceberg 子集，后者多为**静默错答**getMvccPartitionView/resolveTimeTravel/applySnapshot/…须一并；`beginQuerySnapshot` 须与 freshness 同步分流，其 last-modified 标志门控 freshness 是否被读；apply* 须转发并原样回传兄弟句柄）→ S4 `getTableHandle` iceberg 分流（返原始外来句柄，点亮 S3）→ S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（返兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法+连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 +- **✅ §4.4 S1 属性合成纯函数 DONE（`d971410af62`）**：`IcebergSiblingProperties.synthesize(gatewayCatalogProps)`=**整份目录属性逐字复制+注入字面量 `iceberg.catalog.type=hms`**（per-catalog、非 per-table，无需 HmsTableInfo；字面量硬编码因 iceberg 常量 child-first 不可见）。**copy-all（非选择性子集）经对抗核验安全**：与原生 `type=iceberg,iceberg.catalog.type=hms` 目录喂给 IcebergConnector 的**完全同形**（fe-core `createConnectorFromProperties` 也传整份目录属性）、iceberg `create()` 零属性校验→hive 侧 delta 键只被忽略不被误用（`warehouse` 无害=HMS flavor 不从它派生 fs.defaultFS、存量表用自存 location；唯一非缺陷=兄弟用 iceberg 默认缓存 TTL 与原生同）。不改输入、覆盖任何 stray flavor。全绿（4 测+checkstyle+import gate）。 +- **下一步 = §4.4 S2（休眠，序见分解文档）**：`HiveConnector` 持有+懒建兄弟（`context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))`，**单例 per-gateway** volatile 双检、非 per-table 因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash）+`close()` 转发兄弟。仅以 parent-first `Connector` 接口持有、永不强转。之后 S3 元数据每 handle 方法按 `instanceof HiveTableHandle` 守卫转发（**两不相交集**：改写误服务的 hive 实现 + 转发全缺席的 iceberg 子集，后者多为**静默错答**getMvccPartitionView/resolveTimeTravel/applySnapshot/…须一并；`beginQuerySnapshot` 须与 freshness 同步分流，其 last-modified 标志门控 freshness 是否被读；apply* 须转发并原样回传兄弟句柄）→ S4 `getTableHandle` iceberg 分流（返原始外来句柄，点亮 S3）→ S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（返兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法+连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 diff --git a/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md b/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md index 232e67be767e03..576b8652ea4f2f 100644 --- a/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md +++ b/plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md @@ -19,7 +19,7 @@ Design §4.4 proposed a gateway-owned **wrapper handle** `{HiveTableType format; ## 3. Ordered dormant sub-steps (each an independent休眠 commit) - **S0 — fe-core sibling seam `createSiblingConnector` ✅ DONE (this session).** `ConnectorContext.createSiblingConnector(String catalogType, Map properties)` default `return null` (matches every other SPI default's safe-default convention — NOT throw, correcting the design's "UnsupportedOperationException"); `DefaultConnectorContext` override = `return ConnectorFactory.createConnector(catalogType, properties, this)` (passes `this` → sibling shares catalogId/auth/storage suppliers; the sibling's HMS-flavor `HiveCatalog` name defaults to `context.getCatalogName()`, so the shared context is load-bearing). Only fe-core change in the whole feature; hard prerequisite for all plugin work (the hive plugin holds only the `ConnectorContext` interface, so this seam is its ONLY way to build a sibling that inherits the hms catalog's context, and keeps the plugin off `ConnectorFactory` statics). -- **S1 — hive-loader props-synthesis helper (pure).** New `fe-connector-hive` class: `Map synthesize(catalogProps, HmsTableInfo)` injecting **literal** `"iceberg.catalog.type"="hms"` + carrying `"hive.metastore.uris"` + passing through `fs.s3a.*/hadoop.*/dfs.*` + kerberos keys. Must hardcode the literal key/value strings (`IcebergConnectorProperties.TYPE_HMS`/`ICEBERG_CATALOG_TYPE` are child-first, invisible to the hive loader). Minimum working props for hms-flavor: `{iceberg.catalog.type=hms, hive.metastore.uris}`; `warehouse` NOT required. Independent of S0. +- **S1 — hive-loader props-synthesis helper (pure) ✅ DONE (`d971410af62`).** `IcebergSiblingProperties.synthesize(gatewayCatalogProps)` (per-catalog — the sibling is table-agnostic, NO `HmsTableInfo` needed) = **copy the gateway catalog's whole property map verbatim + inject literal `iceberg.catalog.type=hms`** (hardcoded strings; `IcebergConnectorProperties.TYPE_HMS`/`ICEBERG_CATALOG_TYPE` are child-first, invisible to the hive loader). **Copy-all (not a hand-picked subset)** is the robust + verified choice: it is the EXACT map shape the iceberg connector already handles for a native `type=iceberg, iceberg.catalog.type=hms` catalog (fe-core's `createConnectorFromProperties` passes the full catalog props), and iceberg's `create()` does NO property validation, so hive-only delta keys are ignored not misused — **adversarially verified** (every delta key ignored; `warehouse` harmless because the HMS flavor never derives `fs.defaultFS` from it and existing tables use their stored locations; one non-defect note = the sibling uses iceberg's default cache TTL, same as native iceberg-on-hms). Never mutates the input; overrides any stray flavor. Minimum working props for hms-flavor: `{iceberg.catalog.type=hms, hive.metastore.uris|uri}`; `warehouse` NOT required. Independent of S0. Full green (4 tests + checkstyle + import gate). - **S2 — embedded-sibling holder + lazy build + close-forward.** `HiveConnector`: memoized `volatile Connector icebergSibling` = `context.createSiblingConnector("iceberg", S1.synthesize(...))`; forward `close()` to it then null. **Must be ONE sibling per gateway connector** (not per-table): `IcebergConnector` holds per-catalog caches (`latestSnapshotCache`, `manifestCache`, scan→write `rewritableDeleteStash`) shared across its tables — a per-op sibling would fragment them. Held ONLY as the parent-first `Connector` type — never cast. Deps: S0, S1. - **S3 — metadata handle-guard-and-forward.** `HiveConnectorMetadata` every handle-based method: `if (!(handle instanceof HiveTableHandle)) return sibling.getMetadata(session).(…); `. The delegation surface is **two disjoint sets**: (a) override-and-divert the hive-implemented mis-servers (`getTableSchema[+snapshot]`, `getColumnHandles`, `getTableStatistics`, `listPartitions/Names/Values`, `beginQuerySnapshot`, `getTableFreshness`, `getPartitionFreshnessMillis`, `buildTableDescriptor`, `applyFilter`, DDL); (b) forward the wholesale-**absent** iceberg subset that today silently no-ops (`getMvccPartitionView`, `resolveTimeTravel`, `applySnapshot`, `applyRewriteFileScope`, `applyTopnLazyMaterialization`, `applyProjection`, `applyLimit`, `listSupportedSysTables`, `getSysTableHandle`, schema-at-snapshot) — several are **silent wrong answers**, not fail-loud, so a partial gateway ships silent MVCC/time-travel corruption. **`beginQuerySnapshot` must divert together with freshness** (its `isLastModifiedFreshness=true` flag gates whether `getTableFreshness` is even consulted). The handle-in/handle-out `apply*` must forward AND return the sibling's handle **unmodified** (a single un-rewrapped return poisons a downstream `planScan` cast). Deps: S2. - **S4 — `getTableHandle` ICEBERG divert.** When `HiveTableFormatDetector.detect(...)==ICEBERG`, return the sibling's `getTableHandle` (foreign iceberg handle) instead of a HiveTableHandle stamped ICEBERG. The pivot that lights up S3 + makes iceberg's cast succeed. HUDI stays fail-loud/legacy. Deps: S3 (keep adjacent), S2. From b254cdd0648202cbd15117404a626e35755e6be4 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 08:20:29 +0800 Subject: [PATCH 066/334] [feat](catalog) HMS gateway embeds + lazy-builds iceberg sibling connector (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S2 of the hms-cutover sibling-delegation build-out. A flipped hms catalog will be served by the hive plugin acting as a gateway; its iceberg-on-HMS tables are delegated to a sibling iceberg connector built in the iceberg plugin's own child-first classloader (never co-packaged into the hive zip — a second AWS SDK would poison S3 JVM-wide). This step adds only the holder + lazy build + close-forward; there is no consumer yet, so it is fully dormant until hms enters SPI_READY_TYPES. - HiveConnector.getOrCreateIcebergSibling(): volatile double-checked lazy build via context.createSiblingConnector("iceberg", IcebergSiblingProperties .synthesize(properties)), mirroring the existing getOrCreateClient() idiom. One sibling per gateway (IcebergConnector holds per-catalog caches shared across its tables, which a per-op sibling would fragment). Held ONLY as the parent-first Connector interface, never cast (the concrete type is invisible to the hive loader -> a cast would CCE across the loader split). Fails loud (DorisConnectorException, catalog-scoped message) when no iceberg provider is available, and the failure is NOT memoized so a later-available plugin recovers. - close() forwards to the sibling then nulls it (the engine closes only a catalog's primary connector; the gateway owns the sibling lifecycle). No-op when the sibling was never built. Green: fe-connector-hive 194 tests (6 new HiveConnectorSiblingTest) + checkstyle 0 + connector import gate clean. Adversarial review (4 dimensions + verify): 0 confirmed defects. hms confirmed absent from SPI_READY_TYPES -> dormancy holds. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 50 ++++++ .../hive/HiveConnectorSiblingTest.java | 167 ++++++++++++++++++ 2 files changed, 217 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 784c3f3e9cd15e..042014e0a95abd 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -57,6 +57,11 @@ public class HiveConnector implements Connector { // Catalog property key gating the plugin-side Kerberos authenticator (value matches AuthType.KERBEROS). private static final String HADOOP_SECURITY_AUTHENTICATION = "hadoop.security.authentication"; + // The sibling connector type a flipped hms gateway delegates iceberg-on-HMS tables to. A string literal + // (not the iceberg plugin's own type constant, which is child-first and invisible from the hive loader); + // matches the "iceberg" entry in CatalogFactory.SPI_READY_TYPES. + private static final String ICEBERG_CONNECTOR_TYPE = "iceberg"; + private final Map properties; private final ConnectorContext context; private volatile HmsClient hmsClient; @@ -71,6 +76,14 @@ public class HiveConnector implements Connector { // Plugin-owned and dormant until the read cutover wires its query-finish commit (see the manager). private final HiveReadTransactionManager readTxnManager = new HiveReadTransactionManager(); + // Embedded iceberg SIBLING connector: a flipped hms gateway delegates its iceberg-on-HMS tables to it. Built + // once per gateway connector (lazily) in the iceberg plugin's OWN child-first classloader via + // context.createSiblingConnector — never co-packaged into the hive zip (a second AWS SDK would poison S3 + // JVM-wide). Held ONLY as the parent-first Connector interface and NEVER cast: its concrete type is invisible + // to the hive loader, so a cast would CCE across the loader split. Dormant until hms enters SPI_READY_TYPES — + // nothing builds it today. + private volatile Connector icebergSibling; + public HiveConnector(Map properties, ConnectorContext context) { this.properties = Collections.unmodifiableMap(properties); this.context = context; @@ -149,6 +162,36 @@ private HmsClient getOrCreateClient() { return hmsClient; } + /** + * Lazily builds and memoizes the embedded iceberg sibling connector this hive gateway delegates its + * iceberg-on-HMS tables to. There is exactly ONE sibling per gateway connector (not per table): the iceberg + * connector holds per-catalog caches (latest-snapshot, manifest, scan→write delete stash) shared across + * its tables, which a per-op sibling would fragment. The sibling is built through + * {@link ConnectorContext#createSiblingConnector} so its concrete class is loaded by the iceberg plugin's own + * child-first classloader, sharing this gateway catalog's id/auth/storage; it is therefore held ONLY as the + * parent-first {@link Connector} interface and MUST NOT be cast (a cast would CCE across the loader split). + * + *

    Fails loud when no iceberg provider is available (e.g. the plugin is not installed). The failure is NOT + * memoized (a null sibling leaves the field unset), so a later-available plugin recovers on the next access. + */ + Connector getOrCreateIcebergSibling() { + if (icebergSibling == null) { + synchronized (this) { + if (icebergSibling == null) { + Connector sibling = context.createSiblingConnector( + ICEBERG_CONNECTOR_TYPE, IcebergSiblingProperties.synthesize(properties)); + if (sibling == null) { + throw new DorisConnectorException( + "Cannot serve iceberg-on-HMS tables in catalog '" + context.getCatalogName() + + "': the iceberg connector plugin is not available"); + } + icebergSibling = sibling; + } + } + } + return icebergSibling; + } + private HmsClient createClient() { String metastoreUri = properties.get(HiveConnectorProperties.HIVE_METASTORE_URIS); if (metastoreUri == null || metastoreUri.isEmpty()) { @@ -262,5 +305,12 @@ public void close() throws IOException { c.close(); hmsClient = null; } + // Forward close to the embedded iceberg sibling: the engine closes only a catalog's PRIMARY connector, + // so the gateway owns the sibling's lifecycle. No-op when the sibling was never built (dormant path). + Connector sibling = icebergSibling; + if (sibling != null) { + sibling.close(); + icebergSibling = null; + } } } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java new file mode 100644 index 00000000000000..fe410e554d4c22 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java @@ -0,0 +1,167 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Pins the HMS-cutover §4.4 S2 embedded-sibling holder: a flipped hms gateway lazily builds ONE embedded + * iceberg connector (via {@link org.apache.doris.connector.spi.ConnectorContext#createSiblingConnector}) that it + * delegates its iceberg-on-HMS tables to, and forwards {@code close()} to it. + * + *

    The whole surface is dormant until hms enters {@code SPI_READY_TYPES}: no production path calls + * {@code getOrCreateIcebergSibling()} yet, so these assertions are a Rule-9 guard that the holder's contract + * (single sibling per gateway, hms-flavor synthesized props, fail-loud when the plugin is absent, lifecycle + * forwarding) is correct BEFORE the flip wires a consumer. + */ +public class HiveConnectorSiblingTest { + + @Test + public void buildsIcebergSiblingWithSynthesizedHmsFlavorProps() { + Map catalogProps = new HashMap<>(); + catalogProps.put("hive.metastore.uris", "thrift://host:9083"); + catalogProps.put("iceberg.catalog.type", "rest"); // a stray flavor the sibling must override to hms + FakeSibling sibling = new FakeSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(catalogProps, context); + + Connector built = connector.getOrCreateIcebergSibling(); + + Assertions.assertSame(sibling, built, "the accessor must return the context-built sibling"); + // The sibling is always an iceberg connector — the delegate type must reach the seam verbatim. + Assertions.assertEquals("iceberg", context.lastType, "the sibling connector type must be iceberg"); + // Props are the gateway's catalog map + the hms flavor forced on (S1 synthesis), so the embedded + // connector reaches the SAME metastore/storage and always resolves the hms flavor. + Assertions.assertEquals("hms", context.lastProps.get("iceberg.catalog.type"), + "the sibling must be built as the hms flavor, overriding any stray gateway value"); + Assertions.assertEquals("thrift://host:9083", context.lastProps.get("hive.metastore.uris"), + "the gateway's metastore uri must be carried to the sibling"); + } + + @Test + public void memoizesSingleSiblingPerGateway() { + // The iceberg connector holds per-catalog caches (latest-snapshot / manifest / scan->write delete stash) + // shared across its tables; a per-op sibling would fragment them. There must be exactly one build. + RecordingSiblingContext context = new RecordingSiblingContext(new FakeSibling()); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + + Connector first = connector.getOrCreateIcebergSibling(); + Connector second = connector.getOrCreateIcebergSibling(); + + Assertions.assertSame(first, second, "repeated access must return the same single sibling"); + Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once per gateway"); + } + + @Test + public void failsLoudWhenIcebergPluginAbsent() { + // A missing iceberg provider surfaces as a null sibling from the seam; the gateway must fail loud with a + // catalog-scoped message rather than NPE deep in a later delegation. + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + connector::getOrCreateIcebergSibling); + Assertions.assertTrue(ex.getMessage().contains("test_catalog"), + "the failure must name the catalog it could not serve"); + } + + @Test + public void failLoudIsNotMemoized() { + // The absence is not cached: a plugin that becomes available (or a transient factory failure that + // clears) must be picked up on the next access, not permanently poison the gateway. + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + Assertions.assertThrows(DorisConnectorException.class, connector::getOrCreateIcebergSibling); + + FakeSibling sibling = new FakeSibling(); + context.siblingToReturn = sibling; + Assertions.assertSame(sibling, connector.getOrCreateIcebergSibling(), + "a later-available sibling must be built after an earlier fail-loud"); + Assertions.assertEquals(2, context.buildCount, "the failed build must be retried, not memoized"); + } + + @Test + public void closeForwardsToSiblingAndClearsIt() throws Exception { + // The engine closes only the primary connector; the gateway owns the sibling's lifecycle, and a second + // close must not double-close a released sibling. + FakeSibling sibling = new FakeSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + connector.getOrCreateIcebergSibling(); + + connector.close(); + connector.close(); + + Assertions.assertEquals(1, sibling.closeCount, "close must forward to the sibling exactly once"); + } + + @Test + public void closeIsNoOpWhenSiblingNeverBuilt() throws Exception { + // Dormant path: a gateway that never delegated must not build a sibling just to close it. + RecordingSiblingContext context = new RecordingSiblingContext(new FakeSibling()); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + + connector.close(); + + Assertions.assertEquals(0, context.buildCount, "close must not trigger a sibling build"); + } + + /** Records the {@code createSiblingConnector} call and returns a configurable (possibly null) sibling. */ + private static final class RecordingSiblingContext extends FakeConnectorContext { + int buildCount; + String lastType; + Map lastProps; + Connector siblingToReturn; + + RecordingSiblingContext(Connector siblingToReturn) { + this.siblingToReturn = siblingToReturn; + } + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + buildCount++; + lastType = catalogType; + lastProps = properties; + return siblingToReturn; + } + } + + /** A bare {@link Connector} stand-in for the cross-loader iceberg sibling; only close-counting matters here. */ + private static final class FakeSibling implements Connector { + int closeCount; + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public void close() { + closeCount++; + } + } +} From 758dae0970be4c30e94d733a315bc180b5b6b968 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 08:22:40 +0800 Subject: [PATCH 067/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20S2=20sibling-holder=20DONE;=20next=20=3D=20S3=20metadata?= =?UTF-8?q?=20handle-guard-forward?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S2 committed at 119b4fca64b (dormant): HiveConnector.getOrCreateIcebergSibling lazy build + single-per-gateway + never-cast + fail-loud-not-memoized + close forward. Adversarial review wf_c17ad553-53a: 0 confirmed defects. Next step = S3 (HiveConnectorMetadata per-handle instanceof-HiveTableHandle guard-and-forward). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 0b7c910f96922f..cc6c1831c8720f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -60,7 +60,8 @@ 5. **iceberg/hudi-on-HMS 的 snapshot-id 新鲜度**靠 §4.4 兄弟委派（gateway `beginQuerySnapshot` 对 iceberg handle 委派→非 last-modified pin；per-handle pin 标志设计天然支持混合目录）。 - **✅ §4.4a 兄弟连接器 SPI 接缝 DONE（`45dcca32fa8`）**（recon `wf_da155752-ebb` 6维+2 critic + 对抗复审 `wf_caef8bf7-ec2` 3镜头，全 HEAD 核对；分解权威文档 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`，起步必读）。新 `ConnectorContext.createSiblingConnector(String catalogType, Map props)` 默认 **`return null`**（对齐所有其它 SPI 默认的安全默认约定——**非** throw，更正设计 §4.4 的 UnsupportedOperationException）；fe-core `DefaultConnectorContext` 覆写 = `ConnectorFactory.createConnector(catalogType, props, this)`（传 `this` → 兄弟连接器复用本目录 id/auth/storage，其 hms-flavor HiveCatalog 名默认取 `context.getCatalogName()`，故共享 context 承重；fe-core 保持 connector-agnostic、零属性解析——props 由网关合成）；iceberg+paimon `TcclPinningConnectorContext` 装饰器补委派该新方法到 raw context（保持"穷尽 pass-through"契约为真——兄弟自套 TCCL/auth pin，须收未包裹 context 避免双重 pin）。唯一改引擎的一步、后面所有插件步的地基。全绿（spi 3 + fe-core 3 + iceberg 5 + paimon 4 测 + 4 模块 checkstyle 0 + import gate 净）。**⚠关键设计更正（更正设计 §4.4）**：设计的"网关包装句柄 `{format, delegate}`"方案**已证死**——`PluginDrivenScanNode` 选执行器(:209)与执行扫描(:966)用**同一句柄对象**、中间无拆包缝，`IcebergScanPlanProvider` 无条件强转 `(IcebergTableHandle)`(:319/364/468/1009)→包装句柄必 CCE；**改为** `getTableHandle` 对 iceberg 返**原始外来句柄**（iceberg 强转成功）、网关一律用 `instanceof HiveTableHandle`(自有 hive-loader 类型)→hive、否则转发兄弟，**永不强转外来句柄**，零新 SPI。 - **✅ §4.4 S1 属性合成纯函数 DONE（`d971410af62`）**：`IcebergSiblingProperties.synthesize(gatewayCatalogProps)`=**整份目录属性逐字复制+注入字面量 `iceberg.catalog.type=hms`**（per-catalog、非 per-table，无需 HmsTableInfo；字面量硬编码因 iceberg 常量 child-first 不可见）。**copy-all（非选择性子集）经对抗核验安全**：与原生 `type=iceberg,iceberg.catalog.type=hms` 目录喂给 IcebergConnector 的**完全同形**（fe-core `createConnectorFromProperties` 也传整份目录属性）、iceberg `create()` 零属性校验→hive 侧 delta 键只被忽略不被误用（`warehouse` 无害=HMS flavor 不从它派生 fs.defaultFS、存量表用自存 location；唯一非缺陷=兄弟用 iceberg 默认缓存 TTL 与原生同）。不改输入、覆盖任何 stray flavor。全绿（4 测+checkstyle+import gate）。 -- **下一步 = §4.4 S2（休眠，序见分解文档）**：`HiveConnector` 持有+懒建兄弟（`context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))`，**单例 per-gateway** volatile 双检、非 per-table 因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash）+`close()` 转发兄弟。仅以 parent-first `Connector` 接口持有、永不强转。之后 S3 元数据每 handle 方法按 `instanceof HiveTableHandle` 守卫转发（**两不相交集**：改写误服务的 hive 实现 + 转发全缺席的 iceberg 子集，后者多为**静默错答**getMvccPartitionView/resolveTimeTravel/applySnapshot/…须一并；`beginQuerySnapshot` 须与 freshness 同步分流，其 last-modified 标志门控 freshness 是否被读；apply* 须转发并原样回传兄弟句柄）→ S4 `getTableHandle` iceberg 分流（返原始外来句柄，点亮 S3）→ S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（返兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法+连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 +- **✅ §4.4 S2 兄弟持有/懒建/close 转发 DONE（`119b4fca64b`）**（对抗 review `wf_c17ad553-53a`：dormancy-parity/classloader-cce/concurrency-lifecycle/contract-test-intent 4 维 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 新增包级 `getOrCreateIcebergSibling()`：volatile 双检（复刻本文件 `getOrCreateClient()` 惯用法）经 `context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))` 懒建；**单例 per-gateway**（非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash，per-op 会碎片化）；**仅以 parent-first `Connector` 接口持有、永不强转**（外来句柄类型 hive 加载器不可见，强转必 CCE）；插件缺失返 null→**fail-loud** `DorisConnectorException`（含目录名）且**失败不 memoize**（字段留 null 下次重试）。`close()` 追加转发兄弟并置 null（引擎只关主连接器，网关自持兄弟生命周期）、未建时 no-op。全绿（fe-connector-hive 194 测 + 6 新 `HiveConnectorSiblingTest` + checkstyle 0 + import gate 净；HEAD 证实 hms 不在 SPI_READY_TYPES=休眠成立）。⚠**复审记账（refuted，勿改）**：`close()` 在 `hmsClient.close()` 之后转发兄弟，若前者抛 IOException 理论会漏关兄弟——但唯一生产 `HmsClient`=`ThriftHmsClient` 的 `close()` 结构性吞异常（`clientPool.close()` 包在 catch 内）实际不抛，非真缺陷，保持与现有 close 惯用法一致。 +- **下一步 = §4.4 S3（休眠，序见分解文档）**：`HiveConnectorMetadata` 每 handle 方法按 `instanceof HiveTableHandle` 守卫转发（**两不相交集**：改写误服务的 hive 实现 + 转发全缺席的 iceberg 子集，后者多为**静默错答**getMvccPartitionView/resolveTimeTravel/applySnapshot/…须一并；`beginQuerySnapshot` 须与 freshness 同步分流，其 last-modified 标志门控 freshness 是否被读；apply* 须转发并原样回传兄弟句柄）→ S4 `getTableHandle` iceberg 分流（返原始外来句柄，点亮 S3）→ S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（返兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法+连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 From ac96ddce55000555dd5102f06c4af9ceec21d240 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 09:27:25 +0800 Subject: [PATCH 068/334] [feat](catalog) HMS gateway metadata delegates foreign handles to iceberg sibling (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S3 of the hms-cutover sibling-delegation build-out. Makes HiveConnectorMetadata's per-handle methods discriminate on the concrete handle type: a HiveTableHandle runs the existing hive logic byte-identically; a foreign (iceberg-on-HMS) handle is forwarded to the embedded iceberg sibling connector (built dormant in S2). The gateway NEVER casts the foreign handle — its concrete type lives in the iceberg plugin's child-first classloader, so a cast would CCE across the loader split. Wiring: HiveConnector.getMetadata passes this::getOrCreateIcebergSibling as a Supplier; HiveConnectorMetadata.siblingMetadata(session) resolves it per forwarded call (parity with fe-core, which acquires a ConnectorMetadata per operation; the heavy catalog/caches live on the single sibling connector). A 3-arg constructor (hive-only, test) installs a fail-loud NO_ICEBERG_SIBLING supplier. Two guard sets (a code-grounded recon classified the full 67-method surface): - 13 methods hive already overrides (getTableSchema/getColumnHandles/getTable+Column Statistics/estimateDataSizeByListingFiles/applyFilter/listPartitions[Names]/ beginQuerySnapshot/getTable+PartitionFreshness/dropTable/truncateTable) get a pure prefix guard — the hive body is unchanged for a hive handle. - 8 methods hive did NOT override but iceberg does (schema-at-snapshot, getMvccPartitionView, resolveTimeTravel, applySnapshot, applyRewriteFileScope, applyTopnLazyMaterialization, listSupportedSysTables, getSysTableHandle) get NEW guard-and-forward overrides — without them a delegated iceberg table would get hive's SPI default (a SILENT wrong answer: empty MVCC/time-travel/sys-tables or an unthreaded snapshot pin), not a fail-loud. The hive branch reproduces the SPI default. beginQuerySnapshot diverts in lockstep with the two freshness methods (its isLastModifiedFreshness flag gates whether fe-core consults them). Handle-out methods (applyFilter/apply*/getSysTableHandle) return the sibling's handle UNMODIFIED — a rewrap would poison a downstream planScan cast. Excluded by design (recon-verified): getTableHandle (S4), buildTableDescriptor + all name-based view/db/createTable/identifier methods (S6), the inert applyProjection/ applyLimit/listPartitionValues (neither side overrides), and the ALTER-DDL suite + write validations (handle-based but write-cutover, separate substep). Dormant: "hms" is absent from SPI_READY_TYPES and getTableHandle (S4, not yet done) still returns a HiveTableHandle, so every guard is an inert no-op today and a pure-hive query never builds the sibling. Green: fe-connector-hive 197 tests (3 new HiveConnectorMetadataSiblingDelegationTest, including a completeness lock on the exact forwarded-method set) + checkstyle 0 + connector import gate clean. Adversarial review (4 dimensions + verify): 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 5 +- .../connector/hive/HiveConnectorMetadata.java | 166 +++++++++ ...onnectorMetadataSiblingDelegationTest.java | 347 ++++++++++++++++++ 3 files changed, 517 insertions(+), 1 deletion(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 042014e0a95abd..4323c3b7d3a9ae 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -91,7 +91,10 @@ public HiveConnector(Map properties, ConnectorContext context) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - return new HiveConnectorMetadata(getOrCreateClient(), properties, context); + // Pass the sibling supplier (not the built sibling): a pure-hive query never invokes it, so a hive-only + // deployment without the iceberg plugin never builds/throws. The metadata diverts a foreign iceberg + // handle through it (per-handle guard-and-forward). + return new HiveConnectorMetadata(getOrCreateClient(), properties, context, this::getOrCreateIcebergSibling); } @Override diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 1fad270dc9ef93..00b439cb019a5c 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.hive; +import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorColumnStatistics; @@ -35,8 +36,10 @@ import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; +import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.connector.api.pushdown.ConnectorAnd; import org.apache.doris.connector.api.pushdown.ConnectorComparison; import org.apache.doris.connector.api.pushdown.ConnectorExpression; @@ -80,6 +83,7 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.function.Supplier; import java.util.function.ToLongFunction; import java.util.stream.Collectors; @@ -155,6 +159,13 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // Upper bound on partitions listed from HMS for the file-list estimate, matching HiveScanPlanProvider. private static final int MAX_PARTITIONS_FOR_STATS = 100000; + // A Supplier installed by the 3-arg constructor when no iceberg sibling is available (hive-only + // construction, e.g. unit tests exercising only hive-handle paths). It is invoked only when a NON-hive + // handle is delegated — which such a construction never does — so it fails loud instead of NPEing. + private static final Supplier NO_ICEBERG_SIBLING = () -> { + throw new DorisConnectorException("no iceberg sibling connector configured for this hive metadata"); + }; + private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; @@ -163,12 +174,34 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // read from FE Config. The default getEnvironment() is an empty map, so direct-construction tests that // pass a bare context degrade to the hard-coded fallbacks in createTable. private final ConnectorContext context; + // Supplies the embedded iceberg SIBLING connector for delegating a NON-hive (iceberg-on-HMS) table handle + // (see the guard-and-forward methods). Lazy: HiveConnector.getOrCreateIcebergSibling builds it only on first + // use, so a pure-hive query never triggers it. The sibling is used ONLY through the parent-first Connector / + // ConnectorMetadata interfaces — its concrete iceberg types are never cast here (cross-loader CCE). + private final Supplier icebergSiblingSupplier; public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context) { + this(hmsClient, properties, context, NO_ICEBERG_SIBLING); + } + + public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context, + Supplier icebergSiblingSupplier) { this.hmsClient = hmsClient; this.properties = properties; this.context = context; this.typeMappingOptions = buildTypeMappingOptions(properties); + this.icebergSiblingSupplier = icebergSiblingSupplier; + } + + /** + * The embedded iceberg sibling's metadata, for delegating a NON-hive (foreign iceberg) table handle in the + * guard-and-forward methods below. Obtained fresh per call — parity with fe-core, which acquires a + * ConnectorMetadata per operation; the heavy catalog/caches live on the single (memoized) sibling connector, + * so this is cheap. The returned metadata and any handle it produces are used ONLY through the parent-first + * SPI interfaces and MUST NOT be cast (the sibling's concrete iceberg types would CCE across the loader split). + */ + private ConnectorMetadata siblingMetadata(ConnectorSession session) { + return icebergSiblingSupplier.get().getMetadata(session); } // ========== ConnectorSchemaOps ========== @@ -240,6 +273,9 @@ public Optional getTableHandle( @Override public ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getTableSchema(session, handle); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; String dbName = hiveHandle.getDbName(); String tableName = hiveHandle.getTableName(); @@ -295,6 +331,9 @@ public Map getProperties() { @Override public Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getColumnHandles(session, handle); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; HmsTableInfo tableInfo = hmsClient.getTable( hiveHandle.getDbName(), hiveHandle.getTableName()); @@ -485,6 +524,9 @@ private static String parseTrinoViewDefinition(String originalText) { @Override public Optional getTableStatistics( ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getTableStatistics(session, handle); + } Map params = ((HiveTableHandle) handle).getTableParameters(); if (params == null) { return Optional.empty(); @@ -528,6 +570,9 @@ public Optional getTableStatistics( @Override public Optional getColumnStatistics( ConnectorSession session, ConnectorTableHandle handle, String columnName) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getColumnStatistics(session, handle, columnName); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; if (hiveHandle.getTableType() != HiveTableType.HIVE) { return Optional.empty(); @@ -580,6 +625,9 @@ private static long parseLongOrDefault(String value, long defaultValue) { */ @Override public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).estimateDataSizeByListingFiles(session, handle); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; if (hiveHandle.getTableType() != HiveTableType.HIVE) { return -1; @@ -727,6 +775,10 @@ private Configuration buildHadoopConf() { public Optional> applyFilter( ConnectorSession session, ConnectorTableHandle handle, ConnectorFilterConstraint constraint) { + if (!(handle instanceof HiveTableHandle)) { + // Forward AND return the sibling's result UNMODIFIED (a rewrap would poison a downstream scan cast). + return siblingMetadata(session).applyFilter(session, handle, constraint); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); if (partKeyNames == null || partKeyNames.isEmpty()) { @@ -779,6 +831,9 @@ public Optional> applyFilter( */ @Override public List listPartitionNames(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).listPartitionNames(session, handle); + } return collectPartitionNames((HiveTableHandle) handle); } @@ -799,6 +854,9 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH @Override public List listPartitions(ConnectorSession session, ConnectorTableHandle handle, Optional filter) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).listPartitions(session, handle, filter); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); List partitionNames = collectPartitionNames(hiveHandle); @@ -863,6 +921,12 @@ private static Map toPartitionValueMap(String partitionName, Lis @Override public Optional beginQuerySnapshot(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + // Diverts in lockstep with getTableFreshness/getPartitionFreshnessMillis: the pin's + // isLastModifiedFreshness flag (false for an iceberg snapshot-id pin) gates whether fe-core consults + // freshness at all, so half-diverting the pin would corrupt MVCC. + return siblingMetadata(session).beginQuerySnapshot(session, handle); + } return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(-1L).lastModifiedFreshness(true).build()); } @@ -883,6 +947,9 @@ public Optional beginQuerySnapshot(ConnectorSession sessi @Override public Optional getTableFreshness(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getTableFreshness(session, handle); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); if (partKeyNames == null || partKeyNames.isEmpty()) { @@ -921,6 +988,9 @@ public Optional getTableFreshness(ConnectorSession sess @Override public OptionalLong getPartitionFreshnessMillis(ConnectorSession session, ConnectorTableHandle handle, String partitionName) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getPartitionFreshnessMillis(session, handle, partitionName); + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partitions = hmsClient.getPartitions(hiveHandle.getDbName(), hiveHandle.getTableName(), Collections.singletonList(partitionName)); @@ -942,6 +1012,94 @@ private static long lastDdlMillis(Map parameters) { return Long.parseLong(parameters.get(TRANSIENT_LAST_DDL_TIME)) * 1000; } + // ========== Iceberg-on-HMS sibling delegation (forward-absent) ========== + // Handle-based methods hive does NOT implement (it uses the SPI default) but iceberg DOES. Without an + // override here a delegated iceberg-on-HMS table would get hive's SPI default — a SILENT wrong answer + // (empty MVCC/time-travel/sys-tables, or an unthreaded snapshot pin), not a fail-loud. Each forwards a + // foreign handle to the sibling and reproduces the SPI default for a real hive handle. The handle-out + // methods (apply* / getSysTableHandle) return the sibling's handle UNMODIFIED — a rewrap would poison a + // downstream scan cast. + + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle, + ConnectorMvccSnapshot snapshot) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getTableSchema(session, handle, snapshot); + } + // Hive has no schema-at-snapshot; the SPI default ignores the snapshot and returns the latest schema. + return getTableSchema(session, handle); + } + + @Override + public Optional getMvccPartitionView(ConnectorSession session, + ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).getMvccPartitionView(session, handle); + } + // Hive has no range-aware partition view; fe-core builds it from listPartitions (SPI default empty). + return Optional.empty(); + } + + @Override + public Optional resolveTimeTravel(ConnectorSession session, + ConnectorTableHandle handle, ConnectorTimeTravelSpec spec) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).resolveTimeTravel(session, handle, spec); + } + // Hive has no time travel (SPI default empty): an explicit spec on a hive table is unsupported upstream. + return Optional.empty(); + } + + @Override + public ConnectorTableHandle applySnapshot(ConnectorSession session, ConnectorTableHandle handle, + ConnectorMvccSnapshot snapshot) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).applySnapshot(session, handle, snapshot); + } + // Hive's empty pin carries no scan options; the SPI default returns the handle unchanged. + return handle; + } + + @Override + public ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, ConnectorTableHandle handle, + Set rawDataFilePaths) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).applyRewriteFileScope(session, handle, rawDataFilePaths); + } + // Hive has no distributed rewrite scope; the SPI default returns the handle unchanged. + return handle; + } + + @Override + public ConnectorTableHandle applyTopnLazyMaterialization(ConnectorSession session, + ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).applyTopnLazyMaterialization(session, handle); + } + // Hive scan metadata already spans all columns; the SPI default returns the handle unchanged. + return handle; + } + + @Override + public List listSupportedSysTables(ConnectorSession session, ConnectorTableHandle baseTableHandle) { + if (!(baseTableHandle instanceof HiveTableHandle)) { + return siblingMetadata(session).listSupportedSysTables(session, baseTableHandle); + } + // Hive exposes no system tables (SPI default empty). + return Collections.emptyList(); + } + + @Override + public Optional getSysTableHandle(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + if (!(baseTableHandle instanceof HiveTableHandle)) { + // Return the sibling's sys-table handle UNMODIFIED (a rewrap would poison a downstream scan cast). + return siblingMetadata(session).getSysTableHandle(session, baseTableHandle, sysName); + } + // Hive exposes no system tables (SPI default empty). + return Optional.empty(); + } + /** * Renders {@code key1=v1/key2=v2} from partition-key names + values, byte-parity with legacy * {@code HivePartition.getPartitionName} (a raw, unescaped {@code name=value} join). Used only to label @@ -1128,6 +1286,10 @@ public void createTable(ConnectorSession session, ConnectorCreateTableRequest re */ @Override public void dropTable(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).dropTable(session, handle); + return; + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; try { // The handle was just built by the bridge's getTableHandle (which loaded the table), so its @@ -1148,6 +1310,10 @@ public void dropTable(ConnectorSession session, ConnectorTableHandle handle) { */ @Override public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, List partitions) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).truncateTable(session, handle, partitions); + return; + } HiveTableHandle hiveHandle = (HiveTableHandle) handle; try { hmsClient.truncateTable(hiveHandle.getDbName(), hiveHandle.getTableName(), partitions); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java new file mode 100644 index 00000000000000..4a9227c5ad4ecb --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -0,0 +1,347 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorColumnStatistics; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorPartitionInfo; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorTableStatistics; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; +import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorFilterConstraint; +import org.apache.doris.connector.api.pushdown.FilterApplicationResult; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.OptionalLong; +import java.util.Set; + +/** + * Pins the HMS-cutover §4.4 S3 gateway metadata delegation: {@link HiveConnectorMetadata}'s per-handle methods + * route by the concrete handle type — a hive handle runs the existing hive logic, a foreign (iceberg) handle is + * forwarded to the embedded iceberg sibling connector — and NEVER cast the foreign handle. + * + *

    Dormant until hms enters {@code SPI_READY_TYPES}: no production path builds a foreign handle for this + * metadata yet, so these assertions are a Rule-9 guard that the divert contract (forward every per-handle read + + * DROP/TRUNCATE, return the sibling's handle UNMODIFIED, fill the iceberg-only silent gaps) is correct BEFORE the + * flip wires it. The hive-handle byte-parity for these methods is covered by the existing per-method suites.

    + */ +public class HiveConnectorMetadataSiblingDelegationTest { + + /** A foreign (non-hive) handle — the marker type the iceberg sibling's getTableHandle produces post-flip. */ + private static final class ForeignHandle implements ConnectorTableHandle { + } + + private final ForeignHandle foreignHandle = new ForeignHandle(); + private final RecordingSiblingMetadata siblingMetadata = new RecordingSiblingMetadata(); + private final RecordingSiblingConnector siblingConnector = new RecordingSiblingConnector(siblingMetadata); + + /** Metadata wired with a working sibling (hmsClient is null: the hive path is never exercised here). */ + private HiveConnectorMetadata withSibling() { + return new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), + () -> siblingConnector); + } + + private HiveTableHandle hiveHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE).build(); + } + + @Test + public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { + HiveConnectorMetadata md = withSibling(); + + // ---- set (a): methods hive overrides — a foreign handle must NOT run hive logic, it must divert ---- + md.getTableSchema(null, foreignHandle); + md.getColumnHandles(null, foreignHandle); + md.getTableStatistics(null, foreignHandle); + md.getColumnStatistics(null, foreignHandle, "c"); + long size = md.estimateDataSizeByListingFiles(null, foreignHandle); + Optional> filter = md.applyFilter(null, foreignHandle, null); + List partNames = md.listPartitionNames(null, foreignHandle); + md.listPartitions(null, foreignHandle, Optional.empty()); + ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, foreignHandle).orElse(null); + md.getTableFreshness(null, foreignHandle); + md.getPartitionFreshnessMillis(null, foreignHandle, "p"); + md.dropTable(null, foreignHandle); + md.truncateTable(null, foreignHandle, Collections.emptyList()); + + // ---- set (b): methods hive does NOT override — the silent gaps that must be filled by forwarding ---- + md.getTableSchema(null, foreignHandle, null); + md.getMvccPartitionView(null, foreignHandle); + md.resolveTimeTravel(null, foreignHandle, null); + ConnectorTableHandle afterSnapshot = md.applySnapshot(null, foreignHandle, null); + ConnectorTableHandle afterScope = md.applyRewriteFileScope(null, foreignHandle, Collections.emptySet()); + ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(null, foreignHandle); + List sysTables = md.listSupportedSysTables(null, foreignHandle); + Optional sysHandle = md.getSysTableHandle(null, foreignHandle, "snapshots"); + + // Every per-handle method reached the sibling (proves the divert covers the whole surface). + Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_METHODS, siblingMetadata.calls, + "every per-handle read + DROP/TRUNCATE + iceberg-only gap method must forward a foreign handle"); + + // A few return values prove the ANSWER is the sibling's, not hive's default. + Assertions.assertEquals(RecordingSiblingMetadata.SENTINEL_SIZE, size, + "estimateDataSize must return the sibling's value, not hive's -1"); + Assertions.assertEquals(RecordingSiblingMetadata.SENTINEL_SNAPSHOT_ID, pin.getSnapshotId(), + "beginQuerySnapshot must return the sibling's snapshot-id pin, not hive's -1 last-modified pin"); + Assertions.assertEquals(Collections.singletonList("sibling-part"), partNames, + "listPartitionNames must return the sibling's names"); + Assertions.assertEquals(Collections.singletonList("snapshots"), sysTables, + "iceberg-on-HMS system tables must resolve through the sibling (hive exposes none)"); + + // Handle-out methods must return the sibling's handle/result UNMODIFIED (a rewrap poisons a scan cast). + Assertions.assertSame(siblingMetadata.filterResult, filter, "applyFilter must return the sibling result"); + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_HANDLE, afterSnapshot, + "applySnapshot must thread and return the sibling's handle unmodified"); + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_HANDLE, afterScope, + "applyRewriteFileScope must return the sibling's handle unmodified"); + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_HANDLE, afterTopn, + "applyTopnLazyMaterialization must return the sibling's handle unmodified"); + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_HANDLE, sysHandle.orElse(null), + "getSysTableHandle must return the sibling's sys-table handle unmodified"); + } + + @Test + public void hiveHandleRunsHiveBranchAndNeverConsultsSibling() { + HiveConnectorMetadata md = withSibling(); + HiveTableHandle hive = hiveHandle(); + + // The set-(b) + beginQuerySnapshot branches reproduce the SPI default / hive pin WITHOUT the sibling and + // without touching the (null) hmsClient — proving the guard falls through to the hive path for a hive handle. + Assertions.assertFalse(md.getMvccPartitionView(null, hive).isPresent(), "hive has no range partition view"); + Assertions.assertFalse(md.resolveTimeTravel(null, hive, null).isPresent(), "hive has no time travel"); + Assertions.assertSame(hive, md.applySnapshot(null, hive, null), "hive applySnapshot returns the handle"); + Assertions.assertSame(hive, md.applyRewriteFileScope(null, hive, Collections.emptySet()), + "hive applyRewriteFileScope returns the handle"); + Assertions.assertSame(hive, md.applyTopnLazyMaterialization(null, hive), + "hive applyTopnLazyMaterialization returns the handle"); + Assertions.assertTrue(md.listSupportedSysTables(null, hive).isEmpty(), "hive exposes no system tables"); + Assertions.assertFalse(md.getSysTableHandle(null, hive, "snapshots").isPresent(), + "hive exposes no system tables"); + ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, hive).orElse(null); + Assertions.assertNotNull(pin); + Assertions.assertEquals(-1L, pin.getSnapshotId(), "hive's pin is the empty (-1) last-modified pin"); + Assertions.assertTrue(pin.isLastModifiedFreshness(), "hive's pin flags last-modified freshness"); + + Assertions.assertEquals(0, siblingConnector.getMetadataCount, + "a hive handle must never build/consult the iceberg sibling"); + Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle"); + } + + @Test + public void foreignHandleFailsLoudWhenNoSiblingConfigured() { + // The 3-arg constructor (hive-only construction) installs a fail-loud supplier: a foreign handle must + // raise a clear error, not NPE deep in a forward. + HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), + new FakeConnectorContext()); + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableSchema(null, foreignHandle), + "a foreign handle with no sibling configured must fail loud"); + } + + /** A sibling {@link Connector} whose getMetadata hands back the recording metadata and counts the calls. */ + private static final class RecordingSiblingConnector implements Connector { + private final ConnectorMetadata metadata; + private int getMetadataCount; + + RecordingSiblingConnector(ConnectorMetadata metadata) { + this.metadata = metadata; + } + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + getMetadataCount++; + return metadata; + } + } + + /** Records each forwarded method name and returns distinguishable sentinels. */ + private static final class RecordingSiblingMetadata implements ConnectorMetadata { + static final ConnectorTableHandle SIBLING_HANDLE = new ForeignHandle(); + static final long SENTINEL_SIZE = 4242L; + static final long SENTINEL_SNAPSHOT_ID = 99L; + + // The exact set + order of forwarded methods the foreign-handle test drives (a Rule-9 completeness lock: + // dropping a guard, or adding one that should not forward, changes this list and fails the test). + static final List EXPECTED_METHODS = Collections.unmodifiableList(Arrays.asList( + "getTableSchema", "getColumnHandles", "getTableStatistics", "getColumnStatistics", + "estimateDataSizeByListingFiles", "applyFilter", "listPartitionNames", "listPartitions", + "beginQuerySnapshot", "getTableFreshness", "getPartitionFreshnessMillis", "dropTable", + "truncateTable", "getTableSchemaAtSnapshot", "getMvccPartitionView", "resolveTimeTravel", + "applySnapshot", "applyRewriteFileScope", "applyTopnLazyMaterialization", "listSupportedSysTables", + "getSysTableHandle")); + + final List calls = new ArrayList<>(); + final Optional> filterResult = + Optional.of(new FilterApplicationResult<>(SIBLING_HANDLE, null, false)); + + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("getTableSchema"); + return new ConnectorTableSchema("sibling", Collections.emptyList(), "iceberg", Collections.emptyMap()); + } + + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle, + ConnectorMvccSnapshot snapshot) { + calls.add("getTableSchemaAtSnapshot"); + return new ConnectorTableSchema("sibling", Collections.emptyList(), "iceberg", Collections.emptyMap()); + } + + @Override + public Map getColumnHandles(ConnectorSession session, + ConnectorTableHandle handle) { + calls.add("getColumnHandles"); + return Collections.emptyMap(); + } + + @Override + public Optional getTableStatistics(ConnectorSession session, + ConnectorTableHandle handle) { + calls.add("getTableStatistics"); + return Optional.of(new ConnectorTableStatistics(1L, 2L)); + } + + @Override + public Optional getColumnStatistics(ConnectorSession session, + ConnectorTableHandle handle, String columnName) { + calls.add("getColumnStatistics"); + return Optional.empty(); + } + + @Override + public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("estimateDataSizeByListingFiles"); + return SENTINEL_SIZE; + } + + @Override + public Optional> applyFilter(ConnectorSession session, + ConnectorTableHandle handle, ConnectorFilterConstraint constraint) { + calls.add("applyFilter"); + return filterResult; + } + + @Override + public List listPartitionNames(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("listPartitionNames"); + return Collections.singletonList("sibling-part"); + } + + @Override + public List listPartitions(ConnectorSession session, ConnectorTableHandle handle, + Optional filter) { + calls.add("listPartitions"); + return Collections.emptyList(); + } + + @Override + public Optional beginQuerySnapshot(ConnectorSession session, + ConnectorTableHandle handle) { + calls.add("beginQuerySnapshot"); + return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(SENTINEL_SNAPSHOT_ID).build()); + } + + @Override + public Optional getTableFreshness(ConnectorSession session, + ConnectorTableHandle handle) { + calls.add("getTableFreshness"); + return Optional.empty(); + } + + @Override + public OptionalLong getPartitionFreshnessMillis(ConnectorSession session, ConnectorTableHandle handle, + String partitionName) { + calls.add("getPartitionFreshnessMillis"); + return OptionalLong.of(55L); + } + + @Override + public void dropTable(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("dropTable"); + } + + @Override + public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, List partitions) { + calls.add("truncateTable"); + } + + @Override + public Optional getMvccPartitionView(ConnectorSession session, + ConnectorTableHandle handle) { + calls.add("getMvccPartitionView"); + return Optional.empty(); + } + + @Override + public Optional resolveTimeTravel(ConnectorSession session, + ConnectorTableHandle handle, ConnectorTimeTravelSpec spec) { + calls.add("resolveTimeTravel"); + return Optional.empty(); + } + + @Override + public ConnectorTableHandle applySnapshot(ConnectorSession session, ConnectorTableHandle handle, + ConnectorMvccSnapshot snapshot) { + calls.add("applySnapshot"); + return SIBLING_HANDLE; + } + + @Override + public ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, ConnectorTableHandle handle, + Set rawDataFilePaths) { + calls.add("applyRewriteFileScope"); + return SIBLING_HANDLE; + } + + @Override + public ConnectorTableHandle applyTopnLazyMaterialization(ConnectorSession session, + ConnectorTableHandle handle) { + calls.add("applyTopnLazyMaterialization"); + return SIBLING_HANDLE; + } + + @Override + public List listSupportedSysTables(ConnectorSession session, ConnectorTableHandle baseTableHandle) { + calls.add("listSupportedSysTables"); + return Collections.singletonList("snapshots"); + } + + @Override + public Optional getSysTableHandle(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + calls.add("getSysTableHandle"); + return Optional.of(SIBLING_HANDLE); + } + } +} From d09856eafc9b6cbb12cc3b7db263bd62604d59da Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 09:28:40 +0800 Subject: [PATCH 069/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20S3=20metadata=20handle-forward=20DONE;=20next=20=3D=20S4?= =?UTF-8?q?=20getTableHandle=20iceberg=20divert?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit S3 committed at b80c614e5b4 (dormant): HiveConnectorMetadata's 21 per-handle methods guard-and-forward a foreign iceberg handle to the embedded sibling (13 prefix guards on hive-overridden methods + 8 new gap-fill overrides for the silent iceberg-only subset), handle-out methods return the sibling handle unmodified. Recon wf_2676a99b-efa classified the full 67-method surface; adversarial review wf_e393f9ec-b68: 0 findings. Records the scope corrections (buildTableDescriptor -> S6; ALTER-DDL/write-validation deferred to the write cutover; applyProjection/applyLimit/listPartitionValues inert). Next = S4 (getTableHandle returns the raw foreign iceberg handle for ICEBERG tables, lighting up S3). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index cc6c1831c8720f..c037f9b8cfc4a5 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -61,7 +61,8 @@ - **✅ §4.4a 兄弟连接器 SPI 接缝 DONE（`45dcca32fa8`）**（recon `wf_da155752-ebb` 6维+2 critic + 对抗复审 `wf_caef8bf7-ec2` 3镜头，全 HEAD 核对；分解权威文档 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`，起步必读）。新 `ConnectorContext.createSiblingConnector(String catalogType, Map props)` 默认 **`return null`**（对齐所有其它 SPI 默认的安全默认约定——**非** throw，更正设计 §4.4 的 UnsupportedOperationException）；fe-core `DefaultConnectorContext` 覆写 = `ConnectorFactory.createConnector(catalogType, props, this)`（传 `this` → 兄弟连接器复用本目录 id/auth/storage，其 hms-flavor HiveCatalog 名默认取 `context.getCatalogName()`，故共享 context 承重；fe-core 保持 connector-agnostic、零属性解析——props 由网关合成）；iceberg+paimon `TcclPinningConnectorContext` 装饰器补委派该新方法到 raw context（保持"穷尽 pass-through"契约为真——兄弟自套 TCCL/auth pin，须收未包裹 context 避免双重 pin）。唯一改引擎的一步、后面所有插件步的地基。全绿（spi 3 + fe-core 3 + iceberg 5 + paimon 4 测 + 4 模块 checkstyle 0 + import gate 净）。**⚠关键设计更正（更正设计 §4.4）**：设计的"网关包装句柄 `{format, delegate}`"方案**已证死**——`PluginDrivenScanNode` 选执行器(:209)与执行扫描(:966)用**同一句柄对象**、中间无拆包缝，`IcebergScanPlanProvider` 无条件强转 `(IcebergTableHandle)`(:319/364/468/1009)→包装句柄必 CCE；**改为** `getTableHandle` 对 iceberg 返**原始外来句柄**（iceberg 强转成功）、网关一律用 `instanceof HiveTableHandle`(自有 hive-loader 类型)→hive、否则转发兄弟，**永不强转外来句柄**，零新 SPI。 - **✅ §4.4 S1 属性合成纯函数 DONE（`d971410af62`）**：`IcebergSiblingProperties.synthesize(gatewayCatalogProps)`=**整份目录属性逐字复制+注入字面量 `iceberg.catalog.type=hms`**（per-catalog、非 per-table，无需 HmsTableInfo；字面量硬编码因 iceberg 常量 child-first 不可见）。**copy-all（非选择性子集）经对抗核验安全**：与原生 `type=iceberg,iceberg.catalog.type=hms` 目录喂给 IcebergConnector 的**完全同形**（fe-core `createConnectorFromProperties` 也传整份目录属性）、iceberg `create()` 零属性校验→hive 侧 delta 键只被忽略不被误用（`warehouse` 无害=HMS flavor 不从它派生 fs.defaultFS、存量表用自存 location；唯一非缺陷=兄弟用 iceberg 默认缓存 TTL 与原生同）。不改输入、覆盖任何 stray flavor。全绿（4 测+checkstyle+import gate）。 - **✅ §4.4 S2 兄弟持有/懒建/close 转发 DONE（`119b4fca64b`）**（对抗 review `wf_c17ad553-53a`：dormancy-parity/classloader-cce/concurrency-lifecycle/contract-test-intent 4 维 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 新增包级 `getOrCreateIcebergSibling()`：volatile 双检（复刻本文件 `getOrCreateClient()` 惯用法）经 `context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))` 懒建；**单例 per-gateway**（非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash，per-op 会碎片化）；**仅以 parent-first `Connector` 接口持有、永不强转**（外来句柄类型 hive 加载器不可见，强转必 CCE）；插件缺失返 null→**fail-loud** `DorisConnectorException`（含目录名）且**失败不 memoize**（字段留 null 下次重试）。`close()` 追加转发兄弟并置 null（引擎只关主连接器，网关自持兄弟生命周期）、未建时 no-op。全绿（fe-connector-hive 194 测 + 6 新 `HiveConnectorSiblingTest` + checkstyle 0 + import gate 净；HEAD 证实 hms 不在 SPI_READY_TYPES=休眠成立）。⚠**复审记账（refuted，勿改）**：`close()` 在 `hmsClient.close()` 之后转发兄弟，若前者抛 IOException 理论会漏关兄弟——但唯一生产 `HmsClient`=`ThriftHmsClient` 的 `close()` 结构性吞异常（`clientPool.close()` 包在 catch 内）实际不抛，非真缺陷，保持与现有 close 惯用法一致。 -- **下一步 = §4.4 S3（休眠，序见分解文档）**：`HiveConnectorMetadata` 每 handle 方法按 `instanceof HiveTableHandle` 守卫转发（**两不相交集**：改写误服务的 hive 实现 + 转发全缺席的 iceberg 子集，后者多为**静默错答**getMvccPartitionView/resolveTimeTravel/applySnapshot/…须一并；`beginQuerySnapshot` 须与 freshness 同步分流，其 last-modified 标志门控 freshness 是否被读；apply* 须转发并原样回传兄弟句柄）→ S4 `getTableHandle` iceberg 分流（返原始外来句柄，点亮 S3）→ S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（返兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法+连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 +- **✅ §4.4 S3 元数据按句柄守卫转发 DONE（`b80c614e5b4`）**（先 code-grounded recon `wf_2676a99b-efa`：全 67 方法面枚举分类 + 完整性 critic 逐条 HEAD 核对；后对抗 review `wf_e393f9ec-b68`：hive-parity/forward-correctness/completeness-scope/wiring-dormancy 4 维 + verify，**0 findings**）。`HiveConnectorMetadata` 每 handle 方法 `if(!(handle instanceof HiveTableHandle)) return siblingMetadata(session).<同名>(…)`，否则跑原 hive body 字节不变；**外来句柄永不强转**。接线=`HiveConnector.getMetadata` 传 `this::getOrCreateIcebergSibling`（`Supplier`），`siblingMetadata(session)` **每次调用现取**（fe-core 亦 per-op 取 metadata，重状态在兄弟连接器上），3-arg 构造器装 fail-loud `NO_ICEBERG_SIBLING`（仅 hive-only 测用）。**21 方法两集**：①13 个 hive 已覆写方法加纯前缀守卫（getTableSchema[2]/getColumnHandles/get{Table,Column}Statistics/estimateDataSize/applyFilter/listPartitions[Names]/beginQuerySnapshot/get{Table,Partition}Freshness/dropTable/truncateTable）；②8 个 hive 缺席但 iceberg 有的新增守卫-转发覆写（schema-at-snapshot / getMvccPartitionView / resolveTimeTravel / applySnapshot / applyRewriteFileScope / applyTopnLazyMaterialization / listSupportedSysTables / getSysTableHandle）——不补则外来表拿 hive SPI 默认=**静默错答**（空 MVCC/时间旅行/系统表 或 未串快照 pin），hive 分支复刻 SPI 默认。`beginQuerySnapshot` 与两 freshness 同步分流（其 isLastModifiedFreshness 标志门控 freshness 是否被读）；handle-out（applyFilter/apply*/getSysTableHandle）**原样回传兄弟句柄**（rewrap 会毒化下游 planScan 强转）。**recon 校正**：`buildTableDescriptor` **无句柄→归 S6**（分解文档 §3 set-a 误列，已证）；`applyProjection/applyLimit/listPartitionValues` 两边皆未覆写=惰性、**故意不守卫**；ALTER-DDL 全家（addColumn/…/create-drop Branch|Tag / partition-field / renameTable）+ `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 带句柄但 hive 未覆写、属**写切换**（另步，非本步）。全绿（fe-connector-hive 197 测 + 3 新 `HiveConnectorMetadataSiblingDelegationTest`〔含转发方法集完整性锁〕 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES 且 S4 未做→getTableHandle 仍返 HiveTableHandle→守卫今皆 no-op、休眠）。 +- **下一步 = §4.4 S4（休眠，序见分解文档）**：`getTableHandle` 对 iceberg 表分流——`HiveTableFormatDetector.detect(...)==ICEBERG` 时**返兄弟的 `getTableHandle`（原始外来 iceberg 句柄）**而非盖 ICEBERG 章的 HiveTableHandle，点亮 S3 的全部守卫（外来句柄开始真的流入）；HUDI 仍 fail-loud/legacy。之后 S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（`instanceof HiveTableHandle`→hive provider，否则兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法（含 buildTableDescriptor、视图族）按 `detect(db,table)` 分流 + 连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续（ALTER-DDL/写校验并入此）。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 From 46855c217ab60255edb1b543246c96189df63cf5 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 10:28:52 +0800 Subject: [PATCH 070/334] [feat](catalog) HMS gateway diverts iceberg-on-HMS tables to iceberg sibling at getTableHandle (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.4 S4 — the pivot that starts a foreign iceberg handle flowing out of the hive gateway. In HiveConnectorMetadata.getTableHandle, once the format detector resolves the HMS table to ICEBERG, return the embedded iceberg SIBLING connector's OWN table handle (the raw foreign iceberg handle) verbatim instead of a HiveTableHandle stamped ICEBERG. Iceberg's scan/metadata path unconditionally casts the handle to its concrete IcebergTableHandle, so a HiveTableHandle would CCE the moment iceberg cast it; the foreign handle is never cast here (its concrete type is invisible across the classloader split). This activates the S3 guard-and-forward overrides: every gateway consumer discriminates by `instanceof HiveTableHandle` (the gateway's own hive-loader type) and forwards any non-hive handle to the sibling. An empty from the sibling (table vanished / iceberg cannot load) passes through unchanged rather than fabricating a hive handle. HUDI is intentionally NOT diverted — hudi-on-HMS delegation is a later substep, so a hudi table stays on the hive-handle path (byte-unchanged). Plain-hive and view tables are untouched. Dormant until hms enters SPI_READY_TYPES: nothing calls getTableHandle for this connector today. New HiveConnectorMetadataTableHandleDivertTest pins the contract: iceberg returns the sibling's handle unmodified (not a HiveTableHandle), the empty pass-through is proven to be forwarded from the sibling, hive/hudi never consult the sibling, a missing table short-circuits to empty, and an iceberg table with no sibling configured fails loud. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 13 + ...onnectorMetadataTableHandleDivertTest.java | 263 ++++++++++++++++++ 2 files changed, 276 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 00b439cb019a5c..c62c396599562d 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -237,6 +237,19 @@ public Optional getTableHandle( } HmsTableInfo tableInfo = hmsClient.getTable(dbName, tableName); HiveTableType tableType = HiveTableFormatDetector.detect(tableInfo); + // Iceberg-on-HMS divert: an iceberg table registered in this HMS catalog is served by the embedded + // iceberg SIBLING connector, not by hive. Return the sibling's OWN table handle (the raw foreign iceberg + // handle) verbatim — NOT a HiveTableHandle stamped ICEBERG — so iceberg's scan/metadata path, which + // unconditionally casts the handle to its concrete IcebergTableHandle, succeeds. This is the pivot that + // activates the guard-and-forward overrides throughout this class: every gateway consumer discriminates by + // `instanceof HiveTableHandle` (the gateway's OWN hive-loader type) and forwards any non-hive handle to + // the sibling; the foreign iceberg handle is NEVER cast here (its concrete type is invisible across the + // classloader split). HUDI is intentionally NOT diverted — hudi-on-HMS delegation is a later substep, so a + // hudi table stays on the hive-handle path below (dormant, unchanged). Dormant overall until hms enters + // SPI_READY_TYPES: today getTableHandle is never called for this connector. + if (tableType == HiveTableType.ICEBERG) { + return siblingMetadata(session).getTableHandle(session, dbName, tableName); + } // Fail-loud parity with legacy HMSExternalTable.supportedHiveTable(), which threw on a null or // unrecognized input format instead of silently degrading (the old detector returned UNKNOWN). A view // short-circuits: legacy returns true for a view before the format check — a view has no data files so diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java new file mode 100644 index 00000000000000..8112a72fc443ba --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java @@ -0,0 +1,263 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Pins the HMS-cutover §4.4 S4 pivot: {@link HiveConnectorMetadata#getTableHandle} diverts an ICEBERG table to the + * embedded iceberg sibling connector — returning the sibling's OWN (foreign iceberg) handle verbatim — while HIVE + * and HUDI tables keep building a {@link HiveTableHandle} on the hive path and never touch the sibling. + * + *

    WHY (Rule 9): S4 is the step that makes a foreign iceberg handle START flowing out of getTableHandle, which is + * what activates every {@code instanceof HiveTableHandle} guard-and-forward override in {@link HiveConnectorMetadata} + * (S3). The contract that must hold BEFORE the flip wires it:

    + *
      + *
    • An iceberg-on-HMS table must return the iceberg connector's handle unchanged (so iceberg's own + * unconditional {@code (IcebergTableHandle)} cast on the scan/metadata path succeeds) — NOT a HiveTableHandle + * stamped ICEBERG, which would CCE the moment iceberg cast it.
      • + *
    • HUDI is deliberately left on the hive-handle path (its delegation is a later substep); diverting it now + * would silently reroute hudi reads to iceberg.
      • + *
    • A pure-hive table must resolve entirely on the hive path, never building the sibling (a hive-only + * deployment has no iceberg plugin).
      • + *
    + * + *

    Dormant until hms enters {@code SPI_READY_TYPES}: no production path calls getTableHandle for this connector + * yet, so these assertions are a guard, not a live-path test.

    + */ +public class HiveConnectorMetadataTableHandleDivertTest { + + private static final String PARQUET = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String HUDI = "org.apache.hudi.hadoop.HoodieParquetInputFormat"; + + /** The foreign (non-hive) handle the iceberg sibling's getTableHandle produces post-flip. */ + private static final class ForeignHandle implements ConnectorTableHandle { + } + + private final ForeignHandle siblingHandle = new ForeignHandle(); + private final RecordingSibling siblingConnector = new RecordingSibling(siblingHandle); + + @Test + public void icebergTableDivertsAndReturnsSiblingHandleUnmodified() { + HiveConnectorMetadata md = withSibling(icebergTable()); + + Optional handle = md.getTableHandle(null, "db", "t"); + + Assertions.assertTrue(handle.isPresent(), "an existing iceberg-on-HMS table must resolve a handle"); + Assertions.assertSame(siblingHandle, handle.get(), + "getTableHandle must return the iceberg sibling's OWN handle unmodified (a rewrap poisons the " + + "downstream iceberg cast)"); + Assertions.assertFalse(handle.get() instanceof HiveTableHandle, + "an iceberg table must NOT be served a HiveTableHandle stamped ICEBERG"); + Assertions.assertEquals(1, siblingConnector.metadata.getTableHandleCalls, + "the divert must consult the sibling exactly once"); + } + + @Test + public void icebergDivertPropagatesSiblingEmpty() { + // The sibling is authoritative for iceberg existence: if its getTableHandle returns empty (e.g. the + // iceberg catalog cannot load it), the gateway forwards that empty rather than fabricating a hive handle. + siblingConnector.metadata.returnHandle = null; + HiveConnectorMetadata md = withSibling(icebergTable()); + + Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), + "an empty from the sibling must pass through unchanged"); + // Prove the empty was FORWARDED from the sibling (its getTableHandle was consulted), not short-circuited + // to empty by the gateway — otherwise a broken `if (ICEBERG) return Optional.empty()` would pass this test. + Assertions.assertEquals(1, siblingConnector.metadata.getTableHandleCalls, + "the sibling is authoritative for iceberg existence: its getTableHandle must be the source of empty"); + } + + @Test + public void hiveTableBuildsHiveHandleWithoutConsultingSibling() { + HiveConnectorMetadata md = withSibling(hiveTable(PARQUET)); + + Optional handle = md.getTableHandle(null, "db", "t"); + + Assertions.assertTrue(handle.get() instanceof HiveTableHandle, "a hive table resolves a HiveTableHandle"); + Assertions.assertEquals(HiveTableType.HIVE, ((HiveTableHandle) handle.get()).getTableType()); + Assertions.assertEquals(0, siblingConnector.getMetadataCalls, + "a hive table must never build or consult the iceberg sibling"); + } + + @Test + public void hudiTableStaysOnHiveHandlePathNotDiverted() { + // HUDI is NOT diverted in S4: it keeps building a HUDI-stamped HiveTableHandle (its own delegation is a + // later substep). Diverting it here would silently reroute hudi reads to the iceberg sibling. + HiveConnectorMetadata md = withSibling(hiveTable(HUDI)); + + Optional handle = md.getTableHandle(null, "db", "t"); + + Assertions.assertTrue(handle.get() instanceof HiveTableHandle, "a hudi table stays on the hive path"); + Assertions.assertEquals(HiveTableType.HUDI, ((HiveTableHandle) handle.get()).getTableType()); + Assertions.assertEquals(0, siblingConnector.getMetadataCalls, + "a hudi table must NOT be diverted to the iceberg sibling"); + } + + @Test + public void missingTableReturnsEmptyWithoutConsultingSibling() { + HiveConnectorMetadata md = new HiveConnectorMetadata( + new FakeHmsClient(icebergTable(), false), Collections.emptyMap(), new FakeConnectorContext(), + () -> siblingConnector); + + Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), + "a non-existent table short-circuits to empty before any format detection or divert"); + Assertions.assertEquals(0, siblingConnector.getMetadataCalls, "a missing table must not build the sibling"); + } + + @Test + public void icebergTableFailsLoudWhenNoSiblingConfigured() { + // The 3-arg constructor (hive-only construction) installs a fail-loud sibling supplier: an iceberg table + // must raise a clear error, not NPE, when the iceberg plugin is unavailable. + HiveConnectorMetadata md = new HiveConnectorMetadata( + new FakeHmsClient(icebergTable(), true), Collections.emptyMap(), new FakeConnectorContext()); + + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"), + "an iceberg table with no sibling configured must fail loud"); + } + + // ===== helpers ===== + + private HiveConnectorMetadata withSibling(HmsTableInfo tableInfo) { + return new HiveConnectorMetadata(new FakeHmsClient(tableInfo, true), Collections.emptyMap(), + new FakeConnectorContext(), () -> siblingConnector); + } + + private static HmsTableInfo hiveTable(String inputFormat) { + return HmsTableInfo.builder() + .dbName("db").tableName("t").tableType("MANAGED_TABLE") + .inputFormat(inputFormat) + .build(); + } + + private static HmsTableInfo icebergTable() { + return HmsTableInfo.builder() + .dbName("db").tableName("t").tableType("EXTERNAL_TABLE") + .parameters(Collections.singletonMap("table_type", "ICEBERG")) + .build(); + } + + /** A sibling {@link Connector} whose getMetadata hands back a recording metadata and counts the builds. */ + private static final class RecordingSibling implements Connector { + private final RecordingSiblingMetadata metadata; + private int getMetadataCalls; + + RecordingSibling(ConnectorTableHandle handle) { + this.metadata = new RecordingSiblingMetadata(handle); + } + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + getMetadataCalls++; + return metadata; + } + } + + /** Records getTableHandle calls and returns a configurable foreign handle (null -> empty). */ + private static final class RecordingSiblingMetadata implements ConnectorMetadata { + private ConnectorTableHandle returnHandle; + private int getTableHandleCalls; + + RecordingSiblingMetadata(ConnectorTableHandle handle) { + this.returnHandle = handle; + } + + @Override + public Optional getTableHandle(ConnectorSession session, String dbName, + String tableName) { + getTableHandleCalls++; + return Optional.ofNullable(returnHandle); + } + } + + /** Minimal {@link HmsClient} double serving one prebuilt table; the rest fail loud. */ + private static final class FakeHmsClient implements HmsClient { + private final HmsTableInfo table; + private final boolean exists; + + FakeHmsClient(HmsTableInfo table, boolean exists) { + this.table = table; + this.exists = exists; + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return exists; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + return table; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} From 320f75103dc13b98a9caed6273c3c67fa254655f Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 10:29:57 +0800 Subject: [PATCH 071/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20S4=20getTableHandle=20iceberg-divert=20DONE;=20next=20=3D?= =?UTF-8?q?=20S5=20scan-provider=20gateway?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c037f9b8cfc4a5..781ab0aa23c263 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -62,7 +62,8 @@ - **✅ §4.4 S1 属性合成纯函数 DONE（`d971410af62`）**：`IcebergSiblingProperties.synthesize(gatewayCatalogProps)`=**整份目录属性逐字复制+注入字面量 `iceberg.catalog.type=hms`**（per-catalog、非 per-table，无需 HmsTableInfo；字面量硬编码因 iceberg 常量 child-first 不可见）。**copy-all（非选择性子集）经对抗核验安全**：与原生 `type=iceberg,iceberg.catalog.type=hms` 目录喂给 IcebergConnector 的**完全同形**（fe-core `createConnectorFromProperties` 也传整份目录属性）、iceberg `create()` 零属性校验→hive 侧 delta 键只被忽略不被误用（`warehouse` 无害=HMS flavor 不从它派生 fs.defaultFS、存量表用自存 location；唯一非缺陷=兄弟用 iceberg 默认缓存 TTL 与原生同）。不改输入、覆盖任何 stray flavor。全绿（4 测+checkstyle+import gate）。 - **✅ §4.4 S2 兄弟持有/懒建/close 转发 DONE（`119b4fca64b`）**（对抗 review `wf_c17ad553-53a`：dormancy-parity/classloader-cce/concurrency-lifecycle/contract-test-intent 4 维 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 新增包级 `getOrCreateIcebergSibling()`：volatile 双检（复刻本文件 `getOrCreateClient()` 惯用法）经 `context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))` 懒建；**单例 per-gateway**（非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash，per-op 会碎片化）；**仅以 parent-first `Connector` 接口持有、永不强转**（外来句柄类型 hive 加载器不可见，强转必 CCE）；插件缺失返 null→**fail-loud** `DorisConnectorException`（含目录名）且**失败不 memoize**（字段留 null 下次重试）。`close()` 追加转发兄弟并置 null（引擎只关主连接器，网关自持兄弟生命周期）、未建时 no-op。全绿（fe-connector-hive 194 测 + 6 新 `HiveConnectorSiblingTest` + checkstyle 0 + import gate 净；HEAD 证实 hms 不在 SPI_READY_TYPES=休眠成立）。⚠**复审记账（refuted，勿改）**：`close()` 在 `hmsClient.close()` 之后转发兄弟，若前者抛 IOException 理论会漏关兄弟——但唯一生产 `HmsClient`=`ThriftHmsClient` 的 `close()` 结构性吞异常（`clientPool.close()` 包在 catch 内）实际不抛，非真缺陷，保持与现有 close 惯用法一致。 - **✅ §4.4 S3 元数据按句柄守卫转发 DONE（`b80c614e5b4`）**（先 code-grounded recon `wf_2676a99b-efa`：全 67 方法面枚举分类 + 完整性 critic 逐条 HEAD 核对；后对抗 review `wf_e393f9ec-b68`：hive-parity/forward-correctness/completeness-scope/wiring-dormancy 4 维 + verify，**0 findings**）。`HiveConnectorMetadata` 每 handle 方法 `if(!(handle instanceof HiveTableHandle)) return siblingMetadata(session).<同名>(…)`，否则跑原 hive body 字节不变；**外来句柄永不强转**。接线=`HiveConnector.getMetadata` 传 `this::getOrCreateIcebergSibling`（`Supplier`），`siblingMetadata(session)` **每次调用现取**（fe-core 亦 per-op 取 metadata，重状态在兄弟连接器上），3-arg 构造器装 fail-loud `NO_ICEBERG_SIBLING`（仅 hive-only 测用）。**21 方法两集**：①13 个 hive 已覆写方法加纯前缀守卫（getTableSchema[2]/getColumnHandles/get{Table,Column}Statistics/estimateDataSize/applyFilter/listPartitions[Names]/beginQuerySnapshot/get{Table,Partition}Freshness/dropTable/truncateTable）；②8 个 hive 缺席但 iceberg 有的新增守卫-转发覆写（schema-at-snapshot / getMvccPartitionView / resolveTimeTravel / applySnapshot / applyRewriteFileScope / applyTopnLazyMaterialization / listSupportedSysTables / getSysTableHandle）——不补则外来表拿 hive SPI 默认=**静默错答**（空 MVCC/时间旅行/系统表 或 未串快照 pin），hive 分支复刻 SPI 默认。`beginQuerySnapshot` 与两 freshness 同步分流（其 isLastModifiedFreshness 标志门控 freshness 是否被读）；handle-out（applyFilter/apply*/getSysTableHandle）**原样回传兄弟句柄**（rewrap 会毒化下游 planScan 强转）。**recon 校正**：`buildTableDescriptor` **无句柄→归 S6**（分解文档 §3 set-a 误列，已证）；`applyProjection/applyLimit/listPartitionValues` 两边皆未覆写=惰性、**故意不守卫**；ALTER-DDL 全家（addColumn/…/create-drop Branch|Tag / partition-field / renameTable）+ `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 带句柄但 hive 未覆写、属**写切换**（另步，非本步）。全绿（fe-connector-hive 197 测 + 3 新 `HiveConnectorMetadataSiblingDelegationTest`〔含转发方法集完整性锁〕 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES 且 S4 未做→getTableHandle 仍返 HiveTableHandle→守卫今皆 no-op、休眠）。 -- **下一步 = §4.4 S4（休眠，序见分解文档）**：`getTableHandle` 对 iceberg 表分流——`HiveTableFormatDetector.detect(...)==ICEBERG` 时**返兄弟的 `getTableHandle`（原始外来 iceberg 句柄）**而非盖 ICEBERG 章的 HiveTableHandle，点亮 S3 的全部守卫（外来句柄开始真的流入）；HUDI 仍 fail-loud/legacy。之后 S5 `HiveConnector.getScanPlanProvider(handle)` 网关覆写（`instanceof HiveTableHandle`→hive provider，否则兄弟 provider→扫描侧 TCCL 自动 pin）→ S6 按名方法（含 buildTableDescriptor、视图族）按 `detect(db,table)` 分流 + 连接器级能力残留（薄）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）；写/procedure 无 per-handle overload = 读脊柱之外的独立后续（ALTER-DDL/写校验并入此）。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 +- **✅ §4.4 S4 `getTableHandle` iceberg 分流 DONE（`e7c28a39e69`）**（对抗复审 `wf_be91f274-b9b`：divert-correctness/classloader-cce/dormancy-parity/test-intent 4 镜头 + 逐条 verify，**生产码 0 缺陷**，全 HEAD 核对）。`HiveConnectorMetadata.getTableHandle` 在 `HiveTableFormatDetector.detect(tableInfo)==ICEBERG` 时**返 `siblingMetadata(session).getTableHandle(session,db,table)`（原始外来 iceberg 句柄）**、非盖 ICEBERG 的 HiveTableHandle——点亮 S3 全部 `instanceof HiveTableHandle` 守卫（外来句柄开始真流入）。分流放在 fail-loud UNKNOWN 检查**之前**（iceberg 早返，永不触 throw）；detect 内 iceberg(`table_type=ICEBERG`)先于 hudi/hive 判、故 iceberg 优先。**兄弟返 empty 原样透传**（表消失/iceberg 加载不了→不伪造 hive 句柄，兄弟对 iceberg 存在性权威）。**HUDI 不分流**（仍建 HUDI 章 HiveTableHandle、字节不变，hudi 委派属后续子步）；plain-hive/view 不动。网关自身 `hmsClient.tableExists+getTable`（detect 必需）→ 兄弟再自查一次=legacy 双载平价，非缺陷。新 `HiveConnectorMetadataTableHandleDivertTest`(6 测)钉合约：iceberg 返兄弟句柄`assertSame`且非 HiveTableHandle、empty 透传**证经兄弟转发**(getTableHandleCalls==1，复审 Rule-9 补强)、hive/hudi 零咨询兄弟、缺表短路 empty、无兄弟 fail-loud。全绿（fe-connector-hive 203 测〔+6 新〕 + hms 56 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES=休眠成立）。 +- **下一步 = §4.4 S5（休眠，序见分解文档，与 S4 成对）**：`HiveConnector.getScanPlanProvider(ConnectorTableHandle handle)` 网关覆写——`if(handle instanceof HiveTableHandle) return getScanPlanProvider(); return sibling.getScanPlanProvider(handle);`（负判自有 hive-loader 类型=loader-safe；返回的 provider 是 iceberg-loader 所建→`PluginDrivenScanNode.onPluginClassLoader` 按 `provider.getClass().getClassLoader()` **自动 pin 扫描侧 TCCL**，白拿）。之后 S6 按名方法（**含 buildTableDescriptor、视图族 viewExists/getViewDefinition/dropView**）按 `HiveTableFormatDetector.detect(db,table)` 分流 + 连接器级能力残留（薄，per-handle 能力门控 `getCapabilities` 表达不了→留翻闸记账，勿过度构建）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟，见分解文档 §5.1）；写/procedure 无 per-handle overload = 读脊柱之外独立后续（ALTER-DDL/写校验并入此，见分解文档 §6）。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 From 537a695ec5705914db8ff3dcb81e163d7109b9fe Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 10:57:10 +0800 Subject: [PATCH 072/334] [feat](catalog) HMS gateway routes iceberg-on-HMS scans to iceberg sibling scan provider (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.4 S5 — the scan-side pair of the getTableHandle iceberg divert. HiveConnector overrides the per-table scan-provider seam Connector.getScanPlanProvider(handle): a hive handle (the gateway's own hive-loader type) runs the hive scan provider; any foreign handle (the raw iceberg handle the sibling's getTableHandle produced) is delegated to the embedded iceberg sibling's per-handle scan provider, passed through unmodified and never cast. Because the returned sibling provider is built in the iceberg plugin's classloader, PluginDrivenScanNode.onPluginClassLoader auto-pins the scan-thread TCCL to the iceberg loader for free (it keys off provider.getClass().getClassLoader()), so no pinning is needed here. A HUDI-stamped HiveTableHandle is still a HiveTableHandle, so it stays on the hive scan path (its delegation is a later substep). Dormant until hms enters SPI_READY_TYPES: nothing selects a scan provider for this connector today. New HiveConnectorScanProviderDivertTest pins the route: a foreign handle returns the sibling's provider (with the handle reaching the sibling unmodified and the sibling built once), a hive/hudi handle resolves the connector-level hive provider without building or consulting the sibling, and a foreign handle with no iceberg plugin fails loud naming the catalog. The hive/hudi cases stub the no-arg getScanPlanProvider() to isolate the routing from HmsClient construction (a real HmsClient needs a HiveConf, off the unit-test classpath; provider construction is covered by the scan-planning suites). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 21 ++ .../HiveConnectorScanProviderDivertTest.java | 194 ++++++++++++++++++ 2 files changed, 215 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 4323c3b7d3a9ae..fb900a5e2f6640 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.connector.hms.HmsClient; @@ -102,6 +103,26 @@ public ConnectorScanPlanProvider getScanPlanProvider() { return new HiveScanPlanProvider(getOrCreateClient(), properties, readTxnManager); } + /** + * Selects the scan provider for a given table handle — the gateway seam a flipped hms catalog uses to serve + * its iceberg-on-HMS tables from the embedded iceberg sibling. A hive handle (the gateway's OWN hive-loader + * type) runs the hive scan provider; any foreign handle (the raw iceberg handle the sibling's getTableHandle + * produced) is delegated to the sibling's per-handle scan provider. Because the returned sibling provider is + * built in the iceberg plugin's classloader, {@code PluginDrivenScanNode.onPluginClassLoader} auto-pins the + * scan-thread TCCL to the iceberg loader for free (it keys off {@code provider.getClass().getClassLoader()}), + * so no pinning is needed here. The foreign handle is passed through UNMODIFIED and NEVER cast (its concrete + * iceberg type is invisible across the loader split — a cast would CCE). A HUDI table keeps a HiveTableHandle, + * so it stays on the hive scan path (its delegation is a later substep). Pairs with the getTableHandle iceberg + * divert; dormant until hms enters SPI_READY_TYPES (nothing selects a scan provider for this connector today). + */ + @Override + public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + if (handle instanceof HiveTableHandle) { + return getScanPlanProvider(); + } + return getOrCreateIcebergSibling().getScanPlanProvider(handle); + } + @Override public ConnectorWritePlanProvider getWritePlanProvider() { return new HiveWritePlanProvider(getOrCreateClient(), properties, context); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java new file mode 100644 index 00000000000000..caaa5e51d9fbe5 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java @@ -0,0 +1,194 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanRange; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Pins the HMS-cutover §4.4 S5 scan-provider gateway seam: {@link HiveConnector#getScanPlanProvider( + * ConnectorTableHandle)} routes by the concrete handle type — a hive handle runs the hive scan provider, any + * foreign (iceberg-on-HMS) handle is delegated to the embedded iceberg sibling's scan provider — and NEVER casts + * the foreign handle. + * + *

    WHY (Rule 9): this pairs with the getTableHandle iceberg divert. Once getTableHandle returns a foreign + * iceberg handle, the scan of that table must pick the SIBLING's scan provider (iceberg-loader-built, so + * {@code PluginDrivenScanNode.onPluginClassLoader} auto-pins the scan-thread TCCL to the iceberg loader). A hive + * (or hudi-stamped hive) handle must NOT be diverted — a hive-only deployment has no iceberg plugin, and hudi's + * delegation is a later substep. The selection MUST happen at provider-acquisition time, so a wrong route here + * would hand iceberg splits to the hive scanner (or vice versa).

    + * + *

    Dormant until hms enters {@code SPI_READY_TYPES}: no production path selects a scan provider for this + * connector yet, so these assertions are a guard, not a live-path test.

    + */ +public class HiveConnectorScanProviderDivertTest { + + private static final String METASTORE_URI = "thrift://host:9083"; + + /** The foreign (non-hive) handle the iceberg sibling's getTableHandle produces post-flip. */ + private static final class ForeignHandle implements ConnectorTableHandle { + } + + /** + * The connector-level hive scan provider, stubbed so the divert test does not build a real HmsClient. A hive + * handle must resolve to exactly THIS (what the no-arg {@link HiveConnector#getScanPlanProvider()} returns); + * the real no-arg provider construction (which needs a HiveConf, off the unit-test classpath) is covered by + * the scan-planning suites. Distinct instance from any sibling provider so {@code assertSame} is meaningful. + */ + private final ConnectorScanPlanProvider stubbedHiveProvider = new MarkerScanProvider(); + + /** A gateway whose no-arg hive provider is the stub above — isolates the per-handle routing from HmsClient. */ + private HiveConnector gatewayWithStubbedHiveProvider(RecordingSiblingContext context) { + return new HiveConnector(props(), context) { + @Override + public ConnectorScanPlanProvider getScanPlanProvider() { + return stubbedHiveProvider; + } + }; + } + + @Test + public void foreignHandleDelegatesToSiblingScanProviderUnmodified() { + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(props(), context); + ForeignHandle foreign = new ForeignHandle(); + + ConnectorScanPlanProvider provider = connector.getScanPlanProvider(foreign); + + Assertions.assertSame(sibling.provider, provider, + "a foreign handle must return the iceberg sibling's OWN scan provider"); + Assertions.assertSame(foreign, sibling.lastHandle, + "the foreign handle must reach the sibling's per-handle selector UNMODIFIED (a rewrap would " + + "poison the downstream iceberg cast)"); + Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once and consulted"); + } + + @Test + public void hiveHandleUsesHiveProviderWithoutConsultingSibling() { + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = gatewayWithStubbedHiveProvider(context); + HiveTableHandle hive = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE).build(); + + ConnectorScanPlanProvider provider = connector.getScanPlanProvider(hive); + + Assertions.assertSame(stubbedHiveProvider, provider, + "a hive handle resolves to the connector-level hive provider (the no-arg getScanPlanProvider)"); + Assertions.assertEquals(0, context.buildCount, "a hive handle must never build or consult the sibling"); + Assertions.assertNull(sibling.lastHandle, "the sibling's scan provider must not be consulted for a hive handle"); + } + + @Test + public void hudiStampedHiveHandleStaysOnHiveProvider() { + // The route keys on the JVM handle TYPE (HiveTableHandle), not the format enum: a HUDI-stamped hive handle + // is still a HiveTableHandle, so it stays on the hive scan path (hudi delegation is a later substep). + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = gatewayWithStubbedHiveProvider(context); + HiveTableHandle hudi = new HiveTableHandle.Builder("db", "t", HiveTableType.HUDI).build(); + + ConnectorScanPlanProvider provider = connector.getScanPlanProvider(hudi); + + Assertions.assertSame(stubbedHiveProvider, provider, "a hudi-stamped hive handle stays on the hive provider"); + Assertions.assertEquals(0, context.buildCount, "a hudi table must NOT be diverted to the iceberg sibling"); + } + + @Test + public void foreignHandleFailsLoudWhenIcebergPluginAbsent() { + // The seam returns a null sibling when the iceberg plugin is absent; selecting a scan provider for a + // foreign handle must fail loud (naming the catalog), not NPE. + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(props(), context); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> connector.getScanPlanProvider(new ForeignHandle()), + "a foreign handle with no iceberg plugin must fail loud"); + Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); + } + + private static Map props() { + // A metastore uri so the hive-handle branch can build its (offline, lazy-pool) HmsClient; the foreign + // branch never touches it. + Map props = new HashMap<>(); + props.put("hive.metastore.uris", METASTORE_URI); + return props; + } + + /** Records the {@code createSiblingConnector} call and returns a configurable (possibly null) sibling. */ + private static final class RecordingSiblingContext extends FakeConnectorContext { + int buildCount; + Connector siblingToReturn; + + RecordingSiblingContext(Connector siblingToReturn) { + this.siblingToReturn = siblingToReturn; + } + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + buildCount++; + return siblingToReturn; + } + } + + /** A sibling {@link Connector} whose per-handle scan provider is a distinguishable marker, recording the handle. */ + private static final class RecordingSibling implements Connector { + final ConnectorScanPlanProvider provider = new MarkerScanProvider(); + ConnectorTableHandle lastHandle; + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + lastHandle = handle; + return provider; + } + + @Override + public void close() { + } + } + + /** A bare scan provider stand-in; only its identity matters here. */ + private static final class MarkerScanProvider implements ConnectorScanPlanProvider { + @Override + public List planScan(ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter) { + return Collections.emptyList(); + } + } +} From d13aec48e6015786999c11cd60012dabcddaa161 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 10:58:15 +0800 Subject: [PATCH 073/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20S5=20scan-provider=20gateway=20divert=20DONE;=20next=20?= =?UTF-8?q?=3D=20S6=20name-based=20divert=20+=20capability=20residuals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 781ab0aa23c263..d58fb1c4796394 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -63,7 +63,8 @@ - **✅ §4.4 S2 兄弟持有/懒建/close 转发 DONE（`119b4fca64b`）**（对抗 review `wf_c17ad553-53a`：dormancy-parity/classloader-cce/concurrency-lifecycle/contract-test-intent 4 维 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 新增包级 `getOrCreateIcebergSibling()`：volatile 双检（复刻本文件 `getOrCreateClient()` 惯用法）经 `context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))` 懒建；**单例 per-gateway**（非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash，per-op 会碎片化）；**仅以 parent-first `Connector` 接口持有、永不强转**（外来句柄类型 hive 加载器不可见，强转必 CCE）；插件缺失返 null→**fail-loud** `DorisConnectorException`（含目录名）且**失败不 memoize**（字段留 null 下次重试）。`close()` 追加转发兄弟并置 null（引擎只关主连接器，网关自持兄弟生命周期）、未建时 no-op。全绿（fe-connector-hive 194 测 + 6 新 `HiveConnectorSiblingTest` + checkstyle 0 + import gate 净；HEAD 证实 hms 不在 SPI_READY_TYPES=休眠成立）。⚠**复审记账（refuted，勿改）**：`close()` 在 `hmsClient.close()` 之后转发兄弟，若前者抛 IOException 理论会漏关兄弟——但唯一生产 `HmsClient`=`ThriftHmsClient` 的 `close()` 结构性吞异常（`clientPool.close()` 包在 catch 内）实际不抛，非真缺陷，保持与现有 close 惯用法一致。 - **✅ §4.4 S3 元数据按句柄守卫转发 DONE（`b80c614e5b4`）**（先 code-grounded recon `wf_2676a99b-efa`：全 67 方法面枚举分类 + 完整性 critic 逐条 HEAD 核对；后对抗 review `wf_e393f9ec-b68`：hive-parity/forward-correctness/completeness-scope/wiring-dormancy 4 维 + verify，**0 findings**）。`HiveConnectorMetadata` 每 handle 方法 `if(!(handle instanceof HiveTableHandle)) return siblingMetadata(session).<同名>(…)`，否则跑原 hive body 字节不变；**外来句柄永不强转**。接线=`HiveConnector.getMetadata` 传 `this::getOrCreateIcebergSibling`（`Supplier`），`siblingMetadata(session)` **每次调用现取**（fe-core 亦 per-op 取 metadata，重状态在兄弟连接器上），3-arg 构造器装 fail-loud `NO_ICEBERG_SIBLING`（仅 hive-only 测用）。**21 方法两集**：①13 个 hive 已覆写方法加纯前缀守卫（getTableSchema[2]/getColumnHandles/get{Table,Column}Statistics/estimateDataSize/applyFilter/listPartitions[Names]/beginQuerySnapshot/get{Table,Partition}Freshness/dropTable/truncateTable）；②8 个 hive 缺席但 iceberg 有的新增守卫-转发覆写（schema-at-snapshot / getMvccPartitionView / resolveTimeTravel / applySnapshot / applyRewriteFileScope / applyTopnLazyMaterialization / listSupportedSysTables / getSysTableHandle）——不补则外来表拿 hive SPI 默认=**静默错答**（空 MVCC/时间旅行/系统表 或 未串快照 pin），hive 分支复刻 SPI 默认。`beginQuerySnapshot` 与两 freshness 同步分流（其 isLastModifiedFreshness 标志门控 freshness 是否被读）；handle-out（applyFilter/apply*/getSysTableHandle）**原样回传兄弟句柄**（rewrap 会毒化下游 planScan 强转）。**recon 校正**：`buildTableDescriptor` **无句柄→归 S6**（分解文档 §3 set-a 误列，已证）；`applyProjection/applyLimit/listPartitionValues` 两边皆未覆写=惰性、**故意不守卫**；ALTER-DDL 全家（addColumn/…/create-drop Branch|Tag / partition-field / renameTable）+ `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 带句柄但 hive 未覆写、属**写切换**（另步，非本步）。全绿（fe-connector-hive 197 测 + 3 新 `HiveConnectorMetadataSiblingDelegationTest`〔含转发方法集完整性锁〕 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES 且 S4 未做→getTableHandle 仍返 HiveTableHandle→守卫今皆 no-op、休眠）。 - **✅ §4.4 S4 `getTableHandle` iceberg 分流 DONE（`e7c28a39e69`）**（对抗复审 `wf_be91f274-b9b`：divert-correctness/classloader-cce/dormancy-parity/test-intent 4 镜头 + 逐条 verify，**生产码 0 缺陷**，全 HEAD 核对）。`HiveConnectorMetadata.getTableHandle` 在 `HiveTableFormatDetector.detect(tableInfo)==ICEBERG` 时**返 `siblingMetadata(session).getTableHandle(session,db,table)`（原始外来 iceberg 句柄）**、非盖 ICEBERG 的 HiveTableHandle——点亮 S3 全部 `instanceof HiveTableHandle` 守卫（外来句柄开始真流入）。分流放在 fail-loud UNKNOWN 检查**之前**（iceberg 早返，永不触 throw）；detect 内 iceberg(`table_type=ICEBERG`)先于 hudi/hive 判、故 iceberg 优先。**兄弟返 empty 原样透传**（表消失/iceberg 加载不了→不伪造 hive 句柄，兄弟对 iceberg 存在性权威）。**HUDI 不分流**（仍建 HUDI 章 HiveTableHandle、字节不变，hudi 委派属后续子步）；plain-hive/view 不动。网关自身 `hmsClient.tableExists+getTable`（detect 必需）→ 兄弟再自查一次=legacy 双载平价，非缺陷。新 `HiveConnectorMetadataTableHandleDivertTest`(6 测)钉合约：iceberg 返兄弟句柄`assertSame`且非 HiveTableHandle、empty 透传**证经兄弟转发**(getTableHandleCalls==1，复审 Rule-9 补强)、hive/hudi 零咨询兄弟、缺表短路 empty、无兄弟 fail-loud。全绿（fe-connector-hive 203 测〔+6 新〕 + hms 56 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES=休眠成立）。 -- **下一步 = §4.4 S5（休眠，序见分解文档，与 S4 成对）**：`HiveConnector.getScanPlanProvider(ConnectorTableHandle handle)` 网关覆写——`if(handle instanceof HiveTableHandle) return getScanPlanProvider(); return sibling.getScanPlanProvider(handle);`（负判自有 hive-loader 类型=loader-safe；返回的 provider 是 iceberg-loader 所建→`PluginDrivenScanNode.onPluginClassLoader` 按 `provider.getClass().getClassLoader()` **自动 pin 扫描侧 TCCL**，白拿）。之后 S6 按名方法（**含 buildTableDescriptor、视图族 viewExists/getViewDefinition/dropView**）按 `HiveTableFormatDetector.detect(db,table)` 分流 + 连接器级能力残留（薄，per-handle 能力门控 `getCapabilities` 表达不了→留翻闸记账，勿过度构建）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟，见分解文档 §5.1）；写/procedure 无 per-handle overload = 读脊柱之外独立后续（ALTER-DDL/写校验并入此，见分解文档 §6）。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 +- **✅ §4.4 S5 扫描提供者网关分流 DONE（`bb39802751b`）**（对抗复审 `wf_6fc6999e-b80`：route-correctness/classloader-tccl/dormancy-parity/test-intent 4 镜头 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 覆写 per-table 接缝 `getScanPlanProvider(ConnectorTableHandle handle)`——`if(handle instanceof HiveTableHandle) return getScanPlanProvider(); return getOrCreateIcebergSibling().getScanPlanProvider(handle);`（负判自有 hive-loader 类型=loader-safe；外来句柄**原样透传兄弟 per-handle overload、永不强转**）。返回的兄弟 provider 是 iceberg-loader 所建→`PluginDrivenScanNode.onPluginClassLoader`(:516)按 `provider.getClass().getClassLoader()` **自动 pin 扫描侧 TCCL 到 iceberg loader、白拿**（无须本步 pin）。`currentHandle` 构造器由已解析表句柄赋值(:172)、pushdown 变异但从不置 null→路由非 hive 句柄不会误分流。HUDI 章 HiveTableHandle 仍是 HiveTableHandle→留 hive 扫描路径（hudi 委派后续）。SPI 默认(`Connector.getScanPlanProvider(handle)`默认委派 no-arg)→本连接器改前对任意句柄都走 no-arg，改后非 hive 句柄才分流；hms 未 live 故零现存路径受影响。全绿（fe-connector-hive 207 测〔+4 新 `HiveConnectorScanProviderDivertTest`〕 + checkstyle 0 + import gate 净）。**测试隔离**：hive/hudi 例 stub no-arg `getScanPlanProvider()`（真 HmsClient 需 HiveConf、单测类路径无→`getScanNodeProperties` 套另测），只钉路由；foreign/fail-loud 例走真 override。 +- **下一步 = §4.4 S6（休眠，序见分解文档，薄）**：按名方法（**无句柄**：`buildTableDescriptor` + 视图族 `viewExists`/`getViewDefinition`/`dropView`）按 `HiveTableFormatDetector.detect(db,table)` 分流转发兄弟（ICEBERG→兄弟、否则 hive）；连接器级能力残留——`getCapabilities` 是连接器级、表达不了 per-handle 门控（如翻闸时 iceberg/hudi-on-HMS 也会收进 `SUPPORTS_COLUMN_AUTO_ANALYZE` 自动分析、legacy 排除 hudi）→**留翻闸记账，勿过度构建**。S6 完成后 §4.4 读脊柱委派收官。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟，见分解文档 §5.1）；写/procedure/EXECUTE 无 per-handle overload = 读脊柱之外独立后续（新 per-handle SPI overload + ALTER-DDL/写校验并入此，见分解文档 §6）。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 From e5a0d496a2a4e991261ebd87e8345394e550bae8 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 12:21:49 +0800 Subject: [PATCH 074/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20S6=20=3D=20zero-code=20(name-based=20reads=20all=20NO=5FD?= =?UTF-8?q?IVERT);=20read-spine=20S0-S6=20complete;=20next=20=3D=20write?= =?UTF-8?q?=20delegation?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A code-grounded recon (wf_6ac0e047-768: 4 dimension readers + a completeness/correctness critic, all HEAD-verified) proves the §4.4 S6 sketch ("re-run detect(db,table) and forward the view family + buildTableDescriptor for ICEBERG") is unnecessary or actively harmful, so S6 ships ZERO connector code: - buildTableDescriptor: byte-identical output (the sibling is always hms flavor -> same HIVE_TABLE + THiveTable; legacy iceberg-on-HMS also emitted HIVE_TABLE), so a divert only adds a wasteful getTable(detect). - view family (viewExists/getViewDefinition/dropView): legacy-parity for the only reachable objects (iceberg/hudi TABLES return viewExists=false and stay on the getTableHandle divert; an iceberg-on-HMS VIEW stores hive-dialect SQL in HMS view text and is served as a hive view) AND undiscriminable name-only (detect returns UNKNOWN for a view: table_type='iceberg-view' != 'ICEBERG'). - listViewNames stays un-overridden (a divert would double-list views); getTableComment stays SPI-default "" (legacy getComment() returned "" unconditionally -> a divert would surface a comment legacy never showed). - db/catalog methods are format-agnostic. => The §4.4 read-delegation spine (S0-S6) is COMPLETE. Remaining per-table divergences are all write/DDL residuals (dropDatabase-force purge, createTable engine-routing, per-handle getWritePlanProvider/getProcedureOps) or connector-wide capability residuals (SUPPORTS_COLUMN_AUTO_ANALYZE hudi over-admission already documented; supportsLatestSnapshotPreload under-admission) plus the explicit non-goal of first-class iceberg-on-HMS views — all recorded for the flip in the new findings doc. Dormant until hms enters SPI_READY_TYPES. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 6 ++- ...6-name-based-divert-findings-2026-07-08.md | 40 +++++++++++++++++++ 2 files changed, 44 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/hms-s6-name-based-divert-findings-2026-07-08.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d58fb1c4796394..f832eae10be3ba 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**；hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -64,7 +64,9 @@ - **✅ §4.4 S3 元数据按句柄守卫转发 DONE（`b80c614e5b4`）**（先 code-grounded recon `wf_2676a99b-efa`：全 67 方法面枚举分类 + 完整性 critic 逐条 HEAD 核对；后对抗 review `wf_e393f9ec-b68`：hive-parity/forward-correctness/completeness-scope/wiring-dormancy 4 维 + verify，**0 findings**）。`HiveConnectorMetadata` 每 handle 方法 `if(!(handle instanceof HiveTableHandle)) return siblingMetadata(session).<同名>(…)`，否则跑原 hive body 字节不变；**外来句柄永不强转**。接线=`HiveConnector.getMetadata` 传 `this::getOrCreateIcebergSibling`（`Supplier`），`siblingMetadata(session)` **每次调用现取**（fe-core 亦 per-op 取 metadata，重状态在兄弟连接器上），3-arg 构造器装 fail-loud `NO_ICEBERG_SIBLING`（仅 hive-only 测用）。**21 方法两集**：①13 个 hive 已覆写方法加纯前缀守卫（getTableSchema[2]/getColumnHandles/get{Table,Column}Statistics/estimateDataSize/applyFilter/listPartitions[Names]/beginQuerySnapshot/get{Table,Partition}Freshness/dropTable/truncateTable）；②8 个 hive 缺席但 iceberg 有的新增守卫-转发覆写（schema-at-snapshot / getMvccPartitionView / resolveTimeTravel / applySnapshot / applyRewriteFileScope / applyTopnLazyMaterialization / listSupportedSysTables / getSysTableHandle）——不补则外来表拿 hive SPI 默认=**静默错答**（空 MVCC/时间旅行/系统表 或 未串快照 pin），hive 分支复刻 SPI 默认。`beginQuerySnapshot` 与两 freshness 同步分流（其 isLastModifiedFreshness 标志门控 freshness 是否被读）；handle-out（applyFilter/apply*/getSysTableHandle）**原样回传兄弟句柄**（rewrap 会毒化下游 planScan 强转）。**recon 校正**：`buildTableDescriptor` **无句柄→归 S6**（分解文档 §3 set-a 误列，已证）；`applyProjection/applyLimit/listPartitionValues` 两边皆未覆写=惰性、**故意不守卫**；ALTER-DDL 全家（addColumn/…/create-drop Branch|Tag / partition-field / renameTable）+ `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 带句柄但 hive 未覆写、属**写切换**（另步，非本步）。全绿（fe-connector-hive 197 测 + 3 新 `HiveConnectorMetadataSiblingDelegationTest`〔含转发方法集完整性锁〕 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES 且 S4 未做→getTableHandle 仍返 HiveTableHandle→守卫今皆 no-op、休眠）。 - **✅ §4.4 S4 `getTableHandle` iceberg 分流 DONE（`e7c28a39e69`）**（对抗复审 `wf_be91f274-b9b`：divert-correctness/classloader-cce/dormancy-parity/test-intent 4 镜头 + 逐条 verify，**生产码 0 缺陷**，全 HEAD 核对）。`HiveConnectorMetadata.getTableHandle` 在 `HiveTableFormatDetector.detect(tableInfo)==ICEBERG` 时**返 `siblingMetadata(session).getTableHandle(session,db,table)`（原始外来 iceberg 句柄）**、非盖 ICEBERG 的 HiveTableHandle——点亮 S3 全部 `instanceof HiveTableHandle` 守卫（外来句柄开始真流入）。分流放在 fail-loud UNKNOWN 检查**之前**（iceberg 早返，永不触 throw）；detect 内 iceberg(`table_type=ICEBERG`)先于 hudi/hive 判、故 iceberg 优先。**兄弟返 empty 原样透传**（表消失/iceberg 加载不了→不伪造 hive 句柄，兄弟对 iceberg 存在性权威）。**HUDI 不分流**（仍建 HUDI 章 HiveTableHandle、字节不变，hudi 委派属后续子步）；plain-hive/view 不动。网关自身 `hmsClient.tableExists+getTable`（detect 必需）→ 兄弟再自查一次=legacy 双载平价，非缺陷。新 `HiveConnectorMetadataTableHandleDivertTest`(6 测)钉合约：iceberg 返兄弟句柄`assertSame`且非 HiveTableHandle、empty 透传**证经兄弟转发**(getTableHandleCalls==1，复审 Rule-9 补强)、hive/hudi 零咨询兄弟、缺表短路 empty、无兄弟 fail-loud。全绿（fe-connector-hive 203 测〔+6 新〕 + hms 56 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES=休眠成立）。 - **✅ §4.4 S5 扫描提供者网关分流 DONE（`bb39802751b`）**（对抗复审 `wf_6fc6999e-b80`：route-correctness/classloader-tccl/dormancy-parity/test-intent 4 镜头 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 覆写 per-table 接缝 `getScanPlanProvider(ConnectorTableHandle handle)`——`if(handle instanceof HiveTableHandle) return getScanPlanProvider(); return getOrCreateIcebergSibling().getScanPlanProvider(handle);`（负判自有 hive-loader 类型=loader-safe；外来句柄**原样透传兄弟 per-handle overload、永不强转**）。返回的兄弟 provider 是 iceberg-loader 所建→`PluginDrivenScanNode.onPluginClassLoader`(:516)按 `provider.getClass().getClassLoader()` **自动 pin 扫描侧 TCCL 到 iceberg loader、白拿**（无须本步 pin）。`currentHandle` 构造器由已解析表句柄赋值(:172)、pushdown 变异但从不置 null→路由非 hive 句柄不会误分流。HUDI 章 HiveTableHandle 仍是 HiveTableHandle→留 hive 扫描路径（hudi 委派后续）。SPI 默认(`Connector.getScanPlanProvider(handle)`默认委派 no-arg)→本连接器改前对任意句柄都走 no-arg，改后非 hive 句柄才分流；hms 未 live 故零现存路径受影响。全绿（fe-connector-hive 207 测〔+4 新 `HiveConnectorScanProviderDivertTest`〕 + checkstyle 0 + import gate 净）。**测试隔离**：hive/hudi 例 stub no-arg `getScanPlanProvider()`（真 HmsClient 需 HiveConf、单测类路径无→`getScanNodeProperties` 套另测），只钉路由；foreign/fail-loud 例走真 override。 -- **下一步 = §4.4 S6（休眠，序见分解文档，薄）**：按名方法（**无句柄**：`buildTableDescriptor` + 视图族 `viewExists`/`getViewDefinition`/`dropView`）按 `HiveTableFormatDetector.detect(db,table)` 分流转发兄弟（ICEBERG→兄弟、否则 hive）；连接器级能力残留——`getCapabilities` 是连接器级、表达不了 per-handle 门控（如翻闸时 iceberg/hudi-on-HMS 也会收进 `SUPPORTS_COLUMN_AUTO_ANALYZE` 自动分析、legacy 排除 hudi）→**留翻闸记账，勿过度构建**。S6 完成后 §4.4 读脊柱委派收官。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟，见分解文档 §5.1）；写/procedure/EXECUTE 无 per-handle overload = 读脊柱之外独立后续（新 per-handle SPI overload + ALTER-DDL/写校验并入此，见分解文档 §6）。或先补 §4.3 遗留1（`getNewestUpdateVersionOrTime`，须连接器缓存 D2 后）。 +- **✅ §4.4 S6 按名方法/能力残留 = 零代码收官（recon `wf_6ac0e047-768`：4 维读者 + 完整性/正确性 critic 逐条 HEAD 核对；权威结论 = `hms-s6-name-based-divert-findings-2026-07-08.md`，起步必读）。校正分解文档 §3 S6 草案**——所有无句柄**读**方法**均 NO_DIVERT**，S6 无连接器代码改动，只落 findings 文档 + 本残留清单。理由：`buildTableDescriptor` 字节等价（兄弟恒 hms-flavor→同 `HIVE_TABLE`+`THiveTable`；legacy iceberg-on-HMS 亦发 `HIVE_TABLE`；分流只白加一次 `getTable`）；视图族（`viewExists`/`getViewDefinition`/`dropView`）legacy 平价（iceberg 表 `viewExists`=false 留 getTableHandle 分流路径；iceberg-on-HMS 视图存 hive 方言 SQL 于 HMS view text、走 hive 视图路=legacy 平价）**且无句柄不可判别**（`detect` 对视图返 UNKNOWN——iceberg 视图 `table_type='iceberg-view'`≠`'ICEBERG'`）；`listViewNames` 保持不覆写（分流会双列视图=回归）；`getTableComment` 不覆写=SPI 默认 `""`=legacy `getComment()` 恒 `""` 精确平价（分流会显示 legacy 从不显示的 comment=回归）；db/目录级方法（listDatabaseNames/databaseExists/listTableNames/getProperties/create[Database]/getPrimaryKeys/getDatabase）format-agnostic。**⇒ 至此 §4.4 读委派脊柱（S0–S6）全部收官**。 + - ⚠**翻闸残留（findings 文档已详列，休眠期不动）**：①`dropDatabase` force 级联对 iceberg 表裸 `hmsClient.dropTable`（=legacy 平价、无 purge，只泄漏 iceberg 文件）→**写/DDL 委派步**；②`createTable` 引擎路由（新表无可 detect）→写/DDL 步；③`beginTransaction`/`getWritePlanProvider` **无 per-handle overload**→写委派步（新 SPI overload + ALTER-DDL/写校验，见分解文档 §6）；④`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 over-admit hudi（已注于 `HiveConnector.java:139-144`，cost-only、无 per-handle escape）；⑤`supportsLatestSnapshotPreload` under-admit iceberg/hudi（fe-core TableIf 默认 false vs legacy true，latency-only、非 enum-backed）；⑥**first-class iceberg-on-HMS 视图 = 翻闸显式非目标**（legacy 从不经 HMS 目录服务 iceberg ViewCatalog 视图；要做属新特性）。 +- **下一步 = §4.4 写侧委派（读脊柱之外的独立后续）或 §4.3 遗留1**。**写侧委派**（分解文档 §6）：`Connector` 需**新 per-handle SPI overload** `getWritePlanProvider(handle)`/`getProcedureOps(handle)`（现皆连接器级无句柄、`PhysicalPlanTranslator:654/702`+`ConnectorExecuteAction:116` 在拿句柄前解析 provider）→ 网关按 `instanceof HiveTableHandle` 分流兄弟；ALTER-DDL 全家 + `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 写校验 + `dropDatabase` force purge + `createTable` 引擎路由并入此。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟，见分解文档 §5.1）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 diff --git a/plan-doc/tasks/hms-s6-name-based-divert-findings-2026-07-08.md b/plan-doc/tasks/hms-s6-name-based-divert-findings-2026-07-08.md new file mode 100644 index 00000000000000..2a463aacfecd39 --- /dev/null +++ b/plan-doc/tasks/hms-s6-name-based-divert-findings-2026-07-08.md @@ -0,0 +1,40 @@ +# HMS cutover §4.4 S6 — name-based divert + capability residuals: findings (2026-07-08) + +> Code-grounded recon (`wf_6ac0e047-768`: 4 dimension readers — buildTableDescriptor / view-family / other-no-handle / capability-residuals — + a completeness/correctness critic that independently re-verified every call against HEAD). Authoritative scope for §4.4 S6. Trust HEAD over line numbers. + +--- + +## Verdict (headline) + +**S6 requires ZERO new divert code.** Every no-handle *read* method resolves to **NO_DIVERT**; the only per-table divergences are **write/DDL residuals** or the **connector-wide capability residual**, none of which is a "thin name-based read divert". With S0–S5 done, the **§4.4 read-delegation spine is COMPLETE**. S6's deliverable is this document + the HANDOFF residual list — no connector code change. + +This CORRECTS the decomposition's S6 sketch ("re-run `detect(db,table)` and forward view family + `buildTableDescriptor` for ICEBERG"): each of those diverts is proven unnecessary or actively harmful below. + +## Why each no-handle method is NO_DIVERT + +- **`buildTableDescriptor` — NO_DIVERT (byte-identical, a divert is pure waste).** Hive's override is format-agnostic: it unconditionally emits `TTableType.HIVE_TABLE` + `THiveTable(db,table,{})` (`HiveConnectorMetadata.java:382-390`). The sibling is **always** the hms flavor (`IcebergSiblingProperties.synthesize` forces `iceberg.catalog.type=hms`), so iceberg's `buildTableDescriptor` takes its **hms branch** (`IcebergConnectorMetadata.java:677-684`) emitting the **identical** construction — it never reaches the `ICEBERG_TABLE` branch. **Legacy parity confirmed**: `HMSExternalTable.toThrift` emitted `HIVE_TABLE` **unconditionally**, incl. an iceberg-on-HMS table (`dlaType==ICEBERG`). BE does not consult the descriptor table-type for an iceberg-on-HMS scan (JNI scan; iceberg-ness rides the per-file format params). A divert would add a fresh `hmsClient.getTable(detect)` per `toThrift` purely to reproduce identical bytes. **Keep the existing hive override** (it beats the SPI-default `null → SCHEMA_TABLE` fallback); add no fork. +- **View family `viewExists` / `getViewDefinition` / `dropView` — NO_DIVERT (legacy-parity AND undiscriminable name-only).** Two independent reasons: + 1. **Parity.** For the supported iceberg-on-HMS **TABLE** (`table_type=ICEBERG`, no view text), `viewExists`=`isViewTable(getTable)` keys on HMS view-text presence → **false**, keeping it on the table path where `getTableHandle` already diverts (S4). A hive view → true → hive view path. An iceberg-on-HMS **VIEW** stores a **hive-dialect SQL** in HMS `viewOriginalText` (iceberg `HiveViewOperations.newHMSView`), so it is served through the hive view path — byte-parity with legacy `HMSExternalTable.isView`/`getViewText` (which was view-text-only, dlaType-independent). *(This last is a runtime HMS-metadata fact about the bundled `iceberg-hive-metastore` lib, PLAUSIBLE but not verifiable from fe HEAD; it does not change the conclusion either way.)* + 2. **Undiscriminable.** A name-only method has no handle, and `HiveTableFormatDetector.detect` returns **UNKNOWN** for an iceberg view (its `table_type='iceberg-view'` ≠ `'ICEBERG'`, and a view has no data input format). So the "re-run detect and forward" strategy that works for TABLES **cannot classify a view at all**. +- **`listViewNames` — NO_DIVERT (a divert would REGRESS).** Deliberately un-overridden: hive `listTableNames` (HMS `get_all_tables`) already includes every `VIRTUAL_VIEW`, so the SPI-default empty makes fe-core's `addAll` merge a no-op (each view listed once = legacy parity). Overriding to the sibling (`IcebergConnectorMetadata.listViewNames` → `ViewCatalog.listViews`) would **double-list** every view. +- **`getTableComment` — NO_DIVERT (parity; a divert would REGRESS).** Hive does not override it → SPI default `""`. Legacy `HMSExternalTable.getComment()` returned `""` **unconditionally** for all HMS tables incl. iceberg-on-HMS → exact byte-parity. Iceberg's own override returns the real comment, so a divert would surface a comment **legacy never showed**. +- **`getDatabase` / `listDatabaseNames` / `databaseExists` / `listTableNames` / `getProperties` / `supportsCreateDatabase` / `createDatabase` / `getPrimaryKeys` — NO_DIVERT (format-agnostic or inert).** All db/catalog/connector-level; the shared HMS already yields the mixed-catalog union and the sibling wraps the same HMS. Diverting `listTableNames` would **regress** (iceberg's version subtracts views). `getPrimaryKeys` is SPI-default-empty on both connectors (a divert can't change the answer). + +## Residuals for the flip (document, do NOT build in S6) + +1. **`dropDatabase` FORCE cascade — write/DDL residual.** The force branch raw-`hmsClient.dropTable`s each table (no iceberg purge/cleanup). **Legacy parity** (`HiveMetadataOps.dropDbImpl → dropTableImpl(true) → client.dropTable`, no purge) — the HMS entry is removed correctly, it only *leaks* iceberg data/metadata files vs the sibling's `purge=true`. Correcting it needs per-table detect+sibling-forward+location-cleanup → belongs with the **write/DDL delegation substep**, not a thin read divert. +2. **`createTable` engine-routing — write/DDL residual.** Always builds a hive HMS table; a NEW table has nothing to `detect()`. Hive-vs-iceberg CREATE routing is a request/engine decision = write/DDL substep. +3. **`beginTransaction` / `getWritePlanProvider` — write delegation residual.** `HiveConnector.getWritePlanProvider()` has **no per-handle variant** (unlike `getScanPlanProvider(handle)`); routing per-table writes/EXECUTE to an iceberg sibling needs new **per-handle SPI overloads** (see decomposition §6). Separate substep after the read spine. +4. **`SUPPORTS_COLUMN_AUTO_ANALYZE` over-admits hudi-on-HMS — capability residual (already documented at `HiveConnector.java:139-144`).** `getCapabilities` is connector-wide (one EnumSet, no handle) → cannot be per-handle-gated. Legacy `StatisticsUtil.supportAutoAnalyze` admitted HIVE+ICEBERG but **not** HUDI; the connector-wide flag admits hudi too. Cost-only (extra background FULL auto-analyze), not a correctness bug; no per-handle escape. Gate per-handle or explicitly accept at the iceberg/hudi delegation substep. +5. **`supportsLatestSnapshotPreload` under-admits iceberg/hudi-on-HMS — capability-shaped residual (NOT enum-backed).** fe-core `TableIf` default `false`; `PluginDrivenExternalTable` does not override it. Legacy `HMSExternalTable` returned true for HUDI+ICEBERG, false for HIVE → the plugin **under-admits** iceberg/hudi (loses a pre-lock latest-snapshot preload). **Latency-only, no correctness effect** (opt-in via `enable_preload_external_metadata`). Wiring it needs a new capability or per-handle sibling delegation → deferred to the iceberg/hudi delegation substep. +6. **First-class iceberg-on-HMS VIEWS — explicit NON-GOAL at the flip.** Legacy never surfaced iceberg ViewCatalog views through the HMS catalog (only via the separate native `type=iceberg` catalog). Adding them to the mixed catalog is a NEW feature (needs a view-aware discriminator such as `table_type='iceberg-view'` + sibling `viewExists`/`loadView` routing), not an S6 residual. + +## Other capability flags — parity, no residual + +- **`SUPPORTS_VIEW`**: machinery-only; the per-table `isView()` is resolved by `viewExists` (already correct). No divergence. +- **`SUPPORTS_METADATA_PRELOAD`**: legacy returned true unconditionally → exact parity; opt-in, no correctness. +- **`SUPPORTS_MVCC_SNAPSHOT`**: admission-parity (all legacy HMS tables are `MvccTable`); per-table snapshot semantics handled downstream in `beginQuerySnapshot` (per-handle, already diverted). + +## References +- Recon: `wf_6ac0e047-768` (full output `tasks/w4stwou6n.output`). Decomposition: `hms-cutover-sibling-connector-decomposition-2026-07-08.md` (§3 S6 sketch corrected here). Design: `hms-cutover-retype-design-2026-07-07.md`. +- Tracking: apache/doris#65185. From 0e24eb9e0e28a834394149315ff8df2c81d6febf Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 13:18:05 +0800 Subject: [PATCH 075/334] [feat](catalog) HMS gateway metadata delegates iceberg-on-HMS ALTER-DDL + write-validation to sibling (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.4 write-delegation W1 (dormant): extend the S3 handle-guard-forward on HiveConnectorMetadata to the 16 remaining handle-carrying mutating/validation methods so an iceberg-on-HMS table's ALTER-DDL and write-time validation route to the embedded iceberg sibling, while plain-hive stays byte-identical. - 14 ALTER-DDL mutators (renameTable + add/drop/rename/modify/reorder column, create/drop branch & tag, add/drop/replace partition-field): foreign handle -> siblingMetadata().(...); hive handle reproduces the EXACT inherited SPI-default throw ("... not supported"). Foreign handle is never cast. - 2 write validators (validateRowLevelDmlMode / validateStaticPartitionColumns): foreign handle -> sibling (iceberg's real write-mode / static-partition rejections apply); hive handle returns SILENTLY (SPI-default no-op) -- a throw there would newly reject legal plain-hive row-level DML / static-partition INSERTs. Zero new SPI (all 16 already carry a ConnectorTableHandle) and zero fe-core change (ALTER call sites are already handle-based via PluginDrivenExternalCatalog and getTableHandle already S4-diverts iceberg to the sibling). Dormant until "hms" enters SPI_READY_TYPES. Per user sign-off, iceberg-on-HMS gains full native-iceberg write/ALTER capability (an intended upgrade over legacy's read-only reject). Tests: HiveConnectorMetadataSiblingDelegationTest +2 (foreign-handle completeness lock over all 16 + hive-handle throw/no-op + never-consult-sibling). Full module 209 tests green, checkstyle 0, connector-import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 175 +++++++++++++++++- ...onnectorMetadataSiblingDelegationTest.java | 174 +++++++++++++++++ 2 files changed, 346 insertions(+), 3 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index c62c396599562d..b5b13f4ef1fd94 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -29,13 +29,19 @@ import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.ConnectorViewDefinition; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.BranchChange; import org.apache.doris.connector.api.ddl.ConnectorBucketSpec; +import org.apache.doris.connector.api.ddl.ConnectorColumnPosition; import org.apache.doris.connector.api.ddl.ConnectorCreateTableRequest; import org.apache.doris.connector.api.ddl.ConnectorPartitionField; import org.apache.doris.connector.api.ddl.ConnectorPartitionSpec; +import org.apache.doris.connector.api.ddl.DropRefChange; +import org.apache.doris.connector.api.ddl.PartitionFieldChange; +import org.apache.doris.connector.api.ddl.TagChange; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; @@ -1336,9 +1342,172 @@ public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, } } - // RENAME TABLE is intentionally NOT overridden: hive does not support ALTER TABLE RENAME (legacy - // HMSCachedClient has no rename), so the SPI default throw ("RENAME TABLE not supported") preserves the - // pre-flip behavior. + // ========== ConnectorTableOps: ALTER-DDL -- foreign (iceberg) handles divert to the sibling ========== + // + // Every mutating ALTER method already carries a handle. A foreign (iceberg-on-HMS) handle is forwarded to the + // embedded iceberg sibling (which implements the real column / branch / tag / partition-field evolution); the + // foreign handle is NEVER cast. A hive handle reproduces the pre-flip behavior: hive supports none of these + // through this SPI, so its branch throws the exact SPI-default message it inherited before this override. + // Dormant until hms enters SPI_READY_TYPES. + + @Override + public void renameTable(ConnectorSession session, ConnectorTableHandle handle, String newName) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).renameTable(session, handle, newName); + return; + } + // hive does not support ALTER TABLE RENAME (legacy HMSCachedClient has no rename). + throw new DorisConnectorException("RENAME TABLE not supported"); + } + + @Override + public void addColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column, + ConnectorColumnPosition position) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).addColumn(session, handle, column, position); + return; + } + throw new DorisConnectorException("ADD COLUMN not supported"); + } + + @Override + public void addColumns(ConnectorSession session, ConnectorTableHandle handle, List columns) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).addColumns(session, handle, columns); + return; + } + throw new DorisConnectorException("ADD COLUMNS not supported"); + } + + @Override + public void dropColumn(ConnectorSession session, ConnectorTableHandle handle, String columnName) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).dropColumn(session, handle, columnName); + return; + } + throw new DorisConnectorException("DROP COLUMN not supported"); + } + + @Override + public void renameColumn(ConnectorSession session, ConnectorTableHandle handle, String oldName, + String newName) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).renameColumn(session, handle, oldName, newName); + return; + } + throw new DorisConnectorException("RENAME COLUMN not supported"); + } + + @Override + public void modifyColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column, + ConnectorColumnPosition position) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).modifyColumn(session, handle, column, position); + return; + } + throw new DorisConnectorException("MODIFY COLUMN not supported"); + } + + @Override + public void reorderColumns(ConnectorSession session, ConnectorTableHandle handle, List newOrder) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).reorderColumns(session, handle, newOrder); + return; + } + throw new DorisConnectorException("REORDER COLUMNS not supported"); + } + + @Override + public void createOrReplaceBranch(ConnectorSession session, ConnectorTableHandle handle, BranchChange branch) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).createOrReplaceBranch(session, handle, branch); + return; + } + throw new DorisConnectorException("CREATE/REPLACE BRANCH not supported"); + } + + @Override + public void createOrReplaceTag(ConnectorSession session, ConnectorTableHandle handle, TagChange tag) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).createOrReplaceTag(session, handle, tag); + return; + } + throw new DorisConnectorException("CREATE/REPLACE TAG not supported"); + } + + @Override + public void dropBranch(ConnectorSession session, ConnectorTableHandle handle, DropRefChange branch) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).dropBranch(session, handle, branch); + return; + } + throw new DorisConnectorException("DROP BRANCH not supported"); + } + + @Override + public void dropTag(ConnectorSession session, ConnectorTableHandle handle, DropRefChange tag) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).dropTag(session, handle, tag); + return; + } + throw new DorisConnectorException("DROP TAG not supported"); + } + + @Override + public void addPartitionField(ConnectorSession session, ConnectorTableHandle handle, + PartitionFieldChange change) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).addPartitionField(session, handle, change); + return; + } + throw new DorisConnectorException("ADD PARTITION FIELD not supported"); + } + + @Override + public void dropPartitionField(ConnectorSession session, ConnectorTableHandle handle, + PartitionFieldChange change) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).dropPartitionField(session, handle, change); + return; + } + throw new DorisConnectorException("DROP PARTITION FIELD not supported"); + } + + @Override + public void replacePartitionField(ConnectorSession session, ConnectorTableHandle handle, + PartitionFieldChange change) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).replacePartitionField(session, handle, change); + return; + } + throw new DorisConnectorException("REPLACE PARTITION FIELD not supported"); + } + + // ========== ConnectorWriteOps: write validation -- foreign (iceberg) handles divert to the sibling ========== + // + // Both validators carry a handle and run at analysis time. A foreign (iceberg-on-HMS) handle forwards to the + // sibling so iceberg's real write-mode / static-partition rejections apply. A hive handle MUST reproduce the + // permissive SPI default (return silently, NEVER throw) -- a throw here would newly reject legal plain-hive + // row-level DML / static-partition INSERTs. Dormant until hms enters SPI_READY_TYPES. + + @Override + public void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHandle handle, WriteOperation op) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).validateRowLevelDmlMode(session, handle, op); + return; + } + // hive: no per-table row-level DML mode constraint (SPI default no-op). + } + + @Override + public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTableHandle handle, + List staticPartitionColumnNames) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).validateStaticPartitionColumns(session, handle, staticPartitionColumnNames); + return; + } + // hive: no static-partition constraint (SPI default no-op). + } // ========== ConnectorWriteOps: transactions ========== diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 4a9227c5ad4ecb..2888ce0e944aab 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorColumnStatistics; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; @@ -25,8 +26,14 @@ import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.ddl.BranchChange; +import org.apache.doris.connector.api.ddl.ConnectorColumnPosition; +import org.apache.doris.connector.api.ddl.DropRefChange; +import org.apache.doris.connector.api.ddl.PartitionFieldChange; +import org.apache.doris.connector.api.ddl.TagChange; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; @@ -37,6 +44,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.function.Executable; import java.util.ArrayList; import java.util.Arrays; @@ -169,6 +177,73 @@ public void foreignHandleFailsLoudWhenNoSiblingConfigured() { "a foreign handle with no sibling configured must fail loud"); } + @Test + public void everyAlterDdlAndValidateMethodForwardsAForeignHandleToTheSibling() { + HiveConnectorMetadata md = withSibling(); + + // The 14 ALTER-DDL mutators + 2 write validators: a foreign (iceberg-on-HMS) handle must divert, never + // run the hive branch and never be cast. Change objects are null — the guard fires on the handle type + // before any param is touched. + md.renameTable(null, foreignHandle, "new"); + md.addColumn(null, foreignHandle, null, null); + md.addColumns(null, foreignHandle, Collections.emptyList()); + md.dropColumn(null, foreignHandle, "c"); + md.renameColumn(null, foreignHandle, "a", "b"); + md.modifyColumn(null, foreignHandle, null, null); + md.reorderColumns(null, foreignHandle, Collections.emptyList()); + md.createOrReplaceBranch(null, foreignHandle, null); + md.createOrReplaceTag(null, foreignHandle, null); + md.dropBranch(null, foreignHandle, null); + md.dropTag(null, foreignHandle, null); + md.addPartitionField(null, foreignHandle, null); + md.dropPartitionField(null, foreignHandle, null); + md.replacePartitionField(null, foreignHandle, null); + md.validateRowLevelDmlMode(null, foreignHandle, null); + md.validateStaticPartitionColumns(null, foreignHandle, Collections.emptyList()); + + Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_WRITE_METHODS, siblingMetadata.calls, + "every ALTER-DDL mutator + write validator must forward a foreign handle to the sibling"); + } + + @Test + public void hiveHandleAlterDdlThrowsAndValidateIsNoopAndNeverConsultsSibling() { + HiveConnectorMetadata md = withSibling(); + HiveTableHandle hive = hiveHandle(); + + // Group-1: ALTER-DDL for a hive handle throws the EXACT inherited SPI-default message (byte-parity with + // pre-override behavior) without building or consulting the sibling. + assertThrowsMessage(() -> md.renameTable(null, hive, "n"), "RENAME TABLE not supported"); + assertThrowsMessage(() -> md.addColumn(null, hive, null, null), "ADD COLUMN not supported"); + assertThrowsMessage(() -> md.addColumns(null, hive, Collections.emptyList()), "ADD COLUMNS not supported"); + assertThrowsMessage(() -> md.dropColumn(null, hive, "c"), "DROP COLUMN not supported"); + assertThrowsMessage(() -> md.renameColumn(null, hive, "a", "b"), "RENAME COLUMN not supported"); + assertThrowsMessage(() -> md.modifyColumn(null, hive, null, null), "MODIFY COLUMN not supported"); + assertThrowsMessage(() -> md.reorderColumns(null, hive, Collections.emptyList()), + "REORDER COLUMNS not supported"); + assertThrowsMessage(() -> md.createOrReplaceBranch(null, hive, null), "CREATE/REPLACE BRANCH not supported"); + assertThrowsMessage(() -> md.createOrReplaceTag(null, hive, null), "CREATE/REPLACE TAG not supported"); + assertThrowsMessage(() -> md.dropBranch(null, hive, null), "DROP BRANCH not supported"); + assertThrowsMessage(() -> md.dropTag(null, hive, null), "DROP TAG not supported"); + assertThrowsMessage(() -> md.addPartitionField(null, hive, null), "ADD PARTITION FIELD not supported"); + assertThrowsMessage(() -> md.dropPartitionField(null, hive, null), "DROP PARTITION FIELD not supported"); + assertThrowsMessage(() -> md.replacePartitionField(null, hive, null), "REPLACE PARTITION FIELD not supported"); + + // Group-2: validate* for a hive handle MUST return silently — a throw here would newly reject legal + // plain-hive row-level DML / static-partition INSERTs. + md.validateRowLevelDmlMode(null, hive, null); + md.validateStaticPartitionColumns(null, hive, Collections.emptyList()); + + Assertions.assertEquals(0, siblingConnector.getMetadataCount, + "a hive handle must never build/consult the iceberg sibling for ALTER-DDL / validate"); + Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle"); + } + + private static void assertThrowsMessage(Executable exec, String expectedMessage) { + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, exec); + Assertions.assertEquals(expectedMessage, e.getMessage(), + "the hive branch must reproduce the exact inherited SPI-default message"); + } + /** A sibling {@link Connector} whose getMetadata hands back the recording metadata and counts the calls. */ private static final class RecordingSiblingConnector implements Connector { private final ConnectorMetadata metadata; @@ -201,6 +276,14 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata "applySnapshot", "applyRewriteFileScope", "applyTopnLazyMaterialization", "listSupportedSysTables", "getSysTableHandle")); + // The exact set + order of ALTER-DDL / validate methods the foreign-handle write test drives (Rule-9 + // completeness lock for §4.4 W1: dropping a guard, or adding one that should not forward, fails the test). + static final List EXPECTED_WRITE_METHODS = Collections.unmodifiableList(Arrays.asList( + "renameTable", "addColumn", "addColumns", "dropColumn", "renameColumn", "modifyColumn", + "reorderColumns", "createOrReplaceBranch", "createOrReplaceTag", "dropBranch", "dropTag", + "addPartitionField", "dropPartitionField", "replacePartitionField", + "validateRowLevelDmlMode", "validateStaticPartitionColumns")); + final List calls = new ArrayList<>(); final Optional> filterResult = Optional.of(new FilterApplicationResult<>(SIBLING_HANDLE, null, false)); @@ -343,5 +426,96 @@ public Optional getSysTableHandle(ConnectorSession session calls.add("getSysTableHandle"); return Optional.of(SIBLING_HANDLE); } + + // ---- §4.4 W1: ALTER-DDL mutators + write validators (the write-delegation surface) ---- + + @Override + public void renameTable(ConnectorSession session, ConnectorTableHandle handle, String newName) { + calls.add("renameTable"); + } + + @Override + public void addColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column, + ConnectorColumnPosition position) { + calls.add("addColumn"); + } + + @Override + public void addColumns(ConnectorSession session, ConnectorTableHandle handle, List columns) { + calls.add("addColumns"); + } + + @Override + public void dropColumn(ConnectorSession session, ConnectorTableHandle handle, String columnName) { + calls.add("dropColumn"); + } + + @Override + public void renameColumn(ConnectorSession session, ConnectorTableHandle handle, String oldName, + String newName) { + calls.add("renameColumn"); + } + + @Override + public void modifyColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column, + ConnectorColumnPosition position) { + calls.add("modifyColumn"); + } + + @Override + public void reorderColumns(ConnectorSession session, ConnectorTableHandle handle, List newOrder) { + calls.add("reorderColumns"); + } + + @Override + public void createOrReplaceBranch(ConnectorSession session, ConnectorTableHandle handle, + BranchChange branch) { + calls.add("createOrReplaceBranch"); + } + + @Override + public void createOrReplaceTag(ConnectorSession session, ConnectorTableHandle handle, TagChange tag) { + calls.add("createOrReplaceTag"); + } + + @Override + public void dropBranch(ConnectorSession session, ConnectorTableHandle handle, DropRefChange branch) { + calls.add("dropBranch"); + } + + @Override + public void dropTag(ConnectorSession session, ConnectorTableHandle handle, DropRefChange tag) { + calls.add("dropTag"); + } + + @Override + public void addPartitionField(ConnectorSession session, ConnectorTableHandle handle, + PartitionFieldChange change) { + calls.add("addPartitionField"); + } + + @Override + public void dropPartitionField(ConnectorSession session, ConnectorTableHandle handle, + PartitionFieldChange change) { + calls.add("dropPartitionField"); + } + + @Override + public void replacePartitionField(ConnectorSession session, ConnectorTableHandle handle, + PartitionFieldChange change) { + calls.add("replacePartitionField"); + } + + @Override + public void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHandle handle, + WriteOperation op) { + calls.add("validateRowLevelDmlMode"); + } + + @Override + public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTableHandle handle, + List staticPartitionColumnNames) { + calls.add("validateStaticPartitionColumns"); + } } } From 0decdf0a5e3ea590f08780572811e9ff30389dd0 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 13:22:11 +0800 Subject: [PATCH 076/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20write-delegation=20W1=20DONE;=20decomposition=20doc=20W1-?= =?UTF-8?q?W6=20+=20capability=20upgrade=20signed;=20next=20=3D=20W2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add hms-write-delegation-decomposition-2026-07-08.md: authoritative W1-W6 plan from recon wf_f508ac0e-8ec (3 divert shapes; user-signed iceberg-on-HMS capability UPGRADE to full native-iceberg; e2e requirement; flip residuals incl. hudi-on-HMS write flip-blocker). - HANDOFF: W1 landed (e7a96f439e7); next = W2 (getWritePlanProvider(handle) overload + gateway divert + fe-core provider-fetch conversion). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 6 ++- ...ite-delegation-decomposition-2026-07-08.md | 47 +++++++++++++++++++ 2 files changed, 51 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/hms-write-delegation-decomposition-2026-07-08.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index f832eae10be3ba..d3b72836f996c7 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**；hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -66,7 +66,9 @@ - **✅ §4.4 S5 扫描提供者网关分流 DONE（`bb39802751b`）**（对抗复审 `wf_6fc6999e-b80`：route-correctness/classloader-tccl/dormancy-parity/test-intent 4 镜头 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 覆写 per-table 接缝 `getScanPlanProvider(ConnectorTableHandle handle)`——`if(handle instanceof HiveTableHandle) return getScanPlanProvider(); return getOrCreateIcebergSibling().getScanPlanProvider(handle);`（负判自有 hive-loader 类型=loader-safe；外来句柄**原样透传兄弟 per-handle overload、永不强转**）。返回的兄弟 provider 是 iceberg-loader 所建→`PluginDrivenScanNode.onPluginClassLoader`(:516)按 `provider.getClass().getClassLoader()` **自动 pin 扫描侧 TCCL 到 iceberg loader、白拿**（无须本步 pin）。`currentHandle` 构造器由已解析表句柄赋值(:172)、pushdown 变异但从不置 null→路由非 hive 句柄不会误分流。HUDI 章 HiveTableHandle 仍是 HiveTableHandle→留 hive 扫描路径（hudi 委派后续）。SPI 默认(`Connector.getScanPlanProvider(handle)`默认委派 no-arg)→本连接器改前对任意句柄都走 no-arg，改后非 hive 句柄才分流；hms 未 live 故零现存路径受影响。全绿（fe-connector-hive 207 测〔+4 新 `HiveConnectorScanProviderDivertTest`〕 + checkstyle 0 + import gate 净）。**测试隔离**：hive/hudi 例 stub no-arg `getScanPlanProvider()`（真 HmsClient 需 HiveConf、单测类路径无→`getScanNodeProperties` 套另测），只钉路由；foreign/fail-loud 例走真 override。 - **✅ §4.4 S6 按名方法/能力残留 = 零代码收官（recon `wf_6ac0e047-768`：4 维读者 + 完整性/正确性 critic 逐条 HEAD 核对；权威结论 = `hms-s6-name-based-divert-findings-2026-07-08.md`，起步必读）。校正分解文档 §3 S6 草案**——所有无句柄**读**方法**均 NO_DIVERT**，S6 无连接器代码改动，只落 findings 文档 + 本残留清单。理由：`buildTableDescriptor` 字节等价（兄弟恒 hms-flavor→同 `HIVE_TABLE`+`THiveTable`；legacy iceberg-on-HMS 亦发 `HIVE_TABLE`；分流只白加一次 `getTable`）；视图族（`viewExists`/`getViewDefinition`/`dropView`）legacy 平价（iceberg 表 `viewExists`=false 留 getTableHandle 分流路径；iceberg-on-HMS 视图存 hive 方言 SQL 于 HMS view text、走 hive 视图路=legacy 平价）**且无句柄不可判别**（`detect` 对视图返 UNKNOWN——iceberg 视图 `table_type='iceberg-view'`≠`'ICEBERG'`）；`listViewNames` 保持不覆写（分流会双列视图=回归）；`getTableComment` 不覆写=SPI 默认 `""`=legacy `getComment()` 恒 `""` 精确平价（分流会显示 legacy 从不显示的 comment=回归）；db/目录级方法（listDatabaseNames/databaseExists/listTableNames/getProperties/create[Database]/getPrimaryKeys/getDatabase）format-agnostic。**⇒ 至此 §4.4 读委派脊柱（S0–S6）全部收官**。 - ⚠**翻闸残留（findings 文档已详列，休眠期不动）**：①`dropDatabase` force 级联对 iceberg 表裸 `hmsClient.dropTable`（=legacy 平价、无 purge，只泄漏 iceberg 文件）→**写/DDL 委派步**；②`createTable` 引擎路由（新表无可 detect）→写/DDL 步；③`beginTransaction`/`getWritePlanProvider` **无 per-handle overload**→写委派步（新 SPI overload + ALTER-DDL/写校验，见分解文档 §6）；④`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 over-admit hudi（已注于 `HiveConnector.java:139-144`，cost-only、无 per-handle escape）；⑤`supportsLatestSnapshotPreload` under-admit iceberg/hudi（fe-core TableIf 默认 false vs legacy true，latency-only、非 enum-backed）；⑥**first-class iceberg-on-HMS 视图 = 翻闸显式非目标**（legacy 从不经 HMS 目录服务 iceberg ViewCatalog 视图；要做属新特性）。 -- **下一步 = §4.4 写侧委派（读脊柱之外的独立后续）或 §4.3 遗留1**。**写侧委派**（分解文档 §6）：`Connector` 需**新 per-handle SPI overload** `getWritePlanProvider(handle)`/`getProcedureOps(handle)`（现皆连接器级无句柄、`PhysicalPlanTranslator:654/702`+`ConnectorExecuteAction:116` 在拿句柄前解析 provider）→ 网关按 `instanceof HiveTableHandle` 分流兄弟；ALTER-DDL 全家 + `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 写校验 + `dropDatabase` force purge + `createTable` 引擎路由并入此。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟，见分解文档 §5.1）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 +- **§4.4 写委派权威分解 = `hms-write-delegation-decomposition-2026-07-08.md`（起步必读，recon `wf_f508ac0e-8ec` 7 维 + 完整性/分解 critic）。拆成 W1–W6 六个休眠小步，三种委派形态**：Shape A=写/存储过程 getter 加 per-handle 重载（W2/W3/W5）；Shape B=ConnectorMetadata 句柄守卫转发（W1，零新 SPI，**已 DONE**）；Shape C=事务按句柄选择（W4，唯一无先例的新形态）。**签字决策（用户 2026-07-08）：iceberg-on-HMS 升级为完整 native-iceberg 写/DELETE/MERGE/EXECUTE/ALTER 能力**（旧行为全拒=有意升级，非字节平价；对齐 Trino 按句柄委派）。**e2e 硬要求（用户 2026-07-08，memory `hms-iceberg-delegation-needs-e2e`）：每项新增 iceberg 能力（写/ALTER/procedure）最终都须补 e2e 回归（休眠单测只证路由、证不了跨加载器 cast/commit/worker-pool）；实现完成后择机统一补。** +- **下一步 = W2**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例）→ `HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（唯一真覆写）；fe-core 4 处 provider 取用点把已在 1 行后解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator:654/702`、`PhysicalIcebergMergeSink:302`、`PluginDrivenExternalTable:525`；connector-agnostic 合铁律）。**W3**（7 个写能力判定 per-handle 重载，从 W2 自动派生 + fe-core 11 处 analysis-time 门用 `PluginDrivenExternalTable.resolveConnectorTableHandle` 取句柄）依赖 W2；**W4** 事务按句柄（`beginTransaction(session,handle)` 守卫转发，否则 `IcebergWritePlanProvider.currentTransaction` 对 `HiveConnectorTransaction` CCE）依赖 W2；**W5** 存储过程（`getProcedureOps(handle)` + `ConnectorExecuteAction` 解析顺序上提）独立；**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测）。W2+W3+W4+W6=写执行捆绑须全在翻闸前落，W1/W5 可独立提交。 +- **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 diff --git a/plan-doc/tasks/hms-write-delegation-decomposition-2026-07-08.md b/plan-doc/tasks/hms-write-delegation-decomposition-2026-07-08.md new file mode 100644 index 00000000000000..068da7fe63d925 --- /dev/null +++ b/plan-doc/tasks/hms-write-delegation-decomposition-2026-07-08.md @@ -0,0 +1,47 @@ +# HMS cutover §4.4 — write / procedure / DDL delegation: decomposition (2026-07-08) + +> Produced by a code-grounded recon (`wf_f508ac0e-8ec`: 7 dimension readers + completeness critic + decomposition critic, all HEAD-verified) plus lead-engineer independent read of the load-bearing interfaces. Authoritative plan for the remaining §4.4 write-side dormant build-out (the read-delegation spine S0–S6 is DONE). Trust HEAD over line numbers. + +--- + +## 0. Scope + the signed capability decision + +A flipped `hms` catalog is served by the hive plugin as a GATEWAY; iceberg-on-HMS tables delegate to a sibling `IcebergConnector`. The read spine (S0–S6) forwards all reads. This decomposition finishes the **write / ALTER TABLE EXECUTE / ALTER-DDL** side. + +**DECISION — LOCKED (user sign-off 2026-07-08): UPGRADE to full native-iceberg capability.** Post-flip, an iceberg-on-HMS table gains the SAME write / DELETE / MERGE / OVERWRITE / EXECUTE / ALTER-DDL capability as a standalone `type=iceberg` catalog (because the sibling IS an `IcebergConnector`). Legacy rejected all three (writes at `BindSink` "target is not a Hive table"; EXECUTE because `HMSExternalTable` is not a `PluginDrivenExternalTable`; ALTER through hive `HiveMetadataOps`). So the write delegation is a deliberate, documented **upgrade over legacy**, not byte-parity — aligned with Trino (the handle identifies the connector; write/DDL dispatch per-handle) and with Doris's already-shipped `getScanPlanProvider(handle)`. + +**E2E REQUIREMENT (user, 2026-07-08 — memory `hms-iceberg-delegation-needs-e2e`):** every new iceberg-on-HMS capability (write / ALTER / procedure) MUST get an e2e regression test before it is considered done. Dormant unit tests prove ROUTING only, not cross-loader cast / commit / worker-pool behavior. Add them "择机" (at an opportune point) once the write substeps land — one heterogeneous HMS catalog (plain-hive + iceberg-on-HMS), run INSERT/DELETE/MERGE/OVERWRITE + ALTER ADD/RENAME/branch-tag/partition-field + ALTER TABLE EXECUTE, assert same result as a native `type=iceberg` catalog on the same table. + +## 1. The three divert SHAPES (recon-delimited) + +The write/procedure/DDL surface splits into **three structurally distinct** divert mechanisms: + +- **Shape A — per-handle SPI overload on `Connector`** for the write/procedure GETTERS that are connector-level today (no handle): `getWritePlanProvider()` (`Connector.java:74`), the 7 admission views (`supportedWriteOperations`/`supportsWriteBranch`/`requires*`, `:83-125`), `getProcedureOps()` (`:131`). Mirror `getScanPlanProvider(handle)` (`:66-68`): add a `(ConnectorTableHandle)` overload whose default delegates to the no-arg version; `HiveConnector` overrides the ONE real spine (`getWritePlanProvider(handle)` / `getProcedureOps(handle)`) with `instanceof HiveTableHandle` guard-forward. Needs fe-core call-site edits to pass the handle (some sites resolve it one line later; some run at analysis time before a handle exists). +- **Shape B — `ConnectorMetadata` handle-guard-forward** (S3 family, ZERO new SPI) for the mutators that ALREADY carry a handle: 14 ALTER-DDL + 2 write-validators + (`dropTable`/`truncateTable` already done). **✅ W1 landed this.** +- **Shape C — session-bound transaction selection** (`beginTransaction`, no handle today, mints a `HiveConnectorTransaction` unconditionally at `HiveConnectorMetadata:1353`). `IcebergWritePlanProvider.currentTransaction` casts `session.getCurrentTransaction()` to `IcebergConnectorTransaction` → **CCE for an iceberg-on-HMS write**. Neither the scan-seam nor the metadata guard-forward precedent solves this; the design must thread the target handle to the txn-open site (W4). + +## 2. Ordered dormant sub-steps + +- **✅ W1 — ALTER-DDL + write-validation guard-forwards (Shape B, ZERO new SPI). DONE (`e7a96f439e7`).** 14 ALTER-DDL mutators forward foreign handles to the sibling / reproduce the exact SPI-default throw for hive; 2 validators forward / return SILENTLY for hive (a throw would newly reject legal plain-hive DML — the one real trap). No fe-core change (ALTER call sites already handle-based via `PluginDrivenExternalCatalog.resolveAlterHandle`; `getTableHandle` already S4-diverts iceberg). Tests: sibling-delegation +2 (foreign completeness lock over all 16 + hive throw/no-op + never-consult-sibling). 209 module tests green, checkstyle 0, import gate clean. +- **W2 — `getWritePlanProvider(ConnectorTableHandle)` overload + `HiveConnector` divert + fe-core provider-fetch conversion (Shape A spine).** Add `default getWritePlanProvider(handle){ return getWritePlanProvider(); }` on `Connector`; `HiveConnector` overrides = `handle instanceof HiveTableHandle ? getWritePlanProvider() : getOrCreateIcebergSibling().getWritePlanProvider(handle)`. fe-core: 4 sites where the handle is resolved ~1 line later — `PhysicalPlanTranslator:654`(DELETE/MERGE, handle :655) + `:702`(INSERT, handle :703) hoist getTableHandle above the provider fetch; `PhysicalIcebergMergeSink:302`(handle :308); `PluginDrivenExternalTable.fetchSyntheticWriteColumns:525`(handle :531). Connector-agnostic (a per-handle overload call, IRON-RULE OK). Deps: none (W1 optional-adjacent). Risk LOW routing; TCCL deferred to W6. +- **W3 — 7 write-admission per-handle overloads (default-derived) + fe-core admission conversion (Shape A leaves).** Add `supportedWriteOperations(handle)`/`supportsWriteBranch(handle)`/`requires*(handle)`, each null-safe over `getWritePlanProvider(handle)` — so once W2 diverts the provider they auto-return iceberg's answers with NO extra `HiveConnector` override (minimal-surface). fe-core: (a) `PhysicalPlanTranslator:648/697` hoist getTableHandle; (b) the handle-LESS analysis-time gates — `RowLevelDmlRowIdUtils:133`, `IcebergRowLevelDmlTransform:93` (the load-bearing DELETE/MERGE detectors), `InsertOverwriteTableCommand:336/351`, `InsertIntoTableCommand:794`, and the `PluginDrivenExternalTable` requires* wrappers (`115/233/249/263/278` + `BindSink:800/811`) — resolve a handle via `PluginDrivenExternalTable.resolveConnectorTableHandle` (`:100`, verified present). **MEDIUM risk:** `supportedWriteOperations(handle)` is load-bearing — without it iceberg-on-HMS DELETE/MERGE is silently not recognized as row-level DML; `requiresMaterializeStaticPartitionValues` (hive false vs iceberg true) and `requiresPartitionHashWrite` (hive true vs iceberg false) genuinely DIFFER, so a partial conversion mis-plans. Deps: W2. +- **W4 — `beginTransaction(session, handle)` threading (Shape C, the one non-precedented gap).** Add a per-handle `beginTransaction` on `ConnectorWriteOps`; `HiveConnectorMetadata` guard-forwards to `siblingMetadata` for a foreign handle so the session-bound txn is the sibling's `IcebergConnectorTransaction` (else `IcebergWritePlanProvider.currentTransaction` CCEs). Thread the resolved write-target handle into the engine's txn-open call. **DECISION (recommended, needs confirm at W4): option-1 per-handle `beginTransaction(session, handle)` guard-forwarded** (direct S3-family member, matches Trino's handle-bound `beginInsert`). Deps: W2. MEDIUM risk (only genuinely new shape). +- **W5 — `getProcedureOps(ConnectorTableHandle)` overload + `HiveConnector` divert + `ConnectorExecuteAction` reorder.** Add `default getProcedureOps(handle){ return getProcedureOps(); }` (null default); `HiveConnector`: hive handle → inherited null (plain-hive has no procedures, correct), foreign → sibling. fe-core `ConnectorExecuteAction`: hoist the handle-resolution block (`:134-139`) ABOVE the `getProcedureOps()` null-check (`:116`) + `getExecutionMode` (`:123`). Net-new capability (per §0 decision). Reorder changes failure-message ORDER (bad table name now throws "Table not found" before "does not support EXECUTE") — verify no test pins precedence. `ExecuteActionFactory.getSupportedActions` (`:84`, SHOW discovery) has NO live caller → its handle-aware variant deferred. Deps: none (independent of write set). MEDIUM risk (net-new). +- **W6 — write-path TCCL pin verification (flip-time, NOT dormant-testable).** Scan auto-pins via `PluginDrivenScanNode.onPluginClassLoader`; the write path has NO analogue — the sibling `IcebergWritePlanProvider`'s name-based reflection (planWrite `PhysicalPlanTranslator:725`, getWritePartitioning `PhysicalIcebergMergeSink:313`, getSyntheticWriteColumns `PluginDrivenExternalTable:535`) runs on fe-core threads under the app loader. **VERIFY first** whether the sibling-internal `TcclPinningConnectorContext.executeAuthenticated` wraps `planWrite`'s `TDataSink` construction (not just the deep commit); if unpinned, add an fe-core write-path pin keyed off `provider.getClass().getClassLoader()` (connector-agnostic). Deps: W2. The load-bearing unknown of the write step — flip-time e2e only. + +**Ordering rationale.** W1 first (cheapest, no SPI, no fe-core). W2 is the spine every admission derives from. W3 depends on W2 (auto-derive). W4 after the planning-only diverts. W5 independent (schedule after write core, net-new decision). W6 last (flip-time risk). **W2+W3+W4+W6 are the "write execution" bundle — all must land before the flip; W1 and W5 are independently committable.** Every substep is dormant (nothing runs until `hms` enters `SPI_READY_TYPES`). + +## 3. Residuals for the flip (document, do NOT build in the write substeps) + +1. **THE flip line:** add `"hms"` to `CatalogFactory.SPI_READY_TYPES` (`:50`) — the single residual that ends dormancy for the whole write/procedure/DDL spine. +2. **HUDI-on-HMS write parity (flip-blocker for hudi).** `getTableHandle` diverts ONLY ICEBERG → a hudi table keeps a `HiveTableHandle` and (post-W2) routes to the HIVE write provider; `HiveConnectorTransaction` likely does NOT reject non-transactional hudi. The iceberg write step must NOT silently make read-only hudi-on-HMS writable-as-hive — add an explicit hudi-write reject in the hive write path, or close it in the dedicated hudi-delegation substep, before the flip. +3. **Hive ALTER-DDL / RENAME port.** W1's hive branch throws "... not supported" (current connector behavior); if legacy supported any hive ALTER through `HiveMetadataOps`, that would regress at the flip unless the hive connector ports it. Separate hive-DDL concern, not iceberg delegation. +4. **`createTable(format=iceberg)` engine-routing.** A NEW table has no handle to `detect()`; hive-vs-iceberg CREATE routing is a request/engine decision + `pluginCatalogTypeToEngine` has no `"hms"` entry (`CreateTableInfo`). Write/DDL residual. +5. **`dropDatabase` FORCE cascade.** Raw `hmsClient.dropTable` per table, no iceberg purge (legacy parity — only leaks iceberg files). Per-table detect+sibling-forward+cleanup belongs with the write/DDL substep. +6. **§4.5 write-chain items:** read-ACID query-finish commit unwired; `BindSink` explicit-partition-spec INSERT reject not yet ported into the hive plugin (only the LZO reject is, `HiveSinkHelper:85`); `Env.hiveTransactionMgr` relocation; `BIND_BROKER_NAME` relocation. +7. **E2E gate (per §0):** the ONLY place write/procedure/DDL correctness is actually exercised — every substep is dormant + unit tests prove routing only. + +## 4. References +- Recon: `wf_f508ac0e-8ec` (journal in the workflow transcript dir; R6 dropDatabase/createTable reader hit the structured-output retry cap — those items grounded manually + covered by residuals 4/5). Decomposition supersedes design §4.4 wrapper-handle (already corrected in `hms-cutover-sibling-connector-decomposition-2026-07-08.md`). +- Design: `hms-cutover-retype-design-2026-07-07.md` (§4.4/§4.5). Read-spine: `hms-cutover-sibling-connector-decomposition-2026-07-08.md`, `hms-s6-name-based-divert-findings-2026-07-08.md`. +- Tracking: apache/doris#65185. From ce897c2672cd28b16416e96fd35b72acdf1672a7 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 14:06:34 +0800 Subject: [PATCH 077/334] [feat](catalog) HMS gateway selects write plan provider per-handle, diverts iceberg-on-HMS to sibling (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.4 write-delegation W2 (dormant): add a per-handle write-provider seam mirroring the shipped getScanPlanProvider(handle). - Connector: new default getWritePlanProvider(ConnectorTableHandle) delegating to the connector-level getWritePlanProvider() -> behavior-preserving for every single-format connector (jdbc/es/trino/max_compute/paimon/iceberg). - HiveConnector overrides it: a HiveTableHandle uses the hive write provider; a foreign (iceberg-on-HMS) handle is delegated to getOrCreateIcebergSibling() .getWritePlanProvider(handle), passed through UNMODIFIED and NEVER cast. HUDI keeps a HiveTableHandle -> stays on hive (its write delegation is a later step). - fe-core: the 4 provider-fetch sites now pass the resolved table handle (PhysicalPlanTranslator DELETE/MERGE + INSERT, PhysicalIcebergMergeSink, PluginDrivenExternalTable.fetchSyntheticWriteColumns) by hoisting the already-adjacent getTableHandle above the provider fetch. Connector-agnostic (a per-handle overload call), no if(format)/instanceof-HMSExternal branch. The write-admission methods (supportedWriteOperations/requires*) stay connector-level (W3); beginTransaction threading is W4; the write-path TCCL pin is W6. Dormant until "hms" enters SPI_READY_TYPES. Tests: new HiveConnectorWriteProvider DivertTest (4: foreign->sibling unmodified + build-once, hive/hudi->hive no sibling, fail-loud when plugin absent); 4 fe-core mock tests stub the per-handle overload (a plain Mockito mock does not run the interface default). fe-connector-hive 213 + fe-core touched suites green, checkstyle 0 (api/hive/fe-core), import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/api/Connector.java | 13 ++ .../doris/connector/hive/HiveConnector.java | 17 ++ .../HiveConnectorWriteProviderDivertTest.java | 185 ++++++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 14 +- .../translator/PhysicalPlanTranslator.java | 10 +- .../physical/PhysicalIcebergMergeSink.java | 11 +- .../PluginDrivenExternalTableTest.java | 4 + ...ysicalPlanTranslatorAdmissionGateTest.java | 2 + ...lPlanTranslatorIcebergRowLevelDmlTest.java | 2 + .../PhysicalIcebergMergeSinkTest.java | 2 + 10 files changed, 250 insertions(+), 10 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index 4ecddcd8802309..c5c39136b58339 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -75,6 +75,19 @@ default ConnectorWritePlanProvider getWritePlanProvider() { return null; } + /** + * Returns the write plan provider for the given table, allowing one connector to select a different + * provider per table — the write-side analogue of {@link #getScanPlanProvider(ConnectorTableHandle)}. + * + *

    The default ignores {@code handle} and returns the connector-level {@link #getWritePlanProvider()}, so + * every single-format connector is unaffected. A heterogeneous gateway connector (one catalog serving + * multiple table formats) overrides this to delegate to a per-format sub-provider by the concrete + * (connector-defined) handle type; the engine never inspects the format.

    + */ + default ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { + return getWritePlanProvider(); + } + /** * The write operations the engine may perform on this connector — the single admission source. Reads the * write provider's {@link ConnectorWritePlanProvider#supportedOperations()}; no provider ⇒ empty set ⇒ all diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index fb900a5e2f6640..7d7dcb0f76f9c9 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -128,6 +128,23 @@ public ConnectorWritePlanProvider getWritePlanProvider() { return new HiveWritePlanProvider(getOrCreateClient(), properties, context); } + /** + * Per-table write provider: a hive handle uses the hive write provider; a foreign (iceberg-on-HMS) handle is + * delegated to the sibling's per-handle write provider (built in the iceberg plugin's classloader). The + * foreign handle is passed through UNMODIFIED and NEVER cast (its concrete iceberg type is invisible across + * the loader split — a cast would CCE). A HUDI table keeps a HiveTableHandle, so it stays on the hive write + * path (its delegation is a later substep). Mirrors {@link #getScanPlanProvider(ConnectorTableHandle)}; + * dormant until hms enters SPI_READY_TYPES. The returned sibling provider runs its planWrite on fe-core + * threads — the write-path TCCL pin is a separate flip-time concern (write-delegation W6). + */ + @Override + public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { + if (handle instanceof HiveTableHandle) { + return getWritePlanProvider(); + } + return getOrCreateIcebergSibling().getWritePlanProvider(handle); + } + @Override public Set getCapabilities() { // Connector-wide capabilities for the flipped hms catalog, each a faithful port of a legacy diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java new file mode 100644 index 00000000000000..de6745e23dd6bd --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java @@ -0,0 +1,185 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Pins the HMS-cutover §4.4 write-delegation W2 seam: {@link HiveConnector#getWritePlanProvider( + * ConnectorTableHandle)} routes by the concrete handle type — a hive handle runs the hive write provider, any + * foreign (iceberg-on-HMS) handle is delegated to the embedded iceberg sibling's per-handle write provider — and + * NEVER casts the foreign handle. The write-side twin of {@link HiveConnectorScanProviderDivertTest}. + * + *

    WHY (Rule 9): once getTableHandle returns a foreign iceberg handle (S4), a write to that table must pick the + * SIBLING's write provider so an iceberg-on-HMS INSERT/DELETE/MERGE builds an iceberg sink, not a hive one. A hive + * (or hudi-stamped hive) handle must NOT be diverted — a hive-only deployment has no iceberg plugin, and hudi's + * write delegation is a later substep. The foreign handle is passed through UNMODIFIED (a rewrap would poison the + * downstream iceberg cast).

    + * + *

    Dormant until hms enters {@code SPI_READY_TYPES}: no production path selects a write provider for this + * connector yet, so these assertions are a guard, not a live-path test.

    + */ +public class HiveConnectorWriteProviderDivertTest { + + private static final String METASTORE_URI = "thrift://host:9083"; + + /** The foreign (non-hive) handle the iceberg sibling's getTableHandle produces post-flip. */ + private static final class ForeignHandle implements ConnectorTableHandle { + } + + /** + * The connector-level hive write provider, stubbed so the divert test does not build a real HmsClient. A hive + * handle must resolve to exactly THIS (what the no-arg {@link HiveConnector#getWritePlanProvider()} returns); + * the real no-arg provider construction (which needs an HmsClient) is covered by the write-plan suites. + */ + private final ConnectorWritePlanProvider stubbedHiveProvider = new MarkerWriteProvider(); + + /** A gateway whose no-arg hive provider is the stub above — isolates the per-handle routing from HmsClient. */ + private HiveConnector gatewayWithStubbedHiveProvider(RecordingSiblingContext context) { + return new HiveConnector(props(), context) { + @Override + public ConnectorWritePlanProvider getWritePlanProvider() { + return stubbedHiveProvider; + } + }; + } + + @Test + public void foreignHandleDelegatesToSiblingWriteProviderUnmodified() { + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(props(), context); + ForeignHandle foreign = new ForeignHandle(); + + ConnectorWritePlanProvider provider = connector.getWritePlanProvider(foreign); + + Assertions.assertSame(sibling.provider, provider, + "a foreign handle must return the iceberg sibling's OWN write provider"); + Assertions.assertSame(foreign, sibling.lastHandle, + "the foreign handle must reach the sibling's per-handle selector UNMODIFIED (a rewrap would " + + "poison the downstream iceberg cast)"); + Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once and consulted"); + } + + @Test + public void hiveHandleUsesHiveProviderWithoutConsultingSibling() { + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = gatewayWithStubbedHiveProvider(context); + HiveTableHandle hive = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE).build(); + + ConnectorWritePlanProvider provider = connector.getWritePlanProvider(hive); + + Assertions.assertSame(stubbedHiveProvider, provider, + "a hive handle resolves to the connector-level hive provider (the no-arg getWritePlanProvider)"); + Assertions.assertEquals(0, context.buildCount, "a hive handle must never build or consult the sibling"); + Assertions.assertNull(sibling.lastHandle, "the sibling's write provider must not be consulted for a hive handle"); + } + + @Test + public void hudiStampedHiveHandleStaysOnHiveProvider() { + // The route keys on the JVM handle TYPE (HiveTableHandle), not the format enum: a HUDI-stamped hive handle + // is still a HiveTableHandle, so it stays on the hive write path (hudi delegation is a later substep). + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = gatewayWithStubbedHiveProvider(context); + HiveTableHandle hudi = new HiveTableHandle.Builder("db", "t", HiveTableType.HUDI).build(); + + ConnectorWritePlanProvider provider = connector.getWritePlanProvider(hudi); + + Assertions.assertSame(stubbedHiveProvider, provider, "a hudi-stamped hive handle stays on the hive provider"); + Assertions.assertEquals(0, context.buildCount, "a hudi table must NOT be diverted to the iceberg sibling"); + } + + @Test + public void foreignHandleFailsLoudWhenIcebergPluginAbsent() { + // The seam returns a null sibling when the iceberg plugin is absent; selecting a write provider for a + // foreign handle must fail loud (naming the catalog), not NPE. + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(props(), context); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> connector.getWritePlanProvider(new ForeignHandle()), + "a foreign handle with no iceberg plugin must fail loud"); + Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); + } + + private static Map props() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", METASTORE_URI); + return props; + } + + /** Records the {@code createSiblingConnector} call and returns a configurable (possibly null) sibling. */ + private static final class RecordingSiblingContext extends FakeConnectorContext { + int buildCount; + Connector siblingToReturn; + + RecordingSiblingContext(Connector siblingToReturn) { + this.siblingToReturn = siblingToReturn; + } + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + buildCount++; + return siblingToReturn; + } + } + + /** A sibling {@link Connector} whose per-handle write provider is a distinguishable marker, recording the handle. */ + private static final class RecordingSibling implements Connector { + final ConnectorWritePlanProvider provider = new MarkerWriteProvider(); + ConnectorTableHandle lastHandle; + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { + lastHandle = handle; + return provider; + } + + @Override + public void close() { + } + } + + /** A bare write provider stand-in; only its identity matters here. */ + private static final class MarkerWriteProvider implements ConnectorWritePlanProvider { + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + return null; + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 7e26f12b1636b9..fd1d96166a6461 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -522,16 +522,22 @@ private List fetchSyntheticWriteColumns() { if (connector == null) { return Collections.emptyList(); } - ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(); - if (writePlanProvider == null) { - return Collections.emptyList(); - } + // Resolve the handle first so the write provider is selected per-table (a heterogeneous gateway routes + // iceberg-on-HMS to its sibling by the handle type); both null-degrade checks keep the empty fallback. + // Equivalent result for single-format connectors (getWritePlanProvider(handle) defaults to the no-arg + // one); this gated (show-hidden / row-level-DML) path resolves the handle before the provider-null check, + // so a read-only connector now resolves the handle here — a no-op for a write-capable connector, which + // resolved it regardless. ConnectorSession session = pluginCatalog.buildConnectorSession(); ConnectorMetadata metadata = connector.getMetadata(session); Optional handleOpt = resolveConnectorTableHandle(session, metadata); if (!handleOpt.isPresent()) { return Collections.emptyList(); } + ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(handleOpt.get()); + if (writePlanProvider == null) { + return Collections.emptyList(); + } return writePlanProvider.getSyntheticWriteColumns(session, handleOpt.get()); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 212ad7ecdd7627..f91cabd14f39ff 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -649,13 +649,16 @@ private PluginDrivenTableSink buildPluginRowLevelDmlSink( "Connector '" + catalog.getName() + "' (type: " + catalog.getType() + ") does not support row-level DML operations"); } - ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(); + // Resolve the table handle BEFORE selecting the write provider so the provider is chosen per-table (a + // heterogeneous gateway routes iceberg-on-HMS to its sibling by the handle type); byte-identical for + // every single-format connector (getWritePlanProvider(handle) defaults to the connector-level provider). ConnectorTableHandle providerTableHandle = metadata.getTableHandle(connSession, targetTable.getRemoteDbName(), targetTable.getRemoteName()) .orElseThrow(() -> new AnalysisException( "Table not found: " + targetTable.getRemoteDbName() + "." + targetTable.getRemoteName() + " in catalog " + catalog.getName())); + ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(providerTableHandle); providerTableHandle = PluginDrivenScanNode.applyMvccSnapshotPin( metadata, connSession, providerTableHandle, MvccUtil.getSnapshotFromContext(targetTable)); @@ -697,13 +700,16 @@ public PlanFragment visitPhysicalConnectorTableSink( "Connector '" + catalog.getName() + "' (type: " + catalog.getType() + ") does not support INSERT operations"); } - ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(); + // Resolve the table handle BEFORE selecting the write provider so the provider is chosen per-table (a + // heterogeneous gateway routes iceberg-on-HMS to its sibling by the handle type); byte-identical for + // every single-format connector (getWritePlanProvider(handle) defaults to the connector-level provider). ConnectorTableHandle providerTableHandle = metadata.getTableHandle(connSession, targetTable.getRemoteDbName(), targetTable.getRemoteName()) .orElseThrow(() -> new AnalysisException( "Table not found: " + targetTable.getRemoteDbName() + "." + targetTable.getRemoteName() + " in catalog " + catalog.getName())); + ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(providerTableHandle); // Thread the statement's MVCC snapshot pin onto the WRITE handle, reusing the exact scan-side pin // logic so a DML's write anchors at the SAME snapshot its scan read (the pin is keyed by diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java index 17f36dfbf5a6d3..8e5cf5a28993e7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSink.java @@ -299,17 +299,20 @@ private InsertPartitionFieldResult buildInsertPartitionFieldsFromConnector( Map columnExprIdMap) { PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); Connector connector = catalog.getConnector(); - ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(); - if (writePlanProvider == null) { - return new InsertPartitionFieldResult(false, false, null); - } ConnectorSession session = catalog.buildConnectorSession(); ConnectorMetadata metadata = connector.getMetadata(session); + // Resolve the handle first so the write provider is selected per-table (a heterogeneous gateway routes + // iceberg-on-HMS to its sibling by the handle type); both null-degrade checks keep the non-partitioned + // fallback. Byte-identical for single-format connectors (getWritePlanProvider(handle) defaults through). ConnectorTableHandle handle = metadata.getTableHandle( session, table.getRemoteDbName(), table.getRemoteName()).orElse(null); if (handle == null) { return new InsertPartitionFieldResult(false, false, null); } + ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(handle); + if (writePlanProvider == null) { + return new InsertPartitionFieldResult(false, false, null); + } ConnectorWritePartitionSpec spec = writePlanProvider.getWritePartitioning(session, handle); return reconstructPartitionFields(insertPartitionFields, spec, columnExprIdMap); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index 385ab80cd64549..93affd22ffea44 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -90,6 +90,10 @@ private static PluginDrivenExternalTable pluginTable(List synth ConnectorSession session = Mockito.mock(ConnectorSession.class); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getWritePlanProvider()).thenReturn(writeProviderPresent ? provider : null); + // Production now selects the write provider per-handle; a plain Mockito mock does not run the interface + // default, so stub the per-handle overload to the same provider (mirrors the scan seam). + Mockito.when(connector.getWritePlanProvider(Mockito.any())) + .thenReturn(writeProviderPresent ? provider : null); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); Mockito.when(catalog.getConnector()).thenReturn(connector); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java index 2a1a4372508643..da5afa35548c90 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java @@ -144,6 +144,8 @@ private static PluginDrivenExternalTable pluginTable(Set ops) { ConnectorWritePlanProvider provider = Mockito.mock(ConnectorWritePlanProvider.class); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getWritePlanProvider()).thenReturn(provider); + // Production selects the write provider per-handle; a plain mock does not run the interface default. + Mockito.when(connector.getWritePlanProvider(Mockito.any())).thenReturn(provider); Mockito.when(connector.supportedWriteOperations()).thenReturn(ops); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java index 4c34ae9e20751f..f3500b881a6767 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java @@ -265,6 +265,8 @@ private static Plugin pluginTable() { ConnectorWritePlanProvider provider = Mockito.mock(ConnectorWritePlanProvider.class); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getWritePlanProvider()).thenReturn(provider); + // Production selects the write provider per-handle; a plain mock does not run the interface default. + Mockito.when(connector.getWritePlanProvider(Mockito.any())).thenReturn(provider); // The row-level DML gate (buildPluginRowLevelDmlSink) admits on connector.supportedWriteOperations() // containing DELETE/MERGE. On a plain mock the Connector delegator default is not invoked, so stub it // directly (an iceberg connector declares row-level DML support). diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSinkTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSinkTest.java index 11a0d7d5304bcf..7e9d1696fba097 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSinkTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/physical/PhysicalIcebergMergeSinkTest.java @@ -315,6 +315,8 @@ private static PluginDrivenExternalTable pluginTable(ConnectorWritePartitionSpec ConnectorSession session = Mockito.mock(ConnectorSession.class); Connector connector = Mockito.mock(Connector.class); Mockito.when(connector.getWritePlanProvider()).thenReturn(provider); + // Production selects the write provider per-handle; a plain mock does not run the interface default. + Mockito.when(connector.getWritePlanProvider(Mockito.any())).thenReturn(provider); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) .thenReturn(handlePresent ? Optional.of(handle) : Optional.empty()); From db9478022ef5de659db7c7190ac75950fe6bd909 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 14:07:32 +0800 Subject: [PATCH 078/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20write-delegation=20W2=20DONE;=20next=20=3D=20W3=20(per-ha?= =?UTF-8?q?ndle=20write-admission)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W2 landed (84890fe48d3): getWritePlanProvider(handle) overload + HiveConnector divert + 4 fe-core provider-fetch conversions; adversarial review caught + fixed a blocker (4 fe-core mock tests must stub the per-handle overload). Next = W3: the 7 write-admission per-handle overloads (default-derived from getWritePlanProvider( handle)) + fe-core admission-site handle resolution. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d3b72836f996c7..11dad692d161a3 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -67,7 +67,8 @@ - **✅ §4.4 S6 按名方法/能力残留 = 零代码收官（recon `wf_6ac0e047-768`：4 维读者 + 完整性/正确性 critic 逐条 HEAD 核对；权威结论 = `hms-s6-name-based-divert-findings-2026-07-08.md`，起步必读）。校正分解文档 §3 S6 草案**——所有无句柄**读**方法**均 NO_DIVERT**，S6 无连接器代码改动，只落 findings 文档 + 本残留清单。理由：`buildTableDescriptor` 字节等价（兄弟恒 hms-flavor→同 `HIVE_TABLE`+`THiveTable`；legacy iceberg-on-HMS 亦发 `HIVE_TABLE`；分流只白加一次 `getTable`）；视图族（`viewExists`/`getViewDefinition`/`dropView`）legacy 平价（iceberg 表 `viewExists`=false 留 getTableHandle 分流路径；iceberg-on-HMS 视图存 hive 方言 SQL 于 HMS view text、走 hive 视图路=legacy 平价）**且无句柄不可判别**（`detect` 对视图返 UNKNOWN——iceberg 视图 `table_type='iceberg-view'`≠`'ICEBERG'`）；`listViewNames` 保持不覆写（分流会双列视图=回归）；`getTableComment` 不覆写=SPI 默认 `""`=legacy `getComment()` 恒 `""` 精确平价（分流会显示 legacy 从不显示的 comment=回归）；db/目录级方法（listDatabaseNames/databaseExists/listTableNames/getProperties/create[Database]/getPrimaryKeys/getDatabase）format-agnostic。**⇒ 至此 §4.4 读委派脊柱（S0–S6）全部收官**。 - ⚠**翻闸残留（findings 文档已详列，休眠期不动）**：①`dropDatabase` force 级联对 iceberg 表裸 `hmsClient.dropTable`（=legacy 平价、无 purge，只泄漏 iceberg 文件）→**写/DDL 委派步**；②`createTable` 引擎路由（新表无可 detect）→写/DDL 步；③`beginTransaction`/`getWritePlanProvider` **无 per-handle overload**→写委派步（新 SPI overload + ALTER-DDL/写校验，见分解文档 §6）；④`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 over-admit hudi（已注于 `HiveConnector.java:139-144`，cost-only、无 per-handle escape）；⑤`supportsLatestSnapshotPreload` under-admit iceberg/hudi（fe-core TableIf 默认 false vs legacy true，latency-only、非 enum-backed）；⑥**first-class iceberg-on-HMS 视图 = 翻闸显式非目标**（legacy 从不经 HMS 目录服务 iceberg ViewCatalog 视图；要做属新特性）。 - **§4.4 写委派权威分解 = `hms-write-delegation-decomposition-2026-07-08.md`（起步必读，recon `wf_f508ac0e-8ec` 7 维 + 完整性/分解 critic）。拆成 W1–W6 六个休眠小步，三种委派形态**：Shape A=写/存储过程 getter 加 per-handle 重载（W2/W3/W5）；Shape B=ConnectorMetadata 句柄守卫转发（W1，零新 SPI，**已 DONE**）；Shape C=事务按句柄选择（W4，唯一无先例的新形态）。**签字决策（用户 2026-07-08）：iceberg-on-HMS 升级为完整 native-iceberg 写/DELETE/MERGE/EXECUTE/ALTER 能力**（旧行为全拒=有意升级，非字节平价；对齐 Trino 按句柄委派）。**e2e 硬要求（用户 2026-07-08，memory `hms-iceberg-delegation-needs-e2e`）：每项新增 iceberg 能力（写/ALTER/procedure）最终都须补 e2e 回归（休眠单测只证路由、证不了跨加载器 cast/commit/worker-pool）；实现完成后择机统一补。** -- **下一步 = W2**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例）→ `HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（唯一真覆写）；fe-core 4 处 provider 取用点把已在 1 行后解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator:654/702`、`PhysicalIcebergMergeSink:302`、`PluginDrivenExternalTable:525`；connector-agnostic 合铁律）。**W3**（7 个写能力判定 per-handle 重载，从 W2 自动派生 + fe-core 11 处 analysis-time 门用 `PluginDrivenExternalTable.resolveConnectorTableHandle` 取句柄）依赖 W2；**W4** 事务按句柄（`beginTransaction(session,handle)` 守卫转发，否则 `IcebergWritePlanProvider.currentTransaction` 对 `HiveConnectorTransaction` CCE）依赖 W2；**W5** 存储过程（`getProcedureOps(handle)` + `ConnectorExecuteAction` 解析顺序上提）独立；**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测）。W2+W3+W4+W6=写执行捆绑须全在翻闸前落，W1/W5 可独立提交。 +- **✅ W2 DONE（`84890fe48d3`）**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例，对 6 个现存连接器行为不变）；`HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（外来句柄永不强转，hudi 保 HiveTableHandle 留 hive）；fe-core 4 处 provider 取用点把已解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator` DELETE/MERGE+INSERT、`PhysicalIcebergMergeSink`、`PluginDrivenExternalTable.fetchSyntheticWriteColumns`；connector-agnostic 合铁律）。**对抗复审揪出 1 blocker**：4 个 fe-core mock 测只 stub 无参 `getWritePlanProvider()`，Mockito mock 不跑 default→per-handle 返 null→测挂（memory `catalog-spi-fe-core-test-infra`），已补 stub per-handle 重载（照 scan seam）；1 minor=`fetchSyntheticWriteColumns` reorder 对只读连接器（show-hidden 门）多解析一次 handle，注释已改准。全绿（新 `HiveConnectorWriteProviderDivertTest` 4 测；hive 213 + fe-core 触及套 42 测 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。 +- **下一步 = W3**：7 个写能力判定 per-handle 重载（`supportedWriteOperations(handle)`/`supportsWriteBranch(handle)`/5 个 `requires*(handle)`，各 null-safe 从 `getWritePlanProvider(handle)` 派生→W2 已分流故**无需**额外 `HiveConnector` 覆写=最小面）；fe-core 分两组接线：(a) 已有句柄处上提（`PhysicalPlanTranslator:648/697` 的 `supportedWriteOperations`）；(b) analysis-time 无句柄门（`RowLevelDmlRowIdUtils:133`、`IcebergRowLevelDmlTransform:93`〔DELETE/MERGE 探测,吃重〕、`InsertOverwriteTableCommand:336/351`、`InsertIntoTableCommand:794`、`PluginDrivenExternalTable` requires* 包装 115/233/249/263/278 + `BindSink:800/811`）用 `PluginDrivenExternalTable.resolveConnectorTableHandle` 取句柄。**风险 MEDIUM**：`supportedWriteOperations(handle)` 吃重（无它 iceberg-on-HMS DELETE/MERGE 不被认作 row-level DML）；`requiresMaterializeStaticPartitionValues`（hive false vs iceberg true）+`requiresPartitionHashWrite`（hive true vs iceberg false）真差异，半改会误算。**W4** 事务按句柄（`beginTransaction(session,handle)` 守卫转发，否则 `IcebergWritePlanProvider.currentTransaction` 对 `HiveConnectorTransaction` CCE）依赖 W2；**W5** 存储过程（`getProcedureOps(handle)` + `ConnectorExecuteAction` 解析顺序上提）独立；**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测）。W2+W3+W4+W6=写执行捆绑须全在翻闸前落，W1/W5 可独立提交。 - **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 From 3b503a9de73368947c57625f954c9e535fb9b974 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 15:04:10 +0800 Subject: [PATCH 079/334] [feat](catalog) HMS gateway admits writes per-handle: iceberg-on-HMS gets iceberg write ops/branch/partition semantics (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §4.4 write-delegation W3 (dormant): make the write-ADMISSION capabilities per-handle so a flipped hms gateway admits the operations of the table's actual provider (iceberg for iceberg-on-HMS, hive for hive), not the connector-level hive answer. Only 4 of the 7 write capabilities differ between the hive and iceberg providers (HEAD-verified): supportedWriteOperations (hive {INSERT,OVERWRITE} vs iceberg +DELETE,MERGE,REWRITE), supportsWriteBranch (false vs true), requiresPartitionHash Write (true vs false), requiresMaterializeStaticPartitionValues (false vs true). The other 3 (requiresParallelWrite, requiresFullSchemaWriteOrder, requiresPartition LocalSort) are byte-inert for the gateway (hive == iceberg), so they stay no-arg. - Connector: 4 new per-handle overloads, each default-deriving from getWritePlanProvider(handle) exactly as the no-arg views derive from getWritePlanProvider() -> behavior-preserving for every single-format connector, and (via W2's HiveConnector divert) auto-routing per-table with NO extra HiveConnector override. - fe-core: PhysicalPlanTranslator DELETE/MERGE + INSERT gates resolve the handle first and call supportedWriteOperations(handle); PluginDrivenExternalTable adds connectorSupportedWriteOperations()/connectorSupportsWriteBranch() helpers (resolve the handle, call the per-handle overload, DEGRADE to empty/false on a null connector / unresolvable handle -> never a null handle to the gateway) and makes requirePartitionHashOnWrite()/materializeStaticPartitionValues() per-handle; RowLevelDmlRowIdUtils, IcebergRowLevelDmlTransform, InsertInto/OverwriteTable Command route through the table helpers. Connector-agnostic, no if(format) branch. Dormant until "hms" enters SPI_READY_TYPES. Adversarial review: no blocker/major; fixed a neutered degrade test + added null-connector / unresolvable-handle coverage for all 4 differing capabilities. fe-core touched suites green (incl. Plugin DrivenExternalTableTest 28), checkstyle 0 (api + fe-core). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/api/Connector.java | 29 ++++++++ .../datasource/PluginDrivenExternalTable.java | 67 +++++++++++++++++- .../translator/PhysicalPlanTranslator.java | 34 ++++----- .../commands/IcebergRowLevelDmlTransform.java | 4 +- .../plans/commands/RowLevelDmlRowIdUtils.java | 15 ++-- .../insert/InsertIntoTableCommand.java | 6 +- .../insert/InsertOverwriteTableCommand.java | 11 ++- .../PluginDrivenExternalTableTest.java | 69 +++++++++++++++++++ ...ysicalPlanTranslatorAdmissionGateTest.java | 2 + ...lPlanTranslatorIcebergRowLevelDmlTest.java | 3 + .../IcebergRowLevelDmlTransformTest.java | 4 +- .../commands/RowLevelDmlRowIdUtilsTest.java | 24 +++---- .../insert/InsertIntoTableCommandTest.java | 9 +-- .../InsertOverwriteTableCommandTest.java | 16 ++--- 14 files changed, 217 insertions(+), 76 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index c5c39136b58339..dfb891571f4fd4 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -98,12 +98,29 @@ default Set supportedWriteOperations() { return p == null ? EnumSet.noneOf(WriteOperation.class) : p.supportedOperations(); } + /** + * Per-table view of {@link #supportedWriteOperations()}: derives from {@link #getWritePlanProvider( + * ConnectorTableHandle)} so a heterogeneous gateway admits the right operations for {@code handle} (e.g. an + * iceberg-on-HMS table admits DELETE/MERGE that the hive provider does not). The default routes through the + * per-handle provider, so every single-format connector is unaffected. + */ + default Set supportedWriteOperations(ConnectorTableHandle handle) { + ConnectorWritePlanProvider p = getWritePlanProvider(handle); + return p == null ? EnumSet.noneOf(WriteOperation.class) : p.supportedOperations(); + } + /** Null-safe view of {@link ConnectorWritePlanProvider#supportsWriteBranch()}. No provider ⇒ false. */ default boolean supportsWriteBranch() { ConnectorWritePlanProvider p = getWritePlanProvider(); return p != null && p.supportsWriteBranch(); } + /** Per-table view of {@link #supportsWriteBranch()} (derives from the per-handle provider). */ + default boolean supportsWriteBranch(ConnectorTableHandle handle) { + ConnectorWritePlanProvider p = getWritePlanProvider(handle); + return p != null && p.supportsWriteBranch(); + } + /** Null-safe view of {@link ConnectorWritePlanProvider#requiresParallelWrite()}. No provider ⇒ false. */ default boolean requiresParallelWrite() { ConnectorWritePlanProvider p = getWritePlanProvider(); @@ -128,6 +145,12 @@ default boolean requiresPartitionHashWrite() { return p != null && p.requiresPartitionHashWrite(); } + /** Per-table view of {@link #requiresPartitionHashWrite()} (derives from the per-handle provider). */ + default boolean requiresPartitionHashWrite(ConnectorTableHandle handle) { + ConnectorWritePlanProvider p = getWritePlanProvider(handle); + return p != null && p.requiresPartitionHashWrite(); + } + /** * Null-safe view of {@link ConnectorWritePlanProvider#requiresMaterializeStaticPartitionValues()}. No * provider ⇒ false. @@ -137,6 +160,12 @@ default boolean requiresMaterializeStaticPartitionValues() { return p != null && p.requiresMaterializeStaticPartitionValues(); } + /** Per-table view of {@link #requiresMaterializeStaticPartitionValues()} (derives from the per-handle provider). */ + default boolean requiresMaterializeStaticPartitionValues(ConnectorTableHandle handle) { + ConnectorWritePlanProvider p = getWritePlanProvider(handle); + return p != null && p.requiresMaterializeStaticPartitionValues(); + } + /** * Returns the procedure ops for {@code ALTER TABLE EXECUTE} dispatch, or {@code null} if this * connector exposes no table procedures. Procedure-side analogue of {@link #getWritePlanProvider()}. diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index fd1d96166a6461..a949cd6043e606 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -35,6 +35,7 @@ import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.ConnectorViewDefinition; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.systable.PluginDrivenSysTable; @@ -56,11 +57,13 @@ import java.io.IOException; import java.util.ArrayList; import java.util.Collections; +import java.util.EnumSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.Map.Entry; import java.util.Optional; +import java.util.Set; import java.util.stream.Collectors; /** @@ -112,9 +115,57 @@ public boolean supportsParallelWrite() { return false; } Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + // requiresParallelWrite is byte-inert for a heterogeneous gateway (hive and iceberg both true), so the + // connector-level answer needs no per-handle resolution here. return connector != null && connector.requiresParallelWrite(); } + /** + * Resolves this table's connector handle for a per-handle write-capability probe, or empty on any miss (a + * null connector, or an unresolvable handle). A heterogeneous gateway needs the handle to answer write + * capabilities per-table (its iceberg tables differ from its hive tables); a single-format connector ignores + * the handle (the per-handle overloads default to connector-level), so this is byte-identical for it. + */ + private Optional resolveWriteCapabilityHandle(Connector connector) { + ConnectorSession session = ((PluginDrivenExternalCatalog) catalog).buildConnectorSession(); + return resolveConnectorTableHandle(session, connector.getMetadata(session)); + } + + /** + * The write operations the connector admits for THIS table, resolved per-handle so a heterogeneous gateway + * admits DELETE/MERGE/OVERWRITE for its iceberg tables but only INSERT/OVERWRITE for its hive tables. + * Degrades to the empty set (all writes rejected) on any miss, mirroring {@link #fetchSyntheticWriteColumns()}. + */ + public Set connectorSupportedWriteOperations() { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return EnumSet.noneOf(WriteOperation.class); + } + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + if (connector == null) { + return EnumSet.noneOf(WriteOperation.class); + } + return resolveWriteCapabilityHandle(connector) + .map(connector::supportedWriteOperations) + .orElseGet(() -> EnumSet.noneOf(WriteOperation.class)); + } + + /** + * Whether the connector admits branch writes for THIS table, resolved per-handle (iceberg supports + * write-to-branch, hive does not). Degrades to false on any miss. + */ + public boolean connectorSupportsWriteBranch() { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return false; + } + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + if (connector == null) { + return false; + } + return resolveWriteCapabilityHandle(connector) + .map(connector::supportsWriteBranch) + .orElse(false); + } + /** * Returns whether the underlying connector's tables support background per-column auto-analyze. * The statistics auto-collector consults this (in place of the legacy {@code instanceof @@ -246,7 +297,13 @@ public boolean requirePartitionHashOnWrite() { return false; } Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); - return connector != null && connector.requiresPartitionHashWrite(); + if (connector == null) { + return false; + } + // Per-table: hive requires partition-hash writes but iceberg does not, so resolve the handle. + return resolveWriteCapabilityHandle(connector) + .map(connector::requiresPartitionHashWrite) + .orElse(false); } /** @@ -275,7 +332,13 @@ public boolean materializeStaticPartitionValues() { return false; } Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); - return connector != null && connector.requiresMaterializeStaticPartitionValues(); + if (connector == null) { + return false; + } + // Per-table: iceberg retains partition columns (materialize the PARTITION literal), hive does not. + return resolveWriteCapabilityHandle(connector) + .map(connector::requiresMaterializeStaticPartitionValues) + .orElse(false); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index f91cabd14f39ff..0d0f691da282ce 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -643,21 +643,21 @@ private PluginDrivenTableSink buildPluginRowLevelDmlSink( null, col.isAllowNull(), null)) .collect(java.util.stream.Collectors.toList()); - Set writeOps = connector.supportedWriteOperations(); - if (!(writeOps.contains(WriteOperation.DELETE) || writeOps.contains(WriteOperation.MERGE))) { - throw new AnalysisException( - "Connector '" + catalog.getName() + "' (type: " + catalog.getType() - + ") does not support row-level DML operations"); - } - // Resolve the table handle BEFORE selecting the write provider so the provider is chosen per-table (a - // heterogeneous gateway routes iceberg-on-HMS to its sibling by the handle type); byte-identical for - // every single-format connector (getWritePlanProvider(handle) defaults to the connector-level provider). + // Resolve the table handle first so BOTH the write-admission gate and the write provider are chosen + // per-table (a heterogeneous gateway routes iceberg-on-HMS to its sibling by the handle type); + // byte-identical for every single-format connector (the per-handle overloads default to connector-level). ConnectorTableHandle providerTableHandle = metadata.getTableHandle(connSession, targetTable.getRemoteDbName(), targetTable.getRemoteName()) .orElseThrow(() -> new AnalysisException( "Table not found: " + targetTable.getRemoteDbName() + "." + targetTable.getRemoteName() + " in catalog " + catalog.getName())); + Set writeOps = connector.supportedWriteOperations(providerTableHandle); + if (!(writeOps.contains(WriteOperation.DELETE) || writeOps.contains(WriteOperation.MERGE))) { + throw new AnalysisException( + "Connector '" + catalog.getName() + "' (type: " + catalog.getType() + + ") does not support row-level DML operations"); + } ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(providerTableHandle); providerTableHandle = PluginDrivenScanNode.applyMvccSnapshotPin( metadata, connSession, providerTableHandle, MvccUtil.getSnapshotFromContext(targetTable)); @@ -695,20 +695,20 @@ public PlanFragment visitPhysicalConnectorTableSink( // Every write-capable connector builds its own opaque TDataSink via its write-plan // provider (jdbc / maxcompute / iceberg). A connector whose declared write operations do // not include INSERT does not support writes. - if (!connector.supportedWriteOperations().contains(WriteOperation.INSERT)) { - throw new AnalysisException( - "Connector '" + catalog.getName() + "' (type: " + catalog.getType() - + ") does not support INSERT operations"); - } - // Resolve the table handle BEFORE selecting the write provider so the provider is chosen per-table (a - // heterogeneous gateway routes iceberg-on-HMS to its sibling by the handle type); byte-identical for - // every single-format connector (getWritePlanProvider(handle) defaults to the connector-level provider). + // Resolve the table handle first so BOTH the INSERT-admission gate and the write provider are chosen + // per-table (a heterogeneous gateway routes iceberg-on-HMS to its sibling by the handle type); + // byte-identical for every single-format connector (the per-handle overloads default to connector-level). ConnectorTableHandle providerTableHandle = metadata.getTableHandle(connSession, targetTable.getRemoteDbName(), targetTable.getRemoteName()) .orElseThrow(() -> new AnalysisException( "Table not found: " + targetTable.getRemoteDbName() + "." + targetTable.getRemoteName() + " in catalog " + catalog.getName())); + if (!connector.supportedWriteOperations(providerTableHandle).contains(WriteOperation.INSERT)) { + throw new AnalysisException( + "Connector '" + catalog.getName() + "' (type: " + catalog.getType() + + ") does not support INSERT operations"); + } ConnectorWritePlanProvider writePlanProvider = connector.getWritePlanProvider(providerTableHandle); // Thread the statement's MVCC snapshot pin onto the WRITE handle, reusing the exact scan-side pin diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java index 5fea0d3700b00d..9c3bea9c3de594 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransform.java @@ -89,8 +89,8 @@ public boolean handles(TableIf table) { * the {@code ConnectorWriteOps} default {@code false}).

    */ private static boolean pluginConnectorSupportsRowLevelDml(PluginDrivenExternalTable table) { - PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); - Set ops = catalog.getConnector().supportedWriteOperations(); + // Per-handle write-op probe: a heterogeneous gateway admits row-level DML for its iceberg tables only. + Set ops = table.connectorSupportedWriteOperations(); return ops.contains(WriteOperation.DELETE) || ops.contains(WriteOperation.MERGE); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtils.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtils.java index 4ed0e452699f96..ca967991bc7fc6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtils.java @@ -18,10 +18,8 @@ package org.apache.doris.nereids.trees.plans.commands; import org.apache.doris.catalog.Column; -import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.nereids.analyzer.Unbound; import org.apache.doris.nereids.trees.expressions.NamedExpression; @@ -123,14 +121,11 @@ static boolean isRowIdInjectionTarget(ExternalTable table) { } private static boolean pluginConnectorSupportsRowLevelDml(PluginDrivenExternalTable table) { - PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); - Connector connector = catalog.getConnector(); - if (connector == null) { - // A catalog dropped mid-DML-planning nulls its transient connector; degrade to "not a target" - // rather than NPE-aborting the query, mirroring PluginDrivenExternalTable.fetchSyntheticWriteColumns. - return false; - } - Set ops = connector.supportedWriteOperations(); + // Resolved per-handle through the table's write-op probe (a heterogeneous gateway admits row-level DML + // for its iceberg tables but not its hive tables). It degrades to the empty set on a dropped connector / + // unresolvable handle (mirroring fetchSyntheticWriteColumns), so a mid-DML catalog drop is "not a target" + // rather than an NPE. + Set ops = table.connectorSupportedWriteOperations(); return ops.contains(WriteOperation.DELETE) || ops.contains(WriteOperation.MERGE); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java index 21e7be1bd8975a..283f43abba4316 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommand.java @@ -32,7 +32,6 @@ import org.apache.doris.common.util.Util; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.FileScanNode; -import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.doris.RemoteDorisExternalTable; import org.apache.doris.datasource.doris.RemoteOlapTable; @@ -746,9 +745,8 @@ private static boolean connectorSupportsWriteBranch(TableIf targetTable) { if (!(targetTable instanceof PluginDrivenExternalTable)) { return false; } - PluginDrivenExternalCatalog catalog = - (PluginDrivenExternalCatalog) ((PluginDrivenExternalTable) targetTable).getCatalog(); - return catalog.getConnector().supportsWriteBranch(); + // Per-handle: a heterogeneous gateway supports write-to-branch for its iceberg tables but not its hive. + return ((PluginDrivenExternalTable) targetTable).connectorSupportsWriteBranch(); } @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java index 3d5f5a99065135..8f06015b87cc1d 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommand.java @@ -27,7 +27,6 @@ import org.apache.doris.common.UserException; import org.apache.doris.common.util.InternalDatabaseUtil; import org.apache.doris.connector.api.handle.WriteOperation; -import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.doris.RemoteDorisExternalTable; import org.apache.doris.datasource.doris.RemoteOlapTable; @@ -332,8 +331,9 @@ private boolean allowInsertOverwrite(TableIf targetTable) { * {@code PhysicalPlanTranslator}. */ private static boolean pluginConnectorSupportsInsertOverwrite(PluginDrivenExternalTable table) { - PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); - return catalog.getConnector().supportedWriteOperations().contains(WriteOperation.OVERWRITE); + // Per-handle write-op probe (a heterogeneous gateway answers per-table; OVERWRITE happens to be admitted + // by both hive and iceberg, but the probe is resolved uniformly with the other write-op admission gates). + return table.connectorSupportedWriteOperations().contains(WriteOperation.OVERWRITE); } /** @@ -346,9 +346,8 @@ private static boolean pluginConnectorSupportsWriteBranch(TableIf targetTable) { if (!(targetTable instanceof PluginDrivenExternalTable)) { return false; } - PluginDrivenExternalCatalog catalog = - (PluginDrivenExternalCatalog) ((PluginDrivenExternalTable) targetTable).getCatalog(); - return catalog.getConnector().supportsWriteBranch(); + // Per-handle: a heterogeneous gateway supports write-to-branch for its iceberg tables but not its hive. + return ((PluginDrivenExternalTable) targetTable).connectorSupportsWriteBranch(); } private void runInsertCommand(LogicalPlan logicalQuery, InsertCommandContext insertCtx, diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index 93affd22ffea44..c208c1b3fadf78 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -29,6 +29,7 @@ import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.ConnectorViewDefinition; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.qe.ConnectContext; @@ -74,6 +75,74 @@ public void clearCtx() { ConnectContext.remove(); } + // ==================== §4.4 W3: per-handle write-admission capability probes ==================== + + /** + * A CALLS_REAL_METHODS table whose connector answers the write capabilities PER-HANDLE (the overloads a + * heterogeneous gateway diverts). {@code handlePresent=false} models an unresolvable handle. + */ + private static PluginDrivenExternalTable capabilityTable(boolean handlePresent, + Set ops, boolean branch) { + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(handlePresent ? Optional.of(handle) : Optional.empty()); + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); + Mockito.when(connector.supportedWriteOperations(Mockito.any())).thenReturn(ops); + Mockito.when(connector.supportsWriteBranch(Mockito.any())).thenReturn(branch); + Mockito.when(connector.requiresPartitionHashWrite(Mockito.any())).thenReturn(true); + Mockito.when(connector.requiresMaterializeStaticPartitionValues(Mockito.any())).thenReturn(true); + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(catalog.buildConnectorSession()).thenReturn(Mockito.mock(ConnectorSession.class)); + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); + Deencapsulation.setField(table, "catalog", catalog); + return table; + } + + @Test + public void connectorWriteCapabilitiesResolvePerHandle() { + Set ops = EnumSet.of(WriteOperation.INSERT, WriteOperation.DELETE, WriteOperation.MERGE); + PluginDrivenExternalTable table = capabilityTable(true, ops, true); + Assertions.assertEquals(ops, table.connectorSupportedWriteOperations(), + "the write ops must come from the connector's per-handle overload (resolved via the handle)"); + Assertions.assertTrue(table.connectorSupportsWriteBranch(), + "the branch capability must come from the connector's per-handle overload"); + Assertions.assertTrue(table.requirePartitionHashOnWrite(), + "partition-hash-write must come from the connector's per-handle overload"); + Assertions.assertTrue(table.materializeStaticPartitionValues(), + "materialize-static-partition must come from the connector's per-handle overload"); + } + + @Test + public void connectorWriteCapabilitiesDegradeWhenHandleUnresolvable() { + // An unresolvable handle (dropped table / catalog) must degrade to "no writes" — empty op set + false — + // rather than misrouting or NPE-ing, even though the connector WOULD report the capabilities for a handle. + PluginDrivenExternalTable table = capabilityTable(false, EnumSet.of(WriteOperation.DELETE), true); + Assertions.assertTrue(table.connectorSupportedWriteOperations().isEmpty(), + "an unresolvable handle degrades write ops to the empty set"); + Assertions.assertFalse(table.connectorSupportsWriteBranch(), + "an unresolvable handle degrades branch support to false"); + Assertions.assertFalse(table.requirePartitionHashOnWrite(), + "an unresolvable handle degrades partition-hash-write to false"); + Assertions.assertFalse(table.materializeStaticPartitionValues(), + "an unresolvable handle degrades materialize-static-partition to false"); + } + + @Test + public void connectorWriteCapabilitiesDegradeWhenConnectorNull() { + // A catalog dropped mid-planning nulls its transient connector; the probes must degrade (empty / false) + // rather than NPE — this is the null-connector guard the row-id / DML gates rely on. + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalog.getConnector()).thenReturn(null); + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); + Deencapsulation.setField(table, "catalog", catalog); + Assertions.assertTrue(table.connectorSupportedWriteOperations().isEmpty(), + "a null connector degrades write ops to the empty set"); + Assertions.assertFalse(table.connectorSupportsWriteBranch(), "a null connector degrades branch to false"); + } + /** * Builds a CALLS_REAL_METHODS PluginDrivenExternalTable wired to a stubbed connector chain whose * write-plan provider returns {@code synthetic}. {@code writeProviderPresent=false} models a read-only diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java index da5afa35548c90..1d20c02f570b21 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorAdmissionGateTest.java @@ -147,6 +147,8 @@ private static PluginDrivenExternalTable pluginTable(Set ops) { // Production selects the write provider per-handle; a plain mock does not run the interface default. Mockito.when(connector.getWritePlanProvider(Mockito.any())).thenReturn(provider); Mockito.when(connector.supportedWriteOperations()).thenReturn(ops); + // The admission gate now resolves the handle first and consults the per-handle overload. + Mockito.when(connector.supportedWriteOperations(Mockito.any())).thenReturn(ops); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) .thenReturn(Optional.of(handle)); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java index f3500b881a6767..c4ab98e0b94532 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslatorIcebergRowLevelDmlTest.java @@ -272,6 +272,9 @@ private static Plugin pluginTable() { // directly (an iceberg connector declares row-level DML support). Mockito.when(connector.supportedWriteOperations()) .thenReturn(EnumSet.of(WriteOperation.DELETE, WriteOperation.MERGE)); + // Site 1 (row-level DML) now resolves the handle first and consults the per-handle overload. + Mockito.when(connector.supportedWriteOperations(Mockito.any())) + .thenReturn(EnumSet.of(WriteOperation.DELETE, WriteOperation.MERGE)); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) .thenReturn(Optional.of(handle)); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransformTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransformTest.java index 1aca5f00390b84..52bff44c325d48 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransformTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/IcebergRowLevelDmlTransformTest.java @@ -106,7 +106,9 @@ private static PluginDrivenExternalTable pluginTable(boolean supportsDelete, boo } Mockito.when(table.getCatalog()).thenReturn(catalog); Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(connector.supportedWriteOperations()).thenReturn(ops); + // The row-level DML admission probe now resolves per-handle via the table helper; stub it directly. The + // catalog -> connector chain is still needed for checkMode (validateRowLevelDmlMode). + Mockito.when(table.connectorSupportedWriteOperations()).thenReturn(ops); return table; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtilsTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtilsTest.java index 64484135386e05..101493bfa19ebd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtilsTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/RowLevelDmlRowIdUtilsTest.java @@ -17,10 +17,8 @@ package org.apache.doris.nereids.trees.plans.commands; -import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.datasource.ExternalTable; -import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.junit.jupiter.api.Assertions; @@ -47,12 +45,10 @@ private static PluginDrivenExternalTable pluginTableWithCapability(boolean suppo if (supportsMerge) { ops.add(WriteOperation.MERGE); } - Connector connector = Mockito.mock(Connector.class); - Mockito.when(connector.supportedWriteOperations()).thenReturn(ops); - PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); - Mockito.when(catalog.getConnector()).thenReturn(connector); PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); - Mockito.when(table.getCatalog()).thenReturn(catalog); + // The row-id guard now probes the per-handle write ops via the table helper (which resolves the handle + // and calls the connector's per-handle overload); stub the table method directly. + Mockito.when(table.connectorSupportedWriteOperations()).thenReturn(ops); return table; } @@ -91,15 +87,13 @@ public void isRowIdInjectionTargetRejectsUnrelatedExternalTable() { } @Test - public void isRowIdInjectionTargetDegradesWhenConnectorDropped() { - // A catalog dropped mid-DML-planning nulls its transient connector; the guard must degrade to - // "not a target" rather than NPE-aborting the query, mirroring the defensive - // PluginDrivenExternalTable.fetchSyntheticWriteColumns. MUTATION: dropping the null-connector guard - // NPEs here. - PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); - Mockito.when(catalog.getConnector()).thenReturn(null); + public void isRowIdInjectionTargetDegradesWhenNoWriteOps() { + // The per-handle write-op probe degrades to an EMPTY set on a dropped connector / unresolvable handle + // (the guard now lives in PluginDrivenExternalTable.connectorSupportedWriteOperations, covered by its own + // test); an empty set must read as "not a target" rather than admitting row-id injection. MUTATION: + // treating empty ops as a target reddens this. PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); - Mockito.when(table.getCatalog()).thenReturn(catalog); + Mockito.when(table.connectorSupportedWriteOperations()).thenReturn(EnumSet.noneOf(WriteOperation.class)); Assertions.assertFalse(RowLevelDmlRowIdUtils.isRowIdInjectionTarget(table)); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommandTest.java index cf60fcf2ec21d3..4e1cb9ecc57f35 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertIntoTableCommandTest.java @@ -20,8 +20,6 @@ import org.apache.doris.catalog.OlapTable; import org.apache.doris.catalog.TableIf; import org.apache.doris.common.jmockit.Deencapsulation; -import org.apache.doris.connector.api.Connector; -import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.doris.RemoteDorisExternalTable; import org.apache.doris.nereids.NereidsPlanner; @@ -182,11 +180,8 @@ void testSelectInsertExecutorFactoryForRemoteTableWithGroupCommitException() { */ private static PluginDrivenExternalTable pluginTableForWriteBranch(boolean supported) { PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); - PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); - Connector connector = Mockito.mock(Connector.class); - Mockito.when(table.getCatalog()).thenReturn(catalog); - Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(connector.supportsWriteBranch()).thenReturn(supported); + // The @branch gate now probes the per-handle capability via the table helper; stub it directly. + Mockito.when(table.connectorSupportsWriteBranch()).thenReturn(supported); return table; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java index 53e2e51277cd05..22326b1b4a0f39 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/InsertOverwriteTableCommandTest.java @@ -19,9 +19,7 @@ import org.apache.doris.catalog.TableIf; import org.apache.doris.common.jmockit.Deencapsulation; -import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.handle.WriteOperation; -import org.apache.doris.datasource.PluginDrivenExternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.nereids.trees.plans.logical.LogicalPlan; @@ -63,12 +61,9 @@ private static InsertOverwriteTableCommand newCommand() { */ private static PluginDrivenExternalTable pluginTable(boolean supported) { PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); - PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); - Connector connector = Mockito.mock(Connector.class); Set ops = supported ? EnumSet.of(WriteOperation.OVERWRITE) : EnumSet.noneOf(WriteOperation.class); - Mockito.when(table.getCatalog()).thenReturn(catalog); - Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(connector.supportedWriteOperations()).thenReturn(ops); + // The OVERWRITE gate now probes the per-handle write ops via the table helper; stub it directly. + Mockito.when(table.connectorSupportedWriteOperations()).thenReturn(ops); return table; } @@ -78,11 +73,8 @@ private static PluginDrivenExternalTable pluginTable(boolean supported) { */ private static PluginDrivenExternalTable pluginTableForWriteBranch(boolean supported) { PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); - PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); - Connector connector = Mockito.mock(Connector.class); - Mockito.when(table.getCatalog()).thenReturn(catalog); - Mockito.when(catalog.getConnector()).thenReturn(connector); - Mockito.when(connector.supportsWriteBranch()).thenReturn(supported); + // The @branch gate now probes the per-handle capability via the table helper; stub it directly. + Mockito.when(table.connectorSupportsWriteBranch()).thenReturn(supported); return table; } From d4cbee179525fa7d4f7c4a3ef82b8448bd3762cc Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 15:05:28 +0800 Subject: [PATCH 080/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20write-delegation=20W3=20DONE;=20next=20=3D=20W4=20(txn=20?= =?UTF-8?q?per-handle)=20or=20W5=20(procedures)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W3 landed (d1df2e0f7df): write-admission capabilities per-handle (only the 4 differing methods; 3 inert stay no-arg). Records the W3 review residuals (inert-3 rely on hive==iceberg; new getTableHandle round-trip on the live iceberg path). Next = W4 (beginTransaction(session,handle) guard-forward) or W5 (getProcedureOps( handle) + ConnectorExecuteAction reorder), both independently committable. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 11dad692d161a3..c5ea569d2ac594 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -68,7 +68,8 @@ - ⚠**翻闸残留（findings 文档已详列，休眠期不动）**：①`dropDatabase` force 级联对 iceberg 表裸 `hmsClient.dropTable`（=legacy 平价、无 purge，只泄漏 iceberg 文件）→**写/DDL 委派步**；②`createTable` 引擎路由（新表无可 detect）→写/DDL 步；③`beginTransaction`/`getWritePlanProvider` **无 per-handle overload**→写委派步（新 SPI overload + ALTER-DDL/写校验，见分解文档 §6）；④`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 over-admit hudi（已注于 `HiveConnector.java:139-144`，cost-only、无 per-handle escape）；⑤`supportsLatestSnapshotPreload` under-admit iceberg/hudi（fe-core TableIf 默认 false vs legacy true，latency-only、非 enum-backed）；⑥**first-class iceberg-on-HMS 视图 = 翻闸显式非目标**（legacy 从不经 HMS 目录服务 iceberg ViewCatalog 视图；要做属新特性）。 - **§4.4 写委派权威分解 = `hms-write-delegation-decomposition-2026-07-08.md`（起步必读，recon `wf_f508ac0e-8ec` 7 维 + 完整性/分解 critic）。拆成 W1–W6 六个休眠小步，三种委派形态**：Shape A=写/存储过程 getter 加 per-handle 重载（W2/W3/W5）；Shape B=ConnectorMetadata 句柄守卫转发（W1，零新 SPI，**已 DONE**）；Shape C=事务按句柄选择（W4，唯一无先例的新形态）。**签字决策（用户 2026-07-08）：iceberg-on-HMS 升级为完整 native-iceberg 写/DELETE/MERGE/EXECUTE/ALTER 能力**（旧行为全拒=有意升级，非字节平价；对齐 Trino 按句柄委派）。**e2e 硬要求（用户 2026-07-08，memory `hms-iceberg-delegation-needs-e2e`）：每项新增 iceberg 能力（写/ALTER/procedure）最终都须补 e2e 回归（休眠单测只证路由、证不了跨加载器 cast/commit/worker-pool）；实现完成后择机统一补。** - **✅ W2 DONE（`84890fe48d3`）**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例，对 6 个现存连接器行为不变）；`HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（外来句柄永不强转，hudi 保 HiveTableHandle 留 hive）；fe-core 4 处 provider 取用点把已解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator` DELETE/MERGE+INSERT、`PhysicalIcebergMergeSink`、`PluginDrivenExternalTable.fetchSyntheticWriteColumns`；connector-agnostic 合铁律）。**对抗复审揪出 1 blocker**：4 个 fe-core mock 测只 stub 无参 `getWritePlanProvider()`，Mockito mock 不跑 default→per-handle 返 null→测挂（memory `catalog-spi-fe-core-test-infra`），已补 stub per-handle 重载（照 scan seam）；1 minor=`fetchSyntheticWriteColumns` reorder 对只读连接器（show-hidden 门）多解析一次 handle，注释已改准。全绿（新 `HiveConnectorWriteProviderDivertTest` 4 测；hive 213 + fe-core 触及套 42 测 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。 -- **下一步 = W3**：7 个写能力判定 per-handle 重载（`supportedWriteOperations(handle)`/`supportsWriteBranch(handle)`/5 个 `requires*(handle)`，各 null-safe 从 `getWritePlanProvider(handle)` 派生→W2 已分流故**无需**额外 `HiveConnector` 覆写=最小面）；fe-core 分两组接线：(a) 已有句柄处上提（`PhysicalPlanTranslator:648/697` 的 `supportedWriteOperations`）；(b) analysis-time 无句柄门（`RowLevelDmlRowIdUtils:133`、`IcebergRowLevelDmlTransform:93`〔DELETE/MERGE 探测,吃重〕、`InsertOverwriteTableCommand:336/351`、`InsertIntoTableCommand:794`、`PluginDrivenExternalTable` requires* 包装 115/233/249/263/278 + `BindSink:800/811`）用 `PluginDrivenExternalTable.resolveConnectorTableHandle` 取句柄。**风险 MEDIUM**：`supportedWriteOperations(handle)` 吃重（无它 iceberg-on-HMS DELETE/MERGE 不被认作 row-level DML）；`requiresMaterializeStaticPartitionValues`（hive false vs iceberg true）+`requiresPartitionHashWrite`（hive true vs iceberg false）真差异，半改会误算。**W4** 事务按句柄（`beginTransaction(session,handle)` 守卫转发，否则 `IcebergWritePlanProvider.currentTransaction` 对 `HiveConnectorTransaction` CCE）依赖 W2；**W5** 存储过程（`getProcedureOps(handle)` + `ConnectorExecuteAction` 解析顺序上提）独立；**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测）。W2+W3+W4+W6=写执行捆绑须全在翻闸前落，W1/W5 可独立提交。 +- **下一步 = W4 或 W5**（二选一，均独立可提交）。**W4=事务按句柄**：`beginTransaction(session)` 今无句柄、`HiveConnectorMetadata:1353` 无条件建 `HiveConnectorTransaction`，而 `IcebergWritePlanProvider.currentTransaction` 把 `session.getCurrentTransaction()` 强转 `IcebergConnectorTransaction`→iceberg-on-HMS 写 CCE。加 `ConnectorWriteOps.beginTransaction(session, handle)` 默认委派无参版、`HiveConnectorMetadata` 守卫转发兄弟（S3 家族），并把已解析写目标句柄穿到引擎开事务处（`PluginDrivenInsertExecutor`/连接器 sink open 路径）。**唯一无先例的新形态**（beginTransaction 本是无句柄），依赖 W2。**W5=存储过程按句柄**：加 `Connector.getProcedureOps(handle)` 默认（null）+ `HiveConnector` 分流（hive→null 无 procedure，foreign→兄弟）；`ConnectorExecuteAction` 把句柄解析块（:134-139）上提到 `getProcedureOps` 空判（:116）+`getExecutionMode`（:123）**之前**；`ExecuteActionFactory.getSupportedActions`(:84 SHOW 发现)无 live caller、其 handle-aware 变体延后。**net-new 能力**（旧拒 iceberg-on-HMS EXECUTE，已签字升级），reorder 改失败消息序（须验无测钉旧序）。独立于写集。**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测，见分解 §2 W6）。W2+W3+W4+W6=写执行捆绑须全在翻闸前落，W1/W5 可独立提交。 + - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 - **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 From 098b97d64221cb3b83517a43968eac36eec32fad Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 16:16:21 +0800 Subject: [PATCH 081/334] [feat](catalog) HMS gateway opens per-handle transactions: iceberg-on-HMS write/EXECUTE binds the iceberg txn (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W4 of the §4.4 write delegation. The plugin-driven insert executor and the EXECUTE rewrite driver open one connector transaction unconditionally, so a flipped iceberg-on-HMS write would bind a HiveConnectorTransaction that iceberg's write plan downcasts to IcebergConnectorTransaction -> ClassCastException. Select the transaction per-handle so the gateway opens the sibling's transaction for a foreign table (the one non-precedented shape in the write delegation: beginTransaction is inherently handle-less). - ConnectorWriteOps: new default beginTransaction(session, handle) delegating to the no-arg version (mirrors getWritePlanProvider(handle)); the 6 shipped connectors are unchanged (they inherit the default -> their existing no-arg beginTransaction). - HiveConnectorMetadata: override guard-forwards a foreign (non-HiveTableHandle) handle to the sibling (-> IcebergConnectorTransaction); a hive handle falls through to the connector's own HiveConnectorTransaction. The selection MUST be symmetric -- the hive and iceberg write plans downcast the session-bound transaction to different concrete types, so an always-forward (or always-hive) shortcut breaks the opposite side. - fe-core threads the resolved write-target handle into BOTH txn-open sites (else the EXECUTE path keeps a live CCE): PluginDrivenInsertExecutor (the single funnel for INSERT/OVERWRITE/DELETE/MERGE) via a new fail-loud PluginDrivenExternalTable.resolveWriteTargetHandle(), and ConnectorRewriteDriver (EXECUTE, already holds the handle). Connector-agnostic; a null handle is never handed to the gateway (fail-loud, so it cannot misroute a plain write to the sibling). Dormant until hms enters SPI_READY_TYPES. Code-grounded recon + 4-dimension adversarial review + verify pass (CCE clean, dormancy/shipped-parity clean, routing tests mutation-tight); the two verified findings were test-coverage gaps, both closed here: a fail-loud-body test in PluginDrivenExternalTableTest (an orElse(null) downgrade now fails) and a handle-threading assertSame in PluginDrivenInsertExecutorTest (a null 2nd-arg mutant now fails). Residuals (documented, not built here): W6 write-path TCCL pin (opening the txn needs none -- IcebergConnectorTransaction ctor defers SDK/reflection to beginWrite/commit; the commit-path pin is W6); e2e regression for iceberg-on-HMS INSERT/DELETE/MERGE/EXECUTE owed at flip; hudi-on-HMS keeps a HiveTableHandle -> hive txn (later substep, not made worse); once-per-statement getTableHandle at txn-open (cache-backed for paimon/iceberg, uncached for jdbc/maxcompute) consciously accepted; fail-loud re-query slightly widens the failure window on a concurrent DROP / transient metadata error between planning and txn-open. All green: fe-connector-hive 7 (+2 begin-txn divert) + fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30 (+2) / ConnectorRewriteDriverTest 3; checkstyle 0 (api/hive/fe-core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/api/ConnectorWriteOps.java | 15 ++++++ .../connector/hive/HiveConnectorMetadata.java | 17 +++++++ ...onnectorMetadataSiblingDelegationTest.java | 47 +++++++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 17 +++++++ .../execute/ConnectorRewriteDriver.java | 2 +- .../insert/PluginDrivenInsertExecutor.java | 10 +++- .../PluginDrivenExternalTableTest.java | 37 +++++++++++++++ .../execute/ConnectorRewriteDriverTest.java | 6 ++- .../PluginDrivenInsertExecutorTest.java | 26 +++++++++- 9 files changed, 171 insertions(+), 6 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java index c5e60d8d697aa4..c421052f899392 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java @@ -91,4 +91,19 @@ default void validateStaticPartitionColumns(ConnectorSession session, ConnectorT default ConnectorTransaction beginTransaction(ConnectorSession session) { throw new DorisConnectorException("Transactions not supported"); } + + /** + * Per-table view of {@link #beginTransaction(ConnectorSession)}: opens the transaction for the connector + * that owns {@code handle}. The default ignores {@code handle} and returns the connector-level + * {@link #beginTransaction(ConnectorSession)}, so every single-format connector is unaffected. + * + *

    A heterogeneous gateway (one catalog serving multiple table formats) overrides this to route a foreign + * handle to its sibling connector's transaction, so the session-bound transaction's concrete type matches + * the per-handle-selected write plan provider. The no-arg version alone would bind the gateway's own + * transaction and the sibling's write plan would fail to downcast it. Mirrors the per-handle + * {@code getWritePlanProvider(handle)} seam.

    + */ + default ConnectorTransaction beginTransaction(ConnectorSession session, ConnectorTableHandle handle) { + return beginTransaction(session); + } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index b5b13f4ef1fd94..90f93c14401044 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1523,6 +1523,23 @@ public ConnectorTransaction beginTransaction(ConnectorSession session) { return new HiveConnectorTransaction(session.allocateTransactionId(), hmsClient, context); } + /** + * Per-handle transaction open: a FOREIGN (iceberg-on-HMS) handle forwards to the sibling so the + * session-bound transaction is the sibling's {@code IcebergConnectorTransaction} that iceberg's write plan + * downcasts; a hive handle falls through to the connector-level {@link #beginTransaction(ConnectorSession)} + * (a {@code HiveConnectorTransaction} that the hive write plan downcasts). The two write plans downcast to + * DIFFERENT concrete transaction types, so the selection MUST be symmetric — an always-forward (or + * always-hive) shortcut breaks the opposite side. The engine passes the resolved write-target handle + * (never null). Dormant until hms enters SPI_READY_TYPES. + */ + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session).beginTransaction(session, handle); + } + return beginTransaction(session); + } + /** * Drops {@code dbName.tableName} after rejecting a transactional table, mirroring legacy * {@code HiveMetadataOps.dropTableImpl}. Shared by the direct DROP TABLE and the force DROP DATABASE diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 2888ce0e944aab..a2dfc436b923ec 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -33,6 +33,8 @@ import org.apache.doris.connector.api.ddl.TagChange; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; +import org.apache.doris.connector.api.handle.NoOpConnectorTransaction; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.mvcc.ConnectorMvccPartitionView; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; @@ -238,6 +240,44 @@ public void hiveHandleAlterDdlThrowsAndValidateIsNoopAndNeverConsultsSibling() { Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle"); } + @Test + public void beginTransactionForwardsAForeignHandleToTheSibling() { + HiveConnectorMetadata md = withSibling(); + + // A foreign (iceberg-on-HMS) write must open the SIBLING's transaction, so iceberg's write plan can + // downcast the session-bound transaction to IcebergConnectorTransaction — a HiveConnectorTransaction + // (what the unconditional open would bind) would ClassCastException there. + ConnectorTransaction txn = md.beginTransaction(null, foreignHandle); + + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_TXN, txn, + "a foreign handle must open the sibling's transaction, not a hive one"); + Assertions.assertEquals(Collections.singletonList("beginTransaction"), siblingMetadata.calls, + "beginTransaction must forward the foreign handle to the sibling"); + Assertions.assertEquals(1, siblingConnector.getMetadataCount, "the sibling must be consulted once"); + } + + @Test + public void beginTransactionForHiveHandleOpensHiveTxnAndNeverConsultsSibling() { + // A hive handle must fall through to the connector-level beginTransaction. Stub the no-arg factory so the + // test does not build a real HiveConnectorTransaction (which spins a file-system thread pool); the point + // is the per-handle guard routes a hive handle to the connector's OWN transaction, a foreign one to the + // sibling. The selection must be symmetric — hive and iceberg write plans downcast to different types. + ConnectorTransaction hiveTxn = new NoOpConnectorTransaction(70099L, "HIVE"); + HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), + () -> siblingConnector) { + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session) { + return hiveTxn; + } + }; + + Assertions.assertSame(hiveTxn, md.beginTransaction(null, hiveHandle()), + "a hive handle must open the connector-level (hive) transaction, not the sibling's"); + Assertions.assertEquals(0, siblingConnector.getMetadataCount, + "a hive handle must never build/consult the iceberg sibling to open a transaction"); + Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle"); + } + private static void assertThrowsMessage(Executable exec, String expectedMessage) { DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, exec); Assertions.assertEquals(expectedMessage, e.getMessage(), @@ -263,6 +303,7 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { /** Records each forwarded method name and returns distinguishable sentinels. */ private static final class RecordingSiblingMetadata implements ConnectorMetadata { static final ConnectorTableHandle SIBLING_HANDLE = new ForeignHandle(); + static final ConnectorTransaction SIBLING_TXN = new NoOpConnectorTransaction(4243L, "ICEBERG"); static final long SENTINEL_SIZE = 4242L; static final long SENTINEL_SNAPSHOT_ID = 99L; @@ -379,6 +420,12 @@ public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, calls.add("truncateTable"); } + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session, ConnectorTableHandle handle) { + calls.add("beginTransaction"); + return SIBLING_TXN; + } + @Override public Optional getMvccPartitionView(ConnectorSession session, ConnectorTableHandle handle) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index a949cd6043e606..d9672e4ccf6548 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -34,6 +34,7 @@ import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; import org.apache.doris.connector.api.ConnectorViewDefinition; +import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; @@ -106,6 +107,22 @@ protected Optional resolveConnectorTableHandle( return metadata.getTableHandle(session, dbName, getRemoteName()); } + /** + * Resolves this table's write-target {@link ConnectorTableHandle} for the plugin-driven insert executor's + * per-handle transaction selection, failing loud if it cannot be resolved. A heterogeneous gateway needs + * the handle to open the SIBLING connector's transaction for a foreign (iceberg-on-HMS) table; a + * single-format connector ignores it (its {@code beginTransaction} defaults to the connector-level one), so + * this is byte-identical for it. Fails loud rather than returning null: a null handle is not an + * {@code instanceof} the gateway's own handle type and would misroute a plain write to the sibling. + */ + public ConnectorTableHandle resolveWriteTargetHandle() { + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + ConnectorSession session = pluginCatalog.buildConnectorSession(); + return resolveConnectorTableHandle(session, pluginCatalog.getConnector().getMetadata(session)) + .orElseThrow(() -> new DorisConnectorException( + "Cannot resolve the connector table handle for write target " + getName())); + } + /** * Returns whether the underlying connector supports multiple concurrent writers. * Used by the planner to decide GATHER (single writer) vs parallel distribution. diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java index 2cf3dc99ad9bfc..5afb272562884c 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriver.java @@ -127,7 +127,7 @@ public ConnectorProcedureResult run() throws UserException { ConnectorTransaction connectorTx; long txnId; try { - connectorTx = metadata.beginTransaction(session); + connectorTx = metadata.beginTransaction(session, tableHandle); txnId = txnManager.begin(connectorTx); } catch (DorisConnectorException e) { throw new UserException(e.getMessage(), e); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java index 6b6ceb17ce005b..557eb70eff3191 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutor.java @@ -22,9 +22,11 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorWriteOps; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.trees.plans.physical.PhysicalSink; import org.apache.doris.planner.DataSink; @@ -78,7 +80,13 @@ public void beginTransaction() { // it up by id. Connectors whose writes are auto-committed by BE (jdbc) return a degenerate // no-op transaction; maxcompute returns a real one. The connector-specific write session is // created later by planWrite (reached through finalizeSink -> bindDataSink). - connectorTx = writeOps.beginTransaction(connectorSession); + // + // Pass the resolved write-target handle so a heterogeneous gateway (e.g. an iceberg-on-HMS table served + // by the hive plugin) opens the SIBLING connector's transaction, whose concrete type its write plan + // downcasts; a single-format connector ignores the handle (the SPI default delegates to the no-arg + // beginTransaction). resolveWriteTargetHandle fails loud rather than handing the gateway a null handle. + ConnectorTableHandle writeHandle = ((PluginDrivenExternalTable) table).resolveWriteTargetHandle(); + connectorTx = writeOps.beginTransaction(connectorSession, writeHandle); txnId = ((PluginDrivenTransactionManager) transactionManager).begin(connectorTx); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index c208c1b3fadf78..9a880ee634b1a2 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -28,6 +28,7 @@ import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.ConnectorViewDefinition; +import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; @@ -143,6 +144,42 @@ public void connectorWriteCapabilitiesDegradeWhenConnectorNull() { Assertions.assertFalse(table.connectorSupportsWriteBranch(), "a null connector degrades branch to false"); } + // ==================== §4.4 W4: per-handle transaction write-target handle resolution ==================== + + // A CALLS_REAL_METHODS table whose connector resolves the write-target handle to `resolved` (null => empty). + private static PluginDrivenExternalTable writeTargetTable(ConnectorTableHandle resolved) { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(Optional.ofNullable(resolved)); + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(catalog.buildConnectorSession()).thenReturn(Mockito.mock(ConnectorSession.class)); + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); + Deencapsulation.setField(table, "catalog", catalog); + return table; + } + + @Test + public void resolveWriteTargetHandleReturnsTheConnectorResolvedHandle() { + ConnectorTableHandle handle = Mockito.mock(ConnectorTableHandle.class); + Assertions.assertSame(handle, writeTargetTable(handle).resolveWriteTargetHandle(), + "the insert executor threads THIS handle into beginTransaction(session, handle) so a heterogeneous " + + "gateway opens the sibling's transaction for a foreign table"); + } + + @Test + public void resolveWriteTargetHandleFailsLoudWhenUnresolvable() { + // FAILS LOUD rather than returning null: a null handle is not an instanceof the gateway's own handle type + // and would misroute a plain write to the sibling. A downgrade to orElse(null) must break this test. + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> writeTargetTable(null).resolveWriteTargetHandle(), + "an unresolvable write-target handle must fail loud, not return null"); + Assertions.assertTrue(e.getMessage().startsWith("Cannot resolve the connector table handle for write target"), + "the fail-loud message must name the unresolved write target"); + } + /** * Builds a CALLS_REAL_METHODS PluginDrivenExternalTable wired to a stubbed connector chain whose * write-plan provider returns {@code synthetic}. {@code writeProviderPresent=false} models a read-only diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java index cacbe033bf4998..7ac421ffb8deaf 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorRewriteDriverTest.java @@ -89,8 +89,10 @@ public void emptyPlanReturnsZeroRowWithoutOpeningTransaction() throws Exception // Single all-zero row: nothing to rewrite. Assertions.assertEquals(Collections.singletonList(Arrays.asList("0", "0", "0", "0")), result.getRows()); // MUTATION: dropping the empty-groups early return is killed — no transaction may be opened, and no - // group work scheduled, when there is nothing to rewrite. - Mockito.verify(metadata, Mockito.never()).beginTransaction(Mockito.any()); + // group work scheduled, when there is nothing to rewrite. The driver opens the txn via the per-handle + // beginTransaction(session, handle) overload, so watch that one (the single-arg matcher would go + // vacuous once the call site passes the resolved tableHandle). + Mockito.verify(metadata, Mockito.never()).beginTransaction(Mockito.any(), Mockito.any()); } @Test diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java index d54842a40b8759..074dcb9b5c38b5 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/PluginDrivenInsertExecutorTest.java @@ -29,6 +29,7 @@ import org.apache.doris.connector.api.handle.NoOpConnectorTransaction; import org.apache.doris.connector.api.write.ConnectorSinkPlan; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.planner.PluginDrivenTableSink; import org.apache.doris.thrift.TDataSink; import org.apache.doris.transaction.PluginDrivenTransactionManager; @@ -67,10 +68,18 @@ public class PluginDrivenInsertExecutorTest { public void beginTransactionOpensConnectorTxnRegistersGloballyAndStampsTxnId() { PluginDrivenInsertExecutor exec = newUnconstructedExecutor(); StubConnectorTransaction connectorTx = new StubConnectorTransaction(70001L); + FakeTxnWriteOps writeOps = new FakeTxnWriteOps(connectorTx); // Pre-seed the lazy setup so ensureConnectorSetup() short-circuits (no real catalog). Deencapsulation.setField(exec, "connectorSession", ConnectorSessionBuilder.create().build()); - Deencapsulation.setField(exec, "writeOps", new FakeTxnWriteOps(connectorTx)); + Deencapsulation.setField(exec, "writeOps", writeOps); Deencapsulation.setField(exec, "transactionManager", new PluginDrivenTransactionManager()); + // Post-W4 beginTransaction resolves the write-target handle and threads it into the per-handle + // beginTransaction overload (a heterogeneous gateway opens its sibling's transaction for a foreign + // table). Seed a table whose handle resolves. + ConnectorTableHandle writeHandle = new ConnectorTableHandle() { }; + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class); + Mockito.when(table.resolveWriteTargetHandle()).thenReturn(writeHandle); + Deencapsulation.setField(exec, "table", table); exec.beginTransaction(); @@ -83,6 +92,10 @@ public void beginTransactionOpensConnectorTxnRegistersGloballyAndStampsTxnId() { Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().getTxnById(70001L), "the connector txn must be globally registered for the BE block-allocation / " + "commit-data RPCs"); + Mockito.verify(table).resolveWriteTargetHandle(); + Assertions.assertSame(writeHandle, writeOps.handleSeenAtBegin, + "the executor must thread the RESOLVED write-target handle into the per-handle " + + "beginTransaction (a null 2nd arg would misroute a gateway write to the sibling)"); } finally { Env.getCurrentEnv().getGlobalExternalTransactionInfoMgr().removeTxnById(70001L); } @@ -210,9 +223,10 @@ private static PluginDrivenInsertExecutor newUnconstructedExecutor() { return Mockito.mock(PluginDrivenInsertExecutor.class, Mockito.CALLS_REAL_METHODS); } - /** Write ops that hand back a fixed connector transaction from beginTransaction. */ + /** Write ops that hand back a fixed connector transaction and record the handle the executor threads in. */ private static final class FakeTxnWriteOps implements ConnectorWriteOps { private final ConnectorTransaction txn; + private ConnectorTableHandle handleSeenAtBegin; private FakeTxnWriteOps(ConnectorTransaction txn) { this.txn = txn; @@ -222,6 +236,14 @@ private FakeTxnWriteOps(ConnectorTransaction txn) { public ConnectorTransaction beginTransaction(ConnectorSession session) { return txn; } + + // The executor opens the txn through the per-handle overload; capture the handle it passes so the test + // can prove the RESOLVED write-target handle is threaded (not null / not a stale one). + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session, ConnectorTableHandle handle) { + this.handleSeenAtBegin = handle; + return txn; + } } /** Captures the transaction visible on the session at the moment planWrite is invoked. */ From a5e60d1a9759630899eaa96a62282942a1788c5f Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 16:20:08 +0800 Subject: [PATCH 082/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20write-delegation=20W4=20DONE;=20next=20=3D=20W5=20(proced?= =?UTF-8?q?ures=20per-handle)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W4 (transaction per-handle) landed as 5a41e4cc562. Records the change, the two review-found test-coverage gaps closed, and the residuals (W6 pin, e2e, hudi-on-HMS, once-per-statement getTableHandle, fail-loud window). Remaining dormant write substep is W5 (procedures); W6 is flip-time verification. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c5ea569d2ac594..2181a1814be42d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -68,7 +68,8 @@ - ⚠**翻闸残留（findings 文档已详列，休眠期不动）**：①`dropDatabase` force 级联对 iceberg 表裸 `hmsClient.dropTable`（=legacy 平价、无 purge，只泄漏 iceberg 文件）→**写/DDL 委派步**；②`createTable` 引擎路由（新表无可 detect）→写/DDL 步；③`beginTransaction`/`getWritePlanProvider` **无 per-handle overload**→写委派步（新 SPI overload + ALTER-DDL/写校验，见分解文档 §6）；④`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 over-admit hudi（已注于 `HiveConnector.java:139-144`，cost-only、无 per-handle escape）；⑤`supportsLatestSnapshotPreload` under-admit iceberg/hudi（fe-core TableIf 默认 false vs legacy true，latency-only、非 enum-backed）；⑥**first-class iceberg-on-HMS 视图 = 翻闸显式非目标**（legacy 从不经 HMS 目录服务 iceberg ViewCatalog 视图；要做属新特性）。 - **§4.4 写委派权威分解 = `hms-write-delegation-decomposition-2026-07-08.md`（起步必读，recon `wf_f508ac0e-8ec` 7 维 + 完整性/分解 critic）。拆成 W1–W6 六个休眠小步，三种委派形态**：Shape A=写/存储过程 getter 加 per-handle 重载（W2/W3/W5）；Shape B=ConnectorMetadata 句柄守卫转发（W1，零新 SPI，**已 DONE**）；Shape C=事务按句柄选择（W4，唯一无先例的新形态）。**签字决策（用户 2026-07-08）：iceberg-on-HMS 升级为完整 native-iceberg 写/DELETE/MERGE/EXECUTE/ALTER 能力**（旧行为全拒=有意升级，非字节平价；对齐 Trino 按句柄委派）。**e2e 硬要求（用户 2026-07-08，memory `hms-iceberg-delegation-needs-e2e`）：每项新增 iceberg 能力（写/ALTER/procedure）最终都须补 e2e 回归（休眠单测只证路由、证不了跨加载器 cast/commit/worker-pool）；实现完成后择机统一补。** - **✅ W2 DONE（`84890fe48d3`）**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例，对 6 个现存连接器行为不变）；`HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（外来句柄永不强转，hudi 保 HiveTableHandle 留 hive）；fe-core 4 处 provider 取用点把已解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator` DELETE/MERGE+INSERT、`PhysicalIcebergMergeSink`、`PluginDrivenExternalTable.fetchSyntheticWriteColumns`；connector-agnostic 合铁律）。**对抗复审揪出 1 blocker**：4 个 fe-core mock 测只 stub 无参 `getWritePlanProvider()`，Mockito mock 不跑 default→per-handle 返 null→测挂（memory `catalog-spi-fe-core-test-infra`），已补 stub per-handle 重载（照 scan seam）；1 minor=`fetchSyntheticWriteColumns` reorder 对只读连接器（show-hidden 门）多解析一次 handle，注释已改准。全绿（新 `HiveConnectorWriteProviderDivertTest` 4 测；hive 213 + fe-core 触及套 42 测 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。 -- **下一步 = W4 或 W5**（二选一，均独立可提交）。**W4=事务按句柄**：`beginTransaction(session)` 今无句柄、`HiveConnectorMetadata:1353` 无条件建 `HiveConnectorTransaction`，而 `IcebergWritePlanProvider.currentTransaction` 把 `session.getCurrentTransaction()` 强转 `IcebergConnectorTransaction`→iceberg-on-HMS 写 CCE。加 `ConnectorWriteOps.beginTransaction(session, handle)` 默认委派无参版、`HiveConnectorMetadata` 守卫转发兄弟（S3 家族），并把已解析写目标句柄穿到引擎开事务处（`PluginDrivenInsertExecutor`/连接器 sink open 路径）。**唯一无先例的新形态**（beginTransaction 本是无句柄），依赖 W2。**W5=存储过程按句柄**：加 `Connector.getProcedureOps(handle)` 默认（null）+ `HiveConnector` 分流（hive→null 无 procedure，foreign→兄弟）；`ConnectorExecuteAction` 把句柄解析块（:134-139）上提到 `getProcedureOps` 空判（:116）+`getExecutionMode`（:123）**之前**；`ExecuteActionFactory.getSupportedActions`(:84 SHOW 发现)无 live caller、其 handle-aware 变体延后。**net-new 能力**（旧拒 iceberg-on-HMS EXECUTE，已签字升级），reorder 改失败消息序（须验无测钉旧序）。独立于写集。**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测，见分解 §2 W6）。W2+W3+W4+W6=写执行捆绑须全在翻闸前落，W1/W5 可独立提交。 +- **✅ W4 DONE（`5a41e4cc562`）= 事务按句柄（Shape C，唯一无先例的新形态）**：`ConnectorWriteOps` 加 `beginTransaction(session, handle)` 默认重载（委派无参版=对齐 `getWritePlanProvider(handle)` 先例；6 现存连接器继承默认→各自无参 `beginTransaction`，字节不变）；`HiveConnectorMetadata` 覆写守卫转发 foreign 句柄→兄弟（得 `IcebergConnectorTransaction`）、hive 句柄→落连接器自有 `HiveConnectorTransaction`——**必须对称**（hive/iceberg 写计划把 session-bound txn 分别强转不同具体类型，"总转发"/"总 hive"都炸另一侧）。fe-core 把已解析写目标句柄穿进**两个**开事务点（否则 EXECUTE 路径留活 CCE）：`PluginDrivenInsertExecutor`（INSERT/OVERWRITE/DELETE/MERGE 单一漏斗）经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()`、`ConnectorRewriteDriver`（EXECUTE，已持句柄）；connector-agnostic，**绝不给网关传 null 句柄**（fail-loud，故不会把普通写误路由兄弟）。侦察(`wf_a6e430b0-3e9`)+4维对抗复审(`wf_6af16e19-0cc`)+verify（CCE 净、休眠/现存连接器字节平价、路由测 mutation-tight）；2 确认发现均测试覆盖缺口、已补（`PluginDrivenExternalTableTest` fail-loud 体测→`orElse(null)` 降级现挂；`PluginDrivenInsertExecutorTest` 句柄穿透 assertSame→null 二参 mutant 现挂）。全绿（fe-connector-hive 7〔+2 divert〕+ fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30〔+2〕/ ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕+ import gate 净）。**W4 复审记账（residual，休眠期不动）**：①**once-per-statement `getTableHandle` 往返**——每笔 plugin 写开事务时新增一次解析（paimon/iceberg 缓存命中、jdbc/maxcompute 未缓存），有意接受、非 per-row；②fail-loud orElseThrow 开事务时重查——并发 DROP/瞬时元数据错稍扩失败窗（paimon getTableHandle 吞异常成 empty→通用文案盖根因），非正确性回退；③W6 写路径 TCCL pin=**开事务不需**（`IcebergConnectorTransaction` 构造只赋值、SDK/反射推迟到 beginWrite/commit），commit 侧 pin 归 W6；④hudi-on-HMS 保 HiveTableHandle→hive txn（未变差，后续 hudi 委派步）；⑤e2e owed（休眠单测只证路由）。 +- **下一步 = W5=存储过程按句柄**（独立于写集、可独立提交）：加 `Connector.getProcedureOps(handle)` 默认（null）+ `HiveConnector` 分流（hive→null 无 procedure，foreign→兄弟）；`ConnectorExecuteAction` 把句柄解析块（:134-139）上提到 `getProcedureOps` 空判（:116）+`getExecutionMode`（:123）**之前**；`ExecuteActionFactory.getSupportedActions`(:84 SHOW 发现)无 live caller、其 handle-aware 变体延后。**net-new 能力**（旧拒 iceberg-on-HMS EXECUTE，已签字升级），reorder 改失败消息序（须验无测钉旧序）。独立于写集。**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测，见分解 §2 W6）。**写执行捆绑 W2+W3+W4 已全落，仅余 W6（翻闸期验证）**；W1 已落，故写委派休眠小步**仅余 W5（本步，独立可提交）**，W6 属翻闸期。 - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 - **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 From 893d964b79ba0d761874ea6f18a5d7326be892b8 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 17:08:37 +0800 Subject: [PATCH 083/334] [feat](catalog) HMS gateway selects procedures per-handle: iceberg-on-HMS gains EXECUTE via sibling (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W5 of the §4.4 write/procedure/DDL delegation. Procedure-ops selection becomes per-handle, so a flipped hms gateway serves ALTER TABLE ... EXECUTE on its iceberg-on-HMS tables from the embedded iceberg sibling, while plain-hive/hudi (no procedures) stay fail-loud rejected. - Connector: new default getProcedureOps(ConnectorTableHandle) delegating to the no-arg version, mirroring the getScanPlanProvider(handle)/getWritePlanProvider(handle) seams (single-format connectors unaffected). - HiveConnector: overrides it -- a HiveTableHandle inherits the connector-level null (plain-hive/hudi have no procedures); any foreign iceberg-on-HMS handle is delegated to the iceberg sibling, passed through UNMODIFIED and NEVER cast (a cross-loader cast would CCE). - fe-core ConnectorExecuteAction: hoists the session + table-handle resolution above the getProcedureOps null-check and calls the per-handle overload (the no-arg version is null for a gateway). A bad table name now surfaces "Table not found" before "does not support EXECUTE" (documented reorder; no test pins the old order). Stale class javadoc (adapter "never constructed pre-flip") corrected -- iceberg/paimon are already live. - ExecuteActionFactory.getSupportedActions stays on the no-arg getter (no live caller; handle-aware variant deferred). Net-new capability (legacy rejected iceberg-on-HMS EXECUTE), signed off 2026-07-08; e2e owed at the flip. Dormant until hms enters SPI_READY_TYPES. Recon-grounded against HEAD + 4-dimension adversarial review (routing/precedence/CCE/completeness -> per-finding verify): 0 blocker/major/minor, 1 nit (the stale javadoc, fixed). Tests: api 3 per-handle defaults + hive 4 divert (foreign/hive/hudi/plugin-absent) + fe-core per-handle selection. Green: api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3; checkstyle 0 (api/hive/fe-core); import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/api/Connector.java | 14 ++ .../ConnectorProcedureOpsDefaultsTest.java | 66 +++++++ .../doris/connector/hive/HiveConnector.java | 19 ++ .../HiveConnectorProcedureOpsDivertTest.java | 178 ++++++++++++++++++ .../execute/ConnectorExecuteAction.java | 34 ++-- .../execute/ConnectorExecuteActionTest.java | 32 +++- 6 files changed, 330 insertions(+), 13 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index dfb891571f4fd4..a51a6a4f2533ef 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -174,6 +174,20 @@ default ConnectorProcedureOps getProcedureOps() { return null; } + /** + * Returns the procedure ops for the given table, allowing one connector to select a different set of + * procedures per table — the procedure-side analogue of {@link #getScanPlanProvider( + * ConnectorTableHandle)} / {@link #getWritePlanProvider(ConnectorTableHandle)}. + * + *

    The default ignores {@code handle} and returns the connector-level {@link #getProcedureOps()}, so every + * single-format connector is unaffected. A heterogeneous gateway connector (one catalog serving multiple + * table formats) overrides this to delegate a foreign (e.g. iceberg-on-HMS) handle to a sibling connector's + * procedure ops by the concrete (connector-defined) handle type; the engine never inspects the format.

    + */ + default ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { + return getProcedureOps(); + } + /** Returns the set of capabilities this connector supports. */ default Set getCapabilities() { return Collections.emptySet(); diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/procedure/ConnectorProcedureOpsDefaultsTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/procedure/ConnectorProcedureOpsDefaultsTest.java index b7c432003a822a..af45adba1980c5 100644 --- a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/procedure/ConnectorProcedureOpsDefaultsTest.java +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/procedure/ConnectorProcedureOpsDefaultsTest.java @@ -74,6 +74,72 @@ public void getProcedureOpsDefaultsToNull() { + "stay behaviorally unchanged)"); } + @Test + public void getProcedureOpsPerHandleDefaultsToNoArg() { + // A single-format connector overrides only the no-arg getter; the per-handle default must delegate to it + // (NOT return null), so every existing connector routes ALTER TABLE EXECUTE unchanged after the seam is + // added. MUTATION: making the default return null instead of getProcedureOps() -> non-null assert red. + ConnectorProcedureOps only = new BareProcedureOps(); + Connector connector = new Connector() { + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public ConnectorProcedureOps getProcedureOps() { + return only; + } + }; + + Assertions.assertSame(only, connector.getProcedureOps(handle()), + "the per-handle default must delegate to the connector-level no-arg procedure ops"); + } + + @Test + public void getProcedureOpsPerHandleStaysNullWhenConnectorHasNoProcedures() { + // A connector with no procedures (no-arg default returns null) must keep returning null through the + // per-handle seam, so ALTER TABLE EXECUTE is never dispatched to it. + Assertions.assertNull(new BareConnector().getProcedureOps(handle()), + "with no procedure ops at all the per-handle seam stays null"); + } + + @Test + public void getProcedureOpsPerHandleOverrideSelectsPerHandle() { + // A heterogeneous gateway overrides the per-handle seam and returns the SIBLING's ops for a foreign + // handle while a plain (hive) handle keeps the connector-level null, and must NOT fall back to the no-arg + // getter once it has a per-handle answer. MUTATION: keying the override on the no-arg getter (ignoring + // the handle) -> the foreign-handle assert reads null -> red. + ConnectorTableHandle foreignHandle = handle(); + ConnectorProcedureOps siblingOps = new BareProcedureOps(); + Connector gateway = new Connector() { + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public ConnectorProcedureOps getProcedureOps() { + return null; + } + + @Override + public ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { + return handle == foreignHandle ? siblingOps : getProcedureOps(); + } + }; + + Assertions.assertSame(siblingOps, gateway.getProcedureOps(foreignHandle), + "a gateway routes the foreign (iceberg-on-HMS) handle to the sibling's procedure ops"); + Assertions.assertNull(gateway.getProcedureOps(handle()), + "a non-foreign (plain-hive) handle keeps the connector-level null (no procedures)"); + } + + private static ConnectorTableHandle handle() { + return new ConnectorTableHandle() { + }; + } + @Test public void getExecutionModeDefaultsToSingleCall() { // A connector that declares only synchronous procedures inherits SINGLE_CALL for every name, so the diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 7d7dcb0f76f9c9..8a2bc6c12870cc 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -23,6 +23,7 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.connector.hms.HmsClient; @@ -145,6 +146,24 @@ public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle hand return getOrCreateIcebergSibling().getWritePlanProvider(handle); } + /** + * Per-table procedure ops for {@code ALTER TABLE ... EXECUTE}: a hive handle has NO procedures — it inherits + * the connector-level {@code null} (plain-hive exposes none) — while a foreign (iceberg-on-HMS) handle is + * delegated to the sibling's per-handle procedure ops (built in the iceberg plugin's classloader), so an + * iceberg-on-HMS table gains the native iceberg procedures (rollback_to_snapshot, rewrite_data_files, ...). + * The foreign handle is passed through UNMODIFIED and NEVER cast (its concrete iceberg type is invisible + * across the loader split — a cast would CCE). A HUDI table keeps a HiveTableHandle, so it too inherits the + * null (no procedures), same as plain-hive. Mirrors {@link #getWritePlanProvider(ConnectorTableHandle)}; + * dormant until hms enters SPI_READY_TYPES (nothing selects procedure ops for this connector today). + */ + @Override + public ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { + if (handle instanceof HiveTableHandle) { + return getProcedureOps(); + } + return getOrCreateIcebergSibling().getProcedureOps(handle); + } + @Override public Set getCapabilities() { // Connector-wide capabilities for the flipped hms catalog, each a faithful port of a legacy diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java new file mode 100644 index 00000000000000..05ce5863fddeb7 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java @@ -0,0 +1,178 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; +import org.apache.doris.connector.api.procedure.ConnectorProcedureResult; +import org.apache.doris.connector.api.pushdown.ConnectorPredicate; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Pins the HMS-cutover §4.4 write-delegation W5 seam: {@link HiveConnector#getProcedureOps(ConnectorTableHandle)} + * routes {@code ALTER TABLE ... EXECUTE} procedure-ops selection by the concrete handle type — a hive handle has + * no procedures (inherits the connector-level {@code null}), any foreign (iceberg-on-HMS) handle is delegated to + * the embedded iceberg sibling's per-handle procedure ops — and NEVER casts the foreign handle. The + * procedure-side twin of {@link HiveConnectorWriteProviderDivertTest} / {@link HiveConnectorScanProviderDivertTest}. + * + *

    WHY (Rule 9): post-flip an iceberg-on-HMS table gains the native iceberg procedures (rollback_to_snapshot, + * rewrite_data_files, ...), so a foreign handle must pick the SIBLING's ops; a plain-hive (or hudi-stamped hive) + * handle has none and must keep the null, so EXECUTE on it is fail-loud rejected (plain-hive exposes no + * procedures, and hudi's delegation is a later substep). The foreign handle is passed through UNMODIFIED (a + * rewrap would poison the sibling's downstream iceberg cast).

    + * + *

    Dormant until hms enters {@code SPI_READY_TYPES}: no production path selects procedure ops for this + * connector yet, so these assertions are a guard, not a live-path test.

    + */ +public class HiveConnectorProcedureOpsDivertTest { + + private static final String METASTORE_URI = "thrift://host:9083"; + + /** The foreign (non-hive) handle the iceberg sibling's getTableHandle produces post-flip. */ + private static final class ForeignHandle implements ConnectorTableHandle { + } + + @Test + public void foreignHandleDelegatesToSiblingProcedureOpsUnmodified() { + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(props(), context); + ForeignHandle foreign = new ForeignHandle(); + + ConnectorProcedureOps ops = connector.getProcedureOps(foreign); + + Assertions.assertSame(sibling.ops, ops, + "a foreign handle must return the iceberg sibling's OWN procedure ops"); + Assertions.assertSame(foreign, sibling.lastHandle, + "the foreign handle must reach the sibling's per-handle selector UNMODIFIED (a rewrap would " + + "poison the downstream iceberg cast)"); + Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once and consulted"); + } + + @Test + public void hiveHandleHasNoProceduresWithoutConsultingSibling() { + // A hive handle inherits the connector-level null (plain-hive has no procedures). It must NOT build or + // consult the sibling, and — unlike the write/scan seams — it needs no HmsClient (no-arg getProcedureOps + // returns null directly). + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(props(), context); + HiveTableHandle hive = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE).build(); + + Assertions.assertNull(connector.getProcedureOps(hive), + "a hive handle has no procedures — it inherits the connector-level null"); + Assertions.assertEquals(0, context.buildCount, "a hive handle must never build or consult the sibling"); + Assertions.assertNull(sibling.lastHandle, "the sibling's procedure ops must not be consulted for a hive handle"); + } + + @Test + public void hudiStampedHiveHandleHasNoProcedures() { + // The route keys on the JVM handle TYPE (HiveTableHandle), not the format enum: a HUDI-stamped hive handle + // is still a HiveTableHandle, so it inherits the null (no procedures) — hudi delegation is a later substep. + RecordingSibling sibling = new RecordingSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(props(), context); + HiveTableHandle hudi = new HiveTableHandle.Builder("db", "t", HiveTableType.HUDI).build(); + + Assertions.assertNull(connector.getProcedureOps(hudi), + "a hudi-stamped hive handle inherits the connector-level null (no procedures)"); + Assertions.assertEquals(0, context.buildCount, "a hudi table must NOT be diverted to the iceberg sibling"); + } + + @Test + public void foreignHandleFailsLoudWhenIcebergPluginAbsent() { + // The seam returns a null sibling when the iceberg plugin is absent; selecting procedure ops for a + // foreign handle must fail loud (naming the catalog), not NPE. + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(props(), context); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> connector.getProcedureOps(new ForeignHandle()), + "a foreign handle with no iceberg plugin must fail loud"); + Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); + } + + private static Map props() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", METASTORE_URI); + return props; + } + + /** Records the {@code createSiblingConnector} call and returns a configurable (possibly null) sibling. */ + private static final class RecordingSiblingContext extends FakeConnectorContext { + int buildCount; + Connector siblingToReturn; + + RecordingSiblingContext(Connector siblingToReturn) { + this.siblingToReturn = siblingToReturn; + } + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + buildCount++; + return siblingToReturn; + } + } + + /** A sibling {@link Connector} whose per-handle procedure ops are a distinguishable marker, recording the handle. */ + private static final class RecordingSibling implements Connector { + final ConnectorProcedureOps ops = new MarkerProcedureOps(); + ConnectorTableHandle lastHandle; + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { + lastHandle = handle; + return ops; + } + + @Override + public void close() { + } + } + + /** A bare procedure-ops stand-in; only its identity matters here. */ + private static final class MarkerProcedureOps implements ConnectorProcedureOps { + @Override + public List getSupportedProcedures() { + return Collections.emptyList(); + } + + @Override + public ConnectorProcedureResult execute(ConnectorSession session, ConnectorTableHandle table, + String procedureName, Map properties, ConnectorPredicate whereCondition, + List partitionNames) { + return null; + } + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java index 998e19c0ef5204..2c5675b7858616 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteAction.java @@ -74,9 +74,12 @@ * unchecked {@link DorisConnectorException}; this adapter converts it to a {@code UserException} so * {@code ExecuteActionCommand.run()} re-wraps it with the legacy {@code "Failed to execute action:"} prefix.

    * - *

    Dormant pre-cutover. Iceberg tables are {@code IcebergExternalTable} (not - * {@code PluginDrivenExternalTable}) until they enter {@code SPI_READY_TYPES} at P6.6, so this adapter is - * never constructed pre-flip — live {@code ALTER TABLE EXECUTE} still routes to the legacy fe-core actions.

    + *

    Live for the flipped SPI catalogs; per-handle for a gateway. Iceberg and paimon are already in + * {@code SPI_READY_TYPES}, so their tables are {@code PluginDrivenExternalTable}s and {@code ALTER TABLE EXECUTE} + * on them routes through this adapter today. Procedure ops are selected {@link Connector#getProcedureOps( + * org.apache.doris.connector.api.handle.ConnectorTableHandle) per-handle}: a single-format connector just returns + * its connector-level ops, but a flipped {@code hms} gateway (still dormant — not yet in {@code SPI_READY_TYPES}) + * exposes none at the connector level and diverts a foreign iceberg-on-HMS handle to its iceberg sibling.

    */ public class ConnectorExecuteAction implements ExecuteAction { @@ -113,7 +116,22 @@ public void validate(TableNameInfo tableNameInfo, UserIdentity currentUser) thro public ResultSet execute(TableIf ignored) throws UserException { PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); Connector connector = catalog.getConnector(); - ConnectorProcedureOps procedureOps = connector.getProcedureOps(); + + // Resolve the connector session + the target table handle FIRST, so procedure-ops selection is + // PER-HANDLE. A single-format connector's per-handle getProcedureOps(handle) just returns its + // connector-level ops, but a flipped hms GATEWAY exposes no connector-level procedures + // (getProcedureOps() == null) while its iceberg-on-HMS tables do — getProcedureOps(handle) diverts a + // foreign handle to the iceberg sibling, and a plain-hive handle keeps the null. Both dispatch arms + // (single-call and distributed) also need the session/handle. Resolving the handle first means a bad + // table name now surfaces "Table not found" before "does not support EXECUTE". + ConnectorSession session = catalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + ConnectorTableHandle tableHandle = metadata + .getTableHandle(session, table.getRemoteDbName(), table.getRemoteName()) + .orElseThrow(() -> new AnalysisException("Table not found: " + table.getRemoteDbName() + + "." + table.getRemoteName() + " in catalog " + catalog.getName())); + + ConnectorProcedureOps procedureOps = connector.getProcedureOps(tableHandle); if (procedureOps == null) { throw new DdlException("Connector '" + catalog.getName() + "' (type: " + catalog.getType() + ") does not support EXECUTE actions"); @@ -129,14 +147,6 @@ public ResultSet execute(TableIf ignored) throws UserException { throw new DdlException("WHERE condition is not supported for this EXECUTE action"); } - // Resolve the shared connector prerequisites — both dispatch arms (single-call and distributed) need - // the session, the resolved table handle and the partition names. - ConnectorSession session = catalog.buildConnectorSession(); - ConnectorMetadata metadata = connector.getMetadata(session); - ConnectorTableHandle tableHandle = metadata - .getTableHandle(session, table.getRemoteDbName(), table.getRemoteName()) - .orElseThrow(() -> new AnalysisException("Table not found: " + table.getRemoteDbName() - + "." + table.getRemoteName() + " in catalog " + catalog.getName())); List partitionNames = partitionNamesInfo .map(PartitionNamesInfo::getPartitionNames).orElse(Collections.emptyList()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java index 6545342193e2d9..7af1108f31c7af 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/execute/ConnectorExecuteActionTest.java @@ -279,7 +279,9 @@ public void executeReWrapsConnectorExceptionAsUserExceptionWithSameMessage() { @Test public void executeThrowsWhenConnectorExposesNoProcedureOps() { Fixture f = new Fixture(); - Mockito.when(f.connector.getProcedureOps()).thenReturn(null); + // The dispatch selects the ops PER-HANDLE, so a connector with no procedures for this table returns null + // from the per-handle overload (a plain-hive handle in a flipped hms gateway). + Mockito.when(f.connector.getProcedureOps(Mockito.any())).thenReturn(null); ConnectorExecuteAction action = new ConnectorExecuteAction("rollback_to_snapshot", f.props, Optional.empty(), Optional.empty(), f.table); @@ -288,6 +290,31 @@ public void executeThrowsWhenConnectorExposesNoProcedureOps() { "A connector with no procedure ops must fail loud, mirroring the write-path null-provider throw"); } + @Test + public void executeSelectsProcedureOpsByResolvedHandleNotNoArg() { + // W5 gateway scenario: a flipped hms gateway exposes NO connector-level procedures — getProcedureOps() is + // null — but its iceberg-on-HMS table diverts to the sibling's ops via getProcedureOps(handle). Pin that + // the dispatch consults the PER-HANDLE overload with the resolved handle: with the no-arg getter null but + // the per-handle overload wired to the ops, EXECUTE must still run. MUTATION: reverting the dispatch to + // the no-arg getProcedureOps() -> it reads null -> throws "does not support EXECUTE" -> this test red + // (that mutation would wrongly reject every iceberg-on-HMS EXECUTE at the flip). + Fixture f = new Fixture(); + Mockito.when(f.connector.getProcedureOps()).thenReturn(null); + Mockito.when(f.procedureOps.execute(Mockito.any(), Mockito.any(), Mockito.anyString(), + Mockito.anyMap(), Mockito.any(), Mockito.anyList())) + .thenReturn(twoColumnResult(Arrays.asList("100", "200"))); + + ConnectorExecuteAction action = new ConnectorExecuteAction("rollback_to_snapshot", + f.props, Optional.empty(), Optional.empty(), f.table); + Assertions.assertDoesNotThrow(() -> action.execute(f.table)); + + // The ops were selected by the RESOLVED handle, and the body ran with that same handle threaded through. + Mockito.verify(f.connector).getProcedureOps(Mockito.eq(f.handle)); + Mockito.verify(f.procedureOps).execute(Mockito.eq(f.session), Mockito.eq(f.handle), + Mockito.eq("rollback_to_snapshot"), Mockito.eq(f.props), + Mockito.isNull(), Mockito.eq(Collections.emptyList())); + } + @Test public void executeThrowsWhenTableHandleMissing() { Fixture f = new Fixture(); @@ -471,6 +498,9 @@ private static final class Fixture { props.put("snapshot_id", "200"); Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); Mockito.when(connector.getProcedureOps()).thenReturn(procedureOps); + // execute() selects the ops per-handle (getProcedureOps(handle)); a Mockito mock does NOT run the + // default method, so the per-handle overload must be stubbed explicitly or it returns null. + Mockito.when(connector.getProcedureOps(Mockito.any())).thenReturn(procedureOps); Mockito.when(metadata.getTableHandle(Mockito.any(ConnectorSession.class), Mockito.anyString(), Mockito.anyString())).thenReturn(Optional.of(handle)); From 970274cac966043c3f3d9a032259c3c17f3e98dd Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 17:10:59 +0800 Subject: [PATCH 084/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20=C2=A7?= =?UTF-8?q?4.4=20write-delegation=20W5=20DONE;=20read+write=20delegation?= =?UTF-8?q?=20spine=20(S0-S6=20+=20W1-W5)=20complete?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit W5 (procedures per-handle) landed as 8aee62bab20. Records the W5 residuals (paimon extra getTableHandle round-trip, getSupportedActions deferred, e2e owed) and marks the whole §4.4 read+write+procedure+DDL delegation spine done. Next = remaining pre-flip dormant items (candidates: §4.5 write-chain, hudi-on-HMS write-reject, §4.3 getNewestUpdateVersionOrTime[blocked on D2 cache]); W6 TCCL pin is flip-time. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 2181a1814be42d..4c431e7d01ac0f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -69,7 +69,13 @@ - **§4.4 写委派权威分解 = `hms-write-delegation-decomposition-2026-07-08.md`（起步必读，recon `wf_f508ac0e-8ec` 7 维 + 完整性/分解 critic）。拆成 W1–W6 六个休眠小步，三种委派形态**：Shape A=写/存储过程 getter 加 per-handle 重载（W2/W3/W5）；Shape B=ConnectorMetadata 句柄守卫转发（W1，零新 SPI，**已 DONE**）；Shape C=事务按句柄选择（W4，唯一无先例的新形态）。**签字决策（用户 2026-07-08）：iceberg-on-HMS 升级为完整 native-iceberg 写/DELETE/MERGE/EXECUTE/ALTER 能力**（旧行为全拒=有意升级，非字节平价；对齐 Trino 按句柄委派）。**e2e 硬要求（用户 2026-07-08，memory `hms-iceberg-delegation-needs-e2e`）：每项新增 iceberg 能力（写/ALTER/procedure）最终都须补 e2e 回归（休眠单测只证路由、证不了跨加载器 cast/commit/worker-pool）；实现完成后择机统一补。** - **✅ W2 DONE（`84890fe48d3`）**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例，对 6 个现存连接器行为不变）；`HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（外来句柄永不强转，hudi 保 HiveTableHandle 留 hive）；fe-core 4 处 provider 取用点把已解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator` DELETE/MERGE+INSERT、`PhysicalIcebergMergeSink`、`PluginDrivenExternalTable.fetchSyntheticWriteColumns`；connector-agnostic 合铁律）。**对抗复审揪出 1 blocker**：4 个 fe-core mock 测只 stub 无参 `getWritePlanProvider()`，Mockito mock 不跑 default→per-handle 返 null→测挂（memory `catalog-spi-fe-core-test-infra`），已补 stub per-handle 重载（照 scan seam）；1 minor=`fetchSyntheticWriteColumns` reorder 对只读连接器（show-hidden 门）多解析一次 handle，注释已改准。全绿（新 `HiveConnectorWriteProviderDivertTest` 4 测；hive 213 + fe-core 触及套 42 测 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。 - **✅ W4 DONE（`5a41e4cc562`）= 事务按句柄（Shape C，唯一无先例的新形态）**：`ConnectorWriteOps` 加 `beginTransaction(session, handle)` 默认重载（委派无参版=对齐 `getWritePlanProvider(handle)` 先例；6 现存连接器继承默认→各自无参 `beginTransaction`，字节不变）；`HiveConnectorMetadata` 覆写守卫转发 foreign 句柄→兄弟（得 `IcebergConnectorTransaction`）、hive 句柄→落连接器自有 `HiveConnectorTransaction`——**必须对称**（hive/iceberg 写计划把 session-bound txn 分别强转不同具体类型，"总转发"/"总 hive"都炸另一侧）。fe-core 把已解析写目标句柄穿进**两个**开事务点（否则 EXECUTE 路径留活 CCE）：`PluginDrivenInsertExecutor`（INSERT/OVERWRITE/DELETE/MERGE 单一漏斗）经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()`、`ConnectorRewriteDriver`（EXECUTE，已持句柄）；connector-agnostic，**绝不给网关传 null 句柄**（fail-loud，故不会把普通写误路由兄弟）。侦察(`wf_a6e430b0-3e9`)+4维对抗复审(`wf_6af16e19-0cc`)+verify（CCE 净、休眠/现存连接器字节平价、路由测 mutation-tight）；2 确认发现均测试覆盖缺口、已补（`PluginDrivenExternalTableTest` fail-loud 体测→`orElse(null)` 降级现挂；`PluginDrivenInsertExecutorTest` 句柄穿透 assertSame→null 二参 mutant 现挂）。全绿（fe-connector-hive 7〔+2 divert〕+ fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30〔+2〕/ ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕+ import gate 净）。**W4 复审记账（residual，休眠期不动）**：①**once-per-statement `getTableHandle` 往返**——每笔 plugin 写开事务时新增一次解析（paimon/iceberg 缓存命中、jdbc/maxcompute 未缓存），有意接受、非 per-row；②fail-loud orElseThrow 开事务时重查——并发 DROP/瞬时元数据错稍扩失败窗（paimon getTableHandle 吞异常成 empty→通用文案盖根因），非正确性回退；③W6 写路径 TCCL pin=**开事务不需**（`IcebergConnectorTransaction` 构造只赋值、SDK/反射推迟到 beginWrite/commit），commit 侧 pin 归 W6；④hudi-on-HMS 保 HiveTableHandle→hive txn（未变差，后续 hudi 委派步）；⑤e2e owed（休眠单测只证路由）。 -- **下一步 = W5=存储过程按句柄**（独立于写集、可独立提交）：加 `Connector.getProcedureOps(handle)` 默认（null）+ `HiveConnector` 分流（hive→null 无 procedure，foreign→兄弟）；`ConnectorExecuteAction` 把句柄解析块（:134-139）上提到 `getProcedureOps` 空判（:116）+`getExecutionMode`（:123）**之前**；`ExecuteActionFactory.getSupportedActions`(:84 SHOW 发现)无 live caller、其 handle-aware 变体延后。**net-new 能力**（旧拒 iceberg-on-HMS EXECUTE，已签字升级），reorder 改失败消息序（须验无测钉旧序）。独立于写集。**W6** 写路径 TCCL pin=翻闸期验证（非休眠可测，见分解 §2 W6）。**写执行捆绑 W2+W3+W4 已全落，仅余 W6（翻闸期验证）**；W1 已落，故写委派休眠小步**仅余 W5（本步，独立可提交）**，W6 属翻闸期。 +- **✅ W5 DONE（`8aee62bab20`）= 存储过程按句柄**（详见上 🎯 段 W5 条目）。`Connector.getProcedureOps(handle)` per-handle 默认 + `HiveConnector` 分流（hive/hudi→继承 null、foreign→兄弟永不强转）+ fe-core `ConnectorExecuteAction` reorder（句柄解析上提、改调 per-handle）。net-new 升级、reorder 无测钉旧序、顺带修陈旧 javadoc。4 维对抗复审 0 blocker/major/minor + 1 nit（已修）。**⇒ §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官。** + - ⚠**W5 复审记账（休眠期不动）**：①**paimon/无-procedure 连接器 EXECUTE 现多一次 `getTableHandle` 往返**（reorder 先解析句柄再判 null-ops；happy-path iceberg 无新增往返，只错误路径先报 "Table not found"）——有意接受、非正确性回退；②`ExecuteActionFactory.getSupportedActions` 留无参 getter（SHOW 发现，无 live caller；翻闸对 hms 网关会返 hive 连接器级空——handle-aware 变体延后，无消费者故无害）；③e2e owed（休眠单测只证路由；随其它 iceberg-on-HMS 能力翻闸期统一补，memory `hms-iceberg-delegation-needs-e2e`）。 +- **下一步 = 写/procedure/DDL 委派脊柱（读 S0–S6 + 写 W1–W5）已全部收官，转入剩余翻闸前休眠项。候选（择一起步，均独立可提交、均线上零影响；建议先与用户确认优先级）**： + 1. **§4.5 写链收尾**（读-ACID query-finish commit 接线、`BindSink` 显式分区 INSERT 拒 port 进插件、`Env.hiveTransactionMgr` 搬家、`BIND_BROKER_NAME` 搬家、`pluginCatalogTypeToEngine` 无 `hms` 的 engine-map）——延续写路径，为翻闸减负。 + 2. **hudi-on-HMS 写-拒安全网**（hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi）——hudi flip-blocker，防翻闸把只读 hudi-on-HMS 变成可写-as-hive。 + 3. **§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，hive 走 max 分区时间）——**被 D2 连接器缓存阻塞**（探针高频，须先做连接器缓存），故非即时可起。 + （W6 写路径 TCCL pin 属**翻闸期验证**、非休眠可测，见分解 §2 W6。e2e 回归亦翻闸期统一补。） - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 - **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 From 8217d0c476f4ae7f859f3750929bc625cf93eee2 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 17:52:00 +0800 Subject: [PATCH 085/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20write-?= =?UTF-8?q?chain=20(=C2=A74.5)=20decomposition:=204=20dormant=20sub-steps?= =?UTF-8?q?=20(WC1-WC4)=20+=20flip=20residuals?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Code-grounded recon (3 dimension readers + completeness/ordering critics, all HEAD-verified) of the remaining pre-flip write-chain items. Key output: a three-way dormancy taxonomy (A=unreachable, B=value-identical, C=live-with-no-op-default) and a risk-ascending order -- WC1 relocate BIND_BROKER_NAME (B), WC2 engine-map hms cases (A), WC3 partition-spec INSERT reject (C, needs live-connector no-regression test), WC4 read-ACID query-finish commit wiring (C, the hard flip prerequisite). Items 2b/3 carry no dormant work (flip-time deletes R1/R2). Authoritative plan for the write-chain build-out. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- ...ms-write-chain-decomposition-2026-07-08.md | 579 ++++++++++++++++++ 1 file changed, 579 insertions(+) create mode 100644 plan-doc/tasks/hms-write-chain-decomposition-2026-07-08.md diff --git a/plan-doc/tasks/hms-write-chain-decomposition-2026-07-08.md b/plan-doc/tasks/hms-write-chain-decomposition-2026-07-08.md new file mode 100644 index 00000000000000..c898b321cc489e --- /dev/null +++ b/plan-doc/tasks/hms-write-chain-decomposition-2026-07-08.md @@ -0,0 +1,579 @@ +# HMS Cutover — Write-Chain Decomposition (design §4.5) + +**Date:** 2026-07-08 · **Branch:** `catalog-spi-11-hive` · **HEAD:** `75a5d25d746` (W5 done) +**Scope:** decompose the *remaining* pre-flip write-chain items of the HMS-catalog cutover into ordered, +independently-committable **dormant** sub-steps. Planning only — no edits in this doc. + +All line numbers below were re-verified against HEAD `75a5d25d746`. Where the authoritative design doc +(`hms-cutover-retype-design-2026-07-07.md` §4.5) disagrees with HEAD, HEAD wins. + +--- + +## 0. Scope and what is already DONE + +### 0.1 The cutover model (context) +An HMS external catalog (Hive + Iceberg + Hudi tables) is migrating from fe-core's built-in implementation to a +plugin "connector". A flipped `hms` catalog is served by the **HIVE plugin as a GATEWAY**; iceberg-on-HMS tables +delegate to a sibling `IcebergConnector`. The **single flip line** is adding `"hms"` to +`CatalogFactory.SPI_READY_TYPES` (CatalogFactory.java:49-50 — today `{"jdbc","es","trino-connector","max_compute", +"paimon","iceberg"}`, **no `"hms"`**). That flip is **NOT part of this step**. + +**"Dormant" is not one thing.** Three distinct risk classes appear below; the ordered plan (§2) keeps them separate: + +- **Class A — strictly unreachable pre-flip.** New `switch` cases keyed on `getType()=="hms"`; no + `PluginDrivenExternalCatalog` has `getType()=="hms"` until the flip, so the code is literally never entered. + *(Items 5, 5b.)* +- **Class B — live code, value-identical.** Edits a live method but the resolved value / bytes are provably + unchanged for every catalog. *(Item 4.)* +- **Class C — live-exercised, behavior-preserving via a no-op SPI default.** The new fe-core wiring runs **today** + on already-flipped live connectors (es/jdbc/paimon/max_compute/iceberg), relying on a permissive/no-op SPI + default to be inert. Not byte-unchanged; carries live-connector regression surface and **requires a + live-connector no-regression test**, not merely a dormant hive unit test. *(Items 1, 2a.)* + +### 0.2 The delegation spine — already landed and dormant +The read + write + procedure + DDL **delegation** spine is complete on HEAD and dormant: + +- **Read** (scan) delegation via `HiveConnector.getScanPlanProvider(handle)` (per-handle gateway routing to the + iceberg sibling for foreign handles). +- **Write** delegation: the plain-hive write path is already class-forked + (`UnboundTableSinkCreator`/`BindSink`/`PhysicalPlanTranslator`); at the flip it falls through to the live + `UnboundConnectorTableSink → PluginDrivenTableSink → PluginDrivenInsertExecutor` driven by the dormant + `HiveWritePlanProvider`/`HiveConnectorTransaction`. +- **Procedure + write-admission per-handle** delegation (W1–W5, commits `d1df7df`/`5a48c`/`8aee2`/`75a5d`): + `HiveConnectorMetadata.siblingMetadata(...)` forwards foreign (iceberg-on-HMS) handles to the sibling; hive + handles get hive semantics. **This plumbing is present at HEAD** and is a prerequisite that item 2a leans on. + +### 0.3 The five §4.5 write-chain items (this decomposition) +The §4.5 item set is **complete** and matches the design doc verbatim (`hms-cutover-retype-design-2026-07-07.md`: +78-83). No §4.5 item was missed. This doc covers all five, **plus a sixth (item 5b)** that verifying item 5 +surfaces as a firm dormant sub-step (not the "soft follow-up" earlier recon framed it as): + +| # | Item | Class | Remaining pre-flip work? | +|---|------|-------|--------------------------| +| 1 | Read-ACID query-finish commit wiring | C (live, no-op default) | **Yes** — new SPI + fe-core wiring | +| 2a | Reject explicit-partition-spec INSERT | C (live, no-op default) | **Yes** — new SPI + wiring + override | +| 2b | Reject LZO read-only | — | **No dormant work** — done in connector; only a flip-time delete | +| 3 | Delete `Env.hiveTransactionMgr` | — | **No** — flip/delete-time only | +| 4 | Relocate `BIND_BROKER_NAME` | B (value-identical) | **Yes** — constant refactor | +| 5 | `pluginCatalogTypeToEngine` `"hms"→ENGINE_HIVE` | A (unreachable) | **Yes** — one switch case | +| 5b | `PluginDrivenExternalTable` engine-display `"hms"` cases | A (unreachable) | **Yes** — two switch cases (merge into 5) | + +--- + +## 1. Items in detail + +### Item 1 — READ-ACID query-finish commit wiring — **dormant-committable-now (Class C)** + +**HEAD state (unwired — CONFIRMED).** +- Plugin **opens** but never **releases**: `fe-connector-hive/.../HiveScanPlanProvider.java:175-177` + `planAcidScan` does `HiveReadTransaction txn = new HiveReadTransaction(session.getQueryId(), session.getUser(), + ...); readTxnManager.register(txn);`. `register()` (HiveReadTransactionManager.java:47-50) calls `txn.begin()` + (opens the metastore txn) + `txnMap.put(txn.getQueryId(), txn)`. +- The matching release **exists but has zero callers**: `HiveReadTransactionManager.java:52-61` + `public void deregister(String queryId){ HiveReadTransaction txn = txnMap.remove(queryId); if(txn!=null){ try + { txn.commit(); } catch (RuntimeException e) { LOG.warn(...); } } }` — `commit()` → `hmsClient.commitTxn`, + releasing the shared read lock. **Note:** commit failure is *logged and swallowed*, not propagated (verified + HEAD :55-59) — the release is best-effort, **not** "fail-loud". +- The SPI has **no** release method: `ConnectorScanPlanProvider` (fe-connector-api) — no + `releaseReadTransaction`. Default-no-op precedent already exists there (`classifyColumn`, `getDeleteFiles`). +- fe-core query-finish spine **already present and generic**: `QueryFinishCallbackRegistry` + + `QeProcessor.registerQueryFinishCallback` (QeProcessor.java:46 / QeProcessorImpl.java:216-217) invoked inside + `unregisterQuery` via `runAndClear(DebugUtil.printId(queryId))` (QeProcessorImpl.java:212). **Nothing on the + scan side registers a callback today.** No new fe-core lifecycle infra is needed. +- Legacy model to mirror: `HiveScanNode.java:138-149` + `Env.getCurrentHiveTransactionMgr().register(hiveTransaction); ... QeProcessorImpl.INSTANCE + .registerQueryFinishCallback(txnQueryId, () -> Env.getCurrentHiveTransactionMgr().deregister(txnQueryId));`. + +**Touch points (three changes).** +1. **NEW SPI** — `ConnectorScanPlanProvider` (fe-connector-api): add + `default void releaseReadTransaction(String queryId) { }` (no-op default, matching the existing + `classifyColumn`/`getDeleteFiles` default-no-op pattern). +2. **Connector impl** — `HiveScanPlanProvider`: + `@Override public void releaseReadTransaction(String queryId){ readTxnManager.deregister(queryId); }`. + `readTxnManager` (HiveScanPlanProvider.java:86) is the *same* per-connector instance + `HiveConnector.java:79/104` injects into every provider, so `register` (planAcidScan) and `deregister` route + to one manager. +3. **fe-core wiring** — `PluginDrivenScanNode.getSplits()`: after `resolveScanProvider()` returns non-null + (i.e. after the null-guard at ~:911-915, and it must **dominate** `planScan` at ~:965 — register **before** + the `requiredPartitions`-empty short-circuit ~:938), register **unconditionally**: + `String queryId = connectorSession.getQueryId(); QeProcessorImpl.INSTANCE.registerQueryFinishCallback(queryId, + () -> onPluginClassLoader(scanProvider, () -> { scanProvider.releaseReadTransaction(queryId); return null; + }));`. Reuse the existing private-static `onPluginClassLoader` helper (PluginDrivenScanNode.java:516-524) for + the TCCL pin; add an import of `org.apache.doris.qe.QeProcessorImpl`. + +**Why the single `getSplits` site is leak-proof.** Hive can *never* enter connector batch mode: +`HiveScanPlanProvider` overrides neither `supportsBatchScan` (defaults `false`) nor +`planScanForPartitionBatch`, so the batch entry point (`PluginDrivenScanNode.startSplit → +planScanForPartitionBatch`) can never open a hive read txn. `register(txn)` is reachable **only** via +`getSplits → planScan`. Register **before** `planScan` so a `planScan` that opens the metastore txn and then +throws still has its release callback in place (`deregister` on an absent `queryId` is a safe no-op). + +**queryId identity invariant (single string, no conversion).** +`ConnectorSessionBuilder.java:57` (`org.apache.doris.connector`, **not** `datasource`) sets +`b.queryId = ctx.queryId() != null ? DebugUtil.printId(ctx.queryId()) : ""`. So +`session.getQueryId()` == `DebugUtil.printId(...)` == the `runAndClear` key (QeProcessorImpl.java:212) == the +`txnMap` key (`txn.getQueryId()`). One string serves the registry key and the manager key. + +**Traps.** +- **T1 (TCCL — load-bearing).** The callback runs on the `StmtExecutor` thread (unregisterQuery callers + StmtExecutor.java:1035/2188), whose TCCL is the fe-core `app` loader, **not** the plugin loader. + `deregister → txn.commit → hmsClient.commitTxn` dispatches into plugin/hive-metastore/thrift code that + resolves classes by name via TCCL (documented split-brain hazard). The callback **must** wrap + `releaseReadTransaction` in `onPluginClassLoader(scanProvider,...)`; omitting the pin risks + ClassCast/NoClassDef at commit time on a live hive query. +- **T2 (live-but-inert on already-flipped connectors — Class C).** The fe-core registration is + **unconditional**, so it runs **today** for every live plugin scan (es/jdbc/paimon/max_compute/iceberg). For + them the callback pins TCCL and calls the no-op default `releaseReadTransaction` — functionally inert but + genuinely exercised. This is why item 1 is behavior-preserving yet **not strictly dormant**: the default + no-op must never throw; per-query cost is one extra `CopyOnWriteArrayList` entry. **Do NOT** gate it with any + `if(hive)`/`instanceof`/`isTransactional` branch (IRON RULE). +- **T3 (connector-agnostic gate).** `HiveConnector.getScanPlanProvider(handle)` (HiveConnector.java:120-124) + returns the **iceberg sibling** provider for a foreign handle, whose default `releaseReadTransaction` is a + no-op — no hive read txn is opened for iceberg-on-HMS, so nothing to release. +- **T4 (pre-existing multi-ACID-table parity limitation — do NOT fix here).** `HiveReadTransactionManager` keys + `txnMap` by `queryId`; a query joining two full-ACID hive tables runs `planAcidScan` twice with the same + `session.getQueryId()`, so the 2nd `register()` overwrites the 1st and that 1st metastore txn leaks. Legacy + `HiveTransactionMgr` has identical keying — the plugin faithfully preserves parity. Out of scope for item 1. +- **T5 (id scoping).** The release must key on `session.getQueryId()` (== `DebugUtil.printId`). A future + refactor passing a raw `TUniqueId` string would make `runAndClear` never match (silent lock leak) or + `deregister` miss the `txnMap` key. Keep the single-string invariant. + +**Proving test (two, both provable now).** +1. **Connector** (fe-connector-hive, mirror `HiveReadTransactionTest` with its fake `HmsClient`): drive + `planAcidScan → register` (assert `openTxn` fired, shared lock acquired), call + `HiveScanPlanProvider.releaseReadTransaction(queryId)`, assert `commitTxn(txnId)` fired **exactly once** and a + second call is a no-op (idempotent `txnMap.remove`). Encodes WHY: an unreleased read txn leaks the metastore + shared lock. **Do not** assert an exception surfaces on commit failure — HEAD swallows+logs it (T-above). +2. **fe-core** (mirror `PluginDrivenScanNode…SelectionTest`/`MvccPinTest` Mockito style): stub + `resolveScanProvider()` to a Mockito `ConnectorScanPlanProvider`, run `getSplits()`, capture the `Runnable` + passed to a mocked `QeProcessorImpl.registerQueryFinishCallback` (verify registered with + `session.getQueryId()` and **before** `planScan`), invoke it, verify `releaseReadTransaction(queryId)` is + called with TCCL pinned to the provider's classloader. `QueryFinishCallbackRegistryTest` already proves + `runAndClear` fires+clears per-query, so registry semantics need no new test. **Add**: a live-connector + no-regression check (paimon/iceberg scan still green) since the registration is unconditional. + +**Deps / risk.** Independent of items 2/4/5. **Coupled to item 3** as the two halves of the read cutover: +item 1 wires the NEW plugin-side release; item 3 DELETES the legacy `Env.hiveTransactionMgr` + legacy +`HiveScanNode`. Item 1 is a **hard pre-flip prerequisite**: without it, flipped ACID reads register a metastore +txn (HiveScanPlanProvider.java:177) that is never committed → **leaked shared read lock**. No ordering +constraint vs the flip other than "item 1 before flip". + +--- + +### Item 2a — reject explicit-partition-spec INSERT — **dormant-committable-now (Class C)** + +**HEAD state (NOT ported — this is the real remaining pre-check).** +`BindSink.java:667-669` (inside `bindHiveTableSink`, the legacy `UnboundHiveTableSink` path): +`if (!sink.getPartitions().isEmpty()) { throw new AnalysisException("Not support insert with partition spec in +hive catalog."); }`. `sink.getPartitions()` (`UnboundBaseExternalTableSink:69`) is the **dynamic partition-NAME +list** from `PARTITION(p1,p2)` — a *different* field from the static `PARTITION(col='val')` spec +(`staticPartitionKeyValues`). + +On the flip, `CatalogFactory` makes the hms catalog a `PluginDrivenExternalCatalog`, so +`UnboundTableSinkCreator` builds `UnboundConnectorTableSink` and binding goes through **`bindConnectorTableSink`** +(BindSink.java:758-850). Verified at HEAD: `bindConnectorTableSink` reads `sink.getStaticPartitionKeyValues()` +(:768) and calls `checkConnectorStaticPartitions(...)` (:778) but **NEVER reads `sink.getPartitions()`** — so +`INSERT INTO hive PARTITION(p1,p2)` would be **silently accepted** post-flip (a fail-loud regression). +`HiveConnectorMetadata.validateStaticPartitionColumns` is a permissive no-op covering only the static `col=val` +form, not this name-list form. + +**Touch points — mirror the existing `checkConnectorStaticPartitions`/`validateStaticPartitionColumns` seam** +(BindSink.java:727-756 + `ConnectorWriteOps.validateStaticPartitionColumns` :71-74): +1. **NEW neutral SPI** on `ConnectorWriteOps`, e.g. + `default void validateWritePartitionNames(ConnectorSession session, ConnectorTableHandle handle, + List partitionNames) { /* permissive default no-op */ }` (connector owns the message). +2. **fe-core wiring** in `bindConnectorTableSink`: after resolving `table`, guard + `if (!sink.getPartitions().isEmpty())`, resolve the handle (reuse the `checkConnectorStaticPartitions` + pattern / `PluginDrivenExternalTable.resolveConnectorTableHandle`), call the SPI, and + `catch (DorisConnectorException e) { throw new AnalysisException(e.getMessage(), e); }`. **The handle + round-trip + SPI call happen only inside the guard** — so plain `INSERT INTO hive SELECT` (empty + `getPartitions()`) is fully byte-unchanged for every live connector; only an `INSERT … PARTITION(names)` + pays the extra round-trip. +3. **Connector site** `HiveConnectorMetadata`: override — foreign (non-`HiveTableHandle`) → + `siblingMetadata(session).validateWritePartitionNames(...)`; hive handle + non-empty names → + `throw new DorisConnectorException("Not support insert with partition spec in hive catalog.")` (exact legacy + message, **no trailing period difference** — see T-below); empty names → return silently. +4. **flip/delete-time:** delete BindSink.java:667-669 (with the rest of `bindHiveTableSink`). + +**Traps.** +- **T1 (key on the right field).** Must key the reject on `sink.getPartitions()` (dynamic NAME list) **only** — + NOT `staticPartitionKeyValues`. Hive static-partition INSERT `PARTITION(dt='x')` is legal plain-hive and must + stay permissive; conflating the two newly rejects legal writes. +- **T2 (don't key on "table is partitioned").** A normal dynamic-partition hive write uses **no** PARTITION + clause (`getPartitions()` empty), so keying on partitioning would reject **all** writes to partitioned hive + tables. Plain `INSERT … SELECT` → guard false → no reject (parity preserved). +- **T3 (delegate foreign handles).** The hive override must delegate a foreign (iceberg-on-HMS) handle to the + sibling (permissive), else iceberg-on-HMS `PARTITION(...)` writes get rejected where a standalone + `type=iceberg` catalog accepts them — a heterogeneous-vs-standalone divergence. +- **T4 (early-return on empty names).** Return silently on empty names so the sibling-test invariant + "`validate*` for a hive handle returns silently" (`HiveConnectorMetadataSiblingDelegationTest`) still holds. +- **T5 (do NOT inline a generic reject).** `bindConnectorTableSink` is **live** for iceberg/max_compute/paimon, + which today silently ignore `PARTITION(p1,p2)`. A generic throw inlined in fe-core would be a **live behavior + change** (not dormant) and would draft the message in the engine (violates connector-agnostic + + message-in-connector). The permissive-default neutral SPI keeps live connectors' *empty-partition* path + byte-unchanged and the hive reject dormant. **Class C caveat:** for a live connector that *does* pass a + non-empty name-list, the new path runs a real `getTableHandle` round-trip + no-op SPI (behavior-preserving, + not byte-unchanged) — the proving test must confirm those connectors still silently accept. + +**Message-text constraint (hard).** The pre-existing e2e +`regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy:250` asserts on `INSERT … +partition(pt1,pt2)`: `exception "errCode = 2, detailMessage = Not support insert with partition spec in hive +catalog"`. This LIVE test runs on the legacy path today and against the flipped **connector** path post-flip, so +it is the byte-exact parity cross-check: `HiveConnectorMetadata`'s `DorisConnectorException.getMessage()` → +`AnalysisException` **must** reproduce `Not support insert with partition spec in hive catalog` verbatim. (The +legacy fe-core literal has a trailing `.`; the regression matches on a substring without it, so either is fine — +but keep the connector message byte-identical to the legacy literal to be safe.) + +**Proving test.** +- **Connector** (dormant, provable now): extend `HiveConnectorMetadataSiblingDelegationTest` — hive handle + + non-empty names → throws exactly the legacy message; hive handle + empty list → no-op, never consults the + sibling; foreign handle → forwards to sibling. Add `validateWritePartitionNames` to `EXPECTED_WRITE_METHODS` + (the Rule-9 completeness lock). +- **fe-core**: a `bindConnectorTableSink` test asserting a flipped-hms hive `INSERT … PARTITION(p1,p2)` surfaces + `AnalysisException` carrying the legacy message, AND that plain `INSERT INTO hive SELECT` and hive static + `PARTITION(dt='x')` are NOT rejected. Encode WHY (fail-loud parity), not just that a throw occurs. +- **Live-connector no-regression** (Class C): iceberg/paimon INSERT still green (the guard + round-trip must not + reject their partition writes). + +**Deps / risk.** Depends on the sibling-delegation plumbing already landed (W1–W5, present at HEAD). +Independent of items 1/3/4/5. Land the SPI+override+wiring as one dormant commit **before** the flip; the delete +of BindSink.java:667-669 happens at/after the flip. + +--- + +### Item 2b — reject LZO read-only — **DONE in connector; only a flip-time delete remains** + +**HEAD state (three loci; the definitive guard is NOT the BindSink fast-fail).** +1. **Legacy fast-fail** (best-effort, table-SD only): `BindSink.java:671-681` — throws + `"INSERT INTO is not supported for LZO Hive tables (input format: …). LZO tables are read-only in Doris."`. + Its own comment (BindSink.java:673-676) says it is *best-effort* and the **definitive** guard lives + elsewhere. +2. **Legacy definitive guard** (`BaseExternalTableDataSink.getTFileFormatType(String)` :78-89) — rejects LZO + before any other format match. Sole callers are `HiveTableSink` (table SD, ~:126) and every existing + partition SD (~:223). `PluginDrivenTableSink` extends the same base but does **not** call + `getTFileFormatType(String)`. +3. **Connector port** (`HiveSinkHelper.getTFileFormatType` :83-101, **byte-identical message**), wired on the + write-admission path at `HiveWritePlanProvider.buildSink:195` (table SD) **and** `buildExistingPartitions:262` + (every existing partition SD). + +**Classification.** No remaining port work — 2b is done in the connector. The connector port is **PARITY**, not +an upgrade: legacy **already** rejected per-partition LZO via `BaseExternalTableDataSink.getTFileFormatType` +(called for the table SD and every partition SD, per its comment). Earlier recon's "intentionally broader / +documented upgrade" framing was **wrong** — it conflated "the BindSink fast-fail" (table-SD only) with "legacy +behavior" (which includes the definitive per-partition guard). Coverage of plain-hive is unchanged. + +**Remaining touch point (flip/delete-time only).** Delete scope is broader than earlier recon stated — the whole +legacy sink chain dies at flip and includes **all three** loci: +- `BindSink.java:671-681` (the fast-fail block — inside `bindHiveTableSink`, dies with the whole method); +- `HiveTableSink` itself (constructed live at `PhysicalPlanTranslator.java:569 + `new HiveTableSink((HMSExternalTable)...)`); +- `BaseExternalTableDataSink.getTFileFormatType(String)` becomes **dead** once `HiveTableSink` is deleted (no + other caller). The flip author decides whether to also remove the now-orphaned protected method. + +**Traps.** +- The substring-`"lzo"` match is case-insensitive and applied **before** the text/orc/parquet match + (`LzoTextInputFormat` contains `"text"`); the ordering at `HiveSinkHelper` :84-96 (and + `BaseExternalTableDataSink` :86-96) is load-bearing — reordering it after the text match would misclassify + LZO as `FORMAT_CSV_PLAIN` and let a permanently-invisible write through. +- The reject now fires at **plan** time (`planWrite`) rather than analysis time (`BindSink`) — a slightly later + but still pre-BE fail point; matches the design's "push into the write path". + +**Proving test.** Already dormant-proven by `HiveWritePlanProviderTest.planWriteRejectsLzoInputFormat` (asserts +`planWrite` throws `DorisConnectorException` naming LZO for a table-SD LZO input format). A per-partition-LZO +variant (partition SD LZO, table SD plain) would additionally lock the broadened wiring at +`HiveWritePlanProvider:262` — worth adding, not blocking. + +**Deps / risk.** Independent of all other items. The DELETE is part of the flip/delete unit (§3) and must not +precede the flip. + +--- + +### Item 3 — delete `Env.hiveTransactionMgr` + accessors — **flip/delete-time only** + +**HEAD state.** fe-core `Env.java:568` `private HiveTransactionMgr hiveTransactionMgr;`; `Env.java:865` +`this.hiveTransactionMgr = new HiveTransactionMgr();`; `Env.java:1097-1103` +`getHiveTransactionMgr()` / `static getCurrentHiveTransactionMgr()`. The **only live consumer** is legacy fe-core +`HiveScanNode.java:141/147/319`. `HiveScanNode` is still **live** pre-flip: `PhysicalPlanTranslator.java:814`'s +`table instanceof PluginDrivenExternalTable` arm is missed by a plain `HMSExternalTable`, falling to the switch +`case HIVE: new HiveScanNode(...)` (~:834-835). The plugin already holds a complete port +(`HiveConnector.readTxnManager`, `HiveReadTransactionManager`, "plugin-owned and dormant until the read +cutover", HiveReadTransactionManager.java:31-35). One test reference: `OlapInsertExecutorTest.java:264/270` +mocks the static `getCurrentHiveTransactionMgr`. + +**Classification / touch points (DELETE-only, at the read cutover — same change that deletes legacy +`HiveScanNode`).** No SPI to add, no dormant relocation (the relocation already shipped as +`HiveReadTransactionManager`): +1. `Env.java:568` delete field; 2. `Env.java:865` delete ctor init; 3. `Env.java:1097-1099` delete +`getHiveTransactionMgr()`; 4. `Env.java:1101-1103` delete `getCurrentHiveTransactionMgr()`; 5. delete orphaned +fe-core `datasource/hive/HiveTransactionMgr.java` + `HiveTransaction.java` (superseded by the plugin ports); +6. update `OlapInsertExecutorTest.java:264/270` to drop the mock (or the test stops compiling). **No connector +site** — `HiveConnector.readTxnManager` is already wired; **item 1** is what activates it. + +**Traps.** +- Deleting the Env field/accessors while legacy `HiveScanNode` still exists = compile break at + HiveScanNode.java:141/147/319 **and** live ACID hive reads lose their read-lock + write-id snapshot + registration → incorrect ACID snapshots + leaked HMS read locks. Must be **strictly co-sequenced** with + `HiveScanNode` deletion. +- If the flip lands before **item 1** wires the plugin-side release, flipped ACID reads register a + `HiveReadTransaction` that is never committed → **permanent metastore read-lock leak**. So item 1 gates the + flip; item 3 gates on the flip. +- Forgetting `OlapInsertExecutorTest` update = test compile break (silent "skipped test" risk). + +**Proving test.** No dormant unit test can prove deletion in isolation (the field is live today). Proven +post-cutover by: (i) fe-core compiles after `HiveScanNode` + Env manager removal; (ii) existing +`HiveReadTransactionTest` locks `register→begin`/`deregister→commit`; (iii) an e2e ACID-hive read regression on a +flipped hms catalog asserting the HMS read lock is released at query finish (per MEMORY: iceberg/hive-on-HMS new +capabilities each need e2e); (iv) `OlapInsertExecutorTest` updated and green. + +**Deps / risk.** Purely delete-time; **last** in the whole write-chain. Gated behind item 1 (before flip) and +the flip itself. Independent of items 4/5 code-wise. + +--- + +### Item 4 — relocate `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY` — **dormant-committable-now (Class B)** + +**HEAD state.** `HMSExternalCatalog.java:60` `public static final String BIND_BROKER_NAME = "broker.name";`. +`BrokerProperties.java:58` `private static final String BIND_BROKER_NAME_KEY = "broker.name";` (**private**; also +used by `guessIsMe` :64-65 via `equalsIgnoreCase`). The two literals are **identical** (`"broker.name"`). The +generic base reads it: `ExternalCatalog.java:1319-1320` +`public String bindBrokerName(){ return catalogProperty.getProperties().get(HMSExternalCatalog.BIND_BROKER_NAME); +}` — a fe-core **base-class → HMS-subclass** constant dependency (import at ExternalCatalog.java:44). Repo-wide, +`HMSExternalCatalog.BIND_BROKER_NAME` has **exactly one** external reference (ExternalCatalog.java:1320) plus its +definition; `HMSExternalCatalog` appears in `ExternalCatalog.java` **only** at the import (:44) and :1320. +`bindBrokerName()` itself has multiple callers, all unaffected because the resolved key is unchanged +(HiveScanNode.java:125/210/239, DefaultConnectorContext.java:318, HiveTableSink.java:161; mocked in +HiveScanNodeTest.java). + +**Touch points (fe-core only, no connector/SPI surface).** +1. `BrokerProperties.java:58` `private → public` on `BIND_BROKER_NAME_KEY`. +2. `ExternalCatalog.java:1320` `HMSExternalCatalog.BIND_BROKER_NAME → BrokerProperties.BIND_BROKER_NAME_KEY`, + keeping the exact `catalogProperty.getProperties().get(...)` shape. +3. `ExternalCatalog.java:44` remove the now-unused `import …hive.HMSExternalCatalog;`, add + `import …datasource.property.storage.BrokerProperties;` (same fe-core module). +4. `HMSExternalCatalog.java:60` delete the constant. + +This also satisfies an iron-rule cleanup: the generic `ExternalCatalog` base no longer imports/depends on the +HMS subclass. + +**Traps.** +- **Preserve exact lookup semantics.** `bindBrokerName()` uses case-**sensitive** `Map.get("broker.name")`, + whereas `guessIsMe` uses `equalsIgnoreCase`. **Do NOT harmonize** `bindBrokerName` to case-insensitive while + relocating — that would silently change the resolved broker name for every catalog that set a differently-cased + key. Keep `.get(BIND_BROKER_NAME_KEY)` byte-identical to `.get("broker.name")`. +- Because both constants are literally `"broker.name"`, the resolved value is provably unchanged for hms **and** + every non-hms catalog (absent → null). +- Don't delete the HMS copy without the single-reference grep (done) — an outside reference would compile-break. +- Making the key public must not shadow the `@ConnectorProperty(names={"broker.name"})` literal at + BrokerProperties.java:38 — separate, both remain `"broker.name"`. + +**Proving test.** Parity unit test on the **real** method (HiveScanNodeTest *mocks* `bindBrokerName` returning +`""`, so it will NOT catch a regression): (i) `{"broker.name":"b1"}` → `bindBrokerName()=="b1"`; (ii) absent → +`null`; (iii) assert the single-source constant equals `"broker.name"` post-removal. + +**Deps / risk.** Fully independent; committable **now**, no flip gating. Lowest risk (Class B, value-identical). +Safe to land first. + +--- + +### Item 5 — `pluginCatalogTypeToEngine` add `case "hms" → ENGINE_HIVE` — **dormant-committable-now (Class A)** + +**HEAD state.** `CreateTableInfo.java:935-946` (`org.apache.doris.nereids.trees.plans.commands.info`): +`switch(catalog.getType()){ case "max_compute": return ENGINE_MAXCOMPUTE; case "paimon": return ENGINE_PAIMON; +case "iceberg": return ENGINE_ICEBERG; default: return null; }` — **no `"hms"`**, so hms → `null`. +`ENGINE_HIVE = "hive"` exists (CreateTableInfo.java:121). Flip mechanism confirmed: pre-flip `"hms"` → +`HMSExternalCatalog` (CatalogFactory :133-134); with `"hms"` in `SPI_READY_TYPES` → `PluginDrivenExternalCatalog` +(CatalogFactory :110) with `getType()=="hms"`. Consumers: +- `paddingEngineName` (CreateTableInfo.java:911-922): pre-flip hms hits :913 `instanceof HMSExternalCatalog → + ENGINE_HIVE`; **post-flip** it misses :913, reaches :915-916 `instanceof PluginDrivenExternalCatalog && + pluginCatalogTypeToEngine(...) != null` which is **false** (null) → falls to :921 `throw "Current catalog does + not support create table"`. +- `checkEngineWithCatalog` (:386-395) mirrors the same pattern. + +**Touch point (fe-core only, single line).** Add to the switch: +`case "hms": return ENGINE_HIVE;`. Post-flip this makes (i) `paddingEngineName` :915-919 pad `engine=hive` for a +no-ENGINE CREATE on a flipped hms catalog (mirrors legacy :913-914); (ii) `checkEngineWithCatalog` :388-393 +reject a non-hive explicit ENGINE with "This catalog can only use `hive` engine." (mirrors legacy :386-387). + +**Traps.** +- Must return `ENGINE_HIVE` (`"hive"`), never `"hms"`/`"maxcompute"`. +- **Non-interference verified** with the iceberg-only arm `getEffectiveIcebergFormatVersion` (~:1163) which + gates on `ENGINE_ICEBERG.equals(pluginCatalogTypeToEngine(...))` — `hms → ENGINE_HIVE != ENGINE_ICEBERG`, so + hms does NOT trigger iceberg row-lineage/format-version logic (correct: an iceberg-on-HMS table created via + the hms gateway is still a hive-engine CREATE — legacy hms catalogs always use hive engine). +- **Dormancy verified (Class A):** no `PluginDrivenExternalCatalog` has `getType()=="hms"` until `"hms"` enters + `SPI_READY_TYPES`, so the new case is unreachable pre-flip; plain `HMSExternalCatalog` continues through the + `instanceof` arms (:913 / :386) unchanged. + +**IRON-RULE note (tension, not a new violation).** `pluginCatalogTypeToEngine` (and items 5b's switches) switch +on `PluginDrivenExternalCatalog.getType()` — a source-type switch in fe-core, in tension with "no +`switch(dlaType)`/engine-name checks". This is a **pre-existing, maintainer-sanctioned** pattern +(max_compute/paimon/iceberg already committed; javadoc at CreateTableInfo.java:926-933 documents the sync +invariant), so extending it with `"hms"` **conforms** (Rule 11) and is NOT a new violation. Cleaner long-term: +push engine-name behind a connector-provided SPI so fe-core stops switching on `getType()` — out of scope here. + +**Proving test.** Dormant unit test in `CreateTableInfoEngineCatalogTest` (mocks +`PluginDrivenExternalCatalog.getType()`, reflectively invokes private `paddingEngineName`/`checkEngineWithCatalog`): +`registerPluginCatalog("hms_ctl","hms")` then assert (i) no-ENGINE CREATE pads `engineName=="hive"`; (ii) CTAS via +`validateCreateTableAsSelect` pads `"hive"`; (iii) `checkEngineWithCatalog` throws for explicit `ENGINE!=hive`; +(iv) passes for `ENGINE=hive`. Proves the switch entry without an actual flip (`getType` mocked to `"hms"`). + +**Deps / risk.** Fully independent; committable now, live only at the flip. **Merge with item 5b** (below). + +--- + +### Item 5b — `PluginDrivenExternalTable` engine-display `"hms"` cases — **dormant-committable-now (Class A) — MERGE INTO ITEM 5** + +**Why this exists.** Verifying item 5 surfaces a firm dormant sub-step the original 5-item set omitted, and the +`pluginCatalogTypeToEngine` javadoc **mandates** it: CreateTableInfo.java:930-931 says the switch "must stay in +sync" with `PluginDrivenExternalTable.getEngine()/getEngineTableTypeName()`. `hms` IS a CREATE-TABLE-capable +full-adopter, so the sync obligation is triggered. + +**HEAD state.** `PluginDrivenExternalTable.java:988-1021` `getEngine()` and :1023-1043 +`getEngineTableTypeName()` switch on `PluginDrivenExternalCatalog.getType()` with cases +jdbc/es/iceberg/trino-connector/max_compute/paimon and **NO `"hms"`** → default `super.getEngine()` (`"Plugin"`) +and `TableType.PLUGIN_EXTERNAL_TABLE.name()`. A flipped hms table therefore **displays engine `"Plugin"`** in +`SHOW TABLE STATUS` / `information_schema.tables`, where legacy `HMSExternalTable` shows **`"hms"`** — a +user-visible regression at flip. + +**Correct parity values (NOT `"hive"`).** Legacy DISPLAY engine for hms tables is **`"hms"`** — verified +`TableIf.java:270-271` `case HMS_EXTERNAL_TABLE: return "hms";`. This is **distinct** from the CREATE-TABLE +engine (item 5 → `"hive"`). So item 5b must add: +- `getEngine()`: `case "hms": return TableType.HMS_EXTERNAL_TABLE.toEngineName();` (== `"hms"`). +- `getEngineTableTypeName()`: `case "hms": return TableType.HMS_EXTERNAL_TABLE.name();` (== `"HMS_EXTERNAL_TABLE"`). + +Returning `ENGINE_HIVE` here would be a *different* regression (legacy hms tables never displayed `"hive"`). + +**Classification / scope.** Class A (unreachable until a PluginDriven catalog has `getType()=="hms"` — same +dormancy as item 5). **Display-only, NOT a functional break** — verified: `getEngineTableTypeName` feeds only +SHOW output (Env.java:4292/4657); `getEngine()` has **no** write/sink/route/dml callers (grep-confirmed). But it +is a firm dormant-committable sub-step, not a "soft follow-up". + +**Why merge into item 5's commit.** Splitting opens a window where CREATE resolves `engine=hive` but display +shows `"Plugin"` — an internally inconsistent flip. Land item 5 (CREATE engine) + item 5b (display engine) as +**one commit**. + +**Proving test.** Extend the item-5 test harness (or the existing engine-display test for +`PluginDrivenExternalTable`): mock `getType()=="hms"`, assert `getEngine()=="hms"` and +`getEngineTableTypeName()=="HMS_EXTERNAL_TABLE"` (mirrors the jdbc/es/iceberg cases). + +**Deps / risk.** Independent of items 1/2a/3/4. Coupled to item 5 (merge). Low risk (display-only, unreachable). + +--- + +## 2. Ordered dormant sub-steps (WC codes — internal only) + +All four WC commits land **before** the single `SPI_READY_TYPES` flip line. WC1–WC3 are independent leaves with +**no ordering constraint among themselves**; WC4 (item 1) is the one **hard pre-flip prerequisite**. Recommended +order is risk-ascending (land the safest first): + +| Code | Item(s) | Class | What lands | Deps | Risk | +|------|---------|-------|-----------|------|------| +| **WC1** | 4 | B (value-identical) | Relocate `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY`; make key public; drop HMS copy; fix `ExternalCatalog` import | none | **Lowest.** Live method but resolved value provably identical. Only risk = case-sensitivity harmonization (T-item4). | +| **WC2** | 5 **+** 5b | A (unreachable) | `pluginCatalogTypeToEngine` `case "hms"→ENGINE_HIVE`; `PluginDrivenExternalTable.getEngine()` `case "hms"→"hms"`; `getEngineTableTypeName()` `case "hms"→"HMS_EXTERNAL_TABLE"` — **one commit** | none | **Very low.** Strictly unreachable pre-flip. Risk = wrong return value (hive vs hms confusion) — CREATE=hive, DISPLAY=hms. | +| **WC3** | 2a | C (live, no-op default) | New `ConnectorWriteOps.validateWritePartitionNames` (permissive default); `bindConnectorTableSink` guarded wiring; `HiveConnectorMetadata` override (hive→reject, foreign→sibling) | sibling plumbing W1–W5 (present) | **Medium.** New guard runs on live iceberg/mc/paimon **only when `PARTITION(names)` present**; empty case byte-unchanged. Needs live-connector no-regression test. Message must match e2e verbatim. | +| **WC4** | 1 | C (live, no-op default) | New `ConnectorScanPlanProvider.releaseReadTransaction` (no-op default); `HiveScanPlanProvider` override; `PluginDrivenScanNode.getSplits` **unconditional** callback registration with TCCL pin | none (coupled to item 3 at delete-time) | **Highest of the four.** Registration runs on **every** live plugin scan (es/jdbc/paimon/mc/iceberg) calling the no-op default. TCCL pin is load-bearing (T1). **Hard prerequisite for a safe flip.** | + +**Then:** the single flip line — add `"hms"` to `CatalogFactory.SPI_READY_TYPES:50` (**not** a WC step). + +**Ordering rationale.** +- WC1/WC2/WC3 have no code overlap and no flip-gating beyond "before the flip"; any interleaving is safe. Listed + risk-ascending. +- WC4 must merge **before the flip** or flipped ACID reads leak the metastore shared read lock (item 1 T-above / + item 3 trap-b). It has no ordering constraint vs WC1–WC3. +- Class-C commits (WC3, WC4) each **require a live-connector no-regression test** (paimon/iceberg scan + insert + still green), because their fe-core wiring executes today on live connectors relying on no-op SPI defaults. + Class-A/B commits (WC1, WC2) are provable with dormant unit tests alone. + +--- + +## 3. Residuals deferred to the flip / delete phase + +These are **NOT** dormant-committable and must **NOT** precede the flip. The flip flips +`HMSExternalTable → PluginDrivenExternalTable`, so `PhysicalPlanTranslator` selects `PluginDrivenScanNode` over +legacy `HiveScanNode` and `UnboundTableSinkCreator` routes writes through `UnboundConnectorTableSink`; the whole +legacy scan/sink chain then goes dead and is deletable. + +**R1 — Delete `bindHiveTableSink` (ONE deletion, holds BOTH legacy write pre-checks).** `bindHiveTableSink` +(BindSink.java:660) contains **both** the item-2a partition-spec reject (:667-669) **and** the item-2b legacy LZO +fast-fail (:671-681). They are a **single** flip-time delete (remove the whole method), not two — earlier +"two separate deletes / cyclic unit" framing was imprecise. + +**R2 — Delete legacy `HiveScanNode` + item 3.** Co-sequenced as one change: delete `HiveScanNode` +(sole live caller of `Env.getCurrentHiveTransactionMgr` via PhysicalPlanTranslator.java:834 `case HIVE`) together +with `Env.hiveTransactionMgr` field + accessors (Env.java:568/865/1097-1103), the orphaned fe-core +`HiveTransactionMgr.java`/`HiveTransaction.java`, and the `OlapInsertExecutorTest` mock update. **Gated behind +WC4** (item 1 must have wired the plugin-side release before the flip). + +**R3 — Delete `HiveTableSink` + orphaned LZO guard.** `HiveTableSink` (constructed at +PhysicalPlanTranslator.java:569) dies at the flip; `BaseExternalTableDataSink.getTFileFormatType(String)` +(:78-89) becomes dead once `HiveTableSink` is its last caller. Flip author decides whether to remove the +now-orphaned protected method. + +**R4 — The flip line itself.** Add `"hms"` to `CatalogFactory.SPI_READY_TYPES` (§4.5's single flip; explicitly +out of every WC/residual sub-step). + +--- + +## 4. Known unknowns / out-of-scope flags (honest boundaries) + +- **Write-side TCCL pin (design §4.4 item W6) is NOT closed by item 1.** Item 1 handles only the **read**-side + query-finish TCCL pin. The analogous **write**-side pin is open: `IcebergWritePlanProvider.planWrite` + (~:152) builds its `TDataSink` on a fe-core thread under the app loader, and `executeAuthenticated` is only + confirmed deep at ~:672 — whether sink-construction itself is pinned is flagged **OPEN** in + `hms-write-delegation-decomposition-2026-07-08.md:30` (W6: "flip-time, NOT dormant-testable, VERIFY first"). + This is §4.4 scope (not one of the five §4.5 items), so its absence from this decomposition is by design — but + a reader consolidating "remaining pre-flip write items" should know item 1's read-side pin does **not** close + the write-side TCCL story. +- **Multi-ACID-table read-txn leak (item 1 T4).** Pre-existing in both legacy `HiveTransactionMgr` and the + plugin `HiveReadTransactionManager` (both key `txnMap` by `queryId`). Deliberately **out of scope** for item 1 + — fixing it inside WC4 would diverge from legacy parity. +- **`deregister` swallows commit failure.** Not "fail-loud" (HiveReadTransactionManager.java:55-59 logs and + swallows `RuntimeException`; `QueryFinishCallbackRegistry` further isolates per-callback exceptions). Any + item-1 test must assert *exactly-once `commitTxn`* and *idempotent second call*, **not** an exception on + commit failure (that behavior does not exist at HEAD). + +--- + +## 5. References (all HEAD `75a5d25d746`, branch `catalog-spi-11-hive`) + +**Design / prior recon.** +- `plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md` §4.5 (:77-83), §5.1 (phase ordering). +- `plan-doc/tasks/hms-write-delegation-decomposition-2026-07-08.md` (W1–W6; W6 write-side pin OPEN at :30). +- `plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md` (sibling W1–W5 plumbing). + +**Item 1 (read-ACID).** `fe-connector-hive/.../HiveScanPlanProvider.java:86,124-127,175-177`; +`.../HiveReadTransactionManager.java:47-61`; `.../HiveReadTransaction.java` (begin/commit); +`.../HiveConnector.java:79,104,120-124`; `fe-connector-api ConnectorScanPlanProvider.java` (default-no-op +precedent `classifyColumn`/`getDeleteFiles`); `fe-core .../datasource/PluginDrivenScanNode.java:516-524 +(onPluginClassLoader), ~911-915/938/965 (getSplits)`; `.../qe/QeProcessorImpl.java:212,216-217`; +`.../qe/QeProcessor.java:46`; `.../qe/QueryFinishCallbackRegistry.java`; `.../connector/ConnectorSessionBuilder.java:57`; +legacy `.../planner/HiveScanNode.java:138-149,319`. Tests: `HiveReadTransactionTest`, +`QueryFinishCallbackRegistryTest`, `PluginDrivenScanNode…SelectionTest`/`MvccPinTest`. + +**Item 2a/2b (write pre-checks).** `fe-core .../nereids/rules/analysis/BindSink.java:660-681 (bindHiveTableSink: +2a :667-669, 2b :671-681), 727-756 (checkConnectorStaticPartitions), 758-850 (bindConnectorTableSink: :768/778)`; +`.../nereids/trees/plans/commands/UnboundBaseExternalTableSink.java:69`; +`.../nereids/trees/plans/commands/UnboundTableSinkCreator.java:60-63/93-97/128-132`; +`.../planner/BaseExternalTableDataSink.java:78-89`; `.../planner/HiveTableSink.java (~126/223)`; +`.../translator/PhysicalPlanTranslator.java:569`; `fe-connector-hive HiveSinkHelper.java:83-101`; +`HiveWritePlanProvider.java:195,262`; `HiveConnectorMetadata.java (validateStaticPartitionColumns ~:1503-1510, +siblingMetadata)`; `fe-connector-api ConnectorWriteOps.java:71-74`. Tests: +`HiveWritePlanProviderTest.planWriteRejectsLzoInputFormat`, `HiveConnectorMetadataSiblingDelegationTest` +(EXPECTED_WRITE_METHODS lock), e2e `regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy:250`. + +**Item 3 (Env txn mgr).** `fe-core .../catalog/Env.java:568,865,1097-1103`; +`.../datasource/hive/HiveTransactionMgr.java`, `HiveTransaction.java`; `.../planner/HiveScanNode.java:141,147,319`; +`.../translator/PhysicalPlanTranslator.java:814,834-835`; `HiveReadTransactionManager.java:31-35`; +test `OlapInsertExecutorTest.java:264,270`. + +**Item 4 (broker name).** `fe-core .../datasource/hive/HMSExternalCatalog.java:60`; +`.../datasource/property/storage/BrokerProperties.java:38,58,64-65`; +`.../datasource/ExternalCatalog.java:44,1319-1320`. + +**Item 5 / 5b (engine map + display).** `fe-core .../nereids/trees/plans/commands/info/CreateTableInfo.java:121 +(ENGINE_HIVE), 386-395 (checkEngineWithCatalog), 911-922 (paddingEngineName), 926-946 (pluginCatalogTypeToEngine ++ sync javadoc), ~1163 (getEffectiveIcebergFormatVersion)`; +`.../datasource/PluginDrivenExternalTable.java:988-1021 (getEngine), 1023-1043 (getEngineTableTypeName)`; +`.../catalog/TableIf.java:240-278 (toEngineName; HMS_EXTERNAL_TABLE→"hms" :270-271)`; `Env.java:4292,4657` (SHOW +consumers). Flip: `.../datasource/CatalogFactory.java:49-50 (SPI_READY_TYPES), 98,110,133-134`. Test: +`CreateTableInfoEngineCatalogTest`. \ No newline at end of file From 399f59c8770df77fa0414aceffb52c7cab95b15c Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 17:52:16 +0800 Subject: [PATCH 086/334] [refactor](catalog) HMS cutover write-chain WC1: relocate BIND_BROKER_NAME to BrokerProperties (single source) Value-identical relocation (both literals are "broker.name"). The generic ExternalCatalog base no longer depends on the HMS subclass for the broker-name key: - BrokerProperties.BIND_BROKER_NAME_KEY: private -> public (now the single source of truth, read by both guessIsMe case-insensitively and ExternalCatalog.bindBrokerName case-sensitively). - ExternalCatalog.bindBrokerName(): reads BrokerProperties.BIND_BROKER_NAME_KEY instead of HMSExternalCatalog.BIND_BROKER_NAME; drop the now-unused HMSExternalCatalog import, add BrokerProperties. - HMSExternalCatalog.BIND_BROKER_NAME: deleted (verified single external reference at HEAD). Lookup semantics kept byte-identical (case-sensitive Map.get) -- deliberately NOT harmonized to case-insensitive, which would silently change the resolved broker name for a differently-cased key. Iron-rule cleanup: removes a fe-core base-class -> HMS-subclass constant dependency. Dormant/no-op (value provably unchanged for every catalog). Test: BrokerPropertiesTest pins BIND_BROKER_NAME_KEY == "broker.name" (the shared-key regression surface). Green: fe-core BrokerPropertiesTest 6 + HiveScanNodeTest 6; checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../org/apache/doris/datasource/ExternalCatalog.java | 4 ++-- .../doris/datasource/hive/HMSExternalCatalog.java | 2 -- .../datasource/property/storage/BrokerProperties.java | 4 +++- .../property/storage/BrokerPropertiesTest.java | 10 ++++++++++ 4 files changed, 15 insertions(+), 5 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index 7f02071ec307d3..bc68f947e2b644 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -41,13 +41,13 @@ import org.apache.doris.common.util.Util; import org.apache.doris.datasource.connectivity.CatalogConnectivityTestCoordinator; import org.apache.doris.datasource.doris.RemoteDorisExternalDatabase; -import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalDatabase; import org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase; import org.apache.doris.datasource.infoschema.ExternalMysqlDatabase; import org.apache.doris.datasource.lakesoul.LakeSoulExternalDatabase; import org.apache.doris.datasource.metacache.MetaCache; import org.apache.doris.datasource.operations.ExternalMetadataOps; +import org.apache.doris.datasource.property.storage.BrokerProperties; import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.datasource.test.TestExternalDatabase; import org.apache.doris.kerberos.ExecutionAuthenticator; @@ -1418,7 +1418,7 @@ private String getLocalDatabaseName(String dbName, boolean isReplay) { } public String bindBrokerName() { - return catalogProperty.getProperties().get(HMSExternalCatalog.BIND_BROKER_NAME); + return catalogProperty.getProperties().get(BrokerProperties.BIND_BROKER_NAME_KEY); } // ATTN: this method only return all cached databases. diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java index e4157cf72ddadf..aab260f15444e2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalCatalog.java @@ -56,8 +56,6 @@ public class HMSExternalCatalog extends ExternalCatalog { public static final String PARTITION_CACHE_TTL_SECOND = "partition.cache.ttl-second"; public static final String HIVE_STAGING_DIR = "hive.staging_dir"; public static final String DEFAULT_STAGING_BASE_DIR = "/tmp/.doris_staging"; - // broker name for file split and query scan. - public static final String BIND_BROKER_NAME = "broker.name"; // Default is false, if set to true, will get table schema from "remoteTable" instead of from hive metastore. // This is because for some forward compatibility issue of hive metastore, there maybe // "storage schema reading not support" error being thrown. diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/BrokerProperties.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/BrokerProperties.java index a1c9b2e4d369ce..06bc155156922a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/BrokerProperties.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/property/storage/BrokerProperties.java @@ -55,7 +55,9 @@ public static BrokerProperties of(String brokerName, Map origPro return properties; } - private static final String BIND_BROKER_NAME_KEY = "broker.name"; + // Broker name for file split and query scan. The single source of truth for the "broker.name" catalog + // property key (the generic ExternalCatalog.bindBrokerName() reads it; guessIsMe matches it case-insensitively). + public static final String BIND_BROKER_NAME_KEY = "broker.name"; public static boolean guessIsMe(Map props) { if (props == null || props.isEmpty()) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/storage/BrokerPropertiesTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/storage/BrokerPropertiesTest.java index d9fb972e98c377..aaa1a7fb6c06b2 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/property/storage/BrokerPropertiesTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/property/storage/BrokerPropertiesTest.java @@ -36,6 +36,16 @@ void testOfMethod_initializesBrokerNameAndParams() { Assertions.assertEquals("abc", props.getBackendConfigProperties().get("fs.s3a.access.key")); } + @Test + void testBindBrokerNameKeyIsTheSharedBrokerNameLiteral() { + // BIND_BROKER_NAME_KEY is now the SINGLE source of truth for the "broker.name" catalog property, read by + // BOTH guessIsMe (case-insensitive match, below) and the generic ExternalCatalog.bindBrokerName() + // (case-sensitive Map.get). It replaced the former HMSExternalCatalog.BIND_BROKER_NAME copy so the generic + // base no longer depends on the HMS subclass. Pin its value: a drift here silently breaks broker + // resolution for every catalog (both the guessIsMe probe and the bindBrokerName lookup). + Assertions.assertEquals("broker.name", BrokerProperties.BIND_BROKER_NAME_KEY); + } + @Test void testGuessIsMe_returnsTrueWhenBrokerNamePresent() { Map props1 = ImmutableMap.of("broker.name", "test"); From c1629b68476311092dfb6208f5afdd44507f3584 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 17:53:47 +0800 Subject: [PATCH 087/334] [doc](catalog) HMS cutover write-chain: decomposition landed + WC1 (BIND_BROKER_NAME relocation) DONE; next = WC2 Records the write-chain decomposition doc and WC1 (6c01805109c). Sets the risk-ascending path WC2 (engine-map hms cases, unreachable) -> WC3 (partition-spec reject, needs live-connector no-regression test) -> WC4 (read-ACID query-finish commit wiring, the hard flip prerequisite), with the flip-time residuals (R1-R4) and the post- write-chain candidates (hudi write-reject, getNewestUpdateVersionOrTime[blocked on D2]) spelled out. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4c431e7d01ac0f..306a395d183483 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -71,11 +71,13 @@ - **✅ W4 DONE（`5a41e4cc562`）= 事务按句柄（Shape C，唯一无先例的新形态）**：`ConnectorWriteOps` 加 `beginTransaction(session, handle)` 默认重载（委派无参版=对齐 `getWritePlanProvider(handle)` 先例；6 现存连接器继承默认→各自无参 `beginTransaction`，字节不变）；`HiveConnectorMetadata` 覆写守卫转发 foreign 句柄→兄弟（得 `IcebergConnectorTransaction`）、hive 句柄→落连接器自有 `HiveConnectorTransaction`——**必须对称**（hive/iceberg 写计划把 session-bound txn 分别强转不同具体类型，"总转发"/"总 hive"都炸另一侧）。fe-core 把已解析写目标句柄穿进**两个**开事务点（否则 EXECUTE 路径留活 CCE）：`PluginDrivenInsertExecutor`（INSERT/OVERWRITE/DELETE/MERGE 单一漏斗）经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()`、`ConnectorRewriteDriver`（EXECUTE，已持句柄）；connector-agnostic，**绝不给网关传 null 句柄**（fail-loud，故不会把普通写误路由兄弟）。侦察(`wf_a6e430b0-3e9`)+4维对抗复审(`wf_6af16e19-0cc`)+verify（CCE 净、休眠/现存连接器字节平价、路由测 mutation-tight）；2 确认发现均测试覆盖缺口、已补（`PluginDrivenExternalTableTest` fail-loud 体测→`orElse(null)` 降级现挂；`PluginDrivenInsertExecutorTest` 句柄穿透 assertSame→null 二参 mutant 现挂）。全绿（fe-connector-hive 7〔+2 divert〕+ fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30〔+2〕/ ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕+ import gate 净）。**W4 复审记账（residual，休眠期不动）**：①**once-per-statement `getTableHandle` 往返**——每笔 plugin 写开事务时新增一次解析（paimon/iceberg 缓存命中、jdbc/maxcompute 未缓存），有意接受、非 per-row；②fail-loud orElseThrow 开事务时重查——并发 DROP/瞬时元数据错稍扩失败窗（paimon getTableHandle 吞异常成 empty→通用文案盖根因），非正确性回退；③W6 写路径 TCCL pin=**开事务不需**（`IcebergConnectorTransaction` 构造只赋值、SDK/反射推迟到 beginWrite/commit），commit 侧 pin 归 W6；④hudi-on-HMS 保 HiveTableHandle→hive txn（未变差，后续 hudi 委派步）；⑤e2e owed（休眠单测只证路由）。 - **✅ W5 DONE（`8aee62bab20`）= 存储过程按句柄**（详见上 🎯 段 W5 条目）。`Connector.getProcedureOps(handle)` per-handle 默认 + `HiveConnector` 分流（hive/hudi→继承 null、foreign→兄弟永不强转）+ fe-core `ConnectorExecuteAction` reorder（句柄解析上提、改调 per-handle）。net-new 升级、reorder 无测钉旧序、顺带修陈旧 javadoc。4 维对抗复审 0 blocker/major/minor + 1 nit（已修）。**⇒ §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官。** - ⚠**W5 复审记账（休眠期不动）**：①**paimon/无-procedure 连接器 EXECUTE 现多一次 `getTableHandle` 往返**（reorder 先解析句柄再判 null-ops；happy-path iceberg 无新增往返，只错误路径先报 "Table not found"）——有意接受、非正确性回退；②`ExecuteActionFactory.getSupportedActions` 留无参 getter（SHOW 发现，无 live caller；翻闸对 hms 网关会返 hive 连接器级空——handle-aware 变体延后，无消费者故无害）；③e2e owed（休眠单测只证路由；随其它 iceberg-on-HMS 能力翻闸期统一补，memory `hms-iceberg-delegation-needs-e2e`）。 -- **下一步 = 写/procedure/DDL 委派脊柱（读 S0–S6 + 写 W1–W5）已全部收官，转入剩余翻闸前休眠项。候选（择一起步，均独立可提交、均线上零影响；建议先与用户确认优先级）**： - 1. **§4.5 写链收尾**（读-ACID query-finish commit 接线、`BindSink` 显式分区 INSERT 拒 port 进插件、`Env.hiveTransactionMgr` 搬家、`BIND_BROKER_NAME` 搬家、`pluginCatalogTypeToEngine` 无 `hms` 的 engine-map）——延续写路径，为翻闸减负。 - 2. **hudi-on-HMS 写-拒安全网**（hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi）——hudi flip-blocker，防翻闸把只读 hudi-on-HMS 变成可写-as-hive。 - 3. **§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，hive 走 max 分区时间）——**被 D2 连接器缓存阻塞**（探针高频，须先做连接器缓存），故非即时可起。 - （W6 写路径 TCCL pin 属**翻闸期验证**、非休眠可测，见分解 §2 W6。e2e 回归亦翻闸期统一补。） +- **用户已选：写链收尾（§4.5）。权威拆解 = `hms-write-chain-decomposition-2026-07-08.md`（起步必读；code-grounded recon `wf_13e18d30-257`：3 维读者 + 完整性/排序 critic，全 HEAD 核对）**。核心 = 三类休眠 taxonomy（**A**=翻闸前不可达、**B**=值恒等、**C**=活跑靠 no-op SPI 默认——**C 类须补现存连接器无回归测**，非仅休眠 hive 单测）+ 风险升序 4 步： + - **✅ WC1 DONE（`6c01805109c`）= 搬 `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY`**（B 值恒等：两字面量均 `"broker.name"`；`private→public` 成单一真源、generic base 不再依赖 HMS 子类；`.get()` 保**大小写敏感、不 harmonize**；删 HMS 副本〔HEAD 证单引用〕）。全绿（BrokerPropertiesTest 6〔+1 常量真源 pin〕 + HiveScanNodeTest 6 + checkstyle 0）。 + - **下一步 = WC2**（A 不可达、极低风险，可紧接、一 commit）：`CreateTableInfo.pluginCatalogTypeToEngine` 加 `case "hms"→ENGINE_HIVE` + `PluginDrivenExternalTable.getEngine()` 加 `case "hms"→"hms"`、`getEngineTableTypeName()` 加 `case "hms"→"HMS_EXTERNAL_TABLE"`（CREATE=hive、DISPLAY=hms）。 + - **WC3**（item 2a，C 活跑、中风险）：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松默认）+ `bindConnectorTableSink` 守卫接线 + `HiveConnectorMetadata` 覆写（hive→拒显式分区 INSERT、foreign→兄弟）；**须补 paimon/iceberg 无回归测**、文案与 e2e 逐字一致；依赖 W1–W5 兄弟管线（已在）。 + - **WC4**（item 1，C 活跑、最高风险、**翻闸硬前置**）：新 `ConnectorScanPlanProvider.releaseReadTransaction`（no-op 默认）+ `HiveScanPlanProvider` 覆写 deregister + `PluginDrivenScanNode.getSplits` **无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，TCCL pin 经 `onPluginClassLoader` 白拿）；不补则翻闸后 ACID 读泄漏 metastore 共享读锁。**须补现存连接器无回归测**。 + - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** +- **其余翻闸前休眠候选（写链之后）**：**hudi-on-HMS 写-拒安全网**（hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider，须显式拒）；**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 - **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 From 53dc3eddc35100995da93a9196034b73dbc1de14 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 19:12:44 +0800 Subject: [PATCH 088/334] [doc](catalog) Plan: fold hudi into the HMS cutover via sibling delegation (no-regression); hudi now gates the flip Code-grounded recon (4 readers + completeness/ordering critics, all HEAD-verified) turning the three signed decisions (sibling delegation like iceberg / no incremental+time-travel+MVCC regression / plan-first) into an ordered, HEAD-grounded plan (HD-A0..HD-D1, groups A-D). Key results: - The flip is atomic: adding "hms" to SPI_READY_TYPES converts hive + iceberg-on-HMS + hudi-on-HMS at once, so hudi's no-regression build-out is a HARD flip-blocker (the iceberg spine alone is not enough). - PhysicalHudiScan wrinkle RESOLVED: a flipped hudi table is a generic PluginDrivenExternalTable -> LogicalFileScan -> PluginDrivenScanNode (same path as iceberg); incremental/time-travel re-home connector-side via the EXISTING resolveTimeTravel/applySnapshot/INCREMENTAL seam (paimon template) -- NO new fe-core SPI, no source-specific BindRelation/CheckPolicy arm (iron rule). - DV-005 model mismatch resolved: getType()="hudi" is a sibling-only type string (createSiblingConnector bypasses SPI_READY_TYPES); never promote hudi into SPI_READY_TYPES. - ONE new design decision surfaced for sign-off (HD-B1): with two siblings the binary handle discriminator breaks; recommend a neutral Connector.ownsHandle(handle) SPI predicate (dormant-unit-testable) over classloader-identity routing. It mutates the already-landed iceberg spine -- must stay byte-behavior-identical on the iceberg arm. - BIGGEST risk (HD-C3, possibly unfixable under the iron rules): COW-incremental row-level _hoodie_commit_time window filter is skipped on the generic path; needs BE-reader verification (BE tree absent here) + a straddling-base-file e2e fixture. Also surfaced a real snapshot-path bug (HD-A3 partition-value parse infidelity -> silent NULL partition columns on non-hive-style partitioned reads). No code (plan-first per sign-off). HANDOFF updated: hudi elevated from a write-reject footnote to a first-class flip-blocking work-stream. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 9 +- .../hudi-on-hms-delegation-plan-2026-07-08.md | 649 ++++++++++++++++++ 2 files changed, 657 insertions(+), 1 deletion(-) create mode 100644 plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 306a395d183483..90eefed1608b18 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -77,7 +77,14 @@ - **WC3**（item 2a，C 活跑、中风险）：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松默认）+ `bindConnectorTableSink` 守卫接线 + `HiveConnectorMetadata` 覆写（hive→拒显式分区 INSERT、foreign→兄弟）；**须补 paimon/iceberg 无回归测**、文案与 e2e 逐字一致；依赖 W1–W5 兄弟管线（已在）。 - **WC4**（item 1，C 活跑、最高风险、**翻闸硬前置**）：新 `ConnectorScanPlanProvider.releaseReadTransaction`（no-op 默认）+ `HiveScanPlanProvider` 覆写 deregister + `PluginDrivenScanNode.getSplits` **无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，TCCL pin 经 `onPluginClassLoader` 白拿）；不补则翻闸后 ACID 读泄漏 metastore 共享读锁。**须补现存连接器无回归测**。 - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** -- **其余翻闸前休眠候选（写链之后）**：**hudi-on-HMS 写-拒安全网**（hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider，须显式拒）；**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 +- **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** 关键结论： + - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 + - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 + - **DV-005 模型错配已解**：`HudiConnectorProvider.getType()="hudi"` 保留但**只作兄弟类型串**（`createSiblingConnector("hudi",…)` 绕过 `SPI_READY_TYPES`）；**永不**把 `hudi` 加进 `SPI_READY_TYPES`/工厂 case（会重造 DV-005）。 + - **有序子步（HD-A0..HD-D1，4 组）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 + - **⛔ 最大风险 HD-C3（§5.1）**：COW 增量的行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 `CheckPolicy` 注入、通用路径**跳过**且铁律禁 fe-core 复刻；COW 更新重写整个 base 文件把旧 commit 行带进→纯文件级选择返窗口外行=**静默错答**。须连接器把行谓词推进 scan-range/thrift **或** BE reader 按 instant 窗口行级过滤——**须先读 BE Hudi reader 源码判定**（本 checkout 无 BE 树），可能在铁律下无 fe-core 解，且只能翻闸 e2e 验（straddling-base-file fixture）。paimon **非**先例。 + - **翻闸硬门 B1–B6**（见 plan §4.2）+ 两非单测 harness（两加载器路由 fixture + 异构 HMS e2e，含 COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved·非 hive-style/转义分区·写-拒）。**schema 缺失 SIGABRT 方向**（§5.2）已由 critic 对 BE 核对**降级为窄回退非崩溃**（emit-nothing=安全基线走 BE 小写 name-match），但 BE 树不在本 checkout、须落 HD-C4 前复核。 +- **其余翻闸前休眠候选**：**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 - **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 diff --git a/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md b/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md new file mode 100644 index 00000000000000..0bbda3d1c12e8f --- /dev/null +++ b/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md @@ -0,0 +1,649 @@ +# Hudi-on-HMS Delegation Plan (fold Hudi into the HMS-catalog cutover) + +Authoritative, code-grounded plan. HEAD = `75670ae4193d76859c5b261a8ea6bce33d046e00` on branch +`catalog-spi-11-hive`. All line numbers below are read fresh from HEAD (the `gitStatus` snapshot in the +task prompt, `d24e4af`, was stale). Planning only — **no code in this document**. + +--- + +## 0. Scope, signed decisions, and the model-mismatch resolution + +### 0.1 Scope +Fold Hudi into the single HMS-catalog SPI cutover so that, when `"hms"` enters +`CatalogFactory.SPI_READY_TYPES`, a **hudi-on-HMS** table is served by the SPI stack with **no regression** +versus the legacy `HudiScanNode` path. This plan covers: making `fe-connector-hudi` gateway-ready as a +**sibling** connector, the gateway divert (mirroring iceberg), closing the no-regression capability gaps +(incremental read, time travel, MVCC snapshot, partition enumeration, partition-value fidelity, +schema-at-instant), and the read-only write-reject safety net. It also inventories the flip-time dead-code +deletion and the hard flip-blockers. + +### 0.2 The three signed decisions (FIXED — 2026-07-08, do not re-litigate) +1. **Sibling delegation.** Hudi-on-HMS tables are served by **delegating to a hudi SIBLING connector**, exactly + mirroring how iceberg-on-HMS delegates to an iceberg sibling. The `type=hms` hive plugin is the **gateway**; + it builds the hudi sibling via `context.createSiblingConnector("hudi", ...)`, per-handle diverts + scan/metadata to it, and owns its lifecycle. There is **no** standalone `type=hudi` catalog. +2. **No regression.** Hudi incremental read + time travel + MVCC snapshot (all supported by legacy + `HudiScanNode`) must be **closed before the flip**. Leaving them deferred at the flip would regress hudi + users and is not acceptable. +3. **Plan first.** This written plan lands before any code. + +### 0.3 Model-mismatch resolution (DV-005 — RESOLVED, not blocking) +At HEAD, `fe-connector-hudi` declares `HudiConnectorProvider.getType() == "hudi"` +(`HudiConnectorProvider.java:36-38`) as if hudi were a standalone catalog type. But Doris has **no** standalone +`type=hudi` catalog: `CatalogFactory.SPI_READY_TYPES` (`CatalogFactory.java:49-50` = +`{jdbc, es, trino-connector, max_compute, paimon, iceberg}`) contains neither `hms` nor `hudi`; the factory +switch has a `case "hms"` but **no** `case "hudi"`; there is **no** `HudiExternalCatalog` class. Hudi is +parasitic on HMS (`HMSExternalTable.dlaType == HUDI`). + +**Resolution under the sibling model:** `getType() == "hudi"` is **kept** — it is precisely the lookup key that +`context.createSiblingConnector("hudi", ...)` resolves, and `createSiblingConnector` bypasses +`SPI_READY_TYPES` (it goes straight through `DefaultConnectorContext.createSiblingConnector` → +`ConnectorFactory.createConnector`). Therefore: +- **Never** add `"hudi"` to `SPI_READY_TYPES` and **never** add a `case "hudi"` to the factory. Doing so would + build a standalone `PluginDrivenExternalCatalog` around `HudiConnector` with no fe-core catalog class = the + exact DV-005 bug re-created. +- The `HudiConnectorProvider` javadoc (`:30`, *"dedicated Hudi catalogs that connect to HMS"*) is now + **misleading and dangerous** — it invites a future maintainer to "fix" DV-005 by promoting hudi into + `SPI_READY_TYPES`. It must be corrected to *"sibling-only type string, used only via createSiblingConnector + by the hive HMS gateway; never a user-facing catalog type; never add to SPI_READY_TYPES"* (see **HD-A0**). + +The single flip toggle remains **one line**: add `"hms"` to `SPI_READY_TYPES`. That flip converts hive + +iceberg-on-HMS + hudi-on-HMS **simultaneously** (see §4). + +--- + +## 1. Current state + +### 1.1 The hudi sibling connector — what works, what is stubbed (HEAD) + +**Real (the protected ~25% anchor):** +- `HudiConnector` (`HudiConnector.java`) implements `getMetadata` (`:60`), no-arg `getScanPlanProvider()` + (`:65`), `close` (`:103`); builds a `ThriftHmsClient` from raw props via `createClient` (`:80-100`) using + `context::executeAuthenticated` (`:98`). **No** per-handle overloads, **no** write/procedure/transaction + surface, **no** `getCapabilities`. +- `HudiConnectorMetadata` implements `listDatabaseNames`, `databaseExists`, `listTableNames`, `getTableHandle`, + `getColumnHandles`, `applyFilter` (EQ/IN partition pruning → `HudiTableHandle.prunedPartitionPaths`), + `getTableSchema` (`:199`), `getProperties`. COW/MOR detection via `detectHudiTableType` (substring match). + It overrides **none** of `{resolveTimeTravel, applySnapshot, beginQuerySnapshot, getMvccPartitionView, + listPartitions, listPartitionNames, listPartitionValues, getTableFreshness, getPartitionFreshnessMillis, + buildTableDescriptor}` — all inherit SPI no-op defaults. +- `HudiScanPlanProvider` (`HudiScanPlanProvider.java`): snapshot-only split planning. `isCow = + "COPY_ON_WRITE".equals(...)` (`:92`); COW native via `fsView.getLatestBaseFilesBeforeOrOn(partition, + queryInstant)` (`collectCowSplits :204-208`); MOR via `fsView.getLatestMergedFileSlicesBeforeOrOn(...)` + (`collectMorSplits :232-237`) with `useNative = logs.isEmpty() && !filePath.isEmpty()` (`:249`) else a JNI + split (`HudiScanRange` → `THudiFileDesc`). `getScanNodeProperties` emits `file_format_type` / + `table_format_type` + `location.*` passthrough. + +**Stubbed / deferred (the no-regression gaps):** +- `HudiScanPlanProvider.planScan` **always** reads `timeline.lastInstant()` (`:103-108`) and ignores any pin → + time travel and incremental read are not honored. +- No `resolveTimeTravel` / `applySnapshot` / `beginQuerySnapshot` → time-travel / @incr / MVCC-pin all inherit + empty defaults. +- No `listPartitions*` / freshness → a partitioned hudi table's MVCC view is empty (DV-007). +- No `schema_id` / `history_schema_info` / field-ids → schema-evolution degrades to BE name-matching (DV-006). +- **Partition-value parse infidelity (newly surfaced, not in prior batch lists):** + `HudiScanPlanProvider.parsePartitionValues` (`:317-334`) only handles hive-style `k=v` fragments — a + fragment **without** `=` is **silently dropped**, and it never URL-unescapes. Legacy + `HudiPartitionUtils.parsePartitionValues` (`:63-89`) maps **positionally** when a fragment lacks `k=`, has a + single-column-whole-path fallback, and URL-unescapes every value via `FileUtils.unescapePathName`. + Consequence: for Hudi's **default** non-hive-style partitioning (`hive_style_partitioning=false`, paths like + `2024/01`), the connector builds an **empty** partition-value map → BE returns **NULL** partition columns on + a plain snapshot read; escaped chars (`%20`, `%2F`) arrive un-unescaped. This bites the **most basic** + unpruned read (where `resolvePartitions` falls back to `getAllPartitionPaths`, which returns positional + paths). **This is a snapshot-path correctness regression on the "safe anchor" and must be fixed** (see + **HD-A3**). + +### 1.2 Legacy functionality to reproduce (the no-regression bar) +From `datasource/hudi/**` + `PhysicalPlanTranslator.visitPhysicalHudiScan` + `HudiScanNode`: +- **Time travel** `FOR TIME AS OF`: normalize `value.replaceAll("[-: ]", "")` to an instant, resolve on the + completed timeline; **reject** `FOR VERSION AS OF` with *"Hudi does not support FOR VERSION AS OF, please use + FOR TIME AS OF"* (`HudiScanNode` `:206-220`). +- **Incremental read** `@incr(begin,end)`: `LogicalHudiScan.withScanParams` builds + COW/MOR/EmptyIncrementalRelation from `hoodie.datasource.read.begin/end.instanttime`; `getIncrementalSplits` + uses `collectSplits()` (COW) / `collectFileSlices()` (MOR); `getLocationProperties` ships + `incrementalRelation.getHoodieParams()` to BE. **RO-as-RT quirk**: a COW `_ro` table with + `hoodie.query.as.ro.table=true` is read as MOR for incremental. `CheckPolicy` injects a **row-level** + predicate `_hoodie_commit_time > begin AND <= end` (gated on `LogicalHudiScan && HMSExternalTable`). +- **MVCC snapshot / freshness**: `HudiMvccSnapshot` wraps `TablePartitionValues + long timestamp`; + freshness/last-commit instant drives MTMV. +- **Partition enumeration**: `HudiScanNode.getPrunedPartitions` used fe-core `selectedPartitions` (+ dummy + partition for unpartitioned); runtime-filter partition prune via `getPartitionInfoMap`. +- **Schema evolution**: per-split `THudiFileDesc.schema_id` from `InternalSchema` + `addToHistorySchemaInfo` + (`HudiUtils.getSchemaInfo`, nested names lowercased). +- **BE descriptor**: legacy hudi rides `TTableType.HIVE_TABLE` / `THiveTable` (`HudiScanNode extends + HiveScanNode`), with storage props merging BE-format (`AWS_*`) + Hadoop-format (`fs.*`). +- **Read-only**: Doris never writes hudi; INSERT/DELETE/MERGE/EXECUTE are rejected. + +### 1.3 The iceberg sibling template (what to mirror) +- **One sibling holder**: `HiveConnector.java` `:65` `ICEBERG_CONNECTOR_TYPE="iceberg"`, `:87` `volatile + Connector icebergSibling`, `:237-252` `getOrCreateIcebergSibling()` = + `context.createSiblingConnector(ICEBERG_CONNECTOR_TYPE, IcebergSiblingProperties.synthesize(properties))`, + `:362-374` `close()` forwards to it. +- **Property synthesis**: `IcebergSiblingProperties.synthesize` copies the gateway prop map verbatim and + injects the literal `iceberg.catalog.type=hms` flavor. +- **getTableHandle divert**: `HiveConnectorMetadata.getTableHandle` `:256-257` — `if (tableType == + HiveTableType.ICEBERG) return siblingMetadata(session).getTableHandle(...)`, returning the **raw** foreign + iceberg handle (never cast/stamped). HUDI is **explicitly not diverted** (`:254-255` comment). +- **Per-handle guard-and-forward**: every gateway consumer discriminates by the gateway's OWN hive-loader type, + `if (!(handle instanceof HiveTableHandle)) return siblingMetadata(session)...` — ~30 sites in + `HiveConnectorMetadata` (`:295, :353, :546, :592, :647, :797, :853, :876, :943, :969, :1010, :1045, :1055, + :1065, :1075, :1085, :1095, :1104, :1114, :1308, :1332, :1355`, …), all resolving through the single helper + `siblingMetadata(session)` (`:209-210`, `icebergSiblingSupplier.get().getMetadata(session)`). At the + connector level: `getScanPlanProvider(handle)` (`:120-124`), `getWritePlanProvider(handle)` (`:142-146`), + `getProcedureOps(handle)` (`:160-164`) all do `handle instanceof HiveTableHandle ? this : icebergSibling`. +- **Detection**: `HiveTableFormatDetector.detect` already returns `HiveTableType.HUDI` (input-format set or + `flink.connector=hudi`); the `HUDI` enum constant exists. + +**The template's one unsolved-for delta:** it assumes **exactly one** sibling. `ConnectorTableHandle` is a +bare marker — `interface ConnectorTableHandle extends Serializable {}` (zero methods, *"Opaque table handle. +Connector implementations define their own subclasses."*). With a **second** sibling, the binary +`else → iceberg` fallthrough is wrong (a hudi handle would go to the iceberg sibling), and neither foreign +handle can be `instanceof`'d across the loader split. This is the single genuinely-new design problem (§3B). + +--- + +## 2. PhysicalHudiScan routing analysis (the connector-agnostic answer) + +**Question:** after the flip, how does a hudi-on-HMS table reach a hudi-aware scan carrying +incremental/snapshot params, given the "KEY WRINKLE" that hudi uses a distinct `PhysicalHudiScan` node? + +**Answer (HEAD-verified — the wrinkle dissolves; the flipped table rides the GENERIC path):** + +1. **Binding.** `BindRelation` builds `LogicalHudiScan` **only** for `case HMS_EXTERNAL_TABLE` gated on + `hmsTable.getDlaType() == DLAType.HUDI` (`BindRelation.java:603-609`) — i.e. only for a **legacy** + `HMSExternalTable`. The `case PLUGIN_EXTERNAL_TABLE` arm (`:623, :654-658`) **unconditionally** returns a + plain `LogicalFileScan`, carrying `unboundRelation.getScanParams()` (so `@incr`/AS-OF params are **not lost** + at the plan-tree level — resolution just moves connector-side). +2. **Post-flip table class.** When `"hms"` enters `SPI_READY_TYPES`, an hms catalog becomes a + `PluginDrivenExternalCatalog` and its hudi table is a `PluginDrivenMvccExternalTable` (the gateway declares + `SUPPORTS_MVCC_SNAPSHOT` catalog-wide) — `getType() == PLUGIN_EXTERNAL_TABLE`. So it binds to + `LogicalFileScan`, **never** `LogicalHudiScan`. +3. **Translation.** `LogicalFileScan → PhysicalFileScan → PhysicalPlanTranslator.visitPhysicalFileScan` + (`:803`), whose `table instanceof PluginDrivenExternalTable` branch (`:814-822`) routes to the **generic** + `PluginDrivenScanNode` (identical path to iceberg/paimon). `visitPhysicalFileScan` also threads + `getTableSnapshot()`/`getScanParams()` onto the node (`:865-866`), which + `PluginDrivenScanNode.pinMvccSnapshot` → `MvccUtil` resolves via the connector. +4. **The `visitPhysicalHudiScan` fail-loud is DEAD for the flipped path.** Its `PluginDrivenExternalTable` + branch with the incremental throw (`:909-910`) and the time-travel throw (`:913-914`) is **unreachable** — + nothing ever binds a `PluginDrivenExternalTable` to a `LogicalHudiScan`. **Correction to the task framing:** + `visitPhysicalHudiScan:901-919` is *not* the real flip control point; deleting those throws is a no-op for + the flipped path. The real no-regression channel is the generic MVCC seam below. + +**The connector-agnostic seam for incremental + time travel (already built, connector-agnostic, used by +paimon today):** `PluginDrivenMvccExternalTable.loadSnapshot` (`:319`) → `toTimeTravelSpec` (`:328`, mapping +`@incr` → `ConnectorTimeTravelSpec.incremental(params.getMapParams())` at `:416-417`, FOR TIME/VERSION AS OF +otherwise) → `metadata.resolveTimeTravel(session, handle, spec)` (`:347`). An **empty** resolve throws a +kind-specific user error `notFoundMessage(spec)` (`:349`) — i.e. today an `@incr`/AS-OF query on hudi would +**fail loud** (a regression, since legacy served it), not silently read latest. The resolved snapshot is +pinned onto the handle via `metadata.applySnapshot` (`:375`) before `planScan`. **Paimon fully implements +this** (`PaimonConnectorMetadata.resolveTimeTravel :498` with the `INCREMENTAL` case `:552`, `beginQuerySnapshot +:432`) — it is the exact template for hudi. + +**Consequence for the plan:** do **not** try to route a plugin hudi table to `LogicalHudiScan` (that requires a +source-specific `detect==HUDI` arm in `BindRelation` = iron-rule violation, and drags `org.apache.hudi.*` + +`HMSExternalTable` casts into fe-core). Serve hudi-on-HMS through the generic `PluginDrivenScanNode`. Re-home +incremental + time travel **connector-side** via `resolveTimeTravel`/`applySnapshot`/a pin-honoring +`planScan`. Delete the bespoke `LogicalHudiScan`/`PhysicalHudiScan`/`HudiScanNode` + the 4 fe-core +`*IncrementalRelation` classes at flip-time (§4), after porting their logic into `fe-connector-hudi`. + +**No new fe-core SPI is required** for incremental/time-travel — `ConnectorTimeTravelSpec.Kind.INCREMENTAL` +and the whole `loadSnapshot` dispatch already exist. The lift is entirely connector implementation. + +--- + +## 3. Ordered sub-steps + +Legend for each step: **Lands** (what) · **Mode** (dormant-buildable / dormant-verifiable / flip-or-delete) · +**Touch points** · **Deps** · **Risk** · **Proving test**. + +"dormant-buildable" = compiles and lands behind the closed gate but has **no live path** to prove correctness +pre-flip. "dormant-verifiable" = additionally provable by a same-loader unit test. Nothing in Groups A–D +touches `SPI_READY_TYPES`. + +### Group A — Make the hudi connector gateway-ready as a sibling + +#### HD-A0 — DV-005 guardrail (doc-only, zero risk) +- **Lands:** correct `HudiConnectorProvider` / `HudiConnector` javadocs to *"sibling-only type string via + createSiblingConnector; never a user-facing catalog type; never add to SPI_READY_TYPES"*. Add a guard comment + at `CatalogFactory.java:50` next to `SPI_READY_TYPES`. +- **Mode:** dormant-verifiable (doc). **Touch points:** connector (`HudiConnectorProvider.java:26-38`), fe-core + comment (`CatalogFactory.java:49-50`) — comment only, no logic. **Deps:** none. **Risk:** none. **Proving + test:** n/a (review). + +#### HD-A1 — Hudi sibling holder + property synthesis (mirror iceberg S1/S2) +- **Lands:** `HudiSiblingProperties.synthesize(gatewayProps)` returning a **verbatim defensive copy** (hudi + needs **no** flavor key — there is no `iceberg.catalog.type` analogue; `HudiConnector.createClient` reads + `hive.metastore.uris`/`uri` + raw `hadoop.*`/`fs.*`/`dfs.*`/`hive.*`/`s3.*` directly). Add + `HiveConnector.hudiSibling` (`volatile Connector`) + `getOrCreateHudiSibling()` = + `context.createSiblingConnector("hudi", HudiSiblingProperties.synthesize(properties))` (fail-loud when null: + *"hudi connector plugin not available"*); extend `close()` to forward to it. +- **Mode:** dormant-buildable. **Touch points:** connector-hive (`HiveConnector.java` new field + method mirror + of `:237-252`, `close` mirror of `:362-374`), new `HudiSiblingProperties` in connector-hive (do **not** import + hudi-loader constants — hardcode any literal keys, as `IcebergSiblingProperties` does). **Deps:** none. + **Risk:** low — the sibling is lazy and unreferenced until HD-B2. **Proving test:** unit — synthesize returns + a defensive copy equal to input; `getOrCreateHudiSibling` memoizes one instance and `close` forwards (use a + same-loader fake resolved through a stubbed `createSiblingConnector`). +- **Note:** the hudi plugin zip must ship to `connector_plugin_root` and **must not** be co-packaged into the + hive zip (second AWS SDK poisons S3 — same rule as iceberg). + +#### HD-A2 — Kerberos plugin-auth for the hudi sibling (verify / close) +- **Lands:** ensure the hudi sibling gets the same plugin-side UGI `doAs` treatment the hive gateway built, + rather than a plain `context::executeAuthenticated` (`HudiConnector.java:98`) which may **downgrade** a + Kerberized HMS post-flip. Either share the gateway's plugin authenticator or mirror + `HiveConnector.buildPluginAuthenticator`. +- **Mode:** dormant-buildable. **Touch points:** connector-hudi (`HudiConnector.createClient`). **Deps:** none. + **Risk:** medium — a Kerberized hudi-on-HMS regresses at flip if unaddressed; **secondary** (only affects + Kerberos deployments). **Proving test:** flip-time e2e against a Kerberized HMS (no same-loader proof). + +#### HD-A3 — Partition-value parse fidelity (snapshot-path no-regression FIX) +- **Lands:** replace `HudiScanPlanProvider.parsePartitionValues` (`:317-334`) with logic equivalent to legacy + `HudiPartitionUtils.parsePartitionValues` (`:63-89`): positional mapping when a fragment lacks `k=`, + single-column-whole-path fallback, and `FileUtils.unescapePathName` on every value. Also resolve the + **partition-source consistency** gap: `applyFilter` prunes over HMS `listPartitionNames` (hive-style + `k=v/...`) while the unpruned `resolvePartitions` fallback (`:299-307`) lists over Hudi metadata + `getAllPartitionPaths` (positional) — the two return **different-shaped** identifiers. Pick **one** + authoritative partition source keyed on `useHiveSyncPartition` (as legacy did) so the pruned name-shape + equals the `fsView.getLatest*(partitionPath)` relative-path shape. +- **Mode:** dormant-buildable (correctness only provable on BE). **Touch points:** connector-hudi + (`HudiScanPlanProvider.parsePartitionValues`, `resolvePartitions`; port the `HudiPartitionUtils` + positional/unescape logic into the connector). **Deps:** none (independent of HD-A1/B). **Risk:** **HIGH** — + this is a silent NULL-partition-column / wrong-value regression on the **most basic** non-hive-style + partitioned snapshot read; it is *not* covered by any prior batch step. **Proving test:** flip-time e2e over a + `hive_style_partitioning=false` table (path `2024/01`) and a table with an escaped partition value (space, + `/`), asserting non-NULL, correctly-unescaped partition columns vs legacy `HudiScanNode`. + +#### HD-A4 — Force-JNI session flag + COW/MOR detection hardening +- **Lands:** thread the `force_jni` session flag through `ConnectorSession` into `planScan`'s `useNative` + decision **in both directions**: (a) COW-always-native must yield to force-JNI (legacy `canUseNativeReader`); + (b) the MOR no-delta-log → native **downgrade** in `HudiScanRange.populateRangeParams` must **not** fire when + force-JNI is set (legacy guards it with `!isForceJniScanner()` — `force_jni` is a correctness escape hatch). + Harden `detectHudiTableType` so an `UNKNOWN` result cannot silently pick the MOR/JNI path for a COW table. +- **Mode:** dormant-buildable. **Touch points:** connector-hudi (`HudiScanPlanProvider.planScan`, + `HudiScanRange.populateRangeParams`, `HudiConnectorMetadata.detectHudiTableType`). **Deps:** none. **Risk:** + medium — the `planScan.useNative` decision and the `populateRangeParams` downgrade must stay **consistent** or + a slice is planned native but described JNI (or vice-versa). **Proving test:** unit for the session-flag + plumbing (same-loader, with a fake session); flip-time e2e with `force_jni=true` on COW and MOR. + +#### HD-A5 — BE table descriptor + BE-canonical storage props +- **Lands:** `HudiConnectorMetadata.buildTableDescriptor` → `TTableType.HIVE_TABLE` / `THiveTable` (copy the + gateway's `HiveConnectorMetadata` output — legacy hudi rides HIVE_TABLE). In + `HudiScanPlanProvider.getScanNodeProperties`, source BE-facing storage via + `ConnectorContext.getBackendStorageProperties()` (canonical `AWS_*`) **plus** the Hadoop passthrough + (`fs.*`) for the JNI reader — mirroring legacy `getLocationProperties`' dual merge — instead of the current + key-prefix copy, so private-bucket credentials reach BE. +- **Mode:** dormant-buildable. **Touch points:** connector-hudi (`HudiConnectorMetadata.buildTableDescriptor` + new override; `HudiScanPlanProvider.getScanNodeProperties`). **Deps:** none. **Risk:** medium — a generic + `SCHEMA_TABLE` descriptor is likely wrong BE-side; the native COW reader needs `AWS_*`, the JNI MOR reader + needs `fs.*`. Note the **be-java-ext shared classpath** rule: verify the hudi plugin zip carries what the JNI + reader needs (commons-lang / transitive deps) or BE `NoClassDefFound`. **Proving test:** flip-time e2e over an + S3-backed private-bucket hudi table (COW native + MOR JNI). +- **Simplification vs iceberg:** the **name-based** `buildTableDescriptor` path needs **no** hudi divert (unlike + iceberg's ICEBERG_TABLE divert) — legacy hudi's descriptor is identical to hive's. Only the **handle-based** + override here is needed. Do not add a redundant name-based hudi divert. + +### Group B — Gateway divert (mirror iceberg + the ONE new design decision) + +#### HD-B1 — Three-way foreign-handle routing (THE new design decision — needs sign-off) +- **Problem:** with two siblings, the binary `if (handle instanceof HiveTableHandle) hive else iceberg` breaks + — a `HudiTableHandle` (raw hudi-loader type, cast-invisible across the loader split) would be wrong-routed to + the iceberg sibling → cross-loader `ClassCastException` or silently-wrong answers. `ConnectorTableHandle` + carries no format tag (verified: zero methods). This affects **every** per-handle site: the ~30 + `siblingMetadata(session)` forwards in `HiveConnectorMetadata` and the connector-level + `getScanPlanProvider/getWritePlanProvider/getProcedureOps(handle)` (+ any `beginTransaction(handle)`). +- **Two mechanisms (must pick one — this is the flip-blocking decision the plan surfaces for explicit + sign-off):** + - **Option 1 — `ownsHandle` SPI predicate (RECOMMENDED).** Add a neutral parent-first default to the + `Connector` interface: `default boolean ownsHandle(ConnectorTableHandle h) { return false; }`, overridden by + each sibling with its **own** in-loader `instanceof` (`IcebergConnector`: `h instanceof + IcebergTableHandle`; `HudiConnector`: `h instanceof HudiTableHandle`). Gateway routes: + `hive-handle → hive; else icebergSibling.ownsHandle(h) → iceberg; else hudiSibling.ownsHandle(h) → hudi; + else fail-loud`. fe-core **never** calls `ownsHandle` (stays format-agnostic). + - *Cost:* one new parent-first SPI default method. + - *Benefit:* the routing **logic** is **same-loader unit-testable** — each fake overrides `ownsHandle` with + its own class check, and two distinct app-loader fake handle classes are distinguishable. This aligns with + the project's dormancy-testing discipline (every dormant step gets a same-loader unit test). + - **Option 2 — classloader-identity routing (zero new SPI).** Capture each sibling's loader + (`icebergSibling.getClass().getClassLoader()`, `hudiSibling.getClass().getClassLoader()`) and route a + foreign handle by `handle.getClass().getClassLoader() == thatLoader`. Connector-agnostic, keeps the + raw-handle-return invariant, **no** SPI change. + - *Cost / feasibility hazard:* the routing is **NOT** same-loader unit-testable — in a same-loader test both + fakes are app-loader classes, so `handle.getClass().getClassLoader()` collapses to the **same** loader and + the discriminator cannot distinguish them. Correctness is only provable with a two-URLClassLoader fixture + or a flip-time redeploy smoke. +- **Recommendation:** **Option 1 (`ownsHandle`)** — the one-method SPI cost buys dormant-verifiability of the + single most correctness-critical routing seam, which is otherwise invisible until the flip. Option 2 is the + fallback if adding an SPI method is rejected. **Either way**, the gateway's negative/hive test stays on the + gateway's OWN `HiveTableHandle` (loader-safe), and the cross-loader non-CCE guarantee still needs a + two-URLClassLoader fixture (§4). +- **Lands:** replace the single `icebergSiblingSupplier` in `HiveConnectorMetadata` (`:187`) with a handle-aware + resolver (`Function` or a `siblingMetadata(session, handle)` that + dispatches via the chosen mechanism); change the ~30 forward sites to route by handle; change the three + connector-level `HiveConnector.get*Provider(handle)` (`:120-124, :142-146, :160-164`) to the 3-way resolver. +- **Mode:** dormant-verifiable (Option 1) / dormant-buildable (Option 2). **Touch points:** fe-connector-api + (`Connector.java` — one new default, Option 1 only), connector-hive (`HiveConnector`, `HiveConnectorMetadata` + — refactor the live iceberg spine), connector-hudi + connector-iceberg (`ownsHandle` overrides, Option 1). + **Deps:** HD-A1 (needs the hudi sibling holder to reference). **Risk:** **HIGH / flip-blocker** — this + **mutates the already-landed, shared iceberg delegation spine** that the iceberg W-steps depend on; sequence + so existing iceberg per-handle behavior/tests are preserved (the iceberg arm must be byte-behavior-identical + after the refactor). **Proving test:** Option 1 — same-loader unit test with an iceberg-fake and hudi-fake + each overriding `ownsHandle`, asserting each foreign handle routes to its own sibling and a hive handle stays + hive; **plus** a two-URLClassLoader fixture (shared with the iceberg S5 plan) proving no CCE across the loader + split. Option 2 — two-URLClassLoader fixture only. + +#### HD-B2 — getTableHandle HUDI divert + sibling lifecycle (the arming pivot — LAST connector step) +- **Lands:** add, adjacent to the iceberg arm, `if (tableType == HiveTableType.HUDI) return + hudiSiblingMetadata(session).getTableHandle(session, dbName, tableName);` in + `HiveConnectorMetadata.getTableHandle` (`:256`), returning the **raw** `HudiTableHandle`. Keep the + `UNKNOWN && !view` fail-loud below intact. This is the first site that produces a foreign hudi handle, which + lights up the 3-way guard-and-forward everywhere. +- **Mode:** dormant-buildable (dead until `"hms"` enters `SPI_READY_TYPES`). **Touch points:** connector-hive + (`HiveConnectorMetadata.getTableHandle`). **Deps:** **HD-B1 strictly first** (else the foreign hudi handle + mis-routes to iceberg on every per-handle call), **and** all of Group C + HD-D1 complete (else diverting HUDI + regresses incremental/time-travel/MVCC/write-reject at flip). **Risk:** high — this is the pivot; ordering is + load-bearing. **Proving test:** same-loader unit — with the hudi sibling present, `getTableHandle` on a + HUDI-detected table returns the sibling's raw handle (not a `HiveTableHandle`); the gateway/detector must + agree with the sibling's own COW/MOR detection (`HiveTableFormatDetector` is exact-input-format match; + `HudiConnectorMetadata.detectHudiTableType` is substring — ensure they never disagree on HUDI-vs-UNKNOWN). +- **Detection order:** iceberg is checked before hudi (a table carrying both resolves iceberg — legacy parity, + already in the detector). Do not let the HUDI arm swallow the genuine-UNKNOWN fail-loud. + +### Group C — No-regression gap closure (all connector-side; all before HD-B2 / flip) + +All Group C steps build a shared spine: a **pin field on `HudiTableHandle`** (e.g. `queryInstant` + +`incrementalWindow`/`hoodieParams`) that `applySnapshot` stamps and `planScan` honors. Build that spine once +(shared by HD-C2/C3). + +#### HD-C1 — MVCC surface: beginQuerySnapshot / getMvccPartitionView / listPartitions / freshness (DV-007) +- **Lands:** on `HudiConnectorMetadata`: + - `beginQuerySnapshot` → pin the **latest completed instant** as a snapshot-id-kind `ConnectorMvccSnapshot` + (mirroring legacy `HudiUtils.getLastTimeStamp`). **Must NOT** set `lastModifiedFreshness=true` (hudi + instants are monotonic snapshot-ids; that flag routes to the hive-style on-demand freshness probe hudi does + not implement). + - `getMvccPartitionView` → empty (hudi is LIST-partitioned, not RANGE — the generic path then uses + `listPartitions`, exactly like paimon). + - `listPartitions` / `listPartitionNames` → port `HudiUtils.getPartitionValues` + + `HudiPartitionUtils.getAllPartitionNames` + `useHiveSyncPartition`, returning per-partition + `lastModifiedMillis` = the instant (so `getPartitionSnapshot` takes the `MTMVSnapshotIdSnapshot` branch, not + the `-1` sentinel that over-refreshes MTMV). + - `getTableFreshness` / `getPartitionFreshnessMillis` → last-commit instant (decide the freshness model — + snapshot-instant vs last-modified — from legacy `HudiDlaTable`). +- **Mode:** dormant-buildable. **Touch points:** connector-hudi (`HudiConnectorMetadata` new overrides; port + `HudiPartitionUtils`/`HudiUtils` partition-listing fragments). **Deps:** none (independent of the pin spine). + **Risk:** high — without this a partitioned hudi table's MVCC view is empty → the table looks UNPARTITIONED + with a `-1` snapshot id: partition pruning, `selectedPartitionNum`, and MTMV freshness all break vs legacy. + `listLatestPartitions` renders partition items via `HiveUtil.toPartitionValues` with a + `checkState(values.size()==types.size())` — the partition-name format must match the column count or it + throws. **Secondary consequence (elevate to scalability no-regression):** the generic `PluginDrivenScanNode` + batch-mode split throttling keys off `setSelectedPartitions(fileScan.getSelectedPartitions())`, populated + from the MVCC `nameToPartitionItem` — **empty** without `listPartitions` → a large partitioned hudi table + never engages batch mode and `planScan` eagerly enumerates every partition's file slices in one call + (planning-time OOM risk that legacy `HudiScanNode` throttled). **Proving test:** flip-time e2e — partitioned + hudi table shows correct `selectedPartitionNum` under a partition predicate; hudi MTMV detects a new commit; + a many-partition table plans without OOM. +- **MTMV scope decision (needs user input):** if hudi MTMV base tables are in the no-regression bar, freshness + must be real; else document the deferral explicitly (check legacy hudi MTMV support before deciding). + +#### HD-C2 — Time travel (FOR TIME AS OF) via resolveTimeTravel + pin-honoring planScan +- **Lands:** `HudiConnectorMetadata.resolveTimeTravel`: `Kind.TIMESTAMP` → normalize + `value.replaceAll("[-: ]", "")`, validate the instant exists on the completed timeline, return a + `ConnectorMvccSnapshot` carrying the pinned instant (empty return when not-found so fe-core renders + `notFoundMessage`; a `DorisConnectorException` for TZ/parse propagates as-is). `Kind.SNAPSHOT_ID` / + `VERSION_REF` → **fail loud** with the exact legacy message *"Hudi does not support FOR VERSION AS OF, please + use FOR TIME AS OF"* (align tests to this wording; do not reproduce per-action tricks). `applySnapshot` stamps + the pinned instant on the `HudiTableHandle`. `HudiScanPlanProvider.planScan` uses `handle.queryInstant` when + present instead of `timeline.lastInstant()` (`:108`), feeding + `getLatestBaseFilesBeforeOrOn`/`getLatestMergedFileSlicesBeforeOrOn` (both already take an instant — COW & MOR + pin support is a plumb-through, not new merge logic). +- **Mode:** dormant-buildable. **Touch points:** connector-hudi (`HudiConnectorMetadata.resolveTimeTravel` / + `applySnapshot` new; `HudiTableHandle` new `queryInstant` field; `HudiScanPlanProvider.planScan`). **Deps:** + the pin spine (shared with HD-C3); HD-C1's `beginQuerySnapshot` (the latest-instant implicit pin makes a + non-time-travel read byte-identical). **Risk:** medium — the `[-:space]` strip is load-bearing (lexicographic + instant compare); schema-at-instant intersects DV-006 (see HD-C4/C5). **Proving test:** unit for the + VERSION-reject message and timestamp normalization (same-loader); flip-time e2e `FOR TIME AS OF` on COW and + MOR vs legacy. + +#### HD-C3 — Incremental read (@incr) — port IncrementalRelation + close the COW row-level filter +- **Lands:** port `COWIncrementalRelation`, `MORIncrementalRelation`, `EmptyIncrementalRelation`, + `IncrementalRelation` from fe-core `datasource/hudi/source/` **into** `fe-connector-hudi` (recoding split + output to `HudiScanRange`, and re-homing the fe-core helpers they pull in — `LocationPath`, `TableFormatType`, + `HudiPartitionUtils` — **this is a real port, not a verbatim lift**, and the construction driver + `LogicalHudiScan.withScanParams` is HMSExternalTable-coupled and must be re-implemented from the connector's + own metaclient/serde params). Add `HudiConnectorMetadata.resolveTimeTravel` `Kind.INCREMENTAL`: validate the + begin/end window against the timeline, carry begin/end + `hoodie.*` passthrough on the snapshot properties; + `applySnapshot` threads the window onto `HudiTableHandle`; `planScan` branches to incremental split collection + (COW `collectSplits()` / MOR `collectFileSlices()` — dispatch on table type, each throws on the wrong type as + legacy `getIncrementalSplits` does), honoring `fallbackFullTableScan()` → degrade to a normal snapshot scan + (not error); emit `hoodie.datasource.read.begin/end.instanttime` into `getScanNodeProperties`. Reproduce the + **RO-as-RT** quirk connector-side (a COW `_ro` table read as MOR for incremental). `@incr` lists **LATEST** + partitions + LATEST schema (do **not** pin schema for incremental). +- **Mode:** mixed (port = dormant-buildable; correctness = flip-time e2e). **Touch points:** connector-hudi + (new ported relation classes; `HudiConnectorMetadata.resolveTimeTravel`/`applySnapshot`; + `HudiScanPlanProvider.planScan` incremental branch; `HudiScanRange` for MOR JNI params). **Deps:** the pin + spine (HD-C2); HD-A3 (partition parse) for correct partition identifiers. **Risk:** **HIGHEST correctness + landmine — see §5.1.** The COW `_hoodie_commit_time > begin AND <= end` **row-level** predicate that legacy + injected via `CheckPolicy` is **skipped** on the generic `LogicalFileScan` path, and the iron rule forbids + re-adding it in fe-core. Because a COW update **rewrites the whole base file** (carrying unchanged older-commit + rows forward), file-level selection alone returns **out-of-window rows = silent wrong results**. The connector + must push a `_hoodie_commit_time` range predicate through the scan-range/thrift, OR BE's native/JNI reader + must honor the begin/end instant window at **row** granularity — **this must be validated by e2e, not + assumed** (Paimon is **not** a precedent here — its incremental is snapshot-delta, not row-filtering over + carried-forward base rows). **Proving test:** flip-time e2e `@incr(begin,end)` on COW and MOR, with a + deliberately-built fixture whose base file **straddles the window boundary** (older rows carried forward), + asserting only in-window rows return, vs legacy `HudiScanNode`. + +#### HD-C4 — Schema evolution / schema_id / history_schema_info / field-ids (DV-006) +- **Lands:** port field-id onto `HudiColumnHandle` (from Hudi `InternalSchema`), per-commit `InternalSchema` + version tracking (`HudiSchemaCacheValue.getCommitInstantInternalSchema`), an Avro/`InternalSchema` → + `TColumnType`/`TField` converter (port `HudiUtils.getSchemaInfo`), and a + `getScanNodeProperties`/`populateScanLevelParams` hook that sets `current_schema_id` + `history_schema_info` + and per-split `THudiFileDesc.schema_id`. **Every** `history_schema_info` nested `TField.name` **must be + lowercased at every level** (MEMORY: `history_schema_info-lowercase-nested-names`). Tie schema-at-instant to + the time-travel pin (`getTableSchema(pinnedHandle, snapshot)`). Also decide `getTableAvroSchema(true)` (meta + fields) vs the current no-arg (`HudiScanPlanProvider.java:115`, DV-008 gap-2). +- **Mode:** dormant-buildable (correctness only on BE). **Touch points:** connector-hudi (`HudiColumnHandle`, + `HudiConnectorMetadata` schema resolution, `HudiScanRange`/`HudiScanPlanProvider` scan-level params). **Deps:** + HD-C2 (schema-at-instant). **Risk:** hardest sub-area; **scoping decision for the user** (see §4/§5.2). **Do + NOT** ship the half-fix `current==file==-1` schema_id — DV-006 records it degrades to BE `ConstNode` + (case-sensitive) which is **strictly worse** than emitting nothing (BE then uses lowercased + `by_parquet_name`/`by_orc_name` name-match). **Proving test:** flip-time e2e over a schema-evolved + (rename/reorder) hudi table + a mixed-case nested-struct-field table, on real BE (see §5.2 for the + SIGABRT-direction unknown). + +#### HD-C5 — JNI column lists at the pinned instant (time-travel MOR correctness) +- **Lands:** `HudiScanPlanProvider.planScan` currently resolves JNI `column_names`/`column_types` from + `getTableAvroSchema()` (no-arg, latest, no meta fields) **once per scan** (`:114-120`). Under time travel a + pinned-instant MOR/JNI read would ship **LATEST** column lists (schema-at-instant mismatch). Re-derive JNI + columns at the pinned instant (and honor the meta-field decision from HD-C4). Keep `column_names`/`types` as + typed **lists** (un-joined) so comma-bearing hive types are not shattered. +- **Mode:** dormant-buildable. **Touch points:** connector-hudi (`HudiScanPlanProvider.planScan`, + `HudiScanRange`). **Deps:** HD-C2 (pin), HD-C4 (schema/meta-field decision). **Risk:** medium — only bites + pinned-instant MOR/JNI reads over an evolved schema. **Proving test:** flip-time e2e `FOR TIME AS OF` on a + schema-evolved MOR table. + +### Group D — Write-reject safety net (hudi is read-only) + +#### HD-D1 — Explicit read-only reject for hudi write / procedure / DDL +- **Problem:** once HD-B2 diverts HUDI to a foreign handle, that handle is non-hive, so the gateway's + per-handle `getWritePlanProvider(handle)` (`:142-146`), `getProcedureOps(handle)` (`:160-164`), and + `supportedWriteOperations(handle)` (SPI default derives from `getWritePlanProvider(handle)`) — plus the + name-based DDL diverts (createTable/dropTable) — would, under the **binary** discriminator, forward it to the + **iceberg** sibling → a confusing iceberg error or a bad write instead of the legacy read-only rejection. +- **Lands:** under HD-B1's 3-way routing, the hudi arm of the gateway's write/procedure/DDL methods must **not** + forward to any sibling: `supportedWriteOperations(hudiHandle)` → empty (so the generic INSERT/DML gates in + `PhysicalPlanTranslator` throw the standard "does not support INSERT/row-level DML"), + `getWritePlanProvider(hudiHandle)` → throw an explicit *"hudi tables are read-only in this catalog"* + `DorisConnectorException` (defensive — do not rely on the sibling returning null), + `getProcedureOps(hudiHandle)` → null (no procedures, like plain-hive). No `HudiWritePlanProvider` is ever + built. `beginTransaction` for a hudi handle must never be reached (writes rejected before txn open). +- **Mode:** dormant-buildable. **Touch points:** connector-hive (`HiveConnector` write/procedure/txn per-handle + methods; `HiveConnectorMetadata` DDL divert sites). **Deps:** HD-B1. **Risk:** medium — relying on "the hudi + sibling has no writer" is fragile (that only helps if hudi handles are routed **to** the hudi sibling, not to + iceberg); route to an **explicit** reject. **Proving test:** same-loader unit — a hudi-fake handle on the + write/procedure paths yields the explicit reject, not an iceberg-sibling call; flip-time e2e asserts hudi + INSERT/DELETE/MERGE/EXECUTE all fail loud read-only on a heterogeneous HMS catalog (MEMORY: + `hms-iceberg-delegation-needs-e2e` — hudi's e2e is READ parity + write-reject assertion, since hudi is + read-only). + +### 3.x Dependency summary +- HD-A1 → HD-B1 (routing needs the sibling holder). +- HD-B1 **strictly before** HD-B2 (else the first foreign hudi handle mis-routes to iceberg). +- All of Group C + HD-D1 **before** HD-B2 / flip (else the flip regresses live hudi users). +- HD-C2 and HD-C3 share the `HudiTableHandle` pin spine — build once. +- HD-C1 (`listPartitions`) and HD-C3 (incremental) share ported `HudiPartitionUtils`/`HudiUtils` fragments — + port once. +- Port HD-C3/HD-C4 logic **strictly before** the flip-time deletion of fe-core `datasource/hudi/source/*` (the + `*IncrementalRelation` classes are the reference logic; deleting first loses the source of truth). +- HD-B2 is the **last** connector step (arming pivot); the flip must not precede it. + +--- + +## 4. Flip-time residuals + hard flip-blockers + +### 4.1 The flip (one line, all-or-nothing) +Add `"hms"` to `CatalogFactory.SPI_READY_TYPES` (`:49-50`). This converts **hive + iceberg-on-HMS + +hudi-on-HMS simultaneously** — hudi **cannot** be flipped independently. Consequence: the already-complete +iceberg sibling spine is **not enough**; the whole hms flip is now gated on hudi's no-regression build-out. + +### 4.2 Hard flip-blockers (must be GREEN before adding `"hms"`) +- **B1 — 3-way foreign-handle routing (HD-B1).** The binary `else → iceberg` is **actively wrong** once a hudi + handle exists. +- **B2 — time travel (HD-C2) + incremental (HD-C3).** Legacy supports both; without the connector impl, + `resolveTimeTravel` returns empty → `loadSnapshot` **fails loud** on a legacy-supported query = regression. + Includes the **COW row-level `_hoodie_commit_time`** correctness (§5.1) — the single biggest risk. +- **B3 — MVCC / listPartitions (HD-C1).** Else a partitioned hudi table has an empty MVCC view + `-1` snapshot + → pruning / `selectedPartitionNum` / MTMV all break; plus batch-mode / OOM regression. +- **B4 — read-only write/procedure reject (HD-D1).** Else hudi DML mis-routes to the iceberg sibling. +- **B5 — getTableHandle HUDI divert + sibling lifecycle (HD-B2).** +- **B6 — snapshot-path partition-value fidelity (HD-A3).** Silent NULL/wrong partition columns on the most + basic non-hive-style partitioned read. +- **Crash-avoidance (must-fix IF HD-C4 is implemented, independent of DV-006 scope):** any emitted + `history_schema_info` nested `TField.name` must be lowercase (MEMORY) — a mixed-case nested name can drive BE + `StructNode.children.at()` → `std::out_of_range` → whole-DB SIGABRT. Note the **direction unknown** in §5.2: + the safe baseline may be *emit nothing*, in which case this is a property of the fix, not of the flip. +- **Secondary (verify):** Kerberos plugin-auth for the hudi sibling (HD-A2). + +### 4.3 Residuals needing explicit user scoping (may or may not gate the flip) +- **DV-006 schema evolution (HD-C4/C5).** A **narrow** regression: only schema-evolved (rename/reorder) hudi + tables degrade; plain and non-evolved reads are unaffected under the safe baseline. **Not** among the three + named no-regression items (incremental / time-travel / MVCC). User decides whether schema-evolved hudi tables + are in the no-regression target (see §5.2). +- **DV-008 gap-2**: `getTableAvroSchema(true)` meta-field inclusion (column-set difference). +- **Runtime-filter partition prune** (`getPartitionInfoMap` → per-split partition-value map): a pruning + speedup, not a correctness gap (BE still filters) — deferrable. +- **MTMV freshness model** for hudi (HD-C1) — scope with the DV-006 decision. + +### 4.4 Post-flip dead-code deletion (flip-or-delete-time; port first) +Reachable **only** while a legacy `HMSExternalCatalog` with `dlaType==HUDI` exists — zero live callers after +the flip; delete alongside the legacy hive/iceberg deletion: +- fe-core: `LogicalHudiScan`, `PhysicalHudiScan`, `LogicalHudiScanToPhysicalHudiScan`, the + `visitPhysicalHudiScan` method (incl. its dead `PluginDrivenExternalTable` branch + fail-loud throws + `PhysicalPlanTranslator.java:895-925`), the `BindRelation.java:603-609` `dlaType==HUDI` arm, the + `CheckPolicy` `_hoodie_commit_time` predicate injection, `RuleSet`/`AggregateStrategies` hudi entries, + `datasource/hudi/source/*` (`HudiScanNode` + `COW/MOR/Empty IncrementalRelation` + `HudiSplit`), + `datasource/hudi/*` (`HudiMvccSnapshot`, `HudiExternalMetaCache`, `HudiUtils`, `HudiPartitionUtils`, + `HudiSchemaCache*`), `HudiDlaTable`. P3 tracks these as batch-E/T10 (~2403 LOC). +- **Guard:** do not delete `datasource/hudi/source/*` until HD-C3/HD-C4 have ported the reference logic. + +--- + +## 5. Open risks / unknowns + +### 5.1 (BIGGEST) COW-incremental row-level `_hoodie_commit_time` filter — possibly unfixable under the iron rules +Legacy injects `_hoodie_commit_time > begin AND <= end` via `CheckPolicy`, gated on `LogicalHudiScan && +HMSExternalTable`. The flipped hudi table rides the generic `LogicalFileScan` path, so this injection is +**skipped**, and the iron rule forbids a `PluginDrivenExternalTable` arm in `CheckPolicy`/fe-core. In COW, an +update **rewrites the whole base file**, copying unchanged (older-commit) rows forward; a base file selected by +commit-range file selection therefore contains rows **below** the window. Without the row predicate, COW +incremental returns **out-of-window rows = silent wrong results**. **The unknown:** whether BE's native/JNI +Hudi reader applies the begin/end instant window at **row** granularity for COW base files when given +`hoodie.datasource.read.begin/end.instanttime`. If it does not, there is **no iron-rule-compliant fe-core fix** +— the connector must push the predicate through the scan-range/thrift, or BE must do it natively. This is +simultaneously a correctness landmine, possibly unfixable under the constraints, not unit-testable, and only +detectable with a purpose-built straddling-base-file fixture. **Action:** resolve this **first** in HD-C3 by +inspecting the BE Hudi reader (BE tree is absent from this FE checkout — verify against the BE source before +committing to an approach), and build the straddling fixture deliberately. + +### 5.2 Schema-absence SIGABRT direction — recon agents contradicted; reconciled but BE-unverified here +Recon agent 1 claimed *absence* of `history_schema_info` risks a BE `StructNode` `out_of_range` SIGABRT on +mixed-case nested fields (a flip-blocker). The critics reconciled this against BE code +(`table_schema_change_helper.h/.cpp`): when `!params.__isset.history_schema_info` (the connector's current +emit-nothing state), BE uses `by_parquet_name`/`by_orc_name` which **lowercases** file field names → +**case-insensitive** name match, **no** `ConstNode`, **no** SIGABRT. The `ConstNode` `out_of_range` exposure is +reachable **only** via the field-id path, which requires `history_schema_info` to **be set**. **Working +conclusion:** *emitting nothing is the safe baseline*; DV-006 is a **narrow evolved-schema regression, not a +flip-blocker crash** for plain or mixed-case reads; the lowercase-nested-name discipline is a **must-IF-HD-C4-is- +implemented** guard, not a must-fix-before-flip. **Unknown / caveat:** the **BE tree is not present in this +checkout**, so this reconciliation could not be re-verified from source here — **verify against BE +`table_schema_change_helper.*` before finalizing HD-C4 scope**, and treat the SIGABRT-direction as confirmed +only after that read. + +### 5.3 PhysicalHudiScan routing — RESOLVED (not an open risk) +§2 answers it and HEAD confirms it: a flipped hudi-on-HMS table is a `PluginDrivenExternalTable` → +`LogicalFileScan` → `PhysicalFileScan` → `PluginDrivenScanNode`; `visitPhysicalHudiScan`'s plugin branch is +dead post-flip. **Affirm, do not treat as open.** The residual risk is only that a reviewer "fixes" the missing +`PhysicalHudiScan` by re-adding a `dlaType`/`instanceof` arm to `BindRelation` — an iron-rule violation to be +called out in review. + +### 5.4 Dormant-buildable ≠ dormant-verifiable +Every H-step compiles behind the closed gate, but two classes of correctness have **no live path** pre-flip: +- **HD-B1 routing correctness** across the loader split — invisible to same-loader tests. Mechanism choice + decides how invisible (§3B): `ownsHandle` makes the routing **logic** same-loader unit-testable; + classloader-identity does not (both fakes share the app loader). Either way the cross-loader non-CCE guarantee + needs a **two-URLClassLoader fixture**. +- **HD-C1..C5 runtime correctness** (instant-pinned reads, incremental windows, MOR log-merge, schema + evolution, partitioned pruning, partition-value fidelity) — method-unit-testable in isolation, but + end-to-end correctness requires **BE + real hudi tables = flip-time e2e only** (no standalone `type=hudi` + catalog exists to exercise the sibling pre-flip). + +**Pre-committed non-unit harnesses (the literal flip gate, per MEMORY `hms-iceberg-delegation-needs-e2e`):** +(1) a **two-URLClassLoader routing fixture** proving iceberg/hudi foreign handles route to the correct sibling +without CCE (shared with the iceberg S5 plan); (2) a **heterogeneous-HMS e2e** over COW+MOR snapshot reads, +`FOR TIME AS OF`, `@incr(begin,end)` on **both** COW and MOR (incl. the straddling-base-file fixture), +schema-evolved + mixed-case-nested-field tables, non-hive-style + escaped partition values, and partitioned +pruning — asserting **byte-identical** results vs the legacy `HudiScanNode` path (same tables) **and** vs an +independent hudi read; plus hudi INSERT/DELETE/MERGE/EXECUTE fail-loud read-only assertions. + +### 5.5 Refactoring the live iceberg spine (HD-B1) +Generalizing the single `icebergSiblingSupplier` + the binary per-handle discriminators to a 3-way resolver +**mutates already-landed, shared iceberg delegation code** (dormant but live-once-flipped). Sequence to +preserve the existing iceberg per-handle behavior exactly (byte-behavior-identical iceberg arm), and re-run the +iceberg sibling unit/two-loader tests after the refactor. + +--- + +## 6. References +- **iceberg sibling template:** `plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md` + (S0–S6; §2 handle-discrimination contract; §5 no-dormant-testable CCE boundary). +- **Hudi migration task:** `plan-doc/tasks/P3-hudi-migration.md` (batches A–C landed; batch-E/T10 deferrals; + ~2403 LOC dead-code inventory). +- **Deviations:** `plan-doc/deviations-log.md` — **DV-005** (standalone-type mismatch → resolved by sibling), + **DV-006** (schema_id / history_schema_info / field-ids), **DV-007** (listPartitions / MVCC), + **DV-008** (`getTableAvroSchema(true)` meta-field inclusion). +- **Public tracking issue:** apache/doris#65185 (catalog-SPI umbrella). +- **MEMORY:** `plugin-tccl-classloader-gotcha`, `history-schema-info-lowercase-nested-names`, + `hms-iceberg-delegation-needs-e2e`, `plugindriven-no-source-specific-code`, `no-property-parsing-in-fecore`. + +### Key HEAD source anchors (verified at `75670ae4193`) +- `fe/fe-core/.../datasource/CatalogFactory.java:49-50` (SPI_READY_TYPES; the flip line). +- `fe/fe-connector/fe-connector-hive/.../HiveConnector.java:65,87,99,120-124,142-146,160-164,237-252,362-374`. +- `fe/fe-connector/fe-connector-hive/.../HiveConnectorMetadata.java:187,209-210,256-257` + ~30 + `!(handle instanceof HiveTableHandle)` forward sites. +- `fe/fe-connector/fe-connector-hive/.../IcebergSiblingProperties.java` (synthesize template). +- `fe/fe-connector/fe-connector-hudi/.../HudiConnectorProvider.java:36-38` (getType; misleading javadoc `:30`). +- `fe/fe-connector/fe-connector-hudi/.../HudiConnector.java:60,65,80-100,103`. +- `fe/fe-connector/fe-connector-hudi/.../HudiConnectorMetadata.java` (overrides only getTableSchema `:199` + among the MVCC/time-travel set; `applyFilter`, `detectHudiTableType`). +- `fe/fe-connector/fe-connector-hudi/.../HudiScanPlanProvider.java:92,103-108,115,144,204-208,232-249,284-311, + 317-334` (lastInstant pin gap; parsePartitionValues infidelity). +- `fe/fe-core/.../datasource/hudi/HudiPartitionUtils.java:63-89` (legacy positional + unescape parse). +- `fe/fe-core/.../nereids/rules/analysis/BindRelation.java:603-609` (HUDI→LogicalHudiScan HMS-only), + `:623,654-658` (PLUGIN_EXTERNAL_TABLE → LogicalFileScan). +- `fe/fe-core/.../nereids/glue/translator/PhysicalPlanTranslator.java:803-822` (generic PluginDrivenScanNode), + `:865-866` (scanParams/snapshot threading), `:895-925` (dead visitPhysicalHudiScan plugin branch + fail-loud). +- `fe/fe-core/.../datasource/PluginDrivenMvccExternalTable.java:154,161,263,319,328,347,349,353,364,375, + 397-417` (the connector-agnostic MVCC/time-travel/incremental seam). +- `fe/fe-connector/fe-connector-paimon/.../PaimonConnectorMetadata.java:432,498,552` (working resolveTimeTravel + + INCREMENTAL template). +- `fe/fe-connector/fe-connector-api/.../Connector.java:66-67,87-88,107-108,187-188` (per-handle SPI defaults; + **no** `ownsHandle` today — HD-B1 Option 1 adds it). +- `fe/fe-connector/.../ConnectorTableHandle.java` (bare marker, zero methods). \ No newline at end of file From 19ca118e7c568050feead39128f683840ded02b6 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 19:25:37 +0800 Subject: [PATCH 089/334] [feat](catalog) HMS cutover write-chain WC2: map "hms" engine for CREATE + display (dormant) Adds three switch cases so a flipped HMS external catalog (a PluginDrivenExternalCatalog with getType()=="hms") resolves the right engine names. All strictly unreachable pre-flip (Class A): "hms" is not in CatalogFactory.SPI_READY_TYPES, so a type=hms catalog is still a legacy HMSExternalCatalog / HMSExternalTable and none of these cases is entered until the flip. - CreateTableInfo.pluginCatalogTypeToEngine: case "hms" -> ENGINE_HIVE (CREATE engine; mirrors legacy paddingEngineName/checkEngineWithCatalog instanceof HMSExternalCatalog arm). hive != iceberg, so the getEffectiveIcebergFormatVersion arm stays inert for hms. - PluginDrivenExternalTable.getEngine(): case "hms" -> HMS_EXTERNAL_TABLE.toEngineName() == "hms" (DISPLAY engine; legacy HMSExternalTable showed "hms", NOT "hive" -- distinct from the CREATE engine). Falling through to "Plugin" would regress SHOW TABLE STATUS / information_schema.tables. - PluginDrivenExternalTable.getEngineTableTypeName(): case "hms" -> HMS_EXTERNAL_TABLE.name() (keeps the pluginCatalogTypeToEngine sync invariant documented at CreateTableInfo.java). Dormant unit tests (mock getType()=="hms", no actual flip): CreateTableInfoEngineCatalogTest +5 (no-ENGINE pads hive, CTAS pads hive, wrong explicit engine rejected, hive accepted, iceberg-format non-interference lock); PluginDrivenExternalTableEngineTest +2 (engine "hms" and type name HMS_EXTERNAL_TABLE). fe-core BUILD SUCCESS; 31 tests pass (0 fail); checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../datasource/PluginDrivenExternalTable.java | 9 +++ .../plans/commands/info/CreateTableInfo.java | 5 ++ .../PluginDrivenExternalTableEngineTest.java | 20 +++++ .../CreateTableInfoEngineCatalogTest.java | 80 +++++++++++++++++++ 4 files changed, 114 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index d9672e4ccf6548..91367a332d3a74 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -1015,6 +1015,13 @@ public String getEngine() { // TableType.PAIMON_EXTERNAL_TABLE.toEngineName() returns "paimon", // preserving the legacy PaimonExternalTable engine name. return TableType.PAIMON_EXTERNAL_TABLE.toEngineName(); + case "hms": + // Post-flip an HMS external catalog is a PluginDrivenExternalCatalog (type "hms"); + // legacy HMSExternalTable displayed engine "hms" (TableType.HMS_EXTERNAL_TABLE.toEngineName()), + // NOT "hive" — the CREATE-TABLE engine (CreateTableInfo.pluginCatalogTypeToEngine -> "hive") + // is a separate concern. Falling through to "Plugin" would regress SHOW TABLE STATUS / + // information_schema.tables. + return TableType.HMS_EXTERNAL_TABLE.toEngineName(); default: return super.getEngine(); } @@ -1037,6 +1044,8 @@ public String getEngineTableTypeName() { return TableType.MAX_COMPUTE_EXTERNAL_TABLE.name(); case "paimon": return TableType.PAIMON_EXTERNAL_TABLE.name(); + case "hms": + return TableType.HMS_EXTERNAL_TABLE.name(); default: return TableType.PLUGIN_EXTERNAL_TABLE.name(); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java index 44c2a7bb593542..ea939e2c4758d7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfo.java @@ -941,6 +941,11 @@ private static String pluginCatalogTypeToEngine(PluginDrivenExternalCatalog cata return ENGINE_PAIMON; case "iceberg": return ENGINE_ICEBERG; + case "hms": + // A flipped HMS external catalog uses the hive engine for CREATE TABLE (legacy hms + // catalogs always create hive-engine tables); the user-visible DISPLAY engine "hms" + // is a separate concern handled by PluginDrivenExternalTable.getEngine(). + return ENGINE_HIVE; default: return null; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java index aeb5afe25b7708..b99fbd0a45dd19 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableEngineTest.java @@ -133,6 +133,26 @@ public void testIcebergCatalogReturnsIcebergEngineTableTypeName() { "Iceberg catalog tables should report ICEBERG_EXTERNAL_TABLE type name"); } + @Test + public void testHmsCatalogReturnsHmsEngineName() { + // HMS cutover: after the flip an HMS external catalog is a PluginDrivenExternalCatalog (type + // "hms"); legacy HMSExternalTable displayed engine "hms" (TableType.HMS_EXTERNAL_TABLE.toEngineName()), + // NOT "hive" (that is the CREATE-TABLE engine, a separate concern). Without an "hms" case it would + // fall through to "Plugin", regressing SHOW TABLE STATUS / information_schema.tables. + // MUTATION: dropping the hms case -> "Plugin" -> red; returning "hive" -> red. + PluginDrivenExternalTable table = createTableWithCatalogType("hms"); + Assertions.assertEquals("hms", table.getEngine(), + "Hms catalog tables should report engine='hms' (legacy parity), not 'Plugin' or 'hive'"); + } + + @Test + public void testHmsCatalogReturnsHmsEngineTableTypeName() { + PluginDrivenExternalTable table = createTableWithCatalogType("hms"); + Assertions.assertEquals(TableType.HMS_EXTERNAL_TABLE.name(), + table.getEngineTableTypeName(), + "Hms catalog tables should report HMS_EXTERNAL_TABLE type name"); + } + @Test public void testTableTypeIsAlwaysPluginExternalTable() { PluginDrivenExternalTable jdbcTable = createTableWithCatalogType("jdbc"); diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java index da2a9bb36ace02..810b0fd0c4dcdd 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/info/CreateTableInfoEngineCatalogTest.java @@ -62,6 +62,9 @@ public class CreateTableInfoEngineCatalogTest { // Mirror of CreateTableInfo.ENGINE_MAXCOMPUTE (private constant). private static final String ENGINE_MAXCOMPUTE = "maxcompute"; + // Mirror of CreateTableInfo.ENGINE_HIVE (private constant) — the CREATE-TABLE engine a flipped + // hms catalog pads to. + private static final String ENGINE_HIVE = "hive"; // Iceberg catalog-level format-version property keys (literal values of iceberg SDK // CatalogProperties.TABLE_DEFAULT_PREFIX/TABLE_OVERRIDE_PREFIX + TableProperties.FORMAT_VERSION; // spelled out to avoid importing org.apache.iceberg into this nereids test package). @@ -197,6 +200,69 @@ public void jdbcPluginDrivenStillUnsupported() { "jdbc PluginDriven catalog must pass checkEngineWithCatalog (legacy pass-through parity)"); } + // --------------------------------------------------------------------------------------------- + // HMS cutover: a flipped hms external catalog is a PluginDrivenExternalCatalog (type "hms"). + // pluginCatalogTypeToEngine must map "hms" -> ENGINE_HIVE so a no-ENGINE CREATE pads engine=hive + // (legacy hms catalogs always create hive-engine tables) and the catalog-engine consistency check + // still rejects a non-hive explicit ENGINE. Class A (unreachable until "hms" enters + // SPI_READY_TYPES); getType() is mocked to "hms" to prove the switch entry without an actual flip. + // --------------------------------------------------------------------------------------------- + + @Test + public void noEnginePaddedToHiveForPluginDrivenHms() throws Throwable { + registerPluginCatalog("hms_ctl", "hms"); + CreateTableInfo info = newInfo("hms_ctl", null); + + invokePadding(info, "hms_ctl"); + + // Why: a no-ENGINE CREATE TABLE under a flipped hms catalog must auto-pad the hive engine, + // exactly as legacy HMSExternalCatalog did (paddingEngineName :913-914), instead of throwing + // "Current catalog does not support create table". MUTATION: dropping the "hms" case -> + // pluginCatalogTypeToEngine returns null -> throw -> this test fails. + Assertions.assertEquals(ENGINE_HIVE, info.getEngineName(), + "no-ENGINE CREATE TABLE on a PluginDriven hms catalog must pad engine=hive"); + } + + @Test + public void ctasNoEnginePaddedToHiveForHms() { + registerPluginCatalog("hms_ctl", "hms"); + CreateTableInfo info = newInfo("hms_ctl", null); + + // CTAS routes through validateCreateTableAsSelect, whose first action is paddingEngineName. + // The downstream validate(ctx) is heavy and not exercised here; assert only the padding side + // effect. Pre-fix, paddingEngineName throws before setting engineName. + try { + info.validateCreateTableAsSelect(Lists.newArrayList("hms_ctl"), new ArrayList<>(), + Mockito.mock(ConnectContext.class)); + } catch (Exception ignored) { + // Only the engine-padding side effect is under test here. + } + + Assertions.assertEquals(ENGINE_HIVE, info.getEngineName(), + "CTAS into a PluginDriven hms catalog must pad engine=hive via validateCreateTableAsSelect"); + } + + @Test + public void wrongExplicitEngineRejectedForPluginDrivenHms() { + registerPluginCatalog("hms_ctl", "hms"); + // Legacy HMSExternalCatalog rejected any ENGINE != hive ("Hms type catalog can only use `hive` + // engine."); the flipped PluginDriven path mirrors that via checkEngineWithCatalog + the "hms" + // switch entry. An explicit iceberg engine on an hms catalog must be rejected. + CreateTableInfo info = newInfo("hms_ctl", "iceberg"); + + Assertions.assertThrows(AnalysisException.class, () -> invokeCheck(info), + "explicit ENGINE=iceberg on a PluginDriven hms catalog must be rejected"); + } + + @Test + public void correctExplicitEngineHivePassesForPluginDrivenHms() { + registerPluginCatalog("hms_ctl", "hms"); + CreateTableInfo info = newInfo("hms_ctl", ENGINE_HIVE); + + Assertions.assertDoesNotThrow(() -> invokeCheck(info), + "explicit ENGINE=hive on a PluginDriven hms catalog must pass the check"); + } + // --------------------------------------------------------------------------------------------- // ENG-1 / F1: getEffectiveIcebergFormatVersion must consult catalog-level // table-default/override.format-version for a flipped (PluginDrivenExternalCatalog) iceberg @@ -306,4 +372,18 @@ public void catalogLevelFormatVersionIgnoredForNonIcebergPluginCatalog() throws Assertions.assertEquals(2, invokeEffectiveVersion(info), "a non-iceberg PluginDriven catalog must not consult catalog-level format-version"); } + + @Test + public void catalogLevelFormatVersionIgnoredForHmsPluginCatalog() throws Throwable { + // Non-interference lock (item 5 trap): pluginCatalogTypeToEngine("hms") returns ENGINE_HIVE, + // which is != ENGINE_ICEBERG, so the getEffectiveIcebergFormatVersion iceberg arm must NOT fire + // for a flipped hms catalog — an iceberg-on-HMS table created via the hms gateway is still a + // hive-engine CREATE and must not pick up iceberg row-lineage / format-version logic. Even if an + // hms catalog carries a table-default.format-version, it must resolve to 2. + registerPluginCatalogWithProps("hms_ctl", "hms", propMap(TABLE_DEFAULT_FORMAT_VERSION, "3")); + CreateTableInfo info = newInfoWithColumns("hms_ctl", new ArrayList<>()); + + Assertions.assertEquals(2, invokeEffectiveVersion(info), + "a PluginDriven hms catalog must not consult catalog-level format-version (hive != iceberg)"); + } } From b0114ebf91f984766c47a782e56f22e80b0e85f5 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 19:26:30 +0800 Subject: [PATCH 090/334] [doc](catalog) HMS cutover write-chain WC2 DONE (650658b4ce8); next = WC3 (explicit-partition INSERT reject) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 90eefed1608b18..567329727d36a4 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -73,8 +73,8 @@ - ⚠**W5 复审记账（休眠期不动）**：①**paimon/无-procedure 连接器 EXECUTE 现多一次 `getTableHandle` 往返**（reorder 先解析句柄再判 null-ops；happy-path iceberg 无新增往返，只错误路径先报 "Table not found"）——有意接受、非正确性回退；②`ExecuteActionFactory.getSupportedActions` 留无参 getter（SHOW 发现，无 live caller；翻闸对 hms 网关会返 hive 连接器级空——handle-aware 变体延后，无消费者故无害）；③e2e owed（休眠单测只证路由；随其它 iceberg-on-HMS 能力翻闸期统一补，memory `hms-iceberg-delegation-needs-e2e`）。 - **用户已选：写链收尾（§4.5）。权威拆解 = `hms-write-chain-decomposition-2026-07-08.md`（起步必读；code-grounded recon `wf_13e18d30-257`：3 维读者 + 完整性/排序 critic，全 HEAD 核对）**。核心 = 三类休眠 taxonomy（**A**=翻闸前不可达、**B**=值恒等、**C**=活跑靠 no-op SPI 默认——**C 类须补现存连接器无回归测**，非仅休眠 hive 单测）+ 风险升序 4 步： - **✅ WC1 DONE（`6c01805109c`）= 搬 `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY`**（B 值恒等：两字面量均 `"broker.name"`；`private→public` 成单一真源、generic base 不再依赖 HMS 子类；`.get()` 保**大小写敏感、不 harmonize**；删 HMS 副本〔HEAD 证单引用〕）。全绿（BrokerPropertiesTest 6〔+1 常量真源 pin〕 + HiveScanNodeTest 6 + checkstyle 0）。 - - **下一步 = WC2**（A 不可达、极低风险，可紧接、一 commit）：`CreateTableInfo.pluginCatalogTypeToEngine` 加 `case "hms"→ENGINE_HIVE` + `PluginDrivenExternalTable.getEngine()` 加 `case "hms"→"hms"`、`getEngineTableTypeName()` 加 `case "hms"→"HMS_EXTERNAL_TABLE"`（CREATE=hive、DISPLAY=hms）。 - - **WC3**（item 2a，C 活跑、中风险）：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松默认）+ `bindConnectorTableSink` 守卫接线 + `HiveConnectorMetadata` 覆写（hive→拒显式分区 INSERT、foreign→兄弟）；**须补 paimon/iceberg 无回归测**、文案与 e2e 逐字一致；依赖 W1–W5 兄弟管线（已在）。 + - **✅ WC2 DONE（`650658b4ce8`）= engine-map 三 case（A 不可达、极低风险，一 commit）**：`CreateTableInfo.pluginCatalogTypeToEngine` 加 `case "hms"→ENGINE_HIVE`（CREATE 引擎；镜像 legacy paddingEngineName/checkEngineWithCatalog 的 `instanceof HMSExternalCatalog` 臂；`hive != iceberg` 故 `getEffectiveIcebergFormatVersion` 臂对 hms 保持 inert，已加非干扰锁测）+ `PluginDrivenExternalTable.getEngine()` 加 `case "hms"→HMS_EXTERNAL_TABLE.toEngineName()`（== `"hms"`；**DISPLAY 引擎，legacy 显示 "hms" 非 "hive"，与 CREATE 引擎两回事**；落 "Plugin" 会退化 SHOW TABLE STATUS/information_schema.tables）、`getEngineTableTypeName()` 加 `case "hms"→HMS_EXTERNAL_TABLE.name()`（守 pluginCatalogTypeToEngine sync 不变量）。休眠成立已 HEAD 证实（`SPI_READY_TYPES` 无 `"hms"`、CatalogFactory `case "hms"` 仍建 legacy HMSExternalCatalog→无 PluginDriven 目录 getType()=="hms"→三 case 翻闸前不可达）。休眠单测：`CreateTableInfoEngineCatalogTest` +5（no-ENGINE pad hive / CTAS pad hive / 错引擎拒 / hive 过 / iceberg-format 非干扰锁）、`PluginDrivenExternalTableEngineTest` +2（engine `"hms"` + type name `HMS_EXTERNAL_TABLE`）。全绿（fe-core BUILD SUCCESS + 31 测 0 fail + checkstyle 0；纯 fe-core 无连接器改动故 import gate N/A）。 + - **下一步 = WC3**（item 2a，C 活跑、中风险）：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松默认）+ `bindConnectorTableSink` 守卫接线 + `HiveConnectorMetadata` 覆写（hive→拒显式分区 INSERT、foreign→兄弟）；**须补 paimon/iceberg 无回归测**、文案与 e2e 逐字一致；依赖 W1–W5 兄弟管线（已在）。 - **WC4**（item 1，C 活跑、最高风险、**翻闸硬前置**）：新 `ConnectorScanPlanProvider.releaseReadTransaction`（no-op 默认）+ `HiveScanPlanProvider` 覆写 deregister + `PluginDrivenScanNode.getSplits` **无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，TCCL pin 经 `onPluginClassLoader` 白拿）；不补则翻闸后 ACID 读泄漏 metastore 共享读锁。**须补现存连接器无回归测**。 - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** - **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** 关键结论： From 2ac818eceed384028fbdae8a89d8feb3920f1fa1 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 20:03:17 +0800 Subject: [PATCH 091/334] [feat](catalog) HMS cutover write-chain WC3: reject explicit-partition-spec INSERT via neutral SPI (dormant) Ports the legacy hive reject of INSERT INTO hive_tbl PARTITION(p1, p2) (the dynamic partition-NAME list form) into the connector-delegated write path. Legacy fe-core threw "Not support insert with partition spec in hive catalog." in BindSink.bindHiveTableSink; post-flip a hive write binds through bindConnectorTableSink, which never read sink.getPartitions() -- so the reject would be silently lost (a fail-loud regression). This wires it through a neutral SPI so the rejection + message stay in the connector. - fe-connector-api ConnectorWriteOps: new default no-op SPI validateWritePartitionNames(session, handle, partitionNames), mirroring the validateStaticPartitionColumns permissive-default pattern. - fe-connector-hive HiveConnectorMetadata: @Override -- foreign (iceberg-on-HMS) handle forwards to the sibling REGARDLESS of emptiness (accepts PARTITION(names) as a standalone type=iceberg catalog does, no heterogeneous-vs-standalone divergence); a hive handle THROWS the exact legacy message on a non-empty list, returns silently on empty. Foreign handle never cast. - fe-core BindSink: new private checkConnectorWritePartitionNames mirroring checkConnectorStaticPartitions (empty-guard -> resolve handle -> SPI -> DorisConnectorException mapped to AnalysisException), called in bindConnectorTableSink with sink.getPartitions(). Keyed on the dynamic partition-NAME list, NOT staticPartitionKeyValues -- a static PARTITION(col='val') INSERT and a plain INSERT ... SELECT are unaffected. Class C (live wiring, permissive no-op default): the fe-core call runs today on live iceberg/paimon/max_compute/jdbc/es, but the empty-guard makes plain INSERT byte-unchanged and no live connector overrides the SPI (all inherit the no-op), so a non-empty PARTITION(names) on them stays silently accepted as before. The hive throw is dormant until hms enters SPI_READY_TYPES. The flip-time delete of BindSink:667-669 (with the rest of bindHiveTableSink) is a residual, not part of this step. Message byte-parity with the pre-existing e2e test_hive_write_type.groovy:250, which runs on both the legacy and the flipped connector path. Coverage mirrors the peer static-partition seam (connector unit test + e2e; no fe-core unit test precedent for bindConnectorTableSink): HiveConnectorMetadataSiblingDelegationTest +1 (hive rejects non-empty with the legacy message, empty is silent, never consults the sibling) and the EXPECTED_WRITE_METHODS Rule-9 completeness lock extended with validateWritePartitionNames (foreign forward proven regardless of emptiness). Adversarial 4-lens review (routing / message-parity / live-no-regression + dormancy / field-keying + binding-completeness) with a refute pass: 0 confirmed findings. api+hive+core BUILD SUCCESS; sibling test 8/8; checkstyle 0 (api/hive/core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/api/ConnectorWriteOps.java | 18 +++++++++ .../connector/hive/HiveConnectorMetadata.java | 22 +++++++++++ ...onnectorMetadataSiblingDelegationTest.java | 32 +++++++++++++++- .../nereids/rules/analysis/BindSink.java | 37 +++++++++++++++++++ 4 files changed, 108 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java index c421052f899392..fefacb7abaa8cd 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorWriteOps.java @@ -73,6 +73,24 @@ default void validateStaticPartitionColumns(ConnectorSession session, ConnectorT // default: no static-partition constraint } + /** + * Validates that the dynamic partition-NAME list form ({@code INSERT [OVERWRITE] ... PARTITION (p1, p2)} — a + * list of partition column NAMES with no values, distinct from the static {@code PARTITION(col=val)} spec) is + * permitted on {@code handle}, throwing a {@link DorisConnectorException} with a connector-authored message + * otherwise. Called at analysis time, before synthesizing the write plan, so the engine rejects an + * unsupported statement up front (fail loud). + * + *

    The default accepts everything: connectors that ignore the name-list form need not override. A connector + * that must reject it (e.g. hive, where {@code INSERT ... PARTITION(p1, p2)} is unsupported) MUST override + * this and throw, so the rejection — and its message — stay in the connector rather than being drafted by the + * engine. {@code partitionNames} is the list of partition column names from the PARTITION clause (the engine + * calls this only when the list is non-empty).

    + */ + default void validateWritePartitionNames(ConnectorSession session, ConnectorTableHandle handle, + List partitionNames) { + // default: no partition-name-list constraint + } + // ──────────────────── TRANSACTION ──────────────────── /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 90f93c14401044..e4c1f009357ab0 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1509,6 +1509,28 @@ public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTa // hive: no static-partition constraint (SPI default no-op). } + /** + * Rejects the dynamic partition-NAME list form ({@code INSERT ... PARTITION(p1, p2)}) on a hive table with the + * exact legacy message. UNLIKE the two permissive validators above, a hive handle here THROWS on a non-empty + * list — this is the net-new port of the legacy fe-core reject ({@code BindSink.bindHiveTableSink}), not a + * silent no-op. A foreign (iceberg-on-HMS) handle forwards to the sibling, which accepts {@code + * PARTITION(names)} exactly as a standalone {@code type=iceberg} catalog does (no heterogeneous-vs-standalone + * divergence); the forward happens regardless of emptiness (the empty-early-return is hive-only). An empty + * list returns silently for a hive handle (a plain {@code INSERT ... SELECT} or a static {@code + * PARTITION(col='val')} INSERT is legal plain-hive). Dormant until hms enters SPI_READY_TYPES. + */ + @Override + public void validateWritePartitionNames(ConnectorSession session, ConnectorTableHandle handle, + List partitionNames) { + if (!(handle instanceof HiveTableHandle)) { + siblingMetadata(session).validateWritePartitionNames(session, handle, partitionNames); + return; + } + if (partitionNames != null && !partitionNames.isEmpty()) { + throw new DorisConnectorException("Not support insert with partition spec in hive catalog."); + } + } + // ========== ConnectorWriteOps: transactions ========== /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index a2dfc436b923ec..fdd07b6e9aeac9 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -202,11 +202,35 @@ public void everyAlterDdlAndValidateMethodForwardsAForeignHandleToTheSibling() { md.replacePartitionField(null, foreignHandle, null); md.validateRowLevelDmlMode(null, foreignHandle, null); md.validateStaticPartitionColumns(null, foreignHandle, Collections.emptyList()); + // Empty list on purpose: a foreign handle must forward REGARDLESS of emptiness (the empty-early-return is + // hive-only) — this would fail if the empty check were placed before the foreign-handle divert. + md.validateWritePartitionNames(null, foreignHandle, Collections.emptyList()); Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_WRITE_METHODS, siblingMetadata.calls, "every ALTER-DDL mutator + write validator must forward a foreign handle to the sibling"); } + @Test + public void hiveHandleRejectsNonEmptyPartitionNamesWithLegacyMessage() { + HiveConnectorMetadata md = withSibling(); + HiveTableHandle hive = hiveHandle(); + + // Net-new port of the legacy fe-core reject (retired BindSink.bindHiveTableSink): the dynamic + // partition-NAME list form INSERT ... PARTITION(p1, p2) is unsupported on a hive table. UNLIKE the two + // permissive validators, a hive handle here THROWS the EXACT legacy message on a non-empty list. The e2e + // test_hive_write_type.groovy asserts on this literal substring, so it must stay byte-identical. + assertThrowsMessage(() -> md.validateWritePartitionNames(null, hive, Arrays.asList("p1", "p2")), + "Not support insert with partition spec in hive catalog."); + + // An empty list (a plain INSERT ... SELECT or a static PARTITION(col='val') INSERT) is legal plain-hive + // and MUST return silently — a throw here would newly reject legal writes. + md.validateWritePartitionNames(null, hive, Collections.emptyList()); + + Assertions.assertEquals(0, siblingConnector.getMetadataCount, + "a hive handle must never build/consult the iceberg sibling to validate partition names"); + Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle"); + } + @Test public void hiveHandleAlterDdlThrowsAndValidateIsNoopAndNeverConsultsSibling() { HiveConnectorMetadata md = withSibling(); @@ -323,7 +347,7 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata "renameTable", "addColumn", "addColumns", "dropColumn", "renameColumn", "modifyColumn", "reorderColumns", "createOrReplaceBranch", "createOrReplaceTag", "dropBranch", "dropTag", "addPartitionField", "dropPartitionField", "replacePartitionField", - "validateRowLevelDmlMode", "validateStaticPartitionColumns")); + "validateRowLevelDmlMode", "validateStaticPartitionColumns", "validateWritePartitionNames")); final List calls = new ArrayList<>(); final Optional> filterResult = @@ -564,5 +588,11 @@ public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTa List staticPartitionColumnNames) { calls.add("validateStaticPartitionColumns"); } + + @Override + public void validateWritePartitionNames(ConnectorSession session, ConnectorTableHandle handle, + List partitionNames) { + calls.add("validateWritePartitionNames"); + } } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java index 94c6c46bb08c2f..e04f1d2650bce9 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindSink.java @@ -755,6 +755,37 @@ private void checkConnectorStaticPartitions(PluginDrivenExternalTable table, } } + /** + * Connector analogue of the legacy hive partition-spec reject (retired legacy {@code bindHiveTableSink}): + * rejects the dynamic partition-NAME list form ({@code INSERT ... PARTITION(p1, p2)}) through the neutral + * {@code ConnectorMetadata#validateWritePartitionNames} SPI, so the rejection and its message stay in the + * connector (hive rejects, iceberg accepts). A connector {@link DorisConnectorException} is surfaced as the + * analysis-time {@link AnalysisException} the legacy native path threw, preserving the message and exception + * type. The handle round-trip + SPI call happen only when the list is non-empty, so a plain {@code INSERT ... + * SELECT} (empty list) is byte-unchanged for every live connector. Mirrors {@link #checkConnectorStaticPartitions}. + */ + private void checkConnectorWritePartitionNames(PluginDrivenExternalTable table, List partitionNames) { + if (partitionNames == null || partitionNames.isEmpty()) { + return; + } + if (!(table.getCatalog() instanceof PluginDrivenExternalCatalog)) { + return; + } + PluginDrivenExternalCatalog catalog = (PluginDrivenExternalCatalog) table.getCatalog(); + ConnectorSession session = catalog.buildConnectorSession(); + ConnectorMetadata metadata = catalog.getConnector().getMetadata(session); + ConnectorTableHandle handle = metadata.getTableHandle( + session, table.getRemoteDbName(), table.getRemoteName()) + .orElseThrow(() -> new AnalysisException("Table not found: " + + table.getRemoteDbName() + "." + table.getRemoteName() + + " in catalog " + catalog.getName())); + try { + metadata.validateWritePartitionNames(session, handle, partitionNames); + } catch (DorisConnectorException e) { + throw new AnalysisException(e.getMessage(), e); + } + } + private Plan bindConnectorTableSink(MatchingContext> ctx) { UnboundConnectorTableSink sink = ctx.root; Pair pair = bind(ctx.cascadesContext, sink); @@ -777,6 +808,12 @@ private Plan bindConnectorTableSink(MatchingContext bindColumns = selectConnectorSinkBindColumns( table, sink.getColNames(), staticPartitionColNames, sink.isRewrite()); LogicalConnectorTableSink boundSink = new LogicalConnectorTableSink<>( From c2aea71cdf89b9d2d69d346c0e615625a0216314 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 20:04:06 +0800 Subject: [PATCH 092/334] [doc](catalog) HMS cutover write-chain WC3 DONE (ab3b0a63ed1); next = WC4 (read-ACID query-finish commit) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 567329727d36a4..0716398fb21c77 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -74,8 +74,8 @@ - **用户已选：写链收尾（§4.5）。权威拆解 = `hms-write-chain-decomposition-2026-07-08.md`（起步必读；code-grounded recon `wf_13e18d30-257`：3 维读者 + 完整性/排序 critic，全 HEAD 核对）**。核心 = 三类休眠 taxonomy（**A**=翻闸前不可达、**B**=值恒等、**C**=活跑靠 no-op SPI 默认——**C 类须补现存连接器无回归测**，非仅休眠 hive 单测）+ 风险升序 4 步： - **✅ WC1 DONE（`6c01805109c`）= 搬 `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY`**（B 值恒等：两字面量均 `"broker.name"`；`private→public` 成单一真源、generic base 不再依赖 HMS 子类；`.get()` 保**大小写敏感、不 harmonize**；删 HMS 副本〔HEAD 证单引用〕）。全绿（BrokerPropertiesTest 6〔+1 常量真源 pin〕 + HiveScanNodeTest 6 + checkstyle 0）。 - **✅ WC2 DONE（`650658b4ce8`）= engine-map 三 case（A 不可达、极低风险，一 commit）**：`CreateTableInfo.pluginCatalogTypeToEngine` 加 `case "hms"→ENGINE_HIVE`（CREATE 引擎；镜像 legacy paddingEngineName/checkEngineWithCatalog 的 `instanceof HMSExternalCatalog` 臂；`hive != iceberg` 故 `getEffectiveIcebergFormatVersion` 臂对 hms 保持 inert，已加非干扰锁测）+ `PluginDrivenExternalTable.getEngine()` 加 `case "hms"→HMS_EXTERNAL_TABLE.toEngineName()`（== `"hms"`；**DISPLAY 引擎，legacy 显示 "hms" 非 "hive"，与 CREATE 引擎两回事**；落 "Plugin" 会退化 SHOW TABLE STATUS/information_schema.tables）、`getEngineTableTypeName()` 加 `case "hms"→HMS_EXTERNAL_TABLE.name()`（守 pluginCatalogTypeToEngine sync 不变量）。休眠成立已 HEAD 证实（`SPI_READY_TYPES` 无 `"hms"`、CatalogFactory `case "hms"` 仍建 legacy HMSExternalCatalog→无 PluginDriven 目录 getType()=="hms"→三 case 翻闸前不可达）。休眠单测：`CreateTableInfoEngineCatalogTest` +5（no-ENGINE pad hive / CTAS pad hive / 错引擎拒 / hive 过 / iceberg-format 非干扰锁）、`PluginDrivenExternalTableEngineTest` +2（engine `"hms"` + type name `HMS_EXTERNAL_TABLE`）。全绿（fe-core BUILD SUCCESS + 31 测 0 fail + checkstyle 0；纯 fe-core 无连接器改动故 import gate N/A）。 - - **下一步 = WC3**（item 2a，C 活跑、中风险）：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松默认）+ `bindConnectorTableSink` 守卫接线 + `HiveConnectorMetadata` 覆写（hive→拒显式分区 INSERT、foreign→兄弟）；**须补 paimon/iceberg 无回归测**、文案与 e2e 逐字一致；依赖 W1–W5 兄弟管线（已在）。 - - **WC4**（item 1，C 活跑、最高风险、**翻闸硬前置**）：新 `ConnectorScanPlanProvider.releaseReadTransaction`（no-op 默认）+ `HiveScanPlanProvider` 覆写 deregister + `PluginDrivenScanNode.getSplits` **无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，TCCL pin 经 `onPluginClassLoader` 白拿）；不补则翻闸后 ACID 读泄漏 metastore 共享读锁。**须补现存连接器无回归测**。 + - **✅ WC3 DONE（`ab3b0a63ed1`）= 显式分区 INSERT 拒绝（item 2a，C 活跑）**：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松 no-op 默认，镜像 `validateStaticPartitionColumns` 模式）+ `HiveConnectorMetadata` 覆写（foreign→兄弟**无论空非空**都转发〔兄弟接受 `PARTITION(names)`＝独立 iceberg 目录平价，空早返是 hive-only〕、hive+非空→抛**逐字** legacy 文案 `"Not support insert with partition spec in hive catalog."`、hive+空→静默；外来句柄永不强转）+ fe-core `BindSink.checkConnectorWritePartitionNames`（镜像 `checkConnectorStaticPartitions`：空守卫→解析句柄→SPI→`DorisConnectorException`→`AnalysisException`），`bindConnectorTableSink` 用 **`sink.getPartitions()`**（动态分区名列表，**非** `staticPartitionKeyValues`）调用；静态 `PARTITION(col='val')` 与裸 `INSERT...SELECT` 不受影响。**C 活跑但空守卫使裸 INSERT 对现存连接器字节不变、无连接器覆写该 SPI（皆继承 no-op）→ iceberg/paimon/mc 的 `PARTITION(names)` 仍静默接受如旧**；hive 抛拒休眠至 hms 进 SPI_READY_TYPES；翻闸删 `BindSink:667-669`（随 `bindHiveTableSink` 整体）是残留非本步。**覆盖策略=镜像 peer 静态分区缝（连接器单测+e2e，`bindConnectorTableSink` 无 fe-core 单测先例，已核实 HEAD 无同类测）**：`test_hive_write_type.groovy:250` e2e 文案逐字对齐（两路径都跑）+ `HiveConnectorMetadataSiblingDelegationTest` +1（hive 非空抛/空静默/不咨询兄弟）+ `EXPECTED_WRITE_METHODS` Rule-9 完整性锁加 `validateWritePartitionNames`（foreign 空亦转发）。4 维对抗复审（route/message-parity/live-no-regression+dormancy/field-keying+binding-completeness）+ refute 复核 **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + 兄弟测 8/8 + checkstyle 0〔api/hive/core〕 + import gate 净）。 + - **下一步 = WC4**（item 1，C 活跑、最高风险、**翻闸硬前置**）：新 `ConnectorScanPlanProvider.releaseReadTransaction`（no-op 默认）+ `HiveScanPlanProvider` 覆写 deregister + `PluginDrivenScanNode.getSplits` **无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，TCCL pin 经 `onPluginClassLoader` 白拿）；不补则翻闸后 ACID 读泄漏 metastore 共享读锁。**须补现存连接器无回归测**。 - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** - **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 From 11eda0417f035421ef4873231a8506f108f86600 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 20:38:08 +0800 Subject: [PATCH 093/334] [feat](catalog) HMS cutover write-chain WC4: wire read-ACID query-finish commit via neutral SPI (dormant) A transactional (full-ACID / insert-only) hive read opens a metastore read transaction + shared read lock during planScan (HiveScanPlanProvider.planAcidScan -> readTxnManager.register -> openTxn + acquireSharedLock). The matching release (HiveReadTransactionManager.deregister -> commitTxn) had ZERO callers, so a flipped ACID read would leak the metastore shared read lock for the metastore's lifetime. WC4 wires the release through the generic query-finish spine, so fe-core never names hive. - fe-connector-api ConnectorScanPlanProvider: new default no-op releaseReadTransaction(queryId), matching the getDeleteFiles/classifyColumn no-op-default precedent. - fe-connector-hive HiveScanPlanProvider: @Override -> readTxnManager .deregister(queryId). The manager is the SAME per-connector instance HiveConnector injects into register and every provider, so the keys match; deregister commits exactly once, is idempotent, and swallows a commit failure (best-effort). - fe-core PluginDrivenScanNode.getSplits(): UNCONDITIONALLY register a query-finish callback (QeProcessorImpl.registerQueryFinishCallback) right after the scan-provider null-guard -- BEFORE the pruned-to-zero short-circuit and planScan -- so a planScan that opens the txn then throws still has its release in place. queryId is computed ONCE (connectorSession.getQueryId()) == the runAndClear key (DebugUtil.printId) == the connector txnMap key. - fe-core buildReadTransactionReleaseCallback: extracted public-static helper returning the Runnable that pins the provider's plugin classloader (onPluginClassLoader) around the release -- load-bearing, since the callback runs on the StmtExecutor thread (app-loader TCCL) and commitTxn resolves metastore/thrift classes by name. Extracted so the release + TCCL-pin is unit-testable without driving getSplits (mirrors applyMvccSnapshotPin). Class C (live wiring, no-op default): the registration runs today on every live plugin scan (es/jdbc/paimon/max_compute/iceberg); only hive overrides the SPI, so the callback is inert for them (pins TCCL, calls the no-op default, never throws) and runAndClear clears it at query finish. Hive's release is dormant until hms enters SPI_READY_TYPES; the legacy HiveScanNode + Env.hiveTransactionMgr path is untouched (its delete + a live-ACID e2e read are flip-time residuals). Foreign (iceberg-on-HMS) handles route to the iceberg sibling provider whose releaseReadTransaction is the no-op default (no hive read txn opened). Tests: HiveReadTransactionTest +1 (provider.releaseReadTransaction commits via the shared manager exactly once, idempotent repeat, no exception on commit -- swallowed); new PluginDrivenScanNodeReadTxnReleaseTest (callback releases for the query; pins the provider classloader DURING the release against a distinct sentinel TCCL and restores it after). Existing scan-node suites unaffected (none drive getSplits end-to-end). Adversarial 5-lens review (placement/dominance, TCCL pin, queryId-identity + registry, live-no-regression + dormancy, connector-release/idempotency) with a refute pass: 0 confirmed findings. api+hive+core BUILD SUCCESS; HiveReadTransactionTest 4/4; all 16 fe-core scan-node classes 78/78; checkstyle 0 (api/hive/core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../api/scan/ConnectorScanPlanProvider.java | 19 ++++ .../connector/hive/HiveScanPlanProvider.java | 14 +++ .../hive/HiveReadTransactionTest.java | 26 ++++++ .../datasource/PluginDrivenScanNode.java | 33 +++++++ ...luginDrivenScanNodeReadTxnReleaseTest.java | 92 +++++++++++++++++++ 5 files changed, 184 insertions(+) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeReadTxnReleaseTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index 9d7839ff8897c0..dbbf6c1d544828 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -440,4 +440,23 @@ default List getDeleteFiles(TTableFormatFileDesc tableFormatParams) { default String getSerializedTable(Map nodeProperties) { return null; } + + /** + * Releases any per-query read transaction this provider opened, called by the engine when the query + * finishes (via the generic query-finish callback registry). The default is a no-op: connectors that do + * not open a per-query read transaction (every connector except transactional/ACID hive) need not override + * it. + * + *

    A connector that opens a metastore read transaction + shared read lock during {@link #planScan} (hive + * full-ACID / insert-only reads) MUST override this to commit that transaction and release the lock, else + * the shared read lock leaks for the metastore's lifetime. Best-effort: an implementation should log and + * swallow a commit failure rather than propagate (the callback registry isolates exceptions anyway). + * {@code queryId} is the engine query id string ({@link ConnectorSession#getQueryId()}), the same key the + * provider registered the transaction under.

    + * + * @param queryId the finishing query's id (== {@link ConnectorSession#getQueryId()}) + */ + default void releaseReadTransaction(String queryId) { + // default: no per-query read transaction to release + } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 7c569fb476a1ad..c48c4fb774daea 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -208,6 +208,20 @@ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, } } + /** + * Commits and deregisters this query's read transaction (opened by {@link #planAcidScan} via + * {@link HiveReadTransactionManager#register}), releasing the metastore shared read lock. Driven by the + * generic fe-core query-finish callback at query end; without it a transactional-hive read leaks the shared + * read lock for the metastore's lifetime. {@code readTxnManager} is the same per-connector manager that + * {@code register} used (both injected by {@code HiveConnector}), so the {@code queryId} keys match. + * {@code deregister} is idempotent (a no-op for a query that opened no transaction) and swallows a commit + * failure, matching the best-effort SPI contract. + */ + @Override + public void releaseReadTransaction(String queryId) { + readTxnManager.deregister(queryId); + } + /** Encodes each delete-delta as {@code "dir|file1,file2"} for {@link HiveScanRange.Builder#acidInfo}. */ private static List encodeDeleteDeltas(List deltas) { List encoded = new ArrayList<>(deltas.size()); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java index 6f596702a44378..12a7c6aaba04ca 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java @@ -99,6 +99,32 @@ public void testManagerRegisterBeginsAndDeregisterCommits() { Assertions.assertEquals(1, count(client.calls, "commitTxn:5"), "no double commit"); } + @Test + public void testScanProviderReleaseReadTransactionCommitsViaSharedManager() { + RecordingHmsClient client = new RecordingHmsClient(); + client.openTxnReturn = 9L; + // register and the provider's release MUST share the same per-connector manager (HiveConnector injects + // one instance into every provider), so the release finds and commits the txn register opened. + HiveReadTransactionManager mgr = new HiveReadTransactionManager(); + HiveScanPlanProvider provider = new HiveScanPlanProvider(client, new HashMap<>(), mgr); + + // A transactional-hive scan opened a read txn (as planAcidScan does via mgr.register), taking the shared + // read lock. + mgr.register(new HiveReadTransaction("q9", "u", "db", "t", true, client)); + Assertions.assertEquals(1, count(client.calls, "openTxn:u"), "the scan opened a read txn"); + + // The query-finish callback routes through the provider's releaseReadTransaction, which must commit the + // txn (releasing the shared read lock) exactly once — otherwise the lock leaks for the metastore's life. + provider.releaseReadTransaction("q9"); + Assertions.assertEquals(1, count(client.calls, "commitTxn:9"), + "releaseReadTransaction must commit the txn (release the shared read lock) exactly once"); + + // Idempotent: a second release, or a release for a query that opened no txn, is a safe no-op. + provider.releaseReadTransaction("q9"); + provider.releaseReadTransaction("never-opened"); + Assertions.assertEquals(1, count(client.calls, "commitTxn:9"), "no double commit on repeat release"); + } + private static int count(List calls, String prefix) { int n = 0; for (String c : calls) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 7e0ad0c36efc83..8f5a4470766c81 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -51,6 +51,7 @@ import org.apache.doris.planner.PlanNodeId; import org.apache.doris.planner.ScanContext; import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.QeProcessorImpl; import org.apache.doris.qe.SessionVariable; import org.apache.doris.spi.Split; import org.apache.doris.thrift.TColumnCategory; @@ -523,6 +524,26 @@ private static T onPluginClassLoader(ConnectorScanPlanProvider provider, Sup } } + /** + * Builds the query-finish callback that releases a connector's per-query read transaction. Hive full-ACID / + * insert-only reads open a metastore read transaction + shared read lock during {@code planScan}; this + * callback commits it (releasing the lock) when the query finishes. Registered UNCONDITIONALLY for every + * plugin scan in {@link #getSplits} — connector-agnostic, since a connector that opens no read transaction + * inherits the no-op {@link ConnectorScanPlanProvider#releaseReadTransaction} default and the callback is + * inert for it. The release runs on the StmtExecutor thread at query finish, whose TCCL is the fe-core app + * loader, so it MUST be pinned to the provider's plugin classloader ({@link #onPluginClassLoader}) or the + * commit's by-name class resolution (metastore/thrift) would split-brain against the app loader's copies. + * Extracted as a pure function of {@code (scanProvider, queryId)} so the release + TCCL-pin behavior is + * unit-testable without driving a full {@code getSplits}. + */ + public static Runnable buildReadTransactionReleaseCallback( + ConnectorScanPlanProvider scanProvider, String queryId) { + return () -> onPluginClassLoader(scanProvider, () -> { + scanProvider.releaseReadTransaction(queryId); + return null; + }); + } + /** * FIX-E (explain gap): delegates the VERBOSE per-backend block's delete-file lookup to the * connector SPI. The parent {@link FileScanNode#getDeleteFiles} returns empty; a connector that @@ -914,6 +935,18 @@ public List getSplits(int numBackends) throws UserException { return Collections.emptyList(); } + // Register the per-query read-transaction release BEFORE planScan (and before the pruned-to-zero + // short-circuit below), so a planScan that opens a metastore read transaction and then throws still has + // its release callback in place. Unconditional and connector-agnostic: a connector that opens no read + // transaction (every connector except transactional/ACID hive) inherits the no-op + // releaseReadTransaction default, so this is inert for it (the callback only pins TCCL and calls the + // no-op). The callback runs on the StmtExecutor thread at query finish, whose TCCL is the fe-core app + // loader, so the release is pinned to the provider's plugin classloader (see the helper). One string: + // connectorSession.getQueryId() == the query-finish registry key == the connector's txnMap key. + String readTxnQueryId = connectorSession.getQueryId(); + QeProcessorImpl.INSTANCE.registerQueryFinishCallback(readTxnQueryId, + buildReadTransactionReleaseCallback(scanProvider, readTxnQueryId)); + // Push the Nereids partition-pruning result down to the connector so the read session // covers only the surviving partitions. A pruned-to-zero set means no data to read, // mirroring legacy MaxComputeScanNode.getSplits()'s empty-selection short-circuit — UNLESS the diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeReadTxnReleaseTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeReadTxnReleaseTest.java new file mode 100644 index 00000000000000..553a6999b11d62 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeReadTxnReleaseTest.java @@ -0,0 +1,92 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.net.URL; +import java.net.URLClassLoader; +import java.util.concurrent.atomic.AtomicReference; + +/** + * Guards {@link PluginDrivenScanNode#buildReadTransactionReleaseCallback}, the query-finish callback that + * releases a connector's per-query read transaction. Hive full-ACID / insert-only reads open a metastore read + * transaction + shared read lock during {@code planScan}; the engine registers this callback in + * {@code getSplits} to commit it (releasing the lock) when the query finishes. Without the release the shared + * read lock leaks for the metastore's lifetime. + * + *

    Why this matters: the callback runs on the StmtExecutor thread at query finish, whose TCCL is the + * fe-core app loader. {@code releaseReadTransaction -> txn.commit -> hmsClient.commitTxn} resolves + * metastore/thrift classes by name via the TCCL, so the callback MUST pin the provider's plugin classloader for + * the duration of the release, else it split-brains against the app loader's duplicate copies (ClassCast / + * NoClassDef at commit). Each test kills a mutation: (1) dropping the release call / wrong queryId, (2) dropping + * the TCCL pin, (3) leaking the pin.

    + */ +public class PluginDrivenScanNodeReadTxnReleaseTest { + + @Test + public void callbackReleasesReadTransactionForTheQuery() { + ConnectorScanPlanProvider provider = Mockito.mock(ConnectorScanPlanProvider.class); + + Runnable callback = PluginDrivenScanNode.buildReadTransactionReleaseCallback(provider, "query-42"); + // The transaction is released only when the query finishes: merely building the callback must not + // release it yet (a mutation that releases eagerly would leak nothing but breaks the deferred contract). + Mockito.verifyNoInteractions(provider); + + callback.run(); + // MUTATION: dropping the releaseReadTransaction call, or passing a different queryId than the one the + // txn was registered under (so deregister would miss it), is killed here. + Mockito.verify(provider).releaseReadTransaction("query-42"); + } + + @Test + public void callbackPinsProviderClassLoaderDuringReleaseAndRestoresAfter() throws Exception { + ConnectorScanPlanProvider provider = Mockito.mock(ConnectorScanPlanProvider.class); + ClassLoader providerLoader = provider.getClass().getClassLoader(); + + AtomicReference tcclDuringRelease = new AtomicReference<>(); + Mockito.doAnswer(inv -> { + tcclDuringRelease.set(Thread.currentThread().getContextClassLoader()); + return null; + }).when(provider).releaseReadTransaction(Mockito.anyString()); + + Thread current = Thread.currentThread(); + ClassLoader outer = current.getContextClassLoader(); + // A distinct sentinel TCCL (never equal to the provider's mock loader), so a "no pin" mutation — which + // would leave the sentinel in place during the release — is caught by the assertSame below. + try (URLClassLoader sentinel = new URLClassLoader(new URL[0], null)) { + current.setContextClassLoader(sentinel); + + PluginDrivenScanNode.buildReadTransactionReleaseCallback(provider, "q").run(); + + // MUTATION: dropping the onPluginClassLoader pin is killed — during the release the TCCL must be the + // provider's (plugin) classloader, not the caller's app/sentinel loader. + Assertions.assertSame(providerLoader, tcclDuringRelease.get(), + "the release must run with the TCCL pinned to the provider's plugin classloader"); + // MUTATION: leaking the pin (not restoring in a finally) is killed — the caller's TCCL must be back. + Assertions.assertSame(sentinel, current.getContextClassLoader(), + "the TCCL must be restored to the caller's after the release"); + } finally { + current.setContextClassLoader(outer); + } + } +} From 2020dc9b9404b117581802fdf7a3f4542cc50964 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 20:39:58 +0800 Subject: [PATCH 094/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20write-?= =?UTF-8?q?chain=20WC4=20DONE=20(7aeac97c8e8);=20=C2=A74.5=20dormant=20WC1?= =?UTF-8?q?-WC4=20all=20complete;=20next=20thread=20=3D=20hudi=20flip-bloc?= =?UTF-8?q?ker?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 0716398fb21c77..bfa3259f37b58d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -71,11 +71,11 @@ - **✅ W4 DONE（`5a41e4cc562`）= 事务按句柄（Shape C，唯一无先例的新形态）**：`ConnectorWriteOps` 加 `beginTransaction(session, handle)` 默认重载（委派无参版=对齐 `getWritePlanProvider(handle)` 先例；6 现存连接器继承默认→各自无参 `beginTransaction`，字节不变）；`HiveConnectorMetadata` 覆写守卫转发 foreign 句柄→兄弟（得 `IcebergConnectorTransaction`）、hive 句柄→落连接器自有 `HiveConnectorTransaction`——**必须对称**（hive/iceberg 写计划把 session-bound txn 分别强转不同具体类型，"总转发"/"总 hive"都炸另一侧）。fe-core 把已解析写目标句柄穿进**两个**开事务点（否则 EXECUTE 路径留活 CCE）：`PluginDrivenInsertExecutor`（INSERT/OVERWRITE/DELETE/MERGE 单一漏斗）经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()`、`ConnectorRewriteDriver`（EXECUTE，已持句柄）；connector-agnostic，**绝不给网关传 null 句柄**（fail-loud，故不会把普通写误路由兄弟）。侦察(`wf_a6e430b0-3e9`)+4维对抗复审(`wf_6af16e19-0cc`)+verify（CCE 净、休眠/现存连接器字节平价、路由测 mutation-tight）；2 确认发现均测试覆盖缺口、已补（`PluginDrivenExternalTableTest` fail-loud 体测→`orElse(null)` 降级现挂；`PluginDrivenInsertExecutorTest` 句柄穿透 assertSame→null 二参 mutant 现挂）。全绿（fe-connector-hive 7〔+2 divert〕+ fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30〔+2〕/ ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕+ import gate 净）。**W4 复审记账（residual，休眠期不动）**：①**once-per-statement `getTableHandle` 往返**——每笔 plugin 写开事务时新增一次解析（paimon/iceberg 缓存命中、jdbc/maxcompute 未缓存），有意接受、非 per-row；②fail-loud orElseThrow 开事务时重查——并发 DROP/瞬时元数据错稍扩失败窗（paimon getTableHandle 吞异常成 empty→通用文案盖根因），非正确性回退；③W6 写路径 TCCL pin=**开事务不需**（`IcebergConnectorTransaction` 构造只赋值、SDK/反射推迟到 beginWrite/commit），commit 侧 pin 归 W6；④hudi-on-HMS 保 HiveTableHandle→hive txn（未变差，后续 hudi 委派步）；⑤e2e owed（休眠单测只证路由）。 - **✅ W5 DONE（`8aee62bab20`）= 存储过程按句柄**（详见上 🎯 段 W5 条目）。`Connector.getProcedureOps(handle)` per-handle 默认 + `HiveConnector` 分流（hive/hudi→继承 null、foreign→兄弟永不强转）+ fe-core `ConnectorExecuteAction` reorder（句柄解析上提、改调 per-handle）。net-new 升级、reorder 无测钉旧序、顺带修陈旧 javadoc。4 维对抗复审 0 blocker/major/minor + 1 nit（已修）。**⇒ §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官。** - ⚠**W5 复审记账（休眠期不动）**：①**paimon/无-procedure 连接器 EXECUTE 现多一次 `getTableHandle` 往返**（reorder 先解析句柄再判 null-ops；happy-path iceberg 无新增往返，只错误路径先报 "Table not found"）——有意接受、非正确性回退；②`ExecuteActionFactory.getSupportedActions` 留无参 getter（SHOW 发现，无 live caller；翻闸对 hms 网关会返 hive 连接器级空——handle-aware 变体延后，无消费者故无害）；③e2e owed（休眠单测只证路由；随其它 iceberg-on-HMS 能力翻闸期统一补，memory `hms-iceberg-delegation-needs-e2e`）。 -- **用户已选：写链收尾（§4.5）。权威拆解 = `hms-write-chain-decomposition-2026-07-08.md`（起步必读；code-grounded recon `wf_13e18d30-257`：3 维读者 + 完整性/排序 critic，全 HEAD 核对）**。核心 = 三类休眠 taxonomy（**A**=翻闸前不可达、**B**=值恒等、**C**=活跑靠 no-op SPI 默认——**C 类须补现存连接器无回归测**，非仅休眠 hive 单测）+ 风险升序 4 步： +- **✅ 写链收尾（§4.5）休眠四步 WC1–WC4 全部收官（本 session：WC2/WC3/WC4；WC1 上 session）。权威拆解 = `hms-write-chain-decomposition-2026-07-08.md`**。核心 = 三类休眠 taxonomy（**A**=翻闸前不可达、**B**=值恒等、**C**=活跑靠 no-op SPI 默认——**C 类须补现存连接器无回归测**，非仅休眠 hive 单测）+ 风险升序 4 步： - **✅ WC1 DONE（`6c01805109c`）= 搬 `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY`**（B 值恒等：两字面量均 `"broker.name"`；`private→public` 成单一真源、generic base 不再依赖 HMS 子类；`.get()` 保**大小写敏感、不 harmonize**；删 HMS 副本〔HEAD 证单引用〕）。全绿（BrokerPropertiesTest 6〔+1 常量真源 pin〕 + HiveScanNodeTest 6 + checkstyle 0）。 - **✅ WC2 DONE（`650658b4ce8`）= engine-map 三 case（A 不可达、极低风险，一 commit）**：`CreateTableInfo.pluginCatalogTypeToEngine` 加 `case "hms"→ENGINE_HIVE`（CREATE 引擎；镜像 legacy paddingEngineName/checkEngineWithCatalog 的 `instanceof HMSExternalCatalog` 臂；`hive != iceberg` 故 `getEffectiveIcebergFormatVersion` 臂对 hms 保持 inert，已加非干扰锁测）+ `PluginDrivenExternalTable.getEngine()` 加 `case "hms"→HMS_EXTERNAL_TABLE.toEngineName()`（== `"hms"`；**DISPLAY 引擎，legacy 显示 "hms" 非 "hive"，与 CREATE 引擎两回事**；落 "Plugin" 会退化 SHOW TABLE STATUS/information_schema.tables）、`getEngineTableTypeName()` 加 `case "hms"→HMS_EXTERNAL_TABLE.name()`（守 pluginCatalogTypeToEngine sync 不变量）。休眠成立已 HEAD 证实（`SPI_READY_TYPES` 无 `"hms"`、CatalogFactory `case "hms"` 仍建 legacy HMSExternalCatalog→无 PluginDriven 目录 getType()=="hms"→三 case 翻闸前不可达）。休眠单测：`CreateTableInfoEngineCatalogTest` +5（no-ENGINE pad hive / CTAS pad hive / 错引擎拒 / hive 过 / iceberg-format 非干扰锁）、`PluginDrivenExternalTableEngineTest` +2（engine `"hms"` + type name `HMS_EXTERNAL_TABLE`）。全绿（fe-core BUILD SUCCESS + 31 测 0 fail + checkstyle 0；纯 fe-core 无连接器改动故 import gate N/A）。 - **✅ WC3 DONE（`ab3b0a63ed1`）= 显式分区 INSERT 拒绝（item 2a，C 活跑）**：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松 no-op 默认，镜像 `validateStaticPartitionColumns` 模式）+ `HiveConnectorMetadata` 覆写（foreign→兄弟**无论空非空**都转发〔兄弟接受 `PARTITION(names)`＝独立 iceberg 目录平价，空早返是 hive-only〕、hive+非空→抛**逐字** legacy 文案 `"Not support insert with partition spec in hive catalog."`、hive+空→静默；外来句柄永不强转）+ fe-core `BindSink.checkConnectorWritePartitionNames`（镜像 `checkConnectorStaticPartitions`：空守卫→解析句柄→SPI→`DorisConnectorException`→`AnalysisException`），`bindConnectorTableSink` 用 **`sink.getPartitions()`**（动态分区名列表，**非** `staticPartitionKeyValues`）调用；静态 `PARTITION(col='val')` 与裸 `INSERT...SELECT` 不受影响。**C 活跑但空守卫使裸 INSERT 对现存连接器字节不变、无连接器覆写该 SPI（皆继承 no-op）→ iceberg/paimon/mc 的 `PARTITION(names)` 仍静默接受如旧**；hive 抛拒休眠至 hms 进 SPI_READY_TYPES；翻闸删 `BindSink:667-669`（随 `bindHiveTableSink` 整体）是残留非本步。**覆盖策略=镜像 peer 静态分区缝（连接器单测+e2e，`bindConnectorTableSink` 无 fe-core 单测先例，已核实 HEAD 无同类测）**：`test_hive_write_type.groovy:250` e2e 文案逐字对齐（两路径都跑）+ `HiveConnectorMetadataSiblingDelegationTest` +1（hive 非空抛/空静默/不咨询兄弟）+ `EXPECTED_WRITE_METHODS` Rule-9 完整性锁加 `validateWritePartitionNames`（foreign 空亦转发）。4 维对抗复审（route/message-parity/live-no-regression+dormancy/field-keying+binding-completeness）+ refute 复核 **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + 兄弟测 8/8 + checkstyle 0〔api/hive/core〕 + import gate 净）。 - - **下一步 = WC4**（item 1，C 活跑、最高风险、**翻闸硬前置**）：新 `ConnectorScanPlanProvider.releaseReadTransaction`（no-op 默认）+ `HiveScanPlanProvider` 覆写 deregister + `PluginDrivenScanNode.getSplits` **无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，TCCL pin 经 `onPluginClassLoader` 白拿）；不补则翻闸后 ACID 读泄漏 metastore 共享读锁。**须补现存连接器无回归测**。 + - **✅ WC4 DONE（`7aeac97c8e8`）= 读-ACID 查询结束提交接线（item 1，C 活跑、最高风险、翻闸硬前置）**：新 `ConnectorScanPlanProvider.releaseReadTransaction(queryId)`（no-op 默认，对齐 getDeleteFiles/classifyColumn 先例）+ `HiveScanPlanProvider` 覆写→`readTxnManager.deregister`（与 register 共用 HiveConnector 注入的**同一** per-connector 管理器→键匹配；commit 恰一次、幂等、吞 commit 异常=best-effort）+ fe-core `PluginDrivenScanNode.getSplits` 在 scanProvider 空判后**无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，**置于 pruned-to-zero 短路与 planScan 之前**故 planScan 开事务后抛仍有 release 兜底）+ 抽出 `buildReadTransactionReleaseCallback`（public static，回调经 `onPluginClassLoader` pin 插件 loader——回调在 StmtExecutor 线程〔app loader TCCL〕跑、commitTxn 按名解析 metastore/thrift 类故 pin 载荷；抽出以便脱离 getSplits 单测，镜像 `applyMvccSnapshotPin`）。**queryId 单串不变量**：`connectorSession.getQueryId()`（注册键+release 键，只取一次）==`DebugUtil.printId`（runAndClear 键）==连接器 txnMap 键。**C 活跑但仅 hive 覆写 SPI**（已 grep 证 iceberg/paimon/mc/jdbc/es 皆继承 no-op→回调对它们 inert：pin TCCL+调 no-op，永不抛，runAndClear 查询结束清除）；hive release 休眠至 hms 进 SPI_READY_TYPES；legacy `HiveScanNode`+`Env.hiveTransactionMgr` 未动（其删除+live-ACID e2e 读=翻闸残留）；foreign iceberg-on-HMS 句柄路由到兄弟 provider 的 no-op release（未开 hive 读事务）。测试：`HiveReadTransactionTest` +1（provider release 经共享管理器恰一次 commit+幂等+吞异常）、新 `PluginDrivenScanNodeReadTxnReleaseTest`（回调 release+对 distinct sentinel TCCL 证 pin 到 provider loader 且事后还原）；现存 scan-node 套无影响（无一驱动 getSplits 端到端）。5 维对抗复审（placement/dominance·TCCL·queryId-identity+registry·live-no-regression+dormancy·connector-release/idempotency）+ refute **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + HiveReadTransactionTest 4/4 + fe-core 16 个 scan-node 类 78/78 + checkstyle 0〔api/hive/core〕 + import gate 净）。**⇒ 至此 §4.5 写链休眠四步 WC1–WC4 全部收官；写链只余翻闸期删除残留（R1–R4）。** - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** - **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 From 07670dc50473410a6b7710b2032943d43f75f0be Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 21:44:39 +0800 Subject: [PATCH 095/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20hudi?= =?UTF-8?q?=20HD-A0:=20DV-005=20guardrail=20=E2=80=94=20hudi=20is=20siblin?= =?UTF-8?q?g-only,=20never=20a=20standalone=20catalog=20type?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Correct the misleading "dedicated Hudi catalogs" framing that invites a future maintainer to "fix" the standalone-type model mismatch by promoting "hudi" into SPI_READY_TYPES: - HudiConnectorProvider javadoc + getType(): "hudi" is a SIBLING-ONLY type string resolved via createSiblingConnector; not a user-facing catalog type; never add to SPI_READY_TYPES / a factory case. - HudiConnector javadoc: built only as an embedded sibling of the hive hms gateway. - CatalogFactory.SPI_READY_TYPES: replace the stale "hms, hudi still use built-in ExternalCatalog" comment (there is NO HudiExternalCatalog) with a guard — hms uses built-in HMSExternalCatalog until the cutover, hudi is parasitic on HMS and served as an hms-gateway sibling, never added here. Comment/javadoc only; zero logic change. Dormant (hms not in SPI_READY_TYPES). checkstyle 0 (hudi + fe-core). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/hudi/HudiConnector.java | 4 ++++ .../connector/hudi/HudiConnectorProvider.java | 14 ++++++++++++-- .../apache/doris/datasource/CatalogFactory.java | 12 +++++++++--- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index bbf4fff30c0b32..9b7367ca2333ef 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -42,6 +42,10 @@ *

    Phase 1 provides read-only metadata operations (list databases, * list tables, get schema via Hudi's Avro schema). Phase 2 adds scan * planning for COW and MOR tables (snapshot reads).

    + * + *

    Built only as an embedded sibling of the hive {@code hms} gateway (via + * {@code ConnectorContext.createSiblingConnector("hudi", ...)}), never as a standalone {@code type=hudi} + * catalog — see {@link HudiConnectorProvider}.

    */ public class HudiConnector implements Connector { diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorProvider.java index f055aa3a5c5329..993775d904c401 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorProvider.java @@ -27,13 +27,23 @@ * SPI entry point for the Hudi connector plugin. * *

    Registered via {@code META-INF/services/org.apache.doris.connector.spi.ConnectorProvider}. - * The type is {@code "hudi"} for dedicated Hudi catalogs that connect to HMS - * and expose Hudi tables.

    + * + *

    The type {@code "hudi"} is a SIBLING-ONLY type string — NOT a user-facing catalog type. There is no + * {@code type=hudi} catalog and no {@code HudiExternalCatalog}: a hudi table is always parasitic on an HMS + * catalog (legacy {@code HMSExternalTable} with {@code dlaType == HUDI}). After the HMS cutover this connector is + * built only as an embedded sibling of the hive {@code hms} gateway, resolved through + * {@code ConnectorContext.createSiblingConnector("hudi", ...)} — which bypasses + * {@code CatalogFactory.SPI_READY_TYPES}. NEVER add {@code "hudi"} to {@code SPI_READY_TYPES} and never add + * a {@code case "hudi"} to the catalog factory: doing so would build a standalone + * {@code PluginDrivenExternalCatalog} around this connector with no fe-core catalog class backing it (the exact + * model mismatch this type string otherwise invites). */ public class HudiConnectorProvider implements ConnectorProvider { @Override public String getType() { + // Sibling-only lookup key for createSiblingConnector("hudi", ...); see the class javadoc. + // NOT a user-facing catalog type; never add "hudi" to CatalogFactory.SPI_READY_TYPES. return "hudi"; } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java index 1746f9f33d4fbf..4a2a3dcf2766d5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java @@ -43,9 +43,15 @@ public class CatalogFactory { private static final Logger LOG = LogManager.getLogger(CatalogFactory.class); - // Only these catalog types are routed through the SPI connector path. - // Other types (hms, hudi) still use - // their built-in ExternalCatalog implementations until their ConnectorProviders are fully ready. + // Only these catalog types are routed through the SPI connector path; every other type falls through to the + // built-in ExternalCatalog switch below. "hms" is deliberately absent: a hive catalog still uses the built-in + // HMSExternalCatalog until the single-line HMS cutover adds "hms" here (which flips hive + iceberg-on-HMS + + // hudi-on-HMS to the SPI path atomically). + // Do NOT add "hudi": there is no standalone hudi catalog type and no HudiExternalCatalog — a hudi table is + // parasitic on an HMS catalog (HMSExternalTable, dlaType==HUDI) and is served post-cutover as an embedded + // SIBLING of the hms gateway via ConnectorContext.createSiblingConnector("hudi", ...), which bypasses this + // set. Adding "hudi" here would build a standalone PluginDrivenExternalCatalog around HudiConnector with no + // fe-core catalog class backing it. private static final Set SPI_READY_TYPES = ImmutableSet.of("jdbc", "es", "trino-connector", "max_compute", "paimon", "iceberg"); From 3a38e957438ea03400ffaf24e4289e88a0cafc45 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 21:55:25 +0800 Subject: [PATCH 096/334] [feat](catalog) HMS cutover hudi HD-A1: hudi sibling holder + property synthesis (mirror iceberg S1/S2, dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Give the hive hms gateway a second embedded SIBLING connector for hudi-on-HMS, mirroring the reviewed iceberg sibling holder. Dormant: no production path references it until the getTableHandle HUDI divert lands (a later substep). - New HudiSiblingProperties.synthesize(gatewayProps): a verbatim defensive copy of the gateway catalog's whole property map. Unlike IcebergSiblingProperties it injects NO flavor key — hudi has no iceberg.catalog.type analogue; HudiConnector.createClient reads hive.metastore.uris/uri + raw hadoop.*/fs.*/dfs.*/hive.*/s3.* straight from the map. Whole-map copy avoids silently dropping a connectivity key. - HiveConnector: add HUDI_CONNECTOR_TYPE="hudi" literal + volatile hudiSibling field + getOrCreateHudiSibling() (lazy volatile double-checked build via context.createSiblingConnector("hudi", synthesize(props)); fail-loud naming the catalog when the plugin is absent; failure NOT memoized; held ONLY as the parent-first Connector interface, never cast — its concrete type is invisible across the loader split). close() forwards to the hudi sibling too (gateway owns both siblings' lifecycle; each null-guarded, cleared, no-op when never built). Tests: HudiSiblingPropertiesTest (3: verbatim carry, no-flavor-injected, defensive copy) + HiveConnectorSiblingTest hudi cases (build/memoize/fail-loud-not-memoized/close-forward) + a closeForwardsToBothSiblingsIndependently regression guard (adding the hudi arm must not drop/double the iceberg arm). fe-connector-hive 230 tests green, checkstyle 0, import gate net. 4-dimension adversarial review (holder-parity/close-lifecycle/dormancy-synthesis/ test-intent) + verify: 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 50 ++++++++ .../connector/hive/HudiSiblingProperties.java | 54 +++++++++ .../hive/HiveConnectorSiblingTest.java | 112 +++++++++++++++++- .../hive/HudiSiblingPropertiesTest.java | 82 +++++++++++++ 4 files changed, 292 insertions(+), 6 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HudiSiblingProperties.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HudiSiblingPropertiesTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 8a2bc6c12870cc..3a73279d46dcdf 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -64,6 +64,11 @@ public class HiveConnector implements Connector { // matches the "iceberg" entry in CatalogFactory.SPI_READY_TYPES. private static final String ICEBERG_CONNECTOR_TYPE = "iceberg"; + // The sibling connector type a flipped hms gateway delegates hudi-on-HMS tables to. A string literal (hudi + // has NO user-facing catalog type — it is served only via createSiblingConnector); matches the "hudi" type + // string HudiConnectorProvider registers. NEVER add "hudi" to CatalogFactory.SPI_READY_TYPES. + private static final String HUDI_CONNECTOR_TYPE = "hudi"; + private final Map properties; private final ConnectorContext context; private volatile HmsClient hmsClient; @@ -86,6 +91,14 @@ public class HiveConnector implements Connector { // nothing builds it today. private volatile Connector icebergSibling; + // Embedded hudi SIBLING connector: a flipped hms gateway delegates its hudi-on-HMS tables to it. Same + // lifecycle/classloader contract as icebergSibling above — built once per gateway (lazily) in the hudi + // plugin's OWN child-first classloader via context.createSiblingConnector, never co-packaged into the hive + // zip (a second AWS SDK would poison S3 JVM-wide). Held ONLY as the parent-first Connector interface and + // NEVER cast (a cast would CCE across the loader split). Dormant until hms enters SPI_READY_TYPES AND the + // getTableHandle HUDI divert is wired (a later substep) — nothing references it today. + private volatile Connector hudiSibling; + public HiveConnector(Map properties, ConnectorContext context) { this.properties = Collections.unmodifiableMap(properties); this.context = context; @@ -252,6 +265,37 @@ Connector getOrCreateIcebergSibling() { return icebergSibling; } + /** + * Lazily builds and memoizes the embedded hudi sibling connector this hive gateway delegates its + * hudi-on-HMS tables to. Mirrors {@link #getOrCreateIcebergSibling()}: exactly ONE sibling per gateway + * connector (not per table), built through {@link ConnectorContext#createSiblingConnector} so its concrete + * class is loaded by the hudi plugin's own child-first classloader, sharing this gateway catalog's + * id/auth/storage; it is therefore held ONLY as the parent-first {@link Connector} interface and MUST NOT be + * cast (a cast would CCE across the loader split). + * + *

    Fails loud when no hudi provider is available (e.g. the plugin is not installed). The failure is NOT + * memoized (a null sibling leaves the field unset), so a later-available plugin recovers on the next access. + * + *

    Dormant: no production path references it until the getTableHandle HUDI divert lands (a later substep). + */ + Connector getOrCreateHudiSibling() { + if (hudiSibling == null) { + synchronized (this) { + if (hudiSibling == null) { + Connector sibling = context.createSiblingConnector( + HUDI_CONNECTOR_TYPE, HudiSiblingProperties.synthesize(properties)); + if (sibling == null) { + throw new DorisConnectorException( + "Cannot serve hudi-on-HMS tables in catalog '" + context.getCatalogName() + + "': the hudi connector plugin is not available"); + } + hudiSibling = sibling; + } + } + } + return hudiSibling; + } + private HmsClient createClient() { String metastoreUri = properties.get(HiveConnectorProperties.HIVE_METASTORE_URIS); if (metastoreUri == null || metastoreUri.isEmpty()) { @@ -372,5 +416,11 @@ public void close() throws IOException { sibling.close(); icebergSibling = null; } + // Same for the embedded hudi sibling — the gateway owns its lifecycle too. No-op when never built. + Connector hudi = hudiSibling; + if (hudi != null) { + hudi.close(); + hudiSibling = null; + } } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HudiSiblingProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HudiSiblingProperties.java new file mode 100644 index 00000000000000..3b7bd7b366a832 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HudiSiblingProperties.java @@ -0,0 +1,54 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import java.util.HashMap; +import java.util.Map; + +/** + * Synthesizes the catalog-property map for the embedded Hudi sibling connector that a flipped HMS + * gateway delegates its hudi-on-HMS tables to. Mirrors {@link IcebergSiblingProperties} so the two sibling + * paths read identically at the {@code getOrCreate*Sibling} seams. + * + *

    The sibling is built once per gateway catalog (not per table) via + * {@code ConnectorContext.createSiblingConnector("hudi", synthesize(catalogProps))}, sharing the gateway's + * context (metastore auth + storage). Unlike the iceberg sibling there is no flavor key to inject: hudi + * has no {@code iceberg.catalog.type} analogue — {@code HudiConnector.createClient} reads + * {@code hive.metastore.uris}/{@code uri} plus the raw {@code hadoop.*}/{@code fs.*}/{@code dfs.*}/{@code hive.*}/ + * {@code s3.*} storage + kerberos passthrough straight from this map. So synthesis is a plain verbatim copy of + * the gateway catalog's whole property map. Carrying the whole map (rather than a hand-picked subset) is the + * robust choice: it cannot silently drop a connectivity key (the {@code uri} short form, an HDFS-HA + * {@code dfs.*} set, an S3 endpoint override, a kerberos variant, ...); the connector ignores keys it does not + * recognize. + * + *

    A defensive copy is returned so the gateway's own (unmodifiable, shared) property map is never aliased into + * the sibling connector. + */ +final class HudiSiblingProperties { + + private HudiSiblingProperties() { + } + + /** + * Returns a NEW property map = the gateway catalog's properties verbatim (a defensive copy). The input is + * never mutated. No flavor key is injected — a hudi-on-HMS sibling needs none. + */ + static Map synthesize(Map gatewayCatalogProperties) { + return new HashMap<>(gatewayCatalogProperties); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java index fe410e554d4c22..cb6cc64b5e66b5 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java @@ -29,14 +29,16 @@ import java.util.Map; /** - * Pins the HMS-cutover §4.4 S2 embedded-sibling holder: a flipped hms gateway lazily builds ONE embedded - * iceberg connector (via {@link org.apache.doris.connector.spi.ConnectorContext#createSiblingConnector}) that it - * delegates its iceberg-on-HMS tables to, and forwards {@code close()} to it. + * Pins the HMS-cutover embedded-sibling holders: a flipped hms gateway lazily builds ONE embedded iceberg + * connector and ONE embedded hudi connector (each via + * {@link org.apache.doris.connector.spi.ConnectorContext#createSiblingConnector}) that it delegates its + * iceberg-on-HMS / hudi-on-HMS tables to, and forwards {@code close()} to both. * *

    The whole surface is dormant until hms enters {@code SPI_READY_TYPES}: no production path calls - * {@code getOrCreateIcebergSibling()} yet, so these assertions are a Rule-9 guard that the holder's contract - * (single sibling per gateway, hms-flavor synthesized props, fail-loud when the plugin is absent, lifecycle - * forwarding) is correct BEFORE the flip wires a consumer. + * {@code getOrCreateIcebergSibling()} / {@code getOrCreateHudiSibling()} yet, so these assertions are a Rule-9 + * guard that each holder's contract (single sibling per gateway, correctly synthesized props — hms-flavor for + * iceberg, verbatim for hudi — fail-loud when the plugin is absent, independent lifecycle forwarding) is correct + * BEFORE the flip wires a consumer. */ public class HiveConnectorSiblingTest { @@ -130,6 +132,104 @@ public void closeIsNoOpWhenSiblingNeverBuilt() throws Exception { Assertions.assertEquals(0, context.buildCount, "close must not trigger a sibling build"); } + // ---- hudi sibling holder (mirrors the iceberg cases above; hudi synthesizes props verbatim, no flavor) ---- + + @Test + public void buildsHudiSiblingWithVerbatimProps() { + Map catalogProps = new HashMap<>(); + catalogProps.put("hive.metastore.uris", "thrift://host:9083"); + catalogProps.put("iceberg.catalog.type", "rest"); // hudi injects NO flavor: a stray key survives verbatim + FakeSibling sibling = new FakeSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(catalogProps, context); + + Connector built = connector.getOrCreateHudiSibling(); + + Assertions.assertSame(sibling, built, "the accessor must return the context-built sibling"); + // The sibling is always a hudi connector — the delegate type must reach the seam verbatim. + Assertions.assertEquals("hudi", context.lastType, "the sibling connector type must be hudi"); + Assertions.assertEquals("thrift://host:9083", context.lastProps.get("hive.metastore.uris"), + "the gateway's metastore uri must be carried to the sibling"); + // Unlike iceberg, hudi synthesis injects no flavor: a stray gateway key is carried through unchanged + // (there is no iceberg.catalog.type analogue for hudi to force). + Assertions.assertEquals("rest", context.lastProps.get("iceberg.catalog.type"), + "hudi synthesis injects no flavor — a stray gateway key is carried verbatim, not overridden"); + } + + @Test + public void memoizesSingleHudiSiblingPerGateway() { + RecordingSiblingContext context = new RecordingSiblingContext(new FakeSibling()); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + + Connector first = connector.getOrCreateHudiSibling(); + Connector second = connector.getOrCreateHudiSibling(); + + Assertions.assertSame(first, second, "repeated access must return the same single sibling"); + Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once per gateway"); + } + + @Test + public void failsLoudWhenHudiPluginAbsent() { + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + connector::getOrCreateHudiSibling); + Assertions.assertTrue(ex.getMessage().contains("test_catalog"), + "the failure must name the catalog it could not serve"); + Assertions.assertTrue(ex.getMessage().contains("hudi"), + "the failure must name the missing hudi plugin"); + } + + @Test + public void hudiFailLoudIsNotMemoized() { + RecordingSiblingContext context = new RecordingSiblingContext(null); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + Assertions.assertThrows(DorisConnectorException.class, connector::getOrCreateHudiSibling); + + FakeSibling sibling = new FakeSibling(); + context.siblingToReturn = sibling; + Assertions.assertSame(sibling, connector.getOrCreateHudiSibling(), + "a later-available sibling must be built after an earlier fail-loud"); + Assertions.assertEquals(2, context.buildCount, "the failed build must be retried, not memoized"); + } + + @Test + public void closeForwardsToHudiSiblingAndClearsIt() throws Exception { + FakeSibling sibling = new FakeSibling(); + RecordingSiblingContext context = new RecordingSiblingContext(sibling); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + connector.getOrCreateHudiSibling(); + + connector.close(); + connector.close(); + + Assertions.assertEquals(1, sibling.closeCount, "close must forward to the hudi sibling exactly once"); + } + + @Test + public void closeForwardsToBothSiblingsIndependently() throws Exception { + // Regression guard for adding the hudi holder: close() must forward to the iceberg AND the hudi field, + // each exactly once — adding the hudi arm must not drop or double the iceberg arm. Use a type-dispatching + // context so the two siblings are distinct instances. + FakeSibling icebergSibling = new FakeSibling(); + FakeSibling hudiSibling = new FakeSibling(); + FakeConnectorContext context = new FakeConnectorContext() { + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + return "hudi".equals(catalogType) ? hudiSibling : icebergSibling; + } + }; + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + connector.getOrCreateIcebergSibling(); + connector.getOrCreateHudiSibling(); + + connector.close(); + + Assertions.assertEquals(1, icebergSibling.closeCount, "close must forward to the iceberg sibling once"); + Assertions.assertEquals(1, hudiSibling.closeCount, "close must forward to the hudi sibling once"); + } + /** Records the {@code createSiblingConnector} call and returns a configurable (possibly null) sibling. */ private static final class RecordingSiblingContext extends FakeConnectorContext { int buildCount; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HudiSiblingPropertiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HudiSiblingPropertiesTest.java new file mode 100644 index 00000000000000..04d3222c3660d2 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HudiSiblingPropertiesTest.java @@ -0,0 +1,82 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests the pure hudi-sibling property synthesis. Unlike iceberg there is no flavor key to inject: hudi has no + * {@code iceberg.catalog.type} analogue, so synthesis is a verbatim defensive copy of the gateway's catalog map. + */ +public class HudiSiblingPropertiesTest { + + @Test + public void carriesMetastoreStorageAndKerberosKeysVerbatim() { + // The sibling connects to the SAME metastore/storage as the gateway; dropping any of these keys would + // leave the embedded hudi connector unable to reach HMS or object storage. The uri short form must + // survive too (HudiConnector.createClient reads both hive.metastore.uris and uri). + Map in = new HashMap<>(); + in.put("uri", "thrift://host:9083"); + in.put("fs.s3a.access.key", "AK"); + in.put("dfs.nameservices", "ns1"); + in.put("hadoop.security.authentication", "kerberos"); + in.put("hive.metastore.client.principal", "hive/_HOST@REALM"); + + Map out = HudiSiblingProperties.synthesize(in); + + Assertions.assertEquals("thrift://host:9083", out.get("uri"), "the uri short form must be carried"); + Assertions.assertEquals("AK", out.get("fs.s3a.access.key"), "object-storage creds must be carried"); + Assertions.assertEquals("ns1", out.get("dfs.nameservices"), "HDFS-HA config must be carried"); + Assertions.assertEquals("kerberos", out.get("hadoop.security.authentication"), + "kerberos auth mode must be carried"); + Assertions.assertEquals("hive/_HOST@REALM", out.get("hive.metastore.client.principal"), + "kerberos principal must be carried"); + } + + @Test + public void injectsNoFlavorKey() { + // Hudi has no iceberg.catalog.type analogue; synthesis must NOT invent one. The output equals the input. + Map in = new HashMap<>(); + in.put("hive.metastore.uris", "thrift://host:9083"); + + Map out = HudiSiblingProperties.synthesize(in); + + Assertions.assertFalse(out.containsKey("iceberg.catalog.type"), + "hudi synthesis must inject no flavor key"); + Assertions.assertEquals(in, out, "synthesis is a verbatim copy of the gateway property map"); + } + + @Test + public void returnsDefensiveCopyThatDoesNotMutateInput() { + // The gateway holds its catalog properties unmodifiable and shared; the returned map must be a distinct + // instance so a later mutation by the sibling connector cannot corrupt the gateway's own hive path. + Map in = new HashMap<>(); + in.put("hive.metastore.uris", "thrift://host:9083"); + + Map out = HudiSiblingProperties.synthesize(in); + out.put("extra.key", "v"); + + Assertions.assertNotSame(in, out, "synthesis must return a NEW map, not alias the gateway's"); + Assertions.assertFalse(in.containsKey("extra.key"), + "mutating the synthesized map must not affect the gateway's input map"); + } +} From 4a6868485a52d1853a998f721922a1ec47ffd656 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 21:58:09 +0800 Subject: [PATCH 097/334] [doc](catalog) HMS cutover hudi HD-A0 + HD-A1 DONE; next = Group A (A2-A5) + pending HD-B1 sign-off Record the first two hudi flip-blocker substeps as complete (37b85348667 DV-005 guardrail; 28d4781ccac hudi sibling holder + property synthesis). Next thread = rest of Group A (Kerberos, partition-value fidelity FIX, force-JNI, BE descriptor), plus HD-B1 (3-way foreign-handle routing) as the one decision needing user sign-off. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index bfa3259f37b58d..41bba4c9a3d902 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,6 +7,8 @@ # 🎯 当前状态（2026-07-08） +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**下一步 = Group A 余下（HD-A2 Kerberos / HD-A3 分区值保真 FIX〔HIGH〕/ HD-A4 force-JNI / HD-A5 BE 描述符，均无需签字）**；**HD-B1 三路 foreign-handle 路由 = 唯一待用户签字新决策**（已解锁，推荐 Option 1 `ownsHandle` 中立 SPI）。 + **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 **本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： @@ -77,11 +79,15 @@ - **✅ WC3 DONE（`ab3b0a63ed1`）= 显式分区 INSERT 拒绝（item 2a，C 活跑）**：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松 no-op 默认，镜像 `validateStaticPartitionColumns` 模式）+ `HiveConnectorMetadata` 覆写（foreign→兄弟**无论空非空**都转发〔兄弟接受 `PARTITION(names)`＝独立 iceberg 目录平价，空早返是 hive-only〕、hive+非空→抛**逐字** legacy 文案 `"Not support insert with partition spec in hive catalog."`、hive+空→静默；外来句柄永不强转）+ fe-core `BindSink.checkConnectorWritePartitionNames`（镜像 `checkConnectorStaticPartitions`：空守卫→解析句柄→SPI→`DorisConnectorException`→`AnalysisException`），`bindConnectorTableSink` 用 **`sink.getPartitions()`**（动态分区名列表，**非** `staticPartitionKeyValues`）调用；静态 `PARTITION(col='val')` 与裸 `INSERT...SELECT` 不受影响。**C 活跑但空守卫使裸 INSERT 对现存连接器字节不变、无连接器覆写该 SPI（皆继承 no-op）→ iceberg/paimon/mc 的 `PARTITION(names)` 仍静默接受如旧**；hive 抛拒休眠至 hms 进 SPI_READY_TYPES；翻闸删 `BindSink:667-669`（随 `bindHiveTableSink` 整体）是残留非本步。**覆盖策略=镜像 peer 静态分区缝（连接器单测+e2e，`bindConnectorTableSink` 无 fe-core 单测先例，已核实 HEAD 无同类测）**：`test_hive_write_type.groovy:250` e2e 文案逐字对齐（两路径都跑）+ `HiveConnectorMetadataSiblingDelegationTest` +1（hive 非空抛/空静默/不咨询兄弟）+ `EXPECTED_WRITE_METHODS` Rule-9 完整性锁加 `validateWritePartitionNames`（foreign 空亦转发）。4 维对抗复审（route/message-parity/live-no-regression+dormancy/field-keying+binding-completeness）+ refute 复核 **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + 兄弟测 8/8 + checkstyle 0〔api/hive/core〕 + import gate 净）。 - **✅ WC4 DONE（`7aeac97c8e8`）= 读-ACID 查询结束提交接线（item 1，C 活跑、最高风险、翻闸硬前置）**：新 `ConnectorScanPlanProvider.releaseReadTransaction(queryId)`（no-op 默认，对齐 getDeleteFiles/classifyColumn 先例）+ `HiveScanPlanProvider` 覆写→`readTxnManager.deregister`（与 register 共用 HiveConnector 注入的**同一** per-connector 管理器→键匹配；commit 恰一次、幂等、吞 commit 异常=best-effort）+ fe-core `PluginDrivenScanNode.getSplits` 在 scanProvider 空判后**无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，**置于 pruned-to-zero 短路与 planScan 之前**故 planScan 开事务后抛仍有 release 兜底）+ 抽出 `buildReadTransactionReleaseCallback`（public static，回调经 `onPluginClassLoader` pin 插件 loader——回调在 StmtExecutor 线程〔app loader TCCL〕跑、commitTxn 按名解析 metastore/thrift 类故 pin 载荷；抽出以便脱离 getSplits 单测，镜像 `applyMvccSnapshotPin`）。**queryId 单串不变量**：`connectorSession.getQueryId()`（注册键+release 键，只取一次）==`DebugUtil.printId`（runAndClear 键）==连接器 txnMap 键。**C 活跑但仅 hive 覆写 SPI**（已 grep 证 iceberg/paimon/mc/jdbc/es 皆继承 no-op→回调对它们 inert：pin TCCL+调 no-op，永不抛，runAndClear 查询结束清除）；hive release 休眠至 hms 进 SPI_READY_TYPES；legacy `HiveScanNode`+`Env.hiveTransactionMgr` 未动（其删除+live-ACID e2e 读=翻闸残留）；foreign iceberg-on-HMS 句柄路由到兄弟 provider 的 no-op release（未开 hive 读事务）。测试：`HiveReadTransactionTest` +1（provider release 经共享管理器恰一次 commit+幂等+吞异常）、新 `PluginDrivenScanNodeReadTxnReleaseTest`（回调 release+对 distinct sentinel TCCL 证 pin 到 provider loader 且事后还原）；现存 scan-node 套无影响（无一驱动 getSplits 端到端）。5 维对抗复审（placement/dominance·TCCL·queryId-identity+registry·live-no-regression+dormancy·connector-release/idempotency）+ refute **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + HiveReadTransactionTest 4/4 + fe-core 16 个 scan-node 类 78/78 + checkstyle 0〔api/hive/core〕 + import gate 净）。**⇒ 至此 §4.5 写链休眠四步 WC1–WC4 全部收官；写链只余翻闸期删除残留（R1–R4）。** - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** -- **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** 关键结论： +- **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** + - **✅ HD-A0 DONE（`37b85348667`，doc-only）= DV-005 护栏**：修正 `HudiConnectorProvider`/`HudiConnector` javadoc + `CatalogFactory.SPI_READY_TYPES` 注释——`"hudi"` 只是**兄弟类型串**（`createSiblingConnector("hudi",…)` 的查找键，绕过 `SPI_READY_TYPES`），**非**用户可见目录类型、无 `HudiExternalCatalog`；永不加进 `SPI_READY_TYPES`/工厂 case（否则重造 DV-005）。checkstyle 0（hudi+fe-core）。 + - **✅ HD-A1 DONE（`28d4781ccac`）= hudi 兄弟持有 + 属性合成（镜像 iceberg S1/S2，休眠）**：新 `HudiSiblingProperties.synthesize`=网关整份属性**逐字防御性拷贝**（与 iceberg 不同**不注入任何 flavor 键**——hudi 无 `iceberg.catalog.type` 类比，`HudiConnector.createClient` 直读 `hive.metastore.uris`/`uri`+raw `hadoop.*`/`fs.*`/`dfs.*`）+ `HiveConnector` 加 `HUDI_CONNECTOR_TYPE` 常量/`volatile hudiSibling` 字段/`getOrCreateHudiSibling()`（volatile 双检懒建、缺插件 fail-loud 且不 memoize、仅以 parent-first `Connector` 持有永不强转）/`close()` 转发兄弟。休眠（无生产引用，待 HD-B2 getTableHandle 分流点亮）。全绿（fe-connector-hive 230 测〔+3 `HudiSiblingPropertiesTest` +6 hudi holder 测含 `closeForwardsToBothSiblingsIndependently` 回归锁〕+ checkstyle 0 + import gate 净）；4 维对抗复审（holder-parity/close-lifecycle/dormancy-synthesis/test-intent）+ verify **0 发现**。 + - **下一步 = 继续 Group A（无需签字）**：HD-A2（hudi 兄弟 Kerberos plugin-auth，防翻闸后 Kerberized HMS 静默降级）/ **HD-A3（分区值解析保真=快照路径静默 NULL 分区列的真 bug，HIGH）**/ HD-A4（force-JNI session 标志 + COW/MOR 检测硬化）/ HD-A5（BE `HIVE_TABLE` 描述符 + BE-canonical 存储属性）。**HD-B1（三路 foreign-handle 路由机制）= 唯一待用户签字新决策**（HD-A1 已解锁它；推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认——可同加载器单测；会改动已落的 iceberg 活脊柱须保字节等价）。 + - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 - **DV-005 模型错配已解**：`HudiConnectorProvider.getType()="hudi"` 保留但**只作兄弟类型串**（`createSiblingConnector("hudi",…)` 绕过 `SPI_READY_TYPES`）；**永不**把 `hudi` 加进 `SPI_READY_TYPES`/工厂 case（会重造 DV-005）。 - - **有序子步（HD-A0..HD-D1，4 组）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 + - **有序子步（HD-A0..HD-D1，4 组；✅HD-A0/HD-A1 已收官，见上）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 - **⛔ 最大风险 HD-C3（§5.1）**：COW 增量的行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 `CheckPolicy` 注入、通用路径**跳过**且铁律禁 fe-core 复刻；COW 更新重写整个 base 文件把旧 commit 行带进→纯文件级选择返窗口外行=**静默错答**。须连接器把行谓词推进 scan-range/thrift **或** BE reader 按 instant 窗口行级过滤——**须先读 BE Hudi reader 源码判定**（本 checkout 无 BE 树），可能在铁律下无 fe-core 解，且只能翻闸 e2e 验（straddling-base-file fixture）。paimon **非**先例。 - **翻闸硬门 B1–B6**（见 plan §4.2）+ 两非单测 harness（两加载器路由 fixture + 异构 HMS e2e，含 COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved·非 hive-style/转义分区·写-拒）。**schema 缺失 SIGABRT 方向**（§5.2）已由 critic 对 BE 核对**降级为窄回退非崩溃**（emit-nothing=安全基线走 BE 小写 name-match），但 BE 树不在本 checkout、须落 HD-C4 前复核。 - **其余翻闸前休眠候选**：**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 From 922775df7e8edabf2bf5e3032d2f4c8ce37f73df Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 22:03:53 +0800 Subject: [PATCH 098/334] [doc](catalog) HMS cutover hudi HD-B1 SIGNED = Option 1 (Connector.ownsHandle neutral SPI) Record the user's 2026-07-08 sign-off on the one new design decision in the hudi work line: the 3-way foreign-handle routing uses a neutral parent-first Connector.ownsHandle(handle) SPI default (each sibling self-identifies its own handle type in its own classloader), chosen over classloader-identity routing because the routing logic is same-loader unit-testable. Implementation must keep the already-landed iceberg delegation spine byte-behavior-identical. Marked in HANDOFF + the authoritative hudi plan doc. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 4 ++-- plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 41bba4c9a3d902..59be9343f0bbb7 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**下一步 = Group A 余下（HD-A2 Kerberos / HD-A3 分区值保真 FIX〔HIGH〕/ HD-A4 force-JNI / HD-A5 BE 描述符，均无需签字）**；**HD-B1 三路 foreign-handle 路由 = 唯一待用户签字新决策**（已解锁，推荐 Option 1 `ownsHandle` 中立 SPI）。 +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**下一步 = Group A 余下（HD-A2 Kerberos / HD-A3 分区值保真 FIX〔HIGH〕/ HD-A4 force-JNI / HD-A5 BE 描述符，均无需签字）**；**HD-B1 三路 foreign-handle 路由：已签字（2026-07-08）= Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**（各兄弟自判、同加载器可单测；实现时须保 iceberg 活脊柱字节等价）。 **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -82,7 +82,7 @@ - **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** - **✅ HD-A0 DONE（`37b85348667`，doc-only）= DV-005 护栏**：修正 `HudiConnectorProvider`/`HudiConnector` javadoc + `CatalogFactory.SPI_READY_TYPES` 注释——`"hudi"` 只是**兄弟类型串**（`createSiblingConnector("hudi",…)` 的查找键，绕过 `SPI_READY_TYPES`），**非**用户可见目录类型、无 `HudiExternalCatalog`；永不加进 `SPI_READY_TYPES`/工厂 case（否则重造 DV-005）。checkstyle 0（hudi+fe-core）。 - **✅ HD-A1 DONE（`28d4781ccac`）= hudi 兄弟持有 + 属性合成（镜像 iceberg S1/S2，休眠）**：新 `HudiSiblingProperties.synthesize`=网关整份属性**逐字防御性拷贝**（与 iceberg 不同**不注入任何 flavor 键**——hudi 无 `iceberg.catalog.type` 类比，`HudiConnector.createClient` 直读 `hive.metastore.uris`/`uri`+raw `hadoop.*`/`fs.*`/`dfs.*`）+ `HiveConnector` 加 `HUDI_CONNECTOR_TYPE` 常量/`volatile hudiSibling` 字段/`getOrCreateHudiSibling()`（volatile 双检懒建、缺插件 fail-loud 且不 memoize、仅以 parent-first `Connector` 持有永不强转）/`close()` 转发兄弟。休眠（无生产引用，待 HD-B2 getTableHandle 分流点亮）。全绿（fe-connector-hive 230 测〔+3 `HudiSiblingPropertiesTest` +6 hudi holder 测含 `closeForwardsToBothSiblingsIndependently` 回归锁〕+ checkstyle 0 + import gate 净）；4 维对抗复审（holder-parity/close-lifecycle/dormancy-synthesis/test-intent）+ verify **0 发现**。 - - **下一步 = 继续 Group A（无需签字）**：HD-A2（hudi 兄弟 Kerberos plugin-auth，防翻闸后 Kerberized HMS 静默降级）/ **HD-A3（分区值解析保真=快照路径静默 NULL 分区列的真 bug，HIGH）**/ HD-A4（force-JNI session 标志 + COW/MOR 检测硬化）/ HD-A5（BE `HIVE_TABLE` 描述符 + BE-canonical 存储属性）。**HD-B1（三路 foreign-handle 路由机制）= 唯一待用户签字新决策**（HD-A1 已解锁它；推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认——可同加载器单测；会改动已落的 iceberg 活脊柱须保字节等价）。 + - **下一步 = 继续 Group A（无需签字）**：HD-A2（hudi 兄弟 Kerberos plugin-auth，防翻闸后 Kerberized HMS 静默降级）/ **HD-A3（分区值解析保真=快照路径静默 NULL 分区列的真 bug，HIGH）**/ HD-A4（force-JNI session 标志 + COW/MOR 检测硬化）/ HD-A5（BE `HIVE_TABLE` 描述符 + BE-canonical 存储属性）。**HD-B1（三路 foreign-handle 路由机制）已签字（2026-07-08）= Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**（HD-A1 已解锁；各兄弟在自己加载器里 instanceof 自判，可同加载器单测；fe-core 永不调用 ownsHandle；实现时会改动已落的 iceberg 活脊柱〔约 30 处按句柄转发点 + 3 个连接器级 get*Provider〕须保 iceberg 臂字节等价）。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md b/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md index 0bbda3d1c12e8f..ebf7bb58e9ab4f 100644 --- a/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md +++ b/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md @@ -283,7 +283,7 @@ touches `SPI_READY_TYPES`. ### Group B — Gateway divert (mirror iceberg + the ONE new design decision) -#### HD-B1 — Three-way foreign-handle routing (THE new design decision — needs sign-off) +#### HD-B1 — Three-way foreign-handle routing (SIGNED 2026-07-08: **Option 1 — `Connector.ownsHandle(handle)` neutral SPI**; the other option below is retained only for context, do not re-litigate) - **Problem:** with two siblings, the binary `if (handle instanceof HiveTableHandle) hive else iceberg` breaks — a `HudiTableHandle` (raw hudi-loader type, cast-invisible across the loader split) would be wrong-routed to the iceberg sibling → cross-loader `ClassCastException` or silently-wrong answers. `ConnectorTableHandle` From a376aeb5d7ec7486e95b56b65ef5fa8ecb9b3ddb Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 22:15:45 +0800 Subject: [PATCH 099/334] [fix](catalog) HMS cutover hudi HD-A3: partition-value parse fidelity (snapshot-path no-regression FIX, dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace HudiScanPlanProvider.parsePartitionValues (naive split-on-'=') with a byte-faithful port of legacy HudiPartitionUtils.parsePartitionValues, fixing a real snapshot-path correctness regression: Hudi's DEFAULT non-hive-style layout yields relative paths like "2024/01" (no "col=" prefix); the old logic silently DROPPED every prefix-less fragment, so a partitioned hudi table read NULL partition columns on a plain snapshot read, and escaped values (%20, %2F) arrived un-decoded. - Positional mapping: a prefix-less fragment maps to the i-th partition column; a "col="-prefixed fragment contributes its suffix (legacy decides per fragment). - Single-column whole-path fallback when the fragment count mismatches; > 1 column with a mismatch fails loud with the exact legacy message. - Every value URL-unescaped via an inlined unescapePathName (byte-faithful port of Hive FileUtils.unescapePathName, identical to the fe-connector-hive HiveWriteUtils copy) so the connector needs no hive-common dependency. - Method made static + package-private for direct unit testing (no live HoodieTableMetaClient needed); produces a column->value map (the existing consumer shape), not legacy's positional list. Scope: this closes the value-parse regression on the UNPRUNED / most-basic read (getAllPartitionPaths returns Hudi's own relative path = the FileSystemView shape). The PRUNED path (applyFilter) still feeds HMS hive-style partition NAMES, correct for hive-sync'd tables; making the pruned partition SOURCE useHiveSyncPartition-aware for non-hive-style tables belongs to the partition-listing step (which ports that source once) and is likewise closed before the catalog flip — recorded, not silently folded in. Dormant (hms not in SPI_READY_TYPES). New HudiPartitionValuesTest (8: positional / hive-style / mixed / unescape / single-col fallback / prefix-strip / multi-col fail-loud / unpartitioned). fe-connector-hudi 41 + hms 56 + api 61 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiScanPlanProvider.java | 80 ++++++++++-- .../hudi/HudiPartitionValuesTest.java | 118 ++++++++++++++++++ 2 files changed, 189 insertions(+), 9 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 9df29b5166889a..bfc86e84e1d689 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -312,25 +312,87 @@ private List resolvePartitions( } /** - * Parse partition path "year=2024/month=01" into column→value map. + * Parse a Hudi partition's relative path into a column→value map, byte-faithful to legacy + * {@code HudiPartitionUtils.parsePartitionValues}. Handles BOTH hive-style ("year=2024/month=01") and Hudi's + * DEFAULT non-hive-style POSITIONAL ("2024/01") layouts, and URL-unescapes every value: + *

      + *
    • A fragment carrying the "col=" prefix contributes the suffix; a fragment WITHOUT it is mapped + * POSITIONALLY to the i-th partition column. The old split-on-'=' logic silently DROPPED a + * prefix-less fragment, so a non-hive-style partitioned table read NULL partition columns on a plain + * snapshot read — this is the regression this fix closes.
      • + *
    • Single partition column with a mismatched fragment count: the whole path (minus an optional "col=" + * prefix) is that column's value (legacy single-column-whole-path fallback).
      • + *
    • Fragment count != column count with > 1 column: fail loud, exactly like legacy.
      • + *
    • Every value is unescaped via {@link #unescapePathName} (e.g. "%20" → space) — legacy delegated + * to Hive's {@code FileUtils.unescapePathName}; inlined here so the connector needs no hive-common + * dependency (mirrors the fe-connector-hive inlined copy).
      • + *
    + * + *

    Static + package-private for direct unit testing (no live HoodieTableMetaClient needed). + * + *

    NOTE: this derives values from the partition path fed to the FileSystemView. On the UNPRUNED path that + * path is Hudi's own relative path (getAllPartitionPaths) = the shape the FileSystemView uses, so values are + * consistent. The PRUNED path (applyFilter) currently feeds HMS hive-style partition NAMES, which match the + * FileSystemView only for hive-sync'd tables; making the pruned partition SOURCE useHiveSyncPartition-aware + * for non-hive-style tables belongs to the partition-listing step (which ports that source once), and is + * likewise closed before the catalog flip. */ - private Map parsePartitionValues( + static Map parsePartitionValues( String partitionPath, List partKeyNames) { - if (partitionPath == null || partitionPath.isEmpty() - || partKeyNames == null || partKeyNames.isEmpty()) { + if (partKeyNames == null || partKeyNames.isEmpty()) { + // Non-partitioned table (legacy returns an empty value list). The unpartitioned scan path always + // reaches here with an empty key list, so an empty partitionPath needs no separate guard. return Collections.emptyMap(); } Map values = new LinkedHashMap<>(); - String[] parts = partitionPath.split("/"); - for (String part : parts) { - int eq = part.indexOf('='); - if (eq > 0) { - values.put(part.substring(0, eq), part.substring(eq + 1)); + String[] fragments = partitionPath.split("/"); + if (fragments.length != partKeyNames.size()) { + if (partKeyNames.size() == 1) { + String prefix = partKeyNames.get(0) + "="; + String value = partitionPath.startsWith(prefix) + ? partitionPath.substring(prefix.length()) : partitionPath; + values.put(partKeyNames.get(0), unescapePathName(value)); + return values; } + throw new DorisConnectorException( + "Failed to parse partition values of path: " + partitionPath); + } + for (int i = 0; i < fragments.length; i++) { + String prefix = partKeyNames.get(i) + "="; + String raw = fragments[i].startsWith(prefix) + ? fragments[i].substring(prefix.length()) : fragments[i]; + values.put(partKeyNames.get(i), unescapePathName(raw)); } return values; } + /** + * URL-decodes a Hive-escaped path component (e.g. "a%2Fb" → "a/b"). Byte-faithful port of Hive's + * {@code org.apache.hadoop.hive.common.FileUtils.unescapePathName} (identical to the fe-connector-hive + * inlined copy in {@code HiveWriteUtils}), so the connector needs no hive-common dependency. + */ + private static String unescapePathName(String path) { + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < path.length(); i++) { + char c = path.charAt(i); + if (c == '%' && i + 2 < path.length()) { + int code = -1; + try { + code = Integer.parseInt(path.substring(i + 1, i + 3), 16); + } catch (Exception e) { + code = -1; + } + if (code >= 0) { + sb.append((char) code); + i += 2; + continue; + } + } + sb.append(c); + } + return sb.toString(); + } + /** * Detect file format from file path suffix. */ diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java new file mode 100644 index 00000000000000..26ab443398731d --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java @@ -0,0 +1,118 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.DorisConnectorException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.Map; + +/** + * Byte-parity tests for {@link HudiScanPlanProvider#parsePartitionValues} against legacy + * {@code HudiPartitionUtils.parsePartitionValues}. Each case pins WHY the behavior matters for BE-visible + * partition columns on a snapshot read. + */ +public class HudiPartitionValuesTest { + + @Test + public void nonHiveStylePositionalPathMapsByPosition() { + // THE regression this fix closes: Hudi's DEFAULT layout (hive_style_partitioning=false) yields relative + // paths like "2024/01" with NO "col=" prefix. The old split-on-'=' logic dropped every prefix-less + // fragment, so the value map was EMPTY -> BE returned NULL partition columns on a plain snapshot read. + Map values = HudiScanPlanProvider.parsePartitionValues( + "2024/01", Arrays.asList("year", "month")); + + Assertions.assertEquals(2, values.size(), "both positional fragments must map to their columns"); + Assertions.assertEquals("2024", values.get("year")); + Assertions.assertEquals("01", values.get("month")); + } + + @Test + public void hiveStylePathStripsColumnPrefix() { + Map values = HudiScanPlanProvider.parsePartitionValues( + "year=2024/month=01", Arrays.asList("year", "month")); + + Assertions.assertEquals("2024", values.get("year")); + Assertions.assertEquals("01", values.get("month")); + } + + @Test + public void mixedPrefixedAndPositionalFragments() { + // Legacy decides per fragment (startsWith "col=" or not), so a mixed path must resolve each side. + Map values = HudiScanPlanProvider.parsePartitionValues( + "year=2024/01", Arrays.asList("year", "month")); + + Assertions.assertEquals("2024", values.get("year"), "prefixed fragment strips the col= prefix"); + Assertions.assertEquals("01", values.get("month"), "prefix-less fragment maps positionally"); + } + + @Test + public void unescapesEscapedValues() { + // Legacy unescaped every value via Hive's FileUtils.unescapePathName; %20 -> space, %2F -> slash. A + // partition value with an escaped char would otherwise reach BE literally (wrong value). + Map values = HudiScanPlanProvider.parsePartitionValues( + "dt=2024-01-01%2012%3A00%3A00", Collections.singletonList("dt")); + + Assertions.assertEquals("2024-01-01 12:00:00", values.get("dt"), "escaped chars must be decoded"); + } + + @Test + public void singleColumnWholePathFallbackWhenFragmentCountMismatches() { + // Single partition column, path has more '/' fragments than columns: legacy maps the WHOLE path to the + // single column (after stripping an optional "col=" prefix), not throw. + Map values = HudiScanPlanProvider.parsePartitionValues( + "2024/01/01", Collections.singletonList("dt")); + + Assertions.assertEquals(1, values.size()); + Assertions.assertEquals("2024/01/01", values.get("dt"), "whole path maps to the single column"); + } + + @Test + public void singleColumnStripsPrefixInWholePathFallback() { + Map values = HudiScanPlanProvider.parsePartitionValues( + "dt=2024/01/01", Collections.singletonList("dt")); + + Assertions.assertEquals("2024/01/01", values.get("dt"), + "the leading col= prefix is stripped before the whole-path fallback"); + } + + @Test + public void multiColumnFragmentCountMismatchFailsLoud() { + // > 1 partition column and a fragment count that does not match: legacy throws rather than silently + // producing a partial/wrong value map. Fail loud, matching legacy. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> HudiScanPlanProvider.parsePartitionValues( + "2024/01/extra", Arrays.asList("year", "month"))); + Assertions.assertTrue(ex.getMessage().contains("2024/01/extra"), + "the failure must name the offending partition path"); + } + + @Test + public void emptyPartitionKeysReturnsEmptyForUnpartitionedTable() { + // Unpartitioned tables reach here with an empty key list and an empty path; the result must be empty + // (no spurious partition column). + Assertions.assertTrue( + HudiScanPlanProvider.parsePartitionValues("", Collections.emptyList()).isEmpty()); + Assertions.assertTrue( + HudiScanPlanProvider.parsePartitionValues("", null).isEmpty()); + } +} From 34c00c0954a15ac90fa0d3ffbe9f0c570eb20948 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 22:22:20 +0800 Subject: [PATCH 100/334] [feat](catalog) HMS cutover hudi HD-A2: plugin-side Kerberos auth for the hudi sibling (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The hudi connector built its ThriftHmsClient AuthAction from context::executeAuthenticated, which post-flip (the hudi sibling shares the hms gateway's FE-injected context) resolves to NOOP/SIMPLE — silently downgrading a Kerberized HMS. Port HiveConnector's plugin-owned authenticator so the metastore RPC runs under the PLUGIN's own UGI doAs. A hudi sibling runs in its OWN classloader, so it must own its authenticator (sharing the gateway's hive-loader authenticator would split the UGI copy across loaders); hadoop + fe-kerberos are bundled child-first in the hudi plugin zip, so the plugin authenticator and the plugin RPC share one UGI copy (same reasoning as the hive gateway). - pom: add fe-connector-metastore-hms (HMS Kerberos parser via MetaStoreProviders.bindForType("hms") + transitively fe-kerberos), auto-bundled child-first by the existing plugin-zip runtime dependencySet. - HudiConnector: byte-faithful mirror of buildPluginAuthenticator / buildHadoopConf / pluginAuthenticator + createClient auth-action selection. Two Kerberos sources in precedence order (raw storage kerberos, then HMS-metastore kerberos with simple storage via HmsMetaStoreProperties.kerberos()); null (FE-injected path unchanged) for a non-Kerberos catalog. Dormant (hms not in SPI_READY_TYPES); Kerberos-only, provable end-to-end only at flip-time e2e against a Kerberized HMS. New HudiConnectorPluginAuthenticatorTest (5, mirror of the hive test, KDC-free — lazy login). fe-connector-hudi 46 + hms 56 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- fe/fe-connector/fe-connector-hudi/pom.xml | 10 ++ .../doris/connector/hudi/HudiConnector.java | 103 ++++++++++++++++- .../HudiConnectorPluginAuthenticatorTest.java | 107 ++++++++++++++++++ 3 files changed, 219 insertions(+), 1 deletion(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPluginAuthenticatorTest.java diff --git a/fe/fe-connector/fe-connector-hudi/pom.xml b/fe/fe-connector/fe-connector-hudi/pom.xml index 2285112734b753..b9d5456d9fde53 100644 --- a/fe/fe-connector/fe-connector-hudi/pom.xml +++ b/fe/fe-connector/fe-connector-hudi/pom.xml @@ -70,6 +70,16 @@ under the License. ${project.version} + + + ${project.groupId} + fe-connector-metastore-hms + ${project.version} + + org.apache.hudi diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index 9b7367ca2333ef..e2d278f9c6eaef 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -25,14 +25,22 @@ import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.ThriftHmsClient; +import org.apache.doris.connector.metastore.HmsMetaStoreProperties; +import org.apache.doris.connector.metastore.spi.MetaStoreProviders; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.kerberos.HadoopAuthenticator; +import org.apache.doris.kerberos.KerberosAuthSpec; +import org.apache.doris.kerberos.KerberosAuthenticationConfig; +import org.apache.hadoop.conf.Configuration; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import java.io.IOException; import java.util.Collections; import java.util.Map; +import java.util.Optional; +import java.util.concurrent.Callable; /** * Hudi connector implementation. Manages the lifecycle of an @@ -51,10 +59,23 @@ public class HudiConnector implements Connector { private static final Logger LOG = LogManager.getLogger(HudiConnector.class); + // Catalog property key gating the plugin-side Kerberos authenticator (value matches AuthType.KERBEROS). + private static final String HADOOP_SECURITY_AUTHENTICATION = "hadoop.security.authentication"; + private final Map properties; private final ConnectorContext context; private volatile HmsClient hmsClient; + // Lazily-built plugin-side Kerberos authenticator (single-owner auth), null for a non-Kerberos catalog. + // Its doAs acts on the PLUGIN's UserGroupInformation copy — the one this connector's ThriftHmsClient RPC + // reads (hadoop + fe-kerberos bundled child-first in the hudi plugin zip) — not the app-loader copy the + // FE-injected context would use. Mirrors HiveConnector: after the catalog flip the sibling shares the hms + // gateway's context whose executeAuthenticated resolves to NOOP (SIMPLE), which would silently downgrade a + // Kerberos HMS. A hudi sibling runs in its OWN classloader, so it must own its authenticator (sharing the + // gateway's hive-loader authenticator would split the UGI copy across loaders). + private volatile HadoopAuthenticator pluginAuth; + private volatile boolean pluginAuthComputed; + public HudiConnector(Map properties, ConnectorContext context) { this.properties = Collections.unmodifiableMap(properties); this.context = context; @@ -99,10 +120,90 @@ private HmsClient createClient() { LOG.info("Creating Hudi connector HMS client for catalog='{}', uri={}, poolSize={}", context.getCatalogName(), config.getMetastoreUri(), poolSize); - ThriftHmsClient.AuthAction authAction = context::executeAuthenticated; + // For a Kerberos catalog run the metastore RPC under the PLUGIN's UGI doAs (buildPluginAuthenticator), + // NOT the FE-injected context: after the catalog flip that context resolves to NOOP (SIMPLE) auth, which + // would silently downgrade a Kerberos HMS. AuthAction.execute is a generic method ( T execute(...)), + // so it cannot be a lambda — use an anonymous class. ThriftHmsClient.doAs already pins the RPC's TCCL to + // the system classloader; the plugin's HadoopAuthenticator only wraps it in a UGI doAs (no TCCL change). + HadoopAuthenticator auth = pluginAuthenticator(); + ThriftHmsClient.AuthAction authAction; + if (auth != null) { + authAction = new ThriftHmsClient.AuthAction() { + @Override + public T execute(Callable callable) throws Exception { + return auth.doAs(callable::call); + } + }; + } else { + authAction = context::executeAuthenticated; + } return new ThriftHmsClient(config, authAction); } + /** + * Lazily builds and memoizes the plugin-side Kerberos authenticator that {@link #createClient()} wraps the + * metastore RPC under, so the RPC uses the PLUGIN's own {@code UserGroupInformation} copy (hadoop + + * fe-kerberos are bundled child-first in the hudi plugin). Returns {@code null} for a non-Kerberos catalog + * so the FE-injected auth path is preserved unchanged. Construction is cheap — the keytab login is lazy in + * {@code getUGI()} on the first {@code doAs}. Mirrors {@code HiveConnector.pluginAuthenticator}. + */ + private HadoopAuthenticator pluginAuthenticator() { + if (!pluginAuthComputed) { + synchronized (this) { + if (!pluginAuthComputed) { + pluginAuth = buildPluginAuthenticator(properties); + pluginAuthComputed = true; + } + } + } + return pluginAuth; + } + + /** + * Resolves the plugin-side Kerberos authenticator for the catalog, or {@code null} for a non-Kerberos + * catalog. Byte-faithful mirror of {@code HiveConnector.buildPluginAuthenticator} — two Kerberos sources in + * precedence order (mirroring the legacy {@code HMSBaseProperties.initHadoopAuthenticator}): + *

      + *
    1. Storage Kerberos — the raw {@code hadoop.security.authentication=kerberos} passthrough (HDFS + * login). When storage is Kerberos this single login also carries the HMS metastore RPC (same UGI).
      2. + *
    2. HMS-metastore Kerberos with non-Kerberos storage — a secured Hive Metastore whose data + * storage is simple (e.g. a Kerberized HMS over S3). The HMS client principal/keytab facts + * ({@link HmsMetaStoreProperties#kerberos()}, resolved through the shared metastore-spi parser) feed a + * {@link KerberosAuthenticationConfig}, so the {@code doAs} logs in the same client identity fe-core + * used.
      3. + *
    + * Package-visible + static for KDC-free unit testing. + */ + static HadoopAuthenticator buildPluginAuthenticator(Map properties) { + if ("kerberos".equalsIgnoreCase(properties.get(HADOOP_SECURITY_AUTHENTICATION))) { + return HadoopAuthenticator.getHadoopAuthenticator(buildHadoopConf(properties)); + } + HmsMetaStoreProperties hms = (HmsMetaStoreProperties) MetaStoreProviders.bindForType( + HmsClientConfig.METASTORE_TYPE_HMS, properties, Collections.emptyMap()); + Optional spec = hms.kerberos(); + if (spec.isPresent() && spec.get().hasCredentials()) { + Configuration conf = buildHadoopConf(properties); + conf.set("hadoop.security.authentication", "kerberos"); + conf.set("hive.metastore.sasl.enabled", "true"); + return HadoopAuthenticator.getHadoopAuthenticator( + new KerberosAuthenticationConfig(spec.get().getPrincipal(), spec.get().getKeytab(), conf)); + } + return null; + } + + /** + * Builds a plain Hadoop {@link Configuration} from the catalog properties for the authenticator. A plain + * {@code new Configuration()} (NOT {@code HiveConf}) is used deliberately: HiveConf static-init would drag + * hadoop-mapreduce onto the unit-test classpath. The classloader is pinned to the plugin loader so the + * child-first (plugin) copy of the auth classes is resolved. Mirrors {@code HiveConnector.buildHadoopConf}. + */ + private static Configuration buildHadoopConf(Map properties) { + Configuration conf = new Configuration(); + conf.setClassLoader(HudiConnector.class.getClassLoader()); + properties.forEach(conf::set); + return conf; + } + @Override public void close() throws IOException { HmsClient c = hmsClient; diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPluginAuthenticatorTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPluginAuthenticatorTest.java new file mode 100644 index 00000000000000..06a084cde4cdb5 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPluginAuthenticatorTest.java @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.kerberos.HadoopAuthenticator; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Unit tests for {@link HudiConnector#buildPluginAuthenticator(Map)} — the connector-owned plugin-side Kerberos + * authenticator resolution for the hudi sibling. Mirrors {@code HiveConnectorPluginAuthenticatorTest}. + * + *

    The load-bearing case is HMS-metastore Kerberos with simple (non-Kerberos) storage (e.g. a + * Kerberized Hive Metastore over S3). After the catalog flip the hudi sibling shares the hms gateway's + * FE-injected {@code ConnectorContext}, whose {@code executeAuthenticated} resolves to NOOP (SIMPLE) auth, so a + * Kerberos HMS would be silently downgraded unless the connector owns the login itself. A hudi sibling runs in + * its OWN classloader, so it must build its OWN authenticator (sharing the gateway's hive-loader authenticator + * would split the UGI copy across loaders). + * + *

    The actual keytab login is lazy (on first {@code doAs}), so these assertions never touch a KDC. + */ +public class HudiConnectorPluginAuthenticatorTest { + + private static Map props(String... kv) { + Map m = new HashMap<>(); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + /** Storage-level Kerberos (raw hadoop.security.authentication) — the prior behavior, still honored. */ + @Test + public void storageKerberosBuildsAuthenticator() { + HadoopAuthenticator auth = HudiConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hadoop.security.authentication", "kerberos", + "hadoop.kerberos.principal", "doris@EXAMPLE.COM", + "hadoop.kerberos.keytab", "/etc/security/doris.keytab")); + Assertions.assertNotNull(auth, "storage kerberos must yield a plugin authenticator"); + } + + /** + * THE BLOCKER CASE: a Kerberized HMS whose data storage is simple. Storage auth is unset, so the storage + * gate is off; the connector must fall back to the HMS client-principal/keytab facts and still build a + * plugin authenticator (mirroring the fe-core HMS authenticator it replaces). Without this a Kerberized + * hudi-on-HMS table would silently downgrade to SIMPLE at the flip. + */ + @Test + public void hmsMetastoreKerberosWithSimpleStorageBuildsAuthenticator() { + HadoopAuthenticator auth = HudiConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hive.metastore.authentication.type", "kerberos", + "hive.metastore.client.principal", "doris@EXAMPLE.COM", + "hive.metastore.client.keytab", "/etc/security/doris.keytab")); + Assertions.assertNotNull(auth, + "HMS-metastore kerberos with simple storage must yield a plugin authenticator"); + } + + /** A simple-auth HMS builds no authenticator (a spurious one would force needless SIMPLE-vs-Kerberos churn). */ + @Test + public void hmsSimpleAuthReturnsNull() { + HadoopAuthenticator auth = HudiConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hive.metastore.authentication.type", "simple")); + Assertions.assertNull(auth, "simple-auth HMS must not build a plugin authenticator"); + } + + /** A plain HMS with no auth configured builds no authenticator. */ + @Test + public void plainHmsWithoutKerberosReturnsNull() { + HadoopAuthenticator auth = HudiConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083")); + Assertions.assertNull(auth, "plain HMS without kerberos must not build an authenticator"); + } + + /** + * HMS declares kerberos auth-type but the client principal/keytab are blank — the {@code hasCredentials} + * guard must reject it (an authenticator with no login pair would fail obscurely at first doAs). + */ + @Test + public void hmsKerberosWithBlankCredsReturnsNull() { + HadoopAuthenticator auth = HudiConnector.buildPluginAuthenticator( + props("hive.metastore.uris", "thrift://hms:9083", + "hive.metastore.authentication.type", "kerberos")); + Assertions.assertNull(auth, "kerberos HMS without a client principal/keytab pair must not build one"); + } +} From cf8710691cfe82a8c5eb660ba292b9fab5747db6 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 22:31:32 +0800 Subject: [PATCH 101/334] [feat](catalog) HMS cutover hudi HD-A4: force_jni_scanner flag + COW/MOR detection hardening (dormant) Honor the force_jni_scanner escape hatch (legacy HudiScanNode parity) and harden the scan-path COW/MOR decision. - force_jni_scanner: read via ConnectorSession.getSessionProperties() (same key + VariableMgr.toMap channel as the paimon connector's FORCE_JNI_SCANNER; default false). Threaded through planScan in BOTH directions: (a) canUseNativeReader = isCow && !forceJni -> a COW table under force_jni takes the merged-file-slice (JNI) path (legacy HudiScanNode.canUseNativeReader); (b) collectMorSplits' no-delta-log native downgrade gains a !forceJni guard, and the flag is BAKED onto HudiScanRange so populateRangeParams (which has no session) suppresses its own no-log downgrade too -- keeping the plan-time native/JNI decision and the range-time downgrade CONSISTENT (legacy setScanParams' !isForceJniScanner() guard). - Detection hardening: planScan now derives isCow from the authoritative Hudi table config (metaClient.getTableType() == COPY_ON_WRITE) instead of the substring-detected handle type, so an UNKNOWN detection cannot silently pick the wrong read path for a COW table. (getScanNodeProperties' file_format_type default still uses the handle type -- best-effort, per-split overridden.) Dormant (hms not in SPI_READY_TYPES); end-to-end only provable at flip-time e2e with force_jni on COW/MOR. New HudiForceJniTest (6: session-flag read x4 incl. null/default-false + populateRangeParams true/false contrast pair pinning the downgrade suppression). fe-connector-hudi 52 + hms 56 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiScanPlanProvider.java | 45 ++++- .../doris/connector/hudi/HudiScanRange.java | 18 +- .../connector/hudi/HudiForceJniTest.java | 162 ++++++++++++++++++ 3 files changed, 216 insertions(+), 9 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index bfc86e84e1d689..fd80c4eadf1ea3 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -32,6 +32,7 @@ import org.apache.hudi.common.model.BaseFile; import org.apache.hudi.common.model.HoodieBaseFile; import org.apache.hudi.common.model.HoodieLogFile; +import org.apache.hudi.common.model.HoodieTableType; import org.apache.hudi.common.table.HoodieTableMetaClient; import org.apache.hudi.common.table.TableSchemaResolver; import org.apache.hudi.common.table.timeline.HoodieInstant; @@ -75,12 +76,30 @@ public class HudiScanPlanProvider implements ConnectorScanPlanProvider { private static final Logger LOG = LogManager.getLogger(HudiScanPlanProvider.class); + // The force_jni_scanner session flag (VariableMgr.toMap channel, read via + // ConnectorSession.getSessionProperties()). When true, the JNI escape hatch is engaged: a native-eligible + // slice is routed to the JNI reader (dodging native-reader bugs), matching legacy + // HudiScanNode.canUseNativeReader() / setScanParams (sessionVariable.isForceJniScanner()). Same key + read + // path as the paimon connector's FORCE_JNI_SCANNER. Default false, so normal reads are unaffected. + private static final String FORCE_JNI_SCANNER = "force_jni_scanner"; + private final Map properties; public HudiScanPlanProvider(Map properties) { this.properties = properties; } + /** + * Reads the {@code force_jni_scanner} session flag from the SPI session properties. Package-private static + * for offline unit testing. Default false (legacy default) when unset or the session is null. + */ + static boolean isForceJniScannerEnabled(ConnectorSession session) { + if (session == null) { + return false; + } + return Boolean.parseBoolean(session.getSessionProperties().get(FORCE_JNI_SCANNER)); + } + @Override public List planScan( ConnectorSession session, @@ -89,7 +108,6 @@ public List planScan( Optional filter) { HudiTableHandle hudiHandle = (HudiTableHandle) handle; String basePath = hudiHandle.getBasePath(); - boolean isCow = "COPY_ON_WRITE".equals(hudiHandle.getHudiTableType()); Configuration conf = buildHadoopConf(); HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder() @@ -97,6 +115,16 @@ public List planScan( .setBasePath(basePath) .build(); + // Determine COW vs MOR from the Hudi table config (authoritative), NOT the substring-detected handle + // type: an UNKNOWN detection must not silently pick the wrong read path for a COW table (detection + // hardening). + boolean isCow = metaClient.getTableType() == HoodieTableType.COPY_ON_WRITE; + // force_jni_scanner routes even a native-eligible read to the JNI reader (legacy + // HudiScanNode.canUseNativeReader() = !isForceJniScanner() && isCowTable), so a COW table under + // force_jni takes the merged-file-slice (JNI) path too. + boolean forceJni = isForceJniScannerEnabled(session); + boolean useNativeCowPath = isCow && !forceJni; + HoodieTimeline timeline = metaClient.getCommitsAndCompactionTimeline() .filterCompletedInstants(); @@ -144,13 +172,13 @@ public List planScan( Map partValues = parsePartitionValues( partitionPath, hudiHandle.getPartitionKeyNames()); - if (isCow) { + if (useNativeCowPath) { collectCowSplits(fsView, partitionPath, queryInstant, basePath, partValues, ranges); } else { collectMorSplits(fsView, partitionPath, queryInstant, basePath, inputFormat, serdeLib, - columnNames, columnTypes, partValues, ranges); + columnNames, columnTypes, partValues, forceJni, ranges); } } @@ -232,7 +260,7 @@ private void collectMorSplits( String partitionPath, String queryInstant, String basePath, String inputFormat, String serdeLib, List columnNames, List columnTypes, - Map partValues, + Map partValues, boolean forceJni, List ranges) { fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, queryInstant) .forEach(fileSlice -> { @@ -245,8 +273,9 @@ private void collectMorSplits( .map(StoragePath::toString) .collect(Collectors.toList()); - // Dynamic format decision: no logs → native reader - boolean useNative = logs.isEmpty() && !filePath.isEmpty(); + // Dynamic format decision: no logs → native reader, UNLESS force_jni keeps it on JNI + // (legacy HudiScanNode.setScanParams' !isForceJniScanner() guard on the no-log downgrade). + boolean useNative = logs.isEmpty() && !filePath.isEmpty() && !forceJni; String format = useNative ? detectFileFormat(filePath) : "jni"; // For log-only slices, use first log as agency path @@ -259,7 +288,9 @@ private void collectMorSplits( .length(fileSize) .fileSize(fileSize) .fileFormat(format) - .partitionValues(partValues); + .partitionValues(partValues) + // Bake force_jni so populateRangeParams (no session) keeps this slice on JNI too. + .forceJni(forceJni); if (!useNative) { // JNI reader needs full metadata diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java index 7566f9ae1b9084..37263de3760090 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java @@ -64,6 +64,12 @@ public class HudiScanRange implements ConnectorScanRange { private final List deltaLogs; private final List columnNames; private final List columnTypes; + // When true (force_jni_scanner), the JNI escape hatch is engaged for this split: the no-delta-log native + // downgrade in populateRangeParams is suppressed so a native-eligible slice still reads via the JNI reader + // (dodging native-reader bugs). Baked in at plan time by HudiScanPlanProvider from the session flag, so + // populateRangeParams (which has no session) stays CONSISTENT with planScan's native/JNI branch. Legacy + // parity: HudiScanNode.setScanParams guards the same downgrade with !sessionVariable.isForceJniScanner(). + private final boolean forceJni; private HudiScanRange(Builder builder) { this.path = builder.path; @@ -104,6 +110,7 @@ private HudiScanRange(Builder builder) { this.columnTypes = builder.columnTypes != null ? Collections.unmodifiableList(new ArrayList<>(builder.columnTypes)) : Collections.emptyList(); + this.forceJni = builder.forceJni; } @Override @@ -165,9 +172,10 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, boolean isJni = "jni".equalsIgnoreCase(getFileFormat()); - // Dynamic format downgrade: if JNI but no delta logs, use native reader + // Dynamic format downgrade: if JNI but no delta logs, use native reader — UNLESS force_jni is engaged + // (then keep the JNI reader, matching legacy HudiScanNode.setScanParams' !isForceJniScanner() guard). if (isJni) { - if (deltaLogs.isEmpty()) { + if (deltaLogs.isEmpty() && !forceJni) { String dataFilePath = props.getOrDefault( "hudi.data_file_path", ""); if (!dataFilePath.isEmpty()) { @@ -249,6 +257,7 @@ public static class Builder { private List deltaLogs; private List columnNames; private List columnTypes; + private boolean forceJni; public Builder path(String path) { this.path = path; @@ -325,6 +334,11 @@ public Builder columnTypes(List columnTypes) { return this; } + public Builder forceJni(boolean forceJni) { + this.forceJni = forceJni; + return this; + } + public HudiScanRange build() { return new HudiScanRange(this); } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java new file mode 100644 index 00000000000000..8dee54c1341ba3 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java @@ -0,0 +1,162 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.thrift.TFileFormatType; +import org.apache.doris.thrift.TFileRangeDesc; +import org.apache.doris.thrift.TTableFormatFileDesc; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.Map; + +/** + * Tests the {@code force_jni_scanner} escape hatch (legacy {@code HudiScanNode.canUseNativeReader()} / + * {@code setScanParams} parity): reading the session flag and suppressing the no-delta-log native downgrade so a + * native-eligible slice still reads via the JNI reader. + */ +public class HudiForceJniTest { + + @Test + public void sessionFlagTrueEnablesForceJni() { + // Pins the EXACT session key ("force_jni_scanner", same key the paimon connector reads and byte-identical + // to SessionVariable.FORCE_JNI_SCANNER). MUTATION: wrong key -> red. + Assertions.assertTrue(HudiScanPlanProvider.isForceJniScannerEnabled( + sessionWithProps(Collections.singletonMap("force_jni_scanner", "true")))); + } + + @Test + public void sessionFlagUnsetDefaultsFalse() { + // Default false (legacy default): normal reads must be unaffected. MUTATION: defaulting true -> red. + Assertions.assertFalse(HudiScanPlanProvider.isForceJniScannerEnabled( + sessionWithProps(Collections.emptyMap()))); + } + + @Test + public void sessionFlagExplicitFalseIsFalse() { + Assertions.assertFalse(HudiScanPlanProvider.isForceJniScannerEnabled( + sessionWithProps(Collections.singletonMap("force_jni_scanner", "false")))); + } + + @Test + public void nullSessionDefaultsFalse() { + Assertions.assertFalse(HudiScanPlanProvider.isForceJniScannerEnabled(null)); + } + + @Test + public void forceJniSuppressesNoDeltaLogNativeDowngrade() { + // A no-delta-log slice with a parquet base file that would normally downgrade to the native reader must + // STAY on JNI when force_jni was engaged at plan time. This is the escape hatch's whole point. + HudiScanRange range = noDeltaLogParquetRange(true); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertNotEquals(TFileFormatType.FORMAT_PARQUET, rangeDesc.getFormatType(), + "force_jni must NOT downgrade a no-log slice to the native parquet reader"); + Assertions.assertTrue(formatDesc.getHudiParams().isSetColumnNames(), + "JNI fileDesc fields must be set when the slice stays on JNI"); + Assertions.assertEquals("20240101000000000", formatDesc.getHudiParams().getInstantTime()); + } + + @Test + public void withoutForceJniNoDeltaLogDowngradesToNative() { + // The paired contrast: SAME inputs, force_jni off -> the no-log slice downgrades to native parquet and + // sets no JNI fields. Together these pin that the ONLY difference is the force_jni flag. + HudiScanRange range = noDeltaLogParquetRange(false); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertEquals(TFileFormatType.FORMAT_PARQUET, rangeDesc.getFormatType(), + "without force_jni a no-log parquet slice downgrades to the native reader"); + Assertions.assertFalse(formatDesc.getHudiParams().isSetColumnNames(), + "native downgrade must not set JNI fileDesc fields"); + } + + private static HudiScanRange noDeltaLogParquetRange(boolean forceJni) { + return new HudiScanRange.Builder() + .path("s3://bucket/t/base.parquet") + .fileFormat("jni") + .instantTime("20240101000000000") + .serde("serde") + .inputFormat("fmt") + .basePath("s3://bucket/t") + .dataFilePath("s3://bucket/t/base.parquet") + .dataFileLength(456L) + .columnNames(Arrays.asList("x")) + .columnTypes(Arrays.asList("int")) + .forceJni(forceJni) + .build(); + } + + private static ConnectorSession sessionWithProps(Map sessionProps) { + return new ConnectorSession() { + @Override + public String getQueryId() { + return "q"; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return 0; + } + + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + + @Override + public Map getSessionProperties() { + return sessionProps; + } + }; + } +} From afe4c1da5a6a2fcd7ae3fb300e24e02a86d94dfc Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 22:40:07 +0800 Subject: [PATCH 102/334] [feat](catalog) HMS cutover hudi HD-A5: BE HIVE_TABLE descriptor + BE-canonical storage props (dormant) Give hudi tables the correct BE descriptor and BE-facing storage, matching legacy (HudiScanNode extends HiveScanNode -> HIVE_TABLE; getLocationProperties dual merge). - HudiConnectorMetadata.buildTableDescriptor override -> TTableType.HIVE_TABLE + THiveTable (port of legacy HMSExternalTable.toThrift, mirror of HiveConnectorMetadata). Without it the SPI default null makes fe-core build a generic SCHEMA_TABLE descriptor -> BE SchemaTableDescriptor instead of HiveTableDescriptor. No handle in the SPI signature, so the one override serves base + system tables. - HudiScanPlanProvider.getScanNodeProperties storage: now emits BE-canonical static creds from ConnectorContext.getBackendStorageProperties() (AWS_* for object stores / resolved hadoop.dfs.* for HDFS) under location.* for the native (FILE_S3) reader -- BE's native reader understands ONLY canonical keys, so the old raw-alias-only copy 403'd a private bucket. The hadoop-format passthrough (fs.s3a.* etc) is kept AFTER the canonical set (legacy putAll order) for the Hudi JNI reader. Threads the ConnectorContext into the provider (new 2-arg ctor; HudiConnector.getScanPlanProvider passes it). Residual (recorded): a JNI (MOR) read over S3 configured with ONLY Doris aliases (s3.access_key, no fs.s3a.*) still lacks hadoop-format creds -- getBackendStorageProperties gives AWS_* (native), not fs.s3a.*; a hadoop-format normalization hook is a later refinement. Verify the hudi plugin zip carries the JNI reader's transitive deps (be-java-ext shared classpath rule) at flip-time. Dormant (hms not in SPI_READY_TYPES); storage correctness provable only at flip-time e2e over a private bucket. New HudiBackendDescriptorTest (3: HIVE_TABLE descriptor + canonical+passthrough merge + no-context passthrough). fe-connector-hudi 55 + hms 56 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hudi/HudiConnector.java | 2 +- .../connector/hudi/HudiConnectorMetadata.java | 22 ++++ .../connector/hudi/HudiScanPlanProvider.java | 18 ++- .../hudi/HudiBackendDescriptorTest.java | 111 ++++++++++++++++++ 4 files changed, 150 insertions(+), 3 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index e2d278f9c6eaef..242e9ae2651b2b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -88,7 +88,7 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { @Override public ConnectorScanPlanProvider getScanPlanProvider() { - return new HudiScanPlanProvider(properties); + return new HudiScanPlanProvider(properties, context); } private HmsClient getOrCreateClient() { diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 3e43b25230fbb3..e40abdec22467e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -34,6 +34,9 @@ import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientException; import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.thrift.THiveTable; +import org.apache.doris.thrift.TTableDescriptor; +import org.apache.doris.thrift.TTableType; import org.apache.avro.Schema; import org.apache.hadoop.conf.Configuration; @@ -219,6 +222,25 @@ public Map getProperties() { return properties; } + /** + * Builds the BE table descriptor for a hudi table: a {@code TTableType.HIVE_TABLE} carrying a + * {@link THiveTable}, a direct port of legacy hudi (which rode {@code HMSExternalTable.toThrift} / + * {@code HudiScanNode extends HiveScanNode} = HIVE_TABLE). Without this override the SPI default returns + * {@code null} and fe-core ({@code PluginDrivenExternalTable.toThrift}) falls back to a generic + * {@code SCHEMA_TABLE} descriptor, so BE builds a SchemaTableDescriptor instead of a HiveTableDescriptor. + * Mirrors {@code HiveConnectorMetadata.buildTableDescriptor}; the SPI signature carries no handle, so this + * single override serves base and system tables alike. + */ + @Override + public TTableDescriptor buildTableDescriptor(ConnectorSession session, + long tableId, String tableName, String dbName, String remoteName, int numCols, long catalogId) { + THiveTable tHiveTable = new THiveTable(dbName, tableName, new HashMap<>()); + TTableDescriptor desc = new TTableDescriptor( + tableId, TTableType.HIVE_TABLE, numCols, 0, tableName, dbName); + desc.setHiveTable(tHiveTable); + return desc; + } + // ========== Internal helpers ========== /** diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index fd80c4eadf1ea3..f7d0c8ed37d43e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.scan.ConnectorScanRange; +import org.apache.doris.connector.spi.ConnectorContext; import org.apache.avro.Schema; import org.apache.hadoop.conf.Configuration; @@ -84,9 +85,11 @@ public class HudiScanPlanProvider implements ConnectorScanPlanProvider { private static final String FORCE_JNI_SCANNER = "force_jni_scanner"; private final Map properties; + private final ConnectorContext context; - public HudiScanPlanProvider(Map properties) { + public HudiScanPlanProvider(Map properties, ConnectorContext context) { this.properties = properties; + this.context = context; } /** @@ -209,7 +212,18 @@ public Map getScanNodeProperties( props.put("path_partition_keys", String.join(",", partKeys)); } - // Location/storage properties for native and JNI readers + // BE-facing storage for the native + JNI readers, mirroring legacy getLocationProperties' dual merge. + // (1) BE-canonical static credentials (AWS_* for object stores, resolved hadoop.*/dfs.* for HDFS): BE's + // native (FILE_S3) reader understands ONLY these canonical keys, so without them a private bucket + // 403s (the raw catalog aliases s3.access_key/... are useless to it). Sourced from the context's + // single normalization hook. Empty for no context (offline tests) or a credential-less warehouse. + if (context != null) { + context.getBackendStorageProperties().forEach((k, v) -> props.put("location." + k, v)); + } + // (2) Hadoop-format passthrough for the Hudi JNI reader (its own Hadoop FileSystem: fs.s3a.* etc). + // Emitted AFTER the canonical set so an overlapping hadoop key resolves to the catalog's explicit + // value (legacy putAll order: backendStorageProperties then hadoopProperties). The s3./oss./cos./obs. + // Doris aliases are harmless to BE (ignored by both readers) but kept so no configured key is dropped. for (Map.Entry entry : properties.entrySet()) { String key = entry.getKey(); if (key.startsWith("hadoop.") || key.startsWith("fs.") diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java new file mode 100644 index 00000000000000..a8f5b88e13f189 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java @@ -0,0 +1,111 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.thrift.TTableDescriptor; +import org.apache.doris.thrift.TTableType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +/** + * Tests the BE-facing surface for hudi tables: + *

      + *
    • {@code buildTableDescriptor} -> TTableType.HIVE_TABLE + THiveTable (legacy hudi rode HIVE_TABLE; without + * this the SPI default null degrades BE to a generic SchemaTableDescriptor).
      • + *
    • {@code getScanNodeProperties} storage -> BE-canonical creds (for the native FILE_S3 reader) + the + * hadoop-format passthrough (for the Hudi JNI reader), the legacy getLocationProperties dual merge.
      • + *
    + */ +public class HudiBackendDescriptorTest { + + @Test + public void buildTableDescriptorIsHiveTable() { + HudiConnectorMetadata md = new HudiConnectorMetadata(null, Collections.emptyMap()); + + TTableDescriptor desc = md.buildTableDescriptor(null, 7L, "t", "db", "t", 3, 100L); + + Assertions.assertEquals(TTableType.HIVE_TABLE, desc.getTableType(), + "legacy hudi rides HIVE_TABLE; a SCHEMA_TABLE default would build the wrong BE descriptor"); + Assertions.assertNotNull(desc.getHiveTable(), "the HIVE_TABLE descriptor must carry a THiveTable"); + Assertions.assertEquals("db", desc.getHiveTable().getDbName()); + Assertions.assertEquals("t", desc.getHiveTable().getTableName()); + } + + @Test + public void scanNodePropertiesEmitsCanonicalCredsAndHadoopPassthrough() { + // A private-bucket read needs BOTH: BE-canonical AWS_* (native FILE_S3 reader) from the context hook, and + // the hadoop fs.s3a.* passthrough (Hudi JNI reader). The old code emitted only the raw passthrough, so the + // native reader had no usable creds (403). + Map catalogProps = new HashMap<>(); + catalogProps.put("fs.s3a.access.key", "hadoopAK"); + HudiScanPlanProvider provider = new HudiScanPlanProvider( + catalogProps, contextWithBackendProps(Collections.singletonMap("AWS_ACCESS_KEY", "canonAK"))); + + Map result = provider.getScanNodeProperties( + null, new HudiTableHandle("db", "t", "s3://b/t", "COPY_ON_WRITE"), + Collections.emptyList(), Optional.empty()); + + Assertions.assertEquals("canonAK", result.get("location.AWS_ACCESS_KEY"), + "BE-canonical creds must be emitted for the native reader"); + Assertions.assertEquals("hadoopAK", result.get("location.fs.s3a.access.key"), + "the hadoop passthrough must be emitted for the JNI reader"); + } + + @Test + public void scanNodePropertiesWithoutContextStillEmitsHadoopPassthrough() { + // Offline / credential-less warehouse: no context -> no canonical overlay, but the hadoop passthrough + // still flows (so a public bucket / HDFS still reads). + Map catalogProps = new HashMap<>(); + catalogProps.put("fs.s3a.access.key", "hadoopAK"); + HudiScanPlanProvider provider = new HudiScanPlanProvider(catalogProps, null); + + Map result = provider.getScanNodeProperties( + null, new HudiTableHandle("db", "t", "s3://b/t", "COPY_ON_WRITE"), + Collections.emptyList(), Optional.empty()); + + Assertions.assertFalse(result.containsKey("location.AWS_ACCESS_KEY"), + "no context must not synthesize canonical creds"); + Assertions.assertEquals("hadoopAK", result.get("location.fs.s3a.access.key")); + } + + private static ConnectorContext contextWithBackendProps(Map backendProps) { + return new ConnectorContext() { + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public long getCatalogId() { + return 0; + } + + @Override + public Map getBackendStorageProperties() { + return backendProps; + } + }; + } +} From 9ad13aa7cb43bb828e3c7056b9634f9dce7af8f6 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 23:00:31 +0800 Subject: [PATCH 103/334] [fix](catalog) HMS cutover hudi: HudiScanRange.populateRangeParams must set the per-range format explicitly Adversarial review of Group A caught a real defect (masked by the existing tests): populateRangeParams never called rangeDesc.setFormatType() for the native path, relying on the SINGLE node-level file_format_type default, which is wrong whenever a slice's actual format differs from that default: - MOR read-optimized / no-delta-log slice (insert-only or post-compaction): collectMorSplits stamps fileFormat="parquet"/"orc" and skips the JNI metadata, but the MOR node default is "jni" -> the range reached BE as FORMAT_JNI with an all-empty THudiFileDesc -> JNI reader errors/garbage (BLOCKER). - COW under force_jni_scanner: planScan correctly routes to the JNI path, but the COW node default is "parquet" -> the range reached BE as FORMAT_PARQUET, silently defeating the escape hatch (native reader used). - COW ORC table: node default parquet -> an ORC base file read as parquet. Fix (mirrors PaimonScanRange.populateRangeParams): after the no-log downgrade resolution, set the format type EXPLICITLY per range -- FORMAT_JNI on the JNI path, else FORMAT_ORC/FORMAT_PARQUET from the range's own file format (or, for a downgraded "jni" range, the base file suffix). New nativeFormatType helper. Root cause the old tests missed: HudiScanRangeTest/HudiForceJniTest built ranges with .fileFormat("jni") + a data_file_path, a shape collectMorSplits NEVER produces for a native slice (it stamps "parquet" directly), so they exercised the dead downgrade block and passed while production was broken (Rule 9 masking). Added nativeParquet/nativeOrc slice tests that build the range as the planner actually does, and strengthened the force_jni test to assert the range reaches BE as FORMAT_JNI. Dormant (hms not in SPI_READY_TYPES). fe-connector-hudi 57 green (HudiScanRangeTest 4, HudiForceJniTest 6), checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hudi/HudiScanRange.java | 51 +++++++++++++------ .../connector/hudi/HudiForceJniTest.java | 4 +- .../connector/hudi/HudiScanRangeTest.java | 41 +++++++++++++++ 3 files changed, 78 insertions(+), 18 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java index 37263de3760090..c9d644ff7b3b31 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java @@ -172,26 +172,25 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, boolean isJni = "jni".equalsIgnoreCase(getFileFormat()); - // Dynamic format downgrade: if JNI but no delta logs, use native reader — UNLESS force_jni is engaged - // (then keep the JNI reader, matching legacy HudiScanNode.setScanParams' !isForceJniScanner() guard). - if (isJni) { - if (deltaLogs.isEmpty() && !forceJni) { - String dataFilePath = props.getOrDefault( - "hudi.data_file_path", ""); - if (!dataFilePath.isEmpty()) { - String lower = dataFilePath.toLowerCase(); - if (lower.endsWith(".parquet")) { - rangeDesc.setFormatType(TFileFormatType.FORMAT_PARQUET); - isJni = false; - } else if (lower.endsWith(".orc")) { - rangeDesc.setFormatType(TFileFormatType.FORMAT_ORC); - isJni = false; - } - } + // A JNI-format split with no delta logs (a read-optimized / log-less slice) reads natively — UNLESS + // force_jni is engaged (legacy HudiScanNode.setScanParams' !isForceJniScanner() guard). In practice + // collectMorSplits/collectCowSplits already stamp the native format directly, so this only resolves a + // defensively-built "jni"+no-log range. + if (isJni && deltaLogs.isEmpty() && !forceJni) { + String dataFilePath = props.getOrDefault("hudi.data_file_path", ""); + String lower = dataFilePath.toLowerCase(); + if (lower.endsWith(".parquet") || lower.endsWith(".orc")) { + isJni = false; } } + // Set the per-range format EXPLICITLY (mirroring PaimonScanRange): the node-level file_format_type is a + // SINGLE default per table and cannot be correct for every slice — a MOR table mixes native no-log + // slices with JNI log slices, a COW ORC table's node default is parquet, and force_jni keeps a COW slice + // on JNI. Relying on that default silently delivered the wrong reader to BE (an empty THudiFileDesc under + // FORMAT_JNI for a native no-log slice, or the native reader for a force_jni / ORC slice). if (isJni) { + rangeDesc.setFormatType(TFileFormatType.FORMAT_JNI); fileDesc.setInstantTime( props.getOrDefault("hudi.instant_time", "")); fileDesc.setSerde(props.getOrDefault("hudi.serde", "")); @@ -217,6 +216,8 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, if (!columnTypes.isEmpty()) { fileDesc.setColumnTypes(columnTypes); } + } else { + rangeDesc.setFormatType(nativeFormatType(props)); } formatDesc.setHudiParams(fileDesc); @@ -238,6 +239,24 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, } } + /** + * The BE native reader format for a non-JNI slice: from the range's own file format when it is already + * native (collectCowSplits / a no-log MOR slice stamp "parquet"/"orc" directly), else — for a "jni" range + * downgraded above — from the base file suffix. Defaults to parquet (matching {@code detectFileFormat}). + */ + private TFileFormatType nativeFormatType(Map props) { + String fmt = getFileFormat(); + if ("orc".equalsIgnoreCase(fmt)) { + return TFileFormatType.FORMAT_ORC; + } + if ("parquet".equalsIgnoreCase(fmt)) { + return TFileFormatType.FORMAT_PARQUET; + } + String dataFilePath = props.getOrDefault("hudi.data_file_path", ""); + return dataFilePath.toLowerCase().endsWith(".orc") + ? TFileFormatType.FORMAT_ORC : TFileFormatType.FORMAT_PARQUET; + } + /** Builder for constructing HudiScanRange instances. */ public static class Builder { private String path; diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java index 8dee54c1341ba3..34452f148afa10 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiForceJniTest.java @@ -72,8 +72,8 @@ public void forceJniSuppressesNoDeltaLogNativeDowngrade() { TFileRangeDesc rangeDesc = new TFileRangeDesc(); range.populateRangeParams(formatDesc, rangeDesc); - Assertions.assertNotEquals(TFileFormatType.FORMAT_PARQUET, rangeDesc.getFormatType(), - "force_jni must NOT downgrade a no-log slice to the native parquet reader"); + Assertions.assertEquals(TFileFormatType.FORMAT_JNI, rangeDesc.getFormatType(), + "force_jni must keep the slice on the JNI reader (explicit FORMAT_JNI), not the native reader"); Assertions.assertTrue(formatDesc.getHudiParams().isSetColumnNames(), "JNI fileDesc fields must be set when the slice stays on JNI"); Assertions.assertEquals("20240101000000000", formatDesc.getHudiParams().getInstantTime()); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java index 7f8aeeebee8d0e..9a4628233a8af0 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java @@ -62,6 +62,9 @@ public void testJniListsSurviveIntactAndAligned() { THudiFileDesc fileDesc = formatDesc.getHudiParams(); + // A log-bearing MOR slice must set FORMAT_JNI explicitly (not rely on the node default). + Assertions.assertEquals(TFileFormatType.FORMAT_JNI, rangeDesc.getFormatType()); + // Types must NOT be shattered: 3 columns -> 3 type strings (old bug // produced 5: "decimal(10","2)","struct"). Assertions.assertEquals(Arrays.asList("int", "decimal(10,2)", "struct"), @@ -92,4 +95,42 @@ public void testNoDeltaLogsDowngradesToNativeParquet() { Assertions.assertFalse(formatDesc.getHudiParams().isSetColumnTypes()); Assertions.assertFalse(formatDesc.getHudiParams().isSetColumnNames()); } + + @Test + public void nativeParquetSliceExplicitlySetsFormatType() { + // A native slice as collectMorSplits/collectCowSplits actually build it: fileFormat="parquet" (NOT + // "jni"), no JNI metadata. populateRangeParams MUST set FORMAT_PARQUET explicitly — relying on the + // node-level default (jni for a MOR table) shipped an empty THudiFileDesc under FORMAT_JNI to BE. + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/base.parquet") + .fileFormat("parquet") + .fileSize(456L) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertEquals(TFileFormatType.FORMAT_PARQUET, rangeDesc.getFormatType(), + "a native parquet slice must set FORMAT_PARQUET, not inherit the node default"); + Assertions.assertFalse(formatDesc.getHudiParams().isSetColumnNames(), "native slice sets no JNI fields"); + } + + @Test + public void nativeOrcSliceExplicitlySetsFormatType() { + // A COW ORC table's node default is parquet; without an explicit per-range set BE would read the ORC + // base file as parquet. + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/base.orc") + .fileFormat("orc") + .fileSize(456L) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertEquals(TFileFormatType.FORMAT_ORC, rangeDesc.getFormatType(), + "a native orc slice must set FORMAT_ORC, not inherit the parquet node default"); + } } From 70ea6ccfc04e31c71380a0083a7df45b839e8b96 Mon Sep 17 00:00:00 2001 From: morningman Date: Wed, 8 Jul 2026 23:02:39 +0800 Subject: [PATCH 104/334] [doc](catalog) HMS cutover hudi Group A (HD-A0-A5) DONE; next = HD-B1 (signed Option 1) + Group C/D Record all of Group A complete: A0 guardrail, A1 sibling holder, A2 Kerberos, A3 partition-value fidelity FIX, A4 force-JNI + detection hardening, A5 BE HIVE_TABLE descriptor + BE-canonical storage. Includes the adversarial review catch (ce700db35e0): populateRangeParams must set the per-range format explicitly (MOR no-log native slice + COW-force_jni + COW-ORC were mis-formatted via the single node default). Next = HD-B1 three-way routing (signed Option 1 ownsHandle, must keep the iceberg spine byte-identical), then Group C/D, then HD-B2 pivot. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 59be9343f0bbb7..28d39819e8e631 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-08） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**下一步 = Group A 余下（HD-A2 Kerberos / HD-A3 分区值保真 FIX〔HIGH〕/ HD-A4 force-JNI / HD-A5 BE 描述符，均无需签字）**；**HD-B1 三路 foreign-handle 路由：已签字（2026-07-08）= Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**（各兄弟自判、同加载器可单测；实现时须保 iceberg 活脊柱字节等价）。 +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**下一步 = HD-B1 三路 foreign-handle 路由（已签字 2026-07-08 = Option 1 `Connector.ownsHandle(handle)` 中立 SPI；改动已落 iceberg 活脊柱须保字节等价）→ Group C 无回退补齐（MVCC/listPartitions、时间旅行、增量读〔最高风险〕、schema evolution）+ Group D 写-拒安全网 → HD-B2 getTableHandle 分流点亮（最后一步）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -82,12 +82,16 @@ - **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** - **✅ HD-A0 DONE（`37b85348667`，doc-only）= DV-005 护栏**：修正 `HudiConnectorProvider`/`HudiConnector` javadoc + `CatalogFactory.SPI_READY_TYPES` 注释——`"hudi"` 只是**兄弟类型串**（`createSiblingConnector("hudi",…)` 的查找键，绕过 `SPI_READY_TYPES`），**非**用户可见目录类型、无 `HudiExternalCatalog`；永不加进 `SPI_READY_TYPES`/工厂 case（否则重造 DV-005）。checkstyle 0（hudi+fe-core）。 - **✅ HD-A1 DONE（`28d4781ccac`）= hudi 兄弟持有 + 属性合成（镜像 iceberg S1/S2，休眠）**：新 `HudiSiblingProperties.synthesize`=网关整份属性**逐字防御性拷贝**（与 iceberg 不同**不注入任何 flavor 键**——hudi 无 `iceberg.catalog.type` 类比，`HudiConnector.createClient` 直读 `hive.metastore.uris`/`uri`+raw `hadoop.*`/`fs.*`/`dfs.*`）+ `HiveConnector` 加 `HUDI_CONNECTOR_TYPE` 常量/`volatile hudiSibling` 字段/`getOrCreateHudiSibling()`（volatile 双检懒建、缺插件 fail-loud 且不 memoize、仅以 parent-first `Connector` 持有永不强转）/`close()` 转发兄弟。休眠（无生产引用，待 HD-B2 getTableHandle 分流点亮）。全绿（fe-connector-hive 230 测〔+3 `HudiSiblingPropertiesTest` +6 hudi holder 测含 `closeForwardsToBothSiblingsIndependently` 回归锁〕+ checkstyle 0 + import gate 净）；4 维对抗复审（holder-parity/close-lifecycle/dormancy-synthesis/test-intent）+ verify **0 发现**。 - - **下一步 = 继续 Group A（无需签字）**：HD-A2（hudi 兄弟 Kerberos plugin-auth，防翻闸后 Kerberized HMS 静默降级）/ **HD-A3（分区值解析保真=快照路径静默 NULL 分区列的真 bug，HIGH）**/ HD-A4（force-JNI session 标志 + COW/MOR 检测硬化）/ HD-A5（BE `HIVE_TABLE` 描述符 + BE-canonical 存储属性）。**HD-B1（三路 foreign-handle 路由机制）已签字（2026-07-08）= Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**（HD-A1 已解锁；各兄弟在自己加载器里 instanceof 自判，可同加载器单测；fe-core 永不调用 ownsHandle；实现时会改动已落的 iceberg 活脊柱〔约 30 处按句柄转发点 + 3 个连接器级 get*Provider〕须保 iceberg 臂字节等价）。 + - **✅ HD-A2 DONE（`561f777d5f9`）= hudi 兄弟 Kerberos plugin-auth**：端口 `HiveConnector.buildPluginAuthenticator`/`buildHadoopConf`/createClient auth-action（+pom 加 `fe-connector-metastore-hms`＝带 HMS Kerberos 解析器 + 传递 fe-kerberos，插件 zip runtime dependencySet 自动 child-first 打包）；走插件 UGI doAs 否则翻闸后共享网关 NOOP context 静默把 Kerberized HMS 降级 SIMPLE；**hudi 兄弟自有加载器故须自持认证器**（共享 hive 网关的会跨加载器裂 UGI 副本）。5 测（镜像 hive，KDC-free 惰性登录）。 + - **✅ HD-A3 DONE（`19f11063a79`）= 分区值解析保真 FIX**：`parsePartitionValues` 逐字端口 legacy `HudiPartitionUtils.parsePartitionValues`（positional 映射 + 单列整路回退 + `>1` 列 fail-loud + 内联 `unescapePathName`＝Hive FileUtils 字节复刻、无 hive-common 依赖），修非-hive-style 分区（`2024/01`）静默读 NULL 分区列 + 未 unescape 的真 bug；**范围仅值解析（unpruned 最基本读）**；pruned-path HMS-hive-style-vs-Hudi-positional 源一致性（须 `useHiveSyncPartition`）归 **HD-C1 partition-listing 步**（亦翻闸前、无翻闸回退，已记账非静默跳）。8 测。 + - **✅ HD-A4 DONE（`7bc90edf4a3` + 复审修 `ce700db35e0`）= force_jni_scanner + COW/MOR 检测硬化**：读 `force_jni_scanner` session 标志（镜像 paimon），planScan `useNativeCowPath = isCow && !forceJni`（force-JNI 把 COW 也推 JNI 路＝legacy `canUseNativeReader`）+ collectMorSplits 无-log native 降级加 `!forceJni` 守卫；**isCow 改由权威 `metaClient.getTableType()` 定**（非 substring 检测，UNKNOWN 不再静默误选路）。**5 维对抗复审揪出真缺陷（本 session 修）**：`populateRangeParams` 未显式 `setFormatType`、只靠单一 node 默认 → ①MOR 无-log native slice 当 `FORMAT_JNI` 空 THudiFileDesc 发 BE（blocker）②COW+force_jni 当 native（major，架空开关）③COW-ORC 当 parquet——已**镜像 `PaimonScanRange` 逐 range 显式设格式**修好（旧测用 `fileFormat("jni")` 假形状掩盖＝Rule-9 masking，已补 planner 真形状 native-parquet/orc 测）。6+4 测。 + - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 + - **下一步 = HD-B1（见文首「下一步」）**——三路 foreign-handle 路由，已签字 Option 1 `Connector.ownsHandle` 中立 SPI；改动已落 iceberg 活脊柱（约 30 处 `!(handle instanceof HiveTableHandle)` 转发点 + 3 个连接器级 `get*Provider(handle)`）须保 iceberg 臂字节等价；fe-core 永不调用 ownsHandle。然后 Group C（无回退补齐）+ Group D（写-拒）+ HD-B2（分流点亮，最后一步）。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 - **DV-005 模型错配已解**：`HudiConnectorProvider.getType()="hudi"` 保留但**只作兄弟类型串**（`createSiblingConnector("hudi",…)` 绕过 `SPI_READY_TYPES`）；**永不**把 `hudi` 加进 `SPI_READY_TYPES`/工厂 case（会重造 DV-005）。 - - **有序子步（HD-A0..HD-D1，4 组；✅HD-A0/HD-A1 已收官，见上）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 + - **有序子步（HD-A0..HD-D1，4 组；✅Group A（HD-A0–A5）全部收官，见上）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 - **⛔ 最大风险 HD-C3（§5.1）**：COW 增量的行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 `CheckPolicy` 注入、通用路径**跳过**且铁律禁 fe-core 复刻；COW 更新重写整个 base 文件把旧 commit 行带进→纯文件级选择返窗口外行=**静默错答**。须连接器把行谓词推进 scan-range/thrift **或** BE reader 按 instant 窗口行级过滤——**须先读 BE Hudi reader 源码判定**（本 checkout 无 BE 树），可能在铁律下无 fe-core 解，且只能翻闸 e2e 验（straddling-base-file fixture）。paimon **非**先例。 - **翻闸硬门 B1–B6**（见 plan §4.2）+ 两非单测 harness（两加载器路由 fixture + 异构 HMS e2e，含 COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved·非 hive-style/转义分区·写-拒）。**schema 缺失 SIGABRT 方向**（§5.2）已由 critic 对 BE 核对**降级为窄回退非崩溃**（emit-nothing=安全基线走 BE 小写 name-match），但 BE 树不在本 checkout、须落 HD-C4 前复核。 - **其余翻闸前休眠候选**：**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 From 984137129538d5b5b6bd0c48ad2c5f3a36e03af5 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 00:12:37 +0800 Subject: [PATCH 105/334] [feat](catalog) HMS cutover hudi HD-B1: three-way foreign-handle routing via Connector.ownsHandle (dormant) Generalize the flipped hms gateway's binary per-handle discriminator (hive-else-iceberg) to a 3-way router now that a second sibling (hudi) exists: - New neutral SPI Connector.ownsHandle(handle) (default false); IcebergConnector and HudiConnector override it with their OWN in-loader instanceof. fe-core never calls it, so the engine stays format-agnostic. - HiveConnector.resolveSiblingOwner PEEKS the already-built sibling fields (never force-builds) and routes a foreign handle to the sibling that owns it, else fails loud naming the catalog. The 3 connector-level get*Provider(handle) seams + the ~38 HiveConnectorMetadata guard-and-forward sites route through it; the getTableHandle ICEBERG by-type divert keeps the force-build path (renamed icebergSiblingMetadata). - PEEK, not force-build: a hudi-only hms catalog without the iceberg plugin still routes its hudi handles, and the iceberg arm stays byte-behaviour-identical (the owning sibling is always built by getTableHandle before it produces the handle). Dormant until hms enters SPI_READY_TYPES. Foreign handles are never cast. Tests: ownsHandle default (api) + overrides (iceberg/hudi); new HiveConnectorThreeWayRoutingTest (iceberg/hudi/hive routing, unknown fail-loud, peek no-force-build in both directions); updated scan/write/procedure/metadata divert suites for peek + ownsHandle (delegation suite fail-louds the by-type supplier so a site regressing off the peek resolver is caught). 4-lens adversarial review: 0 blocker/major/minor, 2 nits (test tightening) fixed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/api/Connector.java | 18 ++ .../ConnectorScanProviderSelectionTest.java | 13 + .../doris/connector/hive/HiveConnector.java | 73 +++-- .../connector/hive/HiveConnectorMetadata.java | 133 ++++---- ...onnectorMetadataSiblingDelegationTest.java | 25 +- ...onnectorMetadataTableHandleDivertTest.java | 6 +- .../HiveConnectorProcedureOpsDivertTest.java | 22 +- .../HiveConnectorScanProviderDivertTest.java | 24 +- .../HiveConnectorThreeWayRoutingTest.java | 284 ++++++++++++++++++ .../HiveConnectorWriteProviderDivertTest.java | 22 +- .../doris/connector/hudi/HudiConnector.java | 12 + .../hudi/HudiConnectorOwnsHandleTest.java | 62 ++++ .../connector/iceberg/IcebergConnector.java | 12 + .../iceberg/IcebergConnectorTest.java | 15 + 14 files changed, 625 insertions(+), 96 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index a51a6a4f2533ef..9dfbaf5c36e760 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -43,6 +43,24 @@ public interface Connector extends Closeable { /** Returns the metadata interface for the given session. */ ConnectorMetadata getMetadata(ConnectorSession session); + /** + * Whether {@code handle} is one of THIS connector's own concrete {@link ConnectorTableHandle} subclasses. + * + *

    A heterogeneous gateway connector that serves several table formats through embedded sibling + * connectors uses this to route a foreign handle to the sibling that produced it: the sibling's concrete + * handle type is invisible across the plugin classloader split, so the gateway cannot {@code instanceof} it + * directly — it asks each sibling, and the sibling tests its OWN in-loader type. The default returns + * {@code false} (a connector owns no handle it did not define), so every non-gateway connector is + * unaffected.

    + * + *

    fe-core NEVER calls this — it is a connector-to-sibling routing predicate only, so the engine stays + * format-agnostic (it discriminates handles solely by the gateway's own handle type, never by asking a + * connector to classify one).

    + */ + default boolean ownsHandle(ConnectorTableHandle handle) { + return false; + } + /** Returns the scan plan provider for split generation. */ default ConnectorScanPlanProvider getScanPlanProvider() { return null; diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java index b03af7599d0ce5..fe0c66e23dc43c 100644 --- a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java @@ -138,4 +138,17 @@ public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle Assertions.assertNotSame(fallback, gateway.getScanPlanProvider(icebergHandle), "an overriding gateway does not fall back to the connector-level no-arg provider"); } + + @Test + public void ownsHandleDefaultsFalse() { + // A connector owns no handle it did not define. The default must be false so a gateway asking each + // embedded sibling "is this foreign handle yours?" gets a negative from every connector that did not + // produce it, and only the producing sibling (which overrides ownsHandle) claims it. + // MUTATION: flipping the default to true -> a gateway would route a foreign handle to the WRONG sibling. + Connector connector = new FakeConnector() { + }; + + Assertions.assertFalse(connector.ownsHandle(handle()), + "the ownsHandle default must be false (a connector owns no handle it did not define)"); + } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 3a73279d46dcdf..ac108df3325124 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -106,10 +106,41 @@ public HiveConnector(Map properties, ConnectorContext context) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - // Pass the sibling supplier (not the built sibling): a pure-hive query never invokes it, so a hive-only - // deployment without the iceberg plugin never builds/throws. The metadata diverts a foreign iceberg - // handle through it (per-handle guard-and-forward). - return new HiveConnectorMetadata(getOrCreateClient(), properties, context, this::getOrCreateIcebergSibling); + // Two sibling seams for the metadata's guard-and-forward: + // - getOrCreateIcebergSibling (by-TYPE, force-build): only the getTableHandle ICEBERG divert uses it, to + // build+ask the iceberg sibling for a table detected as iceberg (no handle exists yet to route by). + // - resolveSiblingOwner (by-HANDLE, peek): every per-handle site routes a foreign handle to whichever + // ALREADY-BUILT sibling owns it. Passing the resolver (not a built sibling) keeps a pure-hive query + // from ever building/throwing a sibling. + return new HiveConnectorMetadata(getOrCreateClient(), properties, context, + this::getOrCreateIcebergSibling, this::resolveSiblingOwner); + } + + /** + * Resolves which embedded sibling connector OWNS a foreign (non-hive) table handle, for the per-handle + * gateway seams (the connector-level {@code get*Provider(handle)} below and the ~34 guard-and-forward + * methods in {@link HiveConnectorMetadata}). Asks each sibling's {@link Connector#ownsHandle} — the sibling + * tests its OWN in-loader handle type, which is invisible to the hive loader across the plugin split, so the + * gateway can never {@code instanceof} the foreign handle itself. + * + *

    Consults only ALREADY-BUILT siblings (a plain field read, never {@code getOrCreate*}). The owning + * sibling is always already built: a foreign handle can only originate from {@code getTableHandle}'s divert, + * which force-builds that sibling before producing the handle. Reading the field (not force-building) avoids + * demanding an UNRELATED plugin merely to classify a handle — e.g. a hudi-only hms catalog with no iceberg + * plugin must still route its hudi handles without building an iceberg sibling. Fails loud when no built + * sibling owns the handle (an orphan handle is a bug, not a route), naming the catalog. + */ + private Connector resolveSiblingOwner(ConnectorTableHandle handle) { + Connector iceberg = icebergSibling; + if (iceberg != null && iceberg.ownsHandle(handle)) { + return iceberg; + } + Connector hudi = hudiSibling; + if (hudi != null && hudi.ownsHandle(handle)) { + return hudi; + } + throw new DorisConnectorException("Cannot route a foreign table handle in catalog '" + + context.getCatalogName() + "': no embedded sibling connector owns it"); } @Override @@ -125,16 +156,17 @@ public ConnectorScanPlanProvider getScanPlanProvider() { * built in the iceberg plugin's classloader, {@code PluginDrivenScanNode.onPluginClassLoader} auto-pins the * scan-thread TCCL to the iceberg loader for free (it keys off {@code provider.getClass().getClassLoader()}), * so no pinning is needed here. The foreign handle is passed through UNMODIFIED and NEVER cast (its concrete - * iceberg type is invisible across the loader split — a cast would CCE). A HUDI table keeps a HiveTableHandle, - * so it stays on the hive scan path (its delegation is a later substep). Pairs with the getTableHandle iceberg - * divert; dormant until hms enters SPI_READY_TYPES (nothing selects a scan provider for this connector today). + * sibling type is invisible across the loader split — a cast would CCE). A HUDI table (once its divert lands) + * routes to the hudi sibling by the 3-way {@link #resolveSiblingOwner} — a HUDI-stamped HiveTableHandle stays + * hive. Pairs with the getTableHandle diverts; dormant until hms enters SPI_READY_TYPES (nothing selects a + * scan provider for this connector today). */ @Override public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { if (handle instanceof HiveTableHandle) { return getScanPlanProvider(); } - return getOrCreateIcebergSibling().getScanPlanProvider(handle); + return resolveSiblingOwner(handle).getScanPlanProvider(handle); } @Override @@ -143,29 +175,28 @@ public ConnectorWritePlanProvider getWritePlanProvider() { } /** - * Per-table write provider: a hive handle uses the hive write provider; a foreign (iceberg-on-HMS) handle is - * delegated to the sibling's per-handle write provider (built in the iceberg plugin's classloader). The - * foreign handle is passed through UNMODIFIED and NEVER cast (its concrete iceberg type is invisible across - * the loader split — a cast would CCE). A HUDI table keeps a HiveTableHandle, so it stays on the hive write - * path (its delegation is a later substep). Mirrors {@link #getScanPlanProvider(ConnectorTableHandle)}; - * dormant until hms enters SPI_READY_TYPES. The returned sibling provider runs its planWrite on fe-core - * threads — the write-path TCCL pin is a separate flip-time concern (write-delegation W6). + * Per-table write provider: a hive handle uses the hive write provider; a foreign handle is delegated to the + * OWNING sibling's per-handle write provider (resolved 3-way by {@link #resolveSiblingOwner}). The foreign + * handle is passed through UNMODIFIED and NEVER cast (its concrete sibling type is invisible across the loader + * split — a cast would CCE). A HUDI-stamped HiveTableHandle stays on the hive write path. Mirrors {@link + * #getScanPlanProvider(ConnectorTableHandle)}; dormant until hms enters SPI_READY_TYPES. The returned sibling + * provider runs its planWrite on fe-core threads — the write-path TCCL pin is a separate flip-time concern. */ @Override public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { if (handle instanceof HiveTableHandle) { return getWritePlanProvider(); } - return getOrCreateIcebergSibling().getWritePlanProvider(handle); + return resolveSiblingOwner(handle).getWritePlanProvider(handle); } /** * Per-table procedure ops for {@code ALTER TABLE ... EXECUTE}: a hive handle has NO procedures — it inherits - * the connector-level {@code null} (plain-hive exposes none) — while a foreign (iceberg-on-HMS) handle is - * delegated to the sibling's per-handle procedure ops (built in the iceberg plugin's classloader), so an + * the connector-level {@code null} (plain-hive exposes none) — while a foreign handle is delegated to the + * OWNING sibling's per-handle procedure ops (resolved 3-way by {@link #resolveSiblingOwner}), so an * iceberg-on-HMS table gains the native iceberg procedures (rollback_to_snapshot, rewrite_data_files, ...). - * The foreign handle is passed through UNMODIFIED and NEVER cast (its concrete iceberg type is invisible - * across the loader split — a cast would CCE). A HUDI table keeps a HiveTableHandle, so it too inherits the + * The foreign handle is passed through UNMODIFIED and NEVER cast (its concrete sibling type is invisible + * across the loader split — a cast would CCE). A HUDI-stamped HiveTableHandle stays hive and inherits the * null (no procedures), same as plain-hive. Mirrors {@link #getWritePlanProvider(ConnectorTableHandle)}; * dormant until hms enters SPI_READY_TYPES (nothing selects procedure ops for this connector today). */ @@ -174,7 +205,7 @@ public ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { if (handle instanceof HiveTableHandle) { return getProcedureOps(); } - return getOrCreateIcebergSibling().getProcedureOps(handle); + return resolveSiblingOwner(handle).getProcedureOps(handle); } @Override diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index e4c1f009357ab0..6f82a786706ce4 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -89,6 +89,7 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.function.Function; import java.util.function.Supplier; import java.util.function.ToLongFunction; import java.util.stream.Collectors; @@ -172,6 +173,13 @@ public class HiveConnectorMetadata implements ConnectorMetadata { throw new DorisConnectorException("no iceberg sibling connector configured for this hive metadata"); }; + // The by-handle owner resolver installed by the 3-arg constructor (hive-only construction). Invoked only when + // a NON-hive handle reaches a per-handle guard-and-forward site — which such a construction never does — so it + // fails loud instead of NPEing. + private static final Function NO_SIBLING_OWNER = handle -> { + throw new DorisConnectorException("no sibling connector configured for this hive metadata"); + }; + private final HmsClient hmsClient; private final Map properties; private final HmsTypeMapping.Options typeMappingOptions; @@ -180,36 +188,56 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // read from FE Config. The default getEnvironment() is an empty map, so direct-construction tests that // pass a bare context degrade to the hard-coded fallbacks in createTable. private final ConnectorContext context; - // Supplies the embedded iceberg SIBLING connector for delegating a NON-hive (iceberg-on-HMS) table handle - // (see the guard-and-forward methods). Lazy: HiveConnector.getOrCreateIcebergSibling builds it only on first - // use, so a pure-hive query never triggers it. The sibling is used ONLY through the parent-first Connector / - // ConnectorMetadata interfaces — its concrete iceberg types are never cast here (cross-loader CCE). + // Supplies the embedded iceberg SIBLING connector BY TYPE, for the getTableHandle ICEBERG divert only (an + // iceberg-detected table has no handle yet to route by, so the sibling is force-built and asked directly). + // Lazy: HiveConnector.getOrCreateIcebergSibling builds it only on first use, so a pure-hive query never + // triggers it. Used ONLY through the parent-first Connector / ConnectorMetadata interfaces — its concrete + // iceberg types are never cast here (cross-loader CCE). private final Supplier icebergSiblingSupplier; + // Resolves the embedded sibling connector that OWNS a foreign (non-hive) table handle, for the per-handle + // guard-and-forward methods below. Backed by HiveConnector.resolveSiblingOwner (a 3-way ownsHandle dispatch + // over the ALREADY-BUILT iceberg / hudi siblings). Used ONLY through the parent-first Connector / + // ConnectorMetadata interfaces — the owning sibling's concrete types are never cast here (cross-loader CCE). + private final Function siblingOwnerResolver; public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context) { - this(hmsClient, properties, context, NO_ICEBERG_SIBLING); + this(hmsClient, properties, context, NO_ICEBERG_SIBLING, NO_SIBLING_OWNER); } public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context, - Supplier icebergSiblingSupplier) { + Supplier icebergSiblingSupplier, + Function siblingOwnerResolver) { this.hmsClient = hmsClient; this.properties = properties; this.context = context; this.typeMappingOptions = buildTypeMappingOptions(properties); this.icebergSiblingSupplier = icebergSiblingSupplier; + this.siblingOwnerResolver = siblingOwnerResolver; } /** - * The embedded iceberg sibling's metadata, for delegating a NON-hive (foreign iceberg) table handle in the - * guard-and-forward methods below. Obtained fresh per call — parity with fe-core, which acquires a - * ConnectorMetadata per operation; the heavy catalog/caches live on the single (memoized) sibling connector, - * so this is cheap. The returned metadata and any handle it produces are used ONLY through the parent-first - * SPI interfaces and MUST NOT be cast (the sibling's concrete iceberg types would CCE across the loader split). + * The embedded iceberg sibling's metadata resolved BY TYPE, for the getTableHandle ICEBERG divert only (an + * iceberg-detected table has no handle yet, so the sibling is force-built and asked directly). Obtained fresh + * per call — parity with fe-core, which acquires a ConnectorMetadata per operation; the heavy catalog/caches + * live on the single (memoized) sibling connector, so this is cheap. The returned metadata and any handle it + * produces are used ONLY through the parent-first SPI interfaces and MUST NOT be cast (the sibling's concrete + * iceberg types would CCE across the loader split). */ - private ConnectorMetadata siblingMetadata(ConnectorSession session) { + private ConnectorMetadata icebergSiblingMetadata(ConnectorSession session) { return icebergSiblingSupplier.get().getMetadata(session); } + /** + * The OWNING sibling's metadata for a foreign (non-hive) table handle, resolved BY HANDLE (3-way ownsHandle + * dispatch over the already-built iceberg / hudi siblings — see HiveConnector.resolveSiblingOwner). Every + * per-handle guard-and-forward method routes through here so a hudi handle reaches the hudi sibling and an + * iceberg handle the iceberg sibling. Obtained fresh per call; the handle is used ONLY through the + * parent-first SPI interfaces and MUST NOT be cast (cross-loader CCE). + */ + private ConnectorMetadata siblingMetadata(ConnectorSession session, ConnectorTableHandle handle) { + return siblingOwnerResolver.apply(handle).getMetadata(session); + } + // ========== ConnectorSchemaOps ========== @Override @@ -254,7 +282,7 @@ public Optional getTableHandle( // hudi table stays on the hive-handle path below (dormant, unchanged). Dormant overall until hms enters // SPI_READY_TYPES: today getTableHandle is never called for this connector. if (tableType == HiveTableType.ICEBERG) { - return siblingMetadata(session).getTableHandle(session, dbName, tableName); + return icebergSiblingMetadata(session).getTableHandle(session, dbName, tableName); } // Fail-loud parity with legacy HMSExternalTable.supportedHiveTable(), which threw on a null or // unrecognized input format instead of silently degrading (the old detector returned UNKNOWN). A view @@ -293,7 +321,7 @@ public Optional getTableHandle( public ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getTableSchema(session, handle); + return siblingMetadata(session, handle).getTableSchema(session, handle); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; String dbName = hiveHandle.getDbName(); @@ -351,7 +379,7 @@ public Map getProperties() { public Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getColumnHandles(session, handle); + return siblingMetadata(session, handle).getColumnHandles(session, handle); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; HmsTableInfo tableInfo = hmsClient.getTable( @@ -544,7 +572,7 @@ private static String parseTrinoViewDefinition(String originalText) { public Optional getTableStatistics( ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getTableStatistics(session, handle); + return siblingMetadata(session, handle).getTableStatistics(session, handle); } Map params = ((HiveTableHandle) handle).getTableParameters(); if (params == null) { @@ -590,7 +618,7 @@ public Optional getTableStatistics( public Optional getColumnStatistics( ConnectorSession session, ConnectorTableHandle handle, String columnName) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getColumnStatistics(session, handle, columnName); + return siblingMetadata(session, handle).getColumnStatistics(session, handle, columnName); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; if (hiveHandle.getTableType() != HiveTableType.HIVE) { @@ -645,7 +673,7 @@ private static long parseLongOrDefault(String value, long defaultValue) { @Override public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).estimateDataSizeByListingFiles(session, handle); + return siblingMetadata(session, handle).estimateDataSizeByListingFiles(session, handle); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; if (hiveHandle.getTableType() != HiveTableType.HIVE) { @@ -796,7 +824,7 @@ public Optional> applyFilter( ConnectorFilterConstraint constraint) { if (!(handle instanceof HiveTableHandle)) { // Forward AND return the sibling's result UNMODIFIED (a rewrap would poison a downstream scan cast). - return siblingMetadata(session).applyFilter(session, handle, constraint); + return siblingMetadata(session, handle).applyFilter(session, handle, constraint); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); @@ -851,7 +879,7 @@ public Optional> applyFilter( @Override public List listPartitionNames(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).listPartitionNames(session, handle); + return siblingMetadata(session, handle).listPartitionNames(session, handle); } return collectPartitionNames((HiveTableHandle) handle); } @@ -874,7 +902,7 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH public List listPartitions(ConnectorSession session, ConnectorTableHandle handle, Optional filter) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).listPartitions(session, handle, filter); + return siblingMetadata(session, handle).listPartitions(session, handle, filter); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); @@ -944,7 +972,7 @@ public Optional beginQuerySnapshot(ConnectorSession sessi // Diverts in lockstep with getTableFreshness/getPartitionFreshnessMillis: the pin's // isLastModifiedFreshness flag (false for an iceberg snapshot-id pin) gates whether fe-core consults // freshness at all, so half-diverting the pin would corrupt MVCC. - return siblingMetadata(session).beginQuerySnapshot(session, handle); + return siblingMetadata(session, handle).beginQuerySnapshot(session, handle); } return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(-1L).lastModifiedFreshness(true).build()); } @@ -967,7 +995,7 @@ public Optional beginQuerySnapshot(ConnectorSession sessi public Optional getTableFreshness(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getTableFreshness(session, handle); + return siblingMetadata(session, handle).getTableFreshness(session, handle); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); @@ -1008,7 +1036,7 @@ public Optional getTableFreshness(ConnectorSession sess public OptionalLong getPartitionFreshnessMillis(ConnectorSession session, ConnectorTableHandle handle, String partitionName) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getPartitionFreshnessMillis(session, handle, partitionName); + return siblingMetadata(session, handle).getPartitionFreshnessMillis(session, handle, partitionName); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partitions = hmsClient.getPartitions(hiveHandle.getDbName(), @@ -1043,7 +1071,7 @@ private static long lastDdlMillis(Map parameters) { public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getTableSchema(session, handle, snapshot); + return siblingMetadata(session, handle).getTableSchema(session, handle, snapshot); } // Hive has no schema-at-snapshot; the SPI default ignores the snapshot and returns the latest schema. return getTableSchema(session, handle); @@ -1053,7 +1081,7 @@ public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTa public Optional getMvccPartitionView(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).getMvccPartitionView(session, handle); + return siblingMetadata(session, handle).getMvccPartitionView(session, handle); } // Hive has no range-aware partition view; fe-core builds it from listPartitions (SPI default empty). return Optional.empty(); @@ -1063,7 +1091,7 @@ public Optional getMvccPartitionView(ConnectorSessio public Optional resolveTimeTravel(ConnectorSession session, ConnectorTableHandle handle, ConnectorTimeTravelSpec spec) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).resolveTimeTravel(session, handle, spec); + return siblingMetadata(session, handle).resolveTimeTravel(session, handle, spec); } // Hive has no time travel (SPI default empty): an explicit spec on a hive table is unsupported upstream. return Optional.empty(); @@ -1073,7 +1101,7 @@ public Optional resolveTimeTravel(ConnectorSession sessio public ConnectorTableHandle applySnapshot(ConnectorSession session, ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).applySnapshot(session, handle, snapshot); + return siblingMetadata(session, handle).applySnapshot(session, handle, snapshot); } // Hive's empty pin carries no scan options; the SPI default returns the handle unchanged. return handle; @@ -1083,7 +1111,7 @@ public ConnectorTableHandle applySnapshot(ConnectorSession session, ConnectorTab public ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, ConnectorTableHandle handle, Set rawDataFilePaths) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).applyRewriteFileScope(session, handle, rawDataFilePaths); + return siblingMetadata(session, handle).applyRewriteFileScope(session, handle, rawDataFilePaths); } // Hive has no distributed rewrite scope; the SPI default returns the handle unchanged. return handle; @@ -1093,7 +1121,7 @@ public ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, Conn public ConnectorTableHandle applyTopnLazyMaterialization(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).applyTopnLazyMaterialization(session, handle); + return siblingMetadata(session, handle).applyTopnLazyMaterialization(session, handle); } // Hive scan metadata already spans all columns; the SPI default returns the handle unchanged. return handle; @@ -1102,7 +1130,7 @@ public ConnectorTableHandle applyTopnLazyMaterialization(ConnectorSession sessio @Override public List listSupportedSysTables(ConnectorSession session, ConnectorTableHandle baseTableHandle) { if (!(baseTableHandle instanceof HiveTableHandle)) { - return siblingMetadata(session).listSupportedSysTables(session, baseTableHandle); + return siblingMetadata(session, baseTableHandle).listSupportedSysTables(session, baseTableHandle); } // Hive exposes no system tables (SPI default empty). return Collections.emptyList(); @@ -1113,7 +1141,7 @@ public Optional getSysTableHandle(ConnectorSession session ConnectorTableHandle baseTableHandle, String sysName) { if (!(baseTableHandle instanceof HiveTableHandle)) { // Return the sibling's sys-table handle UNMODIFIED (a rewrap would poison a downstream scan cast). - return siblingMetadata(session).getSysTableHandle(session, baseTableHandle, sysName); + return siblingMetadata(session, baseTableHandle).getSysTableHandle(session, baseTableHandle, sysName); } // Hive exposes no system tables (SPI default empty). return Optional.empty(); @@ -1306,7 +1334,7 @@ public void createTable(ConnectorSession session, ConnectorCreateTableRequest re @Override public void dropTable(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).dropTable(session, handle); + siblingMetadata(session, handle).dropTable(session, handle); return; } HiveTableHandle hiveHandle = (HiveTableHandle) handle; @@ -1330,7 +1358,7 @@ public void dropTable(ConnectorSession session, ConnectorTableHandle handle) { @Override public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, List partitions) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).truncateTable(session, handle, partitions); + siblingMetadata(session, handle).truncateTable(session, handle, partitions); return; } HiveTableHandle hiveHandle = (HiveTableHandle) handle; @@ -1353,7 +1381,7 @@ public void truncateTable(ConnectorSession session, ConnectorTableHandle handle, @Override public void renameTable(ConnectorSession session, ConnectorTableHandle handle, String newName) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).renameTable(session, handle, newName); + siblingMetadata(session, handle).renameTable(session, handle, newName); return; } // hive does not support ALTER TABLE RENAME (legacy HMSCachedClient has no rename). @@ -1364,7 +1392,7 @@ public void renameTable(ConnectorSession session, ConnectorTableHandle handle, S public void addColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column, ConnectorColumnPosition position) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).addColumn(session, handle, column, position); + siblingMetadata(session, handle).addColumn(session, handle, column, position); return; } throw new DorisConnectorException("ADD COLUMN not supported"); @@ -1373,7 +1401,7 @@ public void addColumn(ConnectorSession session, ConnectorTableHandle handle, Con @Override public void addColumns(ConnectorSession session, ConnectorTableHandle handle, List columns) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).addColumns(session, handle, columns); + siblingMetadata(session, handle).addColumns(session, handle, columns); return; } throw new DorisConnectorException("ADD COLUMNS not supported"); @@ -1382,7 +1410,7 @@ public void addColumns(ConnectorSession session, ConnectorTableHandle handle, Li @Override public void dropColumn(ConnectorSession session, ConnectorTableHandle handle, String columnName) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).dropColumn(session, handle, columnName); + siblingMetadata(session, handle).dropColumn(session, handle, columnName); return; } throw new DorisConnectorException("DROP COLUMN not supported"); @@ -1392,7 +1420,7 @@ public void dropColumn(ConnectorSession session, ConnectorTableHandle handle, St public void renameColumn(ConnectorSession session, ConnectorTableHandle handle, String oldName, String newName) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).renameColumn(session, handle, oldName, newName); + siblingMetadata(session, handle).renameColumn(session, handle, oldName, newName); return; } throw new DorisConnectorException("RENAME COLUMN not supported"); @@ -1402,7 +1430,7 @@ public void renameColumn(ConnectorSession session, ConnectorTableHandle handle, public void modifyColumn(ConnectorSession session, ConnectorTableHandle handle, ConnectorColumn column, ConnectorColumnPosition position) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).modifyColumn(session, handle, column, position); + siblingMetadata(session, handle).modifyColumn(session, handle, column, position); return; } throw new DorisConnectorException("MODIFY COLUMN not supported"); @@ -1411,7 +1439,7 @@ public void modifyColumn(ConnectorSession session, ConnectorTableHandle handle, @Override public void reorderColumns(ConnectorSession session, ConnectorTableHandle handle, List newOrder) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).reorderColumns(session, handle, newOrder); + siblingMetadata(session, handle).reorderColumns(session, handle, newOrder); return; } throw new DorisConnectorException("REORDER COLUMNS not supported"); @@ -1420,7 +1448,7 @@ public void reorderColumns(ConnectorSession session, ConnectorTableHandle handle @Override public void createOrReplaceBranch(ConnectorSession session, ConnectorTableHandle handle, BranchChange branch) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).createOrReplaceBranch(session, handle, branch); + siblingMetadata(session, handle).createOrReplaceBranch(session, handle, branch); return; } throw new DorisConnectorException("CREATE/REPLACE BRANCH not supported"); @@ -1429,7 +1457,7 @@ public void createOrReplaceBranch(ConnectorSession session, ConnectorTableHandle @Override public void createOrReplaceTag(ConnectorSession session, ConnectorTableHandle handle, TagChange tag) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).createOrReplaceTag(session, handle, tag); + siblingMetadata(session, handle).createOrReplaceTag(session, handle, tag); return; } throw new DorisConnectorException("CREATE/REPLACE TAG not supported"); @@ -1438,7 +1466,7 @@ public void createOrReplaceTag(ConnectorSession session, ConnectorTableHandle ha @Override public void dropBranch(ConnectorSession session, ConnectorTableHandle handle, DropRefChange branch) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).dropBranch(session, handle, branch); + siblingMetadata(session, handle).dropBranch(session, handle, branch); return; } throw new DorisConnectorException("DROP BRANCH not supported"); @@ -1447,7 +1475,7 @@ public void dropBranch(ConnectorSession session, ConnectorTableHandle handle, Dr @Override public void dropTag(ConnectorSession session, ConnectorTableHandle handle, DropRefChange tag) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).dropTag(session, handle, tag); + siblingMetadata(session, handle).dropTag(session, handle, tag); return; } throw new DorisConnectorException("DROP TAG not supported"); @@ -1457,7 +1485,7 @@ public void dropTag(ConnectorSession session, ConnectorTableHandle handle, DropR public void addPartitionField(ConnectorSession session, ConnectorTableHandle handle, PartitionFieldChange change) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).addPartitionField(session, handle, change); + siblingMetadata(session, handle).addPartitionField(session, handle, change); return; } throw new DorisConnectorException("ADD PARTITION FIELD not supported"); @@ -1467,7 +1495,7 @@ public void addPartitionField(ConnectorSession session, ConnectorTableHandle han public void dropPartitionField(ConnectorSession session, ConnectorTableHandle handle, PartitionFieldChange change) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).dropPartitionField(session, handle, change); + siblingMetadata(session, handle).dropPartitionField(session, handle, change); return; } throw new DorisConnectorException("DROP PARTITION FIELD not supported"); @@ -1477,7 +1505,7 @@ public void dropPartitionField(ConnectorSession session, ConnectorTableHandle ha public void replacePartitionField(ConnectorSession session, ConnectorTableHandle handle, PartitionFieldChange change) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).replacePartitionField(session, handle, change); + siblingMetadata(session, handle).replacePartitionField(session, handle, change); return; } throw new DorisConnectorException("REPLACE PARTITION FIELD not supported"); @@ -1493,7 +1521,7 @@ public void replacePartitionField(ConnectorSession session, ConnectorTableHandle @Override public void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHandle handle, WriteOperation op) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).validateRowLevelDmlMode(session, handle, op); + siblingMetadata(session, handle).validateRowLevelDmlMode(session, handle, op); return; } // hive: no per-table row-level DML mode constraint (SPI default no-op). @@ -1503,7 +1531,8 @@ public void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHand public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTableHandle handle, List staticPartitionColumnNames) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).validateStaticPartitionColumns(session, handle, staticPartitionColumnNames); + siblingMetadata(session, handle) + .validateStaticPartitionColumns(session, handle, staticPartitionColumnNames); return; } // hive: no static-partition constraint (SPI default no-op). @@ -1523,7 +1552,7 @@ public void validateStaticPartitionColumns(ConnectorSession session, ConnectorTa public void validateWritePartitionNames(ConnectorSession session, ConnectorTableHandle handle, List partitionNames) { if (!(handle instanceof HiveTableHandle)) { - siblingMetadata(session).validateWritePartitionNames(session, handle, partitionNames); + siblingMetadata(session, handle).validateWritePartitionNames(session, handle, partitionNames); return; } if (partitionNames != null && !partitionNames.isEmpty()) { @@ -1557,7 +1586,7 @@ public ConnectorTransaction beginTransaction(ConnectorSession session) { @Override public ConnectorTransaction beginTransaction(ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session).beginTransaction(session, handle); + return siblingMetadata(session, handle).beginTransaction(session, handle); } return beginTransaction(session); } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index fdd07b6e9aeac9..ab99c1fe178ffb 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -56,6 +56,7 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.function.Supplier; /** * Pins the HMS-cutover §4.4 S3 gateway metadata delegation: {@link HiveConnectorMetadata}'s per-handle methods @@ -77,10 +78,28 @@ private static final class ForeignHandle implements ConnectorTableHandle { private final RecordingSiblingMetadata siblingMetadata = new RecordingSiblingMetadata(); private final RecordingSiblingConnector siblingConnector = new RecordingSiblingConnector(siblingMetadata); - /** Metadata wired with a working sibling (hmsClient is null: the hive path is never exercised here). */ + /** + * The by-TYPE force-build supplier constructor arg. This suite exercises only per-handle (by-handle) sites — + * which must ALL route via the peek resolver — and never calls getTableHandle (the only by-type site), so the + * supplier must never be invoked here. It fails loud if it is, so a per-handle site that regressed from + * {@code siblingMetadata(session, handle)} (peek resolver) to {@code icebergSiblingMetadata(session)} (by-type + * force-build supplier) blows up instead of silently returning the same sibling. + */ + private static final Supplier SUPPLIER_MUST_NOT_BE_USED = () -> { + throw new AssertionError( + "a per-handle site must route via the peek resolver, not the by-type force-build supplier"); + }; + + /** + * Metadata wired so every foreign-handle per-handle site MUST route via the by-handle peek resolver (which + * returns the recording sibling), while the by-type force-build supplier is a fail-loud stub (see + * {@link #SUPPLIER_MUST_NOT_BE_USED}). hmsClient is null: the hive path is never exercised here. This suite + * pins that the per-handle sites FORWARD the whole surface; the 3-way ownsHandle dispatch that PICKS the owner + * is pinned by {@code HiveConnectorThreeWayRoutingTest}. + */ private HiveConnectorMetadata withSibling() { return new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), - () -> siblingConnector); + SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector); } private HiveTableHandle hiveHandle() { @@ -288,7 +307,7 @@ public void beginTransactionForHiveHandleOpensHiveTxnAndNeverConsultsSibling() { // sibling. The selection must be symmetric — hive and iceberg write plans downcast to different types. ConnectorTransaction hiveTxn = new NoOpConnectorTransaction(70099L, "HIVE"); HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), - () -> siblingConnector) { + SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector) { @Override public ConnectorTransaction beginTransaction(ConnectorSession session) { return hiveTxn; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java index 8112a72fc443ba..2872782d8f1d2c 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java @@ -129,7 +129,7 @@ public void hudiTableStaysOnHiveHandlePathNotDiverted() { public void missingTableReturnsEmptyWithoutConsultingSibling() { HiveConnectorMetadata md = new HiveConnectorMetadata( new FakeHmsClient(icebergTable(), false), Collections.emptyMap(), new FakeConnectorContext(), - () -> siblingConnector); + () -> siblingConnector, handle -> siblingConnector); Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), "a non-existent table short-circuits to empty before any format detection or divert"); @@ -150,8 +150,10 @@ public void icebergTableFailsLoudWhenNoSiblingConfigured() { // ===== helpers ===== private HiveConnectorMetadata withSibling(HmsTableInfo tableInfo) { + // getTableHandle diverts iceberg BY TYPE (icebergSiblingSupplier); the by-handle owner resolver is unused + // in this suite, so it just returns the recording sibling. return new HiveConnectorMetadata(new FakeHmsClient(tableInfo, true), Collections.emptyMap(), - new FakeConnectorContext(), () -> siblingConnector); + new FakeConnectorContext(), () -> siblingConnector, handle -> siblingConnector); } private static HmsTableInfo hiveTable(String inputFormat) { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java index 05ce5863fddeb7..5fe0f8922677a6 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorProcedureOpsDivertTest.java @@ -63,16 +63,20 @@ public void foreignHandleDelegatesToSiblingProcedureOpsUnmodified() { RecordingSibling sibling = new RecordingSibling(); RecordingSiblingContext context = new RecordingSiblingContext(sibling); HiveConnector connector = new HiveConnector(props(), context); + // Pre-build the owning sibling, mirroring production: getTableHandle builds the sibling before it produces + // a foreign handle, so the per-handle router only ever PEEKS an already-built one. + connector.getOrCreateIcebergSibling(); ForeignHandle foreign = new ForeignHandle(); ConnectorProcedureOps ops = connector.getProcedureOps(foreign); Assertions.assertSame(sibling.ops, ops, - "a foreign handle must return the iceberg sibling's OWN procedure ops"); + "a foreign handle must return the OWNING sibling's OWN procedure ops"); Assertions.assertSame(foreign, sibling.lastHandle, "the foreign handle must reach the sibling's per-handle selector UNMODIFIED (a rewrap would " + "poison the downstream iceberg cast)"); - Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once and consulted"); + Assertions.assertEquals(1, context.buildCount, + "the router PEEKS the already-built sibling (built once, here explicitly) — it never rebuilds"); } @Test @@ -106,15 +110,15 @@ public void hudiStampedHiveHandleHasNoProcedures() { } @Test - public void foreignHandleFailsLoudWhenIcebergPluginAbsent() { - // The seam returns a null sibling when the iceberg plugin is absent; selecting procedure ops for a - // foreign handle must fail loud (naming the catalog), not NPE. + public void foreignHandleFailsLoudWhenNoBuiltSiblingOwnsIt() { + // No sibling is ever built here, so the 3-way peek router finds no owner and must fail loud (naming the + // catalog), not NPE — the procedure-side stand-in for an orphan handle reaching a per-handle seam. RecordingSiblingContext context = new RecordingSiblingContext(null); HiveConnector connector = new HiveConnector(props(), context); DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, () -> connector.getProcedureOps(new ForeignHandle()), - "a foreign handle with no iceberg plugin must fail loud"); + "a foreign handle no built sibling owns must fail loud"); Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); } @@ -150,6 +154,12 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { return null; } + // Owns the test's ForeignHandle — the 3-way gateway router asks each built sibling this before routing. + @Override + public boolean ownsHandle(ConnectorTableHandle handle) { + return handle instanceof ForeignHandle; + } + @Override public ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { lastHandle = handle; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java index caaa5e51d9fbe5..e00179a9666989 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorScanProviderDivertTest.java @@ -83,16 +83,20 @@ public void foreignHandleDelegatesToSiblingScanProviderUnmodified() { RecordingSibling sibling = new RecordingSibling(); RecordingSiblingContext context = new RecordingSiblingContext(sibling); HiveConnector connector = new HiveConnector(props(), context); + // Pre-build the owning sibling, mirroring production: getTableHandle's divert force-builds the sibling + // before it can produce a foreign handle, so the per-handle router only ever PEEKS an already-built one. + connector.getOrCreateIcebergSibling(); ForeignHandle foreign = new ForeignHandle(); ConnectorScanPlanProvider provider = connector.getScanPlanProvider(foreign); Assertions.assertSame(sibling.provider, provider, - "a foreign handle must return the iceberg sibling's OWN scan provider"); + "a foreign handle must return the OWNING sibling's OWN scan provider"); Assertions.assertSame(foreign, sibling.lastHandle, "the foreign handle must reach the sibling's per-handle selector UNMODIFIED (a rewrap would " + "poison the downstream iceberg cast)"); - Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once and consulted"); + Assertions.assertEquals(1, context.buildCount, + "the router PEEKS the already-built sibling (built once, here explicitly) — it never rebuilds"); } @Test @@ -126,15 +130,16 @@ public void hudiStampedHiveHandleStaysOnHiveProvider() { } @Test - public void foreignHandleFailsLoudWhenIcebergPluginAbsent() { - // The seam returns a null sibling when the iceberg plugin is absent; selecting a scan provider for a - // foreign handle must fail loud (naming the catalog), not NPE. + public void foreignHandleFailsLoudWhenNoBuiltSiblingOwnsIt() { + // No sibling is ever built here (nothing calls getOrCreate*/getTableHandle), so the 3-way peek router + // finds no owner and must fail loud (naming the catalog), not NPE. This stands in for an orphan handle — + // a foreign handle that reached a per-handle seam without its owning sibling built (a bug, not a route). RecordingSiblingContext context = new RecordingSiblingContext(null); HiveConnector connector = new HiveConnector(props(), context); DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, () -> connector.getScanPlanProvider(new ForeignHandle()), - "a foreign handle with no iceberg plugin must fail loud"); + "a foreign handle no built sibling owns must fail loud"); Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); } @@ -172,6 +177,13 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { return null; } + // Owns the test's ForeignHandle — the 3-way gateway router (HiveConnector.resolveSiblingOwner) asks each + // built sibling this before routing, since the foreign handle's concrete type is loader-invisible. + @Override + public boolean ownsHandle(ConnectorTableHandle handle) { + return handle instanceof ForeignHandle; + } + @Override public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { lastHandle = handle; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java new file mode 100644 index 00000000000000..f937d4e5f909f6 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java @@ -0,0 +1,284 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorWriteHandle; +import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; +import org.apache.doris.connector.api.procedure.ConnectorProcedureResult; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorPredicate; +import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanRange; +import org.apache.doris.connector.api.write.ConnectorSinkPlan; +import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Pins the hms-cutover THREE-WAY foreign-handle routing: with TWO embedded siblings (iceberg + hudi) under one + * gateway, the binary {@code else -> iceberg} discriminator no longer works — a hudi handle must not be + * wrong-routed to the iceberg sibling. {@link HiveConnector} routes every per-handle seam by asking each + * ALREADY-BUILT sibling {@link Connector#ownsHandle} (a hive handle stays hive; else the owning sibling by + * {@code ownsHandle}; else fail loud). The concrete foreign handle type is invisible across the plugin + * classloader split, so the gateway can never {@code instanceof} it itself. + * + *

    The same {@code HiveConnector.resolveSiblingOwner} brain backs both the connector-level + * {@code get*Provider(handle)} seams exercised here AND the ~34 per-handle guard-and-forward methods in + * {@link HiveConnectorMetadata} (which forward via the injected resolver — see + * {@link HiveConnectorMetadataSiblingDelegationTest}).

    + * + *

    WHY the PEEK design (Rule 9): routing consults only siblings that are already built (the owning + * sibling is always built first, by getTableHandle, before it can produce the handle). It never force-builds an + * unrelated plugin merely to classify a handle — so a catalog serving only hudi (no iceberg plugin installed) + * still routes its hudi handles, and the iceberg arm stays byte-behaviour-identical (an iceberg handle routes to + * iceberg without ever touching the hudi holder). Dormant until hms enters {@code SPI_READY_TYPES}.

    + */ +public class HiveConnectorThreeWayRoutingTest { + + private static final String METASTORE_URI = "thrift://host:9083"; + + /** Stand-in for the raw iceberg-loader handle the iceberg sibling produces (loader-invisible to the gateway). */ + private static final class IcebergLikeHandle implements ConnectorTableHandle { + } + + /** Stand-in for the raw hudi-loader handle the hudi sibling produces. */ + private static final class HudiLikeHandle implements ConnectorTableHandle { + } + + @Test + public void icebergHandleRoutesToIcebergSiblingAcrossAllSeams() { + RoutingSibling iceberg = new RoutingSibling("iceberg", IcebergLikeHandle.class); + RoutingSibling hudi = new RoutingSibling("hudi", HudiLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(iceberg, hudi); + HiveConnector connector = new HiveConnector(props(), ctx); + connector.getOrCreateIcebergSibling(); + connector.getOrCreateHudiSibling(); + IcebergLikeHandle handle = new IcebergLikeHandle(); + + Assertions.assertSame(iceberg.scanProvider, connector.getScanPlanProvider(handle), + "an iceberg handle must route to the iceberg sibling's scan provider"); + Assertions.assertSame(iceberg.writeProvider, connector.getWritePlanProvider(handle), + "an iceberg handle must route to the iceberg sibling's write provider"); + Assertions.assertSame(iceberg.procedureOps, connector.getProcedureOps(handle), + "an iceberg handle must route to the iceberg sibling's procedure ops"); + Assertions.assertNull(hudi.lastScanHandle, "the hudi sibling must never be consulted for an iceberg handle"); + } + + @Test + public void hudiHandleRoutesToHudiSiblingAcrossAllSeams() { + RoutingSibling iceberg = new RoutingSibling("iceberg", IcebergLikeHandle.class); + RoutingSibling hudi = new RoutingSibling("hudi", HudiLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(iceberg, hudi); + HiveConnector connector = new HiveConnector(props(), ctx); + connector.getOrCreateIcebergSibling(); + connector.getOrCreateHudiSibling(); + HudiLikeHandle handle = new HudiLikeHandle(); + + Assertions.assertSame(hudi.scanProvider, connector.getScanPlanProvider(handle), + "a hudi handle must route to the hudi sibling's scan provider, NOT the iceberg sibling's"); + Assertions.assertSame(hudi.writeProvider, connector.getWritePlanProvider(handle), + "a hudi handle must route to the hudi sibling's write provider"); + Assertions.assertSame(hudi.procedureOps, connector.getProcedureOps(handle), + "a hudi handle must route to the hudi sibling's procedure ops"); + Assertions.assertNull(iceberg.lastScanHandle, "the iceberg sibling must never be consulted for a hudi handle"); + } + + @Test + public void hiveHandleStaysHiveAndConsultsNeitherSibling() { + RoutingSibling iceberg = new RoutingSibling("iceberg", IcebergLikeHandle.class); + RoutingSibling hudi = new RoutingSibling("hudi", HudiLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(iceberg, hudi); + HiveConnector connector = new HiveConnector(props(), ctx); + connector.getOrCreateIcebergSibling(); + connector.getOrCreateHudiSibling(); + HiveTableHandle hive = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE).build(); + + // getProcedureOps needs no HmsClient: a hive handle inherits the connector-level null (plain-hive has none). + Assertions.assertNull(connector.getProcedureOps(hive), "a hive handle has no procedures (connector-level null)"); + Assertions.assertNull(iceberg.lastProcedureHandle, "a hive handle must not consult the iceberg sibling"); + Assertions.assertNull(hudi.lastProcedureHandle, "a hive handle must not consult the hudi sibling"); + } + + @Test + public void unknownForeignHandleFailsLoud() { + RoutingSibling iceberg = new RoutingSibling("iceberg", IcebergLikeHandle.class); + RoutingSibling hudi = new RoutingSibling("hudi", HudiLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(iceberg, hudi); + HiveConnector connector = new HiveConnector(props(), ctx); + connector.getOrCreateIcebergSibling(); + connector.getOrCreateHudiSibling(); + + // A foreign handle owned by NEITHER built sibling is an orphan (a bug, not a route) — fail loud, do not + // silently hand it to an arbitrary sibling. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> connector.getScanPlanProvider(new ConnectorTableHandle() { + }), "a foreign handle no sibling owns must fail loud"); + Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); + } + + @Test + public void hudiHandleRoutesWithoutBuildingIcebergSibling() { + // The PEEK no-regression guarantee: a catalog serving only hudi tables (no iceberg plugin installed) must + // route its hudi handles WITHOUT the router force-building an iceberg sibling. Iceberg is absent (null); + // only the hudi sibling is built. + RoutingSibling hudi = new RoutingSibling("hudi", HudiLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(null, hudi); + HiveConnector connector = new HiveConnector(props(), ctx); + connector.getOrCreateHudiSibling(); + + Assertions.assertSame(hudi.scanProvider, connector.getScanPlanProvider(new HudiLikeHandle()), + "a hudi handle must route to the hudi sibling even with no iceberg plugin present"); + Assertions.assertEquals(0, ctx.icebergBuilds, + "routing a hudi handle must NOT build (or require) the iceberg sibling"); + } + + @Test + public void icebergHandleRoutesWithoutBuildingHudiSibling() { + // Symmetric guarantee — the iceberg arm stays byte-behaviour-identical after adding the hudi holder: an + // iceberg handle routes to iceberg without the router touching the hudi holder (no hudi plugin required). + RoutingSibling iceberg = new RoutingSibling("iceberg", IcebergLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(iceberg, null); + HiveConnector connector = new HiveConnector(props(), ctx); + connector.getOrCreateIcebergSibling(); + + Assertions.assertSame(iceberg.scanProvider, connector.getScanPlanProvider(new IcebergLikeHandle()), + "an iceberg handle must route to the iceberg sibling with no hudi plugin present"); + Assertions.assertEquals(0, ctx.hudiBuilds, + "routing an iceberg handle must NOT build (or require) the hudi sibling"); + } + + private static Map props() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", METASTORE_URI); + return props; + } + + /** A context that builds a distinct sibling per type (iceberg / hudi), each possibly null (absent plugin). */ + private static final class TwoSiblingContext extends FakeConnectorContext { + private final Connector iceberg; + private final Connector hudi; + int icebergBuilds; + int hudiBuilds; + + TwoSiblingContext(Connector iceberg, Connector hudi) { + this.iceberg = iceberg; + this.hudi = hudi; + } + + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + if ("iceberg".equals(catalogType)) { + icebergBuilds++; + return iceberg; + } + if ("hudi".equals(catalogType)) { + hudiBuilds++; + return hudi; + } + return null; + } + } + + /** A sibling that owns exactly one handle type and returns identity-marked, call-recording providers. */ + private static final class RoutingSibling implements Connector { + private final Class ownedType; + final ConnectorScanPlanProvider scanProvider; + final ConnectorWritePlanProvider writeProvider = new MarkerWriteProvider(); + final ConnectorProcedureOps procedureOps = new MarkerProcedureOps(); + ConnectorTableHandle lastScanHandle; + ConnectorTableHandle lastProcedureHandle; + + RoutingSibling(String name, Class ownedType) { + this.ownedType = ownedType; + this.scanProvider = new MarkerScanProvider(); + } + + @Override + public boolean ownsHandle(ConnectorTableHandle handle) { + return ownedType.isInstance(handle); + } + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + + @Override + public ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + lastScanHandle = handle; + return scanProvider; + } + + @Override + public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { + return writeProvider; + } + + @Override + public ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { + lastProcedureHandle = handle; + return procedureOps; + } + + @Override + public void close() { + } + } + + private static final class MarkerScanProvider implements ConnectorScanPlanProvider { + @Override + public List planScan(ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter) { + return Collections.emptyList(); + } + } + + private static final class MarkerWriteProvider implements ConnectorWritePlanProvider { + @Override + public ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle) { + return null; + } + } + + private static final class MarkerProcedureOps implements ConnectorProcedureOps { + @Override + public List getSupportedProcedures() { + return Collections.emptyList(); + } + + @Override + public ConnectorProcedureResult execute(ConnectorSession session, ConnectorTableHandle table, + String procedureName, Map properties, ConnectorPredicate whereCondition, + List partitionNames) { + return null; + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java index de6745e23dd6bd..09f31cbb5a9dfc 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorWriteProviderDivertTest.java @@ -77,16 +77,20 @@ public void foreignHandleDelegatesToSiblingWriteProviderUnmodified() { RecordingSibling sibling = new RecordingSibling(); RecordingSiblingContext context = new RecordingSiblingContext(sibling); HiveConnector connector = new HiveConnector(props(), context); + // Pre-build the owning sibling, mirroring production: getTableHandle builds the sibling before it produces + // a foreign handle, so the per-handle router only ever PEEKS an already-built one. + connector.getOrCreateIcebergSibling(); ForeignHandle foreign = new ForeignHandle(); ConnectorWritePlanProvider provider = connector.getWritePlanProvider(foreign); Assertions.assertSame(sibling.provider, provider, - "a foreign handle must return the iceberg sibling's OWN write provider"); + "a foreign handle must return the OWNING sibling's OWN write provider"); Assertions.assertSame(foreign, sibling.lastHandle, "the foreign handle must reach the sibling's per-handle selector UNMODIFIED (a rewrap would " + "poison the downstream iceberg cast)"); - Assertions.assertEquals(1, context.buildCount, "the sibling must be built exactly once and consulted"); + Assertions.assertEquals(1, context.buildCount, + "the router PEEKS the already-built sibling (built once, here explicitly) — it never rebuilds"); } @Test @@ -120,15 +124,15 @@ public void hudiStampedHiveHandleStaysOnHiveProvider() { } @Test - public void foreignHandleFailsLoudWhenIcebergPluginAbsent() { - // The seam returns a null sibling when the iceberg plugin is absent; selecting a write provider for a - // foreign handle must fail loud (naming the catalog), not NPE. + public void foreignHandleFailsLoudWhenNoBuiltSiblingOwnsIt() { + // No sibling is ever built here, so the 3-way peek router finds no owner and must fail loud (naming the + // catalog), not NPE — the write-side stand-in for an orphan handle reaching a per-handle seam. RecordingSiblingContext context = new RecordingSiblingContext(null); HiveConnector connector = new HiveConnector(props(), context); DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, () -> connector.getWritePlanProvider(new ForeignHandle()), - "a foreign handle with no iceberg plugin must fail loud"); + "a foreign handle no built sibling owns must fail loud"); Assertions.assertTrue(ex.getMessage().contains("test_catalog"), ex.getMessage()); } @@ -164,6 +168,12 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { return null; } + // Owns the test's ForeignHandle — the 3-way gateway router asks each built sibling this before routing. + @Override + public boolean ownsHandle(ConnectorTableHandle handle) { + return handle instanceof ForeignHandle; + } + @Override public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { lastHandle = handle; diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index 242e9ae2651b2b..af0201981825af 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; @@ -86,6 +87,17 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { return new HudiConnectorMetadata(getOrCreateClient(), properties); } + /** + * True for a handle this connector produced (a {@link HudiTableHandle}). Tested against this connector's OWN + * in-loader type, so a heterogeneous hms gateway that embeds this connector as a sibling can route a foreign + * hudi handle here without casting it across the plugin classloader split. Returns false for any other + * connector's handle (e.g. an iceberg sibling's), so the gateway keeps looking. + */ + @Override + public boolean ownsHandle(ConnectorTableHandle handle) { + return handle instanceof HudiTableHandle; + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { return new HudiScanPlanProvider(properties, context); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java new file mode 100644 index 00000000000000..f86b5bc0b28c74 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java @@ -0,0 +1,62 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.ConnectorContext; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; + +/** + * Pins {@link HudiConnector#ownsHandle} for the hms 3-way sibling routing: a flipped hms gateway embeds this + * connector as a sibling and asks it "is this foreign handle yours?" to route a scan/metadata call, because the + * concrete handle type is invisible across the plugin classloader split. + * + *

    ownsHandle must be TRUE for this connector's own {@link HudiTableHandle} and FALSE for any other connector's + * handle (e.g. an iceberg sibling's), so the gateway routes correctly. Dormant until hms enters + * {@code SPI_READY_TYPES}: no production path asks this connector yet, so this is a Rule-9 routing guard. + */ +public class HudiConnectorOwnsHandleTest { + + private static HudiConnector connector() { + return new HudiConnector(Collections.emptyMap(), new ConnectorContext() { + @Override + public String getCatalogName() { + return "test_catalog"; + } + + @Override + public long getCatalogId() { + return 1L; + } + }); + } + + @Test + public void ownsHandleOnlyForHudiTableHandle() { + // MUTATION: returning true unconditionally -> the gateway sends iceberg handles here -> red. + HudiConnector connector = connector(); + Assertions.assertTrue(connector.ownsHandle(new HudiTableHandle("db", "t", "s3://b/t", "COPY_ON_WRITE")), + "a HudiTableHandle is owned by the hudi connector"); + Assertions.assertFalse(connector.ownsHandle(new ConnectorTableHandle() { + }), "a foreign (non-hudi) handle is NOT owned by the hudi connector"); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index ba112eb9fbf3c4..5f8c2157633ff8 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.ConnectorTestResult; import org.apache.doris.connector.api.ConnectorValidationContext; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; @@ -209,6 +210,17 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { newCatalogBackedOps(session), properties, context, latestSnapshotCache); } + /** + * True for a handle this connector produced (an {@link IcebergTableHandle}). Tested against this connector's + * OWN in-loader type, so a heterogeneous hms gateway that embeds this connector as a sibling can route a + * foreign iceberg handle here without casting it across the plugin classloader split. Returns false for any + * other connector's handle (e.g. a hudi sibling's), so the gateway keeps looking. + */ + @Override + public boolean ownsHandle(ConnectorTableHandle handle) { + return handle instanceof IcebergTableHandle; + } + /** * Eagerly validates connectivity during CREATE CATALOG (when {@code test_connection=true}). * Runs two probes so bad configurations fail fast instead of at first query: diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java index fe49aebd87a16a..d7f35004154309 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.ConnectorContractValidator; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.filesystem.properties.StorageProperties; @@ -114,6 +115,20 @@ public void declaresMvccSnapshotCapability() { Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT)); } + @Test + public void ownsHandleOnlyForIcebergTableHandle() { + // WHY (hms 3-way sibling routing): a flipped hms gateway embeds this connector as a sibling and asks it + // "is this foreign handle yours?" to route a scan/metadata call, because the concrete handle type is + // invisible across the plugin classloader split. ownsHandle must be TRUE for this connector's own + // IcebergTableHandle and FALSE for any other connector's handle (e.g. a hudi sibling's), so the gateway + // routes correctly. MUTATION: returning true unconditionally -> the gateway sends hudi handles here -> red. + IcebergConnector connector = new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()); + Assertions.assertTrue(connector.ownsHandle(new IcebergTableHandle("db", "t")), + "an IcebergTableHandle is owned by the iceberg connector"); + Assertions.assertFalse(connector.ownsHandle(new ConnectorTableHandle() { + }), "a foreign (non-iceberg) handle is NOT owned by the iceberg connector"); + } + @Test public void declaresMetadataPreloadCapability() { // WHY (F11): legacy IcebergExternalTable.supportsExternalMetadataPreload returns true so the planner From beea3874503eb3163219cb31f68d3493e0aae60f Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 00:14:19 +0800 Subject: [PATCH 106/334] [doc](catalog) HMS cutover hudi HD-B1 (three-way ownsHandle routing) DONE; next = Group C (no-regression) + Group D, then HD-B2 Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 28d39819e8e631..0d4a2d43f87f79 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,9 +5,9 @@ --- -# 🎯 当前状态（2026-07-08） +# 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**下一步 = HD-B1 三路 foreign-handle 路由（已签字 2026-07-08 = Option 1 `Connector.ownsHandle(handle)` 中立 SPI；改动已落 iceberg 活脊柱须保字节等价）→ Group C 无回退补齐（MVCC/listPartitions、时间旅行、增量读〔最高风险〕、schema evolution）+ Group D 写-拒安全网 → HD-B2 getTableHandle 分流点亮（最后一步）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。下一步 = Group C 无回退补齐（连接器侧，全部翻闸前）：MVCC/listPartitions、时间旅行、增量读〔最高风险〕、schema evolution + Group D 写-拒安全网 → HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -86,7 +86,8 @@ - **✅ HD-A3 DONE（`19f11063a79`）= 分区值解析保真 FIX**：`parsePartitionValues` 逐字端口 legacy `HudiPartitionUtils.parsePartitionValues`（positional 映射 + 单列整路回退 + `>1` 列 fail-loud + 内联 `unescapePathName`＝Hive FileUtils 字节复刻、无 hive-common 依赖），修非-hive-style 分区（`2024/01`）静默读 NULL 分区列 + 未 unescape 的真 bug；**范围仅值解析（unpruned 最基本读）**；pruned-path HMS-hive-style-vs-Hudi-positional 源一致性（须 `useHiveSyncPartition`）归 **HD-C1 partition-listing 步**（亦翻闸前、无翻闸回退，已记账非静默跳）。8 测。 - **✅ HD-A4 DONE（`7bc90edf4a3` + 复审修 `ce700db35e0`）= force_jni_scanner + COW/MOR 检测硬化**：读 `force_jni_scanner` session 标志（镜像 paimon），planScan `useNativeCowPath = isCow && !forceJni`（force-JNI 把 COW 也推 JNI 路＝legacy `canUseNativeReader`）+ collectMorSplits 无-log native 降级加 `!forceJni` 守卫；**isCow 改由权威 `metaClient.getTableType()` 定**（非 substring 检测，UNKNOWN 不再静默误选路）。**5 维对抗复审揪出真缺陷（本 session 修）**：`populateRangeParams` 未显式 `setFormatType`、只靠单一 node 默认 → ①MOR 无-log native slice 当 `FORMAT_JNI` 空 THudiFileDesc 发 BE（blocker）②COW+force_jni 当 native（major，架空开关）③COW-ORC 当 parquet——已**镜像 `PaimonScanRange` 逐 range 显式设格式**修好（旧测用 `fileFormat("jni")` 假形状掩盖＝Rule-9 masking，已补 planner 真形状 native-parquet/orc 测）。6+4 测。 - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 - - **下一步 = HD-B1（见文首「下一步」）**——三路 foreign-handle 路由，已签字 Option 1 `Connector.ownsHandle` 中立 SPI；改动已落 iceberg 活脊柱（约 30 处 `!(handle instanceof HiveTableHandle)` 转发点 + 3 个连接器级 `get*Provider(handle)`）须保 iceberg 臂字节等价；fe-core 永不调用 ownsHandle。然后 Group C（无回退补齐）+ Group D（写-拒）+ HD-B2（分流点亮，最后一步）。 + - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 + - **下一步 = Group C 无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C1** MVCC/listPartitions/freshness（分区 hudi 表 MVCC 视图非空、批模式 split 节流防规划期 OOM；`beginQuerySnapshot` 钉最新 instant 为 snapshot-id-kind、**不设 lastModifiedFreshness**）、**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From edafbcae6427fe0b1fe548653366bcffc335ad60 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 00:59:21 +0800 Subject: [PATCH 107/334] [doc](catalog) HMS cutover hudi: MVCC/listPartitions/freshness step (Group C step 1) authoritative design + HANDOFF Code-grounded design (recon wf_1a09236d-ee0, 5 readers + synthesis, HEAD-verified). Follow the paimon template (override only beginQuerySnapshot + listPartitions/Names/Values; NOT getMvccPartitionView/getTableFreshness/getPartitionFreshnessMillis = dead code under lastModifiedFreshness=false). Pin the latest completed instant as snapshotId; per-partition lastModifiedMillis = the instant; render hive-style partition names (checkState landmine); partition-source consistency (prune-to-zero) + auth/TCCL wiring flagged HIGH. One pending user decision: hudi MTMV freshness scope (legacy is a 0-const stub; recommend real instant freshness). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- ...i-mvcc-partition-step-design-2026-07-09.md | 139 ++++++++++++++++++ 2 files changed, 140 insertions(+), 1 deletion(-) create mode 100644 plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 0d4a2d43f87f79..5000545ec49481 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -87,7 +87,7 @@ - **✅ HD-A4 DONE（`7bc90edf4a3` + 复审修 `ce700db35e0`）= force_jni_scanner + COW/MOR 检测硬化**：读 `force_jni_scanner` session 标志（镜像 paimon），planScan `useNativeCowPath = isCow && !forceJni`（force-JNI 把 COW 也推 JNI 路＝legacy `canUseNativeReader`）+ collectMorSplits 无-log native 降级加 `!forceJni` 守卫；**isCow 改由权威 `metaClient.getTableType()` 定**（非 substring 检测，UNKNOWN 不再静默误选路）。**5 维对抗复审揪出真缺陷（本 session 修）**：`populateRangeParams` 未显式 `setFormatType`、只靠单一 node 默认 → ①MOR 无-log native slice 当 `FORMAT_JNI` 空 THudiFileDesc 发 BE（blocker）②COW+force_jni 当 native（major，架空开关）③COW-ORC 当 parquet——已**镜像 `PaimonScanRange` 逐 range 显式设格式**修好（旧测用 `fileFormat("jni")` 假形状掩盖＝Rule-9 masking，已补 planner 真形状 native-parquet/orc 测）。6+4 测。 - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - - **下一步 = Group C 无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C1** MVCC/listPartitions/freshness（分区 hudi 表 MVCC 视图非空、批模式 split 节流防规划期 OOM；`beginQuerySnapshot` 钉最新 instant 为 snapshot-id-kind、**不设 lastModifiedFreshness**）、**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **下一步 = Group C 无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C1（分区/MVCC/新鲜度）已出权威设计文档 `hudi-mvcc-partition-step-design-2026-07-09.md`（recon `wf_1a09236d-ee0`：5 读者+synthesis，全 HEAD 核对；起步必读）+ 一个待签字决策**——**hudi MTMV 新鲜度**：旧 `HudiDlaTable` 是 `MTMVTimestampSnapshot(0L)` 恒常量桩（真逻辑被注释、从不感知源变化），**推荐实现真 instant 新鲜度=paimon 模型（行为提升非回退，天然消除 -1 过刷新地雷）**。**关键校正（务必先读设计文档）**：①跟 **paimon 模型**——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不 override** `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下是死代码，task brief 那三条是错缝）；②`beginQuerySnapshot` 钉最新 completed instant 为 **snapshotId**（`Long.parseLong(requestedTime())`，空 timeline→`0L` 非 -1），**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`、分区走 `MTMVTimestampSnapshot(instant)`（**非** snapshot-id 分支）；③**partition name 须 hive-style `col=val/...` 渲染**否则通用侧 `HiveUtil.toPartitionValues` 的 `checkState(size==types)` 抛→skip→静默退化 UNPARTITIONED（R1 HIGH）；④**R2 HIGH 分区源一致性**——listPartitions 源须与扫描侧分区源字节一致（useHiveSyncPartition-aware），否则 FE prune-to-zero 短路致 partition-predicate 查询丢行，须 e2e 验（可能须 `ignorePartitionPruneShortCircuit=true`）；⑤**R4 HIGH 须注入 auth+TCCL wrapper 进 `HudiConnectorMetadata`**（现 `getMetadata` 不传 context，新方法建 `HoodieTableMetaClient` 须 plugin UGI doAs + TCCL pin，memory `catalog-spi-plugin-tccl-classloader-gotcha`）；每分区 `lastModifiedMillis`=instant（-1 过刷新/0 旧桩皆错）。**（其余 Group C 步）**、**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md b/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md new file mode 100644 index 00000000000000..8a412275447900 --- /dev/null +++ b/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md @@ -0,0 +1,139 @@ +# Hudi MVCC / listPartitions / freshness step — implementation design (no-regression Group C, step 1) + +Authoritative, code-grounded design. HEAD = `catalog-spi-11-hive`. Recon `wf_1a09236d-ee0` (5 readers + synthesis, +all HEAD-verified; 2 readers hit the structured-output cap and the synthesis re-read those files directly). All the +load-bearing facts below were hand-verified against HEAD after the recon. + +**Goal**: close the no-regression gap so a PARTITIONED hudi-on-HMS table, served post-flip through the GENERIC +`PluginDrivenScanNode` path (like paimon/iceberg), has a correct partition + MVCC-snapshot surface (partition +pruning / `selectedPartitionNum` / SHOW PARTITIONS / TVFs / MTMV freshness). **Zero fe-core changes** — every change +is a connector override consuming the existing SPI (IRON rule respected). + +## The reconciled model: follow the PAIMON template, NOT the hive plumbing + +Paimon (`PaimonConnectorMetadata`) overrides ONLY `beginQuerySnapshot` (:432) + `listPartitions` (:954) + +`listPartitionNames` (:938) + `listPartitionValues` (:960). It does **not** override `getMvccPartitionView`, +`getTableFreshness`, or `getPartitionFreshnessMillis`. Hudi is a snapshot-id connector like paimon, so it mirrors +exactly that surface. `HudiConnectorMetadata` currently overrides **none** of these seven. + +**Correction to the earlier task brief** (verified in `PluginDrivenMvccExternalTable`): with +`lastModifiedFreshness == false`, fe-core NEVER calls `getTableFreshness` (gated at :627) nor +`getPartitionFreshnessMillis` (gated at :591). Overriding them would be DEAD CODE. The task's +"getTableFreshness/getPartitionFreshnessMillis → last-commit instant" bullet targets the wrong seam: the freshness +signal is delivered through `beginQuerySnapshot`'s snapshotId (table) + per-partition `lastModifiedMillis` +(partition). Also, "getPartitionSnapshot takes the snapshot-id branch" is FALSE: on the LIST path +(`getMvccPartitionView` empty) `getPartitionSnapshot` reaches the pin-TIMESTAMP `MTMVTimestampSnapshot(value)` +branch (:607), identical to paimon; the `MTMVSnapshotIdSnapshot` branch (:589) requires a RANGE +`getMvccPartitionView` (iceberg only) — hudi must NOT provide one. + +## Per-method spec (HudiConnectorMetadata) + +1. **`beginQuerySnapshot(session, handle)` — NEW.** Return + `Optional.of(ConnectorMvccSnapshot.builder().snapshotId(instant).build())`, where `instant = + Long.parseLong(timeline.filterCompletedInstants().lastInstant().get().requestedTime())` (the latest COMPLETED + instant; port of `HudiUtils.getLastTimeStamp`, fe-core `HudiUtils.java:271-284`; same timeline the scan already + uses at `HudiScanPlanProvider.java:131-139`). Empty timeline → pin `0L` (legacy parity; `0L >= 0` survives the + `getNewestUpdateVersionOrTime` `v>=0` filter at `PluginDrivenMvccExternalTable.java:714`). **MUST NOT set + `lastModifiedFreshness(true)`** (default false — the one bit separating hudi from the hive precedent). Do NOT set + schemaId (time-travel is a later step). → `getTableSnapshot` returns `MTMVSnapshotIdSnapshot(instant)` (:634). +2. **`getMvccPartitionView` — DO NOT OVERRIDE.** Inherit the SPI default `Optional.empty()`. Keeps the LIST path + (paimon parity). The task's "→ empty" is the inherited default; an explicit override is redundant. +3. **`listPartitions(session, handle, filter)` — NEW.** One `ConnectorPartitionInfo` per partition via a shared + private `collectPartitions(handle)`; `filter` ignored (paimon parity). Unpartitioned → `emptyList()`. + Partition-name SOURCE = port of `HudiExternalMetaCache.loadPartitionNames` (fe-core :195-217), + `useHiveSyncPartition`-aware: + - `true` → `hmsClient.listPartitionNames(db, table, -1)` (already wired at `HudiConnectorMetadata.java:176`) then + unescape each. + - `false` → `HoodieTableMetadata.getAllPartitionPaths` (port of `HudiPartitionUtils.getAllPartitionNames`, + fe-core :42-50; logic already inline in `HudiScanPlanProvider.resolvePartitions:344-352`). + - Defer the `getPartitionNamesBeforeOrEquals` (non-latest time-travel) branch — matches deferred time-travel scope. + `useHiveSyncPartition = Boolean.parseBoolean(properties.getOrDefault("use_hive_sync_partition","false"))` (port of + `HMSExternalTable.useHiveSyncPartition`; key `USE_HIVE_SYNC_PARTITION`). For each raw path: parse values with the + ALREADY-ported `HudiScanPlanProvider.parsePartitionValues(rawPath, partKeyNames)` (HD-A3) and **render a + hive-style name** (see the 7-arg fields below). 7-arg `ConnectorPartitionInfo`. +4. **`listPartitionNames(session, handle)` — NEW.** `collectPartitions` → `getPartitionName()` (paimon parity). +5. **`listPartitionValues(session, handle, cols)` — NEW (recommended, TVF parity).** `collectPartitions` → project + `getPartitionValues()` into `cols` order (paimon parity). +6. **`getTableFreshness` / `getPartitionFreshnessMillis` — DO NOT OVERRIDE** (dead code under flag=false, see above). + +### Per-partition `ConnectorPartitionInfo` (7-arg ctor) +- **partitionName** = HIVE-STYLE `col0=val0/col1=val1/...` in partition-key order (values from `parsePartitionValues`). + **MANDATORY**: the generic consumer rebuilds the item by RE-PARSING the name via `HiveUtil.toPartitionValues` under + `Preconditions.checkState(values.size()==types.size())` (`PluginDrivenMvccExternalTable.java:292-297`). A raw hudi + path (`"2024/01"`, or single-col `"2024"` with no `=`) → wrong count → checkState throws → caught+skipped + (:277-280) → item map short → `isPartitionInvalid()` → silent UNPARTITIONED degrade. So the connector MUST render + hive-style. **Arity precondition**: `partKeyNames.size()` must equal the fe-core partition-column count. +- **partitionValues** = raw parsed `Map` (LinkedHashMap, key order). Backs `listPartitionValues` TVF. +- **properties** = `emptyMap()`. +- **rowCount / sizeBytes / fileCount** = `-1` (UNKNOWN; all -1-tolerant). +- **lastModifiedMillis** = **the instant** (SAME `Long.parseLong(requestedTime())` as the pin; `0L` on empty + timeline). Feeds `MTMVTimestampSnapshot(instant)` (:607). A monotonic non-negative instant is a STABLE marker + (unchanged table → same marker → no refresh; new commit → new instant → refresh). Emitting `-1` (hive names-only + default) → `MTMVTimestampSnapshot(-1)` never equals stored → MTMV always-stale OVER-refresh. Emitting legacy's `0L` + → never detects change (the 0-stub). The instant is strictly more correct than both. + +## Signed / to-sign decisions +- **DECISION (needs user sign-off) — hudi MTMV freshness scope.** Legacy `HudiDlaTable.getPartitionSnapshot`/ + `getTableSnapshot` return `MTMVTimestampSnapshot(0L)` (real logic commented out) — hudi CAN be an MV base but never + auto-refreshes on a source commit. **Recommended = implement REAL instant freshness now** (paimon model above): a + strict superset over the broken 0-stub, near-zero cost (the instant is already computed for the scan), and it + naturally avoids the `-1` over-refresh landmine. The alternative (replicate the 0-stub) is worse (table pin = + instant vs partition = 0 semantic split) and still risks `-1`. **Behavior change to accept: hudi MVs will now + auto-refresh when the base hudi table gets a new commit.** Regardless of A/B, do NOT override + getTableFreshness/getPartitionFreshnessMillis. + +## Risks / landmines (verified) +- **R1 (HIGH) — partition NAME rendering.** Must render hive-style `col=val/...` or silent UNPARTITIONED degrade + (checkState-and-skip). Unit-test the arity directly (`HiveUtil.toPartitionValues(name).size() == partKeys.size()`). +- **R2 (HIGH) — partition-source consistency ⇒ FE prune-to-zero data loss.** Hudi does NOT override + `ignorePartitionPruneShortCircuit` (default false) and its `planScan` ignores `requiredPartitions`; a FE prune over + the NEW listPartitions universe that empties to zero SHORT-CIRCUITS `getSplits` to zero rows WITHOUT calling + planScan (`PluginDrivenScanNode.java:957-969`). If listPartitions omits/mis-names a partition that has data, a + partition-predicate query wrongly returns zero/partial rows — a risk INTRODUCED by this step (today the empty + universe ⇒ NOT_PRUNED ⇒ scan-all). Mitigation: make listPartitions' source byte-identical to the scan's partition + source (same `useHiveSyncPartition` selection = `resolvePartitions`' `getAllPartitionPaths` for non-hive-sync), and + close the TODO at `HudiScanPlanProvider.java:378-383`. **Flip-time e2e MUST verify** a partition-predicate query on + a NON-hive-sync table returns complete rows; decide then whether hudi should set `ignorePartitionPruneShortCircuit + = true`. +- **R3 (MEDIUM) — batch-mode OOM is DORMANT for hudi** (no `supportsBatchScan`); this step supplies the partition + COUNT batch mode keys on but does NOT by itself prevent planning-time OOM. Do not claim it does. +- **R4 (HIGH) — TCCL + UGI on the metaClient calls.** beginQuerySnapshot/listPartitions build a + `HoodieTableMetaClient` + `getAllPartitionPaths` (hudi-bundled reflection + secured storage). The metadata/ + materialize thread is NOT TCCL-pinned and hudi's context is NOT wrapped in `TcclPinningConnectorContext`; secured + HMS/HDFS needs `pluginAuth.doAs` (post-flip `context.executeAuthenticated` is NOOP). **Wiring change required**: + `HudiConnector.getMetadata` currently constructs `new HudiConnectorMetadata(getOrCreateClient(), properties)` with + NO context/auth (`HudiConnector.java:87`). Inject a single `execute(Callable)` wrapper (built by HudiConnector) that + does `pluginAuth.doAs` — or `context.executeAuthenticated` when null — INSIDE a TCCL pin to + `HudiConnector.class.getClassLoader()` (restore-in-finally), so the new metaClient-touching methods run + authenticated + pinned. MEMORY: `catalog-spi-plugin-tccl-classloader-gotcha`. +- **R5 (MEDIUM) — instant encoding.** `Long.parseLong(requestedTime())`; empty → `0L`; malformed → NumberFormatException + fail-loud (legacy parity). requestedTime is a numeric `yyyyMMddHHmmssSSS` string, fits/monotonic in long. +- **R6 (LOW) — no dead overrides.** Do NOT add getMvccPartitionView/getTableFreshness/getPartitionFreshnessMillis; + guard with a test asserting the SPI defaults hold. +- **R7 (LOW, out of scope) — planScan re-derives its own queryInstant** (doesn't read the pin off the handle); matches + current/legacy for a LATEST read; note only. + +## Ports / wiring +- Lift the latest-completed-instant derivation + metaClient build + `getAllPartitionPaths` into a shared + package-private helper reachable from BOTH `HudiConnectorMetadata` and `HudiScanPlanProvider` (they must take the + identical instant; avoid a 3rd copy of the `HoodieTableMetadata.create(...)` dance). The metaClient factory should + run under the plugin auth/UGI + TCCL pin (R4). +- `parsePartitionValues` (HD-A3) + `unescapePathName` are already in the connector — reuse; do NOT pull `HiveUtil` + into the plugin (it runs fe-core-side). +- `HudiConnectorMetadata` ctor gains the auth/TCCL execute-wrapper param; `HudiConnector.getMetadata` passes it. + +## Test plan +- Same-loader unit (module already proves the static-method pattern with HudiPartitionValuesTest/PruningTest): + 1. instant→long (factor into a static helper): `"20240101120000000"` → `20240101120000000L`; empty → `0L`. + 2. hive-style NAME rendering + `HiveUtil.toPartitionValues(name).size()==partKeys.size()` (R1): `"2024/01"`+[year, + month], single-col `"2024"`+[dt] (size 1 not 0), hive-style passthrough, `%`-escaped round-trip. + 3. listPartitions per-partition: `lastModifiedMillis == instant` (assert `!= -1` and non-empty table `!= 0`), + values map, hive-style name. + 4. listPartitionNames == names(listPartitions); unpartitioned → emptyList. + 5. useHiveSyncPartition source selection (stub both sources; assert the branch). + 6. beginQuerySnapshot: `lastModifiedFreshness == false`, `snapshotId == instant`. + 7. dead-code guard: getMvccPartitionView/getTableFreshness/getPartitionFreshnessMillis return SPI defaults. +- Flip-time e2e (per `hms-iceberg-delegation-needs-e2e`): partitioned COW+MOR, hive-sync + non-hive-sync; + `SELECT *` → `partition=N/N`; partition-predicate → complete rows + correct pruned count (R2 non-hive-sync guard); + SHOW PARTITIONS / TVFs; MTMV over a partitioned hudi base (new commit → snapshot changes → refresh; unchanged → + stable, no over-refresh); Kerberos HMS+HDFS (R4). From 73f0030b6eac448387799f2195f55445d438090f Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 01:55:12 +0800 Subject: [PATCH 108/334] [doc](catalog) HMS cutover hudi: sign MVCC/freshness decision = A (real instant freshness, paimon model) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit hudi MTMV freshness scope signed 2026-07-09 = implement real instant freshness now (legacy was a 0-const stub; new path pins the latest completed instant, an intentional improvement — hudi MVs will auto-refresh on a new commit). Recorded in the design doc + HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- ...hudi-mvcc-partition-step-design-2026-07-09.md | 16 ++++++++-------- 2 files changed, 9 insertions(+), 9 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 5000545ec49481..4b61256eee8417 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -87,7 +87,7 @@ - **✅ HD-A4 DONE（`7bc90edf4a3` + 复审修 `ce700db35e0`）= force_jni_scanner + COW/MOR 检测硬化**：读 `force_jni_scanner` session 标志（镜像 paimon），planScan `useNativeCowPath = isCow && !forceJni`（force-JNI 把 COW 也推 JNI 路＝legacy `canUseNativeReader`）+ collectMorSplits 无-log native 降级加 `!forceJni` 守卫；**isCow 改由权威 `metaClient.getTableType()` 定**（非 substring 检测，UNKNOWN 不再静默误选路）。**5 维对抗复审揪出真缺陷（本 session 修）**：`populateRangeParams` 未显式 `setFormatType`、只靠单一 node 默认 → ①MOR 无-log native slice 当 `FORMAT_JNI` 空 THudiFileDesc 发 BE（blocker）②COW+force_jni 当 native（major，架空开关）③COW-ORC 当 parquet——已**镜像 `PaimonScanRange` 逐 range 显式设格式**修好（旧测用 `fileFormat("jni")` 假形状掩盖＝Rule-9 masking，已补 planner 真形状 native-parquet/orc 测）。6+4 测。 - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - - **下一步 = Group C 无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C1（分区/MVCC/新鲜度）已出权威设计文档 `hudi-mvcc-partition-step-design-2026-07-09.md`（recon `wf_1a09236d-ee0`：5 读者+synthesis，全 HEAD 核对；起步必读）+ 一个待签字决策**——**hudi MTMV 新鲜度**：旧 `HudiDlaTable` 是 `MTMVTimestampSnapshot(0L)` 恒常量桩（真逻辑被注释、从不感知源变化），**推荐实现真 instant 新鲜度=paimon 模型（行为提升非回退，天然消除 -1 过刷新地雷）**。**关键校正（务必先读设计文档）**：①跟 **paimon 模型**——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不 override** `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下是死代码，task brief 那三条是错缝）；②`beginQuerySnapshot` 钉最新 completed instant 为 **snapshotId**（`Long.parseLong(requestedTime())`，空 timeline→`0L` 非 -1），**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`、分区走 `MTMVTimestampSnapshot(instant)`（**非** snapshot-id 分支）；③**partition name 须 hive-style `col=val/...` 渲染**否则通用侧 `HiveUtil.toPartitionValues` 的 `checkState(size==types)` 抛→skip→静默退化 UNPARTITIONED（R1 HIGH）；④**R2 HIGH 分区源一致性**——listPartitions 源须与扫描侧分区源字节一致（useHiveSyncPartition-aware），否则 FE prune-to-zero 短路致 partition-predicate 查询丢行，须 e2e 验（可能须 `ignorePartitionPruneShortCircuit=true`）；⑤**R4 HIGH 须注入 auth+TCCL wrapper 进 `HudiConnectorMetadata`**（现 `getMetadata` 不传 context，新方法建 `HoodieTableMetaClient` 须 plugin UGI doAs + TCCL pin，memory `catalog-spi-plugin-tccl-classloader-gotcha`）；每分区 `lastModifiedMillis`=instant（-1 过刷新/0 旧桩皆错）。**（其余 Group C 步）**、**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **下一步 = Group C 无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C1（分区/MVCC/新鲜度）已出权威设计文档 `hudi-mvcc-partition-step-design-2026-07-09.md`（recon `wf_1a09236d-ee0`：5 读者+synthesis，全 HEAD 核对；起步必读）+ 决策已签字**——**hudi MTMV 新鲜度 = A（实现真 instant 新鲜度=paimon 模型，2026-07-09 签字）**：旧 `HudiDlaTable` 是 `MTMVTimestampSnapshot(0L)` 恒常量桩（真逻辑被注释、从不感知源变化），改为钉最新 completed instant（整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`）——**有意行为提升非回退，天然消除 -1 过刷新地雷；hudi MV 从此按 commit 自动刷新**（commit/HANDOFF 须记为有意改进）。**关键校正（务必先读设计文档）**：①跟 **paimon 模型**——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不 override** `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下是死代码，task brief 那三条是错缝）；②`beginQuerySnapshot` 钉最新 completed instant 为 **snapshotId**（`Long.parseLong(requestedTime())`，空 timeline→`0L` 非 -1），**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`、分区走 `MTMVTimestampSnapshot(instant)`（**非** snapshot-id 分支）；③**partition name 须 hive-style `col=val/...` 渲染**否则通用侧 `HiveUtil.toPartitionValues` 的 `checkState(size==types)` 抛→skip→静默退化 UNPARTITIONED（R1 HIGH）；④**R2 HIGH 分区源一致性**——listPartitions 源须与扫描侧分区源字节一致（useHiveSyncPartition-aware），否则 FE prune-to-zero 短路致 partition-predicate 查询丢行，须 e2e 验（可能须 `ignorePartitionPruneShortCircuit=true`）；⑤**R4 HIGH 须注入 auth+TCCL wrapper 进 `HudiConnectorMetadata`**（现 `getMetadata` 不传 context，新方法建 `HoodieTableMetaClient` 须 plugin UGI doAs + TCCL pin，memory `catalog-spi-plugin-tccl-classloader-gotcha`）；每分区 `lastModifiedMillis`=instant（-1 过刷新/0 旧桩皆错）。**（其余 Group C 步）**、**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md b/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md index 8a412275447900..6907540f0b1ebc 100644 --- a/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md +++ b/plan-doc/tasks/hudi-mvcc-partition-step-design-2026-07-09.md @@ -73,14 +73,14 @@ branch (:607), identical to paimon; the `MTMVSnapshotIdSnapshot` branch (:589) r → never detects change (the 0-stub). The instant is strictly more correct than both. ## Signed / to-sign decisions -- **DECISION (needs user sign-off) — hudi MTMV freshness scope.** Legacy `HudiDlaTable.getPartitionSnapshot`/ - `getTableSnapshot` return `MTMVTimestampSnapshot(0L)` (real logic commented out) — hudi CAN be an MV base but never - auto-refreshes on a source commit. **Recommended = implement REAL instant freshness now** (paimon model above): a - strict superset over the broken 0-stub, near-zero cost (the instant is already computed for the scan), and it - naturally avoids the `-1` over-refresh landmine. The alternative (replicate the 0-stub) is worse (table pin = - instant vs partition = 0 semantic split) and still risks `-1`. **Behavior change to accept: hudi MVs will now - auto-refresh when the base hudi table gets a new commit.** Regardless of A/B, do NOT override - getTableFreshness/getPartitionFreshnessMillis. +- **DECISION — hudi MTMV freshness scope — SIGNED 2026-07-09 = A (implement REAL instant freshness now, paimon + model).** Legacy `HudiDlaTable.getPartitionSnapshot`/`getTableSnapshot` return `MTMVTimestampSnapshot(0L)` (real + logic commented out) — hudi CAN be an MV base but never auto-refreshes on a source commit. Chosen: pin the latest + completed instant (table = `MTMVSnapshotIdSnapshot(instant)`, partition = `MTMVTimestampSnapshot(instant)`) — a + strict superset over the broken 0-stub, near-zero cost, naturally avoids the `-1` over-refresh landmine. + **Accepted behavior change: hudi MVs will now auto-refresh when the base hudi table gets a new commit** (record + this in the commit/HANDOFF as an intentional improvement, not a legacy diff). Do NOT override + getTableFreshness/getPartitionFreshnessMillis (dead code under flag=false). ## Risks / landmines (verified) - **R1 (HIGH) — partition NAME rendering.** Must render hive-style `col=val/...` or silent UNPARTITIONED degrade From 0d203019eabdbac6a243cc929480b93ae33e2c8f Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 07:26:24 +0800 Subject: [PATCH 109/334] [feat](catalog) HMS cutover hudi: MVCC snapshot + partition listing surface (Group C step 1, dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds the connector-side MVCC/partition surface a partitioned hudi-on-HMS table needs once served post-flip through the generic PluginDrivenScanNode path (like paimon): partition pruning / selectedPartitionNum / SHOW PARTITIONS / TVFs / MTMV freshness. Dormant (hms not in SPI_READY_TYPES; HD-B2 getTableHandle divert not yet armed). Zero fe-core changes; mirrors the paimon template. HudiConnectorMetadata: - beginQuerySnapshot: pin the latest completed instant as the MVCC snapshot (snapshot-id freshness), lastModifiedFreshness LEFT false so fe-core serves MTMVSnapshotIdSnapshot(instant) for the table and MTMVTimestampSnapshot(instant) per partition. INTENTIONAL improvement over legacy HudiDlaTable (which pinned constant 0L and never auto-refreshed) — a hudi MV now refreshes on a new base commit and stays stable otherwise. Do NOT override getMvccPartitionView/getTableFreshness/getPartitionFreshnessMillis (dead code under flag=false). - listPartitions/listPartitionNames/listPartitionValues via a shared collectPartitions: partition-name SOURCE is use_hive_sync_partition-aware (HMS names, else the hudi metadata listing; empty-HMS falls back), mirroring legacy HudiExternalMetaCache.loadPartitionNames. Per partition: hive-style name, unescaped value map, lastModifiedMillis = the instant (stable non-negative marker). Partition NAME rendering (R1): render col=escape(value) via a Hive makePartName port so the generic model's HiveUtil.toPartitionValues re-parse recovers exactly the connector's values with the right arity. Escaping is load-bearing for a single partition column whose value spans '/' (e.g. TimestampBasedKeyGenerator yyyy/MM/dd -> path "2024/01/02"): without escaping the '/' the re-parse would truncate/collide it. Added escapePathName (inverse of the existing unescapePathName). Auth/TCCL (R4): HudiConnector injects a HudiMetaClientExecutor that runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + a TCCL pin to the hudi plugin classloader (the metadata/MTMV thread is not the TCCL-pinned scan thread; post-flip the shared gateway context is NOOP). Shared statics lifted into HudiScanPlanProvider (buildMetaClient / latestCompletedInstant / listAllPartitionPaths) so the MVCC pin and the scan take the identical instant and one copy of the getAllPartitionPaths dance; resolvePartitions refactored to reuse listAllPartitionPaths (byte-identical). Tests: HudiConnectorPartitionListingTest (18) covers instant->long, hive-style rendering + arity + escaped/slash round-trips (incl. the single-column slash no-collision regression), buildPartitionInfos, listPartitions/Names/Values, use_hive_sync_partition source selection + empty-HMS fallback, beginQuerySnapshot pin (static + instance override), and the dead-code SPI-default guards. Reviewed by a 6-dimension adversarial pass with refute-verification (1 major + 2 test-coverage findings, all fixed here; the R2 applyFilter/prune-to-zero residual is pre-existing dormant scan-path code deferred to the flip per the signed design). fe-connector-hudi 76 tests + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hudi/HudiConnector.java | 35 +- .../connector/hudi/HudiConnectorMetadata.java | 149 +++++- .../hudi/HudiMetaClientExecutor.java | 38 ++ .../connector/hudi/HudiScanPlanProvider.java | 110 ++++- .../hudi/DirectHudiMetaClientExecutor.java | 36 ++ .../hudi/HudiBackendDescriptorTest.java | 3 +- .../HudiConnectorPartitionListingTest.java | 430 ++++++++++++++++++ .../hudi/HudiPartitionPruningTest.java | 3 +- .../connector/hudi/HudiTableTypeTest.java | 3 +- 9 files changed, 794 insertions(+), 13 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiMetaClientExecutor.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/DirectHudiMetaClientExecutor.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPartitionListingTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index af0201981825af..1fc226e3b4a16f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -84,7 +84,40 @@ public HudiConnector(Map properties, ConnectorContext context) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - return new HudiConnectorMetadata(getOrCreateClient(), properties); + return new HudiConnectorMetadata(getOrCreateClient(), properties, metaClientExecutor()); + } + + /** + * Builds the metaClient execute-wrapper the metadata partition/snapshot methods run their + * {@code HoodieTableMetaClient}-touching work inside: a TCCL pin to the hudi plugin classloader (so + * hudi-bundled reflection resolves the plugin's child-first copies) around the plugin UGI {@code doAs} + * (Kerberos) — or the FE-injected {@code context.executeAuthenticated} when this is a non-Kerberos + * catalog — restoring the previous TCCL in a {@code finally}. Mirrors the {@link #createClient()} auth + * choice; the TCCL pin is added because — unlike the HMS thrift RPC ({@code ThriftHmsClient.doAs} pins the + * system loader) — building a metaClient / listing partitions off the (unpinned) planning thread needs the + * plugin loader. See {@link HudiMetaClientExecutor} and memory + * {@code catalog-spi-plugin-tccl-classloader-gotcha}. + */ + private HudiMetaClientExecutor metaClientExecutor() { + return new HudiMetaClientExecutor() { + @Override + public T execute(Callable action) { + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(HudiConnector.class.getClassLoader()); + try { + HadoopAuthenticator auth = pluginAuthenticator(); + if (auth != null) { + return auth.doAs(action::call); + } + return context.executeAuthenticated(action); + } catch (Exception e) { + throw new DorisConnectorException("Hudi metadata operation failed for catalog '" + + context.getCatalogName() + "'", e); + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } + }; } /** diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index e40abdec22467e..90cc8d6009896e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -19,11 +19,13 @@ import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorMetadata; +import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.pushdown.ConnectorAnd; import org.apache.doris.connector.api.pushdown.ConnectorComparison; import org.apache.doris.connector.api.pushdown.ConnectorExpression; @@ -45,6 +47,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.util.AbstractMap; import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; @@ -74,12 +77,19 @@ public class HudiConnectorMetadata implements ConnectorMetadata { private static final Logger LOG = LogManager.getLogger(HudiConnectorMetadata.class); + // Catalog property gating the partition-name source (mirrors legacy HMSExternalTable.USE_HIVE_SYNC_PARTITION). + private static final String USE_HIVE_SYNC_PARTITION = "use_hive_sync_partition"; + private final HmsClient hmsClient; private final Map properties; + // Runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + TCCL pin (see R4). + private final HudiMetaClientExecutor metaClientExecutor; - public HudiConnectorMetadata(HmsClient hmsClient, Map properties) { + public HudiConnectorMetadata(HmsClient hmsClient, Map properties, + HudiMetaClientExecutor metaClientExecutor) { this.hmsClient = hmsClient; this.properties = properties; + this.metaClientExecutor = metaClientExecutor; } // ========== ConnectorSchemaOps ========== @@ -222,6 +232,143 @@ public Map getProperties() { return properties; } + // ========== ConnectorMvccOps (MTMV freshness) ========== + + /** + * Pins the LATEST completed instant as the query-begin MVCC snapshot (snapshot-id freshness, paimon model). + * The generic model then serves {@code MTMVSnapshotIdSnapshot(instant)} for the table and + * {@code MTMVTimestampSnapshot(instant)} per partition, so a hudi materialized view auto-refreshes on a new + * base commit and stays stable otherwise. This is an INTENTIONAL improvement over legacy {@code HudiDlaTable} + * (which pinned a constant {@code 0L} and never detected change), NOT a byte-parity port. + * + *

    {@code lastModifiedFreshness} is deliberately LEFT FALSE — that flag routes a last-modified connector + * (hive) to the on-demand {@code getTableFreshness}/{@code getPartitionFreshnessMillis} probes; a snapshot-id + * connector like hudi keeps the instant pin and pays zero extra metadata calls. {@code schemaId} is left + * default ({@code -1}): explicit time travel is a later step. + */ + @Override + public Optional beginQuerySnapshot( + ConnectorSession session, ConnectorTableHandle handle) { + return buildBeginQuerySnapshot(latestInstant((HudiTableHandle) handle)); + } + + /** Builds the query-begin snapshot from a pinned instant. Static for offline unit testing. */ + static Optional buildBeginQuerySnapshot(long instant) { + return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(instant).build()); + } + + // ========== ConnectorTableOps (partitions) ========== + + /** + * Lists all partitions with metadata. {@code filter} is intentionally ignored (paimon / maxcompute parity): + * the generic model lists the full universe and prunes FE-side. + */ + @Override + public List listPartitions(ConnectorSession session, + ConnectorTableHandle handle, Optional filter) { + return collectPartitions((HudiTableHandle) handle); + } + + @Override + public List listPartitionNames(ConnectorSession session, ConnectorTableHandle handle) { + List partitions = collectPartitions((HudiTableHandle) handle); + List names = new ArrayList<>(partitions.size()); + for (ConnectorPartitionInfo partition : partitions) { + names.add(partition.getPartitionName()); + } + return names; + } + + @Override + public List> listPartitionValues(ConnectorSession session, + ConnectorTableHandle handle, List partitionColumns) { + List partitions = collectPartitions((HudiTableHandle) handle); + List> result = new ArrayList<>(partitions.size()); + for (ConnectorPartitionInfo partition : partitions) { + Map rawValues = partition.getPartitionValues(); + // Preserve the requested partitionColumns order (feeds the partition_values() TVF, whose inner-list + // order must match the input), mirroring PaimonConnectorMetadata.listPartitionValues. + List values = new ArrayList<>(partitionColumns.size()); + for (String column : partitionColumns) { + values.add(rawValues.get(column)); + } + result.add(values); + } + return result; + } + + /** + * Shared partition collector backing {@link #listPartitions}, {@link #listPartitionNames} and + * {@link #listPartitionValues}. Lists the raw partition identifiers from the + * {@code use_hive_sync_partition}-aware source (mirroring legacy + * {@code HudiExternalMetaCache.loadPartitionNames}), then renders one {@link ConnectorPartitionInfo} per + * partition. Unpartitioned → {@code emptyList()} (legacy never lists partitions for an unpartitioned + * table). Explicit time-travel (non-latest) partition listing is a later step. + */ + private List collectPartitions(HudiTableHandle handle) { + List partKeyNames = handle.getPartitionKeyNames(); + if (partKeyNames == null || partKeyNames.isEmpty()) { + return Collections.emptyList(); + } + if (useHiveSyncPartition()) { + // hive-sync tables register their partitions in HMS: list the names from there (already authed via + // hmsClient, no metaClient), like legacy. The instant still comes from the timeline. If HMS has none + // (a hive-sync table not yet synced), fall back to the hudi metadata listing (legacy parity). + List hmsNames = hmsClient.listPartitionNames( + handle.getDbName(), handle.getTableName(), -1); + if (hmsNames != null && !hmsNames.isEmpty()) { + return buildPartitionInfos(hmsNames, partKeyNames, latestInstant(handle)); + } + LOG.warn("hive-sync hudi table {}.{} has no HMS partitions; " + + "falling back to hudi metadata partition listing", + handle.getDbName(), handle.getTableName()); + } + // Non-hive-sync (or hive-sync HMS-empty fallback): the instant and the partition paths both come from + // the metaClient, built ONCE under the plugin auth + TCCL pin. Byte-consistent with the scan's unpruned + // partition source (resolvePartitions -> listAllPartitionPaths), which is the R2 prune-to-zero guard. + Map.Entry> listing = metaClientExecutor.execute(() -> { + HoodieTableMetaClient metaClient = + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), handle.getBasePath()); + return new AbstractMap.SimpleImmutableEntry<>( + HudiScanPlanProvider.latestCompletedInstant(metaClient), + HudiScanPlanProvider.listAllPartitionPaths(metaClient)); + }); + return buildPartitionInfos(listing.getValue(), partKeyNames, listing.getKey()); + } + + /** + * Renders one {@link ConnectorPartitionInfo} per raw partition path. {@code partitionName} = hive-style + * (for the fe-core re-parse), {@code partitionValues} = the unescaped value map (for the TVF), + * {@code lastModifiedMillis} = the pinned instant (a stable, monotonic freshness marker feeding + * {@code MTMVTimestampSnapshot}; row/size/file counts stay {@code UNKNOWN}). Static + package-private for + * offline unit testing. + */ + static List buildPartitionInfos( + List rawPaths, List partKeyNames, long instant) { + List result = new ArrayList<>(rawPaths.size()); + for (String rawPath : rawPaths) { + // Parse the unescaped values ONCE; render the hive-style name from the SAME values so the name and + // the values map agree by construction (the name re-parses back to these values in fe-core). + Map values = HudiScanPlanProvider.parsePartitionValues(rawPath, partKeyNames); + String name = HudiScanPlanProvider.renderHiveStylePartitionName(partKeyNames, values); + result.add(new ConnectorPartitionInfo(name, values, Collections.emptyMap(), + ConnectorPartitionInfo.UNKNOWN, ConnectorPartitionInfo.UNKNOWN, + instant, ConnectorPartitionInfo.UNKNOWN)); + } + return result; + } + + /** Pins the latest completed instant, building the metaClient under the plugin auth + TCCL pin. */ + private long latestInstant(HudiTableHandle handle) { + return metaClientExecutor.execute(() -> + HudiScanPlanProvider.latestCompletedInstant( + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), handle.getBasePath()))); + } + + private boolean useHiveSyncPartition() { + return Boolean.parseBoolean(properties.getOrDefault(USE_HIVE_SYNC_PARTITION, "false")); + } + /** * Builds the BE table descriptor for a hudi table: a {@code TTableType.HIVE_TABLE} carrying a * {@link THiveTable}, a direct port of legacy hudi (which rode {@code HMSExternalTable.toThrift} / diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiMetaClientExecutor.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiMetaClientExecutor.java new file mode 100644 index 00000000000000..159b451b6c1990 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiMetaClientExecutor.java @@ -0,0 +1,38 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import java.util.concurrent.Callable; + +/** + * Runs a Hudi {@code HoodieTableMetaClient}-touching action under the plugin's Kerberos UGI {@code doAs} and a + * TCCL pin to the hudi plugin classloader. + * + *

    Built by {@link HudiConnector} and injected into {@link HudiConnectorMetadata} so the partition-listing / + * MVCC-snapshot metadata methods — which build a live metaClient off the query-planning / MTMV-refresh thread, + * NOT the TCCL-pinned scan thread ({@code PluginDrivenScanNode.onPluginClassLoader}) — resolve hudi-bundled + * reflection against the plugin's child-first copies and authenticate to a secured HMS/HDFS (post-flip the + * FE-injected {@code context.executeAuthenticated} is NOOP for a sibling). See + * {@code HudiConnector.metaClientExecutor()} and memory {@code catalog-spi-plugin-tccl-classloader-gotcha}.

    + * + *

    A generic method (not a lambda target): the implementation is an anonymous class in {@link HudiConnector}. + * Checked exceptions from {@code action} are wrapped by the implementation.

    + */ +interface HudiMetaClientExecutor { + T execute(Callable action); +} diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index f7d0c8ed37d43e..7025ea18844019 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -342,20 +342,63 @@ private List resolvePartitions( } try { - HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder() - .enable(HoodieTableMetadataUtil.isFilesPartitionAvailable(metaClient)) - .build(); - HoodieLocalEngineContext engineCtx = new HoodieLocalEngineContext(metaClient.getStorageConf()); - HoodieTableMetadata tableMetadata = HoodieTableMetadata.create( - engineCtx, metaClient.getStorage(), metadataConfig, - metaClient.getBasePath().toString(), true); - return tableMetadata.getAllPartitionPaths(); + return listAllPartitionPaths(metaClient); } catch (Exception e) { throw new DorisConnectorException( "Failed to list partitions for " + handle.getBasePath(), e); } } + /** + * Builds a {@link HoodieTableMetaClient} from a Hadoop {@link Configuration} and base path. Package-private + * static so the metadata path ({@link HudiConnectorMetadata}) builds the metaClient the same way the scan + * does, from inside the plugin-auth + TCCL pin its execute-wrapper supplies. + */ + static HoodieTableMetaClient buildMetaClient(Configuration conf, String basePath) { + return HoodieTableMetaClient.builder() + .setConf(new org.apache.hudi.storage.hadoop.HadoopStorageConfiguration(conf)) + .setBasePath(basePath) + .build(); + } + + /** + * Returns the LATEST completed instant as a numeric long ({@code yyyyMMddHHmmssSSS}), or {@code 0L} when + * the timeline has none. Byte-faithful port of legacy {@code HudiUtils.getLastTimeStamp} and the same + * timeline {@link #planScan} reads at query time — so the MVCC pin and the scan take the identical instant. + */ + static long latestCompletedInstant(HoodieTableMetaClient metaClient) { + Optional requestedTime = metaClient.getCommitsAndCompactionTimeline() + .filterCompletedInstants().lastInstant().toJavaOptional() + .map(HoodieInstant::requestedTime); + return requestedTimeToInstant(requestedTime); + } + + /** + * Pure numeric mapping backing {@link #latestCompletedInstant}: a present {@code requestedTime} parses to a + * long ({@code Long.parseLong}, fail-loud on malformed = legacy parity); absent → {@code 0L} (legacy + * empty-timeline sentinel, {@code >= 0} so it survives the dictionary-refresh filter). Extracted so the + * empty/value semantics are unit-testable without a live metaClient. + */ + static long requestedTimeToInstant(Optional requestedTime) { + return requestedTime.map(Long::parseLong).orElse(0L); + } + + /** + * Lists ALL partition relative paths from the Hudi metadata table (COW/MOR agnostic). Byte-faithful port of + * legacy {@code HudiPartitionUtils.getAllPartitionNames}; extracted so both {@link #resolvePartitions} and + * the metadata partition-listing path share one copy of the {@code HoodieTableMetadata.create(...)} dance. + */ + static List listAllPartitionPaths(HoodieTableMetaClient metaClient) throws Exception { + HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder() + .enable(HoodieTableMetadataUtil.isFilesPartitionAvailable(metaClient)) + .build(); + HoodieLocalEngineContext engineCtx = new HoodieLocalEngineContext(metaClient.getStorageConf()); + HoodieTableMetadata tableMetadata = HoodieTableMetadata.create( + engineCtx, metaClient.getStorage(), metadataConfig, + metaClient.getBasePath().toString(), true); + return tableMetadata.getAllPartitionPaths(); + } + /** * Parse a Hudi partition's relative path into a column→value map, byte-faithful to legacy * {@code HudiPartitionUtils.parsePartitionValues}. Handles BOTH hive-style ("year=2024/month=01") and Hudi's @@ -411,6 +454,57 @@ static Map parsePartitionValues( return values; } + /** + * Renders a Hive-style partition name ({@code "col0=val0/col1=val1/..."}) from a column→value map, in + * partition-key order, ESCAPING each value with {@link #escapePathName} (the canonical Hive {@code + * makePartName}). + * + *

    MANDATORY for the generic MVCC model: fe-core rebuilds the partition item by re-parsing this + * name via {@code HiveUtil.toPartitionValues} under a {@code checkState(values.size()==types.size())}. A raw + * positional path ({@code "2024/01"}) would yield the wrong value count → the partition is skipped + * → silent UNPARTITIONED degrade, so a hive-style name is required. Escaping is MANDATORY too: + * {@code HiveUtil.toPartitionValues} splits on {@code '/'}, so a value that itself spans {@code '/'} (e.g. a + * single partition column with a {@code yyyy/MM/dd} output format → path {@code "2024/01/02"}) must be + * escaped ({@code "%2F"}) or the re-parse would truncate/collide it. Since {@code escapePathName} is the + * exact inverse of {@link #escapePathName}'s unescape (the same set {@code HiveUtil.toPartitionValues} uses), + * the re-parse recovers EXACTLY the values {@link #parsePartitionValues} produced. Static + package-private + * for direct unit testing. + */ + static String renderHiveStylePartitionName(List partKeyNames, Map values) { + StringBuilder sb = new StringBuilder(); + for (String col : partKeyNames) { + if (sb.length() > 0) { + sb.append('/'); + } + sb.append(col).append('=').append(escapePathName(values.get(col))); + } + return sb.toString(); + } + + // Hive FileUtils.charToEscape minus the control range: escaped so a partition VALUE containing one of these + // survives the round-trip through HiveUtil.toPartitionValues (which url-unescapes). '/' and '=' are the + // load-bearing ones (structural to the re-parse); the rest mirror Hive for name faithfulness. + private static final String CHARS_TO_ESCAPE = "\"#%'*/:=?\\{[]^"; + + /** + * URL-encodes a partition value into a Hive-escaped path component (e.g. {@code "a/b"} → {@code + * "a%2Fb"}). Byte-faithful port of Hive's {@code org.apache.hadoop.hive.common.FileUtils.escapePathName} and + * the exact inverse of {@link #unescapePathName}, so a rendered hive-style name re-parses (unescapes) back + * to the original value. Inlined so the connector needs no hive-common dependency. + */ + private static String escapePathName(String value) { + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < value.length(); i++) { + char c = value.charAt(i); + if (c < 0x20 || c == 0x7F || CHARS_TO_ESCAPE.indexOf(c) >= 0) { + sb.append('%').append(String.format("%02X", (int) c)); + } else { + sb.append(c); + } + } + return sb.toString(); + } + /** * URL-decodes a Hive-escaped path component (e.g. "a%2Fb" → "a/b"). Byte-faithful port of Hive's * {@code org.apache.hadoop.hive.common.FileUtils.unescapePathName} (identical to the fe-connector-hive diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/DirectHudiMetaClientExecutor.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/DirectHudiMetaClientExecutor.java new file mode 100644 index 00000000000000..088e288c1eeca1 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/DirectHudiMetaClientExecutor.java @@ -0,0 +1,36 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import java.util.concurrent.Callable; + +/** + * Test {@link HudiMetaClientExecutor} that runs the action directly (no plugin auth, no TCCL pin). Used by + * tests that do not exercise the metaClient-touching partition/snapshot paths; a checked exception surfaces + * as an unchecked wrapper so a mis-set-up fixture fails loud. + */ +final class DirectHudiMetaClientExecutor implements HudiMetaClientExecutor { + @Override + public T execute(Callable action) { + try { + return action.call(); + } catch (Exception e) { + throw new RuntimeException(e); + } + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java index a8f5b88e13f189..13234ecad5d2db 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java @@ -42,7 +42,8 @@ public class HudiBackendDescriptorTest { @Test public void buildTableDescriptorIsHiveTable() { - HudiConnectorMetadata md = new HudiConnectorMetadata(null, Collections.emptyMap()); + HudiConnectorMetadata md = new HudiConnectorMetadata(null, Collections.emptyMap(), + new DirectHudiMetaClientExecutor()); TTableDescriptor desc = md.buildTableDescriptor(null, 7L, "t", "db", "t", 3, 100L); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPartitionListingTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPartitionListingTest.java new file mode 100644 index 00000000000000..a6693234d15006 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorPartitionListingTest.java @@ -0,0 +1,430 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorPartitionInfo; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.AbstractMap; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.OptionalLong; +import java.util.concurrent.Callable; + +/** + * Tests the Hudi MVCC / listPartitions / freshness surface added so a PARTITIONED hudi-on-HMS table, served + * post-flip through the GENERIC {@code PluginDrivenScanNode} path (like paimon), has a correct partition + + * MVCC-snapshot surface. Each assertion pins WHY the behavior matters: + *

      + *
    • the query-begin pin is a snapshot-id (instant) freshness, NOT a last-modified one (paimon model);
      • + *
    • per-partition names are HIVE-STYLE so the generic model's {@code HiveUtil.toPartitionValues} re-parse + * matches the partition-column arity (a raw positional name silently degrades to UNPARTITIONED);
      • + *
    • {@code lastModifiedMillis} is the pinned instant (a stable freshness marker), never {@code -1};
      • + *
    • the partition-name SOURCE follows {@code use_hive_sync_partition} (HMS vs hudi metadata);
      • + *
    • the dead-code MVCC/freshness seams are NOT overridden (SPI defaults hold).
      • + *
    + */ +public class HudiConnectorPartitionListingTest { + + private static final List YEAR_MONTH = Arrays.asList("year", "month"); + + // ── item 1: instant string → long (pure) ────────────────────────────────────────────────────────── + + @Test + public void requestedTimeParsesToLong() { + Assertions.assertEquals(20240101120000000L, + HudiScanPlanProvider.requestedTimeToInstant(Optional.of("20240101120000000"))); + } + + @Test + public void emptyTimelinePinsZero() { + // Empty timeline pins 0L (>= 0 so it survives getNewestUpdateVersionOrTime's v>=0 filter), NOT -1. + Assertions.assertEquals(0L, HudiScanPlanProvider.requestedTimeToInstant(Optional.empty())); + } + + // ── item 2: hive-style NAME rendering + arity round-trip ─────────────────────────────────────────── + + @Test + public void positionalPathRendersHiveStyleWithCorrectArity() { + // Hudi's DEFAULT layout is positional "2024/01" with NO "col=" prefix. Rendering MUST inject the keys + // so the generic re-parse yields exactly partKeys.size() values (else checkState throws -> the + // partition is dropped -> silent UNPARTITIONED degrade). + String name = render("2024/01", YEAR_MONTH); + Assertions.assertEquals("year=2024/month=01", name); + assertRoundTrips(name, YEAR_MONTH, Arrays.asList("2024", "01")); + } + + @Test + public void singleColumnPositionalPathRendersOneSegment() { + // Single partition column with a bare value: size 1, not 0. + String name = render("2024", Collections.singletonList("dt")); + Assertions.assertEquals("dt=2024", name); + assertRoundTrips(name, Collections.singletonList("dt"), Collections.singletonList("2024")); + } + + @Test + public void hiveStylePathIsRenderedIdempotently() { + String name = render("year=2024/month=01", YEAR_MONTH); + Assertions.assertEquals("year=2024/month=01", name); + assertRoundTrips(name, YEAR_MONTH, Arrays.asList("2024", "01")); + } + + @Test + public void onDiskEscapedValueRoundTripsToRealValue() { + // A value escaped on disk ("a%20b" = "a b") parses to the real value and the rendered name re-parses + // back to it. (Space is not in Hive's escape set, so the rendered value carries a literal space.) + List dt = Collections.singletonList("dt"); + Assertions.assertEquals("a b", + HudiScanPlanProvider.parsePartitionValues("dt=a%20b", dt).get("dt")); + String name = render("dt=a%20b", dt); + Assertions.assertEquals("dt=a b", name); + assertRoundTrips(name, dt, Collections.singletonList("a b")); + } + + @Test + public void singleColumnSlashValueIsEscapedSoItRoundTrips() { + // THE bug this guards: a single partition column whose value spans '/' (e.g. TimestampBasedKeyGenerator + // OutputDateFormat=yyyy/MM/dd -> path "2024/01/02"). The value is the WHOLE path; without escaping the + // '/', HiveUtil.toPartitionValues would truncate it to "2024". Escaping to "%2F" makes it round-trip. + List dt = Collections.singletonList("dt"); + Assertions.assertEquals("2024/01/02", + HudiScanPlanProvider.parsePartitionValues("2024/01/02", dt).get("dt")); + String name = render("2024/01/02", dt); + Assertions.assertEquals("dt=2024%2F01%2F02", name); + assertRoundTrips(name, dt, Collections.singletonList("2024/01/02")); + } + + @Test + public void distinctSlashValuedPartitionsDoNotCollide() { + // Two distinct single-column slash paths must render distinct names AND re-parse to distinct values — + // else the generic model collapses them onto one partition key, corrupting MTMV per-partition tracking. + List dt = Collections.singletonList("dt"); + String a = render("2024/01/02", dt); + String b = render("2024/03/04", dt); + Assertions.assertNotEquals(a, b); + assertRoundTrips(a, dt, Collections.singletonList("2024/01/02")); + assertRoundTrips(b, dt, Collections.singletonList("2024/03/04")); + } + + // ── item 3/4: buildPartitionInfos + listPartitions/Names/Values ──────────────────────────────────── + + @Test + public void buildPartitionInfosStampsInstantAndValues() { + List infos = HudiConnectorMetadata.buildPartitionInfos( + Arrays.asList("2024/01", "2024/02"), YEAR_MONTH, 20240101120000000L); + + Assertions.assertEquals(2, infos.size()); + ConnectorPartitionInfo first = infos.get(0); + Assertions.assertEquals("year=2024/month=01", first.getPartitionName()); + Assertions.assertEquals("2024", first.getPartitionValues().get("year")); + Assertions.assertEquals("01", first.getPartitionValues().get("month")); + // lastModifiedMillis == the instant (a stable non-negative marker), NOT the -1 UNKNOWN sentinel. + Assertions.assertEquals(20240101120000000L, first.getLastModifiedMillis()); + Assertions.assertNotEquals(ConnectorPartitionInfo.UNKNOWN, first.getLastModifiedMillis()); + // row/size/file counts stay UNKNOWN (not collected on the hot path). + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, first.getRowCount()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, first.getSizeBytes()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, first.getFileCount()); + } + + @Test + public void listPartitionNamesMatchesListPartitions() { + HudiConnectorMetadata md = metadata(false, + new RecordingHmsClient(null), stub(new AbstractMap.SimpleImmutableEntry<>( + 99L, Arrays.asList("2024/01", "2024/02")))); + ConnectorTableHandle handle = partitioned(); + + List names = md.listPartitionNames(null, handle); + List parts = md.listPartitions(null, handle, Optional.empty()); + + Assertions.assertEquals(Arrays.asList("year=2024/month=01", "year=2024/month=02"), names); + Assertions.assertEquals(names.size(), parts.size()); + for (int i = 0; i < names.size(); i++) { + Assertions.assertEquals(names.get(i), parts.get(i).getPartitionName()); + } + } + + @Test + public void listPartitionValuesProjectsRequestedColumnOrder() { + HudiConnectorMetadata md = metadata(false, + new RecordingHmsClient(null), stub(new AbstractMap.SimpleImmutableEntry<>( + 99L, Collections.singletonList("2024/01")))); + // Request in reversed order: inner list order must follow the input columns, not the storage order. + List> values = md.listPartitionValues(null, partitioned(), Arrays.asList("month", "year")); + Assertions.assertEquals(Collections.singletonList(Arrays.asList("01", "2024")), values); + } + + @Test + public void unpartitionedTableListsNothing() { + HudiConnectorMetadata md = metadata(false, new RecordingHmsClient(null), + new DirectHudiMetaClientExecutor()); + ConnectorTableHandle handle = new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(Collections.emptyList()).build(); + Assertions.assertTrue(md.listPartitions(null, handle, Optional.empty()).isEmpty()); + Assertions.assertTrue(md.listPartitionNames(null, handle).isEmpty()); + } + + // ── item 5: use_hive_sync_partition source selection ─────────────────────────────────────────────── + + @Test + public void hiveSyncTableListsPartitionsFromHms() { + RecordingHmsClient hms = new RecordingHmsClient(Arrays.asList("year=2024/month=01", "year=2024/month=02")); + // Stub returns the instant (a Long) for the timeline call; the partition NAMES must come from HMS. + HudiConnectorMetadata md = metadata(true, hms, stub(7L)); + + List names = md.listPartitionNames(null, partitioned()); + + Assertions.assertEquals(Arrays.asList("year=2024/month=01", "year=2024/month=02"), names); + Assertions.assertTrue(hms.listPartitionNamesCalled, "hive-sync must list partitions from HMS"); + } + + @Test + public void hiveSyncWithEmptyHmsFallsBackToMetaClientListing() { + // A hive-sync table not yet synced to HMS: empty HMS must fall through to the hudi metadata listing + // (the prune-to-zero guard), stamped with the metaClient instant — NOT return zero partitions. + RecordingHmsClient hms = new RecordingHmsClient(Collections.emptyList()); + HudiConnectorMetadata md = metadata(true, hms, + stub(new AbstractMap.SimpleImmutableEntry<>(5L, Collections.singletonList("2024/01")))); + + List parts = md.listPartitions(null, partitioned(), Optional.empty()); + + Assertions.assertTrue(hms.listPartitionNamesCalled, "hive-sync must first consult HMS"); + Assertions.assertEquals(1, parts.size()); + Assertions.assertEquals("year=2024/month=01", parts.get(0).getPartitionName()); + Assertions.assertEquals(5L, parts.get(0).getLastModifiedMillis()); + } + + @Test + public void nonHiveSyncTableNeverConsultsHms() { + RecordingHmsClient hms = new RecordingHmsClient(null); // throws if listPartitionNames is called + HudiConnectorMetadata md = metadata(false, hms, + stub(new AbstractMap.SimpleImmutableEntry<>(88L, Collections.singletonList("2024/01")))); + + List parts = md.listPartitions(null, partitioned(), Optional.empty()); + + Assertions.assertEquals(1, parts.size()); + Assertions.assertEquals("year=2024/month=01", parts.get(0).getPartitionName()); + Assertions.assertEquals(88L, parts.get(0).getLastModifiedMillis()); + Assertions.assertFalse(hms.listPartitionNamesCalled, "non-hive-sync must NOT consult HMS"); + } + + // ── item 6: beginQuerySnapshot ───────────────────────────────────────────────────────────────────── + + @Test + public void beginQuerySnapshotPinsInstantWithoutLastModifiedFlag() { + Optional snapshot = + HudiConnectorMetadata.buildBeginQuerySnapshot(20240101120000000L); + Assertions.assertTrue(snapshot.isPresent()); + Assertions.assertEquals(20240101120000000L, snapshot.get().getSnapshotId()); + // The one bit that separates hudi (snapshot-id) from hive (last-modified): MUST stay false, else the + // generic model would route hudi to the on-demand getTableFreshness probe (which hudi does not provide). + Assertions.assertFalse(snapshot.get().isLastModifiedFreshness()); + // Time travel is a later step: schemaId stays default (-1 => latest schema). + Assertions.assertEquals(-1L, snapshot.get().getSchemaId()); + } + + @Test + public void beginQuerySnapshotOverrideThreadsInstantIntoPin() { + // Drive the ACTUAL SPI override (not just the static helper): the stub executor returns the instant + // latestInstant would read off the timeline, so the override must thread it into the pin unchanged and + // leave lastModifiedFreshness false. Guards a mutation of the override body to empty / a wrong instant. + HudiConnectorMetadata md = metadata(false, new RecordingHmsClient(null), stub(20240101120000000L)); + Optional snapshot = md.beginQuerySnapshot(null, partitioned()); + Assertions.assertTrue(snapshot.isPresent()); + Assertions.assertEquals(20240101120000000L, snapshot.get().getSnapshotId()); + Assertions.assertFalse(snapshot.get().isLastModifiedFreshness()); + } + + // ── item 7: dead-code guard (SPI defaults hold) ──────────────────────────────────────────────────── + + @Test + public void mvccPartitionViewAndFreshnessSeamsAreNotOverridden() { + HudiConnectorMetadata md = metadata(false, new RecordingHmsClient(null), + new DirectHudiMetaClientExecutor()); + ConnectorTableHandle handle = partitioned(); + // A snapshot-id connector must NOT provide a range view (that would flip getPartitionSnapshot to the + // MTMVSnapshotIdSnapshot branch) nor the last-modified freshness probes (dead code under flag=false). + Assertions.assertFalse(md.getMvccPartitionView(null, handle).isPresent()); + Assertions.assertFalse(md.getTableFreshness(null, handle).isPresent()); + Assertions.assertEquals(OptionalLong.empty(), + md.getPartitionFreshnessMillis(null, handle, "year=2024/month=01")); + } + + // ── helpers ──────────────────────────────────────────────────────────────────────────────────────── + + private static HudiTableHandle partitioned() { + return new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(YEAR_MONTH).build(); + } + + private static HudiConnectorMetadata metadata(boolean useHiveSync, HmsClient hms, + HudiMetaClientExecutor executor) { + Map props = useHiveSync + ? Collections.singletonMap("use_hive_sync_partition", "true") + : Collections.emptyMap(); + return new HudiConnectorMetadata(hms, props, executor); + } + + /** Executor that ignores the action and returns a canned value (stubs out the live metaClient). */ + private static HudiMetaClientExecutor stub(Object cannedReturn) { + return new HudiMetaClientExecutor() { + @Override + @SuppressWarnings("unchecked") + public T execute(Callable action) { + return (T) cannedReturn; + } + }; + } + + /** Renders the hive-style name for a raw partition path the way buildPartitionInfos does (parse then render). */ + private static String render(String rawPath, List partKeys) { + return HudiScanPlanProvider.renderHiveStylePartitionName( + partKeys, HudiScanPlanProvider.parsePartitionValues(rawPath, partKeys)); + } + + /** Asserts the rendered name re-parses (fe-core-style) to exactly the expected values, correct arity. */ + private static void assertRoundTrips(String name, List partKeys, List expectedValues) { + List reparsed = hiveToPartitionValues(name); + Assertions.assertEquals(partKeys.size(), reparsed.size(), + "re-parsed value count must equal the partition-column count (else checkState throws)"); + Assertions.assertEquals(expectedValues, reparsed); + } + + /** + * Local mirror of fe-core {@code HiveUtil.toPartitionValues} (cannot import fe-core from a connector test): + * the generic model re-parses the rendered name EXACTLY this way under + * {@code checkState(values.size() == types.size())}. + */ + private static List hiveToPartitionValues(String partitionName) { + List result = new ArrayList<>(); + int start = 0; + while (true) { + while (start < partitionName.length() && partitionName.charAt(start) != '=') { + start++; + } + start++; + int end = start; + while (end < partitionName.length() && partitionName.charAt(end) != '/') { + end++; + } + if (start > partitionName.length()) { + break; + } + result.add(unescape(partitionName.substring(start, end))); + start = end + 1; + } + return result; + } + + private static String unescape(String path) { + StringBuilder sb = new StringBuilder(); + for (int i = 0; i < path.length(); i++) { + char c = path.charAt(i); + if (c == '%' && i + 2 < path.length()) { + int code; + try { + code = Integer.parseInt(path.substring(i + 1, i + 3), 16); + } catch (Exception e) { + code = -1; + } + if (code >= 0) { + sb.append((char) code); + i += 2; + continue; + } + } + sb.append(c); + } + return sb.toString(); + } + + /** Minimal {@link HmsClient} double: records whether listPartitionNames was called; the rest fail loud. */ + private static final class RecordingHmsClient implements HmsClient { + private final List partitionNames; + private boolean listPartitionNamesCalled; + + RecordingHmsClient(List partitionNames) { + this.partitionNames = partitionNames; + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + listPartitionNamesCalled = true; + if (partitionNames == null) { + throw new UnsupportedOperationException("listPartitionNames must not be called here"); + } + return partitionNames; + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java index af6b59a532be0b..d6000b6cd40b27 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java @@ -162,7 +162,8 @@ public void testUnpartitionedTableIsNotTouched() { private Optional> applyFilter( HudiTableHandle handle, ConnectorExpression expr) { HudiConnectorMetadata metadata = new HudiConnectorMetadata( - new FakeHmsClient(PARTITIONS), Collections.emptyMap()); + new FakeHmsClient(PARTITIONS), Collections.emptyMap(), + new DirectHudiMetaClientExecutor()); return metadata.applyFilter(null, handle, new ConnectorFilterConstraint(expr)); } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java index ef172b9dc17ce6..5d1aee3f137181 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTableTypeTest.java @@ -51,7 +51,8 @@ private String detect(String inputFormat, Map parameters) { .parameters(parameters) .build(); HudiConnectorMetadata metadata = - new HudiConnectorMetadata(new FakeHmsClient(info), Collections.emptyMap()); + new HudiConnectorMetadata(new FakeHmsClient(info), Collections.emptyMap(), + new DirectHudiMetaClientExecutor()); Optional handle = metadata.getTableHandle(null, "db", "t"); Assertions.assertTrue(handle.isPresent()); return ((HudiTableHandle) handle.get()).getHudiTableType(); From 7be84c95fed6d6a80ea92e624a9c492f3098d8e8 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 07:27:48 +0800 Subject: [PATCH 110/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20hudi?= =?UTF-8?q?=20HD-C1=20(MVCC=20snapshot=20+=20partition=20listing=20+=20fre?= =?UTF-8?q?shness)=20DONE;=20next=20=3D=20Group=20C=E4=BD=99=20(time=20tra?= =?UTF-8?q?vel=20/=20incremental=20/=20schema-evolution)=20+=20Group=20D,?= =?UTF-8?q?=20then=20HD-B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4b61256eee8417..56346e10c1db58 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。下一步 = Group C 无回退补齐（连接器侧，全部翻闸前）：MVCC/listPartitions、时间旅行、增量读〔最高风险〕、schema evolution + Group D 写-拒安全网 → HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。下一步 = Group C 余项无回退补齐（连接器侧，全部翻闸前）：时间旅行（HD-C2）、增量读〔最高风险 HD-C3〕、schema evolution（HD-C4/C5，用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -87,7 +87,8 @@ - **✅ HD-A4 DONE（`7bc90edf4a3` + 复审修 `ce700db35e0`）= force_jni_scanner + COW/MOR 检测硬化**：读 `force_jni_scanner` session 标志（镜像 paimon），planScan `useNativeCowPath = isCow && !forceJni`（force-JNI 把 COW 也推 JNI 路＝legacy `canUseNativeReader`）+ collectMorSplits 无-log native 降级加 `!forceJni` 守卫；**isCow 改由权威 `metaClient.getTableType()` 定**（非 substring 检测，UNKNOWN 不再静默误选路）。**5 维对抗复审揪出真缺陷（本 session 修）**：`populateRangeParams` 未显式 `setFormatType`、只靠单一 node 默认 → ①MOR 无-log native slice 当 `FORMAT_JNI` 空 THudiFileDesc 发 BE（blocker）②COW+force_jni 当 native（major，架空开关）③COW-ORC 当 parquet——已**镜像 `PaimonScanRange` 逐 range 显式设格式**修好（旧测用 `fileFormat("jni")` 假形状掩盖＝Rule-9 masking，已补 planner 真形状 native-parquet/orc 测）。6+4 测。 - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - - **下一步 = Group C 无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C1（分区/MVCC/新鲜度）已出权威设计文档 `hudi-mvcc-partition-step-design-2026-07-09.md`（recon `wf_1a09236d-ee0`：5 读者+synthesis，全 HEAD 核对；起步必读）+ 决策已签字**——**hudi MTMV 新鲜度 = A（实现真 instant 新鲜度=paimon 模型，2026-07-09 签字）**：旧 `HudiDlaTable` 是 `MTMVTimestampSnapshot(0L)` 恒常量桩（真逻辑被注释、从不感知源变化），改为钉最新 completed instant（整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`）——**有意行为提升非回退，天然消除 -1 过刷新地雷；hudi MV 从此按 commit 自动刷新**（commit/HANDOFF 须记为有意改进）。**关键校正（务必先读设计文档）**：①跟 **paimon 模型**——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不 override** `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下是死代码，task brief 那三条是错缝）；②`beginQuerySnapshot` 钉最新 completed instant 为 **snapshotId**（`Long.parseLong(requestedTime())`，空 timeline→`0L` 非 -1），**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`、分区走 `MTMVTimestampSnapshot(instant)`（**非** snapshot-id 分支）；③**partition name 须 hive-style `col=val/...` 渲染**否则通用侧 `HiveUtil.toPartitionValues` 的 `checkState(size==types)` 抛→skip→静默退化 UNPARTITIONED（R1 HIGH）；④**R2 HIGH 分区源一致性**——listPartitions 源须与扫描侧分区源字节一致（useHiveSyncPartition-aware），否则 FE prune-to-zero 短路致 partition-predicate 查询丢行，须 e2e 验（可能须 `ignorePartitionPruneShortCircuit=true`）；⑤**R4 HIGH 须注入 auth+TCCL wrapper 进 `HudiConnectorMetadata`**（现 `getMetadata` 不传 context，新方法建 `HoodieTableMetaClient` 须 plugin UGI doAs + TCCL pin，memory `catalog-spi-plugin-tccl-classloader-gotcha`）；每分区 `lastModifiedMillis`=instant（-1 过刷新/0 旧桩皆错）。**（其余 Group C 步）**、**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 + - **下一步 = Group C 余项无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From cc1de5e673f0f90fc3c1e078ccf39d5ae0f5ea57 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 08:24:03 +0800 Subject: [PATCH 111/334] [feat](catalog) HMS cutover hudi HD-C2: FOR TIME AS OF time travel (dormant) Serve hudi-on-HMS FOR TIME AS OF connector-side (mirroring paimon; byte-faithful to legacy HudiScanNode), so a hudi table read through the generic PluginDrivenScanNode path after the catalog flip honors an explicit instant with no regression. Establishes the query-instant pin spine on HudiTableHandle that incremental read (next step) reuses. - HudiTableHandle: add nullable String queryInstant (+ getter, Builder setter, copies in ctor and toBuilder so it preserves prunedPartitionPaths). No equals/hashCode (reference identity kept; pin excluded from identity, mirroring paimon). - HudiConnectorMetadata.resolveTimeTravel: TIMESTAMP -> strip [-: ] (legacy parity, no session-TZ, no epoch-millis, no timeline validation) and pin it via an internal MVCC property; SNAPSHOT_ID/VERSION_REF -> throw the byte-for-byte legacy reject message ("Hudi does not support `FOR VERSION AS OF`, please use `FOR TIME AS OF`"); other kinds -> Optional.empty() (SPI default; INCREMENTAL is a later step). - HudiConnectorMetadata.applySnapshot: property present -> stamp queryInstant via toBuilder (preserves prunedPartitionPaths, which applyFilter set earlier at scan time); empty properties (query-begin latest pin) or null -> handle UNCHANGED, so every non-time-travel read stays byte-identical (planScan falls back to timeline.lastInstant()). - HudiScanPlanProvider.planScan: honor handle.getQueryInstant() at the single instant chokepoint; drives COW/MOR before-or-on file selection AND the MOR-JNI THudiFileDesc instantTime consistently. Zero fe-core changes (pin flows via the generic MVCC seam: loadSnapshot -> resolveTimeTravel -> ConnectorMvccSnapshot property -> applyMvccSnapshotPin -> applySnapshot -> handle -> planScan). Dormant until "hms" enters SPI_READY_TYPES. Design: plan-doc/tasks/hudi-time-travel-step-design-2026-07-09.md. 5-reader recon + reconciliation critic + 4-lens adversarial review (no-regression / legacy-parity / pin-plumbing / test-strength) + refutation verify: 0 confirmed defects. Deferrals (documented, e2e-owed): schema-at-instant on schema-on-read evolved tables and the partition-SET-at-instant (dropped-partition edge) both read LATEST -> later steps; planScan honoring is only e2e-provable (offline unit tests cover routing/normalization/ handle-threading). Tests: HudiTimeTravelTest (9). fe-connector-hudi 85/85 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiConnectorMetadata.java | 71 +++++++ .../connector/hudi/HudiScanPlanProvider.java | 8 +- .../doris/connector/hudi/HudiTableHandle.java | 18 ++ .../connector/hudi/HudiTimeTravelTest.java | 198 ++++++++++++++++++ 4 files changed, 294 insertions(+), 1 deletion(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTimeTravelTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 90cc8d6009896e..302adf2e480a68 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -23,9 +23,11 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.connector.api.pushdown.ConnectorAnd; import org.apache.doris.connector.api.pushdown.ConnectorComparison; import org.apache.doris.connector.api.pushdown.ConnectorExpression; @@ -80,6 +82,12 @@ public class HudiConnectorMetadata implements ConnectorMetadata { // Catalog property gating the partition-name source (mirrors legacy HMSExternalTable.USE_HIVE_SYNC_PARTITION). private static final String USE_HIVE_SYNC_PARTITION = "use_hive_sync_partition"; + // Internal MVCC-snapshot property carrying the FOR TIME AS OF instant from resolveTimeTravel to applySnapshot + // (FE-internal transport, paimon scan-options model). It is NEVER serialized to BE: fe-core only feeds the + // snapshot to applySnapshot, which consumes this property and stamps HudiTableHandle.queryInstant. Not a + // BE-facing key. + private static final String HUDI_QUERY_INSTANT_PROPERTY = "hudi.query-instant"; + private final HmsClient hmsClient; private final Map properties; // Runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + TCCL pin (see R4). @@ -257,6 +265,69 @@ static Optional buildBeginQuerySnapshot(long instant) { return Optional.of(ConnectorMvccSnapshot.builder().snapshotId(instant).build()); } + /** + * Resolves an explicit {@code FOR TIME AS OF} / {@code FOR VERSION AS OF} into a pinned snapshot, owning all + * hudi-specific parsing (byte-faithful to legacy {@code HudiScanNode}). The generic seam + * ({@code PluginDrivenMvccExternalTable.loadSnapshot}) turns an empty result into a "not found" error and + * lets a thrown exception propagate verbatim. + * + *
      + *
    • {@code TIMESTAMP} ({@code FOR TIME AS OF}) — strip {@code [-: ]} from the value and pin it as a + * completed-timeline instant read BEFORE-OR-ON at scan time (legacy {@code HudiScanNode.java:211}). NO + * epoch-millis conversion, NO session time zone (unlike paimon), NO timeline-existence validation: + * legacy validates nothing and never errors on a well-formed {@code FOR TIME AS OF} (a too-early / + * future instant simply reads empty / latest via {@code getLatest*BeforeOrOn}). So ALWAYS return a + * non-empty pin — never empty, never throw for a not-found timestamp. {@code spec.isDigital()} is + * ignored (legacy strips regardless). The instant rides {@link #HUDI_QUERY_INSTANT_PROPERTY}; + * {@link #applySnapshot} stamps it onto the handle.
      • + *
    • {@code SNAPSHOT_ID} / {@code VERSION_REF} ({@code FOR VERSION AS OF}, numeric / named) — hudi + * rejects both with the byte-for-byte legacy message ({@code HudiScanNode.java:209}). It is THROWN (not + * {@code Optional.empty()}) so the exact wording reaches the user; an empty return would surface + * fe-core's wrong-domain "can't find snapshot" text.
      • + *
    • Other kinds ({@code TAG}/{@code BRANCH} — hudi has none; {@code INCREMENTAL} — a later + * {@code @incr} step) → {@code Optional.empty()} = the SPI default (no worse than not overriding).
      • + *
    + */ + @Override + public Optional resolveTimeTravel(ConnectorSession session, + ConnectorTableHandle handle, ConnectorTimeTravelSpec spec) { + switch (spec.getKind()) { + case TIMESTAMP: + return Optional.of(ConnectorMvccSnapshot.builder() + .property(HUDI_QUERY_INSTANT_PROPERTY, spec.getStringValue().replaceAll("[-: ]", "")) + .build()); + case SNAPSHOT_ID: + case VERSION_REF: + throw new DorisConnectorException( + "Hudi does not support `FOR VERSION AS OF`, please use `FOR TIME AS OF`"); + default: + return Optional.empty(); + } + } + + /** + * Threads a pinned {@code FOR TIME AS OF} instant onto the handle BEFORE planScan. Reads it from the + * snapshot's {@link #HUDI_QUERY_INSTANT_PROPERTY} (set by {@link #resolveTimeTravel}) and stamps it via + * {@code toBuilder().queryInstant(...)}, which PRESERVES any {@code prunedPartitionPaths} applyFilter set + * earlier (applyFilter runs before applySnapshot at scan time, so a rebuild-from-scratch would drop the + * pruning). For the query-begin latest pin ({@code beginQuerySnapshot} carries only a {@code snapshotId}, NO + * property) — and for a null snapshot — the handle is returned UNCHANGED, so a plain read stays + * byte-identical to today (planScan falls back to {@code timeline.lastInstant()}). Mirrors paimon's + * empty-properties / invalid-pin no-op. + */ + @Override + public ConnectorTableHandle applySnapshot(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { + if (snapshot == null) { + return handle; + } + String queryInstant = snapshot.getProperties().get(HUDI_QUERY_INSTANT_PROPERTY); + if (queryInstant == null) { + return handle; + } + return ((HudiTableHandle) handle).toBuilder().queryInstant(queryInstant).build(); + } + // ========== ConnectorTableOps (partitions) ========== /** diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 7025ea18844019..4044e12aaf97ad 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -136,7 +136,13 @@ public List planScan( LOG.info("No completed instants on timeline for {}, returning empty splits", basePath); return Collections.emptyList(); } - String queryInstant = lastInstant.get().requestedTime(); + // FOR TIME AS OF pins an explicit instant (applySnapshot stamped it on the handle); a plain read has + // none and reads the latest completed instant (byte-identical to before this step). This single local + // drives every downstream instant use: COW/MOR getLatest*BeforeOrOn file selection AND the MOR-JNI + // THudiFileDesc.instantTime, so FE file selection and the BE merge instant stay consistent. + String queryInstant = hudiHandle.getQueryInstant() != null + ? hudiHandle.getQueryInstant() + : lastInstant.get().requestedTime(); // Resolve column names and types for JNI reader List columnNames; diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java index 67d3c9d8c271cd..32729be2a8958b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java @@ -49,6 +49,11 @@ public class HudiTableHandle implements ConnectorTableHandle { // Set after applyFilter for partition pruning private final List prunedPartitionPaths; + // Set after applySnapshot for FOR TIME AS OF time travel: the completed-timeline instant the scan reads + // BEFORE-OR-ON (a String like "20240101120000", not a numeric snapshot id). Null = no time travel; the scan + // reads the latest completed instant (byte-identical to a plain snapshot read). + private final String queryInstant; + private HudiTableHandle(Builder builder) { this.dbName = builder.dbName; this.tableName = builder.tableName; @@ -63,6 +68,7 @@ private HudiTableHandle(Builder builder) { ? Collections.unmodifiableMap(builder.tableParameters) : Collections.emptyMap(); this.prunedPartitionPaths = builder.prunedPartitionPaths; + this.queryInstant = builder.queryInstant; } /** Legacy constructor for Phase 1 compatibility (metadata-only). */ @@ -106,6 +112,11 @@ public List getPrunedPartitionPaths() { return prunedPartitionPaths; } + /** The FOR TIME AS OF instant the scan reads before-or-on, or {@code null} for a latest read. */ + public String getQueryInstant() { + return queryInstant; + } + /** Returns a builder pre-populated with this handle's state, for creating modified copies. */ public Builder toBuilder() { Builder b = new Builder(dbName, tableName, basePath, hudiTableType); @@ -114,6 +125,7 @@ public Builder toBuilder() { b.partitionKeyNames = this.partitionKeyNames; b.tableParameters = this.tableParameters; b.prunedPartitionPaths = this.prunedPartitionPaths; + b.queryInstant = this.queryInstant; return b; } @@ -136,6 +148,7 @@ public static final class Builder { private List partitionKeyNames; private Map tableParameters; private List prunedPartitionPaths; + private String queryInstant; public Builder(String dbName, String tableName, String basePath, String hudiTableType) { this.dbName = dbName; @@ -169,6 +182,11 @@ public Builder prunedPartitionPaths(List val) { return this; } + public Builder queryInstant(String val) { + this.queryInstant = val; + return this; + } + public HudiTableHandle build() { return new HudiTableHandle(this); } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTimeTravelTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTimeTravelTest.java new file mode 100644 index 00000000000000..3f38545306858a --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiTimeTravelTest.java @@ -0,0 +1,198 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests the Hudi {@code FOR TIME AS OF} time-travel surface (resolveTimeTravel + applySnapshot + the + * queryInstant pin on the handle) added so a hudi-on-HMS table served post-flip through the GENERIC + * {@code PluginDrivenScanNode} path honors an explicit instant, byte-faithfully to legacy {@code HudiScanNode}. + * Each assertion pins WHY the behavior matters: + *
      + *
    • {@code FOR TIME AS OF} is a pure lexical {@code replaceAll("[-: ]","")} of the value (legacy + * {@code HudiScanNode.java:211}) — NO session time zone, NO epoch-millis conversion, NO timeline + * validation, and it NEVER errors on a well-formed value (a too-early/future instant reads empty/latest + * via before-or-on);
      • + *
    • {@code FOR VERSION AS OF} (numeric snapshot-id or named ref) is rejected by THROWING the byte-for-byte + * legacy message, so the exact wording reaches the user (an empty return would surface fe-core's + * wrong-domain "can't find snapshot" text);
      • + *
    • applySnapshot stamps the instant via {@code toBuilder()} so it PRESERVES applyFilter's + * prunedPartitionPaths (applyFilter runs first at scan time);
      • + *
    • the query-begin latest pin (empty properties) leaves the handle UNCHANGED, so a plain read stays + * byte-identical to before this step.
      • + *
    + * + *

    resolveTimeTravel and applySnapshot never touch the hmsClient or the metaClient executor, so the metadata + * is built with {@code null} collaborators: any accidental metadata access would NPE, which itself proves the + * fully-offline / no-validation contract. + */ +public class HudiTimeTravelTest { + + private static final String VERSION_REJECT_MESSAGE = + "Hudi does not support `FOR VERSION AS OF`, please use `FOR TIME AS OF`"; + private static final List YEAR_MONTH = Arrays.asList("year", "month"); + + // ── FOR TIME AS OF: normalize + pin ──────────────────────────────────────────────────────────────── + + @Test + public void resolveTimestampStripsSeparatorsAndPinsInstantOntoHandle() { + HudiConnectorMetadata md = metadata(); + // "2024-01-01 12:00:00" -> "20240101120000": dashes/colons/space stripped, nothing else. Drive the + // full path resolveTimeTravel -> applySnapshot so the private carrier property is exercised end-to-end. + ConnectorMvccSnapshot pin = resolveTimestamp(md, "2024-01-01 12:00:00"); + HudiTableHandle pinned = (HudiTableHandle) md.applySnapshot(null, partitioned(), pin); + Assertions.assertEquals("20240101120000", pinned.getQueryInstant(), + "FOR TIME AS OF must strip [-: ] and pin the value verbatim (legacy HudiScanNode:211) — " + + "no session-TZ shift, no epoch-millis conversion"); + } + + @Test + public void resolveTimestampIgnoresDigitalFlagAndDoesNoConversion() { + HudiConnectorMetadata md = metadata(); + // A digital (epoch-millis-looking) value has no [-: ] to strip, so it passes through VERBATIM — legacy + // hudi never treated FOR TIME AS OF as epoch-millis (unlike paimon); it is compared lexically before-or-on. + ConnectorMvccSnapshot pin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.timestamp("1704067200000", true)).orElseThrow(AssertionError::new); + HudiTableHandle pinned = (HudiTableHandle) md.applySnapshot(null, partitioned(), pin); + Assertions.assertEquals("1704067200000", pinned.getQueryInstant()); + } + + @Test + public void resolveTimestampIsPermissiveAndFullyOffline() { + // A too-early instant (before any commit) resolves to a NON-EMPTY pin, NOT Optional.empty(): legacy + // never validates FOR TIME AS OF against the timeline and never errors — before-or-on simply reads + // empty. The metadata's hmsClient + executor are null, so a present result also proves resolveTimeTravel + // performs ZERO metadata access (any timeline lookup would NPE here). + HudiConnectorMetadata md = metadata(); + Optional pin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.timestamp("1999-01-01 00:00:00", false)); + Assertions.assertTrue(pin.isPresent(), + "well-formed FOR TIME AS OF must always pin (permissive, legacy parity), never return empty"); + } + + // ── FOR VERSION AS OF: reject with the byte-for-byte legacy message ───────────────────────────────── + + @Test + public void resolveSnapshotIdRejectsWithByteForByteLegacyMessage() { + HudiConnectorMetadata md = metadata(); + // FOR VERSION AS OF arrives as SNAPSHOT_ID. Must THROW (not empty) so the exact legacy string + // reaches the user verbatim (it propagates as-is through loadSnapshot, which has no try/catch). + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> md.resolveTimeTravel(null, partitioned(), ConnectorTimeTravelSpec.snapshotId("123"))); + Assertions.assertEquals(VERSION_REJECT_MESSAGE, ex.getMessage()); + } + + @Test + public void resolveVersionRefRejectsWithByteForByteLegacyMessage() { + HudiConnectorMetadata md = metadata(); + // FOR VERSION AS OF '' arrives as VERSION_REF — hudi rejects it identically (it has no + // tags/branches), with the SAME message. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> md.resolveTimeTravel(null, partitioned(), ConnectorTimeTravelSpec.versionRef("v1.0"))); + Assertions.assertEquals(VERSION_REJECT_MESSAGE, ex.getMessage()); + } + + // ── applySnapshot: stamp preserving pruning / no-op for the latest pin ────────────────────────────── + + @Test + public void applySnapshotStampsInstantPreservingPrunedPartitions() { + HudiConnectorMetadata md = metadata(); + // applyFilter runs BEFORE applySnapshot at scan time, so a pruned handle must keep its pruning after the + // pin. Guards a rebuild-from-scratch mutation (which would silently turn a pruned scan into a full scan). + List pruned = Arrays.asList("year=2024/month=01", "year=2024/month=02"); + HudiTableHandle prunedHandle = partitioned().toBuilder().prunedPartitionPaths(pruned).build(); + ConnectorMvccSnapshot pin = resolveTimestamp(md, "2024-06-01 00:00:00"); + HudiTableHandle stamped = (HudiTableHandle) md.applySnapshot(null, prunedHandle, pin); + Assertions.assertEquals("20240601000000", stamped.getQueryInstant()); + Assertions.assertEquals(pruned, stamped.getPrunedPartitionPaths()); + } + + @Test + public void applySnapshotLeavesLatestPinUnchanged() { + HudiConnectorMetadata md = metadata(); + // The query-begin latest pin (beginQuerySnapshot output) carries ONLY a snapshotId, NO query-instant + // property. applySnapshot must return the handle UNCHANGED so planScan falls back to timeline.lastInstant() + // — the no-regression guard for every ordinary (non-time-travel) hudi read. + ConnectorMvccSnapshot latestPin = + HudiConnectorMetadata.buildBeginQuerySnapshot(20240101120000000L).orElseThrow(AssertionError::new); + HudiTableHandle base = partitioned(); + HudiTableHandle result = (HudiTableHandle) md.applySnapshot(null, base, latestPin); + Assertions.assertSame(base, result, "latest pin must not rebuild the handle"); + Assertions.assertNull(result.getQueryInstant()); + } + + @Test + public void applySnapshotWithNullSnapshotIsUnchanged() { + HudiConnectorMetadata md = metadata(); + HudiTableHandle base = partitioned(); + Assertions.assertSame(base, md.applySnapshot(null, base, null)); + } + + // ── handle field round-trip ───────────────────────────────────────────────────────────────────────── + + @Test + public void toBuilderRoundTripsQueryInstantAndPreservesEveryField() { + Map params = Collections.singletonMap("k", "v"); + HudiTableHandle original = new HudiTableHandle.Builder("db", "t", "s3://b/t", "MERGE_ON_READ") + .inputFormat("org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat") + .serdeLib("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe") + .partitionKeyNames(YEAR_MONTH) + .tableParameters(params) + .prunedPartitionPaths(Arrays.asList("year=2024/month=01")) + .queryInstant("20240101120000") + .build(); + HudiTableHandle copy = original.toBuilder().build(); + Assertions.assertEquals("20240101120000", copy.getQueryInstant()); + Assertions.assertEquals(original.getInputFormat(), copy.getInputFormat()); + Assertions.assertEquals(original.getSerdeLib(), copy.getSerdeLib()); + Assertions.assertEquals(original.getPartitionKeyNames(), copy.getPartitionKeyNames()); + Assertions.assertEquals(original.getTableParameters(), copy.getTableParameters()); + Assertions.assertEquals(original.getPrunedPartitionPaths(), copy.getPrunedPartitionPaths()); + // A fresh handle carries no pin (null), so a plain read stays on timeline.lastInstant(). + Assertions.assertNull(partitioned().getQueryInstant()); + } + + // ── helpers ──────────────────────────────────────────────────────────────────────────────────────── + + /** resolveTimeTravel/applySnapshot never touch hmsClient or the executor → null collaborators are safe. */ + private static HudiConnectorMetadata metadata() { + return new HudiConnectorMetadata(null, Collections.emptyMap(), null); + } + + private static HudiTableHandle partitioned() { + return new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(YEAR_MONTH).build(); + } + + private static ConnectorMvccSnapshot resolveTimestamp(HudiConnectorMetadata md, String value) { + return md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.timestamp(value, false)).orElseThrow(AssertionError::new); + } +} From ccaf103efb71a32969e16e375662f9d889234aa8 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 08:27:52 +0800 Subject: [PATCH 112/334] =?UTF-8?q?[doc](catalog)=20HMS=20cutover=20hudi?= =?UTF-8?q?=20HD-C2=20(FOR=20TIME=20AS=20OF=20time=20travel)=20DONE;=20nex?= =?UTF-8?q?t=20=3D=20Group=20C=E4=BD=99=20(HD-C3=20=E5=A2=9E=E9=87=8F?= =?UTF-8?q?=E8=AF=BB)=20+=20HD-C4/C5=20+=20Group=20D,=20then=20HD-B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design doc for the time-travel step + HANDOFF roll-forward. HD-C2 committed at cf4f0e1b756 (dormant). Two plan corrections recorded in the design (both surfaced by the 5-reader recon + reconciliation critic and confirmed by the 4-lens adversarial review, 0 confirmed defects): FOR TIME AS OF is byte-faithful-permissive (no timeline validation, never errors — the written plan's "validate + notFound" was a paimon-ism that would regress); FOR VERSION AS OF is rejected by THROWING the byte-for-byte legacy message (empty-return would surface fe-core's wrong-domain "can't find snapshot"). Documented deferrals (e2e-owed, not silent): schema-at-instant (HD-C4), partition-SET- at-instant dropped-partition edge, and planScan-honoring (only e2e-provable). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +- ...hudi-time-travel-step-design-2026-07-09.md | 155 ++++++++++++++++++ 2 files changed, 158 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/hudi-time-travel-step-design-2026-07-09.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 56346e10c1db58..c98a6fe3f34762 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。下一步 = Group C 余项无回退补齐（连接器侧，全部翻闸前）：时间旅行（HD-C2）、增量读〔最高风险 HD-C3〕、schema evolution（HD-C4/C5，用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。下一步 = Group C 余项无回退补齐（连接器侧，全部翻闸前）：增量读〔最高风险 HD-C3〕、schema evolution（HD-C4/C5，用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -88,7 +88,8 @@ - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 - - **下一步 = Group C 余项无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C2** 时间旅行（`resolveTimeTravel` TIMESTAMP + `FOR VERSION AS OF` 逐字拒 + pin-honoring planScan + HudiTableHandle 加 `queryInstant`）、**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 + - **下一步 = Group C 余项无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-time-travel-step-design-2026-07-09.md b/plan-doc/tasks/hudi-time-travel-step-design-2026-07-09.md new file mode 100644 index 00000000000000..570e32a94cf005 --- /dev/null +++ b/plan-doc/tasks/hudi-time-travel-step-design-2026-07-09.md @@ -0,0 +1,155 @@ +# Hudi time travel (FOR TIME AS OF) — connector-side design (HD-C2) + +Authoritative, code-grounded design for the hudi `FOR TIME AS OF` step. HEAD = branch `catalog-spi-11-hive` +(after HD-C1 `bbe6cfcd647`). Recon = `wf_e0b62364-d20` (5 HEAD-grounded readers + reconciliation critic, all +line anchors re-verified). This design is the mirror of the paimon time-travel template and the +byte-faithful reproduction of legacy `HudiScanNode` FOR TIME AS OF. **No fe-core changes.** + +--- + +## 0. Scope + +After the HMS-catalog cutover a hudi-on-HMS table is a generic `PluginDrivenMvccExternalTable` served by +`fe-connector-hudi` as a sibling. This step implements **`FOR TIME AS OF`** connector-side so a hudi table +served through the generic scan path reads at the pinned instant with **no regression** vs legacy +`HudiScanNode`. It also establishes the **query-instant pin spine on `HudiTableHandle`** that HD-C3 +(incremental read) reuses. + +Out of scope (documented deferrals): schema-at-instant (`FOR TIME AS OF` on a schema-*evolved* table still +reads the LATEST schema) → HD-C4; `@incr` incremental read → HD-C3; `@tag`/`@branch` → hudi has no tags/branches. + +## 1. The mechanism (how the pin reaches planScan) — verified + +The pin flows through the generic MVCC seam exactly like paimon. NO fe-core edit is needed; all four SPI hooks +already dispatch generically: + +1. fe-core `PluginDrivenMvccExternalTable.loadSnapshot` (`:347`) calls `metadata.resolveTimeTravel(session, + handle, spec)` for an explicit `FOR TIME AS OF` and stores the resolved `PluginDrivenMvccSnapshot` in the + statement context (`StatementContext.loadSnapshots`, keyed by table + version selector). The `pinnedHandle` + it computes at `:375` is **discarded** (used only for `getTableSchema`); the surviving carrier is the opaque + `ConnectorMvccSnapshot`. +2. At scan time `PluginDrivenScanNode.pinMvccSnapshot` (`:749`) does a **version-aware** lookup + (`MvccUtil.getSnapshotFromContext` keyed by *this scan's* `getQueryTableSnapshot()` selector), so a + `FOR TIME AS OF` query retrieves the **resolveTimeTravel-resolved** snapshot (my property), while a normal + read retrieves the query-begin latest-pin snapshot (empty properties). +3. `applyMvccSnapshotPin` (`:732`) unwraps `getConnectorSnapshot()` and calls + `metadata.applySnapshot(session, currentHandle, connectorSnapshot)`; the result becomes `currentHandle` + (`:768`), which is exactly the handle passed to `scanProvider.planScan(session, currentHandle, ...)` + (`:998-1000`). + +**Consequence:** the pin must live on the `ConnectorMvccSnapshot` (a String property) and be re-derived by +`HudiConnectorMetadata.applySnapshot` onto the `HudiTableHandle`; a value stamped only on `loadSnapshot`'s +handle would never reach `planScan`. `ConnectorMvccSnapshot.getProperties()` is **not** serialized to BE for a +plugin scan (fe-core only feeds it to `applySnapshot`); the instant reaches BE **only** via the per-range +`THudiFileDesc.instantTime` that `planScan` already stamps for MOR-JNI slices. So carrying the instant as a +handle field is both necessary and sufficient. + +## 2. Decisions + +### D1 — TIMESTAMP is permissive; NO timeline validation. **(plan correction)** +Legacy `HudiScanNode` never validates a `FOR TIME AS OF` value against the timeline: it only does +`value.replaceAll("[-: ]", "")` (`HudiScanNode.java:211`) and `getLatest{BaseFiles,MergedFileSlices}BeforeOrOn` +(`:419`/`:435`) — **before-or-on** file selection. A value after all commits reads the latest slice; a value +before the earliest commit reads **empty** (0 rows). Legacy **never errors** on a well-formed FOR TIME AS OF +(only the VERSION rejection). The written HD-C2 plan text ("validate the instant exists … empty ⇒ +notFoundMessage") is a paimon-ism that would ADD an error where legacy read empty = a regression. **Corrected:** +`resolveTimeTravel(TIMESTAMP)` normalizes and **always returns a non-empty pin** for a well-formed value; it +never returns empty and never throws for not-found. This is byte-faithful, keeps the method fully offline +unit-testable (no live metaClient, zero extra round-trip), and preserves before-or-on at `planScan` unchanged. + +### D2 — SNAPSHOT_ID / VERSION_REF fail loud by THROWING the byte-for-byte legacy message. **(mechanism correction)** +Legacy rejects `FOR VERSION AS OF` for hudi (`HudiScanNode.java:208-209`). fe-core's `toTimeTravelSpec` maps a +digital `FOR VERSION AS OF` → `SNAPSHOT_ID` and a non-digital one → `VERSION_REF` +(`PluginDrivenMvccExternalTable.java:404-407`); **both** must be rejected. Returning `Optional.empty()` would +surface fe-core's WRONG-DOMAIN `notFoundMessage` ("can't find snapshot by id" / "…by tag"). To surface the +exact legacy string the connector must **throw** a `DorisConnectorException` (unchecked; propagates verbatim — +no try/catch at `loadSnapshot :347-350`): + +> ``Hudi does not support `FOR VERSION AS OF`, please use `FOR TIME AS OF` `` (verbatim incl. both backtick pairs) + +### D3 — Digital flag ignored (legacy parity). +Legacy strips `[-: ]` regardless of `digital`; it does **no** epoch-millis conversion and **no** session +time-zone parse (unlike paimon). `resolveTimeTravel(TIMESTAMP)` ignores `spec.isDigital()` and just normalizes +the string. (A digital epoch-millis value passes through verbatim and reads before-or-on lexically — a faithful +legacy quirk.) + +### D4 — Pin carrier = MVCC property → handle field. +`resolveTimeTravel(TIMESTAMP)` stashes the normalized instant STRING in +`ConnectorMvccSnapshot.property(HUDI_QUERY_INSTANT_PROPERTY, normalized)` (FE-internal transport, paimon +scan-options model). `applySnapshot` reads that property and stamps +`hudiHandle.toBuilder().queryInstant(v).build()` — **`toBuilder` preserves `prunedPartitionPaths`** (applyFilter +runs before applySnapshot, so a pruned time-travel scan must not lose its pruning). Do **not** overload +`snapshotId` (opaque to fe-core for the TT pin; planScan needs the string; property-presence cleanly +discriminates explicit-TT from latest-pin). `snapshotId` left default — the TT pin never enters an MTMV +comparison (MTMV uses the query-begin pin). + +### D5 — applySnapshot is a no-op for the latest pin (no-regression guard). +`applySnapshot` is also invoked at scan time on the **query-begin** (latest) pin, which carries **empty** +properties (`beginQuerySnapshot` sets only `snapshotId`). When the `HUDI_QUERY_INSTANT_PROPERTY` is absent (or +snapshot null) `applySnapshot` returns the handle **unchanged** → `planScan` falls back to +`timeline.lastInstant()` → **byte-identical** to today for every ordinary read. Mirrors paimon's +`snapshotId<0 ⇒ return handle` guard. + +### D6 — planScan single switch point. +`HudiScanPlanProvider.java:139` changes from `String queryInstant = lastInstant.get().requestedTime();` to +`hudiHandle.getQueryInstant() != null ? hudiHandle.getQueryInstant() : lastInstant.get().requestedTime()`. +All three downstream consumers read this one local — COW `getLatestBaseFilesBeforeOrOn` (`:250`), MOR +`getLatestMergedFileSlicesBeforeOrOn` (`:279`), MOR-JNI `builder.instantTime(queryInstant)` (`:311` → +`THudiFileDesc.instantTime`) — so FE file selection and the BE merge instant stay consistent. The +empty-timeline early-return (`:135-138`) is kept: `lastInstant.get()` is only touched in the null-pin branch +(guaranteed present past `:138`), so no NPE; a pinned instant on a never-committed table still early-returns +empty (correct). Schema/columns (`:144-157`) already resolve LATEST with no instant arg → NOT a switch site +(schema-at-instant is HD-C4). + +### D7 — TAG / BRANCH / INCREMENTAL → `Optional.empty()` (unchanged SPI-default behavior). +The `resolveTimeTravel` override handles TIMESTAMP (pin) and SNAPSHOT_ID/VERSION_REF (throw); every other kind +returns `Optional.empty()` = the same result as not overriding, so no regression. `INCREMENTAL` is HD-C3 (which +will add that case); hudi has no `@tag`/`@branch`. Dormant until HD-B2, so no live behavior depends on this. + +## 3. Implementation checklist (ordered) + +1. **`HudiTableHandle.java`** — add `private final String queryInstant;` + `getQueryInstant()`; Builder field + + `queryInstant(String)` fluent setter; copy in the private ctor and in `toBuilder()` (mirroring + `prunedPartitionPaths`). **No** equals/hashCode (handle has none; keep reference identity, matching paimon's + intent to exclude the pin from identity). +2. **`HudiConnectorMetadata.java`** — override `resolveTimeTravel` (TIMESTAMP → pin property; SNAPSHOT_ID + + VERSION_REF → throw D2 message; default → empty) and `applySnapshot` (property present → stamp via + `toBuilder().queryInstant(...)`; else unchanged). Add the `HUDI_QUERY_INSTANT_PROPERTY` constant. Do **not** + override 3-arg `getTableSchema` (clean latest-schema deferral to HD-C4). +3. **`HudiScanPlanProvider.java:139`** — honor `handle.getQueryInstant()` (D6). + +## 4. Test plan (offline, same-loader; mirror `HudiConnectorPartitionListingTest`) + +- T1 `resolveTimeTravel(TIMESTAMP "2024-01-01 12:00:00")` → pin property == `"20240101120000"`; no TZ shift. +- T2 `resolveTimeTravel(SNAPSHOT_ID)` → throws `DorisConnectorException` with the byte-for-byte D2 message. +- T3 `resolveTimeTravel(VERSION_REF)` → throws the same byte-for-byte message. +- T4 `resolveTimeTravel(TIMESTAMP)` is permissive/offline: non-empty pin, ZERO metaClient interaction (D1). +- T5 `applySnapshot` with the property → `handle.getQueryInstant()==normalized` AND `prunedPartitionPaths` + preserved (set pruning on the input handle, assert it survives — guards toBuilder-not-rebuild). +- T6 `applySnapshot` with the empty-properties latest pin (`beginQuerySnapshot` output) → handle UNCHANGED, + `getQueryInstant()==null` (the no-regression guard). +- T7 `HudiTableHandle.toBuilder().queryInstant(x).build()` round-trips and preserves every other field. + +**e2e owed (flip-time, per `hms-iceberg-delegation-needs-e2e`):** `planScan` honoring the pin is not +offline-provable (it builds a live metaClient). A real `FOR TIME AS OF` regression test on actual COW + MOR +hudi tables (schema-stable = assert byte-faithful data pin; schema-evolved = assert/document the HD-C4 +latest-schema gap) is required before the flip. Offline tests prove routing/normalization/handle-threading only. + +## 5. Residuals +- **Schema-at-instant (HD-C4):** FOR TIME AS OF an OLD instant on a schema-EVOLVED table reads the LATEST + schema/columns (`getTableAvroSchema()` no-arg). No-op for non-evolved tables (legacy's fallback also ignores + the timestamp — it only reads schema-at-instant when `hoodie.schema.on.read.enable=true`). Real gap only for + schema-on-read evolved tables → HD-C4. +- **Partition-set-at-instant (deferred; surfaced by HD-C2 review, not a HD-C2 code defect):** HD-C2 pins the + DATA instant for file selection, but the partition *universe* is still resolved at LATEST + (`resolvePartitions` → `listAllPartitionPaths` = current partitions; `HudiConnectorMetadata.collectPartitions` + already flags "explicit time-travel (non-latest) partition listing is a later step"). Legacy pinned the + at-instant write-partition set (`getPartitionNamesBeforeOrEquals(queryInstant)`). Consequence: a partition + **dropped after** the pinned instant could be silently omitted from a `FOR TIME AS OF` result (row loss), + IF Hudi's `getAllPartitionPaths` tombstones it (unverified library semantic). Partitions ADDED after the + instant are harmless (before-or-on yields no files). This is a no-regression item for time-travel + completeness to close/validate before the flip (candidate: fold into the schema/partition-at-instant work + or a small follow-up), tracked here so the row-loss edge is **not silent**. Validated via the same COW/MOR + `FOR TIME AS OF` e2e (build a dropped-partition-then-travel-back fixture). +- **Pin spine reuse:** the `queryInstant` field + `planScan` honoring is the spine HD-C3 extends with the + incremental window/hoodie params. From a8680b78e2ea31a939842a65eed2a0857cf7b34b Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 09:32:16 +0800 Subject: [PATCH 113/334] [doc](catalog) HMS cutover hudi HD-C3 (incremental read): recon + FE-only neutral-SPI design + sign-off MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two code-grounded recon workflows (BE row-filter verdict + FE-only feasibility), both HEAD/BE-source verified. Signed decisions (2026-07-09): architecture = FE-only neutral SPI (connector supplies a synthetic _hoodie_commit_time window predicate via a new connector-agnostic ConnectorMetadata.getSyntheticScanPredicates default-empty SPI; fe-core injects it as an analysis-time LogicalFilter; BE applies it with its existing scan-conjunct machinery) = NO BE change, NO source-specific fe-core branch; D-C3-1 = expose all 5 _hoodie_* meta columns VISIBLE (legacy SELECT * parity). Corrects the stale 'no BE tree in checkout' claim. Planning only; next = implement INC-1..INC-5 (design doc §4). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 4 +- ...hudi-incremental-step-design-2026-07-09.md | 224 ++++++++++++++++++ 2 files changed, 226 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c98a6fe3f34762..1af6d52547aa9b 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -89,13 +89,13 @@ - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 - - **下一步 = Group C 余项无回退补齐（连接器侧，全部在 HD-B2/翻闸前）**：**HD-C3 增量读 @incr〔最高风险 §5.1：COW 行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 CheckPolicy、通用路径跳过且铁律禁 fe-core 复刻；COW 重写整 base 文件带旧 commit 行→纯文件级选择返窗口外行=静默错答，须先读 BE Hudi reader 判能否连接器/BE 行级过滤，本 checkout 无 BE 树〕**、HD-C4/C5 schema evolution（DV-006，**用户须定范围**：仅 schema-evolved 表回退，非三大无回退项）→ **Group D** HD-D1 写-拒安全网（hudi 只读，3 路路由的 hudi 臂显式拒写/procedure、绝不转发兄弟）→ **HD-B2** getTableHandle 分流 HUDI（武装 pivot，**最后一步**，须 HD-B1✅ + Group C + HD-D1 全完）。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C3 增量读 = recon + 设计 + 签字 DONE（本 session，尚未写实现码；权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**下一步 = 实现 5 步休眠拆解**（设计 §4）：INC-1 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（extend HD-C2 脊柱）→ INC-2 移植 COW/MOR/Empty IncrementalRelation（re-home `LocationPath`/`TableFormatType`/`HudiPartitionUtils`，修 `MORIncrementalRelation:92` 潜 bug）→ INC-3 增量 `planScan`（COW/MOR/RO-as-RT 单点/fallback；**不发** hoodie.* 到 BE）→ INC-4 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列 → INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 - **DV-005 模型错配已解**：`HudiConnectorProvider.getType()="hudi"` 保留但**只作兄弟类型串**（`createSiblingConnector("hudi",…)` 绕过 `SPI_READY_TYPES`）；**永不**把 `hudi` 加进 `SPI_READY_TYPES`/工厂 case（会重造 DV-005）。 - **有序子步（HD-A0..HD-D1，4 组；✅Group A（HD-A0–A5）全部收官，见上）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 - - **⛔ 最大风险 HD-C3（§5.1）**：COW 增量的行级 `_hoodie_commit_time > begin AND <= end` 谓词旧走 `CheckPolicy` 注入、通用路径**跳过**且铁律禁 fe-core 复刻；COW 更新重写整个 base 文件把旧 commit 行带进→纯文件级选择返窗口外行=**静默错答**。须连接器把行谓词推进 scan-range/thrift **或** BE reader 按 instant 窗口行级过滤——**须先读 BE Hudi reader 源码判定**（本 checkout 无 BE 树），可能在铁律下无 fe-core 解，且只能翻闸 e2e 验（straddling-base-file fixture）。paimon **非**先例。 + - **✅ 最大风险 HD-C3（§5.1）已解（本 session）**：COW 增量行级 `_hoodie_commit_time > begin AND <= end` 谓词的正确性落点已判定 = **FE-only 中立 SPI、零 BE 改**（详见上 HD-C3 条目 + 设计文档）。侦察坐实 BE 两读取路径不自窗口过滤但**已应用扫描 conjunct**，故连接器供中立 SPI 谓词、fe-core 分析期注入 `LogicalFilter` 引用可见 `_hoodie_commit_time` 列即可，BE 无改。**旧交接「本 checkout 无 BE 树/可能无 fe-core 解」已证伪**（BE 树在、已读 reader 源码）。straddling-base-file fixture 仍是翻闸 e2e 硬门（离线单测证 routing/窗口解析/句柄穿透，行正确性只 e2e 可证）。paimon **非**先例（其 incremental 是 snapshot-delta 非行过滤）。 - **翻闸硬门 B1–B6**（见 plan §4.2）+ 两非单测 harness（两加载器路由 fixture + 异构 HMS e2e，含 COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved·非 hive-style/转义分区·写-拒）。**schema 缺失 SIGABRT 方向**（§5.2）已由 critic 对 BE 核对**降级为窄回退非崩溃**（emit-nothing=安全基线走 BE 小写 name-match），但 BE 树不在本 checkout、须落 HD-C4 前复核。 - **其余翻闸前休眠候选**：**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 diff --git a/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md b/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md new file mode 100644 index 00000000000000..0579cd6533d3c0 --- /dev/null +++ b/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md @@ -0,0 +1,224 @@ +# Hudi `@incr` Incremental Read — connector step design (FE-only, no BE change) + +Authoritative, code-grounded design for the incremental-read no-regression gap closure (the highest-risk +Group-C item). HEAD = branch `catalog-spi-11-hive`. Two recon workflows back this doc (all HEAD-verified): +- `w2ififdgn` / `wf_4b001028-a0c` — BE row-filtering verdict (the §5.1 landmine), port inventory. +- `wp3k0gcvx` / `wf_05d58d84-143` — FE-only neutral-SPI feasibility verdict. +Full recon output: `/tmp/claude-1000/.../tasks/w2ififdgn.output` and `.../wp3k0gcvx.output` (this session). + +--- + +## 0. The problem (why incremental is the hard one) + +A COW update **rewrites the whole base file**, copying unchanged older-commit rows forward. So selecting the +base files touched in a `(begin, end]` commit window is **not enough** — those files also contain +out-of-window rows. Correct incremental read needs a **row-level** filter `_hoodie_commit_time > begin AND +<= end`. + +Legacy delivered that row filter with a **source-specific fe-core injection**: +- `CheckPolicy.java:83-88` — an analysis rule — wraps a `LogicalFilter` carrying + `LogicalHudiScan.generateIncrementalExpression()` (`LogicalHudiScan.java:129-148`), gated on + `relation instanceof LogicalHudiScan && getTable() instanceof HMSExternalTable`. +- The filter references the `_hoodie_commit_time` **meta column** (legacy hudi exposes all 5 `_hoodie_*` meta + columns as visible), which materializes it into the scan tuple; BE reads it and applies the conjunct via its + normal scan-conjunct machinery; a projection above drops it. + +Post-flip, a hudi-on-HMS table is a generic `PluginDrivenExternalTable` → `LogicalFileScan` (`BindRelation.java:654`), +which matches **neither** `instanceof LogicalHudiScan` **nor** `HMSExternalTable`. So the legacy injection is +structurally dead, and the **iron rule forbids** re-adding a `dlaType==HUDI` / `PluginDrivenExternalTable` +arm to `CheckPolicy`/fe-core. + +### §5.1 verdict — where row correctness can live (both recons, cross-verified by hand) + +BE does **not** window-filter on its own, on any path: +- Native reader (`be/src/format/table/hudi_reader.cpp:29-83`, `format_v2/table/hudi_reader.cpp`) reads the whole + base file; zero commit-time logic. `THudiFileDesc` (`PlanNodes.thrift:409-422`) has only a scalar + `instant_time`, no begin/end window. +- JNI reader forwards every `hoodie.*` scan property to the Java scanner, but + `HadoopHudiJniScanner.java:124-131` **drops** them (only lifts `hadoop_conf.*`), builds one + `HoodieRealtimeFileSplit` at `instant_time`, and returns all rows. +- **BUT** both BE paths **do** apply pushed scan conjuncts (native: `table_reader.h:303-313` expression filters; + JNI: `finalize_jni_block` `VExprContext::filter_block`, `format_v2/jni/hudi_jni_reader.cpp:159-162`). + +⇒ Row correctness = **a `_hoodie_commit_time` conjunct must reach the scan node**. There are exactly two ways: +- **(A) BE-synthesized** — new BE code reads the window from a carrier and synthesizes+applies the filter. Rejected: + touches C++/JNI in two reader trees + a thrift/protocol change; higher blast radius; **user does not want BE changes**. +- **(B) FE-only neutral SPI (CHOSEN)** — the connector supplies the predicate through a **neutral, connector-agnostic** + SPI; a generic fe-core analysis rule injects it as a `LogicalFilter` on a **hidden** `_hoodie_commit_time` column + it exposes. BE applies it with **zero BE change** (it already applies scan conjuncts). This is the "re-home + source-specific fe-core logic into a neutral SPI" pattern used throughout this migration, and mirrors Trino's + unenforced/residual-predicate model. + +**Feasibility of (B): CONFIRMED (recon `wp3k0gcvx`).** The only non-obvious constraint: inject at the **logical +(analysis) layer**, NOT the physical scan node. A scan node cannot mint a slot for an unprojected column +(`generateTupleDesc` only loops `scan.getOutput()`, `PhysicalPlanTranslator.java:896-898` — no `addSlot`); but an +analysis-time `LogicalFilter` referencing a hidden column **does** force materialization (legacy proof). The window +is available at analysis time — `@incr` bounds ride `scanParams` on the plugin `LogicalFileScan` at bind time +(`BindRelation.java:654`), and the MVCC snapshot is resolved during analysis (`StatementContext.loadSnapshots`, +retrieved later by `PluginDrivenScanNode.pinMvccSnapshot`, `:755-759`). + +--- + +## 1. The neutral SPI + fe-core mechanism (the row-correctness core) + +### 1.1 Expose the 5 `_hoodie_*` meta columns as VISIBLE (connector-side, generic) — SIGNED D-C3-1 +Per D-C3-1, the connector exposes all 5 hudi meta columns (`_hoodie_commit_time`, `_hoodie_commit_seqno`, +`_hoodie_record_key`, `_hoodie_partition_path`, `_hoodie_file_name`) as **visible** STRING columns in the hudi +schema (port legacy `getTableAvroSchema(true)` / `HMSExternalTable.initHudiSchema`). Being visible, they are in +the plugin scan output (`getFullSchema()` → `LogicalFileScan.computePluginDrivenOutput():214-224`) and bindable by +name — so `_hoodie_commit_time` is materializable by the injected filter (§1.4) with **no** hidden-column path +needed. `SELECT *`/`DESCRIBE` now match legacy. (The `ConnectorColumn.invisible()` / +`ConnectorColumnConverter.java:81-82` hidden path — iceberg v3 row-lineage's mechanism — is available but NOT +used here given D-C3-1.) + +✅ **SIGNED D-C3-1 (visibility parity, user 2026-07-09) = (ii) all 5 `_hoodie_*` meta columns VISIBLE = full +legacy `SELECT *`/`DESCRIBE` parity** (the no-regression bar is legacy `HudiScanNode`). The connector's hudi +schema must expose `_hoodie_commit_time`, `_hoodie_commit_seqno`, `_hoodie_record_key`, +`_hoodie_partition_path`, `_hoodie_file_name` as visible columns (port legacy `HMSExternalTable.initHudiSchema` +meta-column set / `getTableAvroSchema(true)` — DV-008 gap-2). This also makes `_hoodie_commit_time` naturally +materializable for the incremental filter (no separate hidden-column path needed) and restores legacy +`SELECT _hoodie_commit_time`. Changes the current (dormant) connector's `SELECT *` to add these 5 columns — no +live regression (hms not yet in `SPI_READY_TYPES`). Because the columns are visible, §1.3/§1.4's slot binding +resolves them the same way; the `invisible()`/hidden-column path is NOT needed. + +### 1.2 Neutral SPI: connector supplies a synthetic scan predicate +Add a neutral default to `ConnectorMetadata` (parent-first, default = none; fe-core NEVER discriminates by source): + +``` +// Connector may require an extra scan-level predicate (e.g. a CDC/incremental commit-time window) that the +// engine must apply. Default = empty. iceberg/paimon/jdbc/... inherit empty → plans byte-identical. +default List getSyntheticScanPredicates( + ConnectorSession session, ConnectorTableHandle handle, TableScanParams scanParams) { + return List.of(); +} +``` +- Returns **`ConnectorExpression`** (the existing pushdown expression type — `ConnectorColumnRef` + `ConnectorLiteral.ofString` + + `ConnectorComparison{GT,LE}` + `ConnectorAnd` already express `_hoodie_commit_time > 'begin' AND <= 'end'`, + recon-confirmed). This keeps the SPI **general and Trino-aligned** (a connector residual predicate), not a + hudi-shaped struct. +- Hudi connector resolves `(begin, end]` from `scanParams` (resolving an omitted `endTime` via its own + metaClient, exactly as legacy `withScanParams` does — inside `HudiMetaClientExecutor.execute` for TCCL/auth), + and returns the range `ConnectorExpression` over `_hoodie_commit_time`. Non-incremental scans → empty. +- **Simpler fallback if the reverse converter proves heavy:** a `SyntheticScanPredicate{colName, lowerExclusive, + upperInclusive}` struct + fe-core builds native Nereids exprs directly (no converter). Recorded as fallback; + primary is the general `ConnectorExpression` form the user asked for. + +### 1.3 fe-core: reverse `ConnectorExpression → Nereids Expression` (bounded, new) +No reverse converter exists (only forward `Expr/Nereids → ConnectorExpression`: +`ExprToConnectorExpressionConverter`, `NereidsToConnectorExpressionConverter`). Build a bounded reverse for the +shape {ColumnRef, StringLiteral, GT/LE/…comparisons, And/Or} → Nereids `SlotReference`/`StringLiteral`/ +`GreaterThan`/`LessThanEqual`/`And`, binding `ConnectorColumnRef.name` → the scan output's `SlotReference` by name +(the scan's `getLogicalProperties().getOutput()`). ~150–250 LOC, connector-agnostic, reusable. + +### 1.4 fe-core: neutral analysis rule (the injection locus) +A neutral rule (either a new analysis rule on `LogicalFileScan`, or a neutral hook **replacing** the +`instanceof LogicalHudiScan` branch at `CheckPolicy.java:83-88`) that, for any plugin `LogicalFileScan` with +`scanParams`: +1. calls `connector.getSyntheticScanPredicates(session, handle, scanParams)`; +2. converts each `ConnectorExpression` → bound Nereids `Expression` (§1.3), resolving column refs against the + scan output; **no-op if the named slot is absent** (legacy `timeField==null` short-circuit, + `LogicalHudiScan.java:140-142`); +3. wraps a `LogicalFilter` over the returned conjuncts. + +iceberg/paimon return empty → **no filter added, plan byte-identical**. This is the iron-rule guarantee: fe-core +calls the SPI unconditionally; only hudi answers. Keep both bounds **string-typed** (`StringLiteral`) — a numeric +coercion would silently change lexicographic instant comparison. + +**Dormancy:** pre-flip, legacy hudi still binds to `LogicalHudiScan` and the OLD `CheckPolicy` branch serves it; +the new neutral SPI returns empty for every live plugin connector (iceberg/paimon), so the rule adds nothing. +Do NOT delete the old `CheckPolicy:83-88` branch until the flip (it is live for legacy hudi). + +### 1.5 Flip-time (NOT now) +Remove the incremental/time-travel `throw` in `visitPhysicalHudiScan` (`PhysicalPlanTranslator.java:909-912`) — it +is dead for the plugin path anyway — and delete the legacy `CheckPolicy:83-88` hudi branch + `LogicalHudiScan` + +`datasource/hudi/source/*` alongside the rest of the legacy hudi deletion. + +--- + +## 2. Connector-side incremental read (the split-selection port) + +Independent of the row-filter mechanism; this is the "which files" half, all in `fe-connector-hudi`, dormant. + +### 2.1 `resolveTimeTravel(INCREMENTAL)` + `applySnapshot` (extend HD-C2 spine) +- `resolveTimeTravel` currently returns `Optional.empty()` for `INCREMENTAL` (`HudiConnectorMetadata.java:303-304`). + Add the `INCREMENTAL` case: parse `spec.getIncrementalParams()` (begin/end), resolve the window against the + completed timeline **inside `HudiMetaClientExecutor.execute`** (TCCL-pin+auth), and return a + `ConnectorMvccSnapshot` that (a) pins schema/freshness at **LATEST** (mirror paimon; empty table → −1) and + (b) carries begin/end + a mode marker via new internal property keys (mirror `HUDI_QUERY_INSTANT_PROPERTY`, + `:85-89`), e.g. `HUDI_BEGIN_INSTANT_PROPERTY` / `HUDI_END_INSTANT_PROPERTY` / `HUDI_INCREMENTAL_MODE`. + **Never return empty for a valid window** — `PluginDrivenMvccExternalTable.loadSnapshot` fail-loud + `notFoundMessage` has no INCREMENTAL arm (`:348-350`), so empty → wrong-domain error. +- `applySnapshot` reads those properties and stamps `beginInstant`/`endInstant`(+mode) onto `HudiTableHandle` + via `toBuilder()` (preserve `prunedPartitionPaths`, as HD-C2 did). Absent → handle unchanged (snapshot path + byte-identical). + +### 2.2 Port the IncrementalRelation family into the connector +| Legacy (`fe-core datasource/hudi/source/`) | Action | Re-home | +|---|---|---| +| `IncrementalRelation` (interface) | Port as connector-internal interface | keep shape (`collectSplits`/`collectFileSlices`/`getStartTs`/`getEndTs`/`getHoodieParams`) | +| `COWIncrementalRelation` (`:74-240`) | Port | `LocationPath`, `TableFormatType.HUDI`, `HudiPartitionUtils.parsePartitionValues`, `spi.Split`/`HudiSplit`→`HudiScanRange` | +| `MORIncrementalRelation` (`:64-205`) | Port; **fix** the `:92` latent bug (`LATEST_TIME.equals(latestTime)` should test `endTimestamp`; COW does it right at `:98`) | `spi.Split` | +| `EmptyIncrementalRelation` (`:29-71`) | Port; short-circuit to empty split list (its `incr.operation`/`includeStartTime` keys are inert now — BE isn't the row filter) | — | +| `LogicalHudiScan.withScanParams` driver (`:232-281`) | **Re-implement** (HMSExternalTable-coupled) inside `resolveTimeTravel` using the connector's own metaClient/storage plumbing (HD-C1/C2) | — | +| `HudiScanNode.getIncrementalSplits` gating (`:381-403,468-484,556-568`) | **Re-implement** in `planScan` onto existing `collectCowSplits`/`collectMorSplits` | — | +| `CheckPolicy`/`generateIncrementalExpression` row filter | **Do NOT port to fe-core** — becomes the neutral SPI of §1 | — | + +### 2.3 `planScan` incremental branch +When the handle carries a window: branch to incremental split enumeration — COW base files over the range +(native OK), MOR merged file-slices at `endTs` (JNI, `THudiFileDesc.instant_time=endTs`). Then: +- **RO-as-RT quirk (single connector locus):** a COW `_ro` table with serde `hoodie.query.as.ro.table=true` + (name endsWith `_ro`) → treat as MOR for incremental. Legacy duplicates this in two places + (`LogicalHudiScan.java:251-260` + `HudiScanNode.java:186-199`); collapse to ONE check in + `HudiScanPlanProvider` split-selection. +- **`fallbackFullTableScan` degrade:** check `shouldFullTableScan()` **before** calling the ported + `collectSplits`/`collectFileSlices` (which throw when `fullTableScan`); on true, fall through to the normal + latest-snapshot partition scan (not error). +- **Scan-node properties:** with the FE-filter approach, BE ignores hoodie incremental params, so **do NOT** emit + `hoodie.datasource.read.begin/end.instanttime` (the connector selects files; fe-core injects the row filter). + This is simpler than the BE-carrier design. +- `@incr` lists **LATEST** partitions + **LATEST** schema (do NOT pin schema for incremental). + +--- + +## 3. Iron-rule & correctness landmines +1. **No hudi branch in fe-core.** The neutral rule keys on the SPI returning empty for non-hudi; the only real + relocation is `CheckPolicy:83-88` → neutral SPI (flip-time). Column exposure via `invisible()` is already clean. +2. **Empty-window / slot-absent over-read.** SPI returns empty for non-incremental scans; the rule no-ops if the + named slot is absent (legacy `:140-142`). +3. **String typing.** Keep bounds `StringLiteral`; lexicographic instant compare (Hudi instants are fixed-width sortable). +4. **MOR JNI required_fields.** The hidden `_hoodie_commit_time` materializes as a REGULAR file slot (via the + filter reference) → flows through `FileQueryScanNode` required-slots automatically. Assert in MOR e2e. +5. **Materialize-then-discard.** Projection above the scan drops the hidden column when unselected (matches legacy). +6. **TCCL wrapping.** Any new metaClient/timeline touch from `resolveTimeTravel`/the SPI runs on the metadata + thread → wrap in `HudiMetaClientExecutor.execute`. `planScan` is already pinned. +7. **MOR `endTs` bug** (`MORIncrementalRelation.java:92`) — fix on port. + +--- + +## 4. Proposed dormant-commit decomposition (each independently committable + same-loader unit test) +- **INC-1 — handle pin spine + `resolveTimeTravel(INCREMENTAL)` + `applySnapshot`** (connector, dormant). Window + resolution → LATEST-pinned snapshot carrying begin/end; handle stamped. *Test:* non-empty snapshot with window + props; handle carries window + preserves pruning; never empty for a valid window. +- **INC-2 — port IncrementalRelation family** (connector, dormant). COW/MOR/Empty + interface; re-home helpers; + fix `MOR:92`. *Test:* start/end resolution, commit-range selection, Empty path, throw-on-fullTableScan. +- **INC-3 — incremental `planScan`** (connector, dormant). COW/MOR/RO-as-RT/fallback branch; no hoodie params + emitted. *Test:* split set + fallback degrades to snapshot path (not throw) + RO-as-RT routes MOR. +- **INC-4 — neutral synthetic-predicate SPI + fe-core rule + reverse converter + hidden `_hoodie_commit_time`** + (fe-core + connector, dormant-in-effect). *Test:* SPI empty for iceberg/paimon (plan unchanged); hudi returns + range expr; rule wraps `LogicalFilter`; converter builds bound Nereids exprs; hidden column not in `SELECT *`. +- **INC-5 — flip (live)** — remove the `visitPhysicalHudiScan` incremental throw; delete legacy `CheckPolicy` + hudi branch + legacy hudi source. *Closing e2e:* §5 below. + +## 5. Flip-time e2e (the literal correctness gate; per memory `hms-iceberg-delegation-needs-e2e`) +COW + MOR tables, each ≥3 commits; `SELECT ... incr('beginTime'=c1,'endTime'=c2)` returns EXACTLY the rows written +in `(c1, c2]` — **including the linchpin case where a base file touched in the window also carries forward +older-commit rows** (proves the `_hoodie_commit_time` filter, not just file selection). Plus: RO-as-RT table, +`endTime='latest'` sentinel, empty-timeline (Empty relation), fallback-full-table-scan (archived instant). Assert +byte-parity vs the legacy HMS hudi catalog on identical data. + +## 6. Decisions +- **D-C3-1 (visibility parity) — SIGNED (user 2026-07-09) = all 5 `_hoodie_*` meta columns VISIBLE (legacy parity).** (§1.1) +- **SPI return shape** — general `ConnectorExpression` (recommended, Trino-aligned) vs simple string-bounds struct + (cheaper fallback). Implementer's call unless the reverse converter proves heavier than ~250 LOC. (§1.2) +- **Overall architecture — SIGNED (user 2026-07-09) = FE-only neutral SPI, NO BE change** (user requirement; + recon `wp3k0gcvx` confirmed feasible + simpler/safer than BE). (§0/§1) From fc5faa5606a32eea7c3d608c5dacb6e32ed7cd90 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 10:32:38 +0800 Subject: [PATCH 114/334] [feat](catalog) HMS cutover hudi HD-C3 INC-1: @incr window resolution spine (dormant) Connector-only, dormant (hms not in SPI_READY_TYPES). First of five INC steps for hudi @incr incremental read; extends the HD-C2 FOR TIME AS OF spine. - resolveTimeTravel(INCREMENTAL): resolve the (begin, end] completed-timeline window in ONE connector locus (consolidating legacy's per-relation COW/MOR resolution). begin required (byte-faithful fail-loud message), "earliest"->"000", end default-to-latest, "latest"->latest completed instant tested on the RESOLVED end value (legacy COW form, inherently avoiding the dead-code MOR:92 bug); empty timeline -> (000,000] without the begin-required check. Returns a NON-EMPTY property-only pin (snapshotId/schemaId inert: fe-core's INCREMENTAL loadSnapshot branch lists latest partitions + latest schema and reads only the window props). - applySnapshot: stamp begin/endInstant onto HudiTableHandle (mutually exclusive with the FOR TIME AS OF queryInstant carrier), preserving prunedPartitionPaths. - HudiTableHandle: new beginInstant/endInstant fields + Builder/toBuilder. - HudiScanPlanProvider: extract latestCompletedInstantTime(String) from latestCompletedInstant (byte-identical delegate). The single getCommitsAndCompactionTimeline resolves per table type to exactly the timeline legacy uses per type (verified vs hudi-common 1.0.2 bytecode: COW = getCommitAndReplaceTimeline = legacy COW's metaClient.getCommitTimeline, incl. replacecommit/clustering; MOR = getWriteTimeline) => byte-parity for both. Deferred by design (documented, fail-loud, not silent): populateMetaFields fail-loud + hollow-commit USE_TRANSITION_TIME variant -> INC-2 (relation-family port); raw hoodie.datasource.read.*.instanttime window keys -> INC-4. Adversarial review (4 dims, each finding refuted) + hudi-1.0.2 bytecode verification: 0 confirmed defects (blocker on COW timeline parity refuted). fe-connector-hudi 96 tests (HudiIncrementalTest +11) + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiConnectorMetadata.java | 120 ++++++++- .../connector/hudi/HudiScanPlanProvider.java | 33 ++- .../doris/connector/hudi/HudiTableHandle.java | 38 +++ .../connector/hudi/HudiIncrementalTest.java | 252 ++++++++++++++++++ 4 files changed, 427 insertions(+), 16 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 302adf2e480a68..0bbfb0fc487a4f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -88,6 +88,25 @@ public class HudiConnectorMetadata implements ConnectorMetadata { // BE-facing key. private static final String HUDI_QUERY_INSTANT_PROPERTY = "hudi.query-instant"; + // Internal MVCC-snapshot properties carrying the resolved @incr (begin, end] window from resolveTimeTravel to + // applySnapshot (same FE-internal transport as HUDI_QUERY_INSTANT_PROPERTY). NEVER serialized to BE: fe-core's + // INCREMENTAL loadSnapshot branch lists the LATEST partitions + LATEST schema itself and only carries these on + // the pin; applySnapshot consumes them to stamp HudiTableHandle.begin/endInstant. Not BE-facing keys. + private static final String HUDI_INCREMENTAL_BEGIN_PROPERTY = "hudi.incremental-begin"; + private static final String HUDI_INCREMENTAL_END_PROPERTY = "hudi.incremental-end"; + + // The @incr window param keys fe-core threads verbatim via getIncrementalParams() (byte-faithful to legacy + // LogicalHudiScan.withScanParams, which reads "beginTime"/"endTime" from the scan params). + private static final String INCR_BEGIN_TIME_KEY = "beginTime"; + private static final String INCR_END_TIME_KEY = "endTime"; + + // Window sentinels (byte-faithful to legacy IncrementalRelation.EARLIEST_TIME / LATEST_TIME; "000" is the + // earliest-resolved / empty-timeline bound). fe-core's legacy IncrementalRelation constants live in fe-core + // and must not be imported across the plugin boundary, so they are re-declared here. + private static final String INCR_EARLIEST_SENTINEL = "earliest"; + private static final String INCR_LATEST_SENTINEL = "latest"; + private static final String INCR_ZERO_INSTANT = "000"; + private final HmsClient hmsClient; private final Map properties; // Runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + TCCL pin (see R4). @@ -284,8 +303,12 @@ static Optional buildBeginQuerySnapshot(long instant) { * rejects both with the byte-for-byte legacy message ({@code HudiScanNode.java:209}). It is THROWN (not * {@code Optional.empty()}) so the exact wording reaches the user; an empty return would surface * fe-core's wrong-domain "can't find snapshot" text. - *

  • Other kinds ({@code TAG}/{@code BRANCH} — hudi has none; {@code INCREMENTAL} — a later - * {@code @incr} step) → {@code Optional.empty()} = the SPI default (no worse than not overriding).
    • + *
  • {@code INCREMENTAL} ({@code @incr(...)}) — see {@link #resolveIncremental}: resolves the + * {@code (begin, end]} window and returns a NON-EMPTY property-only pin. NEVER empty for a valid + * window (the generic {@code loadSnapshot} fail-loud has no INCREMENTAL arm, so empty would surface a + * wrong-domain "can't resolve time travel" message).
    • + *
  • Other kinds ({@code TAG}/{@code BRANCH} — hudi has none) → {@code Optional.empty()} = the + * SPI default (no worse than not overriding).
    • * */ @Override @@ -300,20 +323,85 @@ public Optional resolveTimeTravel(ConnectorSession sessio case VERSION_REF: throw new DorisConnectorException( "Hudi does not support `FOR VERSION AS OF`, please use `FOR TIME AS OF`"); + case INCREMENTAL: + return Optional.of(resolveIncremental((HudiTableHandle) handle, spec.getIncrementalParams())); default: return Optional.empty(); } } /** - * Threads a pinned {@code FOR TIME AS OF} instant onto the handle BEFORE planScan. Reads it from the - * snapshot's {@link #HUDI_QUERY_INSTANT_PROPERTY} (set by {@link #resolveTimeTravel}) and stamps it via - * {@code toBuilder().queryInstant(...)}, which PRESERVES any {@code prunedPartitionPaths} applyFilter set - * earlier (applyFilter runs before applySnapshot at scan time, so a rebuild-from-scratch would drop the - * pruning). For the query-begin latest pin ({@code beginQuerySnapshot} carries only a {@code snapshotId}, NO + * Resolves an {@code @incr(...)} incremental read into a NON-EMPTY property-only pin carrying the resolved + * {@code (begin, end]} completed-timeline window. Consolidates legacy's per-relation window resolution + * (spread byte-identically across {@code COWIncrementalRelation}/{@code MORIncrementalRelation} constructors) + * into ONE connector locus, so the resolved bounds are on the handle for both file selection (a later step) + * and the synthetic {@code _hoodie_commit_time} filter. Runs the single metaClient touch (latest completed + * instant) under {@link HudiMetaClientExecutor#execute} (TCCL pin + plugin UGI {@code doAs}). + * + *
      + *
    • Empty completed timeline → the {@code (000, 000]} window — legacy {@code withScanParams} + * short-circuits to {@code EmptyIncrementalRelation} before the begin-required check, so a + * missing {@code beginTime} is NOT an error here; the window selects nothing.
      • + *
    • {@code beginTime} is required (legacy fail-loud, byte-for-byte message); {@code "earliest"} → + * {@code "000"}.
      • + *
    • {@code endTime} defaults to the latest completed instant; {@code "latest"} → the latest completed + * instant. The sentinel test is on the RESOLVED end value (legacy COW form, + * {@code COWIncrementalRelation:98}); the single locus inherently avoids the dead-code MOR bug + * ({@code MORIncrementalRelation:92}, which tested {@code latestTime} and so never fired).
      • + *
    + * + *

    The pin is property-only: {@code snapshotId}/{@code schemaId} are inert because fe-core's INCREMENTAL + * {@code loadSnapshot} branch lists the LATEST partitions + LATEST schema itself and reads only these window + * properties. COW/MOR/RO-as-RT file selection is deferred to {@code planScan}; {@link #applySnapshot} stamps + * the window onto the handle.

    + */ + private ConnectorMvccSnapshot resolveIncremental(HudiTableHandle handle, Map params) { + Optional latestTime = metaClientExecutor.execute(() -> + HudiScanPlanProvider.latestCompletedInstantTime( + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), handle.getBasePath()))); + String begin; + String end; + if (!latestTime.isPresent()) { + // Empty completed timeline: legacy short-circuits to EmptyIncrementalRelation (begin = end = "000") + // WITHOUT the begin-required check. + begin = INCR_ZERO_INSTANT; + end = INCR_ZERO_INSTANT; + } else { + begin = params.get(INCR_BEGIN_TIME_KEY); + if (begin == null) { + throw new DorisConnectorException("Specify the begin instant time to pull from using " + + "option hoodie.datasource.read.begin.instanttime"); + } + if (INCR_EARLIEST_SENTINEL.equals(begin)) { + begin = INCR_ZERO_INSTANT; + } + end = params.getOrDefault(INCR_END_TIME_KEY, latestTime.get()); + if (INCR_LATEST_SENTINEL.equals(end)) { + end = latestTime.get(); + } + } + return ConnectorMvccSnapshot.builder() + .property(HUDI_INCREMENTAL_BEGIN_PROPERTY, begin) + .property(HUDI_INCREMENTAL_END_PROPERTY, end) + .build(); + } + + /** + * Threads a resolved pin onto the handle BEFORE planScan, reading the FE-internal carrier properties set by + * {@link #resolveTimeTravel} and stamping via {@code toBuilder()}, which PRESERVES any + * {@code prunedPartitionPaths} applyFilter set earlier (applyFilter runs before applySnapshot at scan time, + * so a rebuild-from-scratch would drop the pruning). Two mutually exclusive carriers: + * + *
      + *
    • {@link #HUDI_QUERY_INSTANT_PROPERTY} ({@code FOR TIME AS OF}) → stamp {@code queryInstant}.
      • + *
    • {@link #HUDI_INCREMENTAL_BEGIN_PROPERTY}/{@link #HUDI_INCREMENTAL_END_PROPERTY} ({@code @incr}) + * → stamp {@code begin/endInstant}.
      • + *
    + * + *

    For the query-begin latest pin ({@code beginQuerySnapshot} carries only a {@code snapshotId}, NO * property) — and for a null snapshot — the handle is returned UNCHANGED, so a plain read stays * byte-identical to today (planScan falls back to {@code timeline.lastInstant()}). Mirrors paimon's - * empty-properties / invalid-pin no-op. + * empty-properties / invalid-pin no-op.

    */ @Override public ConnectorTableHandle applySnapshot(ConnectorSession session, @@ -321,11 +409,19 @@ public ConnectorTableHandle applySnapshot(ConnectorSession session, if (snapshot == null) { return handle; } - String queryInstant = snapshot.getProperties().get(HUDI_QUERY_INSTANT_PROPERTY); - if (queryInstant == null) { - return handle; + Map properties = snapshot.getProperties(); + String queryInstant = properties.get(HUDI_QUERY_INSTANT_PROPERTY); + if (queryInstant != null) { + return ((HudiTableHandle) handle).toBuilder().queryInstant(queryInstant).build(); + } + String beginInstant = properties.get(HUDI_INCREMENTAL_BEGIN_PROPERTY); + if (beginInstant != null) { + return ((HudiTableHandle) handle).toBuilder() + .beginInstant(beginInstant) + .endInstant(properties.get(HUDI_INCREMENTAL_END_PROPERTY)) + .build(); } - return ((HudiTableHandle) handle).toBuilder().queryInstant(queryInstant).build(); + return handle; } // ========== ConnectorTableOps (partitions) ========== diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 4044e12aaf97ad..3986b9523b06a2 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -367,16 +367,41 @@ static HoodieTableMetaClient buildMetaClient(Configuration conf, String basePath .build(); } + /** + * Returns the LATEST completed instant as its raw {@code requestedTime} String (e.g. + * {@code yyyyMMddHHmmssSSS}, compared lexicographically), or {@code Optional.empty()} when the timeline has + * no completed instants. Reads the same {@code getCommitsAndCompactionTimeline().filterCompletedInstants()} + * as {@link #latestCompletedInstant} / {@link #planScan}. + * + *

    This ONE shared helper is byte-parity with legacy COW/MOR incremental {@code latestTime} for BOTH table + * types, because {@code metaClient.getCommitsAndCompactionTimeline()} resolves per table type to exactly the + * timeline legacy uses per type (verified against hudi-common 1.0.2 bytecode): + *

      + *
    • COW → {@code getActiveTimeline().getCommitAndReplaceTimeline()} = {@code {commit, replacecommit, + * clustering}} — identical to what legacy COW's {@code metaClient.getCommitTimeline()} returns + * (that metaClient method ALSO delegates to {@code getCommitAndReplaceTimeline()}, so it is NOT + * commit-only; it includes the replacecommit/clustering instants COW produces via INSERT OVERWRITE / + * clustering).
      • + *
    • MOR → {@code getActiveTimeline().getWriteTimeline()} — identical to legacy MOR's + * {@code metaClient.getCommitsAndCompactionTimeline()}.
      • + *
    + * Both legacy and this helper take {@code lastInstant().requestedTime()} under the default hollow-commit + * policy; the {@code USE_TRANSITION_TIME} completion-time variant is a documented deferral (see the + * incremental-read step design). + */ + static Optional latestCompletedInstantTime(HoodieTableMetaClient metaClient) { + return metaClient.getCommitsAndCompactionTimeline() + .filterCompletedInstants().lastInstant().toJavaOptional() + .map(HoodieInstant::requestedTime); + } + /** * Returns the LATEST completed instant as a numeric long ({@code yyyyMMddHHmmssSSS}), or {@code 0L} when * the timeline has none. Byte-faithful port of legacy {@code HudiUtils.getLastTimeStamp} and the same * timeline {@link #planScan} reads at query time — so the MVCC pin and the scan take the identical instant. */ static long latestCompletedInstant(HoodieTableMetaClient metaClient) { - Optional requestedTime = metaClient.getCommitsAndCompactionTimeline() - .filterCompletedInstants().lastInstant().toJavaOptional() - .map(HoodieInstant::requestedTime); - return requestedTimeToInstant(requestedTime); + return requestedTimeToInstant(latestCompletedInstantTime(metaClient)); } /** diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java index 32729be2a8958b..281aebb8f8e3af 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java @@ -54,6 +54,15 @@ public class HudiTableHandle implements ConnectorTableHandle { // reads the latest completed instant (byte-identical to a plain snapshot read). private final String queryInstant; + // Set after applySnapshot for an @incr incremental read: the resolved (begin, end] completed-timeline window + // (String instants like "20240101120000"; empty timeline / "earliest" collapse to "000") the scan reads. + // Both bounds are FULLY resolved by HudiConnectorMetadata.resolveTimeTravel (EARLIEST -> "000", an omitted / + // "latest" end -> the latest completed instant), so downstream file selection + the synthetic commit-time + // filter consume them directly. A non-null beginInstant is the incremental-read marker; null = not an + // incremental read. + private final String beginInstant; + private final String endInstant; + private HudiTableHandle(Builder builder) { this.dbName = builder.dbName; this.tableName = builder.tableName; @@ -69,6 +78,8 @@ private HudiTableHandle(Builder builder) { : Collections.emptyMap(); this.prunedPartitionPaths = builder.prunedPartitionPaths; this.queryInstant = builder.queryInstant; + this.beginInstant = builder.beginInstant; + this.endInstant = builder.endInstant; } /** Legacy constructor for Phase 1 compatibility (metadata-only). */ @@ -117,6 +128,19 @@ public String getQueryInstant() { return queryInstant; } + /** + * The resolved incremental-read begin instant (exclusive lower bound of the {@code (begin, end]} window), + * or {@code null} for a non-incremental read. A non-null value is the incremental-read marker. + */ + public String getBeginInstant() { + return beginInstant; + } + + /** The resolved incremental-read end instant (inclusive upper bound), or {@code null} if non-incremental. */ + public String getEndInstant() { + return endInstant; + } + /** Returns a builder pre-populated with this handle's state, for creating modified copies. */ public Builder toBuilder() { Builder b = new Builder(dbName, tableName, basePath, hudiTableType); @@ -126,6 +150,8 @@ public Builder toBuilder() { b.tableParameters = this.tableParameters; b.prunedPartitionPaths = this.prunedPartitionPaths; b.queryInstant = this.queryInstant; + b.beginInstant = this.beginInstant; + b.endInstant = this.endInstant; return b; } @@ -149,6 +175,8 @@ public static final class Builder { private Map tableParameters; private List prunedPartitionPaths; private String queryInstant; + private String beginInstant; + private String endInstant; public Builder(String dbName, String tableName, String basePath, String hudiTableType) { this.dbName = dbName; @@ -187,6 +215,16 @@ public Builder queryInstant(String val) { return this; } + public Builder beginInstant(String val) { + this.beginInstant = val; + return this; + } + + public Builder endInstant(String val) { + this.endInstant = val; + return this; + } + public HudiTableHandle build() { return new HudiTableHandle(this); } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java new file mode 100644 index 00000000000000..a59328167aa27f --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java @@ -0,0 +1,252 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.Callable; + +/** + * Tests the Hudi {@code @incr(...)} incremental-read window-resolution surface (INC-1): the + * {@code resolveTimeTravel(INCREMENTAL)} case + {@code applySnapshot} + the {@code begin/endInstant} pin on the + * handle, added so a hudi-on-HMS table served post-flip through the GENERIC {@code PluginDrivenScanNode} path + * resolves an incremental window byte-faithfully to legacy {@code COW/MORIncrementalRelation} — but consolidated + * into ONE connector locus. Each assertion pins WHY the behavior matters: + *
      + *
    • {@code beginTime} is required with the byte-for-byte legacy fail-loud message, so a missing bound reaches + * the user verbatim (an empty return would surface fe-core's wrong-domain "can't resolve time travel" + * text, since {@code loadSnapshot} has no INCREMENTAL not-found arm);
      • + *
    • an omitted / {@code "latest"} end bound resolves to the latest completed instant — the sentinel test is + * on the RESOLVED end value (legacy COW form), which is why {@code end="latest"} yields the instant, not + * the literal (guarding against the dead-code MOR bug that tested {@code latestTime});
      • + *
    • an empty completed timeline yields the {@code (000, 000]} window WITHOUT the begin-required check (legacy + * {@code withScanParams} short-circuits to {@code EmptyIncrementalRelation} first);
      • + *
    • applySnapshot stamps the window via {@code toBuilder()} so it PRESERVES applyFilter's prunedPartitionPaths + * and does not cross-contaminate the {@code FOR TIME AS OF} queryInstant carrier.
      • + *
    + * + *

    Unlike {@code FOR TIME AS OF}, INCREMENTAL resolution DOES touch the metaClient (to resolve the latest + * completed instant), so the metadata is built with a STUB {@link HudiMetaClientExecutor} that returns a canned + * latest-instant {@code Optional} without building a live metaClient — the same offline pattern as the + * partition-listing tests. + */ +public class HudiIncrementalTest { + + private static final List YEAR_MONTH = Arrays.asList("year", "month"); + private static final String LATEST = "20240102030405006"; + private static final String BEGIN_REQUIRED_MESSAGE = + "Specify the begin instant time to pull from using option hoodie.datasource.read.begin.instanttime"; + + // ── window resolution: begin/end pinned onto the handle ───────────────────────────────────────────── + + @Test + public void explicitBeginAndEndAreCarriedVerbatimOntoHandle() { + // Both bounds explicit and non-sentinel: they pass through unchanged (latestTime is resolved but unused). + // Drive the full path resolveTimeTravel -> applySnapshot so the FE-internal carrier properties are + // exercised end-to-end. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + HudiTableHandle pinned = resolveAndApply(md, window("20240101000000", "20240101120000")); + Assertions.assertEquals("20240101000000", pinned.getBeginInstant()); + Assertions.assertEquals("20240101120000", pinned.getEndInstant()); + } + + @Test + public void omittedEndDefaultsToLatestCompletedInstant() { + // No endTime param -> end defaults to the latest completed instant (legacy getOrDefault(end-key, + // latestTime)). Guards a mutation that would leave end null / empty for an open-ended @incr window. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + HudiTableHandle pinned = resolveAndApply(md, window("20240101000000", null)); + Assertions.assertEquals("20240101000000", pinned.getBeginInstant()); + Assertions.assertEquals(LATEST, pinned.getEndInstant(), + "an omitted endTime must resolve to the latest completed instant"); + } + + @Test + public void latestSentinelResolvesEndToTheLatestCompletedInstant() { + // end="latest" must resolve to the instant, NOT stay the literal "latest". The sentinel test is on the + // RESOLVED end value (COWIncrementalRelation:98); the dead-code MOR bug (MORIncrementalRelation:92 tested + // latestTime, so end="latest" was left unresolved) is inherently avoided by the single locus. This + // assertion KILLS a mutation back to the buggy MOR form. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + HudiTableHandle pinned = resolveAndApply(md, window("20240101000000", "latest")); + Assertions.assertEquals(LATEST, pinned.getEndInstant(), + "end=\"latest\" must resolve to the latest completed instant, not the literal sentinel"); + } + + @Test + public void earliestSentinelResolvesBeginToZero() { + // begin="earliest" -> "000" (legacy EARLIEST_TIME). Guards dropping the sentinel mapping. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + HudiTableHandle pinned = resolveAndApply(md, window("earliest", "20240101120000")); + Assertions.assertEquals("000", pinned.getBeginInstant(), + "begin=\"earliest\" must collapse to \"000\" (legacy EARLIEST_TIME)"); + } + + // ── fail-loud + empty-timeline ─────────────────────────────────────────────────────────────────────── + + @Test + public void missingBeginThrowsByteForByteLegacyMessageWhenTimelineNonEmpty() { + // Non-empty timeline (stub returns a latest instant) + no beginTime -> THROW the byte-for-byte legacy + // message (it propagates as-is through loadSnapshot). An empty return would surface fe-core's wrong-domain + // "can't resolve time travel" text. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.incremental(window(null, "20240101120000")))); + Assertions.assertEquals(BEGIN_REQUIRED_MESSAGE, ex.getMessage()); + } + + @Test + public void emptyTimelineYieldsZeroWindowWithoutBeginRequiredCheck() { + // Empty completed timeline (stub returns Optional.empty()) short-circuits to the (000, 000] window BEFORE + // the begin-required check — so a MISSING beginTime is NOT an error here (legacy withScanParams builds + // EmptyIncrementalRelation first). The window selects nothing. + HudiConnectorMetadata md = metadata(stub(Optional.empty())); + HudiTableHandle pinned = resolveAndApply(md, window(null, null)); + Assertions.assertEquals("000", pinned.getBeginInstant()); + Assertions.assertEquals("000", pinned.getEndInstant()); + } + + @Test + public void resolveIncrementalNeverReturnsEmptyForAValidWindow() { + // The generic loadSnapshot fail-loud has no INCREMENTAL not-found arm, so INCREMENTAL must ALWAYS pin. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + Optional pin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.incremental(window("20240101000000", null))); + Assertions.assertTrue(pin.isPresent(), "a valid @incr window must always pin, never return empty"); + } + + // ── applySnapshot: stamp preserving pruning / carrier isolation ───────────────────────────────────── + + @Test + public void applySnapshotStampsWindowPreservingPrunedPartitions() { + // applyFilter runs BEFORE applySnapshot at scan time, so a pruned handle must keep its pruning after the + // window pin. Guards a rebuild-from-scratch mutation (which would silently turn a pruned incremental scan + // into a full scan). + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + List pruned = Arrays.asList("year=2024/month=01", "year=2024/month=02"); + HudiTableHandle prunedHandle = partitioned().toBuilder().prunedPartitionPaths(pruned).build(); + ConnectorMvccSnapshot pin = md.resolveTimeTravel(null, prunedHandle, + ConnectorTimeTravelSpec.incremental(window("20240101000000", "20240101120000"))) + .orElseThrow(AssertionError::new); + HudiTableHandle stamped = (HudiTableHandle) md.applySnapshot(null, prunedHandle, pin); + Assertions.assertEquals("20240101000000", stamped.getBeginInstant()); + Assertions.assertEquals("20240101120000", stamped.getEndInstant()); + Assertions.assertEquals(pruned, stamped.getPrunedPartitionPaths(), + "the incremental pin must preserve applyFilter's partition pruning"); + Assertions.assertNull(stamped.getQueryInstant(), + "an incremental pin must not set the FOR TIME AS OF queryInstant carrier"); + } + + @Test + public void timeTravelPinDoesNotSetIncrementalWindow() { + // Cross-isolation the other way: a FOR TIME AS OF pin must leave begin/endInstant null (the two carriers + // are mutually exclusive). Guards accidental cross-wiring in applySnapshot. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + ConnectorMvccSnapshot pin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.timestamp("2024-01-01 12:00:00", false)).orElseThrow(AssertionError::new); + HudiTableHandle stamped = (HudiTableHandle) md.applySnapshot(null, partitioned(), pin); + Assertions.assertEquals("20240101120000", stamped.getQueryInstant()); + Assertions.assertNull(stamped.getBeginInstant()); + Assertions.assertNull(stamped.getEndInstant()); + } + + @Test + public void applySnapshotLeavesLatestPinUnchanged() { + // The query-begin latest pin (beginQuerySnapshot output) carries ONLY a snapshotId, NO window property. + // applySnapshot must return the handle UNCHANGED so a plain read stays byte-identical. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + ConnectorMvccSnapshot latestPin = + HudiConnectorMetadata.buildBeginQuerySnapshot(20240101120000000L).orElseThrow(AssertionError::new); + HudiTableHandle base = partitioned(); + HudiTableHandle result = (HudiTableHandle) md.applySnapshot(null, base, latestPin); + Assertions.assertSame(base, result, "a latest pin must not rebuild the handle"); + Assertions.assertNull(result.getBeginInstant()); + Assertions.assertNull(result.getEndInstant()); + } + + // ── handle field round-trip ───────────────────────────────────────────────────────────────────────── + + @Test + public void toBuilderRoundTripsWindowFields() { + // Guards the toBuilder() copy lines for begin/endInstant (a dropped copy would silently lose the window + // when applyFilter/applySnapshot rebuild the handle). + HudiTableHandle original = new HudiTableHandle.Builder("db", "t", "s3://b/t", "MERGE_ON_READ") + .partitionKeyNames(YEAR_MONTH) + .beginInstant("20240101000000") + .endInstant("20240101120000") + .build(); + HudiTableHandle copy = original.toBuilder().build(); + Assertions.assertEquals("20240101000000", copy.getBeginInstant()); + Assertions.assertEquals("20240101120000", copy.getEndInstant()); + // A fresh handle carries no window (null), so a plain read is not treated as incremental. + Assertions.assertNull(partitioned().getBeginInstant()); + Assertions.assertNull(partitioned().getEndInstant()); + } + + // ── helpers ──────────────────────────────────────────────────────────────────────────────────────── + + private static HudiConnectorMetadata metadata(HudiMetaClientExecutor executor) { + return new HudiConnectorMetadata(null, Collections.emptyMap(), executor); + } + + private static HudiTableHandle partitioned() { + return new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(YEAR_MONTH).build(); + } + + /** Builds the raw @incr param map fe-core threads via getIncrementalParams() (null entries omitted). */ + private static Map window(String beginTime, String endTime) { + Map params = new HashMap<>(); + if (beginTime != null) { + params.put("beginTime", beginTime); + } + if (endTime != null) { + params.put("endTime", endTime); + } + return params; + } + + private static HudiTableHandle resolveAndApply(HudiConnectorMetadata md, Map params) { + ConnectorMvccSnapshot pin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.incremental(params)).orElseThrow(AssertionError::new); + return (HudiTableHandle) md.applySnapshot(null, partitioned(), pin); + } + + /** Executor that ignores the action and returns a canned value (stubs out the live metaClient). */ + private static HudiMetaClientExecutor stub(Object cannedReturn) { + return new HudiMetaClientExecutor() { + @Override + @SuppressWarnings("unchecked") + public T execute(Callable action) { + return (T) cannedReturn; + } + }; + } +} From 2b5a2df1fd5615b5496335a6769b0afc8eeefd85 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 10:34:53 +0800 Subject: [PATCH 115/334] [doc](catalog) HMS cutover hudi HD-C3 INC-1 DONE; next = INC-2 (port IncrementalRelation family) INC-1 (@incr window resolution spine, 9327261ec52) recorded. Adversarial review outcome documented: COW timeline byte-parity blocker REFUTED via hudi-1.0.2 bytecode (metaClient.getCommitTimeline delegates to getCommitAndReplaceTimeline, not the commit-only timeline method the review conflated it with). Three findings deferred by design and now explicitly owned downstream: populateMetaFields fail-loud + hollow-commit handling -> INC-2 (relation-family port); raw hoodie.* window keys -> INC-4. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 ++- ...hudi-incremental-step-design-2026-07-09.md | 38 +++++++++++++++++-- 2 files changed, 37 insertions(+), 6 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 1af6d52547aa9b..23ce16cd434393 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。下一步 = Group C 余项无回退补齐（连接器侧，全部翻闸前）：增量读〔最高风险 HD-C3〕、schema evolution（HD-C4/C5，用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。下一步 = HD-C3 余 4 步（INC-2 移植 IncrementalRelation 家族〔含 INC-1 延后的 populateMetaFields fail-loud + hollow-commit 处理〕 → INC-3 增量 planScan → INC-4 中立谓词 SPI+fe-core 规则+反向转换器+5 可见元数据列 → INC-5 翻闸接线）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -89,7 +89,8 @@ - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 - - **✅ HD-C3 增量读 = recon + 设计 + 签字 DONE（本 session，尚未写实现码；权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**下一步 = 实现 5 步休眠拆解**（设计 §4）：INC-1 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（extend HD-C2 脊柱）→ INC-2 移植 COW/MOR/Empty IncrementalRelation（re-home `LocationPath`/`TableFormatType`/`HudiPartitionUtils`，修 `MORIncrementalRelation:92` 潜 bug）→ INC-3 增量 `planScan`（COW/MOR/RO-as-RT 单点/fallback；**不发** hoodie.* 到 BE）→ INC-4 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列 → INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C3 INC-1 DONE（`9327261ec52`，休眠）= @incr 窗口解析脊柱**（recon `wf_3398c7fb-bfb` 4 读者+parity critic〔scope critic StructuredOutput 上限失败，已用我亲手字节码核对补〕 + 对抗复审 `wf_13bea565-a47` 4 维〔各 finding 逐条 refute〕）。连接器侧扩展 HD-C2 脊柱：`resolveTimeTravel(INCREMENTAL)` **单一 locus** `resolveIncremental` 解析 (begin,end]〔begin 必填逐字 fail-loud、`"earliest"`→`"000"`、end 缺省→最新 completed instant、`"latest"`→最新〔测**已解析 end 值**=legacy COW 形，天然规避 `MOR:92` 死码 bug〕、空时间线→`(000,000]` 免 begin 检查〕→ **NON-EMPTY property-only pin**（snapshotId/schemaId inert：fe-core INCREMENTAL `loadSnapshot` 分支自列 latest 分区+latest schema、只读窗口属性）；`applySnapshot` 钉 `begin/endInstant`（与 FOR TIME `queryInstant` 载体互斥、保 prunedPartitionPaths）。新 `HudiTableHandle.begin/endInstant` + `HudiScanPlanProvider.latestCompletedInstantTime(String)`（从 `latestCompletedInstant` 字节等价抽出）。**复审关键裁决**：①**blocker（COW 时间线 byte-parity）被 hudi-1.0.2 字节码驳倒**——单 `getCommitsAndCompactionTimeline()` 按类型解析到 legacy 逐类型时间线（COW→`getCommitAndReplaceTimeline`={commit,replacecommit,clustering}=legacy COW 的 `metaClient.getCommitTimeline()`〔该 metaClient 方法**非** commit-only，委派给 getCommitAndReplaceTimeline〕；MOR→`getWriteTimeline`），review+adversarial-verify 都把 metaClient 方法与 `BaseHoodieTimeline.getCommitTimeline()` 时间线方法混淆了；②major/minor/nit 均**按设计延后**（见下）。`HudiIncrementalTest` +11、fe-connector-hudi 96 测 + checkstyle 0 + import gate 净。**⚠延后（已文档化 fail-loud、非静默丢，INC-2/4 须接）**：(a) **`populateMetaFields()` fail-loud→INC-2**（"meta fields disabled" 拒，结构上属 relation 构造器=INC-2 移植；INC-1 窗口解析不查，INC-2 移植时在 planScan 强制、同为查询规划期失败；INC-2 须字节忠实移植+e2e 加 meta-fields-disabled fixture）；(b) **hollow-commit `USE_TRANSITION_TIME`（completionTime）+ hollow 抛→INC-2/INC-3**（默认 FAIL 策略用 requestedTime，非默认策略绑 relation 文件选择）；(c) **原生 `hoodie.datasource.read.*.instanttime` 窗口键→INC-4**（INC-1 读标准 `@incr` 别名 beginTime/endTime，是否也收原生 hoodie 键归 INC-4）。 + - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **下一步 INC-2** 移植 COW/MOR/Empty IncrementalRelation（re-home `LocationPath`/`TableFormatType`/`HudiPartitionUtils`，修 `MORIncrementalRelation:92` 潜 bug；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit 处理**）→ INC-3 增量 `planScan`（COW/MOR/RO-as-RT 单点/fallback；**不发** hoodie.* 到 BE）→ INC-4 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列（**+ INC-1 延后的原生 hoodie.* 窗口键决策**）→ INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md b/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md index 0579cd6533d3c0..7984060988601e 100644 --- a/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md +++ b/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md @@ -196,11 +196,41 @@ When the handle carries a window: branch to incremental split enumeration — CO --- ## 4. Proposed dormant-commit decomposition (each independently committable + same-loader unit test) -- **INC-1 — handle pin spine + `resolveTimeTravel(INCREMENTAL)` + `applySnapshot`** (connector, dormant). Window - resolution → LATEST-pinned snapshot carrying begin/end; handle stamped. *Test:* non-empty snapshot with window - props; handle carries window + preserves pruning; never empty for a valid window. +- **✅ INC-1 DONE (`9327261ec52`) — handle pin spine + `resolveTimeTravel(INCREMENTAL)` + `applySnapshot`** + (connector, dormant). Window resolution consolidated into ONE connector locus (`resolveIncremental`): begin + required (byte-faithful fail-loud), `"earliest"`→`"000"`, end default-to-latest, `"latest"`→latest completed + instant tested on the RESOLVED end value (legacy COW form, inherently avoiding the dead-code `MOR:92` bug); + empty timeline → `(000, 000]` without the begin-required check; NON-EMPTY property-only pin (snapshotId/schemaId + inert — fe-core's INCREMENTAL `loadSnapshot` branch lists LATEST partitions + LATEST schema and reads only the + window props). `applySnapshot` stamps `begin/endInstant` (mutually exclusive with the FOR TIME `queryInstant` + carrier), preserving `prunedPartitionPaths`. New `HudiTableHandle.begin/endInstant` + `HudiScanPlanProvider. + latestCompletedInstantTime(String)` (extracted from `latestCompletedInstant`). `HudiIncrementalTest` +11 (stubbed + executor). Adversarial review (4 dims) + hudi-1.0.2 **bytecode** verification: 0 confirmed defects. + - **Timeline byte-parity (blocker REFUTED, bytecode-verified):** the single `getCommitsAndCompactionTimeline()` + resolves per table type to exactly legacy's per-type timeline — COW → `getActiveTimeline().getCommitAndReplaceTimeline()` + = `{commit, replacecommit, clustering}` = legacy COW's `metaClient.getCommitTimeline()` (that metaClient method + is NOT commit-only; it delegates to `getCommitAndReplaceTimeline`, so it includes the replacecommit/clustering + a COW table produces via INSERT OVERWRITE / clustering); MOR → `getWriteTimeline()` = legacy MOR's + `getCommitsAndCompactionTimeline()`. (The review's "COW uses commit-only `getCommitTimeline`" premise conflated + the metaClient method with the timeline-level `BaseHoodieTimeline.getCommitTimeline()` — different methods.) + - ⚠**Deferred by design (documented, fail-loud, NOT silently dropped) — INC-2/INC-4 must pick up:** + 1. **`populateMetaFields()` fail-loud → INC-2** (relation-family port): legacy COW/MOR throw + `"Incremental queries are not supported when meta fields are disabled"` (`COWIncrementalRelation:81-83` / + `MORIncrementalRelation:73-75`) for a non-empty timeline, before the begin check. That guard lives + structurally in the relation constructors = INC-2's port. INC-1 (window resolution) does not check it; + it will be enforced when INC-2 ports the relations at planScan (same query-planning-time failure, one step + later). **INC-2 MUST port this check byte-faithfully; add a meta-fields-disabled fixture to the §5 e2e.** + 2. **Hollow-commit `USE_TRANSITION_TIME` latestTime (`getCompletionTime()` vs `requestedTime()`) + the + FAIL-on-hollow-commit throw → INC-2/INC-3** (relation port): INC-1 uses `requestedTime()` (default FAIL + policy). The non-default `hoodie.read.timeline.holes.resolution.policy` variant + `handleHollowCommitIfNeeded` + are tied to the relation's file selection. + 3. **Raw `hoodie.datasource.read.{begin,end}.instanttime` window keys → INC-4:** INC-1 reads the standard + `@incr` aliases `beginTime`/`endTime` from `getIncrementalParams()` (= legacy `withScanParams`). Whether the + neutral SPI should also accept the raw hoodie keys (legacy `withScanParams:244-248` forwards `hoodie.*`) is + an INC-4 decision. - **INC-2 — port IncrementalRelation family** (connector, dormant). COW/MOR/Empty + interface; re-home helpers; - fix `MOR:92`. *Test:* start/end resolution, commit-range selection, Empty path, throw-on-fullTableScan. + fix `MOR:92`; **port the `populateMetaFields()` fail-loud + hollow-commit handling deferred from INC-1 (above).** + *Test:* start/end resolution, commit-range selection, Empty path, throw-on-fullTableScan, meta-fields-disabled throw. - **INC-3 — incremental `planScan`** (connector, dormant). COW/MOR/RO-as-RT/fallback branch; no hoodie params emitted. *Test:* split set + fallback degrades to snapshot path (not throw) + RO-as-RT routes MOR. - **INC-4 — neutral synthetic-predicate SPI + fe-core rule + reverse converter + hidden `_hoodie_commit_time`** From 1ddf8362db413b2e0bfd6e5960c1aa7c7fc2d985 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 12:29:14 +0800 Subject: [PATCH 116/334] [feat](catalog) HMS cutover hudi HD-C3 INC-2: port IncrementalRelation family (dormant) Port the legacy hudi COW/MOR/Empty IncrementalRelation family + interface from fe-core (datasource/hudi/source) into fe-connector-hudi, dormant (not yet wired into planScan = INC-3). Approach A (single window authority, user-signed): the ported relations consume the ALREADY-RESOLVED (begin, end] window that resolveIncremental stamped on HudiTableHandle and do FILE SELECTION only; they do NOT re-resolve the window. So the dead MORIncrementalRelation:92 sentinel bug vanishes by construction, and the file set can never diverge from the later synthetic _hoodie_commit_time row filter (INC-4). - IncrementalRelation (interface) + COW/MOR/Empty relations, connector-internal, package-private. Split type re-homed spi.Split/HudiSplit -> HudiScanRange; LocationPath/TableFormatType/HudiPartitionUtils dropped (fe-core). COW collectSplits -> List (path + length/fileSize from HoodieWriteStat.getFileSizeInBytes(), format via detectFileFormat, partition values via the connector's Map parsePartitionValues on getTableConfig().getPartitionFields() = legacy COW:212 parity). MOR collectFileSlices -> List verbatim. Everything from the archived-flag computation onward is byte-faithful to legacy; only the window-parse prologue is removed. getHoodieParams/getStartTs dropped (dead in the FE-filter model). - Ported the INC-1-deferred guards as shared statics on IncrementalRelation, with byte-for-byte messages re-typed to DorisConnectorException: checkIncrementalMetaFields (meta-fields-disabled, first ctor guard, non-empty timeline only via upstream Empty routing), checkStateTransitionTimeFullTableScan, decideArchivalFullTableScan, checkNotFullTableScan. - resolveIncremental made hollow-commit-policy-aware for the END axis: under hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME it resolves the default/"latest" end on the completion-time axis (new latestCompletedInstantTime overload: getCompletionTime vs requestedTime) so the single handle-stamped end matches the ported relation's completion-time file selection AND the later row filter -- no under-read of the final in-window commit. Inert under the default policy (byte-identical) and dormant (planScan reads only queryInstant, never begin/endInstant). Dormant: nothing wires the relations into planScan (INC-3). Zero fe-core imports; no fe-core edits. HEAD-grounded recon + 5-dim clean-room adversarial review (byte-parity / dormancy-iron-rule / guard-correctness = 0 findings; 3 test/doc- honesty findings fixed; MOR:92 + END-axis crux self-verified). The constructor- level file selection (timeline/metadata/fs) + the completion-time axis are e2e-only (design section 5, now mandating meta-fields-disabled + USE_TRANSITION_TIME fixtures). Green: fe-connector-hudi full suite (HudiIncrementalRelationTest 13 + HudiIncrementalTest 12; 0 failures) + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../hudi/COWIncrementalRelation.java | 229 ++++++++++++++++++ .../hudi/EmptyIncrementalRelation.java | 59 +++++ .../connector/hudi/HudiConnectorMetadata.java | 22 +- .../connector/hudi/HudiScanPlanProvider.java | 27 ++- .../connector/hudi/IncrementalRelation.java | 128 ++++++++++ .../hudi/MORIncrementalRelation.java | 177 ++++++++++++++ .../hudi/HudiIncrementalRelationTest.java | 166 +++++++++++++ .../connector/hudi/HudiIncrementalTest.java | 16 ++ 8 files changed, 817 insertions(+), 7 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java new file mode 100644 index 00000000000000..20dc2eeef7b139 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java @@ -0,0 +1,229 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.GlobPattern; +import org.apache.hadoop.fs.Path; +import org.apache.hudi.common.fs.FSUtils; +import org.apache.hudi.common.model.FileSlice; +import org.apache.hudi.common.model.HoodieCommitMetadata; +import org.apache.hudi.common.model.HoodieReplaceCommitMetadata; +import org.apache.hudi.common.model.HoodieWriteStat; +import org.apache.hudi.common.table.HoodieTableMetaClient; +import org.apache.hudi.common.table.timeline.HoodieInstant; +import org.apache.hudi.common.table.timeline.HoodieTimeline; +import org.apache.hudi.common.table.timeline.TimelineUtils; +import org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling; +import org.apache.hudi.common.util.Option; +import org.apache.hudi.storage.StoragePath; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Consumer; +import java.util.stream.Collectors; + +/** + * Selects the base files a COW {@code @incr(...)} read must scan over a resolved {@code (begin, end]} window. + * Connector-internal port of legacy {@code datasource.hudi.source.COWIncrementalRelation}, with the window-parse + * prologue removed (the window is resolved ONCE by {@link HudiConnectorMetadata#resolveTimeTravel} and consumed + * here, see {@link IncrementalRelation}) and the split type re-homed from fe-core {@code HudiSplit} to + * {@link HudiScanRange}. Everything from the archived-flag computation onward is byte-faithful to legacy. + * + *

    {@link #collectSplits()} yields native ranges directly (COW has only base files); {@link #collectFileSlices()} + * is unsupported (the MOR shape). + */ +final class COWIncrementalRelation implements IncrementalRelation { + + private final Map optParams; + private final HoodieTableMetaClient metaClient; + private final HollowCommitHandling hollowCommitHandling; + private final boolean startInstantArchived; + private final boolean endInstantArchived; + private final boolean fullTableScan; + private final FileSystem fs; + private final Map fileToWriteStat; + private final Collection filteredRegularFullPaths; + private final Collection filteredMetaBootstrapFullPaths; + + private final String startTs; + private final String endTs; + + COWIncrementalRelation(HoodieTableMetaClient metaClient, Configuration configuration, + String startTs, String endTs, HollowCommitHandling hollowCommitHandling, + Map optParams) throws IOException { + this.optParams = optParams; + this.metaClient = metaClient; + this.hollowCommitHandling = hollowCommitHandling; + this.startTs = startTs; + this.endTs = endTs; + HoodieTimeline commitTimeline = TimelineUtils.handleHollowCommitIfNeeded( + metaClient.getCommitTimeline().filterCompletedInstants(), metaClient, hollowCommitHandling); + // Meta-fields guard (ported from INC-1's deferral list): fail loud before any file selection. It is the + // first guard because the empty-completed-timeline case routes to EmptyIncrementalRelation upstream, so + // this relation is built only for a non-empty timeline (legacy's "non-empty only" semantics). + IncrementalRelation.checkIncrementalMetaFields(metaClient.getTableConfig().populateMetaFields()); + + startInstantArchived = commitTimeline.isBeforeTimelineStarts(startTs); + endInstantArchived = commitTimeline.isBeforeTimelineStarts(endTs); + + HoodieTimeline commitsTimelineToReturn; + if (hollowCommitHandling == HollowCommitHandling.USE_TRANSITION_TIME) { + commitsTimelineToReturn = commitTimeline.findInstantsInRangeByCompletionTime(startTs, endTs); + } else { + commitsTimelineToReturn = commitTimeline.findInstantsInRange(startTs, endTs); + } + List commitsToReturn = commitsTimelineToReturn.getInstants(); + + // todo: support configuration hoodie.datasource.read.incr.filters + StoragePath basePath = metaClient.getBasePath(); + Map regularFileIdToFullPath = new HashMap<>(); + Map metaBootstrapFileIdToFullPath = new HashMap<>(); + HoodieTimeline replacedTimeline = commitsTimelineToReturn.getCompletedReplaceTimeline(); + Map replacedFile = new HashMap<>(); + for (HoodieInstant instant : replacedTimeline.getInstants()) { + HoodieReplaceCommitMetadata metadata = metaClient.getActiveTimeline() + .readReplaceCommitMetadata(instant); + metadata.getPartitionToReplaceFileIds().forEach( + (key, value) -> value.forEach( + e -> replacedFile.put(e, FSUtils.constructAbsolutePath(basePath, key).toString()))); + } + + fileToWriteStat = new HashMap<>(); + for (HoodieInstant commit : commitsToReturn) { + HoodieCommitMetadata metadata = metaClient.getActiveTimeline().readCommitMetadata(commit); + metadata.getPartitionToWriteStats().forEach((partition, stats) -> { + for (HoodieWriteStat stat : stats) { + fileToWriteStat.put(FSUtils.constructAbsolutePath(basePath, stat.getPath()).toString(), stat); + } + }); + if (HoodieTimeline.METADATA_BOOTSTRAP_INSTANT_TS.equals(commit.requestedTime())) { + metadata.getFileIdAndFullPaths(basePath).forEach((k, v) -> { + if (!(replacedFile.containsKey(k) && v.startsWith(replacedFile.get(k)))) { + metaBootstrapFileIdToFullPath.put(k, v); + } + }); + } else { + metadata.getFileIdAndFullPaths(basePath).forEach((k, v) -> { + if (!(replacedFile.containsKey(k) && v.startsWith(replacedFile.get(k)))) { + regularFileIdToFullPath.put(k, v); + } + }); + } + } + + if (!metaBootstrapFileIdToFullPath.isEmpty()) { + // filer out meta bootstrap files that have had more commits since metadata bootstrap + metaBootstrapFileIdToFullPath.entrySet().removeIf(e -> regularFileIdToFullPath.containsKey(e.getKey())); + } + String pathGlobPattern = optParams.getOrDefault("hoodie.datasource.read.incr.path.glob", ""); + if ("".equals(pathGlobPattern)) { + filteredRegularFullPaths = regularFileIdToFullPath.values(); + filteredMetaBootstrapFullPaths = metaBootstrapFileIdToFullPath.values(); + } else { + GlobPattern globMatcher = new GlobPattern("*" + pathGlobPattern); + filteredRegularFullPaths = regularFileIdToFullPath.values().stream().filter(globMatcher::matches) + .collect(Collectors.toList()); + filteredMetaBootstrapFullPaths = metaBootstrapFileIdToFullPath.values().stream() + .filter(globMatcher::matches).collect(Collectors.toList()); + } + + fs = new Path(basePath.toUri().getPath()).getFileSystem(configuration); + fullTableScan = shouldFullTableScan(); + } + + private boolean shouldFullTableScan() throws IOException { + boolean fallbackToFullTableScan = Boolean.parseBoolean( + optParams.getOrDefault("hoodie.datasource.read.incr.fallback.fulltablescan.enable", "false")); + if (IncrementalRelation.decideArchivalFullTableScan( + fallbackToFullTableScan, startInstantArchived, endInstantArchived, hollowCommitHandling)) { + return true; + } + if (fallbackToFullTableScan) { + for (String path : filteredMetaBootstrapFullPaths) { + if (!fs.exists(new Path(path))) { + return true; + } + } + for (String path : filteredRegularFullPaths) { + if (!fs.exists(new Path(path))) { + return true; + } + } + } + return false; + } + + @Override + public List collectFileSlices() { + throw new UnsupportedOperationException(); + } + + @Override + public List collectSplits() { + IncrementalRelation.checkNotFullTableScan(fullTableScan); + if (filteredRegularFullPaths.isEmpty() && filteredMetaBootstrapFullPaths.isEmpty()) { + return Collections.emptyList(); + } + List splits = new ArrayList<>(); + // Partition-column NAMES come from the hudi table config (byte-faithful to legacy COW:212), NOT the + // HMS-sourced handle.partitionKeyNames the snapshot path uses; the two coincide for hive-synced tables. + Option partitionColumns = metaClient.getTableConfig().getPartitionFields(); + List partitionNames = partitionColumns.isPresent() ? Arrays.asList(partitionColumns.get()) + : Collections.emptyList(); + + Consumer generatorSplit = baseFile -> { + HoodieWriteStat stat = fileToWriteStat.get(baseFile); + splits.add(new HudiScanRange.Builder() + .path(baseFile) + .start(0) + // length + fileSize both from the write stat, matching legacy HudiSplit(0, size, size, ...). + .length(stat.getFileSizeInBytes()) + .fileSize(stat.getFileSizeInBytes()) + .fileFormat(HudiScanPlanProvider.detectFileFormat(baseFile)) + .partitionValues( + HudiScanPlanProvider.parsePartitionValues(stat.getPartitionPath(), partitionNames)) + .build()); + }; + + for (String baseFile : filteredMetaBootstrapFullPaths) { + generatorSplit.accept(baseFile); + } + for (String baseFile : filteredRegularFullPaths) { + generatorSplit.accept(baseFile); + } + return splits; + } + + @Override + public boolean fallbackFullTableScan() { + return fullTableScan; + } + + @Override + public String getEndTs() { + return endTs; + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java new file mode 100644 index 00000000000000..70ea92557db809 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java @@ -0,0 +1,59 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.hudi.common.model.FileSlice; + +import java.util.Collections; +import java.util.List; + +/** + * The {@code @incr} relation for an empty completed timeline: it selects nothing. Connector-internal port of + * legacy {@code datasource.hudi.source.EmptyIncrementalRelation}. The scan planner routes the empty-timeline + * case here (legacy {@code LogicalHudiScan.withScanParams:261-262}), so COW/MOR are never built empty. + * + *

    Legacy's {@code getHoodieParams()} (which set {@code hoodie.datasource.read.incr.operation} / + * {@code includeStartTime} on the BE-facing params) is intentionally NOT ported: the connector uses the FE-side + * synthetic-predicate model for row correctness and emits NO {@code hoodie.datasource.read.*} incremental keys + * to BE, so those keys are inert (see the incremental-read step design). {@code getStartTs()} is likewise + * dropped (the resolved window lives on the handle); {@code getEndTs()} returns the legacy {@code "000"} bound. + */ +final class EmptyIncrementalRelation implements IncrementalRelation { + + private static final String EMPTY_TS = "000"; + + @Override + public List collectFileSlices() { + return Collections.emptyList(); + } + + @Override + public List collectSplits() { + return Collections.emptyList(); + } + + @Override + public boolean fallbackFullTableScan() { + return false; + } + + @Override + public String getEndTs() { + return EMPTY_TS; + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 0bbfb0fc487a4f..5cf6f7d33f09d6 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -107,6 +107,15 @@ public class HudiConnectorMetadata implements ConnectorMetadata { private static final String INCR_LATEST_SENTINEL = "latest"; private static final String INCR_ZERO_INSTANT = "000"; + // The legacy hollow-commit policy key and the one non-default value that shifts window resolution. Under + // USE_TRANSITION_TIME legacy resolves the default / "latest" end on the completion-time axis + // (COWIncrementalRelation:94-96 / MORIncrementalRelation:88-90) and its file selection uses + // findInstantsInRangeByCompletionTime; resolving the end here on the SAME axis keeps ONE handle-stamped end + // correct for both file selection and the later synthetic _hoodie_commit_time row filter. Any other value + // (or absent) -> the default requested-time axis = byte-identical to the pre-policy behaviour. + private static final String INCR_HOLLOW_POLICY_KEY = "hoodie.read.timeline.holes.resolution.policy"; + private static final String INCR_STATE_TRANSITION_POLICY = "USE_TRANSITION_TIME"; + private final HmsClient hmsClient; private final Map properties; // Runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + TCCL pin (see R4). @@ -347,7 +356,12 @@ public Optional resolveTimeTravel(ConnectorSession sessio *

  • {@code endTime} defaults to the latest completed instant; {@code "latest"} → the latest completed * instant. The sentinel test is on the RESOLVED end value (legacy COW form, * {@code COWIncrementalRelation:98}); the single locus inherently avoids the dead-code MOR bug - * ({@code MORIncrementalRelation:92}, which tested {@code latestTime} and so never fired).
    • + * ({@code MORIncrementalRelation:92}, which tested {@code latestTime} and so never fired). The + * "latest completed instant" is taken on the COMPLETION-time axis under the {@code USE_TRANSITION_TIME} + * hollow-commit policy (legacy parity) so the ONE resolved end matches the ported relation's + * completion-time file selection AND the later synthetic {@code _hoodie_commit_time} row filter — the + * connector never lets the file set and the row filter diverge on the window. An explicitly-supplied + * {@code endTime} is used verbatim on either axis (the user supplies an axis-appropriate value). * * *

    The pin is property-only: {@code snapshotId}/{@code schemaId} are inert because fe-core's INCREMENTAL @@ -356,9 +370,13 @@ public Optional resolveTimeTravel(ConnectorSession sessio * the window onto the handle.

    */ private ConnectorMvccSnapshot resolveIncremental(HudiTableHandle handle, Map params) { + // Resolve the latest completed instant on the completion-time axis when the hollow-commit policy is + // USE_TRANSITION_TIME (legacy parity, see INCR_HOLLOW_POLICY_KEY); the default axis is requested-time. + boolean useCompletionTime = INCR_STATE_TRANSITION_POLICY.equals(params.get(INCR_HOLLOW_POLICY_KEY)); Optional latestTime = metaClientExecutor.execute(() -> HudiScanPlanProvider.latestCompletedInstantTime( - HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), handle.getBasePath()))); + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), handle.getBasePath()), + useCompletionTime)); String begin; String end; if (!latestTime.isPresent()) { diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 3986b9523b06a2..44ba0c20a9083a 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -386,13 +386,28 @@ static HoodieTableMetaClient buildMetaClient(Configuration conf, String basePath * {@code metaClient.getCommitsAndCompactionTimeline()}. * * Both legacy and this helper take {@code lastInstant().requestedTime()} under the default hollow-commit - * policy; the {@code USE_TRANSITION_TIME} completion-time variant is a documented deferral (see the - * incremental-read step design). + * policy; the {@code USE_TRANSITION_TIME} completion-time variant is served by the overload below. */ static Optional latestCompletedInstantTime(HoodieTableMetaClient metaClient) { + return latestCompletedInstantTime(metaClient, false); + } + + /** + * The LATEST completed instant on the requested-time axis (default) or the completion-time axis when + * {@code useCompletionTime} is true. The completion-time axis is legacy's {@code USE_TRANSITION_TIME} + * hollow-commit policy path: legacy COW/MOR derive the default / {@code "latest"} end via + * {@code lastInstant().getCompletionTime()} rather than {@code requestedTime()} under that policy + * ({@code COWIncrementalRelation:94-96} / {@code MORIncrementalRelation:88-90}), and both their file + * selection ({@code findInstantsInRangeByCompletionTime}) and the row filter consume that completion-time + * end. {@link HudiConnectorMetadata#resolveTimeTravel} resolves the end on the SAME axis so the ONE + * handle-stamped end is correct for both — the connector never lets the file set and the row filter diverge + * on the window. Completion time {@code >=} requested time for any instant, so a requested-time end fed into + * completion-time selection would silently drop the final in-window commit (under-read). + */ + static Optional latestCompletedInstantTime(HoodieTableMetaClient metaClient, boolean useCompletionTime) { return metaClient.getCommitsAndCompactionTimeline() .filterCompletedInstants().lastInstant().toJavaOptional() - .map(HoodieInstant::requestedTime); + .map(instant -> useCompletionTime ? instant.getCompletionTime() : instant.requestedTime()); } /** @@ -564,9 +579,11 @@ private static String unescapePathName(String path) { } /** - * Detect file format from file path suffix. + * Detect file format from file path suffix. Package-private static so the ported incremental relations + * ({@code COWIncrementalRelation}) stamp the required explicit {@code fileFormat} on their native + * {@link HudiScanRange}s the same way the snapshot COW path does. */ - private static String detectFileFormat(String filePath) { + static String detectFileFormat(String filePath) { if (filePath == null || filePath.isEmpty()) { return "parquet"; } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java new file mode 100644 index 00000000000000..0208b1be4bee99 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java @@ -0,0 +1,128 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.DorisConnectorException; + +import org.apache.hudi.common.model.FileSlice; +import org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling; + +import java.util.List; + +/** + * Selects the files an {@code @incr(...)} incremental read must scan over a resolved {@code (begin, end]} + * commit-time window. Connector-internal port of legacy {@code datasource.hudi.source.IncrementalRelation}, + * with the split type re-homed from fe-core {@code spi.Split}/{@code HudiSplit} to the connector's + * {@link HudiScanRange} (fe-core is off-limits across the plugin boundary). + * + *

    Window is already resolved. Unlike legacy — where each relation constructor re-parsed / re-resolved + * the window from {@code optParams} — the ported relations take the ALREADY-RESOLVED {@code startTs}/{@code endTs} + * that {@link HudiConnectorMetadata#resolveTimeTravel} stamped on the {@link HudiTableHandle}, and do FILE + * SELECTION only. This keeps a SINGLE window authority (the handle) feeding both the file set here and the + * synthetic {@code _hoodie_commit_time} row filter a later step injects, so the two can never disagree on the + * window; the dead {@code MORIncrementalRelation:92} sentinel bug (which tested {@code latestTime} instead of the + * end) vanishes by construction because no sentinel re-resolution survives in the relation. + * + *

    Two shapes, mirroring legacy's own asymmetry: a COW relation yields native {@link HudiScanRange}s directly + * ({@link #collectSplits()}); a MOR relation yields merged {@link FileSlice}s ({@link #collectFileSlices()}) that + * the scan planner later turns into JNI ranges at {@code endTs}. Each relation supports only its own shape and + * throws {@link UnsupportedOperationException} for the other. + * + *

    DORMANT. Nothing wires these relations into {@code HudiScanPlanProvider.planScan} yet (that is the + * next step); they are ported + unit-testable in isolation. The empty-completed-timeline case routes to + * {@link EmptyIncrementalRelation} in the scan planner (legacy {@code LogicalHudiScan.withScanParams:261-262}), + * so COW/MOR are never constructed on an empty timeline and their meta-fields guard fires only for a non-empty + * one (legacy parity). + */ +interface IncrementalRelation { + + /** Merged file slices at {@code endTs} for the MOR path (COW throws {@link UnsupportedOperationException}). */ + List collectFileSlices(); + + /** Native base-file ranges for the COW path (MOR throws {@link UnsupportedOperationException}). */ + List collectSplits(); + + /** + * Whether the window fell back to a full-table scan (archived instant / missing file). The scan planner + * checks this BEFORE calling {@link #collectSplits()}/{@link #collectFileSlices()} and, when true, degrades + * to the normal latest-snapshot partition scan instead of the incremental path (legacy + * {@code HudiScanNode.getSplits:470}). + */ + boolean fallbackFullTableScan(); + + /** The resolved inclusive window end (the MOR-JNI merge instant); {@code "000"} for the empty relation. */ + String getEndTs(); + + /** + * Fail loud when a table has meta fields disabled, byte-for-byte with legacy + * ({@code COWIncrementalRelation:81-83} / {@code MORIncrementalRelation:73-75}) but re-typed to the + * connector's {@link DorisConnectorException} (the user-visible message is preserved, exactly as INC-1 + * re-typed the begin-required throw). Ported here from INC-1's deferral list. Because the empty timeline + * routes to {@link EmptyIncrementalRelation} upstream, this is reached only for a non-empty timeline + * (legacy's "non-empty only" semantics), so it stays the FIRST guard in each COW/MOR constructor. + */ + static void checkIncrementalMetaFields(boolean populateMetaFields) { + if (!populateMetaFields) { + throw new DorisConnectorException( + "Incremental queries are not supported when meta fields are disabled"); + } + } + + /** + * Fail loud when the {@code USE_TRANSITION_TIME} hollow-commit policy meets a full-table-scan fallback, + * byte-for-byte with legacy ({@code COWIncrementalRelation:178-180} inside {@code shouldFullTableScan}, + * {@code MORIncrementalRelation:104-106} in the constructor). Shared so both relations emit the identical + * message. + */ + static void checkStateTransitionTimeFullTableScan(HollowCommitHandling policy, boolean fullTableScan) { + if (policy == HollowCommitHandling.USE_TRANSITION_TIME && fullTableScan) { + throw new DorisConnectorException( + "Cannot use stateTransitionTime while enables full table scan"); + } + } + + /** + * Fail loud when a resolved window fell back to a full-table scan, byte-for-byte with legacy + * ({@code COWIncrementalRelation:206} / {@code MORIncrementalRelation:177}), re-typed to the connector's + * {@link DorisConnectorException}. A DEFENSIVE invariant: the scan planner is contracted to check + * {@link #fallbackFullTableScan()} and degrade to the snapshot path BEFORE calling {@link #collectSplits()} + * / {@link #collectFileSlices()}, so this normally never fires. Extracted (like the other guards) so the + * message is byte-verifiable offline. + */ + static void checkNotFullTableScan(boolean fullTableScan) { + if (fullTableScan) { + throw new DorisConnectorException("Fallback to full table scan"); + } + } + + /** + * The archival half of legacy COW {@code shouldFullTableScan} ({@code COWIncrementalRelation:177-182}): a + * full-table scan is required when the fallback is enabled AND either bound is archived (before the + * timeline start); under {@code USE_TRANSITION_TIME} that combination is rejected instead. Pure booleans in, + * so it is unit-testable as a matrix; the file-existence half of {@code shouldFullTableScan} stays in the + * COW relation because it probes the filesystem. + */ + static boolean decideArchivalFullTableScan(boolean fallbackEnabled, boolean startArchived, + boolean endArchived, HollowCommitHandling policy) { + if (fallbackEnabled && (startArchived || endArchived)) { + checkStateTransitionTimeFullTableScan(policy, true); + return true; + } + return false; + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java new file mode 100644 index 00000000000000..09503e575cdc22 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java @@ -0,0 +1,177 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.GlobPattern; +import org.apache.hudi.common.model.BaseFile; +import org.apache.hudi.common.model.FileSlice; +import org.apache.hudi.common.model.HoodieCommitMetadata; +import org.apache.hudi.common.table.HoodieTableMetaClient; +import org.apache.hudi.common.table.timeline.HoodieInstant; +import org.apache.hudi.common.table.timeline.HoodieTimeline; +import org.apache.hudi.common.table.timeline.TimelineUtils; +import org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling; +import org.apache.hudi.common.table.view.HoodieTableFileSystemView; +import org.apache.hudi.hadoop.utils.HoodieInputFormatUtils; +import org.apache.hudi.metadata.HoodieTableMetadataUtil; +import org.apache.hudi.storage.StoragePathInfo; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +/** + * Selects the merged file slices a MOR {@code @incr(...)} read must scan over a resolved {@code (begin, end]} + * window. Connector-internal port of legacy {@code datasource.hudi.source.MORIncrementalRelation}, with the + * window-parse prologue removed (the window is resolved ONCE by + * {@link HudiConnectorMetadata#resolveTimeTravel} and consumed here, see {@link IncrementalRelation}). The dead + * {@code MORIncrementalRelation:92} sentinel bug (which tested {@code latestTime} instead of the end and so never + * fired) is gone by construction — no sentinel re-resolution survives in the relation. + * + *

    {@link #collectFileSlices()} yields the merged slices for the scan planner to turn into JNI ranges at + * {@code endTs}; {@link #collectSplits()} is unsupported (the COW shape). + */ +final class MORIncrementalRelation implements IncrementalRelation { + + private final Map optParams; + private final HoodieTableMetaClient metaClient; + private final HoodieTimeline timeline; + private final HollowCommitHandling hollowCommitHandling; + private final String startTimestamp; + private final String endTimestamp; + private final boolean startInstantArchived; + private final boolean endInstantArchived; + private final List includedCommits; + private final List commitsMetadata; + private final List affectedFilesInCommits; + private final boolean fullTableScan; + private final String globPattern; + + MORIncrementalRelation(HoodieTableMetaClient metaClient, Configuration configuration, + String startTs, String endTs, HollowCommitHandling hollowCommitHandling, + Map optParams) throws IOException { + this.optParams = optParams; + this.metaClient = metaClient; + this.hollowCommitHandling = hollowCommitHandling; + this.startTimestamp = startTs; + this.endTimestamp = endTs; + timeline = metaClient.getCommitsAndCompactionTimeline().filterCompletedInstants(); + // Meta-fields guard (ported from INC-1's deferral list): fail loud before any file selection. First guard + // because the empty-completed-timeline case routes to EmptyIncrementalRelation upstream, so this relation + // is built only for a non-empty timeline (legacy's "non-empty only" semantics). + IncrementalRelation.checkIncrementalMetaFields(metaClient.getTableConfig().populateMetaFields()); + + startInstantArchived = timeline.isBeforeTimelineStarts(startTimestamp); + endInstantArchived = timeline.isBeforeTimelineStarts(endTimestamp); + + includedCommits = getIncludedCommits(); + commitsMetadata = getCommitsMetadata(); + affectedFilesInCommits = HoodieInputFormatUtils.listAffectedFilesForCommits(configuration, + metaClient.getBasePath(), commitsMetadata); + fullTableScan = shouldFullTableScan(); + IncrementalRelation.checkStateTransitionTimeFullTableScan(hollowCommitHandling, fullTableScan); + globPattern = optParams.getOrDefault("hoodie.datasource.read.incr.path.glob", ""); + } + + private List getIncludedCommits() { + if (!startInstantArchived || !endInstantArchived) { + // If endTimestamp commit is not archived, will filter instants + // before endTimestamp. + if (hollowCommitHandling == HollowCommitHandling.USE_TRANSITION_TIME) { + return timeline.findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp).getInstants(); + } else { + return timeline.findInstantsInRange(startTimestamp, endTimestamp).getInstants(); + } + } else { + return timeline.getInstants(); + } + } + + private List getCommitsMetadata() throws IOException { + List result = new ArrayList<>(); + for (HoodieInstant commit : includedCommits) { + result.add(TimelineUtils.getCommitMetadata(commit, timeline)); + } + return result; + } + + private boolean shouldFullTableScan() throws IOException { + boolean should = Boolean.parseBoolean( + optParams.getOrDefault("hoodie.datasource.read.incr.fallback.fulltablescan.enable", "false")) && ( + startInstantArchived || endInstantArchived); + if (should) { + return true; + } + for (StoragePathInfo fileStatus : affectedFilesInCommits) { + if (!metaClient.getStorage().exists(fileStatus.getPath())) { + return true; + } + } + return false; + } + + @Override + public boolean fallbackFullTableScan() { + return fullTableScan; + } + + @Override + public String getEndTs() { + return endTimestamp; + } + + @Override + public List collectFileSlices() { + if (includedCommits.isEmpty()) { + return Collections.emptyList(); + } + IncrementalRelation.checkNotFullTableScan(fullTableScan); + HoodieTimeline scanTimeline; + if (hollowCommitHandling == HollowCommitHandling.USE_TRANSITION_TIME) { + scanTimeline = metaClient.getCommitsAndCompactionTimeline() + .findInstantsInRangeByCompletionTime(startTimestamp, endTimestamp); + } else { + scanTimeline = TimelineUtils.handleHollowCommitIfNeeded( + metaClient.getCommitsAndCompactionTimeline(), metaClient, hollowCommitHandling) + .findInstantsInRange(startTimestamp, endTimestamp); + } + String latestCommit = includedCommits.get(includedCommits.size() - 1).requestedTime(); + HoodieTableFileSystemView fsView = new HoodieTableFileSystemView(metaClient, scanTimeline, + affectedFilesInCommits); + Stream fileSlices = HoodieTableMetadataUtil.getWritePartitionPaths(commitsMetadata) + .stream().flatMap(relativePartitionPath -> + fsView.getLatestMergedFileSlicesBeforeOrOn(relativePartitionPath, latestCommit)); + if ("".equals(globPattern)) { + return fileSlices.collect(Collectors.toList()); + } + GlobPattern globMatcher = new GlobPattern("*" + globPattern); + return fileSlices.filter(fileSlice -> globMatcher.matches(fileSlice.getBaseFile().map(BaseFile::getPath) + .or(fileSlice.getLatestLogFile().map(f -> f.getPath().toString())).get())) + .collect(Collectors.toList()); + } + + @Override + public List collectSplits() { + throw new UnsupportedOperationException(); + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java new file mode 100644 index 00000000000000..dcc3d2d93ecd14 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java @@ -0,0 +1,166 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.DorisConnectorException; + +import org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Tests the OFFLINE-verifiable surface of the ported {@code @incr} IncrementalRelation family (INC-2): the pure + * fail-loud guards extracted onto {@link IncrementalRelation} and the {@link EmptyIncrementalRelation}. Each + * assertion pins WHY the behavior matters. + * + *

    Coverage scope (Rule 12 — no over-claim). The COW/MOR relation CONSTRUCTORS do all timeline + + * metadata + filesystem work EAGERLY on a live {@code HoodieTableMetaClient}, and the connector deliberately has + * no Mockito / no hudi write deps, so the file-SELECTION pipeline (commit-range selection, write-stat → + * {@link HudiScanRange} mapping, file-slice selection, {@code fs.exists} full-table-scan probes, the meta-fields + * guard on a REAL disabled table, and the {@code USE_TRANSITION_TIME} completion-time END axis) is inherently + * e2e-only and is DEFERRED to the flip-time e2e (design §5 now mandates a meta-fields-disabled fixture and a + * USE_TRANSITION_TIME completion-axis fixture). Until that e2e lands, the completion-time END axis is UNVERIFIED + * at unit level (the stubbed executor swallows it). These unit tests cover ONLY the pure decisions they name; + * they do NOT prove file selection or the axis resolution. + */ +public class HudiIncrementalRelationTest { + + // ── meta-fields fail-loud (ported from INC-1 deferral; byte-for-byte message) ──────────────────────────── + + @Test + public void metaFieldsDisabledThrowsByteForByteLegacyMessage() { + // Legacy COW:81-83 / MOR:73-75 reject incremental on a meta-fields-disabled table. The message must be + // byte-for-byte (re-typed to DorisConnectorException, the connector's fail-loud type) so it reaches the + // user verbatim. This guard is the FIRST check in each ported COW/MOR constructor. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> IncrementalRelation.checkIncrementalMetaFields(false)); + Assertions.assertEquals( + "Incremental queries are not supported when meta fields are disabled", ex.getMessage()); + } + + @Test + public void metaFieldsEnabledIsNoOp() { + // A normal (meta-fields-enabled) table must pass the guard. Guards a mutation that inverts the condition. + Assertions.assertDoesNotThrow(() -> IncrementalRelation.checkIncrementalMetaFields(true)); + } + + // ── state-transition-time + full-table-scan rejection (byte-for-byte message) ──────────────────────────── + + @Test + public void stateTransitionTimeWithFullTableScanThrows() { + // Legacy COW:178-180 / MOR:104-106 reject USE_TRANSITION_TIME combined with a full-table-scan fallback. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> IncrementalRelation.checkStateTransitionTimeFullTableScan( + HollowCommitHandling.USE_TRANSITION_TIME, true)); + Assertions.assertEquals( + "Cannot use stateTransitionTime while enables full table scan", ex.getMessage()); + } + + @Test + public void stateTransitionTimeWithoutFullTableScanIsNoOp() { + // USE_TRANSITION_TIME alone (no full-table scan) is fine — only the COMBINATION is rejected. + Assertions.assertDoesNotThrow(() -> IncrementalRelation.checkStateTransitionTimeFullTableScan( + HollowCommitHandling.USE_TRANSITION_TIME, false)); + } + + @Test + public void fullTableScanUnderDefaultPolicyIsNoOp() { + // A full-table scan under the default FAIL policy must NOT throw — the throw is specific to + // USE_TRANSITION_TIME. Guards a mutation that drops the policy condition. + Assertions.assertDoesNotThrow(() -> IncrementalRelation.checkStateTransitionTimeFullTableScan( + HollowCommitHandling.FAIL, true)); + } + + // ── archival full-table-scan decision matrix (COW pure gate) ───────────────────────────────────────────── + + @Test + public void archivalFullTableScanRequiresFallbackEnabled() { + // Fallback disabled -> never a full-table scan on archival, even if a bound is archived. Guards the + // fallback-gates-everything semantics (legacy COW:177). + Assertions.assertFalse(IncrementalRelation.decideArchivalFullTableScan( + false, true, true, HollowCommitHandling.FAIL)); + } + + @Test + public void archivalFullTableScanRequiresAnArchivedBound() { + // Fallback enabled but NEITHER bound archived -> the archival gate does not fire (the file-existence + // probe, not tested here, may still trigger). Guards a mutation that drops the archived condition. + Assertions.assertFalse(IncrementalRelation.decideArchivalFullTableScan( + true, false, false, HollowCommitHandling.FAIL)); + } + + @Test + public void archivalFullTableScanFiresWhenStartOrEndArchived() { + // Either archived bound (with fallback) triggers a full-table scan under a non-transition policy. + Assertions.assertTrue(IncrementalRelation.decideArchivalFullTableScan( + true, true, false, HollowCommitHandling.FAIL)); + Assertions.assertTrue(IncrementalRelation.decideArchivalFullTableScan( + true, false, true, HollowCommitHandling.BLOCK)); + } + + @Test + public void archivalFullTableScanRejectsStateTransitionTime() { + // The archival trigger under USE_TRANSITION_TIME is rejected (legacy COW:178-180), not returned as true. + Assertions.assertThrows(DorisConnectorException.class, + () -> IncrementalRelation.decideArchivalFullTableScan( + true, true, false, HollowCommitHandling.USE_TRANSITION_TIME)); + } + + @Test + public void stateTransitionTimeRejectionIsGatedOnAnArchivedBound() { + // USE_TRANSITION_TIME alone must NOT reject: legacy nests the transition-time throw INSIDE the archival + // trigger `fallback && (startArchived||endArchived)` (COW:177-181), so a non-archived window returns false + // even under USE_TRANSITION_TIME. Kills a mutation that hoists the state-transition check above the + // archival guard (rejecting any USE_TRANSITION_TIME+fallback window). + Assertions.assertFalse(IncrementalRelation.decideArchivalFullTableScan( + true, false, false, HollowCommitHandling.USE_TRANSITION_TIME)); + } + + // ── fallback-to-full-table-scan defensive throw (byte-for-byte message) ────────────────────────────────── + + @Test + public void fullTableScanFallbackThrowsByteForByteLegacyMessage() { + // The defensive guard COW.collectSplits / MOR.collectFileSlices fire when the window fell back to a full + // scan (legacy COW:206 / MOR:177). Byte-for-byte message, re-typed to DorisConnectorException. The scan + // planner is contracted to degrade before calling collect*, so this rarely fires — but the message must + // still reach the user verbatim if it does. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> IncrementalRelation.checkNotFullTableScan(true)); + Assertions.assertEquals("Fallback to full table scan", ex.getMessage()); + } + + @Test + public void notFullTableScanIsNoOp() { + // A window that did NOT fall back must pass the guard (the normal incremental path). Guards a condition + // inversion. + Assertions.assertDoesNotThrow(() -> IncrementalRelation.checkNotFullTableScan(false)); + } + + // ── empty relation (empty completed timeline) ──────────────────────────────────────────────────────────── + + @Test + public void emptyRelationSelectsNothing() { + // The empty-timeline relation selects no files on either shape and never falls back. Its end bound is the + // legacy "000" sentinel. Guards against a mutation returning non-empty / a non-"000" bound. + EmptyIncrementalRelation empty = new EmptyIncrementalRelation(); + Assertions.assertTrue(empty.collectSplits().isEmpty(), "empty relation must select no splits"); + Assertions.assertTrue(empty.collectFileSlices().isEmpty(), "empty relation must select no file slices"); + Assertions.assertFalse(empty.fallbackFullTableScan(), "empty relation never falls back to a full scan"); + Assertions.assertEquals("000", empty.getEndTs(), "empty relation end bound is the legacy \"000\""); + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java index a59328167aa27f..e2ac2b227e8c19 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java @@ -108,6 +108,22 @@ public void earliestSentinelResolvesBeginToZero() { "begin=\"earliest\" must collapse to \"000\" (legacy EARLIEST_TIME)"); } + @Test + public void useTransitionTimePolicyStillResolvesWindow() { + // A USE_TRANSITION_TIME hollow-commit policy param must not break window resolution: resolveIncremental + // computes the completion-time axis (useCompletionTime=true) and still pins a valid (begin, end]. The stub + // executor returns a canned latest regardless of the axis, so the requested-vs-completion AXIS itself is + // NOT verified here — it is deferred to the §5 USE_TRANSITION_TIME completion-axis e2e fixture. This test + // only guards that adding the policy branch did not crash resolution or drop the pin. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + Map params = window("20240101000000", null); + params.put("hoodie.read.timeline.holes.resolution.policy", "USE_TRANSITION_TIME"); + HudiTableHandle pinned = resolveAndApply(md, params); + Assertions.assertEquals("20240101000000", pinned.getBeginInstant()); + Assertions.assertEquals(LATEST, pinned.getEndInstant(), + "USE_TRANSITION_TIME must still resolve an omitted end to the (canned) latest completed instant"); + } + // ── fail-loud + empty-timeline ─────────────────────────────────────────────────────────────────────── @Test From 06d3442bdb0d7564c18a9038ed642aed6f42e399 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 12:32:11 +0800 Subject: [PATCH 117/334] [doc](catalog) HMS cutover hudi HD-C3 INC-2 DONE; next = INC-3 (incremental planScan) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit INC-2 (port IncrementalRelation family) landed dormant in d64a25f28fc under the user-signed Approach A (single window authority). Update HANDOFF 🎯 + the Hudi section + the 5-step decomposition line; next dormant step = INC-3 (incremental planScan, wiring COW/MOR/RO-as-RT split enumeration, no hoodie.* to BE). Also add to the incremental-step design §5 flip-time e2e gate the two fixtures INC-2's port made load-bearing: a meta-fields-disabled table (the checkIncrementalMetaFields rejection) and a hollow/inflight-commit table under hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME (the completion- time END axis — the only level at which INC-2's axis fix is verifiable; the offline unit tests cannot drive it). Closes the review's test/doc-honesty finding that the test docstrings cited a §5 fixture that did not yet exist. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- .../tasks/hudi-incremental-step-design-2026-07-09.md | 9 +++++++-- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 23ce16cd434393..3136637b30ea75 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。下一步 = HD-C3 余 4 步（INC-2 移植 IncrementalRelation 家族〔含 INC-1 延后的 populateMetaFields fail-loud + hollow-commit 处理〕 → INC-3 增量 planScan → INC-4 中立谓词 SPI+fe-core 规则+反向转换器+5 可见元数据列 → INC-5 翻闸接线）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。下一步 = HD-C3 余 3 步（INC-3 增量 planScan → INC-4 中立谓词 SPI+fe-core 规则+反向转换器+5 可见元数据列 → INC-5 翻闸接线）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -90,7 +90,8 @@ - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 - **✅ HD-C3 INC-1 DONE（`9327261ec52`，休眠）= @incr 窗口解析脊柱**（recon `wf_3398c7fb-bfb` 4 读者+parity critic〔scope critic StructuredOutput 上限失败，已用我亲手字节码核对补〕 + 对抗复审 `wf_13bea565-a47` 4 维〔各 finding 逐条 refute〕）。连接器侧扩展 HD-C2 脊柱：`resolveTimeTravel(INCREMENTAL)` **单一 locus** `resolveIncremental` 解析 (begin,end]〔begin 必填逐字 fail-loud、`"earliest"`→`"000"`、end 缺省→最新 completed instant、`"latest"`→最新〔测**已解析 end 值**=legacy COW 形，天然规避 `MOR:92` 死码 bug〕、空时间线→`(000,000]` 免 begin 检查〕→ **NON-EMPTY property-only pin**（snapshotId/schemaId inert：fe-core INCREMENTAL `loadSnapshot` 分支自列 latest 分区+latest schema、只读窗口属性）；`applySnapshot` 钉 `begin/endInstant`（与 FOR TIME `queryInstant` 载体互斥、保 prunedPartitionPaths）。新 `HudiTableHandle.begin/endInstant` + `HudiScanPlanProvider.latestCompletedInstantTime(String)`（从 `latestCompletedInstant` 字节等价抽出）。**复审关键裁决**：①**blocker（COW 时间线 byte-parity）被 hudi-1.0.2 字节码驳倒**——单 `getCommitsAndCompactionTimeline()` 按类型解析到 legacy 逐类型时间线（COW→`getCommitAndReplaceTimeline`={commit,replacecommit,clustering}=legacy COW 的 `metaClient.getCommitTimeline()`〔该 metaClient 方法**非** commit-only，委派给 getCommitAndReplaceTimeline〕；MOR→`getWriteTimeline`），review+adversarial-verify 都把 metaClient 方法与 `BaseHoodieTimeline.getCommitTimeline()` 时间线方法混淆了；②major/minor/nit 均**按设计延后**（见下）。`HudiIncrementalTest` +11、fe-connector-hudi 96 测 + checkstyle 0 + import gate 净。**⚠延后（已文档化 fail-loud、非静默丢，INC-2/4 须接）**：(a) **`populateMetaFields()` fail-loud→INC-2**（"meta fields disabled" 拒，结构上属 relation 构造器=INC-2 移植；INC-1 窗口解析不查，INC-2 移植时在 planScan 强制、同为查询规划期失败；INC-2 须字节忠实移植+e2e 加 meta-fields-disabled fixture）；(b) **hollow-commit `USE_TRANSITION_TIME`（completionTime）+ hollow 抛→INC-2/INC-3**（默认 FAIL 策略用 requestedTime，非默认策略绑 relation 文件选择）；(c) **原生 `hoodie.datasource.read.*.instanttime` 窗口键→INC-4**（INC-1 读标准 `@incr` 别名 beginTime/endTime，是否也收原生 hoodie 键归 INC-4）。 - - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **下一步 INC-2** 移植 COW/MOR/Empty IncrementalRelation（re-home `LocationPath`/`TableFormatType`/`HudiPartitionUtils`，修 `MORIncrementalRelation:92` 潜 bug；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit 处理**）→ INC-3 增量 `planScan`（COW/MOR/RO-as-RT 单点/fallback；**不发** hoodie.* 到 BE）→ INC-4 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列（**+ INC-1 延后的原生 hoodie.* 窗口键决策**）→ INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C3 INC-2 DONE（`d64a25f28fc`，休眠）= IncrementalRelation 家族移植**（HEAD-grounded recon `wf_50a59f91-14e` 3 读者〔窗口协调/边界/API+可测性〕+完整性 critic + 5 维对抗复审 `wf_b247a285-d64`〔byte-parity/dormancy-iron-rule/guard-correctness 三镜头 **0 发现**；test-strength 3 test/doc-honesty finding 已修；mor92-and-axis 镜头 API stall 死→crux 亲手 grep+读自证〕）。COW/MOR/Empty + `IncrementalRelation` 接口移植进 `fe-connector-hudi`（连接器内部、package-private）。**用户签字方案 A（单一窗口权威，见本 session 决策）**：移植关系**消费句柄已解析的 (begin,end] 窗口只挑文件、自己不再解析**（构造器删窗口解析 prologue）→ 死码 `MORIncrementalRelation:92` sentinel bug **自然消失**、文件集永不与 INC-4 `_hoodie_commit_time` 行过滤的 end 打架。split 型 re-home `spi.Split`/`HudiSplit`→`HudiScanRange`（丢 `LocationPath`/`TableFormatType`/`HudiPartitionUtils` fe-core 型）；COW `collectSplits`→`List`（path + length/fileSize 取 `HoodieWriteStat.getFileSizeInBytes()`、format 走 `detectFileFormat`、分区值走连接器 Map `parsePartitionValues` on `metaClient.getTableConfig().getPartitionFields()`=legacy COW:212 平价，**非** HMS handle keys）；MOR `collectFileSlices`→`List` 逐字；archived-flag 起字节忠实 legacy，仅删窗口 prologue。INC-1 延后守卫作 `IncrementalRelation` 共享静态、逐字文案 re-type 成 `DorisConnectorException`：`checkIncrementalMetaFields`（meta-fields-disabled 首守卫、非空时间线 only 靠上游 Empty 路由）/`checkStateTransitionTimeFullTableScan`/`decideArchivalFullTableScan`/`checkNotFullTableScan`。**`resolveIncremental` END 轴按策略**：`hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME` 下 default/`latest` end 走 completion-time 轴（新 `latestCompletedInstantTime(mc, useCompletionTime)` 重载：getCompletionTime vs requestedTime）→ 单一 handle-stamped end 与关系 `findInstantsInRangeByCompletionTime` 选片同轴、无末批漏读；默认策略字节不变、休眠（planScan 只读 `queryInstant` 从不读 begin/endInstant）。`getHoodieParams`/`getStartTs` 丢弃（FE-filter 模型下已死）。全绿（`HudiIncrementalRelationTest` 13 + `HudiIncrementalTest` 12 + 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**⚠ e2e-only（休眠单测只覆盖纯守卫+Empty；构造器 timeline/metadata/fs 选片 + completion-time 轴须 e2e，设计 §5 已加 meta-fields-disabled + USE_TRANSITION_TIME fixture）**。**下一步 = INC-3 增量 planScan（接线 COW/MOR/RO-as-RT 单点/fallback，不发 hoodie.* 到 BE）**。 + - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **下一步 INC-3** 增量 `planScan`（COW/MOR/RO-as-RT 单点/fallback；**不发** hoodie.* 到 BE）→ INC-4 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列（**+ INC-1 延后的原生 hoodie.* 窗口键决策**）→ INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md b/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md index 7984060988601e..2e53896e547b2a 100644 --- a/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md +++ b/plan-doc/tasks/hudi-incremental-step-design-2026-07-09.md @@ -243,8 +243,13 @@ When the handle carries a window: branch to incremental split enumeration — CO COW + MOR tables, each ≥3 commits; `SELECT ... incr('beginTime'=c1,'endTime'=c2)` returns EXACTLY the rows written in `(c1, c2]` — **including the linchpin case where a base file touched in the window also carries forward older-commit rows** (proves the `_hoodie_commit_time` filter, not just file selection). Plus: RO-as-RT table, -`endTime='latest'` sentinel, empty-timeline (Empty relation), fallback-full-table-scan (archived instant). Assert -byte-parity vs the legacy HMS hudi catalog on identical data. +`endTime='latest'` sentinel, empty-timeline (Empty relation), fallback-full-table-scan (archived instant), +**meta-fields-disabled table** (the `checkIncrementalMetaFields` rejection — mandated §4 INC-1 deferral-1), and a +**hollow / inflight-commit table read under `hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME`** +(the completion-time END axis: a table where `getCompletionTime() != requestedTime()` for the last in-window +commit, asserting the end resolves on the completion axis so the final commit's rows are NOT under-read — this is +the ONLY level at which the axis fix from INC-2 is verified; the offline unit tests cannot drive it, INC-1 +deferral-2). Assert byte-parity vs the legacy HMS hudi catalog on identical data. ## 6. Decisions - **D-C3-1 (visibility parity) — SIGNED (user 2026-07-09) = all 5 `_hoodie_*` meta columns VISIBLE (legacy parity).** (§1.1) From f51f2c5d9ab0f993f24ea2373f2ac85ef1570898 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 14:21:43 +0800 Subject: [PATCH 118/334] [feat](catalog) HMS cutover hudi HD-C3 INC-3: incremental planScan wiring (dormant) Wire the ported (dormant) IncrementalRelation family into HudiScanPlanProvider.planScan so a Hudi @incr read selects the files its resolved (begin, end] window touches, instead of the latest-snapshot scan. Connector-side and DORMANT (hms not in SPI_READY_TYPES); FE-only, no BE change; no hoodie.* incremental params emitted to BE. - planScan: add a `getBeginInstant() != null` incremental branch (after schema resolution, before the fsView build) that builds the COW/MOR relation via metaClient.getTableType() and returns its file selection, or degrades to the latest-snapshot scan on fallbackFullTableScan() (legacy getSplits:470). - Extract buildMorRange from collectMorSplits (byte-identical for the snapshot path); the incremental MOR path reuses it, stamping the JNI merge instant at relation.getEndTs() and per-slice partition values from the Hudi table-config partition fields (not the HMS handle keys), matching legacy getIncrementalSplits. - Route COW->collectSplits (native) / MOR->collectFileSlices by relation type. Signed behavior change: force_jni + COW + @incr no longer crashes (legacy called collectFileSlices() on a COW relation -> UnsupportedOperationException); a COW incremental read now always reads native. MOR still honors force_jni. - Thread the raw @incr option params (glob / fallback / hollow-commit policy) onto HudiTableHandle: resolveIncremental copies them into the FE-internal MVCC snapshot properties under a "hudi.incr-opt." prefix, applySnapshot reconstructs them into a new incrementalParams field. FE-only transport (never serialized to BE); planScan derives HollowCommitHandling via IncrementalRelation .hollowCommitHandling, same policy key resolveIncremental's END axis uses. - The legacy RO-as-RT flip is subsumed by metaClient.getTableType() (hoodie.properties-authoritative), so it is not ported and no serde params are threaded. Row-level (begin, end] filtering is a later step (INC-4). Tests (offline): HudiIncrementalPlanScanTest (incrementalRanges routing + buildMorRange mapping via a fake IncrementalRelation + hand-built FileSlice), HudiIncrementalTest +3 (param threading round-trip + namespace isolation), HudiIncrementalRelationTest +3 (hollowCommitHandling FAIL default / USE_TRANSITION_TIME / bogus-throw). Full hudi module green, checkstyle 0, connector-import gate clean. 4-dimension adversarial review: 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiConnectorMetadata.java | 35 ++- .../connector/hudi/HudiScanPlanProvider.java | 222 +++++++++++++---- .../doris/connector/hudi/HudiTableHandle.java | 25 ++ .../connector/hudi/IncrementalRelation.java | 16 ++ .../hudi/HudiIncrementalPlanScanTest.java | 227 ++++++++++++++++++ .../hudi/HudiIncrementalRelationTest.java | 36 +++ .../connector/hudi/HudiIncrementalTest.java | 61 +++++ 7 files changed, 570 insertions(+), 52 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 5cf6f7d33f09d6..a3e07bea8bda72 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -95,6 +95,16 @@ public class HudiConnectorMetadata implements ConnectorMetadata { private static final String HUDI_INCREMENTAL_BEGIN_PROPERTY = "hudi.incremental-begin"; private static final String HUDI_INCREMENTAL_END_PROPERTY = "hudi.incremental-end"; + // Prefix under which the RAW @incr option params ride the same FE-internal MVCC-snapshot transport, so the + // ported IncrementalRelation family reads its glob / fallback / hollow-commit-policy options at planScan + // exactly as legacy did (planScan has only the handle, not the original scan params). Each raw key becomes + // "hudi.incr-opt." in the snapshot properties and is stripped back in applySnapshot into + // HudiTableHandle.incrementalParams. The "hudi." prefix (never "hoodie.") keeps it namespace-consistent with + // the other FE-internal carriers and guarantees no raw hoodie.* key masquerades as a BE-facing property (and + // the transport is FE-only regardless: fe-core consumes the snapshot ONLY via applySnapshot, never + // serializing its properties to BE). + private static final String HUDI_INCR_OPT_PREFIX = "hudi.incr-opt."; + // The @incr window param keys fe-core threads verbatim via getIncrementalParams() (byte-faithful to legacy // LogicalHudiScan.withScanParams, which reads "beginTime"/"endTime" from the scan params). private static final String INCR_BEGIN_TIME_KEY = "beginTime"; @@ -398,10 +408,19 @@ private ConnectorMvccSnapshot resolveIncremental(HudiTableHandle handle, Map { + if (v != null) { + builder.property(HUDI_INCR_OPT_PREFIX + k, v); + } + }); + return builder.build(); } /** @@ -434,9 +453,19 @@ public ConnectorTableHandle applySnapshot(ConnectorSession session, } String beginInstant = properties.get(HUDI_INCREMENTAL_BEGIN_PROPERTY); if (beginInstant != null) { + // Reconstruct the raw @incr option params from their prefixed carriers (the begin/end keys use a + // different "hudi.incremental-*" namespace, so they are not collected here). + Map incrementalParams = new HashMap<>(); + for (Map.Entry entry : properties.entrySet()) { + if (entry.getKey().startsWith(HUDI_INCR_OPT_PREFIX)) { + incrementalParams.put( + entry.getKey().substring(HUDI_INCR_OPT_PREFIX.length()), entry.getValue()); + } + } return ((HudiTableHandle) handle).toBuilder() .beginInstant(beginInstant) .endInstant(properties.get(HUDI_INCREMENTAL_END_PROPERTY)) + .incrementalParams(incrementalParams) .build(); } return handle; diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 44ba0c20a9083a..94f45318f413c6 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -31,6 +31,7 @@ import org.apache.hudi.common.config.HoodieMetadataConfig; import org.apache.hudi.common.engine.HoodieLocalEngineContext; import org.apache.hudi.common.model.BaseFile; +import org.apache.hudi.common.model.FileSlice; import org.apache.hudi.common.model.HoodieBaseFile; import org.apache.hudi.common.model.HoodieLogFile; import org.apache.hudi.common.model.HoodieTableType; @@ -38,15 +39,19 @@ import org.apache.hudi.common.table.TableSchemaResolver; import org.apache.hudi.common.table.timeline.HoodieInstant; import org.apache.hudi.common.table.timeline.HoodieTimeline; +import org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling; import org.apache.hudi.common.table.view.FileSystemViewManager; import org.apache.hudi.common.table.view.HoodieTableFileSystemView; +import org.apache.hudi.common.util.Option; import org.apache.hudi.metadata.HoodieTableMetadata; import org.apache.hudi.metadata.HoodieTableMetadataUtil; import org.apache.hudi.storage.StoragePath; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.io.IOException; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; @@ -70,8 +75,9 @@ * * * - *

    Scope: Snapshot reads of non-incremental tables. - * Incremental reads, schema evolution, and time travel are deferred.

    + *

    Scope: snapshot reads, {@code FOR TIME AS OF} time travel (a pinned {@code queryInstant}), and {@code @incr} + * incremental FILE selection (a resolved {@code (begin, end]} window). Incremental row-level filtering to the + * window (a synthetic {@code _hoodie_commit_time} predicate) and schema evolution are deferred to later steps.

    */ public class HudiScanPlanProvider implements ConnectorScanPlanProvider { @@ -162,6 +168,34 @@ public List planScan( columnTypes = Collections.emptyList(); } + String inputFormat = hudiHandle.getInputFormat(); + String serdeLib = hudiHandle.getSerdeLib(); + + // @incr incremental read: a non-null beginInstant is the marker (INC-1 stamped the resolved (begin, end] + // window onto the handle). Select the files the window touches via the ported IncrementalRelation family + // instead of the latest-snapshot partition scan below. NOTE: this selects FILES only — row-level + // filtering to (begin, end] is a LATER step (an FE-side synthetic _hoodie_commit_time predicate), so a + // bare @incr read would over-read until that lands (harmless while the connector is dormant). The COW-vs- + // MOR relation choice is driven by the SAME isCow the snapshot path uses (metaClient.getTableType(), + // hoodie.properties-authoritative): this SUBSUMES the legacy RO-as-RT flip (LogicalHudiScan:251-260 / + // HudiScanNode:187-199), which existed only because legacy classifies from the hive inputFormat (a MOR + // _ro facade uses HoodieParquetInputFormat, so legacy misreads it as COW and the flip re-routes it to + // MOR). metaClient.getTableType() reads MERGE_ON_READ for that facade directly, so no flip — and no serde + // params are needed on the handle. Do NOT restore the flip. + if (hudiHandle.getBeginInstant() != null) { + IncrementalRelation relation = buildIncrementalRelation(metaClient, conf, hudiHandle, isCow); + Optional> incrementalRanges = incrementalRanges(relation, isCow, forceJni, + basePath, inputFormat, serdeLib, columnNames, columnTypes, partitionFieldNames(metaClient)); + if (incrementalRanges.isPresent()) { + LOG.info("Hudi incremental scan planning: {}.{} window=({}, {}] splits={}", + hudiHandle.getDbName(), hudiHandle.getTableName(), + hudiHandle.getBeginInstant(), hudiHandle.getEndInstant(), incrementalRanges.get().size()); + return incrementalRanges.get(); + } + // relation.fallbackFullTableScan() (archived instant / missing file) → degrade to the normal + // latest-snapshot partition scan below (legacy HudiScanNode.getSplits:470), reading the latest instant. + } + // Build file system view via FileSystemViewManager (Hudi 1.0.2 API) HoodieMetadataConfig metadataConfig = HoodieMetadataConfig.newBuilder() .enable(HoodieTableMetadataUtil.isFilesPartitionAvailable(metaClient)) @@ -173,9 +207,6 @@ public List planScan( // Resolve partitions List partitionPaths = resolvePartitions(hudiHandle, metaClient); - String inputFormat = hudiHandle.getInputFormat(); - String serdeLib = hudiHandle.getSerdeLib(); - List ranges = new ArrayList<>(); for (String partitionPath : partitionPaths) { Map partValues = parsePartitionValues( @@ -283,50 +314,143 @@ private void collectMorSplits( Map partValues, boolean forceJni, List ranges) { fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, queryInstant) - .forEach(fileSlice -> { - Optional baseFileOpt = fileSlice.getBaseFile().toJavaOptional(); - String filePath = baseFileOpt.map(BaseFile::getPath).orElse(""); - long fileSize = baseFileOpt.map(BaseFile::getFileSize).orElse(0L); - - List logs = fileSlice.getLogFiles() - .map(HoodieLogFile::getPath) - .map(StoragePath::toString) - .collect(Collectors.toList()); - - // Dynamic format decision: no logs → native reader, UNLESS force_jni keeps it on JNI - // (legacy HudiScanNode.setScanParams' !isForceJniScanner() guard on the no-log downgrade). - boolean useNative = logs.isEmpty() && !filePath.isEmpty() && !forceJni; - String format = useNative ? detectFileFormat(filePath) : "jni"; - - // For log-only slices, use first log as agency path - String agencyPath = filePath.isEmpty() && !logs.isEmpty() - ? logs.get(0) : filePath; - - HudiScanRange.Builder builder = new HudiScanRange.Builder() - .path(agencyPath) - .start(0) - .length(fileSize) - .fileSize(fileSize) - .fileFormat(format) - .partitionValues(partValues) - // Bake force_jni so populateRangeParams (no session) keeps this slice on JNI too. - .forceJni(forceJni); - - if (!useNative) { - // JNI reader needs full metadata - builder.instantTime(queryInstant) - .serde(serdeLib) - .inputFormat(inputFormat) - .basePath(basePath) - .dataFilePath(filePath) - .dataFileLength(fileSize) - .deltaLogs(logs) - .columnNames(columnNames) - .columnTypes(columnTypes); - } - - ranges.add(builder.build()); - }); + .forEach(fileSlice -> ranges.add(buildMorRange(fileSlice, partValues, queryInstant, + forceJni, basePath, inputFormat, serdeLib, columnNames, columnTypes))); + } + + /** + * Builds one MOR {@link HudiScanRange} from a merged {@link FileSlice}, shared by the snapshot MOR path + * ({@link #collectMorSplits}) and the {@code @incr} MOR path ({@link #incrementalRanges}). Byte-faithful port + * of legacy {@code HudiScanNode.generateHudiSplit}: a slice with no delta logs reads natively (parquet/orc) + * UNLESS {@code force_jni} keeps it on JNI, a log-only slice uses its first log as the agency path, and a JNI + * slice carries the full merge metadata. The {@code jniInstant} is the merge instant BE reads: the snapshot + * path passes its {@code queryInstant}, the incremental path passes the resolved window END + * ({@code relation.getEndTs()}). Package-private static so the mapping is unit-testable with a hand-built + * {@link FileSlice} and reused by both paths without duplication. + */ + static HudiScanRange buildMorRange(FileSlice fileSlice, Map partValues, String jniInstant, + boolean forceJni, String basePath, String inputFormat, String serdeLib, + List columnNames, List columnTypes) { + Optional baseFileOpt = fileSlice.getBaseFile().toJavaOptional(); + String filePath = baseFileOpt.map(BaseFile::getPath).orElse(""); + long fileSize = baseFileOpt.map(BaseFile::getFileSize).orElse(0L); + + List logs = fileSlice.getLogFiles() + .map(HoodieLogFile::getPath) + .map(StoragePath::toString) + .collect(Collectors.toList()); + + // Dynamic format decision: no logs → native reader, UNLESS force_jni keeps it on JNI + // (legacy HudiScanNode.setScanParams' !isForceJniScanner() guard on the no-log downgrade). + boolean useNative = logs.isEmpty() && !filePath.isEmpty() && !forceJni; + String format = useNative ? detectFileFormat(filePath) : "jni"; + + // For log-only slices, use first log as agency path + String agencyPath = filePath.isEmpty() && !logs.isEmpty() + ? logs.get(0) : filePath; + + HudiScanRange.Builder builder = new HudiScanRange.Builder() + .path(agencyPath) + .start(0) + .length(fileSize) + .fileSize(fileSize) + .fileFormat(format) + .partitionValues(partValues) + // Bake force_jni so populateRangeParams (no session) keeps this slice on JNI too. + .forceJni(forceJni); + + if (!useNative) { + // JNI reader needs full metadata + builder.instantTime(jniInstant) + .serde(serdeLib) + .inputFormat(inputFormat) + .basePath(basePath) + .dataFilePath(filePath) + .dataFileLength(fileSize) + .deltaLogs(logs) + .columnNames(columnNames) + .columnTypes(columnTypes); + } + + return builder.build(); + } + + /** + * Builds the ported {@link IncrementalRelation} for a resolved {@code @incr} window: a COW relation when + * {@code isCow} (metaClient-authoritative), else a MOR relation. The relation consumes the ALREADY-RESOLVED + * {@code begin/endInstant} from the handle (INC-1) and the raw {@code @incr} option params (glob / fallback / + * hollow-commit policy) threaded onto the handle, and does file selection only. Runs inline in {@code + * planScan}, reusing its metaClient + Hadoop conf, so the relation's filesystem/timeline I/O inherits the + * scan thread's plugin classloader pin (the same context the snapshot path's metaClient I/O runs in). The + * ctor's {@link IOException} is re-typed to {@link DorisConnectorException} (parity with {@link + * #resolvePartitions}). + */ + private static IncrementalRelation buildIncrementalRelation(HoodieTableMetaClient metaClient, + Configuration conf, HudiTableHandle handle, boolean isCow) { + Map optParams = handle.getIncrementalParams(); + HollowCommitHandling policy = IncrementalRelation.hollowCommitHandling(optParams); + try { + return isCow + ? new COWIncrementalRelation(metaClient, conf, handle.getBeginInstant(), + handle.getEndInstant(), policy, optParams) + : new MORIncrementalRelation(metaClient, conf, handle.getBeginInstant(), + handle.getEndInstant(), policy, optParams); + } catch (IOException e) { + throw new DorisConnectorException( + "Failed to build incremental relation for " + handle.getBasePath(), e); + } + } + + /** + * The incremental split set for a resolved {@code @incr} window, or {@link Optional#empty()} to signal the + * caller must DEGRADE to the normal latest-snapshot scan. Byte-parity with legacy {@code + * HudiScanNode.getSplits:470} + {@code getIncrementalSplits}, but routing on the relation TYPE + * ({@code isCow}) rather than legacy's {@code canUseNativeReader()}: + *
      + *
    • {@code relation.fallbackFullTableScan()} (an archived instant / missing file) → + * {@link Optional#empty()} = degrade to the latest-snapshot scan (NOT an error), legacy {@code :470}.
      • + *
    • COW → {@link IncrementalRelation#collectSplits()} yields native ranges directly. + * {@code force_jni} is intentionally IGNORED for a COW incremental read (it always reads native) + * — a signed, deliberate deviation from legacy, which routes {@code force_jni}+COW to the MOR-style + * branch and calls {@code collectFileSlices()} on a COW relation → {@code UnsupportedOperationException} + * (a latent legacy crash). Routing on the relation type never calls the unsupported shape.
      • + *
    • MOR → {@link IncrementalRelation#collectFileSlices()} (a FLAT cross-partition slice list) turned + * into JNI ranges at the resolved window END ({@code relation.getEndTs()}), with per-slice partition + * values parsed from the slice's own partition path against the Hudi table-config partition fields + * (the same non-handle source the COW relation uses). {@code force_jni} still keeps a no-log MOR slice + * on JNI via {@link #buildMorRange}.
      • + *
    + * Package-private static, pure over the {@link IncrementalRelation} contract, so file-selection routing + + * the degrade decision are unit-testable with a fake relation (no live metaClient). + */ + static Optional> incrementalRanges(IncrementalRelation relation, boolean isCow, + boolean forceJni, String basePath, String inputFormat, String serdeLib, + List columnNames, List columnTypes, List partitionFieldNames) { + if (relation.fallbackFullTableScan()) { + return Optional.empty(); + } + List ranges = new ArrayList<>(); + if (isCow) { + ranges.addAll(relation.collectSplits()); + return Optional.of(ranges); + } + String endTs = relation.getEndTs(); + for (FileSlice fileSlice : relation.collectFileSlices()) { + Map partValues = parsePartitionValues(fileSlice.getPartitionPath(), partitionFieldNames); + ranges.add(buildMorRange(fileSlice, partValues, endTs, forceJni, + basePath, inputFormat, serdeLib, columnNames, columnTypes)); + } + return Optional.of(ranges); + } + + /** + * The Hudi table-config partition-field names (byte-faithful to legacy {@code HudiScanNode:391-393}), the + * source the incremental MOR path parses per-slice partition values against — NOT the HMS-sourced + * handle partition keys the snapshot path uses (the two coincide only for hive-synced tables). + */ + private static List partitionFieldNames(HoodieTableMetaClient metaClient) { + Option fields = metaClient.getTableConfig().getPartitionFields(); + return fields.isPresent() ? Arrays.asList(fields.get()) : Collections.emptyList(); } /** diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java index 281aebb8f8e3af..21fdc50cf90c6e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiTableHandle.java @@ -63,6 +63,13 @@ public class HudiTableHandle implements ConnectorTableHandle { private final String beginInstant; private final String endInstant; + // Set after applySnapshot for an @incr incremental read: the RAW @incr option params fe-core threaded via + // getIncrementalParams() (e.g. hoodie.datasource.read.incr.path.glob / ...fallback.fulltablescan.enable / + // hoodie.read.timeline.holes.resolution.policy). planScan feeds this map straight to the ported + // IncrementalRelation constructors as their optParams (and derives HollowCommitHandling from it), so the + // relations read the glob / fallback / policy exactly as legacy did. Empty for a non-incremental read. + private final Map incrementalParams; + private HudiTableHandle(Builder builder) { this.dbName = builder.dbName; this.tableName = builder.tableName; @@ -80,6 +87,9 @@ private HudiTableHandle(Builder builder) { this.queryInstant = builder.queryInstant; this.beginInstant = builder.beginInstant; this.endInstant = builder.endInstant; + this.incrementalParams = builder.incrementalParams != null + ? Collections.unmodifiableMap(builder.incrementalParams) + : Collections.emptyMap(); } /** Legacy constructor for Phase 1 compatibility (metadata-only). */ @@ -141,6 +151,14 @@ public String getEndInstant() { return endInstant; } + /** + * The raw {@code @incr} option params (glob / fallback-full-table-scan / hollow-commit policy), fed verbatim + * to the ported incremental relations at scan time. Empty (never null) for a non-incremental read. + */ + public Map getIncrementalParams() { + return incrementalParams; + } + /** Returns a builder pre-populated with this handle's state, for creating modified copies. */ public Builder toBuilder() { Builder b = new Builder(dbName, tableName, basePath, hudiTableType); @@ -152,6 +170,7 @@ public Builder toBuilder() { b.queryInstant = this.queryInstant; b.beginInstant = this.beginInstant; b.endInstant = this.endInstant; + b.incrementalParams = this.incrementalParams; return b; } @@ -177,6 +196,7 @@ public static final class Builder { private String queryInstant; private String beginInstant; private String endInstant; + private Map incrementalParams; public Builder(String dbName, String tableName, String basePath, String hudiTableType) { this.dbName = dbName; @@ -225,6 +245,11 @@ public Builder endInstant(String val) { return this; } + public Builder incrementalParams(Map val) { + this.incrementalParams = val; + return this; + } + public HudiTableHandle build() { return new HudiTableHandle(this); } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java index 0208b1be4bee99..6c5ffffd08bc5e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java @@ -23,6 +23,7 @@ import org.apache.hudi.common.table.timeline.TimelineUtils.HollowCommitHandling; import java.util.List; +import java.util.Map; /** * Selects the files an {@code @incr(...)} incremental read must scan over a resolved {@code (begin, end]} @@ -83,6 +84,21 @@ static void checkIncrementalMetaFields(boolean populateMetaFields) { } } + /** + * Maps the {@code @incr} hollow-commit policy option to its {@link HollowCommitHandling} enum, byte-faithful + * to legacy ({@code COWIncrementalRelation:74-75} / {@code MORIncrementalRelation:76-77}): the policy key + * defaults to {@code FAIL} and any explicit value is passed to {@link HollowCommitHandling#valueOf} (which + * THROWS {@link IllegalArgumentException} on a bogus value — legacy parity, same terminal error). The + * policy-key literal is RE-DECLARED here rather than imported from fe-core's {@code IncrementalRelation} + * across the plugin boundary, and matches {@code HudiConnectorMetadata.INCR_HOLLOW_POLICY_KEY} (the ONE place + * the END-axis resolution reads the same key). Called by {@code HudiScanPlanProvider.planScan} so the ported + * relation's OWN completion-time file selection uses the SAME policy that drove the window END resolution. + */ + static HollowCommitHandling hollowCommitHandling(Map optParams) { + return HollowCommitHandling.valueOf( + optParams.getOrDefault("hoodie.read.timeline.holes.resolution.policy", "FAIL")); + } + /** * Fail loud when the {@code USE_TRANSITION_TIME} hollow-commit policy meets a full-table-scan fallback, * byte-for-byte with legacy ({@code COWIncrementalRelation:178-180} inside {@code shouldFullTableScan}, diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java new file mode 100644 index 00000000000000..40e04bb93a96de --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java @@ -0,0 +1,227 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.scan.ConnectorScanRange; + +import org.apache.hudi.common.model.FileSlice; +import org.apache.hudi.common.model.HoodieBaseFile; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests the OFFLINE-verifiable core of the INC-3 incremental {@code planScan} wiring: the pure static helpers + * {@link HudiScanPlanProvider#incrementalRanges} (COW/MOR routing + fallback degrade) and + * {@link HudiScanPlanProvider#buildMorRange} (file-slice → JNI range mapping). Both are exercised without a + * live {@code HoodieTableMetaClient} — {@code incrementalRanges} via a FAKE {@link IncrementalRelation}, and + * {@code buildMorRange} via a hand-built {@link FileSlice}. + * + *

    Coverage scope (Rule 12 — no over-claim). These cover the routing + mapping DECISIONS only. Building + * the real relation (which does eager timeline/metadata/filesystem I/O) and the actual file SELECTION + * (commit-range, write-stat mapping, {@code fs.exists} full-table-scan probes, the completion-time END axis) are + * e2e-only (design §5) — same boundary as {@link HudiIncrementalRelationTest}. Row-level filtering of the read to + * the {@code (begin, end]} window is a LATER step (an FE-side synthetic {@code _hoodie_commit_time} predicate), + * NOT this file-selection step. + */ +public class HudiIncrementalPlanScanTest { + + private static final List YEAR_MONTH = Arrays.asList("year", "month"); + private static final String END_TS = "20240101120000"; + private static final String BASE_PATH = "s3://b/t"; + private static final String INPUT_FORMAT = "org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat"; + private static final String SERDE = "org.apache.hudi.hadoop.realtime.HoodieParquetRealtimeInputFormat"; + + // ── fallback → degrade (Optional.empty), collect* NOT called ───────────────────────────────────────────── + + @Test + public void fallbackFullTableScanDegradesToSnapshotScanWithoutCollecting() { + // relation.fallbackFullTableScan()==true must yield Optional.empty() = the caller degrades to the normal + // latest-snapshot scan (legacy HudiScanNode.getSplits:470), and NEITHER collect method is invoked (both + // throw to prove they are not called). Kills a mutation inverting the fallback check. + FakeRelation fallback = new FakeRelation(); + fallback.fallback = true; + fallback.collectSplitsThrows = true; + fallback.collectFileSlicesThrows = true; + Optional> result = HudiScanPlanProvider.incrementalRanges( + fallback, /*isCow*/ true, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Assertions.assertFalse(result.isPresent(), + "fallbackFullTableScan must signal degrade to the snapshot scan (Optional.empty)"); + } + + // ── COW → collectSplits (native ranges returned verbatim); force_jni IGNORED for COW ───────────────────── + + @Test + public void cowRoutesToCollectSplitsAndNeverCallsCollectFileSlices() { + // A COW incremental read returns the relation's native collectSplits() ranges directly and must NEVER + // call collectFileSlices() (which throws on a COW relation = the shape contract). Kills a mutation that + // routes COW through the MOR branch (which would reproduce the legacy UnsupportedOperationException crash). + HudiScanRange native1 = new HudiScanRange.Builder().path("s3://b/t/f1.parquet").fileFormat("parquet").build(); + FakeRelation cow = new FakeRelation(); + cow.splits = Collections.singletonList(native1); + cow.collectFileSlicesThrows = true; + Optional> result = HudiScanPlanProvider.incrementalRanges( + cow, /*isCow*/ true, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals(1, result.get().size()); + Assertions.assertSame(native1, result.get().get(0), "COW ranges must be the relation's collectSplits output"); + } + + @Test + public void cowIncrementalIgnoresForceJniAndStaysNative() { + // The signed graceful deviation: under force_jni a COW incremental read STILL reads native (collectSplits), + // instead of legacy's route to collectFileSlices() on a COW relation → UnsupportedOperationException. So + // force_jni=true must NOT change COW routing. collectFileSlices throws to prove it is not taken. + HudiScanRange native1 = new HudiScanRange.Builder().path("s3://b/t/f1.parquet").fileFormat("parquet").build(); + FakeRelation cow = new FakeRelation(); + cow.splits = Collections.singletonList(native1); + cow.collectFileSlicesThrows = true; + Optional> result = HudiScanPlanProvider.incrementalRanges( + cow, /*isCow*/ true, /*forceJni*/ true, BASE_PATH, INPUT_FORMAT, SERDE, + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Assertions.assertTrue(result.isPresent()); + Assertions.assertSame(native1, result.get().get(0), + "COW incremental must stay native even under force_jni (never call collectFileSlices)"); + } + + // ── MOR → collectFileSlices mapped to JNI ranges at endTs; collectSplits NEVER called ──────────────────── + + @Test + public void morRoutesToCollectFileSlicesMappedAtEndTs() { + // A MOR incremental read maps each collectFileSlices() slice to a JNI range at the resolved window END + // (relation.getEndTs()), with per-slice partition values from the slice's OWN partition path. It must + // NEVER call collectSplits() (which throws on a MOR relation). Uses force_jni so a base-only (no-log) + // slice stays on the JNI reader, exercising the instantTime=endTs stamping. + FileSlice slice = baseOnlySlice("year=2024/month=01", + "s3://b/t/year=2024/month=01/fileid-1_0_20240101000000.parquet"); + FakeRelation mor = new FakeRelation(); + mor.fileSlices = Collections.singletonList(slice); + mor.collectSplitsThrows = true; + Optional> result = HudiScanPlanProvider.incrementalRanges( + mor, /*isCow*/ false, /*forceJni*/ true, BASE_PATH, INPUT_FORMAT, SERDE, + Arrays.asList("c1"), Arrays.asList("int"), YEAR_MONTH); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals(1, result.get().size()); + HudiScanRange range = (HudiScanRange) result.get().get(0); + Assertions.assertEquals("jni", range.getFileFormat(), "force_jni keeps the no-log MOR slice on JNI"); + Assertions.assertEquals(END_TS, range.getProperties().get("hudi.instant_time"), + "the JNI merge instant must be the resolved window END (getEndTs), not the latest instant"); + Map partValues = range.getPartitionValues(); + Assertions.assertEquals("2024", partValues.get("year")); + Assertions.assertEquals("01", partValues.get("month"), + "partition values must be parsed from the slice's own partition path against the table-config fields"); + } + + @Test + public void morEmptySliceListYieldsEmptyRangesButTakesMorBranch() { + // A MOR relation with no slices returns Optional.of([]) — present (not degrade) but empty. collectSplits + // throws to prove the MOR (collectFileSlices) branch was taken, not the COW branch. + FakeRelation mor = new FakeRelation(); + mor.fileSlices = Collections.emptyList(); + mor.collectSplitsThrows = true; + Optional> result = HudiScanPlanProvider.incrementalRanges( + mor, /*isCow*/ false, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Assertions.assertTrue(result.isPresent()); + Assertions.assertTrue(result.get().isEmpty()); + } + + // ── buildMorRange mapping (hand-built FileSlice) ───────────────────────────────────────────────────────── + + @Test + public void buildMorRangeStampsJniInstantAndMetadataForForceJniSlice() { + // buildMorRange on a base-only slice with force_jni: a JNI range carrying instantTime=jniInstant + serde + + // input format + base path + column lists. Pins the exact JNI metadata BE needs. + FileSlice slice = baseOnlySlice("year=2024/month=01", + "s3://b/t/year=2024/month=01/fileid-1_0_20240101000000.parquet"); + Map partValues = HudiScanPlanProvider.parsePartitionValues( + slice.getPartitionPath(), YEAR_MONTH); + HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, partValues, END_TS, /*forceJni*/ true, + BASE_PATH, INPUT_FORMAT, SERDE, Arrays.asList("c1"), Arrays.asList("int")); + Assertions.assertEquals("jni", range.getFileFormat()); + Assertions.assertEquals(END_TS, range.getProperties().get("hudi.instant_time")); + Assertions.assertEquals(SERDE, range.getProperties().get("hudi.serde")); + Assertions.assertEquals(BASE_PATH, range.getProperties().get("hudi.base_path")); + } + + @Test + public void buildMorRangeDowngradesNoLogSliceToNativeWithoutForceJni() { + // A base-only (no delta log) slice WITHOUT force_jni reads natively: format from the base file suffix, and + // NO JNI instant metadata (the native reader needs only the path). Guards the native downgrade parity. + FileSlice slice = baseOnlySlice("year=2024/month=01", + "s3://b/t/year=2024/month=01/fileid-1_0_20240101000000.parquet"); + HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, Collections.emptyMap(), END_TS, + /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, + Arrays.asList("c1"), Arrays.asList("int")); + Assertions.assertEquals("parquet", range.getFileFormat(), + "a no-log slice without force_jni must downgrade to the native parquet reader"); + } + + // ── helpers ──────────────────────────────────────────────────────────────────────────────────────────── + + /** A MOR file slice with a single base file and no delta logs. */ + private static FileSlice baseOnlySlice(String partitionPath, String baseFilePath) { + FileSlice slice = new FileSlice(partitionPath, "20240101000000", "fileid-1"); + slice.setBaseFile(new HoodieBaseFile(baseFilePath)); + return slice; + } + + /** A recording fake {@link IncrementalRelation}: canned outputs, throws on the shape it should not serve. */ + private static final class FakeRelation implements IncrementalRelation { + private List splits = new ArrayList<>(); + private List fileSlices = new ArrayList<>(); + private boolean fallback; + private String endTs = END_TS; + private boolean collectSplitsThrows; + private boolean collectFileSlicesThrows; + + @Override + public List collectFileSlices() { + if (collectFileSlicesThrows) { + throw new UnsupportedOperationException("collectFileSlices must not be called on this route"); + } + return fileSlices; + } + + @Override + public List collectSplits() { + if (collectSplitsThrows) { + throw new UnsupportedOperationException("collectSplits must not be called on this route"); + } + return splits; + } + + @Override + public boolean fallbackFullTableScan() { + return fallback; + } + + @Override + public String getEndTs() { + return endTs; + } + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java index dcc3d2d93ecd14..ef41bcc74e7ba9 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java @@ -23,6 +23,10 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + /** * Tests the OFFLINE-verifiable surface of the ported {@code @incr} IncrementalRelation family (INC-2): the pure * fail-loud guards extracted onto {@link IncrementalRelation} and the {@link EmptyIncrementalRelation}. Each @@ -151,6 +155,38 @@ public void notFullTableScanIsNoOp() { Assertions.assertDoesNotThrow(() -> IncrementalRelation.checkNotFullTableScan(false)); } + // ── hollow-commit policy → HollowCommitHandling enum (byte-faithful to legacy COW:74-75 / MOR:76-77) ───── + + @Test + public void hollowCommitHandlingDefaultsToFailWhenPolicyAbsent() { + // No policy param -> FAIL (legacy getOrDefault(policyKey, "FAIL")). This is the default axis + // (requested-time); guards a mutation that drops the "FAIL" default (which would NPE valueOf(null)). + Assertions.assertEquals(HollowCommitHandling.FAIL, + IncrementalRelation.hollowCommitHandling(Collections.emptyMap())); + } + + @Test + public void hollowCommitHandlingReadsUseTransitionTime() { + // The one non-default value that matters: USE_TRANSITION_TIME switches the relation's own file selection + // to the completion-time axis (COW:93-97 / MOR:100-104), which must match the END axis resolveIncremental + // resolved on the SAME policy. Guards dropping/misreading the policy key. + Map params = new HashMap<>(); + params.put("hoodie.read.timeline.holes.resolution.policy", "USE_TRANSITION_TIME"); + Assertions.assertEquals(HollowCommitHandling.USE_TRANSITION_TIME, + IncrementalRelation.hollowCommitHandling(params)); + } + + @Test + public void hollowCommitHandlingThrowsOnBogusPolicyLikeLegacy() { + // A bogus policy value reaches HollowCommitHandling.valueOf and THROWS IllegalArgumentException — legacy + // parity (legacy also valueOf's it, COW:74-75 / MOR:76-77). Same terminal error, one phase later (at + // planScan rather than the relation ctor). Guards a mutation that would swallow the bad value. + Map params = new HashMap<>(); + params.put("hoodie.read.timeline.holes.resolution.policy", "NOT_A_POLICY"); + Assertions.assertThrows(IllegalArgumentException.class, + () -> IncrementalRelation.hollowCommitHandling(params)); + } + // ── empty relation (empty completed timeline) ──────────────────────────────────────────────────────────── @Test diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java index e2ac2b227e8c19..31649cc8c6fc70 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java @@ -207,6 +207,67 @@ public void applySnapshotLeavesLatestPinUnchanged() { Assertions.assertNull(result.getEndInstant()); } + // ── raw @incr option-param threading (glob / fallback / policy → handle) ──────────────────────────── + + @Test + public void resolveThreadsRawIncrParamsOntoHandleExcludingWindowCarriers() { + // The raw @incr option params (glob / fallback / hollow policy) must ride the FE-internal transport to + // the handle so planScan can feed them to the ported relations; the begin/end WINDOW carriers (their own + // "hudi.incremental-*" keys) must NOT leak into that opt-param map. Guards both the copy AND the namespace + // isolation (a mutation that dropped the prefix filter would pollute the relations' optParams with the + // internal carriers). + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + Map params = window("20240101000000", "20240101120000"); + params.put("hoodie.datasource.read.incr.path.glob", "*/2024/*"); + params.put("hoodie.datasource.read.incr.fallback.fulltablescan.enable", "true"); + params.put("hoodie.read.timeline.holes.resolution.policy", "USE_TRANSITION_TIME"); + HudiTableHandle pinned = resolveAndApply(md, params); + + Map incr = pinned.getIncrementalParams(); + Assertions.assertEquals("*/2024/*", incr.get("hoodie.datasource.read.incr.path.glob")); + Assertions.assertEquals("true", incr.get("hoodie.datasource.read.incr.fallback.fulltablescan.enable")); + Assertions.assertEquals("USE_TRANSITION_TIME", incr.get("hoodie.read.timeline.holes.resolution.policy")); + // The resolved window rides begin/endInstant separately; the FE-internal carriers must not leak into the + // opt-param map the relations read. + Assertions.assertFalse(incr.containsKey("hudi.incremental-begin"), + "the FE-internal begin carrier must not appear in the relations' opt-param map"); + Assertions.assertFalse(incr.containsKey("hudi.incremental-end"), + "the FE-internal end carrier must not appear in the relations' opt-param map"); + // beginTime/endTime aliases are carried verbatim (legacy passed the full optParams); they are inert for + // the relations (which read only glob/fallback/policy) but round-trip fidelity is asserted. + Assertions.assertEquals("20240101000000", incr.get("beginTime")); + Assertions.assertEquals("20240101120000", incr.get("endTime")); + // And the window itself still lands on the dedicated fields. + Assertions.assertEquals("20240101000000", pinned.getBeginInstant()); + Assertions.assertEquals("20240101120000", pinned.getEndInstant()); + } + + @Test + public void nonIncrementalPinsLeaveIncrementalParamsEmpty() { + // A FOR TIME AS OF pin and a plain (latest) handle must both carry an EMPTY incrementalParams — only an + // @incr read populates it. Guards accidental cross-wiring in applySnapshot. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + ConnectorMvccSnapshot ttPin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.timestamp("2024-01-01 12:00:00", false)).orElseThrow(AssertionError::new); + HudiTableHandle ttStamped = (HudiTableHandle) md.applySnapshot(null, partitioned(), ttPin); + Assertions.assertTrue(ttStamped.getIncrementalParams().isEmpty(), + "a FOR TIME AS OF pin must not populate incrementalParams"); + Assertions.assertTrue(partitioned().getIncrementalParams().isEmpty(), + "a plain handle has empty incrementalParams"); + } + + @Test + public void toBuilderRoundTripsIncrementalParams() { + // Guards the toBuilder() copy line for incrementalParams: a dropped copy would silently lose glob/ + // fallback/policy when applyFilter/applySnapshot rebuild the handle, so the relations would read defaults. + Map params = new HashMap<>(); + params.put("hoodie.datasource.read.incr.path.glob", "*/x/*"); + HudiTableHandle original = new HudiTableHandle.Builder("db", "t", "s3://b/t", "MERGE_ON_READ") + .beginInstant("20240101000000").endInstant("20240101120000").incrementalParams(params).build(); + HudiTableHandle copy = original.toBuilder().build(); + Assertions.assertEquals("*/x/*", copy.getIncrementalParams().get("hoodie.datasource.read.incr.path.glob")); + } + // ── handle field round-trip ───────────────────────────────────────────────────────────────────────── @Test From 2f7b3741a0853e893309ed55acefe85c91f98cc8 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 14:23:26 +0800 Subject: [PATCH 119/334] [doc](catalog) HMS cutover hudi HD-C3 INC-3 DONE; next = INC-4 (neutral synthetic-predicate SPI) INC-3 (incremental planScan wiring, 0fea5bc6ba4) done: planScan selects @incr files via the ported IncrementalRelation family, degrades to snapshot on fallback, threads raw @incr opt-params onto the handle; force_jni+COW made graceful (user-signed); RO-as-RT subsumed by metaClient.getTableType(); row-level filter deferred to INC-4. 4-dimension adversarial review: 0 findings. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 3136637b30ea75..195db1dec90edb 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。下一步 = HD-C3 余 3 步（INC-3 增量 planScan → INC-4 中立谓词 SPI+fe-core 规则+反向转换器+5 可见元数据列 → INC-5 翻闸接线）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。下一步 = HD-C3 余 2 步（INC-4 中立谓词 SPI+fe-core 规则+反向转换器+5 可见元数据列 → INC-5 翻闸接线）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -90,8 +90,9 @@ - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 - **✅ HD-C3 INC-1 DONE（`9327261ec52`，休眠）= @incr 窗口解析脊柱**（recon `wf_3398c7fb-bfb` 4 读者+parity critic〔scope critic StructuredOutput 上限失败，已用我亲手字节码核对补〕 + 对抗复审 `wf_13bea565-a47` 4 维〔各 finding 逐条 refute〕）。连接器侧扩展 HD-C2 脊柱：`resolveTimeTravel(INCREMENTAL)` **单一 locus** `resolveIncremental` 解析 (begin,end]〔begin 必填逐字 fail-loud、`"earliest"`→`"000"`、end 缺省→最新 completed instant、`"latest"`→最新〔测**已解析 end 值**=legacy COW 形，天然规避 `MOR:92` 死码 bug〕、空时间线→`(000,000]` 免 begin 检查〕→ **NON-EMPTY property-only pin**（snapshotId/schemaId inert：fe-core INCREMENTAL `loadSnapshot` 分支自列 latest 分区+latest schema、只读窗口属性）；`applySnapshot` 钉 `begin/endInstant`（与 FOR TIME `queryInstant` 载体互斥、保 prunedPartitionPaths）。新 `HudiTableHandle.begin/endInstant` + `HudiScanPlanProvider.latestCompletedInstantTime(String)`（从 `latestCompletedInstant` 字节等价抽出）。**复审关键裁决**：①**blocker（COW 时间线 byte-parity）被 hudi-1.0.2 字节码驳倒**——单 `getCommitsAndCompactionTimeline()` 按类型解析到 legacy 逐类型时间线（COW→`getCommitAndReplaceTimeline`={commit,replacecommit,clustering}=legacy COW 的 `metaClient.getCommitTimeline()`〔该 metaClient 方法**非** commit-only，委派给 getCommitAndReplaceTimeline〕；MOR→`getWriteTimeline`），review+adversarial-verify 都把 metaClient 方法与 `BaseHoodieTimeline.getCommitTimeline()` 时间线方法混淆了；②major/minor/nit 均**按设计延后**（见下）。`HudiIncrementalTest` +11、fe-connector-hudi 96 测 + checkstyle 0 + import gate 净。**⚠延后（已文档化 fail-loud、非静默丢，INC-2/4 须接）**：(a) **`populateMetaFields()` fail-loud→INC-2**（"meta fields disabled" 拒，结构上属 relation 构造器=INC-2 移植；INC-1 窗口解析不查，INC-2 移植时在 planScan 强制、同为查询规划期失败；INC-2 须字节忠实移植+e2e 加 meta-fields-disabled fixture）；(b) **hollow-commit `USE_TRANSITION_TIME`（completionTime）+ hollow 抛→INC-2/INC-3**（默认 FAIL 策略用 requestedTime，非默认策略绑 relation 文件选择）；(c) **原生 `hoodie.datasource.read.*.instanttime` 窗口键→INC-4**（INC-1 读标准 `@incr` 别名 beginTime/endTime，是否也收原生 hoodie 键归 INC-4）。 - - **✅ HD-C3 INC-2 DONE（`d64a25f28fc`，休眠）= IncrementalRelation 家族移植**（HEAD-grounded recon `wf_50a59f91-14e` 3 读者〔窗口协调/边界/API+可测性〕+完整性 critic + 5 维对抗复审 `wf_b247a285-d64`〔byte-parity/dormancy-iron-rule/guard-correctness 三镜头 **0 发现**；test-strength 3 test/doc-honesty finding 已修；mor92-and-axis 镜头 API stall 死→crux 亲手 grep+读自证〕）。COW/MOR/Empty + `IncrementalRelation` 接口移植进 `fe-connector-hudi`（连接器内部、package-private）。**用户签字方案 A（单一窗口权威，见本 session 决策）**：移植关系**消费句柄已解析的 (begin,end] 窗口只挑文件、自己不再解析**（构造器删窗口解析 prologue）→ 死码 `MORIncrementalRelation:92` sentinel bug **自然消失**、文件集永不与 INC-4 `_hoodie_commit_time` 行过滤的 end 打架。split 型 re-home `spi.Split`/`HudiSplit`→`HudiScanRange`（丢 `LocationPath`/`TableFormatType`/`HudiPartitionUtils` fe-core 型）；COW `collectSplits`→`List`（path + length/fileSize 取 `HoodieWriteStat.getFileSizeInBytes()`、format 走 `detectFileFormat`、分区值走连接器 Map `parsePartitionValues` on `metaClient.getTableConfig().getPartitionFields()`=legacy COW:212 平价，**非** HMS handle keys）；MOR `collectFileSlices`→`List` 逐字；archived-flag 起字节忠实 legacy，仅删窗口 prologue。INC-1 延后守卫作 `IncrementalRelation` 共享静态、逐字文案 re-type 成 `DorisConnectorException`：`checkIncrementalMetaFields`（meta-fields-disabled 首守卫、非空时间线 only 靠上游 Empty 路由）/`checkStateTransitionTimeFullTableScan`/`decideArchivalFullTableScan`/`checkNotFullTableScan`。**`resolveIncremental` END 轴按策略**：`hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME` 下 default/`latest` end 走 completion-time 轴（新 `latestCompletedInstantTime(mc, useCompletionTime)` 重载：getCompletionTime vs requestedTime）→ 单一 handle-stamped end 与关系 `findInstantsInRangeByCompletionTime` 选片同轴、无末批漏读；默认策略字节不变、休眠（planScan 只读 `queryInstant` 从不读 begin/endInstant）。`getHoodieParams`/`getStartTs` 丢弃（FE-filter 模型下已死）。全绿（`HudiIncrementalRelationTest` 13 + `HudiIncrementalTest` 12 + 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**⚠ e2e-only（休眠单测只覆盖纯守卫+Empty；构造器 timeline/metadata/fs 选片 + completion-time 轴须 e2e，设计 §5 已加 meta-fields-disabled + USE_TRANSITION_TIME fixture）**。**下一步 = INC-3 增量 planScan（接线 COW/MOR/RO-as-RT 单点/fallback，不发 hoodie.* 到 BE）**。 - - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **下一步 INC-3** 增量 `planScan`（COW/MOR/RO-as-RT 单点/fallback；**不发** hoodie.* 到 BE）→ INC-4 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列（**+ INC-1 延后的原生 hoodie.* 窗口键决策**）→ INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C3 INC-2 DONE（`d64a25f28fc`，休眠）= IncrementalRelation 家族移植**（HEAD-grounded recon `wf_50a59f91-14e` 3 读者〔窗口协调/边界/API+可测性〕+完整性 critic + 5 维对抗复审 `wf_b247a285-d64`〔byte-parity/dormancy-iron-rule/guard-correctness 三镜头 **0 发现**；test-strength 3 test/doc-honesty finding 已修；mor92-and-axis 镜头 API stall 死→crux 亲手 grep+读自证〕）。COW/MOR/Empty + `IncrementalRelation` 接口移植进 `fe-connector-hudi`（连接器内部、package-private）。**用户签字方案 A（单一窗口权威，见本 session 决策）**：移植关系**消费句柄已解析的 (begin,end] 窗口只挑文件、自己不再解析**（构造器删窗口解析 prologue）→ 死码 `MORIncrementalRelation:92` sentinel bug **自然消失**、文件集永不与 INC-4 `_hoodie_commit_time` 行过滤的 end 打架。split 型 re-home `spi.Split`/`HudiSplit`→`HudiScanRange`（丢 `LocationPath`/`TableFormatType`/`HudiPartitionUtils` fe-core 型）；COW `collectSplits`→`List`（path + length/fileSize 取 `HoodieWriteStat.getFileSizeInBytes()`、format 走 `detectFileFormat`、分区值走连接器 Map `parsePartitionValues` on `metaClient.getTableConfig().getPartitionFields()`=legacy COW:212 平价，**非** HMS handle keys）；MOR `collectFileSlices`→`List` 逐字；archived-flag 起字节忠实 legacy，仅删窗口 prologue。INC-1 延后守卫作 `IncrementalRelation` 共享静态、逐字文案 re-type 成 `DorisConnectorException`：`checkIncrementalMetaFields`（meta-fields-disabled 首守卫、非空时间线 only 靠上游 Empty 路由）/`checkStateTransitionTimeFullTableScan`/`decideArchivalFullTableScan`/`checkNotFullTableScan`。**`resolveIncremental` END 轴按策略**：`hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME` 下 default/`latest` end 走 completion-time 轴（新 `latestCompletedInstantTime(mc, useCompletionTime)` 重载：getCompletionTime vs requestedTime）→ 单一 handle-stamped end 与关系 `findInstantsInRangeByCompletionTime` 选片同轴、无末批漏读；默认策略字节不变、休眠（planScan 只读 `queryInstant` 从不读 begin/endInstant）。`getHoodieParams`/`getStartTs` 丢弃（FE-filter 模型下已死）。全绿（`HudiIncrementalRelationTest` 13 + `HudiIncrementalTest` 12 + 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**⚠ e2e-only（休眠单测只覆盖纯守卫+Empty；构造器 timeline/metadata/fs 选片 + completion-time 轴须 e2e，设计 §5 已加 meta-fields-disabled + USE_TRANSITION_TIME fixture）**。**下一步 = INC-4（中立 synthetic-predicate SPI + fe-core 分析期规则 + 反向 ConnectorExpression→Nereids 转换器 + 暴露 5 可见 `_hoodie_*` 元数据列）**。 + - **✅ HD-C3 INC-3 DONE（`0fea5bc6ba4`，休眠）= 增量 planScan 接线**（对抗复审 `wf_99c5b718-c11` 4 维〔snapshot byte-parity / incremental-correctness / dormancy-transport / test-intent〕+ 逐条 adversarial-verify **0 发现**）。`HudiScanPlanProvider.planScan` 加 `getBeginInstant()!=null` 增量支路（置于 schema 解析后、fsView 构建前，纯早返；非增量读〔含 FOR TIME AS OF〕字节不变）：`buildIncrementalRelation` 按 `isCow`(=`metaClient.getTableType()==COPY_ON_WRITE`，权威) 建 COW/MOR 关系（IOException 重型为 `DorisConnectorException`）；`incrementalRanges` **纯静态**路由（fake 关系可离线测）——`fallbackFullTableScan()`→`Optional.empty()`→降级到最新快照扫描（legacy `getSplits:470`，读 latest；两载体互斥故 @incr 句柄无 queryInstant）；COW→`collectSplits()` 原生；MOR→`collectFileSlices()` 逐片经**抽出的** `buildMorRange`（从 `collectMorSplits` 内联 lambda 提取、快照路径字节不变）转 JNI range 钉 `getEndTs()`（窗口 END=BE 合并时刻，非 latest）、分区值按 `metaClient.getTableConfig().getPartitionFields()` 逐片解析（=legacy `HudiScanNode:391-393`，**非** HMS handle 键）。**RO-as-RT 翻转 = 被 metaClient.getTableType() 天然覆盖、不移植、不需 serde 参数上句柄**（旧翻转只为补偿 legacy 按 hive inputFormat 把 MOR `_ro` 门面误判 COW；连接器读 hoodie.properties 直接得 MERGE_ON_READ）。**用户签字方案 B（graceful，2026-07-09）**：`force_jni+COW+增量` 旧代码 `collectFileSlices()` on COW relation→`UnsupportedOperationException` 崩溃，改按**关系类型**路由→COW 增量恒原生（忽略 force_jni），不崩溃；MOR 仍经 buildMorRange 尊重 force_jni（无 log 片留 JNI）。**参数穿透**：`resolveIncremental` 把原始 @incr 选项（glob/fallback/hollow-policy）复制进 MVCC 快照属性（前缀 `hudi.incr-opt.`、跳 null 值），`applySnapshot` 前缀剥离重建进**新** `HudiTableHandle.incrementalParams`（**BE 安全**：快照属性只经 `PluginDrivenScanNode.applyMvccSnapshotPin`→`applySnapshot` 消费、从不入 BE thrift〔亲手 + 复审双证〕；begin/end 载体走 `hudi.incremental-*` 不同命名空间不混入；`toBuilder` 复制）；planScan 由 `IncrementalRelation.hollowCommitHandling(optParams)`=`valueOf(getOrDefault(policyKey,"FAIL"))` 派生策略，与 `resolveIncremental` END 轴 `INCR_HOLLOW_POLICY_KEY` 同源（文件集与窗口 END 同轴）。**行级过滤（`_hoodie_commit_time` 谓词）= INC-4；本步只挑 FILE，裸 @incr 会多读直到 INC-4（休眠无影响）**。全绿（`HudiIncrementalPlanScanTest` 7〔fake IncrementalRelation + 手造 FileSlice：fallback 降级/COW 路由不碰 collectFileSlices/COW 忽略 force_jni/MOR 钉 endTs+逐片分区值/buildMorRange JNI+native 降级〕+ `HudiIncrementalTest` 15〔+3 参数穿透+命名空间隔离+非增量 pin 空 incrementalParams〕+ `HudiIncrementalRelationTest` 16〔+3 hollowCommitHandling FAIL 默认/USE_TRANSITION_TIME/bogus 抛〕+ 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**下一步 = INC-4**。 + - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **下一步 INC-4** 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列（**+ INC-1 延后的原生 hoodie.* 窗口键决策**）→ INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From d55fd2cadb037f5e861969558204a385f3d5da02 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 15:30:11 +0800 Subject: [PATCH 120/334] [feat](catalog) HMS cutover hudi HD-C3 INC-4a: neutral synthetic-predicate SPI + hudi override + hive delegation (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Connector-side half of the incremental-read row-correctness gap closure. A COW base file rewritten inside an @incr (begin, end] window also carries forward older-commit rows, so file selection alone is not enough — the read needs a row-level `_hoodie_commit_time > begin AND <= end` filter. Legacy delivered that via a source-specific fe-core injection (CheckPolicy + LogicalHudiScan); this re-homes it into a connector-neutral SPI (the fe-core rule + reverse converter land in INC-4b). - ConnectorMetadata.getSyntheticScanPredicates(session, handle, snapshot): new parent-first default returning List.of() (a connector "residual predicate", Trino-aligned). iceberg/paimon/jdbc inherit empty -> plans byte-identical. Takes the resolved ConnectorMvccSnapshot (a connector-api type) — NOT fe-core's analysis.TableScanParams, which the connector-api module cannot import and the import gate forbids. - HudiConnectorMetadata override: reads the SAME resolved (begin, end] window off the snapshot props that applySnapshot consumes (single window authority — the file set and the row filter can never diverge), emits `_hoodie_commit_time > begin AND <= end` as two flat STRING-typed conjuncts (lexicographic instant compare, byte-faithful to LogicalHudiScan.generateIncrementalExpression). Empty for any non-incremental pin (plain / FOR TIME AS OF). - HiveConnectorMetadata delegating override: routes a foreign (iceberg/hudi) handle to its owning sibling (mirrors resolveTimeTravel/applySnapshot) so a hudi-on-HMS @incr read gets its window filter; without it the foreign handle would inherit the empty default -> silent out-of-window over-read. - getTableAvroSchema() -> getTableAvroSchema(true) in HudiConnectorMetadata + HudiScanPlanProvider (lockstep): explicit `true` = legacy parity, byte-identical for the common populate.meta.fields=true table, and makes _hoodie_commit_time a robust visible name-bindable column (the row-filter's binding precondition) regardless of the config default. Dormant: hms not in SPI_READY_TYPES; hudi getTableHandle divert (which routes a hudi handle to the hudi sibling) is a later substep, so the hive delegation is not yet exercised for hudi. Tests: HudiIncrementalTest +3 (range emit / TT-pin empty / plain+null empty), HudiSchemaParityTest +1 (_hoodie_commit_time visible STRING), sibling-delegation +1 divert + completeness-lock entry + hive-handle empty. fe-connector-hudi 127 / fe-connector-hive 236 green, checkstyle 0, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/api/ConnectorMetadata.java | 23 ++++++++ .../connector/hive/HiveConnectorMetadata.java | 13 +++++ ...onnectorMetadataSiblingDelegationTest.java | 21 ++++++- .../connector/hudi/HudiConnectorMetadata.java | 57 ++++++++++++++++++- .../connector/hudi/HudiScanPlanProvider.java | 5 +- .../connector/hudi/HudiIncrementalTest.java | 55 ++++++++++++++++++ .../connector/hudi/HudiSchemaParityTest.java | 24 ++++++++ 7 files changed, 194 insertions(+), 4 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java index cdd96016893841..4ba5a7ca2d8033 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java @@ -22,10 +22,12 @@ import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; import java.io.Closeable; import java.io.IOException; import java.util.Collections; +import java.util.List; import java.util.Map; import java.util.Optional; import java.util.OptionalLong; @@ -164,6 +166,27 @@ default ConnectorTableHandle applySnapshot(ConnectorSession session, return handle; // default: connectors without time-travel ignore the pin } + /** + * Returns extra scan-level predicates the engine MUST apply for {@code handle} at the pinned + * {@code snapshot} — a connector "residual predicate" the read cannot enforce by file selection alone. + * The canonical case is an incremental / CDC commit-time window: a rewritten base file also carries + * forward out-of-window rows, so a ROW-LEVEL commit-time filter is required for correctness. The engine + * reverse-converts each returned {@link ConnectorExpression} into its native predicate and wraps a filter + * over the scan, binding column references to the connector's own (visible) output columns by name. + * + *

    The predicate is expressed in the connector-neutral {@link ConnectorExpression} pushdown grammar, + * NOT a source-specific shape — the engine NEVER discriminates by connector here; it applies whatever the + * connector returns. This mirrors the engine-agnostic residual-predicate model.

    + * + *

    The default returns an EMPTY list: a connector with no synthetic scan predicate adds nothing, so the + * plan is byte-identical. iceberg/paimon/jdbc/... inherit this empty default; only a connector that opts in + * (e.g. hudi incremental read) returns a non-empty list, and only for the scans that need it.

    + */ + default List getSyntheticScanPredicates(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { + return List.of(); // default: connectors without a residual scan predicate add nothing + } + /** * Threads a per-group rewrite file scope into the table handle BEFORE {@code planScan}, so the * distributed {@code rewrite_data_files} driver can scope each per-group INSERT-SELECT scan to only the diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 6f82a786706ce4..9f8e1b23e748ae 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1107,6 +1107,19 @@ public ConnectorTableHandle applySnapshot(ConnectorSession session, ConnectorTab return handle; } + @Override + public List getSyntheticScanPredicates(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { + if (!(handle instanceof HiveTableHandle)) { + // Route a foreign (iceberg / hudi) handle to its owning sibling so a hudi-on-HMS @incr read gets + // its row-level `_hoodie_commit_time` window filter. Without this the foreign handle would inherit + // the empty SPI default -> no filter -> out-of-window rows leak (a SILENT correctness bug). + return siblingMetadata(session, handle).getSyntheticScanPredicates(session, handle, snapshot); + } + // Plain hive has no synthetic scan predicate (SPI default empty). + return List.of(); + } + @Override public ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, ConnectorTableHandle handle, Set rawDataFilePaths) { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index ab99c1fe178ffb..27613586e55a18 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorTableStatistics; +import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.ddl.BranchChange; import org.apache.doris.connector.api.ddl.ConnectorColumnPosition; @@ -40,6 +41,7 @@ import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.pushdown.ConnectorFilterConstraint; import org.apache.doris.connector.api.pushdown.FilterApplicationResult; @@ -130,6 +132,7 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { md.getMvccPartitionView(null, foreignHandle); md.resolveTimeTravel(null, foreignHandle, null); ConnectorTableHandle afterSnapshot = md.applySnapshot(null, foreignHandle, null); + List predicates = md.getSyntheticScanPredicates(null, foreignHandle, null); ConnectorTableHandle afterScope = md.applyRewriteFileScope(null, foreignHandle, Collections.emptySet()); ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(null, foreignHandle); List sysTables = md.listSupportedSysTables(null, foreignHandle); @@ -159,6 +162,9 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { "applyTopnLazyMaterialization must return the sibling's handle unmodified"); Assertions.assertSame(RecordingSiblingMetadata.SIBLING_HANDLE, sysHandle.orElse(null), "getSysTableHandle must return the sibling's sys-table handle unmodified"); + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_PREDICATES, predicates, + "getSyntheticScanPredicates must return the sibling's residual predicates unmodified — a " + + "hudi-on-HMS @incr read gets its row filter from the hudi sibling, not hive's empty default"); } @Test @@ -171,6 +177,8 @@ public void hiveHandleRunsHiveBranchAndNeverConsultsSibling() { Assertions.assertFalse(md.getMvccPartitionView(null, hive).isPresent(), "hive has no range partition view"); Assertions.assertFalse(md.resolveTimeTravel(null, hive, null).isPresent(), "hive has no time travel"); Assertions.assertSame(hive, md.applySnapshot(null, hive, null), "hive applySnapshot returns the handle"); + Assertions.assertTrue(md.getSyntheticScanPredicates(null, hive, null).isEmpty(), + "plain hive has no synthetic scan predicate"); Assertions.assertSame(hive, md.applyRewriteFileScope(null, hive, Collections.emptySet()), "hive applyRewriteFileScope returns the handle"); Assertions.assertSame(hive, md.applyTopnLazyMaterialization(null, hive), @@ -349,6 +357,8 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata static final ConnectorTransaction SIBLING_TXN = new NoOpConnectorTransaction(4243L, "ICEBERG"); static final long SENTINEL_SIZE = 4242L; static final long SENTINEL_SNAPSHOT_ID = 99L; + static final List SIBLING_PREDICATES = Collections.singletonList( + new ConnectorColumnRef("sibling-pred", ConnectorType.of("STRING"))); // The exact set + order of forwarded methods the foreign-handle test drives (a Rule-9 completeness lock: // dropping a guard, or adding one that should not forward, changes this list and fails the test). @@ -357,8 +367,8 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata "estimateDataSizeByListingFiles", "applyFilter", "listPartitionNames", "listPartitions", "beginQuerySnapshot", "getTableFreshness", "getPartitionFreshnessMillis", "dropTable", "truncateTable", "getTableSchemaAtSnapshot", "getMvccPartitionView", "resolveTimeTravel", - "applySnapshot", "applyRewriteFileScope", "applyTopnLazyMaterialization", "listSupportedSysTables", - "getSysTableHandle")); + "applySnapshot", "getSyntheticScanPredicates", "applyRewriteFileScope", + "applyTopnLazyMaterialization", "listSupportedSysTables", "getSysTableHandle")); // The exact set + order of ALTER-DDL / validate methods the foreign-handle write test drives (Rule-9 // completeness lock for §4.4 W1: dropping a guard, or adding one that should not forward, fails the test). @@ -490,6 +500,13 @@ public ConnectorTableHandle applySnapshot(ConnectorSession session, ConnectorTab return SIBLING_HANDLE; } + @Override + public List getSyntheticScanPredicates(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { + calls.add("getSyntheticScanPredicates"); + return SIBLING_PREDICATES; + } + @Override public ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, ConnectorTableHandle handle, Set rawDataFilePaths) { diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index a3e07bea8bda72..10a299bb3ceea8 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -126,6 +126,11 @@ public class HudiConnectorMetadata implements ConnectorMetadata { private static final String INCR_HOLLOW_POLICY_KEY = "hoodie.read.timeline.holes.resolution.policy"; private static final String INCR_STATE_TRANSITION_POLICY = "USE_TRANSITION_TIME"; + // The Hudi per-record commit-time meta column the synthetic incremental row filter references + // (HoodieRecord.COMMIT_TIME_METADATA_FIELD; lower-cased in the connector schema). Byte-faithful to legacy + // LogicalHudiScan.generateIncrementalExpression, which matched the scan-output slot by this exact name. + private static final String HUDI_COMMIT_TIME_COLUMN = "_hoodie_commit_time"; + private final HmsClient hmsClient; private final Map properties; // Runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + TCCL pin (see R4). @@ -471,6 +476,51 @@ public ConnectorTableHandle applySnapshot(ConnectorSession session, return handle; } + /** + * Supplies the ROW-LEVEL correctness filter for an {@code @incr} incremental read as a connector-neutral + * residual predicate (the neutral synthetic-predicate SPI). A COW base file rewritten inside the + * {@code (begin, end]} window also carries forward older-commit rows, so selecting the touched files is NOT + * enough — the engine must additionally apply {@code _hoodie_commit_time > begin AND _hoodie_commit_time + * <= end} at the row level. This is exactly the filter legacy injected via + * {@code LogicalHudiScan.generateIncrementalExpression}, re-homed here so fe-core stays source-agnostic + * (it reverse-converts the returned {@link ConnectorExpression}s and wraps a filter; it never branches on + * the connector). + * + *

    The window is read from the SAME resolved {@code snapshot} that {@link #applySnapshot} consumes + * ({@link #HUDI_INCREMENTAL_BEGIN_PROPERTY}/{@link #HUDI_INCREMENTAL_END_PROPERTY}, stamped ONCE by + * {@link #resolveIncremental}), so the row filter and the file-selection window are the single same + * resolution and can never diverge on an advancing timeline. Both bounds are STRING literals over a STRING + * column ref — lexicographic compare over fixed-width Hudi instants, byte-faithful to legacy (a numeric + * coercion would silently corrupt the ordering).

    + * + *

    Returns an EMPTY list for any non-incremental read (a plain latest read or {@code FOR TIME AS OF} + * pin carries no {@code (begin, end]} window), so those plans stay byte-identical.

    + */ + @Override + public List getSyntheticScanPredicates(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { + if (snapshot == null) { + return Collections.emptyList(); + } + Map properties = snapshot.getProperties(); + String begin = properties.get(HUDI_INCREMENTAL_BEGIN_PROPERTY); + String end = properties.get(HUDI_INCREMENTAL_END_PROPERTY); + if (begin == null || end == null) { + // Not an incremental read (plain / FOR TIME AS OF pins carry no window) -> no synthetic filter. + return Collections.emptyList(); + } + ConnectorType stringType = ConnectorType.of("STRING"); + org.apache.doris.connector.api.pushdown.ConnectorColumnRef commitTime = + new org.apache.doris.connector.api.pushdown.ConnectorColumnRef(HUDI_COMMIT_TIME_COLUMN, stringType); + ConnectorExpression lower = new ConnectorComparison( + ConnectorComparison.Operator.GT, commitTime, ConnectorLiteral.ofString(begin)); + ConnectorExpression upper = new ConnectorComparison( + ConnectorComparison.Operator.LE, commitTime, ConnectorLiteral.ofString(end)); + // Two flat conjuncts: fe-core ANDs them into one LogicalFilter (byte-faithful to legacy + // ImmutableSet.of(great, less)); no ConnectorAnd wrapper is needed. + return List.of(lower, upper); + } + // ========== ConnectorTableOps (partitions) ========== /** @@ -616,7 +666,12 @@ private List getSchemaFromMetaClient(String basePath) { .setBasePath(basePath) .build(); TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); - Schema avroSchema = schemaResolver.getTableAvroSchema(); + // include the 5 `_hoodie_*` meta columns (byte-faithful to legacy getHudiTableSchema, which passes + // `true` unconditionally at HiveMetaStoreClientHelper). The no-arg overload only includes them when + // hoodie.populate.meta.fields defaults true, so the explicit `true` (a) restores legacy SELECT * + // / DESCRIBE parity even for a populate.meta.fields=false table and (b) guarantees `_hoodie_commit_time` + // is a visible, name-bindable output column for the synthetic incremental row filter to reference. + Schema avroSchema = schemaResolver.getTableAvroSchema(true); return avroSchemaToColumns(avroSchema); } catch (Exception e) { LOG.warn("Failed to get schema from Hudi MetaClient for path '{}': {}", diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 94f45318f413c6..a81d7a712e6d5a 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -155,7 +155,10 @@ public List planScan( List columnTypes; try { TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); - Schema avroSchema = schemaResolver.getTableAvroSchema(); + // include the 5 `_hoodie_*` meta columns (explicit `true` = legacy parity, in lockstep with + // HudiConnectorMetadata.getSchemaFromMetaClient) so the JNI reader's column list matches the + // exposed schema; byte-identical for the common populate.meta.fields=true table. + Schema avroSchema = schemaResolver.getTableAvroSchema(true); columnNames = avroSchema.getFields().stream() .map(Schema.Field::name).collect(Collectors.toList()); columnTypes = avroSchema.getFields().stream() diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java index 31649cc8c6fc70..5556b2ab21d355 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalTest.java @@ -17,9 +17,14 @@ package org.apache.doris.connector.hudi; +import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -268,6 +273,56 @@ public void toBuilderRoundTripsIncrementalParams() { Assertions.assertEquals("*/x/*", copy.getIncrementalParams().get("hoodie.datasource.read.incr.path.glob")); } + // ── synthetic scan predicate (the neutral row-level @incr filter SPI) ─────────────────────────────── + + @Test + public void syntheticScanPredicatesEmitStringTypedCommitTimeWindow() { + // The @incr row filter is required because a COW base file rewritten inside the window ALSO carries + // forward out-of-window rows. The SPI reads the window off the SAME resolved pin applySnapshot consumes + // (single window authority, so file selection and the row filter can never diverge) and emits + // `_hoodie_commit_time > begin AND <= end` as two flat conjuncts, STRING-typed both sides for + // lexicographic instant compare. Byte-faithful to legacy LogicalHudiScan.generateIncrementalExpression. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + ConnectorMvccSnapshot pin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.incremental(window("20240101000000", "20240101120000"))) + .orElseThrow(AssertionError::new); + + List predicates = md.getSyntheticScanPredicates(null, partitioned(), pin); + + ConnectorColumnRef commitTime = new ConnectorColumnRef("_hoodie_commit_time", ConnectorType.of("STRING")); + Assertions.assertEquals(Arrays.asList( + new ConnectorComparison(ConnectorComparison.Operator.GT, commitTime, + ConnectorLiteral.ofString("20240101000000")), + new ConnectorComparison(ConnectorComparison.Operator.LE, commitTime, + ConnectorLiteral.ofString("20240101120000"))), + predicates, + "an @incr read must emit `_hoodie_commit_time > begin AND <= end`, STRING-typed on both sides"); + } + + @Test + public void syntheticScanPredicatesAreEmptyForTimeTravelPin() { + // A FOR TIME AS OF pin carries a queryInstant, NOT a (begin, end] window -> no row filter (its rows are + // already correct by file selection at the instant). Guards a mutation that would emit a bogus window. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + ConnectorMvccSnapshot ttPin = md.resolveTimeTravel(null, partitioned(), + ConnectorTimeTravelSpec.timestamp("2024-01-01 12:00:00", false)).orElseThrow(AssertionError::new); + Assertions.assertTrue(md.getSyntheticScanPredicates(null, partitioned(), ttPin).isEmpty(), + "a FOR TIME AS OF pin must not produce a synthetic row filter"); + } + + @Test + public void syntheticScanPredicatesAreEmptyForPlainAndNullPin() { + // The query-begin latest pin (beginQuerySnapshot) carries only a snapshotId, no window -> a plain read + // gets NO synthetic filter, so its plan is byte-identical to today. A null snapshot is likewise a no-op. + HudiConnectorMetadata md = metadata(stub(Optional.of(LATEST))); + ConnectorMvccSnapshot latestPin = + HudiConnectorMetadata.buildBeginQuerySnapshot(20240101120000000L).orElseThrow(AssertionError::new); + Assertions.assertTrue(md.getSyntheticScanPredicates(null, partitioned(), latestPin).isEmpty(), + "a plain latest pin must not produce a synthetic row filter"); + Assertions.assertTrue(md.getSyntheticScanPredicates(null, partitioned(), null).isEmpty(), + "a null snapshot must not produce a synthetic row filter"); + } + // ── handle field round-trip ───────────────────────────────────────────────────────────────────────── @Test diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java index 9ae752484e5efc..18241a11a4712e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java @@ -121,6 +121,30 @@ public void testColumnTypeStringsMirrorLegacyColTypes() { } } + @Test + public void metaCommitTimeColumnIsExposedVisibleAndStringForRowFilterBinding() { + // The synthetic incremental row filter references a scan-output slot named EXACTLY "_hoodie_commit_time" + // (byte-faithful to legacy LogicalHudiScan.generateIncrementalExpression). getTableAvroSchema(true) always + // carries the 5 `_hoodie_*` meta fields as nullable strings; avroSchemaToColumns must preserve the + // commit-time field as a VISIBLE STRING column with that exact lower-case name, or the filter would + // silently fail to bind and the incremental scan would over-read out-of-window rows. + String metaInclusive = + "{\"type\":\"record\",\"name\":\"hudi_t\",\"fields\":[" + + "{\"name\":\"_hoodie_commit_time\",\"type\":[\"null\",\"string\"],\"default\":null}," + + "{\"name\":\"id\",\"type\":\"long\"}" + + "]}"; + List columns = + HudiConnectorMetadata.avroSchemaToColumns(new Schema.Parser().parse(metaInclusive)); + ConnectorColumn commitTime = columns.stream() + .filter(c -> "_hoodie_commit_time".equals(c.getName())) + .findFirst().orElseThrow(() -> new AssertionError( + "_hoodie_commit_time must be exposed as a column for the incremental row filter to bind")); + Assertions.assertTrue(commitTime.isVisible(), + "_hoodie_commit_time must be VISIBLE (legacy SELECT * parity + the row-filter's slot binding)"); + Assertions.assertEquals(ConnectorType.of("STRING"), commitTime.getType(), + "_hoodie_commit_time must be STRING so the window compare is lexicographic over Hudi instants"); + } + @Test public void testTopLevelNameLoweredButNestedStructNamePreserved() { List columns = HudiConnectorMetadata.avroSchemaToColumns(schema()); From ea726300ba9a2437ddefe1c7e222778f0dc754e6 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 15:38:48 +0800 Subject: [PATCH 121/334] [feat](catalog) HMS cutover hudi HD-C3 INC-4b: reverse ConnectorExpression -> Nereids converter (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fe-core bounded reverse converter, the twin of the forward NereidsToConnectorExpressionConverter / ExprToConnectorExpressionConverter. Turns a connector-neutral residual predicate (from getSyntheticScanPredicates) back into a bound Nereids Expression, binding ConnectorColumnRef -> scan-output SlotReference by name. Connector-agnostic: it only maps node types; the column names + literal values come from the connector payload (zero source-specific logic). Used by the INC-4c analysis rule (next commit). TOTAL + FAIL-LOUD, the deliberate INVERSE of the forward converter's drop-on-unknown contract: the forward path drops unrepresentable nodes because dropping WIDENS a write-conflict filter (safe); dropping a conjunct from a scan residual predicate UNDER-filters and leaks rows the connector meant to exclude (a correctness bug). So this throws AnalysisException on any node outside the supported grammar, on a column ref absent from the scan output, and on a non-string literal (a numeric coercion would silently break lexicographic instant compare) — never silently returns null. Supported grammar (what a residual predicate uses today): ConnectorAnd, ConnectorComparison EQ/LT/LE/GT/GE, ConnectorColumnRef bound by name, STRING ConnectorLiteral. Standalone + unit-tested (dormant, no caller yet). Tests: ConnectorExpressionToNereidsConverterTest 8/8 (slot binding / all 5 operators / And / single-conjunct unwrap / 4 fail-loud arms), checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- ...ConnectorExpressionToNereidsConverter.java | 163 ++++++++++++++++++ ...ectorExpressionToNereidsConverterTest.java | 147 ++++++++++++++++ 2 files changed, 310 insertions(+) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverter.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverterTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverter.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverter.java new file mode 100644 index 00000000000000..2f925b47f184e1 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverter.java @@ -0,0 +1,163 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.nereids.exceptions.AnalysisException; +import org.apache.doris.nereids.trees.expressions.And; +import org.apache.doris.nereids.trees.expressions.EqualTo; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.GreaterThan; +import org.apache.doris.nereids.trees.expressions.GreaterThanEqual; +import org.apache.doris.nereids.trees.expressions.LessThan; +import org.apache.doris.nereids.trees.expressions.LessThanEqual; +import org.apache.doris.nereids.trees.expressions.SlotReference; +import org.apache.doris.nereids.trees.expressions.literal.StringLiteral; + +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** + * Converts a connector-neutral {@link ConnectorExpression} residual predicate (as returned by + * {@code ConnectorMetadata.getSyntheticScanPredicates}) BACK into a bound Nereids {@link Expression}. It is the + * reverse of the forward converters {@link NereidsToConnectorExpressionConverter} / + * {@link ExprToConnectorExpressionConverter} (same package) and is connector-agnostic: it only maps + * {@code ConnectorExpression} node types to Nereids nodes — the column names and literal values come entirely + * from the connector's payload, so there is ZERO source-specific logic here. + * + *

    The analysis-time rule that injects a connector's synthetic scan predicate (e.g. hudi's incremental + * {@code _hoodie_commit_time} window) calls this to turn each returned {@code ConnectorExpression} into a + * {@code LogicalFilter} conjunct, binding {@link ConnectorColumnRef}s to the scan-output {@link SlotReference}s + * by name.

    + * + *

    TOTAL and FAIL-LOUD — the deliberate INVERSE of the forward converter's drop-on-unknown contract. + * The forward converter returns {@code null} for unrepresentable nodes because dropping a conjunct only WIDENS + * a write-conflict filter (safe). Here the opposite holds: dropping a conjunct from a residual SCAN predicate + * UNDER-filters and leaks rows the connector meant to exclude (a correctness bug). So this throws on any node + * outside the supported grammar, on any column reference that does not resolve to a scan-output slot, and on + * any non-string literal — never silently returning null.

    + * + *

    Supported grammar (what a connector residual predicate uses today): {@link ConnectorAnd}, + * {@link ConnectorComparison} with an order/equality operator ({@code EQ/LT/LE/GT/GE}), {@link ConnectorColumnRef} + * bound by name, and STRING {@link ConnectorLiteral}s. Extend the dispatch (and add a + * {@code ConnectorType -> Nereids DataType} mapper for typed literals) when a connector needs a wider shape.

    + */ +public final class ConnectorExpressionToNereidsConverter { + + private ConnectorExpressionToNereidsConverter() { + } + + /** + * Converts {@code expr} into a bound Nereids {@link Expression}, resolving {@link ConnectorColumnRef}s + * against {@code boundSlots} (scan-output slots keyed by column name). + * + * @throws AnalysisException on an unsupported node, an unresolvable column reference, or a non-string literal + */ + public static Expression convert(ConnectorExpression expr, Map boundSlots) { + if (expr instanceof ConnectorAnd) { + return convertAnd((ConnectorAnd) expr, boundSlots); + } + if (expr instanceof ConnectorComparison) { + return convertComparison((ConnectorComparison) expr, boundSlots); + } + throw new AnalysisException( + "cannot reverse-convert connector residual predicate node: " + describe(expr)); + } + + private static Expression convertAnd(ConnectorAnd and, Map boundSlots) { + List conjuncts = and.getConjuncts(); + if (conjuncts.isEmpty()) { + throw new AnalysisException("cannot reverse-convert an empty ConnectorAnd"); + } + List converted = new ArrayList<>(conjuncts.size()); + for (ConnectorExpression conjunct : conjuncts) { + converted.add(convert(conjunct, boundSlots)); + } + // Nereids And(List) requires >= 2 children; a single conjunct is returned bare (mirrors the forward + // convertAnd's single-conjunct unwrap). + return converted.size() == 1 ? converted.get(0) : new And(converted); + } + + private static Expression convertComparison(ConnectorComparison cmp, Map boundSlots) { + Expression left = convertLeaf(cmp.getLeft(), boundSlots); + Expression right = convertLeaf(cmp.getRight(), boundSlots); + switch (cmp.getOperator()) { + case EQ: + return new EqualTo(left, right); + case LT: + return new LessThan(left, right); + case LE: + return new LessThanEqual(left, right); + case GT: + return new GreaterThan(left, right); + case GE: + return new GreaterThanEqual(left, right); + default: + // NE / EQ_FOR_NULL have no single-node Nereids inverse and no residual predicate emits them + // today. Fail loud rather than approximate (an approximation could change filtered rows). + throw new AnalysisException( + "unsupported comparison operator in connector residual predicate: " + cmp.getOperator()); + } + } + + private static Expression convertLeaf(ConnectorExpression operand, Map boundSlots) { + if (operand instanceof ConnectorColumnRef) { + String columnName = ((ConnectorColumnRef) operand).getColumnName(); + SlotReference slot = boundSlots.get(columnName); + if (slot == null) { + // Fail loud: silently dropping a predicate that references a column the scan does not output would + // UNDER-filter and leak rows the connector meant to exclude. Post visible-meta-column exposure this + // is unreachable for a correct hudi table; if it fires it surfaces a real binding bug, not wrong + // results. + throw new AnalysisException("connector residual predicate references column '" + columnName + + "' which is not in the scan output"); + } + return slot; + } + if (operand instanceof ConnectorLiteral) { + return convertLiteral((ConnectorLiteral) operand); + } + throw new AnalysisException( + "cannot reverse-convert connector comparison operand: " + describe(operand)); + } + + private static Expression convertLiteral(ConnectorLiteral literal) { + // Only STRING literals are supported today (residual predicates compare fixed-width instant strings + // lexicographically). Generalizing to typed literals needs a ConnectorType -> Nereids DataType mapper (the + // forward path only maps DataType -> ConnectorType); until then fail loud on any other type so a numeric + // coercion can never silently change compare semantics. + if (!"STRING".equalsIgnoreCase(literal.getType().getTypeName())) { + throw new AnalysisException("unsupported connector residual-predicate literal type: " + + literal.getType().getTypeName()); + } + Object value = literal.getValue(); + if (value == null) { + throw new AnalysisException("a null connector residual-predicate literal is not supported"); + } + return new StringLiteral(value.toString()); + } + + private static String describe(ConnectorExpression expr) { + return expr == null ? "null" : expr.getClass().getSimpleName(); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverterTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverterTest.java new file mode 100644 index 00000000000000..4f8ee0c232472a --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/ConnectorExpressionToNereidsConverterTest.java @@ -0,0 +1,147 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.connector.api.ConnectorType; +import org.apache.doris.connector.api.pushdown.ConnectorAnd; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorComparison; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.api.pushdown.ConnectorOr; +import org.apache.doris.nereids.exceptions.AnalysisException; +import org.apache.doris.nereids.trees.expressions.And; +import org.apache.doris.nereids.trees.expressions.EqualTo; +import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.GreaterThan; +import org.apache.doris.nereids.trees.expressions.GreaterThanEqual; +import org.apache.doris.nereids.trees.expressions.LessThan; +import org.apache.doris.nereids.trees.expressions.LessThanEqual; +import org.apache.doris.nereids.trees.expressions.SlotReference; +import org.apache.doris.nereids.trees.expressions.literal.StringLiteral; +import org.apache.doris.nereids.types.StringType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +/** + * Unit tests for the reverse {@link ConnectorExpressionToNereidsConverter}: it turns a connector-neutral + * residual predicate (e.g. hudi's incremental {@code _hoodie_commit_time > c1 AND <= c2}) back into a bound + * Nereids {@link Expression}, binding column refs to scan-output slots by name. + * + *

    The converter is TOTAL + FAIL-LOUD (the inverse of the forward converter's safe drop-on-unknown): dropping + * a conjunct from a scan residual predicate would UNDER-filter and leak rows, so anything outside the supported + * grammar throws rather than returning null. These tests pin both the happy conversion and every fail-loud arm.

    + */ +public class ConnectorExpressionToNereidsConverterTest { + + private static final ConnectorType STRING = ConnectorType.of("STRING"); + + private final SlotReference commitTime = new SlotReference("_hoodie_commit_time", StringType.INSTANCE); + + private Map slots() { + Map m = new HashMap<>(); + m.put("_hoodie_commit_time", commitTime); + return m; + } + + private static ConnectorComparison cmp(ConnectorComparison.Operator op, String bound) { + return new ConnectorComparison(op, + new ConnectorColumnRef("_hoodie_commit_time", STRING), ConnectorLiteral.ofString(bound)); + } + + @Test + public void convertsComparisonBindingColumnRefToTheScanSlot() { + Expression result = ConnectorExpressionToNereidsConverter.convert( + cmp(ConnectorComparison.Operator.GT, "c1"), slots()); + Assertions.assertTrue(result instanceof GreaterThan, "GT must map to a Nereids GreaterThan"); + Assertions.assertSame(commitTime, result.child(0), + "the column ref must bind to the SAME scan-output slot instance (a fresh slot would be unbound)"); + Assertions.assertEquals(new StringLiteral("c1"), result.child(1), + "the bound must be a STRING literal so the compare is lexicographic over instants"); + } + + @Test + public void mapsEachOrderEqualityOperatorToItsNereidsNode() { + Assertions.assertTrue(convert(ConnectorComparison.Operator.EQ) instanceof EqualTo); + Assertions.assertTrue(convert(ConnectorComparison.Operator.LT) instanceof LessThan); + Assertions.assertTrue(convert(ConnectorComparison.Operator.LE) instanceof LessThanEqual); + Assertions.assertTrue(convert(ConnectorComparison.Operator.GT) instanceof GreaterThan); + Assertions.assertTrue(convert(ConnectorComparison.Operator.GE) instanceof GreaterThanEqual); + } + + private Expression convert(ConnectorComparison.Operator op) { + return ConnectorExpressionToNereidsConverter.convert(cmp(op, "c1"), slots()); + } + + @Test + public void convertsConnectorAndToNereidsAnd() { + ConnectorExpression window = new ConnectorAnd(Arrays.asList( + cmp(ConnectorComparison.Operator.GT, "c1"), cmp(ConnectorComparison.Operator.LE, "c2"))); + Expression result = ConnectorExpressionToNereidsConverter.convert(window, slots()); + Assertions.assertTrue(result instanceof And, "a ConnectorAnd must map to a Nereids And"); + Assertions.assertEquals(2, result.children().size(), "the And must keep both conjuncts"); + Assertions.assertTrue(result.child(0) instanceof GreaterThan); + Assertions.assertTrue(result.child(1) instanceof LessThanEqual); + } + + @Test + public void singleConjunctConnectorAndIsUnwrapped() { + // Nereids And(List) rejects < 2 children, so a 1-element ConnectorAnd must return the bare conjunct. + ConnectorExpression single = new ConnectorAnd( + Collections.singletonList(cmp(ConnectorComparison.Operator.GT, "c1"))); + Expression result = ConnectorExpressionToNereidsConverter.convert(single, slots()); + Assertions.assertTrue(result instanceof GreaterThan, "a single-conjunct AND must unwrap, not build And(1)"); + } + + @Test + public void failsLoudWhenColumnNotInScanOutput() { + // Dropping a residual predicate whose column is absent would UNDER-filter (leak out-of-window rows) — the + // inverse of the forward converter's safe drop. So an unresolvable column MUST throw, not no-op. + Assertions.assertThrows(AnalysisException.class, () -> ConnectorExpressionToNereidsConverter.convert( + cmp(ConnectorComparison.Operator.GT, "c1"), Collections.emptyMap())); + } + + @Test + public void failsLoudOnUnsupportedNode() { + ConnectorExpression or = new ConnectorOr(Arrays.asList( + cmp(ConnectorComparison.Operator.GT, "c1"), cmp(ConnectorComparison.Operator.LE, "c2"))); + Assertions.assertThrows(AnalysisException.class, + () -> ConnectorExpressionToNereidsConverter.convert(or, slots())); + } + + @Test + public void failsLoudOnNonStringLiteral() { + // A numeric literal must NOT be silently coerced to a string (would change lexicographic compare). + ConnectorExpression gt = new ConnectorComparison(ConnectorComparison.Operator.GT, + new ConnectorColumnRef("_hoodie_commit_time", STRING), ConnectorLiteral.ofInt(42)); + Assertions.assertThrows(AnalysisException.class, + () -> ConnectorExpressionToNereidsConverter.convert(gt, slots())); + } + + @Test + public void failsLoudOnUnsupportedComparisonOperator() { + Assertions.assertThrows(AnalysisException.class, () -> ConnectorExpressionToNereidsConverter.convert( + cmp(ConnectorComparison.Operator.NE, "c1"), slots())); + } +} From 7d881d7a327c3f972bce198b578db45728335f99 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 16:23:15 +0800 Subject: [PATCH 122/334] [feat](catalog) HMS cutover hudi HD-C3 INC-4c: neutral analysis rule wires the synthetic-predicate SPI (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit fe-core half that activates the row-correctness mechanism (dormant in effect — iceberg/paimon return empty and hudi-on-HMS is not yet routed). Turns a connector's neutral residual predicate into a LogicalFilter at analysis time, where an output SlotReference for the referenced column already exists (a scan node cannot mint a slot for an unprojected column, so injection MUST be logical). - PluginDrivenExternalTable.getSyntheticScanPredicates(MvccSnapshot): public fe-core entry that does the connector/session/handle plumbing (mirrors resolveWriteTargetHandle), unwraps the resolved ConnectorMvccSnapshot from the analysis-time PluginDrivenMvccSnapshot, and calls the SPI. Empty for a non-plugin snapshot, an unresolvable handle, or a non-opting connector. - CheckPolicy: a NEUTRAL arm added beside the legacy LogicalHudiScan arm — for a plugin LogicalFileScan carrying scanParams it retrieves the resolved snapshot (StatementContext.getSnapshot), asks the table for its synthetic predicates, and reverse-converts each (ConnectorExpressionToNereidsConverter) binding column refs to the scan output slots by name, wrapping them into the row-policy filter. Keys ONLY on the generic PluginDrivenExternalTable + the SPI returning empty — zero source discrimination. iceberg/paimon return empty -> no filter -> plan byte-identical. The legacy LogicalHudiScan arm is untouched (deleted at flip). Why analysis-layer + this window source: the resolved (begin, end] window rides the ConnectorMvccSnapshot (resolved by loadSnapshots at analysis), NOT a stamped handle — the handle's begin/endInstant are set only at scan-node build (applySnapshot), which is too late to materialize the _hoodie_commit_time slot. Reading the snapshot is the SAME single resolution the scan-time file selection uses, so the row filter and file set can never diverge. Dormant: hms not in SPI_READY_TYPES; hudi getTableHandle divert is a later substep. For live iceberg/paimon @branch/@tag scans the rule fires but the SPI returns empty -> byte-identical plan (the extra getTableHandle is a same-statement connector cache hit). Tests: PluginDrivenExternalTableTest +3 (delegates / empty for non-plugin snapshot / empty for unresolvable handle) = 33; ConnectorExpressionToNereidsConverterTest 8; CheckRowPolicyTest 6 (no regression on the shared rule); fe-core compile 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../datasource/PluginDrivenExternalTable.java | 31 ++++++++++ .../nereids/rules/analysis/CheckPolicy.java | 58 ++++++++++++++++++ .../PluginDrivenExternalTableTest.java | 61 +++++++++++++++++++ 3 files changed, 150 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 91367a332d3a74..55d42cf23c28db 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -37,6 +37,8 @@ import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.systable.PluginDrivenSysTable; @@ -123,6 +125,35 @@ public ConnectorTableHandle resolveWriteTargetHandle() { "Cannot resolve the connector table handle for write target " + getName())); } + /** + * Returns the connector's synthetic scan predicates for this table at the given resolved MVCC + * {@code snapshot} — a connector "residual predicate" the read cannot enforce by file selection alone + * (e.g. a hudi incremental {@code _hoodie_commit_time} commit-time window), expressed in the neutral + * {@link ConnectorExpression} grammar. The analysis-time synthetic-predicate rule reverse-converts these + * into a {@code LogicalFilter} over this table's scan. + * + *

    The {@code snapshot} is the one {@link org.apache.doris.datasource.mvcc.MvccTable#loadSnapshot} + * resolved at analysis time (retrieved from {@code StatementContext}), so the row-filter window is the + * SAME single resolution the scan-time {@code applySnapshot} threads onto the handle — file selection and + * the row filter can never diverge. Returns empty when the snapshot is not a plugin MVCC snapshot, the + * handle cannot be resolved, or the connector has no residual predicate (iceberg/paimon/... and every + * non-incremental read inherit the empty SPI default) — so the plan stays byte-identical.

    + */ + public List getSyntheticScanPredicates(MvccSnapshot snapshot) { + if (!(snapshot instanceof PluginDrivenMvccSnapshot) || !(catalog instanceof PluginDrivenExternalCatalog)) { + return Collections.emptyList(); + } + ConnectorMvccSnapshot connectorSnapshot = ((PluginDrivenMvccSnapshot) snapshot).getConnectorSnapshot(); + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = pluginCatalog.getConnector().getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + if (!handleOpt.isPresent()) { + return Collections.emptyList(); + } + return metadata.getSyntheticScanPredicates(session, handleOpt.get(), connectorSnapshot); + } + /** * Returns whether the underlying connector supports multiple concurrent writers. * Used by the planner to decide GATHER (single writer) vs parallel distribution. diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckPolicy.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckPolicy.java index 8c100726716d9f..132306486f54ea 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckPolicy.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/CheckPolicy.java @@ -17,15 +17,23 @@ package org.apache.doris.nereids.rules.analysis; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.datasource.ConnectorExpressionToNereidsConverter; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable; +import org.apache.doris.datasource.mvcc.MvccSnapshot; +import org.apache.doris.nereids.StatementContext; import org.apache.doris.nereids.analyzer.UnboundRelation; import org.apache.doris.nereids.rules.Rule; import org.apache.doris.nereids.rules.RuleType; import org.apache.doris.nereids.trees.expressions.Expression; +import org.apache.doris.nereids.trees.expressions.Slot; +import org.apache.doris.nereids.trees.expressions.SlotReference; import org.apache.doris.nereids.trees.plans.Plan; import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate; import org.apache.doris.nereids.trees.plans.logical.LogicalCheckPolicy; import org.apache.doris.nereids.trees.plans.logical.LogicalCheckPolicy.RelatedPolicy; +import org.apache.doris.nereids.trees.plans.logical.LogicalFileScan; import org.apache.doris.nereids.trees.plans.logical.LogicalFilter; import org.apache.doris.nereids.trees.plans.logical.LogicalHudiScan; import org.apache.doris.nereids.trees.plans.logical.LogicalPlan; @@ -37,8 +45,11 @@ import com.google.common.collect.ImmutableList; +import java.util.Collections; +import java.util.HashMap; import java.util.LinkedHashSet; import java.util.List; +import java.util.Map; import java.util.Set; /** @@ -81,11 +92,22 @@ public List buildRules() { // replace incremental params as AND expression if (relation instanceof LogicalHudiScan) { + // Legacy hudi-on-HMS incremental read (LogicalHudiScan); deleted at the cutover. LogicalHudiScan hudiScan = (LogicalHudiScan) relation; if (hudiScan.getTable() instanceof HMSExternalTable) { combineFilter.addAll(hudiScan.generateIncrementalExpression( hudiScan.getLogicalProperties().getOutput())); } + } else if (relation instanceof LogicalFileScan + && ((LogicalFileScan) relation).getTable() instanceof PluginDrivenExternalTable + && ((LogicalFileScan) relation).getScanParams().isPresent()) { + // Neutral synthetic-predicate injection for an SPI-driven (plugin) scan: the + // connector supplies a residual predicate the engine must apply (e.g. a hudi @incr + // _hoodie_commit_time commit-time window), which fe-core reverse-converts into an + // AND filter WITHOUT branching on the source. iceberg/paimon/... return empty, so + // nothing is added and the plan stays byte-identical (the iron-rule guarantee). + combineFilter.addAll(collectConnectorSyntheticPredicates( + (LogicalFileScan) relation, ctx.cascadesContext.getStatementContext())); } RelatedPolicy relatedPolicy = checkPolicy.findPolicy(relation, ctx.cascadesContext); @@ -108,6 +130,42 @@ public List buildRules() { ); } + /** + * Collects the connector's synthetic scan predicates for a plugin {@link LogicalFileScan} and reverse-converts + * them into bound Nereids conjuncts. The connector-neutral {@link ConnectorExpression}s (e.g. a hudi @incr + * {@code _hoodie_commit_time} window) come from the SPI; fe-core only binds their column refs to the scan's + * output slots by name and maps the node shapes back to Nereids — it never branches on the source. Empty for + * every non-opting connector (iceberg/paimon/...) and every non-incremental read, so the plan is unchanged. + */ + private Set collectConnectorSyntheticPredicates(LogicalFileScan scan, + StatementContext statementContext) { + PluginDrivenExternalTable table = (PluginDrivenExternalTable) scan.getTable(); + // The MVCC snapshot resolved at analysis time (StatementContext.loadSnapshots) carries the + // connector-resolved window — the SAME single resolution the scan-time applySnapshot threads onto the + // handle, so the row filter and the file selection can never diverge. + MvccSnapshot snapshot = statementContext + .getSnapshot(table, scan.getTableSnapshot(), scan.getScanParams()) + .orElse(null); + if (snapshot == null) { + return Collections.emptySet(); + } + List predicates = table.getSyntheticScanPredicates(snapshot); + if (predicates.isEmpty()) { + return Collections.emptySet(); + } + Map boundSlots = new HashMap<>(); + for (Slot slot : scan.getLogicalProperties().getOutput()) { + if (slot instanceof SlotReference) { + boundSlots.put(slot.getName(), (SlotReference) slot); + } + } + Set result = new LinkedHashSet<>(); + for (ConnectorExpression predicate : predicates) { + result.add(ConnectorExpressionToNereidsConverter.convert(predicate, boundSlots)); + } + return result; + } + // logicalView() or logicalSubQueryAlias(logicalView()) private boolean isView(Plan plan) { if (plan instanceof LogicalView) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index 9a880ee634b1a2..945a269350122a 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -31,7 +31,11 @@ import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.pushdown.ConnectorColumnRef; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.qe.ConnectContext; import com.google.common.collect.ImmutableList; @@ -180,6 +184,63 @@ public void resolveWriteTargetHandleFailsLoudWhenUnresolvable() { "the fail-loud message must name the unresolved write target"); } + // ============= HD-C3 INC-4: synthetic scan predicate (connector residual predicate) plumbing ============= + + // A CALLS_REAL_METHODS table whose connector resolves the handle to `resolved` (null => empty) and returns + // `predicates` from getSyntheticScanPredicates. + private static PluginDrivenExternalTable syntheticPredicateTable(ConnectorTableHandle resolved, + List predicates) { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + Mockito.when(metadata.getTableHandle(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(Optional.ofNullable(resolved)); + Mockito.when(metadata.getSyntheticScanPredicates(Mockito.any(), Mockito.any(), Mockito.any())) + .thenReturn(predicates); + Connector connector = Mockito.mock(Connector.class); + Mockito.when(connector.getMetadata(Mockito.any())).thenReturn(metadata); + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.when(catalog.buildConnectorSession()).thenReturn(Mockito.mock(ConnectorSession.class)); + PluginDrivenExternalTable table = Mockito.mock(PluginDrivenExternalTable.class, Mockito.CALLS_REAL_METHODS); + Deencapsulation.setField(table, "catalog", catalog); + return table; + } + + private static PluginDrivenMvccSnapshot pluginSnapshot() { + return new PluginDrivenMvccSnapshot(ConnectorMvccSnapshot.builder().build(), + Collections.emptyMap(), Collections.emptyMap()); + } + + @Test + public void getSyntheticScanPredicatesDelegatesToTheConnectorForAPluginSnapshot() { + // The analysis rule retrieves the resolved snapshot and asks the connector for its residual predicate + // (the hudi @incr _hoodie_commit_time window); the table threads it verbatim from the SPI. + List sentinel = Collections.singletonList( + new ConnectorColumnRef("_hoodie_commit_time", ConnectorType.of("STRING"))); + PluginDrivenExternalTable table = + syntheticPredicateTable(Mockito.mock(ConnectorTableHandle.class), sentinel); + Assertions.assertSame(sentinel, table.getSyntheticScanPredicates(pluginSnapshot()), + "the residual predicate must be threaded verbatim from the connector SPI"); + } + + @Test + public void getSyntheticScanPredicatesEmptyForNonPluginSnapshot() { + // A non-plugin MvccSnapshot carries no ConnectorMvccSnapshot to hand the SPI -> empty (and the guard + // avoids a ClassCastException). The rule then adds no filter. + MvccSnapshot foreign = Mockito.mock(MvccSnapshot.class); + Assertions.assertTrue(syntheticPredicateTable(Mockito.mock(ConnectorTableHandle.class), + Collections.emptyList()).getSyntheticScanPredicates(foreign).isEmpty(), + "a non-plugin snapshot must yield no synthetic predicates"); + } + + @Test + public void getSyntheticScanPredicatesEmptyWhenHandleUnresolvable() { + // An unresolvable handle (concurrent DROP / transient metadata error) degrades to empty rather than + // handing the gateway a null handle -> the rule simply adds no filter. + Assertions.assertTrue(syntheticPredicateTable(null, Collections.emptyList()) + .getSyntheticScanPredicates(pluginSnapshot()).isEmpty(), + "an unresolvable handle must degrade to no synthetic predicates"); + } + /** * Builds a CALLS_REAL_METHODS PluginDrivenExternalTable wired to a stubbed connector chain whose * write-plan provider returns {@code synthetic}. {@code writeProviderPresent=false} models a read-only From bf7c15f62e7d8ae92be31f20913b3ce0c81e6773 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 16:38:24 +0800 Subject: [PATCH 123/334] [test](catalog) HMS cutover hudi HD-C3 INC-4 review fix: honest scope for the meta-column unit test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial review (6 dims, each finding verified) came back with 0 confirmed defects on correctness / byte-parity / dormancy / iron-rule / gateway / schema plumbing, and 1 minor test-honesty finding: the added HudiSchemaParityTest meta-column test's javadoc overclaimed that it guards the getTableAvroSchema()->getTableAvroSchema(true) flip, but it hand-feeds a meta-inclusive avro schema straight into avroSchemaToColumns and never invokes getTableAvroSchema (which needs a live metaClient). So reverting the flip survives the unit suite; the real guard for the populate.meta.fields=false exposure is e2e. Fix: rename to avroSchemaToColumnsPreservesMetaCommitTimeAsBindableStringColumn and correct the javadoc to state what the unit test actually locks — that avroSchemaToColumns preserves a meta-named field as an exact-lower-case, STRING, visible column (the reverse-converter's binding precondition) — and to state explicitly that the end-to-end getTableAvroSchema(true) meta-column exposure is an e2e guard, not this unit test. Assertions (exact name / STRING type / visible) unchanged. HudiSchemaParityTest 4/4, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiSchemaParityTest.java | 15 +++++++++------ 1 file changed, 9 insertions(+), 6 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java index 18241a11a4712e..18df581efdba11 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java @@ -122,12 +122,15 @@ public void testColumnTypeStringsMirrorLegacyColTypes() { } @Test - public void metaCommitTimeColumnIsExposedVisibleAndStringForRowFilterBinding() { - // The synthetic incremental row filter references a scan-output slot named EXACTLY "_hoodie_commit_time" - // (byte-faithful to legacy LogicalHudiScan.generateIncrementalExpression). getTableAvroSchema(true) always - // carries the 5 `_hoodie_*` meta fields as nullable strings; avroSchemaToColumns must preserve the - // commit-time field as a VISIBLE STRING column with that exact lower-case name, or the filter would - // silently fail to bind and the incremental scan would over-read out-of-window rows. + public void avroSchemaToColumnsPreservesMetaCommitTimeAsBindableStringColumn() { + // The synthetic incremental row filter binds a ConnectorColumnRef to a scan-output slot named EXACTLY + // "_hoodie_commit_time" (byte-faithful to legacy LogicalHudiScan.generateIncrementalExpression). This test + // pins avroSchemaToColumns' HALF of that binding precondition: GIVEN an avro schema that carries the meta + // field (which the getSchemaFromMetaClient getTableAvroSchema(true) call guarantees at runtime), + // avroSchemaToColumns must surface it as a column with that exact lower-case name, STRING type, and + // visible — never dropped/renamed/mistyped/hidden — or the filter's ConnectorColumnRef fails to bind. + // The getTableAvroSchema(true) call itself runs only against a live metaClient, so the end-to-end + // meta-column EXPOSURE for a populate.meta.fields=false table is an e2e guard, NOT this unit test. String metaInclusive = "{\"type\":\"record\",\"name\":\"hudi_t\",\"fields\":[" + "{\"name\":\"_hoodie_commit_time\",\"type\":[\"null\",\"string\"],\"default\":null}," From 2a411bb97148cd6f5340b043449e72c19ee2ff15 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 16:41:20 +0800 Subject: [PATCH 124/334] [doc](catalog) HMS cutover hudi HD-C3 INC-4 DONE; next = INC-5 (flip-time wiring) Roll the handoff: INC-4 (neutral synthetic-predicate SPI + reverse converter + analysis rule + visible meta columns) is done across 3 dormant commits + a review fix, adversarially reviewed (5/6 dims 0 findings, 1 minor test-honesty fixed). Records the corrected SPI signature (ConnectorMvccSnapshot, not fe-core TableScanParams), the required HiveConnectorMetadata delegation, the analysis-layer injection rationale, and the e2e/flip residuals. Next = INC-5. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 195db1dec90edb..ae2941bda10b68 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。下一步 = HD-C3 余 2 步（INC-4 中立谓词 SPI+fe-core 规则+反向转换器+5 可见元数据列 → INC-5 翻闸接线）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**下一步 = HD-C3 余 1 步（INC-5 翻闸接线：删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + legacy hudi source）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -91,8 +91,9 @@ - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 - **✅ HD-C3 INC-1 DONE（`9327261ec52`，休眠）= @incr 窗口解析脊柱**（recon `wf_3398c7fb-bfb` 4 读者+parity critic〔scope critic StructuredOutput 上限失败，已用我亲手字节码核对补〕 + 对抗复审 `wf_13bea565-a47` 4 维〔各 finding 逐条 refute〕）。连接器侧扩展 HD-C2 脊柱：`resolveTimeTravel(INCREMENTAL)` **单一 locus** `resolveIncremental` 解析 (begin,end]〔begin 必填逐字 fail-loud、`"earliest"`→`"000"`、end 缺省→最新 completed instant、`"latest"`→最新〔测**已解析 end 值**=legacy COW 形，天然规避 `MOR:92` 死码 bug〕、空时间线→`(000,000]` 免 begin 检查〕→ **NON-EMPTY property-only pin**（snapshotId/schemaId inert：fe-core INCREMENTAL `loadSnapshot` 分支自列 latest 分区+latest schema、只读窗口属性）；`applySnapshot` 钉 `begin/endInstant`（与 FOR TIME `queryInstant` 载体互斥、保 prunedPartitionPaths）。新 `HudiTableHandle.begin/endInstant` + `HudiScanPlanProvider.latestCompletedInstantTime(String)`（从 `latestCompletedInstant` 字节等价抽出）。**复审关键裁决**：①**blocker（COW 时间线 byte-parity）被 hudi-1.0.2 字节码驳倒**——单 `getCommitsAndCompactionTimeline()` 按类型解析到 legacy 逐类型时间线（COW→`getCommitAndReplaceTimeline`={commit,replacecommit,clustering}=legacy COW 的 `metaClient.getCommitTimeline()`〔该 metaClient 方法**非** commit-only，委派给 getCommitAndReplaceTimeline〕；MOR→`getWriteTimeline`），review+adversarial-verify 都把 metaClient 方法与 `BaseHoodieTimeline.getCommitTimeline()` 时间线方法混淆了；②major/minor/nit 均**按设计延后**（见下）。`HudiIncrementalTest` +11、fe-connector-hudi 96 测 + checkstyle 0 + import gate 净。**⚠延后（已文档化 fail-loud、非静默丢，INC-2/4 须接）**：(a) **`populateMetaFields()` fail-loud→INC-2**（"meta fields disabled" 拒，结构上属 relation 构造器=INC-2 移植；INC-1 窗口解析不查，INC-2 移植时在 planScan 强制、同为查询规划期失败；INC-2 须字节忠实移植+e2e 加 meta-fields-disabled fixture）；(b) **hollow-commit `USE_TRANSITION_TIME`（completionTime）+ hollow 抛→INC-2/INC-3**（默认 FAIL 策略用 requestedTime，非默认策略绑 relation 文件选择）；(c) **原生 `hoodie.datasource.read.*.instanttime` 窗口键→INC-4**（INC-1 读标准 `@incr` 别名 beginTime/endTime，是否也收原生 hoodie 键归 INC-4）。 - **✅ HD-C3 INC-2 DONE（`d64a25f28fc`，休眠）= IncrementalRelation 家族移植**（HEAD-grounded recon `wf_50a59f91-14e` 3 读者〔窗口协调/边界/API+可测性〕+完整性 critic + 5 维对抗复审 `wf_b247a285-d64`〔byte-parity/dormancy-iron-rule/guard-correctness 三镜头 **0 发现**；test-strength 3 test/doc-honesty finding 已修；mor92-and-axis 镜头 API stall 死→crux 亲手 grep+读自证〕）。COW/MOR/Empty + `IncrementalRelation` 接口移植进 `fe-connector-hudi`（连接器内部、package-private）。**用户签字方案 A（单一窗口权威，见本 session 决策）**：移植关系**消费句柄已解析的 (begin,end] 窗口只挑文件、自己不再解析**（构造器删窗口解析 prologue）→ 死码 `MORIncrementalRelation:92` sentinel bug **自然消失**、文件集永不与 INC-4 `_hoodie_commit_time` 行过滤的 end 打架。split 型 re-home `spi.Split`/`HudiSplit`→`HudiScanRange`（丢 `LocationPath`/`TableFormatType`/`HudiPartitionUtils` fe-core 型）；COW `collectSplits`→`List`（path + length/fileSize 取 `HoodieWriteStat.getFileSizeInBytes()`、format 走 `detectFileFormat`、分区值走连接器 Map `parsePartitionValues` on `metaClient.getTableConfig().getPartitionFields()`=legacy COW:212 平价，**非** HMS handle keys）；MOR `collectFileSlices`→`List` 逐字；archived-flag 起字节忠实 legacy，仅删窗口 prologue。INC-1 延后守卫作 `IncrementalRelation` 共享静态、逐字文案 re-type 成 `DorisConnectorException`：`checkIncrementalMetaFields`（meta-fields-disabled 首守卫、非空时间线 only 靠上游 Empty 路由）/`checkStateTransitionTimeFullTableScan`/`decideArchivalFullTableScan`/`checkNotFullTableScan`。**`resolveIncremental` END 轴按策略**：`hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME` 下 default/`latest` end 走 completion-time 轴（新 `latestCompletedInstantTime(mc, useCompletionTime)` 重载：getCompletionTime vs requestedTime）→ 单一 handle-stamped end 与关系 `findInstantsInRangeByCompletionTime` 选片同轴、无末批漏读；默认策略字节不变、休眠（planScan 只读 `queryInstant` 从不读 begin/endInstant）。`getHoodieParams`/`getStartTs` 丢弃（FE-filter 模型下已死）。全绿（`HudiIncrementalRelationTest` 13 + `HudiIncrementalTest` 12 + 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**⚠ e2e-only（休眠单测只覆盖纯守卫+Empty；构造器 timeline/metadata/fs 选片 + completion-time 轴须 e2e，设计 §5 已加 meta-fields-disabled + USE_TRANSITION_TIME fixture）**。**下一步 = INC-4（中立 synthetic-predicate SPI + fe-core 分析期规则 + 反向 ConnectorExpression→Nereids 转换器 + 暴露 5 可见 `_hoodie_*` 元数据列）**。 - - **✅ HD-C3 INC-3 DONE（`0fea5bc6ba4`，休眠）= 增量 planScan 接线**（对抗复审 `wf_99c5b718-c11` 4 维〔snapshot byte-parity / incremental-correctness / dormancy-transport / test-intent〕+ 逐条 adversarial-verify **0 发现**）。`HudiScanPlanProvider.planScan` 加 `getBeginInstant()!=null` 增量支路（置于 schema 解析后、fsView 构建前，纯早返；非增量读〔含 FOR TIME AS OF〕字节不变）：`buildIncrementalRelation` 按 `isCow`(=`metaClient.getTableType()==COPY_ON_WRITE`，权威) 建 COW/MOR 关系（IOException 重型为 `DorisConnectorException`）；`incrementalRanges` **纯静态**路由（fake 关系可离线测）——`fallbackFullTableScan()`→`Optional.empty()`→降级到最新快照扫描（legacy `getSplits:470`，读 latest；两载体互斥故 @incr 句柄无 queryInstant）；COW→`collectSplits()` 原生；MOR→`collectFileSlices()` 逐片经**抽出的** `buildMorRange`（从 `collectMorSplits` 内联 lambda 提取、快照路径字节不变）转 JNI range 钉 `getEndTs()`（窗口 END=BE 合并时刻，非 latest）、分区值按 `metaClient.getTableConfig().getPartitionFields()` 逐片解析（=legacy `HudiScanNode:391-393`，**非** HMS handle 键）。**RO-as-RT 翻转 = 被 metaClient.getTableType() 天然覆盖、不移植、不需 serde 参数上句柄**（旧翻转只为补偿 legacy 按 hive inputFormat 把 MOR `_ro` 门面误判 COW；连接器读 hoodie.properties 直接得 MERGE_ON_READ）。**用户签字方案 B（graceful，2026-07-09）**：`force_jni+COW+增量` 旧代码 `collectFileSlices()` on COW relation→`UnsupportedOperationException` 崩溃，改按**关系类型**路由→COW 增量恒原生（忽略 force_jni），不崩溃；MOR 仍经 buildMorRange 尊重 force_jni（无 log 片留 JNI）。**参数穿透**：`resolveIncremental` 把原始 @incr 选项（glob/fallback/hollow-policy）复制进 MVCC 快照属性（前缀 `hudi.incr-opt.`、跳 null 值），`applySnapshot` 前缀剥离重建进**新** `HudiTableHandle.incrementalParams`（**BE 安全**：快照属性只经 `PluginDrivenScanNode.applyMvccSnapshotPin`→`applySnapshot` 消费、从不入 BE thrift〔亲手 + 复审双证〕；begin/end 载体走 `hudi.incremental-*` 不同命名空间不混入；`toBuilder` 复制）；planScan 由 `IncrementalRelation.hollowCommitHandling(optParams)`=`valueOf(getOrDefault(policyKey,"FAIL"))` 派生策略，与 `resolveIncremental` END 轴 `INCR_HOLLOW_POLICY_KEY` 同源（文件集与窗口 END 同轴）。**行级过滤（`_hoodie_commit_time` 谓词）= INC-4；本步只挑 FILE，裸 @incr 会多读直到 INC-4（休眠无影响）**。全绿（`HudiIncrementalPlanScanTest` 7〔fake IncrementalRelation + 手造 FileSlice：fallback 降级/COW 路由不碰 collectFileSlices/COW 忽略 force_jni/MOR 钉 endTs+逐片分区值/buildMorRange JNI+native 降级〕+ `HudiIncrementalTest` 15〔+3 参数穿透+命名空间隔离+非增量 pin 空 incrementalParams〕+ `HudiIncrementalRelationTest` 16〔+3 hollowCommitHandling FAIL 默认/USE_TRANSITION_TIME/bogus 抛〕+ 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**下一步 = INC-4**。 - - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **下一步 INC-4** 中立 SPI + fe-core 规则 + reverse `ConnectorExpression→Nereids` 转换（~150-250 LOC）+ 暴露 5 可见元数据列（**+ INC-1 延后的原生 hoodie.* 窗口键决策**）→ INC-5 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C3 INC-3 DONE（`0fea5bc6ba4`，休眠）= 增量 planScan 接线**（对抗复审 `wf_99c5b718-c11` 4 维〔snapshot byte-parity / incremental-correctness / dormancy-transport / test-intent〕+ 逐条 adversarial-verify **0 发现**）。`HudiScanPlanProvider.planScan` 加 `getBeginInstant()!=null` 增量支路（置于 schema 解析后、fsView 构建前，纯早返；非增量读〔含 FOR TIME AS OF〕字节不变）：`buildIncrementalRelation` 按 `isCow`(=`metaClient.getTableType()==COPY_ON_WRITE`，权威) 建 COW/MOR 关系（IOException 重型为 `DorisConnectorException`）；`incrementalRanges` **纯静态**路由（fake 关系可离线测）——`fallbackFullTableScan()`→`Optional.empty()`→降级到最新快照扫描（legacy `getSplits:470`，读 latest；两载体互斥故 @incr 句柄无 queryInstant）；COW→`collectSplits()` 原生；MOR→`collectFileSlices()` 逐片经**抽出的** `buildMorRange`（从 `collectMorSplits` 内联 lambda 提取、快照路径字节不变）转 JNI range 钉 `getEndTs()`（窗口 END=BE 合并时刻，非 latest）、分区值按 `metaClient.getTableConfig().getPartitionFields()` 逐片解析（=legacy `HudiScanNode:391-393`，**非** HMS handle 键）。**RO-as-RT 翻转 = 被 metaClient.getTableType() 天然覆盖、不移植、不需 serde 参数上句柄**（旧翻转只为补偿 legacy 按 hive inputFormat 把 MOR `_ro` 门面误判 COW；连接器读 hoodie.properties 直接得 MERGE_ON_READ）。**用户签字方案 B（graceful，2026-07-09）**：`force_jni+COW+增量` 旧代码 `collectFileSlices()` on COW relation→`UnsupportedOperationException` 崩溃，改按**关系类型**路由→COW 增量恒原生（忽略 force_jni），不崩溃；MOR 仍经 buildMorRange 尊重 force_jni（无 log 片留 JNI）。**参数穿透**：`resolveIncremental` 把原始 @incr 选项（glob/fallback/hollow-policy）复制进 MVCC 快照属性（前缀 `hudi.incr-opt.`、跳 null 值），`applySnapshot` 前缀剥离重建进**新** `HudiTableHandle.incrementalParams`（**BE 安全**：快照属性只经 `PluginDrivenScanNode.applyMvccSnapshotPin`→`applySnapshot` 消费、从不入 BE thrift〔亲手 + 复审双证〕；begin/end 载体走 `hudi.incremental-*` 不同命名空间不混入；`toBuilder` 复制）；planScan 由 `IncrementalRelation.hollowCommitHandling(optParams)`=`valueOf(getOrDefault(policyKey,"FAIL"))` 派生策略，与 `resolveIncremental` END 轴 `INCR_HOLLOW_POLICY_KEY` 同源（文件集与窗口 END 同轴）。**行级过滤（`_hoodie_commit_time` 谓词）= INC-4；本步只挑 FILE，裸 @incr 会多读直到 INC-4（休眠无影响）**。全绿（`HudiIncrementalPlanScanTest` 7〔fake IncrementalRelation + 手造 FileSlice：fallback 降级/COW 路由不碰 collectFileSlices/COW 忽略 force_jni/MOR 钉 endTs+逐片分区值/buildMorRange JNI+native 降级〕+ `HudiIncrementalTest` 15〔+3 参数穿透+命名空间隔离+非增量 pin 空 incrementalParams〕+ `HudiIncrementalRelationTest` 16〔+3 hollowCommitHandling FAIL 默认/USE_TRANSITION_TIME/bogus 抛〕+ 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。 + - **✅ HD-C3 INC-4 DONE（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠）= 中立 synthetic-predicate SPI（行级 `_hoodie_commit_time` 窗口过滤）**（**用户签字方案 A**=通用 `ConnectorExpression` 而非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f`：row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing 5 维 **0 发现** + test-intent 1 minor〔meta-col 测 javadoc overclaim + isVisible 空断言〕已修）。**问题=COW base 文件在窗口内被重写会带出窗口外旧行，只挑文件不够、须行级 `_hoodie_commit_time > begin AND <= end`；legacy 靠 source-specific fe-core 注入（`CheckPolicy`+`LogicalHudiScan.generateIncrementalExpression`），翻闸后死码+违铁律，须挪进中立 SPI（对齐 Trino 残余谓词）**。①新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,ConnectorMvccSnapshot)` 默认空——**纠正设计硬伤**：原签名收 fe-core `analysis.TableScanParams`，连接器-api 不依赖 fe-core+import gate 禁→编译不过，改收连接器-api `ConnectorMvccSnapshot`；hudi override 从**同一** `resolveIncremental` 盖的 `HUDI_INCREMENTAL_BEGIN/END_PROPERTY` 读窗口（**单一窗口权威**，与 applySnapshot 文件选择同源、永不发散）发 `> begin`(GT)+`<= end`(LE) 两 flat STRING conjunct（词典序 instant 比较，byte-faithful legacy）；非增量 pin（plain/FOR TIME）返空。②**HiveConnectorMetadata 委派 override**（**recon 揪出漏项**：无它→foreign hudi 句柄继承空默认→静默 over-read）转发兄弟，镜像 resolveTimeTravel/applySnapshot、外来句柄永不强转、hive 句柄返空。③fe-core 新 `ConnectorExpressionToNereidsConverter`（**TOTAL+FAIL-LOUD**，与 forward 转换器 drop-on-unknown **相反**：漏 conjunct=under-filter 漏行；仅 {ConnectorAnd, GT/LE/EQ/LT/GE, ColumnRef 按名绑, STRING literal}，And(size≥2) 守卫+单 conjunct 解包，其余/未解析列/非 STRING literal 抛 `AnalysisException`）。④`CheckPolicy` **中立臂**（keys `LogicalFileScan`+`instanceof PluginDrivenExternalTable`+`scanParams.isPresent()`——**零 source-specific**，legacy `LogicalHudiScan` 臂在前先命中不重叠；从 `StatementContext.getSnapshot` 取分析期已解析快照→`PluginDrivenExternalTable.getSyntheticScanPredicates(MvccSnapshot)`〔连接器/句柄/快照解包 plumbing，镜像 resolveWriteTargetHandle〕→反向转换绑扫描输出 slot→注入 `LogicalFilter`）。**关键=注入必在分析层非扫描节点**（扫描节点无法为未投影列造 slot；窗口经 `ConnectorMvccSnapshot` 而非 scan-node 才盖的 handle→analysis-time 可达）。⑤`getTableAvroSchema()→getTableAvroSchema(true)`（连接器 schema+扫描侧 lockstep）=legacy 平价、common(populate.meta.fields=true)表字节不变、robust 暴露 `_hoodie_commit_time` 可绑列。**iceberg/paimon 返空默认→计划字节不变；DORMANT**（hms 未进 SPI_READY_TYPES + hudi getTableHandle 分流=HD-B2 后续步；live iceberg/paimon @branch/@tag 命中臂但 SPI 空→字节不变，多一次 same-statement 缓存命中 getTableHandle）。全绿（hudi `HudiIncrementalTest` 18+`HudiSchemaParityTest` 4 / hive `HiveConnectorMetadataSiblingDelegationTest` 8〔+divert+完整性锁+hive 空〕/ fe-core `ConnectorExpressionToNereidsConverterTest` 8+`PluginDrivenExternalTableTest` 33+`CheckRowPolicyTest` 6 无回归 + api/hive/hudi/fe-core checkstyle 0 + import gate 净）。**下一步 = INC-5 翻闸接线**。**⚠残留（e2e/翻闸，非静默）**：①`getTableAvroSchema(true)` 对 populate.meta.fields=false 表的 SELECT*/DESCRIBE 元数据列暴露只 e2e 可证（单测只覆盖 `avroSchemaToColumns` 名/型/可见保真=绑定前提，非 live getTableAvroSchema）；②行级过滤正确性（straddling-base-file carry-forward）+ 兄弟委派跨加载器只 e2e 可证；③原生 `hoodie.datasource.read.*.instanttime` 窗口键是否收（INC-1 延后项）=INC-5/e2e 决策；④INC-5 翻闸删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`。 + - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **✅ INC-4 DONE（`c692f4fe763`/`537afcb37f6`/`81b20559f45`+复审修 `48bc2b207cd`）** 中立 SPI（**收 `ConnectorMvccSnapshot` 非 `TableScanParams`**——设计原签名编译不过已纠正）+ fe-core 反向 `ConnectorExpression→Nereids` 转换器（~100 LOC 实测，非设计估的 150-250；TOTAL+fail-loud）+ CheckPolicy 中立臂（keys `PluginDrivenExternalTable`+scanParams，注入分析层）+ 可见元数据列（`getTableAvroSchema(true)`）〔原生 hoodie.* 窗口键决策=延后 INC-5/e2e〕→ **下一步 INC-5** 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From 3bc14571c56534410db6f1dd680b5a2946c11d87 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 18:00:56 +0800 Subject: [PATCH 125/334] [doc](catalog) HMS cutover hudi HD-C4/C5 schema-evolution recon + design + signoff MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Corrects the stale "next = INC-5" framing (INC-5 is flip-time live deletion, not a dormant step — its three targets still serve live legacy hudi). Real next dormant step = HD-C4/C5 schema evolution; user signed off FULL parity (not the narrow emit-nothing regression). Recon wf_85bd47a0-0aa (6 HEAD-grounded readers + completeness/decomposition critics). Authoritative design = hudi-schema-evolution-step-design-2026-07-09.md. Key rulings: mirror paimon (multi-entry history + per-split schema_id), NOT iceberg single-entry; lowercase every nesting level (v1 StructNode SIGABRT guard); keep current_schema_id=-1 (a real id colliding a split id drives the banned case-sensitive ConstNode degrade); field-id source = mode-aware commit-metadata InternalSchema; per-split schema_id native-branch only; zero BE change, zero new connector-api SPI (all seams pre-exist). getColumnHandles runs before the pin, so C5b re-resolves the -1 entry against the pinned schema. 6-step decomposition C4a..C4d (steady-state) + C5a..C5b (time-travel). User signed off a 2-part split: Part 1 = C4a-C4d + consolidated adversarial review, Part 2 = C5a-C5b; user starts Part 1 (C4a) in a fresh session. All dormant; flip-time e2e owed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 +- ...schema-evolution-step-design-2026-07-09.md | 120 ++++++++++++++++++ 2 files changed, 122 insertions(+), 1 deletion(-) create mode 100644 plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index ae2941bda10b68..5e03be747fe531 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**下一步 = HD-C3 余 1 步（INC-5 翻闸接线：删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + legacy hudi source）→ HD-C4/C5 schema evolution（用户须定范围）+ Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**用户将在下个 session 开始 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器，完全可单测）**。之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -94,6 +94,7 @@ - **✅ HD-C3 INC-3 DONE（`0fea5bc6ba4`，休眠）= 增量 planScan 接线**（对抗复审 `wf_99c5b718-c11` 4 维〔snapshot byte-parity / incremental-correctness / dormancy-transport / test-intent〕+ 逐条 adversarial-verify **0 发现**）。`HudiScanPlanProvider.planScan` 加 `getBeginInstant()!=null` 增量支路（置于 schema 解析后、fsView 构建前，纯早返；非增量读〔含 FOR TIME AS OF〕字节不变）：`buildIncrementalRelation` 按 `isCow`(=`metaClient.getTableType()==COPY_ON_WRITE`，权威) 建 COW/MOR 关系（IOException 重型为 `DorisConnectorException`）；`incrementalRanges` **纯静态**路由（fake 关系可离线测）——`fallbackFullTableScan()`→`Optional.empty()`→降级到最新快照扫描（legacy `getSplits:470`，读 latest；两载体互斥故 @incr 句柄无 queryInstant）；COW→`collectSplits()` 原生；MOR→`collectFileSlices()` 逐片经**抽出的** `buildMorRange`（从 `collectMorSplits` 内联 lambda 提取、快照路径字节不变）转 JNI range 钉 `getEndTs()`（窗口 END=BE 合并时刻，非 latest）、分区值按 `metaClient.getTableConfig().getPartitionFields()` 逐片解析（=legacy `HudiScanNode:391-393`，**非** HMS handle 键）。**RO-as-RT 翻转 = 被 metaClient.getTableType() 天然覆盖、不移植、不需 serde 参数上句柄**（旧翻转只为补偿 legacy 按 hive inputFormat 把 MOR `_ro` 门面误判 COW；连接器读 hoodie.properties 直接得 MERGE_ON_READ）。**用户签字方案 B（graceful，2026-07-09）**：`force_jni+COW+增量` 旧代码 `collectFileSlices()` on COW relation→`UnsupportedOperationException` 崩溃，改按**关系类型**路由→COW 增量恒原生（忽略 force_jni），不崩溃；MOR 仍经 buildMorRange 尊重 force_jni（无 log 片留 JNI）。**参数穿透**：`resolveIncremental` 把原始 @incr 选项（glob/fallback/hollow-policy）复制进 MVCC 快照属性（前缀 `hudi.incr-opt.`、跳 null 值），`applySnapshot` 前缀剥离重建进**新** `HudiTableHandle.incrementalParams`（**BE 安全**：快照属性只经 `PluginDrivenScanNode.applyMvccSnapshotPin`→`applySnapshot` 消费、从不入 BE thrift〔亲手 + 复审双证〕；begin/end 载体走 `hudi.incremental-*` 不同命名空间不混入；`toBuilder` 复制）；planScan 由 `IncrementalRelation.hollowCommitHandling(optParams)`=`valueOf(getOrDefault(policyKey,"FAIL"))` 派生策略，与 `resolveIncremental` END 轴 `INCR_HOLLOW_POLICY_KEY` 同源（文件集与窗口 END 同轴）。**行级过滤（`_hoodie_commit_time` 谓词）= INC-4；本步只挑 FILE，裸 @incr 会多读直到 INC-4（休眠无影响）**。全绿（`HudiIncrementalPlanScanTest` 7〔fake IncrementalRelation + 手造 FileSlice：fallback 降级/COW 路由不碰 collectFileSlices/COW 忽略 force_jni/MOR 钉 endTs+逐片分区值/buildMorRange JNI+native 降级〕+ `HudiIncrementalTest` 15〔+3 参数穿透+命名空间隔离+非增量 pin 空 incrementalParams〕+ `HudiIncrementalRelationTest` 16〔+3 hollowCommitHandling FAIL 默认/USE_TRANSITION_TIME/bogus 抛〕+ 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。 - **✅ HD-C3 INC-4 DONE（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠）= 中立 synthetic-predicate SPI（行级 `_hoodie_commit_time` 窗口过滤）**（**用户签字方案 A**=通用 `ConnectorExpression` 而非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f`：row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing 5 维 **0 发现** + test-intent 1 minor〔meta-col 测 javadoc overclaim + isVisible 空断言〕已修）。**问题=COW base 文件在窗口内被重写会带出窗口外旧行，只挑文件不够、须行级 `_hoodie_commit_time > begin AND <= end`；legacy 靠 source-specific fe-core 注入（`CheckPolicy`+`LogicalHudiScan.generateIncrementalExpression`），翻闸后死码+违铁律，须挪进中立 SPI（对齐 Trino 残余谓词）**。①新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,ConnectorMvccSnapshot)` 默认空——**纠正设计硬伤**：原签名收 fe-core `analysis.TableScanParams`，连接器-api 不依赖 fe-core+import gate 禁→编译不过，改收连接器-api `ConnectorMvccSnapshot`；hudi override 从**同一** `resolveIncremental` 盖的 `HUDI_INCREMENTAL_BEGIN/END_PROPERTY` 读窗口（**单一窗口权威**，与 applySnapshot 文件选择同源、永不发散）发 `> begin`(GT)+`<= end`(LE) 两 flat STRING conjunct（词典序 instant 比较，byte-faithful legacy）；非增量 pin（plain/FOR TIME）返空。②**HiveConnectorMetadata 委派 override**（**recon 揪出漏项**：无它→foreign hudi 句柄继承空默认→静默 over-read）转发兄弟，镜像 resolveTimeTravel/applySnapshot、外来句柄永不强转、hive 句柄返空。③fe-core 新 `ConnectorExpressionToNereidsConverter`（**TOTAL+FAIL-LOUD**，与 forward 转换器 drop-on-unknown **相反**：漏 conjunct=under-filter 漏行；仅 {ConnectorAnd, GT/LE/EQ/LT/GE, ColumnRef 按名绑, STRING literal}，And(size≥2) 守卫+单 conjunct 解包，其余/未解析列/非 STRING literal 抛 `AnalysisException`）。④`CheckPolicy` **中立臂**（keys `LogicalFileScan`+`instanceof PluginDrivenExternalTable`+`scanParams.isPresent()`——**零 source-specific**，legacy `LogicalHudiScan` 臂在前先命中不重叠；从 `StatementContext.getSnapshot` 取分析期已解析快照→`PluginDrivenExternalTable.getSyntheticScanPredicates(MvccSnapshot)`〔连接器/句柄/快照解包 plumbing，镜像 resolveWriteTargetHandle〕→反向转换绑扫描输出 slot→注入 `LogicalFilter`）。**关键=注入必在分析层非扫描节点**（扫描节点无法为未投影列造 slot；窗口经 `ConnectorMvccSnapshot` 而非 scan-node 才盖的 handle→analysis-time 可达）。⑤`getTableAvroSchema()→getTableAvroSchema(true)`（连接器 schema+扫描侧 lockstep）=legacy 平价、common(populate.meta.fields=true)表字节不变、robust 暴露 `_hoodie_commit_time` 可绑列。**iceberg/paimon 返空默认→计划字节不变；DORMANT**（hms 未进 SPI_READY_TYPES + hudi getTableHandle 分流=HD-B2 后续步；live iceberg/paimon @branch/@tag 命中臂但 SPI 空→字节不变，多一次 same-statement 缓存命中 getTableHandle）。全绿（hudi `HudiIncrementalTest` 18+`HudiSchemaParityTest` 4 / hive `HiveConnectorMetadataSiblingDelegationTest` 8〔+divert+完整性锁+hive 空〕/ fe-core `ConnectorExpressionToNereidsConverterTest` 8+`PluginDrivenExternalTableTest` 33+`CheckRowPolicyTest` 6 无回归 + api/hive/hudi/fe-core checkstyle 0 + import gate 净）。**下一步 = INC-5 翻闸接线**。**⚠残留（e2e/翻闸，非静默）**：①`getTableAvroSchema(true)` 对 populate.meta.fields=false 表的 SELECT*/DESCRIBE 元数据列暴露只 e2e 可证（单测只覆盖 `avroSchemaToColumns` 名/型/可见保真=绑定前提，非 live getTableAvroSchema）；②行级过滤正确性（straddling-base-file carry-forward）+ 兄弟委派跨加载器只 e2e 可证；③原生 `hoodie.datasource.read.*.instanttime` 窗口键是否收（INC-1 延后项）=INC-5/e2e 决策；④INC-5 翻闸删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`。 - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **✅ INC-4 DONE（`c692f4fe763`/`537afcb37f6`/`81b20559f45`+复审修 `48bc2b207cd`）** 中立 SPI（**收 `ConnectorMvccSnapshot` 非 `TableScanParams`**——设计原签名编译不过已纠正）+ fe-core 反向 `ConnectorExpression→Nereids` 转换器（~100 LOC 实测，非设计估的 150-250；TOTAL+fail-loud）+ CheckPolicy 中立臂（keys `PluginDrivenExternalTable`+scanParams，注入分析层）+ 可见元数据列（`getTableAvroSchema(true)`）〔原生 hoodie.* 窗口键决策=延后 INC-5/e2e〕→ **下一步 INC-5** 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 + - **✅ HD-C4/C5 schema-evolution recon + 设计 + 签字（recon `wf_85bd47a0-0aa` 6 读者+完整性/拆解 2 critic〔iron-rule 读者 StructuredOutput 上限失败，gap 由 critic+亲手核对补〕；权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`，起步必读）= 完整对齐老实现（用户签字 B，非窄退化；memory `hudi-schema-evolution-full-parity-signoff`）**。**BE 核实**：原生机器已就位、**零 BE 改**（`be/src/format_v2/table/hudi_reader.cpp` 已读 `hudi_params.schema_id`+`can_map_by_history_schema`+`annotate_file_schema_from_history`，与 paimon 同）。**架构裁决**：**模板=paimon 多条历史(-1+每引用版本)+每-split schema_id**（非 iceberg 单条；`THudiFileDesc` 有 f12 schema_id 而 `TIcebergFileDesc` 无=load-bearing divergence）；**小写逐层=iceberg**（format_v2 大小写不敏感但同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader→`StructNode.children.at()` raw name→std::out_of_range→整库 SIGABRT）；**current_schema_id 保持 -1**（真 id 撞某 split id→v1 `current==split`→ConstNode 大小写敏感逐字=禁的半吊子退化；paimon+legacy 均 -1，BE `find_external_root_field` 认 -1 条目为 target overlay）；**field-id 源=mode-aware commit-metadata InternalSchema**（stable across renames；非 naive `convert(latest avro)` 否则 -1 条目 id≠per-file id→BY_FIELD_ID 失配）；**每-split schema_id 只盖原生分支**（JNI/MOR-realtime 不看 schema_id、读 `column_names/types`@instant→C5b 全靠 FE 供）；**零新 SPI**（3-arg `getTableSchema`+hive 委派已发 `HiveConnectorMetadata:1070-1078`、`populateScanLevelParams` no-op、`getScanNodeProperties`、`ConnectorColumn.withUniqueId`、`THudiFileDesc` f12、`TFileScanRangeParams` 25/26 全在；100% 连接器内部）。**关键结构事实**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:973` vs `:979`）→句柄 field-id 是 latest-keyed→时间旅行改名列须 **C5b 按 pinned schema 重解 -1 条目**（iceberg full-pinned/empty-requested 式），非靠句柄；**C4c 硬序在 C4d 前**（字典present 而 per-split schema_id 仍 -1 = 禁的 current==file==-1）。**§6 签字决策（对齐 legacy 字节忠实）**：历史集=被选中文件涉及版本的并集(非全量枚举、自洽=每发出 split id 必在史、=legacy per-referenced-file、C4c/C4d 共享 resolver 背靠背落)；非演进表 FOR TIME=latest-fallback(schema 从未变=字节等价)；scalar `TColumnType`=STRING placeholder(BE 只用作 nested-vs-scalar 判别，不 port legacy 全 avro→Doris 型表)；field-id 留句柄(=legacy `updateHudiColumnUniqueId`)。**用户签字两部分拆分（2026-07-09）**：**Part 1=C4a–C4d + 完成后统一对抗复审；Part 2=C5a–C5b。用户下个 session 起 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器：`InternalSchema/Types.Field→TSchema/TField`+base64 encode/apply 往返，逐层小写、STRING placeholder、id+name+is_optional 每层，wired 进空，完全可单测，零 fe-core import）**。**e2e 硬门（切换期，非静默）**：改名/调序/加/删列 COW+MOR 读(native BY_FIELD_ID engage)、混合大小写嵌套结构不崩、演进表 FOR TIME(COW 列表+MOR JNI 列表@instant)、非演进 FOR TIME 等价、@incr latest schema、异构 HMS vs 独立 hudi 目录同表比行（memory `hms-iceberg-delegation-needs-e2e`）。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md b/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md new file mode 100644 index 00000000000000..3e0ef043c0c9b6 --- /dev/null +++ b/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md @@ -0,0 +1,120 @@ +# Hudi-on-HMS schema-evolution parity — authoritative step design (HD-C4/HD-C5) + +> Recon: `wf_85bd47a0-0aa` (6 HEAD-grounded readers + completeness + decomposition critics; iron-rule reader failed, gap closed by the critics + hand verification). All file:line HEAD-verified. Signed-off scope = **full parity** (memory `hudi-schema-evolution-full-parity-signoff`, 2026-07-09). This is the hardest hudi sub-area. Every sub-step is a **dormant** connector-internal commit (hms not in `SPI_READY_TYPES`; hudi handles not yet diverted) with a same-loader unit test where possible; correctness is only observable on BE at flip-time e2e. + +--- + +## 0. Problem + +Post-flip a hudi-on-HMS table is a generic `PluginDrivenExternalTable`. Today the hudi connector emits **nothing** for schema-history → BE `can_map_by_history_schema` returns false (`be/src/format_v2/table/schema_history_util.cpp:124`) → `BY_NAME` fallback (case-insensitive `to_lower` matching). That safe baseline is correct for plain/add/drop/reorder reads but a **renamed** column reads NULL on old files (its old physical name no longer matches the new table name). To reach legacy parity the connector must drive BE's `BY_FIELD_ID` path by emitting field-ids + a per-version schema history + a per-split schema id, and must resolve schema **at the pinned instant** for time travel. + +--- + +## 1. BE contract (what FE must populate — NO BE code changes) + +BE's native format_v2 hudi machinery is **already in place and identical to paimon** (`be/src/format_v2/table/hudi_reader.cpp:30-52` mirrors `paimon_reader.cpp:36-55`). Field-id mapping engages **only** when ALL of the following are populated together: + +1. **`TFileScanRangeParams.current_schema_id`** (PlanNodes.thrift field 25) — **keep `-1`** (the sentinel). BE `find_external_root_field` (`be/src/format_v2/table_reader.cpp:254`) selects the history `TSchema` whose `schema_id == current_schema_id` as the *target overlay* and stamps each projected table column `identifier = TYPE_INT(field.id)` from it (`:514`). A real id equal to any per-split id would drive the **v1** engine's `current==split → ConstNode` case-sensitive verbatim-name path (`be/src/format/table/table_schema_change_helper.h:243`) = **the banned half-fix**. `-1` is `!= ` every split id (splits are `>=0`) ⇒ always the field-id path. +2. **`TFileScanRangeParams.history_schema_info`** (field 26, `list`) covering: (a) one `TSchema{schema_id=-1}` — the *target/current* overlay, built from the **requested** columns (names == BE scan slots by construction, the CI-969249 `StructNode` DCHECK invariant); (b) one `TSchema` per distinct per-split `schema_id` in the scan. Each `TField` MUST carry `id` (InternalSchema field id, **stable across renames**), `name` (**physical file column name in THAT version**), `type` (`TColumnType`), and `nestedField` for struct/array/map — `id`+`name` at **every** nesting level. +3. **`THudiFileDesc.schema_id`** (field 12, **native reader only**) = the InternalSchema version of the commit that wrote that base file; `>=0` and present in `history_schema_info`. Unset/`-1` ⇒ `BY_NAME` (safe, no evolution). + +**Mapping mechanics** (`schema_history_util.cpp:132`, `column_mapper.cpp:102`): file columns are stamped with the split-schema's field-ids by `to_lower(name)` match against the physical file columns; table columns are stamped with the `-1`-entry ids; `FieldIdMatcher` maps table→file **purely by integer id** — a rename works because the same id appears under a different name in the `-1` vs split `TSchema`. A missing id ⇒ `ranges::find_if` end ⇒ nullptr ⇒ column becomes missing/const **silently** (no throw). ⇒ **populate `id` at EVERY level of EVERY `TSchema`** or you get silent corruption strictly worse than `BY_NAME`. + +**JNI / MOR-realtime path** (`be/src/format_v2/jni/hudi_jni_reader.cpp`) consumes **no** schema_id / history / field-id. It forwards `hudi_params.column_names` + `column_types` (fields 8/9) + `instant_time` verbatim to `HadoopHudiJniScanner` and projects by `table_column.name`. So MOR evolution correctness is **entirely FE-supplied** = HD-C5. + +**Lowercase rule (load-bearing).** format_v2 hudi is fully case-insensitive (`to_lower` everywhere) and cannot SIGABRT. But the **same** `history_schema_info` thrift is consumed by the **v1** `format/table` hudi reader (`table_schema_change_helper.cpp:393` builds `StructNode` children keyed by the **raw** history `TField.name`; `helper.h:143/162/167` then `children.at(lowercased_table_name)` → `std::out_of_range` → uncaught → whole-process SIGABRT). ⇒ the FE emitter MUST lowercase **every** `TField.name` at **every** nesting level (mirror the iceberg `DROP_AND_ADD` fix; memory `catalog-spi-history-schema-info-lowercase-nested-names`). + +--- + +## 2. Template = PAIMON (not iceberg), lowercase = ICEBERG + +| Concern | iceberg | paimon | **hudi = ** | +|---|---|---|---| +| BE path | `by_file_field_id` (ids in parquet/orc metadata) | `by_table_field_id` (FE-supplied history) | **paimon** — needs per-version history + per-split id | +| history entries | ONE (`-1` only) | `-1` + one per `SchemaManager.listAllIds()` | **paimon-style multi-entry** | +| per-split `schema_id` | none (`TIcebergFileDesc` has no such field) | `RawFile.schemaId()`, native branch only | **per-file, native branch only** | +| `current_schema_id` | `-1` | `-1` | **`-1`** | +| `-1` entry source | requested columns | requested columns (name-match pinned schema, latest fallback) | **requested column handles** (CI-969249) | +| lowercase | **every level** (`IcebergSchemaUtils.buildField:245`, `Locale.ROOT`) | top-level only (`PaimonScanPlanProvider:1521`) | **iceberg — every level** | +| scalar `TColumnType` | STRING placeholder (nested-vs-scalar discriminator) | STRING placeholder | **STRING placeholder** (BE ignores scalar tag on field-id path; do NOT port legacy full Doris-type map) | + +Reference emitters: `PaimonScanPlanProvider.buildSchemaEvolutionParam:1363-1395` / `buildField:1532-1578` / `PaimonScanRange.populateRangeParams:238-241` (per-split, native-only) / `encode+apply:1462/1477`. Iceberg lowercase call site: `IcebergSchemaUtils.buildField:245`. + +--- + +## 3. What to port from legacy (fe-core → connector) + +Legacy flow (`HudiScanNode` + `HudiUtils` + `HudiSchemaCacheKey/Value` + `HMSExternalTable.initHudiSchema` + `HiveMetaStoreClientHelper.getHudiTableSchema`): + +- **Field-id source = MODE-AWARE InternalSchema** (`HiveMetaStoreClientHelper.getHudiTableSchema:829-857`): evolution on ⇒ `getTableInternalSchemaFromCommitMetadata(instant)` (stable ids); evolution off ⇒ `AvroInternalSchemaConverter.convert(getTableAvroSchema(true))` (positional ids, versionId 0). **DO NOT** source the `-1`-entry/handle ids from a naive `convert(latest avro)` when evolution is on — the completeness critic's crux: naive-convert ids won't match the per-file `getCommitInstantInternalSchema` ids ⇒ `BY_FIELD_ID` mismatch. Same logical column = same id across versions **only** through the mode-aware path. +- **Per-file `schema_id`** (`HudiScanNode.setHudiParams:304-323`): native branch only; `FSUtils.getCommitTime()` → `InternalSchemaCache.searchSchemaAndCache` (evolution) = real versionId, else `convert()` = **0** (never `-1`; JAR-confirmed). JNI branch never sets it. +- **`TField` builder** (`HudiUtils.getSchemaInfo:350-421`): port as a self-contained `Types.Field → TField` converter — but **add every-level lowercasing** (legacy lowercases nothing, `:363`) and emit STRING-placeholder scalar type (legacy emits full Doris type `:407-410` — drop it). +- **Schema-at-instant** (`HudiScanNode.doInitialize:206-224`, `getSchemaCacheValue` keyed by `(nameMapping, queryInstant)`): resolve columns/colTypes/ids AT the pinned instant. **Legacy limitation to mirror:** `getTableInternalSchemaFromCommitMetadata` honors the instant **only** when schema.on.read is enabled; else falls back to LATEST avro. For a non-evolution table this is byte-equivalent (schema never changed). +- **metaClient/storage coupling** (HD-C1/C2): `hmsTable.getHudiClient()` → connector metaClient; every timeline/InternalSchema/`searchSchemaAndCache` touch on the metadata thread must be wrapped in `HudiMetaClientExecutor.execute` (legacy left `getHudiTableSchema`/`getCommitInstantInternalSchema` UNWRAPPED — the port must be STRICTER, iron rule). `getScanNodeProperties`/`planScan` already run on the TCCL-pinned scan thread. + +--- + +## 4. Architecture placement & iron rules + +- **100% connector-internal to `fe-connector-hudi`. NO new connector-api SPI method** — every seam pre-exists: `ConnectorTableOps.getTableSchema(session,handle,snapshot)` 3-arg default (+ hive gateway delegation ALREADY shipped `HiveConnectorMetadata.java:1070-1078`); `ConnectorScanPlanProvider.populateScanLevelParams` default no-op; `getScanNodeProperties`; `ConnectorColumn.withUniqueId`; `THudiFileDesc` field 12; `TFileScanRangeParams` 25/26. +- **fe-core stays source-agnostic** — it only plumbs the base64 `hudi.schema_evolution` scan-node prop and the per-split `hudi.schema_id` range prop; zero schema logic, zero `if(format)`. +- **Sibling delegation already reaches the hudi path** (recon-confirmed): `HiveConnector.getScanPlanProvider(handle):165-169` routes a foreign hudi handle to `resolveSiblingOwner(handle).getScanPlanProvider(handle)` (so `getScanNodeProperties`/`populateScanLevelParams` run); `getColumnHandles` delegated at `HiveConnectorMetadata:382`. No new hive-gateway change. +- **`getColumnHandles` runs BEFORE the MVCC pin** (`PluginDrivenScanNode.java:973` vs `pinMvccSnapshot:979`) ⇒ handle field-ids are LATEST-keyed. Fine for steady-state (InternalSchema ids stable). Under time travel a renamed column's pinned name is absent from the latest-built handle map ⇒ dropped from `columns`. ⇒ **C5b MUST re-resolve the `-1` entry against the pinned schema** (iceberg full-pinned / empty-requested pattern), not rely on the handle. + +--- + +## 5. Dormant-commit decomposition (ascending same-loader testability; C4* before C5*) + +- **HD-C4a — `HudiSchemaUtils` converter (new connector class).** Self-contained `InternalSchema`/`Types.Field → TSchema/TStructField/TField` builder + `encode`/`apply` round-trip on a throwaway `TFileScanRangeParams` (fields 25/26). Lowercase EVERY level (mirror `IcebergSchemaUtils.buildField:245`, NOT legacy verbatim, NOT paimon top-only); scalar `TColumnType` = STRING placeholder; `id`+`name`+`is_optional` at every level. Pure library, wired into nothing yet. Zero fe-core import. *Test (fully same-loader):* hand-build an InternalSchema with a mixed-case nested struct child; assert every emitted `TField.name` lowercased at every level, current/history round-trips base64, scalar `type.type==STRING`, nested ARRAY/MAP/STRUCT tags. Mirror `HudiSchemaParityTest`. **deps: none.** +- **HD-C4b — field-ids onto `HudiColumnHandle`.** Add `int fieldId` (default `ConnectorColumn.UNSET_UNIQUE_ID=-1`; ctor+getter; **KEEP equals/hashCode by name** like `IcebergColumnHandle:55-69` — do NOT add id to identity). Source ids from the **mode-aware** InternalSchema (mirror legacy `updateHudiColumnUniqueId` + `initHudiSchema`), via `ConnectorColumn.withUniqueId`; `getColumnHandles` threads `col.getUniqueId()` onto the handle. *Test (fully same-loader):* build an Avro/InternalSchema; assert each `ConnectorColumn.getUniqueId()==` the InternalSchema field id (non-evolution convert path); evolution-mode commit-metadata id source is e2e. **deps: none.** (Paimon-style provider-side name-resolution was considered and **rejected** in favor of legacy fidelity + reuse for future nested-prune field-ids; see §6.) +- **HD-C4c — per-split `THudiFileDesc.schema_id` (native branch ONLY).** `HudiScanRange.Builder` gains `schemaId`; `populateRangeParams` sets `fileDesc.setSchemaId` ONLY in the native else-branch (`HudiScanRange.java:219-221`), NEVER the JNI branch (`:192-218`) — legacy/paimon parity. Provider computes per-file id in `collectCowSplits` and the native no-log MOR read-optimized slice that **downgrades to native** (`HudiScanRange:179-183` — NOT only COW): `FSUtils.getCommitTime(fileName) → InternalSchemaCache.searchSchemaAndCache` (evolution) else fallback InternalSchema id `0`. Runs on the TCCL-pinned scan thread. Factor a **shared static per-file→schemaId resolver** into `HudiSchemaUtils` that C4d reuses. *Test:* stamping is same-loader (`HudiScanRangeTest`: Builder→populateRangeParams asserts field 12 set on native, UNSET on JNI); the id value is e2e. **deps: shares resolver with C4d.** +- **HD-C4d — scan-level dict emission + apply (steady-state / no-pin).** `getScanNodeProperties` builds a base64 `hudi.schema_evolution` prop = `current_schema_id(-1)` + the `-1` entry from the **requested** columns arg + one entry per **referenced-split** InternalSchema version (via the C4c shared resolver + C4a converter). OVERRIDE `populateScanLevelParams` to copy fields 25/26. Gate OFF for `force_jni`/MOR-realtime-only handles (paimon gate parity). *Test:* encode/decode round-trip same-loader; history-set completeness + BE engage are e2e. **deps: C4a; shares C4c resolver. HARD-ORDER after C4c** (a dormant prefix flipping with the dict present while per-split `schema_id` is unset(`-1`) is the forbidden `current==file==-1` v1-ConstNode degrade — land C4c+C4d back-to-back). +- **HD-C5a — schema-at-instant column list (`getTableSchema` 3-arg override).** Override `ConnectorTableOps.getTableSchema(session,handle,snapshot)` on `HudiConnectorMetadata` (ABSENT today → SPI default returns latest = HD-C2 residual #1). Key off `HudiTableHandle.getQueryInstant()` (NOT `snapshot.getSchemaId()` which stays `-1` for hudi); null instant → delegate to the 2-arg latest path (shared build-path so latest/at-instant can't drift). Resolve InternalSchema AT the instant WRAPPED in `HudiMetaClientExecutor.execute` (also closes the existing unwrapped `getSchemaFromMetaClient` gap). Hive delegation already shipped ⇒ no hive change. *Test:* null-instant→latest delegation branch is same-loader; the at-instant read is e2e. **deps: C4b.** +- **HD-C5b — scan-side at-instant: MOR/JNI column list + `-1` entry at pinned instant.** (i) `planScan` JNI `column_names`/`column_types` re-derived from the queryInstant-scoped schema (replacing latest `getTableAvroSchema(true)`) — legacy `HudiScanNode:222-224`; keep BOTH schema loci in lockstep. (ii) The dict `-1` entry re-resolved against the **pinned** InternalSchema (iceberg full-pinned / empty-requested) because handle ids are latest-keyed — else a renamed column drops a BE slot. `current_schema_id` stays `-1`. All off-scan-thread timeline touches under `HudiMetaClientExecutor.execute`. *Test:* almost entirely e2e; FE-unit asserts JNI col list + `-1` entry are derived from the instant-resolved schema, not latest. **deps: C4c, C4d, C5a.** + +--- + +## 6. Resolved design decisions (mirror legacy byte-faithfully — consistent with the full-parity sign-off) + +- **D1 — history-set enumeration = union of REFERENCED-split ids** (not all-versions, not lazy-per-split). Hudi has no `SchemaManager.listAllIds()`. The C4c per-file resolver already computes each selected split's `schema_id`; the C4d dict emits `-1` + exactly those referenced ids. Self-consistent by construction (every emitted split id is in history ⇒ no BE fail-loud "miss schema info"), cheaper than enumerating history, and byte-faithful to legacy's per-referenced-file `putHistorySchemaInfo` (just up-front at plan level, not lazy). ⇒ C4c and C4d share the resolver and land back-to-back. +- **D2 — `current_schema_id` stays `-1`, `-1` entry made pinned-correct under time travel** (C5b), rejecting the time-travel reader's "emit a real current_schema_id" (would risk the v1 `ConstNode` case-sensitive degrade; paimon + legacy both emit `-1`). +- **D3 — non-evolution FOR TIME AS OF mirrors legacy's latest-fallback.** For a non-evolution table the schema never changed via InternalSchema, so "latest" == "at-instant" — byte-equivalent. Evolution (schema.on.read) tables get true schema-at-instant. No extension beyond legacy. +- **D4 — keep field-id on the handle (C4b).** Mirrors legacy `updateHudiColumnUniqueId` + the plan mandate; the steady-state `-1` entry reads ids from the requested handles directly; same field-id also unblocks future nested-column-prune. Paimon-style provider-side name-resolution was the simpler alternative but diverges from legacy and would still need C5b re-resolve under pin anyway. +- **D5 — scalar `TColumnType` = STRING placeholder** (BE by_table_field_id uses `type.type` only as a nested-vs-scalar discriminator; don't port the legacy full avro→Doris type map). + +--- + +## 7. Correctness gates / landmines + +1. **Half-fix ban**: never engage `BY_FIELD_ID` (split id `>=0` + history set) with any `TField.id` unset → silent column drop. Populate `id` at every level of every `TSchema`. +2. **`current==split==-1` ban**: keep `current_schema_id=-1` with splits `>=0`. +3. **Split-entry names = physical file column names at that commit** (BE stamps file cols by name to get ids). +4. **Lowercase every nested level** (v1 SIGABRT guard). +5. **TCCL/UGI**: every new timeline/InternalSchema/`searchSchemaAndCache` touch under `HudiMetaClientExecutor.execute`. +6. **JNI omits schema_id**; only native base-file / native-downgraded MOR-RO slices carry it. +7. **`@incr` lists LATEST schema** (both sides; PluginDrivenMvccExternalTable INCREMENTAL branch + legacy leaves snapshot null) — intended, not a missing pin. + +--- + +## 8. Flip-time e2e (per memory `hms-iceberg-delegation-needs-e2e`) + +Heterogeneous-HMS catalog, hudi tables vs a standalone hudi catalog, same rows: +- COW + MOR schema-evolved (rename + reorder + add + drop) read (native BY_FIELD_ID engages, renamed column reads correctly on old files). +- Mixed-case nested-struct-field table (no SIGABRT on any reachable path). +- `FOR TIME AS OF` on a schema-evolved COW table (schema-at-instant column list) and a schema-evolved MOR table (JNI column list @instant). +- Non-evolution `FOR TIME AS OF` (latest-fallback byte-equivalence). +- `@incr` over an evolved table (latest schema). + +--- + +## 9. Status — DESIGN SIGNED OFF (2026-07-09), split into two parts + +User signed off the full design + all §6 decisions, with an explicit **two-part split** (2026-07-09): +- **Part 1 = HD-C4 steady-state read evolution (C4a→C4b→C4c→C4d)** as independent dormant commits, each with its §5 same-loader test, THEN a consolidated **adversarial review** of the whole steady-state part before Part 2. C4c+C4d land back-to-back (D1 shared resolver + prefix-flip safety). +- **Part 2 = HD-C5 time-travel-over-evolved (C5a→C5b)** as independent dormant commits + review, next work line after Part 1. + +User will **start Part 1 in a fresh session**. Owed at flip: the §8 e2e. + +**First actionable step (Part 1, C4a):** new connector-internal `HudiSchemaUtils` — pure `InternalSchema`/`Types.Field → TSchema/TStructField/TField` converter + base64 encode/apply round-trip on a throwaway `TFileScanRangeParams` (fields 25/26); lowercase every level (mirror `IcebergSchemaUtils.buildField`), STRING-placeholder scalar tag, `id`+`name`+`is_optional` at every level; wired into nothing yet; fully same-loader unit-testable. Zero fe-core import. + +Related: [[hudi-on-hms-delegation-plan-2026-07-08.md]] §HD-C4/C5, `hudi-time-travel-step-design-2026-07-09.md` (pin spine), recon `wf_85bd47a0-0aa`, memories `hudi-schema-evolution-full-parity-signoff` + `catalog-spi-history-schema-info-lowercase-nested-names`. From 8e490ba66d0016bc5ba8efce913a1a3df95d2e1c Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 18:21:42 +0800 Subject: [PATCH 126/334] [feat](catalog) HMS cutover hudi HD-C4a: dormant HudiSchemaUtils schema-evolution converter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First step of the hudi-on-HMS schema-evolution parity work (HD-C4, steady-state read evolution). Adds a self-contained connector-internal HudiSchemaUtils that converts a hudi InternalSchema / Types.Field into the BE native-reader schema dictionary thrift (TSchema / TStructField / TField) and round-trips it through a base64 TFileScanRangeParams carrier (fields current_schema_id 25 + history_schema_info 26). Pure library, wired into nothing yet; later steps (C4c/C4d) reuse buildSchemaInfo + encode to emit the -1 target entry, the per-referenced-version history, and the per-split schema_id. Port of legacy HudiUtils.getSchemaInfo(InternalSchema / List / Types.Field) with two deliberate deviations, both pinned by mutation-tight tests: - lowercase EVERY nesting level (Locale.ROOT, mirroring IcebergSchemaUtils.buildField; legacy lowercased nothing) — the same history_schema_info thrift also feeds the v1 format/table hudi reader whose StructNode is keyed by the lowercase Doris slot name, so a mixed-case nested name would std::out_of_range-SIGABRT the whole BE; - scalar TColumnType = STRING placeholder (BE reads type.type only as a nested-vs-scalar discriminator on the field-id path), dropping legacy's full fromAvroHudiTypeToDorisType map. id + name + is_optional are carried at every nesting level (legacy parity). encode fails loud on serialize error; applySchemaEvolution is a no-op on null/empty and fails loud on a corrupt prop (mirrors iceberg/paimon). Template = paimon (per-version multi-entry + per-split schema_id), lowercase = iceberg (every level). Zero fe-core import; zero new connector-api SPI. DORMANT (hms not in SPI_READY_TYPES; hudi handles not yet diverted). Test HudiSchemaUtilsTest (11, fully same-loader): hand-built InternalSchema with a mixed-case nested struct child asserts every-level lowercasing, STRING scalar tag, struct/array/map nested tags, id/is_optional preserved, the InternalSchema-keyed vs explicit -1 schema id, and the base64 encode/apply round-trip (+ null/empty no-op, corrupt fail-loud). Full hudi module 138/138 green; checkstyle 0; import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hudi/HudiSchemaUtils.java | 215 +++++++++++++++ .../connector/hudi/HudiSchemaUtilsTest.java | 251 ++++++++++++++++++ 2 files changed, 466 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java new file mode 100644 index 00000000000000..10ac57ca2508f2 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java @@ -0,0 +1,215 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.thrift.TColumnType; +import org.apache.doris.thrift.TFileScanRangeParams; +import org.apache.doris.thrift.TPrimitiveType; +import org.apache.doris.thrift.schema.external.TArrayField; +import org.apache.doris.thrift.schema.external.TField; +import org.apache.doris.thrift.schema.external.TFieldPtr; +import org.apache.doris.thrift.schema.external.TMapField; +import org.apache.doris.thrift.schema.external.TNestedField; +import org.apache.doris.thrift.schema.external.TSchema; +import org.apache.doris.thrift.schema.external.TStructField; + +import org.apache.hudi.internal.schema.InternalSchema; +import org.apache.hudi.internal.schema.Types; +import org.apache.thrift.TDeserializer; +import org.apache.thrift.TSerializer; +import org.apache.thrift.protocol.TBinaryProtocol; + +import java.util.Base64; +import java.util.List; +import java.util.Locale; + +/** + * Builds the native-reader schema dictionary ({@code current_schema_id} + {@code history_schema_info}) so BE + * matches file↔table columns BY FIELD ID across schema evolution (rename/reorder), instead of falling back + * to NAME matching (which silently reads NULL for a renamed column on its old files). Self-contained + * hudi→thrift port of legacy {@code HudiUtils.getSchemaInfo(InternalSchema)} + {@code getSchemaInfo( + * List<Types.Field>)} + {@code getSchemaInfo(Types.Field)} (zero fe-core import). + * + *

    Template = paimon, lowercase = iceberg. Like paimon ({@code by_table_field_id}) hudi supplies the + * per-file schema to BE, so the dictionary needs a per-committed-version history ({@code -1} target entry + one + * entry per referenced split {@code schema_id}) — unlike iceberg's single {@code -1} entry. Field ids come from + * hudi's {@link InternalSchema} (stable across renames). This class is only the pure {@code InternalSchema -> + * thrift} converter + the base64 transport round-trip; the dictionary orchestration (the {@code -1} entry from + * the requested columns, the per-referenced-version entries, the per-split {@code schema_id}) lands in later + * steps that reuse {@link #buildSchemaInfo} + {@link #encode}.

    + * + *

    Two deliberate deviations from legacy ({@code HudiUtils.getSchemaInfo}): + *

      + *
    • Lowercase EVERY nesting level ({@code Locale.ROOT}, mirroring + * {@code IcebergSchemaUtils.buildField}) — legacy lowercased nothing. The same {@code history_schema_info} + * thrift is also consumed by the v1 {@code format/table} hudi reader, whose {@code StructNode} is keyed by + * the RAW history {@code TField.name} then looked up by the LOWERCASED Doris slot name; a mixed-case + * nested field name there throws {@code std::out_of_range} and SIGABRTs the whole BE (mirror of the iceberg + * {@code DROP_AND_ADD} fix).
      • + *
    • Scalar {@code TColumnType} = {@code STRING} placeholder — legacy emitted the full Doris type + * ({@code fromAvroHudiTypeToDorisType(...).toColumnTypeThrift()}); BE uses {@code type.type} only as a + * nested-vs-scalar discriminator on the field-id path, so a single placeholder suffices and avoids porting + * the avro→Doris type map.
      • + *
    + * {@code id} + {@code name} + {@code is_optional} are carried at EVERY nesting level (legacy parity — a missing + * {@code id} at any level makes BE's field-id matcher silently drop the column, worse than {@code BY_NAME}).

    + */ +public final class HudiSchemaUtils { + + // Legacy parity: current_schema_id is the -1 sentinel ("latest"/target); the current/target schema is also + // pushed into history_schema_info under this id. BE's find_external_root_field selects the -1 entry as the + // table-side overlay; a real id equal to any per-split id would drive the banned v1 case-sensitive path. + static final long CURRENT_SCHEMA_ID = -1L; + + private static final Base64.Encoder BASE64_ENCODER = Base64.getEncoder(); + + private HudiSchemaUtils() { + } + + /** + * Build one {@link TSchema} from a hudi {@link InternalSchema}, keyed by the schema's own committed id + * (a per-version history entry). Port of legacy {@code HudiUtils.getSchemaInfo(InternalSchema)}. + */ + static TSchema buildSchemaInfo(InternalSchema internalSchema) { + return buildSchemaInfo(internalSchema.schemaId(), internalSchema.getRecord().fields()); + } + + /** + * Build one {@link TSchema} from an explicit schema id + top-level fields. The later {@code -1} target entry + * (built from the requested columns) reuses this with {@link #CURRENT_SCHEMA_ID}. + */ + static TSchema buildSchemaInfo(long schemaId, List fields) { + TSchema tSchema = new TSchema(); + tSchema.setSchemaId(schemaId); + tSchema.setRootField(buildStructField(fields)); + return tSchema; + } + + private static TStructField buildStructField(List fields) { + TStructField structField = new TStructField(); + for (Types.Field field : fields) { + structField.addToFields(fieldPtr(buildField(field))); + } + return structField; + } + + /** + * Recursively convert a hudi {@link Types.Field} to a {@link TField}, carrying {@code id} / {@code name} + * (lowercased with {@code Locale.ROOT} at every level) / {@code is_optional}, a nested-vs-scalar + * {@code type.type} tag ({@code STRING} placeholder for scalars), and the nested {@code array}/{@code map}/ + * {@code struct} structure. Port of legacy {@code HudiUtils.getSchemaInfo(Types.Field)}. + */ + static TField buildField(Types.Field field) { + TField tField = new TField(); + // Lowercase every level (Locale.ROOT): BE's table-side StructNode is looked up by the lowercase Doris + // slot name; a mixed-case nested name would SIGABRT the v1 reader (see class javadoc). + tField.setName(field.name().toLowerCase(Locale.ROOT)); + tField.setId(field.fieldId()); + tField.setIsOptional(field.isOptional()); + + TColumnType columnType = new TColumnType(); + TNestedField nestedField = new TNestedField(); + switch (field.type().typeId()) { + case ARRAY: { + columnType.setType(TPrimitiveType.ARRAY); + Types.ArrayType arrayType = (Types.ArrayType) field.type(); + TArrayField arrayField = new TArrayField(); + arrayField.setItemField(fieldPtr(buildField(arrayType.fields().get(0)))); + nestedField.setArrayField(arrayField); + tField.setNestedField(nestedField); + break; + } + case MAP: { + columnType.setType(TPrimitiveType.MAP); + Types.MapType mapType = (Types.MapType) field.type(); + TMapField mapField = new TMapField(); + mapField.setKeyField(fieldPtr(buildField(mapType.fields().get(0)))); + mapField.setValueField(fieldPtr(buildField(mapType.fields().get(1)))); + nestedField.setMapField(mapField); + tField.setNestedField(nestedField); + break; + } + case RECORD: { + columnType.setType(TPrimitiveType.STRUCT); + Types.RecordType recordType = (Types.RecordType) field.type(); + nestedField.setStructField(buildStructField(recordType.fields())); + tField.setNestedField(nestedField); + break; + } + default: + // Scalar: BE reads type.type only as a nested-vs-scalar discriminator on the field-id path (it + // never inspects the specific scalar tag), so a single placeholder is sufficient and avoids + // replicating the full hudi->Doris primitive mapping (drops legacy fromAvroHudiTypeToDorisType). + columnType.setType(TPrimitiveType.STRING); + break; + } + tField.setType(columnType); + return tField; + } + + /** + * Serialize the schema dictionary into a base64 {@code TBinaryProtocol} blob, carried by a throwaway + * {@link TFileScanRangeParams} (the exact thrift target so {@link #applySchemaEvolution} only copies the two + * fields back). Mirrors iceberg/paimon. + */ + static String encode(long currentSchemaId, List history) { + TFileScanRangeParams carrier = new TFileScanRangeParams(); + carrier.setCurrentSchemaId(currentSchemaId); + carrier.setHistorySchemaInfo(history); + try { + byte[] bytes = new TSerializer(new TBinaryProtocol.Factory()).serialize(carrier); + return BASE64_ENCODER.encodeToString(bytes); + } catch (Exception | LinkageError e) { + // Catch LinkageError (e.g. IncompatibleClassChangeError from a thrift classloader split) too: wrapped + // as a RuntimeException it surfaces as a clean per-query failure instead of escaping the connection + // handler as an uncaught Error and killing the whole mysql session (mirrors iceberg/paimon). + throw new RuntimeException("Failed to serialize hudi schema-evolution info", e); + } + } + + /** + * Decode the prop produced by {@link #encode} and copy {@code current_schema_id} + {@code history_schema_info} + * onto the real scan params. Fail loud on a decode error — the prop is produced by us, so a failure is a real + * bug, and silently dropping it would re-introduce the silent wrong-rows risk on schema-evolved native reads. + * A {@code null}/empty prop (e.g. another connector's props map) is a no-op. + */ + static void applySchemaEvolution(TFileScanRangeParams params, String encoded) { + if (encoded == null || encoded.isEmpty()) { + return; + } + try { + byte[] bytes = Base64.getDecoder().decode(encoded); + TFileScanRangeParams carrier = new TFileScanRangeParams(); + new TDeserializer(new TBinaryProtocol.Factory()).deserialize(carrier, bytes); + if (carrier.isSetCurrentSchemaId()) { + params.setCurrentSchemaId(carrier.getCurrentSchemaId()); + } + if (carrier.isSetHistorySchemaInfo()) { + params.setHistorySchemaInfo(carrier.getHistorySchemaInfo()); + } + } catch (Exception e) { + throw new RuntimeException("Failed to apply hudi schema-evolution info to scan params", e); + } + } + + private static TFieldPtr fieldPtr(TField field) { + TFieldPtr ptr = new TFieldPtr(); + ptr.setFieldPtr(field); + return ptr; + } +} diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java new file mode 100644 index 00000000000000..2ec39b6443f7fe --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java @@ -0,0 +1,251 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.thrift.TFileScanRangeParams; +import org.apache.doris.thrift.TPrimitiveType; +import org.apache.doris.thrift.schema.external.TField; +import org.apache.doris.thrift.schema.external.TFieldPtr; +import org.apache.doris.thrift.schema.external.TSchema; + +import org.apache.hudi.internal.schema.InternalSchema; +import org.apache.hudi.internal.schema.Types; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; + +/** + * Same-loader unit tests for {@link HudiSchemaUtils} — the HD-C4a {@code InternalSchema -> thrift} converter that + * later steps (C4c/C4d) turn into the native-reader schema dictionary. A wrong/missing entry makes BE either + * silently read NULL for a renamed column or SIGABRT the whole process on a mixed-case nested name, so these + * assertions pin the two deliberate deviations from legacy {@code HudiUtils.getSchemaInfo} (every-level + * lowercasing + STRING-placeholder scalar) plus the id/optional/nested structure legacy already carried. No + * Mockito, no live metaClient: the {@link InternalSchema} is hand-built via {@link Types}. + * + *

    End-to-end BE field-id matching is only observable on the flip-time docker e2e (native + * {@code by_table_field_id}); these tests cover the FE-side thrift shape the dictionary is built from.

    + */ +public class HudiSchemaUtilsTest { + + // A hand-built hudi InternalSchema exercising the casing boundary + every nested container: + // Id INT (required -> is_optional=false) field id 1 + // Name STRING (optional) field id 2 + // Addr STRUCT field id 3 (child Street id 4) <- mixed-case nested + // Tags ARRAY field id 5 (element id 6) + // Props MAP field id 7 (key id 8, value id 9) + // Mixed-case top-level names (Id/Name/Addr/Tags/Props) and a mixed-case nested struct child (Street) + // exercise the every-level lowercasing crux. schemaId 42 proves the InternalSchema-keyed path uses the + // schema's OWN committed id (a per-version history entry), not the -1 sentinel. + private static final long SCHEMA_ID = 42L; + + private static InternalSchema buildInternalSchema() { + Types.Field street = Types.Field.get(4, true, "Street", Types.StringType.get()); + Types.RecordType addrType = Types.RecordType.get(Collections.singletonList(street)); + List fields = Arrays.asList( + Types.Field.get(1, false, "Id", Types.IntType.get()), + Types.Field.get(2, true, "Name", Types.StringType.get()), + Types.Field.get(3, true, "Addr", addrType), + Types.Field.get(5, true, "Tags", Types.ArrayType.get(6, true, Types.StringType.get())), + Types.Field.get(7, true, "Props", + Types.MapType.get(8, 9, Types.StringType.get(), Types.IntType.get()))); + return new InternalSchema(SCHEMA_ID, Types.RecordType.get(fields)); + } + + /** Index a struct's top-level {@link TField} children by name (preserving order for ordering assertions). */ + private static Map topFields(TSchema schema) { + Map byName = new LinkedHashMap<>(); + for (TFieldPtr ptr : schema.getRootField().getFields()) { + byName.put(ptr.getFieldPtr().getName(), ptr.getFieldPtr()); + } + return byName; + } + + private static TField structChild(TField parent, String name) { + for (TFieldPtr ptr : parent.getNestedField().getStructField().getFields()) { + if (ptr.getFieldPtr().getName().equals(name)) { + return ptr.getFieldPtr(); + } + } + throw new AssertionError("no nested field named " + name); + } + + // --- schema id: InternalSchema-keyed uses the schema's own id; explicit-id keys off the sentinel --- + + @Test + public void schemaKeyedOffInternalSchemaCommittedId() { + // A per-version history entry must carry the InternalSchema's OWN committed id so BE can resolve a native + // file's schema_id against it. MUTATION: hard-code -1 in buildSchemaInfo(InternalSchema) -> the history + // entry no longer matches any file's committed schema_id -> BE "miss table/file schema info" -> red. + TSchema schema = HudiSchemaUtils.buildSchemaInfo(buildInternalSchema()); + Assertions.assertEquals(SCHEMA_ID, schema.getSchemaId()); + } + + @Test + public void explicitSchemaIdIsCarried() { + // The later -1 target entry is built via buildSchemaInfo(CURRENT_SCHEMA_ID, fields). MUTATION: ignore the + // schemaId arg -> the target entry loses the -1 sentinel BE selects as the table-side overlay -> red. + List fields = buildInternalSchema().getRecord().fields(); + TSchema schema = HudiSchemaUtils.buildSchemaInfo(HudiSchemaUtils.CURRENT_SCHEMA_ID, fields); + Assertions.assertEquals(-1L, schema.getSchemaId()); + } + + // --- top-level: field ids + lowercased names + is_optional (legacy parity) --- + + @Test + public void topLevelFieldsCarryHudiFieldIdsAndLowercasedNames() { + // The dictionary's top-level names must be LOWERCASED (Locale.ROOT) so BE's table-side StructNode keys + // match the lowercase Doris scan slots, while the id is the hudi InternalSchema field id (the rename-safe + // join key). MUTATION: keep the hudi case ("Id") -> the lowercase slot lookup misses -> red here, silent + // NULL on BE. + Map fields = topFields(HudiSchemaUtils.buildSchemaInfo(buildInternalSchema())); + + Assertions.assertEquals(Arrays.asList("id", "name", "addr", "tags", "props"), + Arrays.asList(fields.keySet().toArray())); + Assertions.assertFalse(fields.containsKey("Id")); + Assertions.assertFalse(fields.containsKey("Name")); + + Assertions.assertEquals(1, fields.get("id").getId()); + Assertions.assertEquals(2, fields.get("name").getId()); + Assertions.assertEquals(3, fields.get("addr").getId()); + Assertions.assertEquals(5, fields.get("tags").getId()); + Assertions.assertEquals(7, fields.get("props").getId()); + } + + @Test + public void isOptionalMirrorsHudiNullability() { + // Legacy carries hudi's own optional flag at every level (getSchemaInfo sets is_optional from + // field.isOptional()). A required column stays is_optional=false; an optional one true. MUTATION: + // hard-code true (leak iceberg's force-nullable habit) -> the required "id" flips -> red. + Map fields = topFields(HudiSchemaUtils.buildSchemaInfo(buildInternalSchema())); + Assertions.assertFalse(fields.get("id").isIsOptional()); + Assertions.assertTrue(fields.get("name").isIsOptional()); + } + + // --- scalar placeholder + nested struct/array/map carry field ids at every level --- + + @Test + public void scalarFieldsUseStringPlaceholder() { + // BE reads type.type only as a nested-vs-scalar discriminator on the field-id path, so every scalar is a + // single STRING placeholder regardless of the real hudi type — the deliberate deviation from legacy's full + // fromAvroHudiTypeToDorisType map. MUTATION: port the real type (INT for "id") -> not STRING -> red; the + // assertion pins the simplification as intentional. + Map fields = topFields(HudiSchemaUtils.buildSchemaInfo(buildInternalSchema())); + Assertions.assertEquals(TPrimitiveType.STRING, fields.get("id").getType().getType()); + Assertions.assertEquals(TPrimitiveType.STRING, fields.get("name").getType().getType()); + } + + @Test + public void nestedStructChildIsLowercasedAndCarriesId() { + // THE SIGABRT crux (mirror of the iceberg DROP_AND_ADD fix): a mixed-case nested struct child ("Street") + // must be emitted LOWERCASED. The same history_schema_info thrift feeds the v1 format/table hudi reader, + // whose StructNode is looked up by the lowercase Doris slot name; keeping the hudi case makes BE's + // children.at("street") throw std::out_of_range -> whole-process SIGABRT. MUTATION (the bug): emit + // field.name() verbatim for struct children -> the lowercase key is absent / the mixed-case key leaks -> red. + Map fields = topFields(HudiSchemaUtils.buildSchemaInfo(buildInternalSchema())); + TField addr = fields.get("addr"); + Assertions.assertEquals(TPrimitiveType.STRUCT, addr.getType().getType()); + + TField street = structChild(addr, "street"); + Assertions.assertEquals(4, street.getId()); + Assertions.assertEquals(TPrimitiveType.STRING, street.getType().getType()); + // ... and the hudi-cased name must NOT leak through (that is exactly what aborts BE). + Assertions.assertThrows(AssertionError.class, () -> structChild(addr, "Street")); + } + + @Test + public void arrayElementCarriesIdAndType() { + // Faithful to legacy: the array element is a nested TField carrying its own hudi field id (BE field-id + // matches nested fields too). MUTATION: drop the element field id -> BE cannot map the element -> red. + Map fields = topFields(HudiSchemaUtils.buildSchemaInfo(buildInternalSchema())); + TField tags = fields.get("tags"); + Assertions.assertEquals(TPrimitiveType.ARRAY, tags.getType().getType()); + + TField element = tags.getNestedField().getArrayField().getItemField().getFieldPtr(); + Assertions.assertEquals(6, element.getId()); + Assertions.assertEquals(TPrimitiveType.STRING, element.getType().getType()); + } + + @Test + public void mapKeyAndValueCarryIdsAndTypes() { + // Faithful to legacy: map key (index 0) + value (index 1) each carry their own hudi field id. MUTATION: + // swap or drop the key/value ids -> BE mis-maps the map entries -> red. + Map fields = topFields(HudiSchemaUtils.buildSchemaInfo(buildInternalSchema())); + TField props = fields.get("props"); + Assertions.assertEquals(TPrimitiveType.MAP, props.getType().getType()); + + TField key = props.getNestedField().getMapField().getKeyField().getFieldPtr(); + TField value = props.getNestedField().getMapField().getValueField().getFieldPtr(); + Assertions.assertEquals(8, key.getId()); + Assertions.assertEquals(9, value.getId()); + Assertions.assertEquals(TPrimitiveType.STRING, key.getType().getType()); + Assertions.assertEquals(TPrimitiveType.STRING, value.getType().getType()); + } + + // --- round-trip through the base64 prop transport (what C4d's getScanNodeProperties/apply will do) --- + + @Test + public void encodeApplyRoundTripsThroughBase64() { + // encode() serializes the dict; applySchemaEvolution() copies current_schema_id + history_schema_info back + // onto the real params — the exact transport C4d round-trips through the scan-node props. MUTATION: drop + // either copied field in applySchemaEvolution -> the params are missing it -> red. The nested "street" + // stays lowercased after the round-trip (proves the whole tree survives serialization). + TSchema history = HudiSchemaUtils.buildSchemaInfo(buildInternalSchema()); + String encoded = HudiSchemaUtils.encode(HudiSchemaUtils.CURRENT_SCHEMA_ID, + Collections.singletonList(history)); + + TFileScanRangeParams params = new TFileScanRangeParams(); + HudiSchemaUtils.applySchemaEvolution(params, encoded); + + Assertions.assertTrue(params.isSetCurrentSchemaId()); + Assertions.assertEquals(-1L, params.getCurrentSchemaId()); + Assertions.assertEquals(1, params.getHistorySchemaInfoSize()); + TSchema decoded = params.getHistorySchemaInfo().get(0); + Assertions.assertEquals(SCHEMA_ID, decoded.getSchemaId()); + Map decodedTop = topFields(decoded); + Assertions.assertEquals(5, decodedTop.size()); + Assertions.assertEquals("street", + structChild(decodedTop.get("addr"), "street").getName().toLowerCase(Locale.ROOT)); + } + + @Test + public void applyIsNoOpForNullOrEmpty() { + // A null/empty prop (e.g. another connector's props map, or a handle that emits no dict) must leave the + // params untouched, not throw. MUTATION: drop the guard -> Base64.decode(null) NPEs -> red. + TFileScanRangeParams params = new TFileScanRangeParams(); + HudiSchemaUtils.applySchemaEvolution(params, null); + HudiSchemaUtils.applySchemaEvolution(params, ""); + Assertions.assertFalse(params.isSetCurrentSchemaId()); + Assertions.assertFalse(params.isSetHistorySchemaInfo()); + } + + @Test + public void applyFailsLoudOnCorruptProp() { + // The prop is produced by us, so a decode failure is a real bug — fail loud rather than silently drop it + // (a dropped dict re-introduces the silent wrong-rows risk on schema-evolved native reads). MUTATION: + // swallow the exception -> no throw -> red. + TFileScanRangeParams params = new TFileScanRangeParams(); + Assertions.assertThrows(RuntimeException.class, + () -> HudiSchemaUtils.applySchemaEvolution(params, "!!!not-base64-thrift!!!")); + } +} From 8084eef1a7bb6dc7f31110f32c067101cbd27c75 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 18:23:06 +0800 Subject: [PATCH 127/334] [doc](catalog) HMS cutover hudi HD-C4a DONE; next = C4b (field-id onto HudiColumnHandle) Per-round HANDOFF update: mark HD-C4a (the pure HudiSchemaUtils InternalSchema->thrift converter, dormant commit d29bb52173d) done in the current-status line and the hudi section, and set the next step to C4b (field-ids onto HudiColumnHandle from the mode-aware InternalSchema). Records the two deliberate legacy deviations (every-level Locale.ROOT lowercasing, STRING-placeholder scalar), the paimon-template/iceberg-lowercase architecture, the 11 same-loader tests, and the full-green verification. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 5e03be747fe531..49dfb2347fdba5 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**用户将在下个 session 开始 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器，完全可单测）**。之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`，mode-aware InternalSchema 取 id，`equals/hashCode` 保持按名不加 id）→ C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）→ C4d（扫描级 `hudi.schema_evolution` 字典发射+`populateScanLevelParams` apply，C4c/C4d 背靠背落〔D1 共享 resolver + prefix-flip 安全〕）→ 完成后统一对抗复审 → Part 2 C5a/C5b。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -95,6 +95,7 @@ - **✅ HD-C3 INC-4 DONE（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠）= 中立 synthetic-predicate SPI（行级 `_hoodie_commit_time` 窗口过滤）**（**用户签字方案 A**=通用 `ConnectorExpression` 而非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f`：row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing 5 维 **0 发现** + test-intent 1 minor〔meta-col 测 javadoc overclaim + isVisible 空断言〕已修）。**问题=COW base 文件在窗口内被重写会带出窗口外旧行，只挑文件不够、须行级 `_hoodie_commit_time > begin AND <= end`；legacy 靠 source-specific fe-core 注入（`CheckPolicy`+`LogicalHudiScan.generateIncrementalExpression`），翻闸后死码+违铁律，须挪进中立 SPI（对齐 Trino 残余谓词）**。①新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,ConnectorMvccSnapshot)` 默认空——**纠正设计硬伤**：原签名收 fe-core `analysis.TableScanParams`，连接器-api 不依赖 fe-core+import gate 禁→编译不过，改收连接器-api `ConnectorMvccSnapshot`；hudi override 从**同一** `resolveIncremental` 盖的 `HUDI_INCREMENTAL_BEGIN/END_PROPERTY` 读窗口（**单一窗口权威**，与 applySnapshot 文件选择同源、永不发散）发 `> begin`(GT)+`<= end`(LE) 两 flat STRING conjunct（词典序 instant 比较，byte-faithful legacy）；非增量 pin（plain/FOR TIME）返空。②**HiveConnectorMetadata 委派 override**（**recon 揪出漏项**：无它→foreign hudi 句柄继承空默认→静默 over-read）转发兄弟，镜像 resolveTimeTravel/applySnapshot、外来句柄永不强转、hive 句柄返空。③fe-core 新 `ConnectorExpressionToNereidsConverter`（**TOTAL+FAIL-LOUD**，与 forward 转换器 drop-on-unknown **相反**：漏 conjunct=under-filter 漏行；仅 {ConnectorAnd, GT/LE/EQ/LT/GE, ColumnRef 按名绑, STRING literal}，And(size≥2) 守卫+单 conjunct 解包，其余/未解析列/非 STRING literal 抛 `AnalysisException`）。④`CheckPolicy` **中立臂**（keys `LogicalFileScan`+`instanceof PluginDrivenExternalTable`+`scanParams.isPresent()`——**零 source-specific**，legacy `LogicalHudiScan` 臂在前先命中不重叠；从 `StatementContext.getSnapshot` 取分析期已解析快照→`PluginDrivenExternalTable.getSyntheticScanPredicates(MvccSnapshot)`〔连接器/句柄/快照解包 plumbing，镜像 resolveWriteTargetHandle〕→反向转换绑扫描输出 slot→注入 `LogicalFilter`）。**关键=注入必在分析层非扫描节点**（扫描节点无法为未投影列造 slot；窗口经 `ConnectorMvccSnapshot` 而非 scan-node 才盖的 handle→analysis-time 可达）。⑤`getTableAvroSchema()→getTableAvroSchema(true)`（连接器 schema+扫描侧 lockstep）=legacy 平价、common(populate.meta.fields=true)表字节不变、robust 暴露 `_hoodie_commit_time` 可绑列。**iceberg/paimon 返空默认→计划字节不变；DORMANT**（hms 未进 SPI_READY_TYPES + hudi getTableHandle 分流=HD-B2 后续步；live iceberg/paimon @branch/@tag 命中臂但 SPI 空→字节不变，多一次 same-statement 缓存命中 getTableHandle）。全绿（hudi `HudiIncrementalTest` 18+`HudiSchemaParityTest` 4 / hive `HiveConnectorMetadataSiblingDelegationTest` 8〔+divert+完整性锁+hive 空〕/ fe-core `ConnectorExpressionToNereidsConverterTest` 8+`PluginDrivenExternalTableTest` 33+`CheckRowPolicyTest` 6 无回归 + api/hive/hudi/fe-core checkstyle 0 + import gate 净）。**下一步 = INC-5 翻闸接线**。**⚠残留（e2e/翻闸，非静默）**：①`getTableAvroSchema(true)` 对 populate.meta.fields=false 表的 SELECT*/DESCRIBE 元数据列暴露只 e2e 可证（单测只覆盖 `avroSchemaToColumns` 名/型/可见保真=绑定前提，非 live getTableAvroSchema）；②行级过滤正确性（straddling-base-file carry-forward）+ 兄弟委派跨加载器只 e2e 可证；③原生 `hoodie.datasource.read.*.instanttime` 窗口键是否收（INC-1 延后项）=INC-5/e2e 决策；④INC-5 翻闸删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`。 - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **✅ INC-4 DONE（`c692f4fe763`/`537afcb37f6`/`81b20559f45`+复审修 `48bc2b207cd`）** 中立 SPI（**收 `ConnectorMvccSnapshot` 非 `TableScanParams`**——设计原签名编译不过已纠正）+ fe-core 反向 `ConnectorExpression→Nereids` 转换器（~100 LOC 实测，非设计估的 150-250；TOTAL+fail-loud）+ CheckPolicy 中立臂（keys `PluginDrivenExternalTable`+scanParams，注入分析层）+ 可见元数据列（`getTableAvroSchema(true)`）〔原生 hoodie.* 窗口键决策=延后 INC-5/e2e〕→ **下一步 INC-5** 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - **✅ HD-C4/C5 schema-evolution recon + 设计 + 签字（recon `wf_85bd47a0-0aa` 6 读者+完整性/拆解 2 critic〔iron-rule 读者 StructuredOutput 上限失败，gap 由 critic+亲手核对补〕；权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`，起步必读）= 完整对齐老实现（用户签字 B，非窄退化；memory `hudi-schema-evolution-full-parity-signoff`）**。**BE 核实**：原生机器已就位、**零 BE 改**（`be/src/format_v2/table/hudi_reader.cpp` 已读 `hudi_params.schema_id`+`can_map_by_history_schema`+`annotate_file_schema_from_history`，与 paimon 同）。**架构裁决**：**模板=paimon 多条历史(-1+每引用版本)+每-split schema_id**（非 iceberg 单条；`THudiFileDesc` 有 f12 schema_id 而 `TIcebergFileDesc` 无=load-bearing divergence）；**小写逐层=iceberg**（format_v2 大小写不敏感但同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader→`StructNode.children.at()` raw name→std::out_of_range→整库 SIGABRT）；**current_schema_id 保持 -1**（真 id 撞某 split id→v1 `current==split`→ConstNode 大小写敏感逐字=禁的半吊子退化；paimon+legacy 均 -1，BE `find_external_root_field` 认 -1 条目为 target overlay）；**field-id 源=mode-aware commit-metadata InternalSchema**（stable across renames；非 naive `convert(latest avro)` 否则 -1 条目 id≠per-file id→BY_FIELD_ID 失配）；**每-split schema_id 只盖原生分支**（JNI/MOR-realtime 不看 schema_id、读 `column_names/types`@instant→C5b 全靠 FE 供）；**零新 SPI**（3-arg `getTableSchema`+hive 委派已发 `HiveConnectorMetadata:1070-1078`、`populateScanLevelParams` no-op、`getScanNodeProperties`、`ConnectorColumn.withUniqueId`、`THudiFileDesc` f12、`TFileScanRangeParams` 25/26 全在；100% 连接器内部）。**关键结构事实**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:973` vs `:979`）→句柄 field-id 是 latest-keyed→时间旅行改名列须 **C5b 按 pinned schema 重解 -1 条目**（iceberg full-pinned/empty-requested 式），非靠句柄；**C4c 硬序在 C4d 前**（字典present 而 per-split schema_id 仍 -1 = 禁的 current==file==-1）。**§6 签字决策（对齐 legacy 字节忠实）**：历史集=被选中文件涉及版本的并集(非全量枚举、自洽=每发出 split id 必在史、=legacy per-referenced-file、C4c/C4d 共享 resolver 背靠背落)；非演进表 FOR TIME=latest-fallback(schema 从未变=字节等价)；scalar `TColumnType`=STRING placeholder(BE 只用作 nested-vs-scalar 判别，不 port legacy 全 avro→Doris 型表)；field-id 留句柄(=legacy `updateHudiColumnUniqueId`)。**用户签字两部分拆分（2026-07-09）**：**Part 1=C4a–C4d + 完成后统一对抗复审；Part 2=C5a–C5b。用户下个 session 起 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器：`InternalSchema/Types.Field→TSchema/TField`+base64 encode/apply 往返，逐层小写、STRING placeholder、id+name+is_optional 每层，wired 进空，完全可单测，零 fe-core import）**。**e2e 硬门（切换期，非静默）**：改名/调序/加/删列 COW+MOR 读(native BY_FIELD_ID engage)、混合大小写嵌套结构不崩、演进表 FOR TIME(COW 列表+MOR JNI 列表@instant)、非演进 FOR TIME 等价、@incr latest schema、异构 HMS vs 独立 hudi 目录同表比行（memory `hms-iceberg-delegation-needs-e2e`）。 + - **✅ HD-C4a DONE（`d29bb52173d`，休眠，Part 1 首步）= 纯 `HudiSchemaUtils` 转换器**（新连接器内部类，`fe-connector-hudi`）。自足 `InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` 构建器 + base64 `encode`/`applySchemaEvolution` 往返 on 抛弃型 `TFileScanRangeParams`（f25 `current_schema_id` + f26 `history_schema_info`）。**移植 legacy `HudiUtils.getSchemaInfo(InternalSchema/List/Types.Field)` 两处有意偏离**（均由 mutation-tight 测钉）：①**逐层小写**（`Locale.ROOT`，镜像 `IcebergSchemaUtils.buildField`；legacy 不小写）——同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader，其 `StructNode` 按小写 Doris slot 名查，混大小写嵌套名→`std::out_of_range`→整库 SIGABRT；②**scalar `TColumnType`=STRING placeholder**（BE field-id 路只用 `type.type` 作 nested-vs-scalar 判别）——丢 legacy `fromAvroHudiTypeToDorisType` 全型表。`id`+`name`+`is_optional` 每层保真（legacy 平价）。`encode` serialize 错 fail-loud（catch `Exception|LinkageError` 防 thrift 类加载器裂）；`applySchemaEvolution` null/empty no-op、corrupt fail-loud（镜像 iceberg/paimon）。**模板=paimon（后续 C4c/C4d 复用 `buildSchemaInfo`+`encode` 发 -1 目标条 + 每引用版本历史 + 每-split schema_id），小写=iceberg 每层。wired 进空、零 fe-core import、零新连接器-api SPI。DORMANT**（hms 未进 `SPI_READY_TYPES`、hudi 句柄未分流）。`HudiSchemaUtilsTest` 11 全 same-loader（手搭混大小写嵌套 struct 子字段的 `InternalSchema` 断言每层小写/嵌套 struct·array·map 标签+每层 field-id/STRING scalar/InternalSchema-keyed 与显式 -1 schema id/base64 往返）。全绿（hudi 模块 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`）**。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From b60a9ff889e0e160d9bb5b64419d2f6aca294df5 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 19:08:31 +0800 Subject: [PATCH 128/334] [feat](catalog) HMS cutover hudi HD-C4b: dormant field-ids onto HudiColumnHandle from the mode-aware InternalSchema Second step of the hudi-on-HMS schema-evolution parity work. Sources each column's Hudi InternalSchema field id (the rename-safe join key BE uses for BY_FIELD_ID matching) and carries it on the column handle, so a later step's dictionary and the BE reader can match a renamed column on its old files by id instead of reading NULL. - HudiColumnHandle gains an int fieldId (ctor + getFieldId() + toString). Field id is deliberately NOT part of equals/hashCode: identity stays name+type (mirror IcebergColumnHandle, which keeps identity by name and does not fold the id in). - HudiConnectorMetadata.getSchemaFromMetaClient resolves the mode-aware InternalSchema for the LATEST commit and attaches each column's top-level field id. Mirror of legacy HiveMetaStoreClientHelper.getHudiTableSchema: schema.on.read on -> commit-metadata InternalSchema (STABLE ids across renames); off -> AvroInternalSchemaConverter.convert (latest avro) (positional ids). The no-arg getTableInternalSchemaFromCommitMetadata() pins the latest commit (steady-state / no time-travel pin; an at-instant variant is a later step). On any resolution failure the columns are returned unchanged (ids stay UNSET -> BE BY_NAME, the safe baseline) rather than dropping the whole schema. - getColumnHandles threads col.getUniqueId() onto the handle (mirror iceberg). Adaptation from legacy (documented): legacy sources columns AND ids from the same InternalSchema and zips positionally; the connector keeps its independent getTableAvroSchema(true) column source (preserving the shipped meta-column-inclusive schema), so ids are matched BY NAME in attachTopLevelFieldIds -- equivalent for the aligned column set and robust when the commit-metadata InternalSchema omits a column (e.g. the _hoodie_* meta fields): an unmatched column keeps UNSET_UNIQUE_ID -> BY_NAME (correct, meta columns are never renamed). Only the top-level id is threaded onto the handle; nested field ids for the schema dictionary come straight from the InternalSchema via HudiSchemaUtils (C4a). NOTE(C5a): this metaClient touch is not yet wrapped in HudiMetaClientExecutor.execute -- closing the existing unwrapped getSchemaFromMetaClient auth/TCCL gap is deferred to the at-instant step, which restructures this read. Zero fe-core import; zero new connector-api SPI. DORMANT (hudi is deliberately NOT in SPI_READY_TYPES; hudi handles not yet diverted -- no live catalog reaches getColumnHandles). The evolution-mode commit-metadata id source is flip-time e2e. Test HudiColumnFieldIdTest (4, fully same-loader): attachTopLevelFieldIds sources ids by (case-insensitive) name from convert(avro) InternalSchema; an unmatched column keeps UNSET; the handle carries the field id; the field id is not part of handle identity. Full hudi module 142/142 green; checkstyle 0; import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiColumnHandle.java | 20 ++- .../connector/hudi/HudiConnectorMetadata.java | 67 +++++++- .../connector/hudi/HudiColumnFieldIdTest.java | 144 ++++++++++++++++++ 3 files changed, 225 insertions(+), 6 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java index 6579aa2476ee56..4a3a54e2e4266a 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java @@ -22,8 +22,9 @@ import java.util.Objects; /** - * Column handle for Hudi tables, carrying column name, type, and - * whether the column is a partition key. + * Column handle for Hudi tables, carrying column name, type, + * whether the column is a partition key, and the Hudi InternalSchema + * field id (for schema-evolution BY_FIELD_ID matching; HD-C4b). */ public class HudiColumnHandle implements ConnectorColumnHandle { @@ -32,11 +33,18 @@ public class HudiColumnHandle implements ConnectorColumnHandle { private final String name; private final String typeName; private final boolean isPartitionKey; + // Hudi InternalSchema field id (stable across renames), sourced from the mode-aware InternalSchema and + // threaded here by HudiConnectorMetadata.getColumnHandles. ConnectorColumn.UNSET_UNIQUE_ID (-1) when no id + // was resolved (e.g. a _hoodie_* meta column absent from a commit-metadata schema -> BE falls back to + // BY_NAME). Deliberately NOT part of equals/hashCode: the handle's identity stays name+type (mirror + // IcebergColumnHandle, which keeps identity by name and does not fold the field id in). + private final int fieldId; - public HudiColumnHandle(String name, String typeName, boolean isPartitionKey) { + public HudiColumnHandle(String name, String typeName, boolean isPartitionKey, int fieldId) { this.name = Objects.requireNonNull(name); this.typeName = Objects.requireNonNull(typeName); this.isPartitionKey = isPartitionKey; + this.fieldId = fieldId; } public String getName() { @@ -51,6 +59,10 @@ public boolean isPartitionKey() { return isPartitionKey; } + public int getFieldId() { + return fieldId; + } + public String getColumnName() { return name; } @@ -74,7 +86,7 @@ public int hashCode() { @Override public String toString() { - return "HudiColumnHandle{" + name + ":" + typeName + return "HudiColumnHandle{" + name + ":" + typeName + "[" + fieldId + "]" + (isPartitionKey ? " [partition]" : "") + "}"; } } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 10a299bb3ceea8..6732e9d93a019f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -46,6 +46,10 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hudi.common.table.HoodieTableMetaClient; import org.apache.hudi.common.table.TableSchemaResolver; +import org.apache.hudi.common.util.Option; +import org.apache.hudi.internal.schema.InternalSchema; +import org.apache.hudi.internal.schema.Types; +import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -205,9 +209,11 @@ public Map getColumnHandles( for (ConnectorColumn col : schema.getColumns()) { boolean isPartKey = partKeyNames != null && partKeyNames.contains(col.getName()); + // Thread the Hudi InternalSchema field id (set by getSchemaFromMetaClient) onto the handle so + // schema-evolution reads match old files BY FIELD ID (HD-C4b). UNSET (-1) -> BE BY_NAME. handles.put(col.getName(), new HudiColumnHandle(col.getName(), - col.getType().getTypeName(), isPartKey)); + col.getType().getTypeName(), isPartKey, col.getUniqueId())); } return handles; } @@ -672,7 +678,8 @@ private List getSchemaFromMetaClient(String basePath) { // / DESCRIBE parity even for a populate.meta.fields=false table and (b) guarantees `_hoodie_commit_time` // is a visible, name-bindable output column for the synthetic incremental row filter to reference. Schema avroSchema = schemaResolver.getTableAvroSchema(true); - return avroSchemaToColumns(avroSchema); + List columns = avroSchemaToColumns(avroSchema); + return attachHudiFieldIds(schemaResolver, avroSchema, columns); } catch (Exception e) { LOG.warn("Failed to get schema from Hudi MetaClient for path '{}': {}", basePath, e.getMessage()); @@ -680,6 +687,62 @@ private List getSchemaFromMetaClient(String basePath) { } } + /** + * Resolve the mode-aware InternalSchema for the LATEST commit and attach each column's top-level field id + * (HD-C4b). Mirror of legacy {@code HiveMetaStoreClientHelper.getHudiTableSchema}: when {@code + * hoodie.schema.on.read.enable} is on, the ids come from the commit-metadata {@link InternalSchema} (STABLE + * across renames, so a renamed column keeps its id and BE matches its old files BY FIELD ID); otherwise from + * {@code AvroInternalSchemaConverter.convert(latest avro)} (positional ids). The no-arg {@code + * getTableInternalSchemaFromCommitMetadata()} pins the latest commit — steady-state / no time-travel pin (an + * at-instant variant is a later step). + * + *

    On ANY resolution failure the columns are returned unchanged (ids stay {@code UNSET_UNIQUE_ID} -> BE + * BY_NAME, the safe baseline) rather than dropping the whole schema — a schema-evolution id hiccup must not + * fail a plain read.

    + * + *

    Unlike legacy (which sources columns AND ids from the same InternalSchema and zips positionally), the + * connector keeps its independent {@code getTableAvroSchema(true)} column source (preserving the shipped + * meta-column-inclusive schema), so ids are matched BY NAME in {@link #attachTopLevelFieldIds}. NOTE(C5a): + * this metaClient touch is not yet wrapped in {@code HudiMetaClientExecutor.execute} — closing the existing + * unwrapped {@code getSchemaFromMetaClient} auth/TCCL gap is deferred to the at-instant step (C5a), which + * restructures this whole read.

    + */ + private List attachHudiFieldIds(TableSchemaResolver schemaResolver, Schema latestAvro, + List columns) { + try { + Option fromCommit = schemaResolver.getTableInternalSchemaFromCommitMetadata(); + InternalSchema internalSchema = fromCommit.isPresent() + ? fromCommit.get() + : AvroInternalSchemaConverter.convert(latestAvro); + return attachTopLevelFieldIds(columns, internalSchema); + } catch (Exception e) { + LOG.warn("Failed to resolve Hudi field ids; falling back to name-based (BY_NAME) matching: {}", + e.getMessage()); + return columns; + } + } + + /** + * Attach each column's top-level field id from {@code internalSchema}, matched by (lower-cased) name. Port of + * legacy {@code HudiUtils.updateHudiColumnUniqueId} at the top level only: the handle carries the top-level + * field id, while nested field ids for the BE schema dictionary come straight from the InternalSchema via + * {@link HudiSchemaUtils}. A column with no matching InternalSchema field (e.g. a {@code _hoodie_*} meta + * column absent from a commit-metadata schema) keeps {@link ConnectorColumn#UNSET_UNIQUE_ID} -> BE BY_NAME + * (correct: meta columns are never renamed). Package-private + static for same-loader unit testing. + */ + static List attachTopLevelFieldIds(List columns, InternalSchema internalSchema) { + Map idByName = new HashMap<>(); + for (Types.Field field : internalSchema.getRecord().fields()) { + idByName.put(field.name().toLowerCase(Locale.ROOT), field.fieldId()); + } + List result = new ArrayList<>(columns.size()); + for (ConnectorColumn col : columns) { + Integer id = idByName.get(col.getName().toLowerCase(Locale.ROOT)); + result.add(id != null ? col.withUniqueId(id) : col); + } + return result; + } + /** * Fallback: read schema from HMS if MetaClient fails. */ diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java new file mode 100644 index 00000000000000..d23f2395b88a0f --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java @@ -0,0 +1,144 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorColumn; + +import org.apache.avro.Schema; +import org.apache.hudi.internal.schema.InternalSchema; +import org.apache.hudi.internal.schema.Types; +import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Locale; +import java.util.Map; + +/** + * Same-loader unit tests for HD-C4b: sourcing Hudi InternalSchema field ids onto the columns + * ({@link HudiConnectorMetadata#attachTopLevelFieldIds}) and carrying them on {@link HudiColumnHandle}. The + * field id is the rename-safe join key BE uses for schema-evolution BY_FIELD_ID matching; without it a renamed + * column reads NULL on its old files. + * + *

    Covers the non-evolution {@code AvroInternalSchemaConverter.convert(avro)} id source (positional ids). The + * evolution-mode commit-metadata id source ({@code getTableInternalSchemaFromCommitMetadata}) needs a live + * metaClient with schema.on.read commit history and is exercised only by the flip-time docker e2e.

    + */ +public class HudiColumnFieldIdTest { + + // A representative Avro schema with mixed-case top-level names (Id/Name/Addr) exercising the case-insensitive + // name match, plus a nested struct (only the TOP-LEVEL id is threaded onto the handle; nested ids for the BE + // dictionary come straight from the InternalSchema via HudiSchemaUtils, not from here). + private static final String SCHEMA_JSON = + "{\"type\":\"record\",\"name\":\"hudi_t\",\"fields\":[" + + "{\"name\":\"Id\",\"type\":[\"null\",\"int\"],\"default\":null}," + + "{\"name\":\"Name\",\"type\":\"string\"}," + + "{\"name\":\"Addr\",\"type\":[\"null\",{\"type\":\"record\",\"name\":\"addr_t\"," + + "\"fields\":[{\"name\":\"Street\",\"type\":[\"null\",\"string\"],\"default\":null}]}]," + + "\"default\":null}" + + "]}"; + + private static Schema parse(String json) { + return new Schema.Parser().parse(json); + } + + /** Expected top-level id per (lower-cased) name, read back from the InternalSchema (not hard-coded). */ + private static Map expectedIds(InternalSchema internalSchema) { + Map ids = new HashMap<>(); + for (Types.Field field : internalSchema.getRecord().fields()) { + ids.put(field.name().toLowerCase(Locale.ROOT), field.fieldId()); + } + return ids; + } + + @Test + public void fieldIdsSourcedByNameFromInternalSchema() { + // getSchemaFromMetaClient builds columns from getTableAvroSchema(true) and sources ids from the mode-aware + // InternalSchema; for a non-evolution table that InternalSchema is convert(latest avro). Each column must + // carry the InternalSchema field id for its (lower-cased) name — the rename-safe BE join key. MUTATION: + // leave uniqueId UNSET / source a fabricated positional value -> the id != the InternalSchema id -> red. + Schema avro = parse(SCHEMA_JSON); + List columns = HudiConnectorMetadata.avroSchemaToColumns(avro); + InternalSchema internalSchema = AvroInternalSchemaConverter.convert(avro); + Map expected = expectedIds(internalSchema); + + List attached = HudiConnectorMetadata.attachTopLevelFieldIds(columns, internalSchema); + + Assertions.assertEquals(3, attached.size()); + for (ConnectorColumn col : attached) { + Integer want = expected.get(col.getName()); + Assertions.assertNotNull(want, "no InternalSchema field for column " + col.getName()); + Assertions.assertEquals(want.intValue(), col.getUniqueId(), + "field id mismatch for column " + col.getName()); + } + // Mixed-case avro name "Id" is matched case-insensitively to the lower-cased column "id" (the byte name BE + // keys by). MUTATION: drop the toLowerCase on either side -> the mixed-case name misses -> UNSET -> red. + Map attachedById = new HashMap<>(); + for (ConnectorColumn col : attached) { + attachedById.put(col.getName(), col.getUniqueId()); + } + Assertions.assertTrue(attachedById.containsKey("id")); + Assertions.assertTrue(attachedById.containsKey("name")); + Assertions.assertTrue(attachedById.containsKey("addr")); + Assertions.assertNotEquals(ConnectorColumn.UNSET_UNIQUE_ID, attachedById.get("addr").intValue()); + } + + @Test + public void unmatchedColumnKeepsUnsetId() { + // A column with no matching InternalSchema field (e.g. a _hoodie_* meta column absent from a + // commit-metadata schema) must keep UNSET_UNIQUE_ID so BE falls back to BY_NAME — never a wrong id. + // Here the columns include an extra "_hoodie_commit_time" that the (data-only) InternalSchema lacks. + // MUTATION: default a matched-but-missing column to 0 / to a neighbour's id -> not UNSET -> red. + Schema dataAvro = parse(SCHEMA_JSON); + List columns = new ArrayList<>( + HudiConnectorMetadata.avroSchemaToColumns(dataAvro)); + // An extra column not present in the (data-only) InternalSchema, reusing an existing column's type. + columns.add(new ConnectorColumn("_hoodie_commit_time", columns.get(1).getType(), "", true, null)); + InternalSchema internalSchema = AvroInternalSchemaConverter.convert(dataAvro); + + List attached = HudiConnectorMetadata.attachTopLevelFieldIds(columns, internalSchema); + + ConnectorColumn attachedExtra = attached.get(attached.size() - 1); + Assertions.assertEquals("_hoodie_commit_time", attachedExtra.getName()); + Assertions.assertEquals(ConnectorColumn.UNSET_UNIQUE_ID, attachedExtra.getUniqueId()); + // the real data columns are still resolved + Assertions.assertNotEquals(ConnectorColumn.UNSET_UNIQUE_ID, attached.get(0).getUniqueId()); + } + + @Test + public void handleCarriesFieldId() { + // getColumnHandles threads col.getUniqueId() onto the handle. MUTATION: hard-code UNSET / drop the arg + // -> getFieldId() != the sourced id -> red. + HudiColumnHandle handle = new HudiColumnHandle("c", "int", false, 7); + Assertions.assertEquals(7, handle.getFieldId()); + } + + @Test + public void fieldIdIsNotPartOfHandleIdentity() { + // The field id must NOT enter equals/hashCode (mirror IcebergColumnHandle: identity by name, not id). + // Otherwise a plan that keys handles by identity would treat the same column with a differently-resolved + // id as two columns. MUTATION: fold fieldId into equals/hashCode -> these two become unequal -> red. + HudiColumnHandle a = new HudiColumnHandle("c", "int", false, 7); + HudiColumnHandle b = new HudiColumnHandle("c", "int", false, ConnectorColumn.UNSET_UNIQUE_ID); + Assertions.assertEquals(a, b); + Assertions.assertEquals(a.hashCode(), b.hashCode()); + } +} From fb832e29f073aa0ccdd1ef51a64b045ed8063206 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 19:09:31 +0800 Subject: [PATCH 129/334] [doc](catalog) HMS cutover hudi HD-C4b DONE; next = C4c (per-split THudiFileDesc.schema_id) Per-round HANDOFF update: mark HD-C4b (field-ids onto HudiColumnHandle from the mode-aware InternalSchema, dormant commit d54dc2d3ac5) done in the current-status line and the hudi section; next step = C4c (per-split THudiFileDesc.schema_id, native branch only). Records the by-name id matching adaptation (vs legacy positional), the identity rule (fieldId not in equals/hashCode), the fail-safe-to-BY_NAME degrade, the C5a-owed HudiMetaClientExecutor wrapping, and the honest dormancy (hudi deliberately excluded from SPI_READY_TYPES). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 49dfb2347fdba5..890c4ae609f3b4 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`，mode-aware InternalSchema 取 id，`equals/hashCode` 保持按名不加 id）→ C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）→ C4d（扫描级 `hudi.schema_evolution` 字典发射+`populateScanLevelParams` apply，C4c/C4d 背靠背落〔D1 共享 resolver + prefix-flip 安全〕）→ 完成后统一对抗复审 → Part 2 C5a/C5b。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）→ C4d（扫描级 `hudi.schema_evolution` 字典发射+`populateScanLevelParams` apply，C4c/C4d 背靠背落〔D1 共享 resolver + prefix-flip 安全〕）→ 完成后统一对抗复审 → Part 2 C5a/C5b。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -96,6 +96,7 @@ - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **✅ INC-4 DONE（`c692f4fe763`/`537afcb37f6`/`81b20559f45`+复审修 `48bc2b207cd`）** 中立 SPI（**收 `ConnectorMvccSnapshot` 非 `TableScanParams`**——设计原签名编译不过已纠正）+ fe-core 反向 `ConnectorExpression→Nereids` 转换器（~100 LOC 实测，非设计估的 150-250；TOTAL+fail-loud）+ CheckPolicy 中立臂（keys `PluginDrivenExternalTable`+scanParams，注入分析层）+ 可见元数据列（`getTableAvroSchema(true)`）〔原生 hoodie.* 窗口键决策=延后 INC-5/e2e〕→ **下一步 INC-5** 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - **✅ HD-C4/C5 schema-evolution recon + 设计 + 签字（recon `wf_85bd47a0-0aa` 6 读者+完整性/拆解 2 critic〔iron-rule 读者 StructuredOutput 上限失败，gap 由 critic+亲手核对补〕；权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`，起步必读）= 完整对齐老实现（用户签字 B，非窄退化；memory `hudi-schema-evolution-full-parity-signoff`）**。**BE 核实**：原生机器已就位、**零 BE 改**（`be/src/format_v2/table/hudi_reader.cpp` 已读 `hudi_params.schema_id`+`can_map_by_history_schema`+`annotate_file_schema_from_history`，与 paimon 同）。**架构裁决**：**模板=paimon 多条历史(-1+每引用版本)+每-split schema_id**（非 iceberg 单条；`THudiFileDesc` 有 f12 schema_id 而 `TIcebergFileDesc` 无=load-bearing divergence）；**小写逐层=iceberg**（format_v2 大小写不敏感但同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader→`StructNode.children.at()` raw name→std::out_of_range→整库 SIGABRT）；**current_schema_id 保持 -1**（真 id 撞某 split id→v1 `current==split`→ConstNode 大小写敏感逐字=禁的半吊子退化；paimon+legacy 均 -1，BE `find_external_root_field` 认 -1 条目为 target overlay）；**field-id 源=mode-aware commit-metadata InternalSchema**（stable across renames；非 naive `convert(latest avro)` 否则 -1 条目 id≠per-file id→BY_FIELD_ID 失配）；**每-split schema_id 只盖原生分支**（JNI/MOR-realtime 不看 schema_id、读 `column_names/types`@instant→C5b 全靠 FE 供）；**零新 SPI**（3-arg `getTableSchema`+hive 委派已发 `HiveConnectorMetadata:1070-1078`、`populateScanLevelParams` no-op、`getScanNodeProperties`、`ConnectorColumn.withUniqueId`、`THudiFileDesc` f12、`TFileScanRangeParams` 25/26 全在；100% 连接器内部）。**关键结构事实**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:973` vs `:979`）→句柄 field-id 是 latest-keyed→时间旅行改名列须 **C5b 按 pinned schema 重解 -1 条目**（iceberg full-pinned/empty-requested 式），非靠句柄；**C4c 硬序在 C4d 前**（字典present 而 per-split schema_id 仍 -1 = 禁的 current==file==-1）。**§6 签字决策（对齐 legacy 字节忠实）**：历史集=被选中文件涉及版本的并集(非全量枚举、自洽=每发出 split id 必在史、=legacy per-referenced-file、C4c/C4d 共享 resolver 背靠背落)；非演进表 FOR TIME=latest-fallback(schema 从未变=字节等价)；scalar `TColumnType`=STRING placeholder(BE 只用作 nested-vs-scalar 判别，不 port legacy 全 avro→Doris 型表)；field-id 留句柄(=legacy `updateHudiColumnUniqueId`)。**用户签字两部分拆分（2026-07-09）**：**Part 1=C4a–C4d + 完成后统一对抗复审；Part 2=C5a–C5b。用户下个 session 起 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器：`InternalSchema/Types.Field→TSchema/TField`+base64 encode/apply 往返，逐层小写、STRING placeholder、id+name+is_optional 每层，wired 进空，完全可单测，零 fe-core import）**。**e2e 硬门（切换期，非静默）**：改名/调序/加/删列 COW+MOR 读(native BY_FIELD_ID engage)、混合大小写嵌套结构不崩、演进表 FOR TIME(COW 列表+MOR JNI 列表@instant)、非演进 FOR TIME 等价、@incr latest schema、异构 HMS vs 独立 hudi 目录同表比行（memory `hms-iceberg-delegation-needs-e2e`）。 - **✅ HD-C4a DONE（`d29bb52173d`，休眠，Part 1 首步）= 纯 `HudiSchemaUtils` 转换器**（新连接器内部类，`fe-connector-hudi`）。自足 `InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` 构建器 + base64 `encode`/`applySchemaEvolution` 往返 on 抛弃型 `TFileScanRangeParams`（f25 `current_schema_id` + f26 `history_schema_info`）。**移植 legacy `HudiUtils.getSchemaInfo(InternalSchema/List/Types.Field)` 两处有意偏离**（均由 mutation-tight 测钉）：①**逐层小写**（`Locale.ROOT`，镜像 `IcebergSchemaUtils.buildField`；legacy 不小写）——同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader，其 `StructNode` 按小写 Doris slot 名查，混大小写嵌套名→`std::out_of_range`→整库 SIGABRT；②**scalar `TColumnType`=STRING placeholder**（BE field-id 路只用 `type.type` 作 nested-vs-scalar 判别）——丢 legacy `fromAvroHudiTypeToDorisType` 全型表。`id`+`name`+`is_optional` 每层保真（legacy 平价）。`encode` serialize 错 fail-loud（catch `Exception|LinkageError` 防 thrift 类加载器裂）；`applySchemaEvolution` null/empty no-op、corrupt fail-loud（镜像 iceberg/paimon）。**模板=paimon（后续 C4c/C4d 复用 `buildSchemaInfo`+`encode` 发 -1 目标条 + 每引用版本历史 + 每-split schema_id），小写=iceberg 每层。wired 进空、零 fe-core import、零新连接器-api SPI。DORMANT**（hms 未进 `SPI_READY_TYPES`、hudi 句柄未分流）。`HudiSchemaUtilsTest` 11 全 same-loader（手搭混大小写嵌套 struct 子字段的 `InternalSchema` 断言每层小写/嵌套 struct·array·map 标签+每层 field-id/STRING scalar/InternalSchema-keyed 与显式 -1 schema id/base64 往返）。全绿（hudi 模块 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`）**。 + - **✅ HD-C4b DONE（`d54dc2d3ac5`，休眠，Part 1 第二步）= field-id 上 `HudiColumnHandle`**（源自 mode-aware InternalSchema）。①`HudiColumnHandle` 加 `int fieldId`（ctor+`getFieldId()`+toString）——**不入 equals/hashCode**（身份保 name+typeName，镜像 `IcebergColumnHandle` 按名不折 id）。②`HudiConnectorMetadata.getSchemaFromMetaClient` 解析 mode-aware InternalSchema：no-arg `TableSchemaResolver.getTableInternalSchemaFromCommitMetadata()` 钉**最新 commit**（schema.on.read 开→commit-metadata InternalSchema〔改名跨版本**稳定 id**〕；关→`AvroInternalSchemaConverter.convert(latest avro)` 位置 id）——镜像 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`（steady-state/无时间旅行 pin，at-instant 变体留 C5a）；解析失败降级 columns 不带 id（UNSET→BE BY_NAME）而非丢整 schema。③`getColumnHandles` 穿 `col.getUniqueId()` 上 handle（镜像 iceberg）。**与 legacy 有意偏离（已文档化）**：legacy 列源+id 源同一 InternalSchema 位置 zip；连接器保 `getTableAvroSchema(true)` 独立列源（保住已发的含 meta 列 schema），故 `attachTopLevelFieldIds` **按名匹配**——对齐列集等价、且当 commit-metadata InternalSchema 缺某列（如 `_hoodie_*` meta）时该列留 UNSET→BY_NAME（正确，meta 列从不改名）。只顶层 id 上 handle；嵌套 field-id 由 InternalSchema 经 `HudiSchemaUtils` 直发（C4a）。**⚠残留**：`NOTE(C5a)` 此 metaClient touch 尚未 `HudiMetaClientExecutor.execute` 包裹——closing 既有 `getSchemaFromMetaClient` 未包裹 auth/TCCL gap 留 C5a（它重构此读）；evolution-mode commit-metadata id 源仅 e2e 可证（单测覆盖非演进 `convert(avro)` 路）。DORMANT（hudi 故意不在 `SPI_READY_TYPES`、句柄未分流→无 live 目录达 `getColumnHandles`）。`HudiColumnFieldIdTest` 4 same-loader（按名取 id/未命中 UNSET/handle 载 id/id 不入身份）。全绿（hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）**。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From 4d81aa9db1440ac903e5df5747a8697d91defcb9 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 19:44:32 +0800 Subject: [PATCH 130/334] [feat](catalog) HMS cutover hudi HD-C4c: dormant per-split THudiFileDesc.schema_id (native branch only) Third step of the hudi-on-HMS schema-evolution parity work. Stamps each native base file with the InternalSchema version of the commit that wrote it (THudiFileDesc.schema_id, field 12) so BE's native field-id reader can look up that file's columns in the schema dictionary and match them to the current table by id (rename-safe). NATIVE branch only: the JNI merge reader reads column_names/types @instant and consumes no schema_id. - HudiSchemaUtils: two shared static resolvers. resolveTableInternalSchema = the mode-aware table InternalSchema for the latest commit (schema.on.read on -> commit metadata, stable ids; off -> convert(latest avro), version 0) + evolution flag; the field-id step (HudiConnectorMetadata) is refactored to reuse it (single source of truth for the id source). resolveFileInternalSchema = the per-file version: evolution on -> FSUtils.getCommitTime(fileName) -> InternalSchemaCache.searchSchemaAndCache; off -> the base schema (version 0). Port of legacy HudiScanNode.setHudiParams. - HudiScanRange.Builder gains a nullable schemaId; the ctor carries it as the hudi.schema_id prop only when set; populateRangeParams stamps fileDesc.setSchemaId ONLY in the native else-branch (mirror PaimonScanRange), NEVER the JNI branch. Absent -> BE BY_NAME. - HudiScanPlanProvider.planScan resolves the base InternalSchema once and builds a per-file schema_id resolver (base-file path -> version); collectCowSplits (COW base files, always native) and the native no-log MOR slice in buildMorRange stamp it. The @incr path passes a null resolver (@incr lists the latest schema; no per-file dict). Base-schema-unresolved -> null resolver -> no stamp; a per-file resolution failure logs and skips that file -> both degrade to BY_NAME rather than failing the scan. Ordering: schema_id present WITHOUT the dictionary (still to come) means BE does not engage the field-id path -> BY_NAME (the safe baseline). This is the safe prefix; the dict step (current_schema_id=-1 + history_schema_info) MUST land after this one (a dict-present + schema_id-unset intermediate would be the forbidden current==file==-1 v1-ConstNode degrade). Zero fe-core import; zero new connector-api SPI. DORMANT (hudi is deliberately NOT in SPI_READY_TYPES; handles not diverted). The schema_id VALUE is flip-time e2e; only the stamping mechanism (native sets field 12, JNI never) is same-loader tested. Tests: HudiScanRangeTest +3 (native slice stamps field 12; JNI slice never stamps even with a schema_id set; native slice without a schema_id leaves it unset); HudiIncrementalPlanScanTest buildMorRange callers assert native downgrade carries the resolver's id while a force-jni slice does not. Full hudi module 145/145 green; checkstyle 0; import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiConnectorMetadata.java | 8 +-- .../connector/hudi/HudiScanPlanProvider.java | 60 +++++++++++++++---- .../doris/connector/hudi/HudiScanRange.java | 20 +++++++ .../doris/connector/hudi/HudiSchemaUtils.java | 57 ++++++++++++++++++ .../hudi/HudiIncrementalPlanScanTest.java | 8 ++- .../connector/hudi/HudiScanRangeTest.java | 60 +++++++++++++++++++ 6 files changed, 195 insertions(+), 18 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 6732e9d93a019f..03d29580ad8aea 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -46,10 +46,8 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hudi.common.table.HoodieTableMetaClient; import org.apache.hudi.common.table.TableSchemaResolver; -import org.apache.hudi.common.util.Option; import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.internal.schema.Types; -import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -710,10 +708,8 @@ private List getSchemaFromMetaClient(String basePath) { private List attachHudiFieldIds(TableSchemaResolver schemaResolver, Schema latestAvro, List columns) { try { - Option fromCommit = schemaResolver.getTableInternalSchemaFromCommitMetadata(); - InternalSchema internalSchema = fromCommit.isPresent() - ? fromCommit.get() - : AvroInternalSchemaConverter.convert(latestAvro); + InternalSchema internalSchema = + HudiSchemaUtils.resolveTableInternalSchema(schemaResolver, latestAvro).internalSchema; return attachTopLevelFieldIds(columns, internalSchema); } catch (Exception e) { LOG.warn("Failed to resolve Hudi field ids; falling back to name-based (BY_NAME) matching: {}", diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index a81d7a712e6d5a..50d432c34c4737 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -57,6 +57,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.function.Function; import java.util.stream.Collectors; /** @@ -153,6 +154,10 @@ public List planScan( // Resolve column names and types for JNI reader List columnNames; List columnTypes; + // The mode-aware table InternalSchema (+ evolution flag), resolved ONCE, drives the per-file + // THudiFileDesc.schema_id stamped on native slices for BE's field-id path. Null when unresolved -> + // no schema_id -> BE BY_NAME (the safe baseline). schema_id is dormant/inert until the dict is emitted. + HudiSchemaUtils.ResolvedInternalSchema resolvedSchema = null; try { TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); // include the 5 `_hoodie_*` meta columns (explicit `true` = legacy parity, in lockstep with @@ -164,6 +169,7 @@ public List planScan( columnTypes = avroSchema.getFields().stream() .map(f -> HudiTypeMapping.toHiveTypeString(f.schema())) .collect(Collectors.toList()); + resolvedSchema = HudiSchemaUtils.resolveTableInternalSchema(schemaResolver, avroSchema); } catch (Exception e) { LOG.warn("Failed to resolve Hudi schema for JNI reader, JNI splits may fail: {}", e.getMessage()); @@ -171,6 +177,21 @@ public List planScan( columnTypes = Collections.emptyList(); } + // Per-file native-reader schema_id resolver (base-file path -> version), or null to skip stamping + // (base schema unresolved -> BY_NAME). A per-file resolution failure logs and returns null for that file + // (BY_NAME) rather than failing the whole scan. Runs on this TCCL-pinned scan thread. + final HudiSchemaUtils.ResolvedInternalSchema baseSchema = resolvedSchema; + Function schemaIdResolver = baseSchema == null ? null + : filePath -> { + try { + return HudiSchemaUtils.resolveFileInternalSchema(filePath, + baseSchema.enableSchemaEvolution, baseSchema.internalSchema, metaClient).schemaId(); + } catch (Exception e) { + LOG.warn("Failed to resolve Hudi per-file schema_id for {}: {}", filePath, e.getMessage()); + return null; + } + }; + String inputFormat = hudiHandle.getInputFormat(); String serdeLib = hudiHandle.getSerdeLib(); @@ -217,11 +238,11 @@ public List planScan( if (useNativeCowPath) { collectCowSplits(fsView, partitionPath, queryInstant, - basePath, partValues, ranges); + basePath, partValues, ranges, schemaIdResolver); } else { collectMorSplits(fsView, partitionPath, queryInstant, basePath, inputFormat, serdeLib, - columnNames, columnTypes, partValues, forceJni, ranges); + columnNames, columnTypes, partValues, forceJni, ranges, schemaIdResolver); } } @@ -286,21 +307,27 @@ private void collectCowSplits( String partitionPath, String queryInstant, String basePath, Map partValues, - List ranges) { + List ranges, + Function schemaIdResolver) { fsView.getLatestBaseFilesBeforeOrOn(partitionPath, queryInstant) .forEach(baseFile -> { String filePath = baseFile.getPath(); long fileSize = baseFile.getFileSize(); String format = detectFileFormat(filePath); - ranges.add(new HudiScanRange.Builder() + HudiScanRange.Builder builder = new HudiScanRange.Builder() .path(filePath) .start(0) .length(fileSize) .fileSize(fileSize) .fileFormat(format) - .partitionValues(partValues) - .build()); + .partitionValues(partValues); + // COW base files always read native -> stamp the per-file schema version for BE's field-id path. + Long schemaId = schemaIdResolver == null ? null : schemaIdResolver.apply(filePath); + if (schemaId != null) { + builder.schemaId(schemaId); + } + ranges.add(builder.build()); }); } @@ -315,10 +342,11 @@ private void collectMorSplits( String basePath, String inputFormat, String serdeLib, List columnNames, List columnTypes, Map partValues, boolean forceJni, - List ranges) { + List ranges, + Function schemaIdResolver) { fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, queryInstant) .forEach(fileSlice -> ranges.add(buildMorRange(fileSlice, partValues, queryInstant, - forceJni, basePath, inputFormat, serdeLib, columnNames, columnTypes))); + forceJni, basePath, inputFormat, serdeLib, columnNames, columnTypes, schemaIdResolver))); } /** @@ -330,10 +358,15 @@ private void collectMorSplits( * path passes its {@code queryInstant}, the incremental path passes the resolved window END * ({@code relation.getEndTs()}). Package-private static so the mapping is unit-testable with a hand-built * {@link FileSlice} and reused by both paths without duplication. + * + *

    {@code schemaIdResolver} (base-file path -> native schema version) is applied ONLY to a native + * (no-log, non-force-jni) slice — the JNI merge reader consumes no schema_id. {@code null} skips stamping + * (the {@code @incr} path passes null: {@code @incr} lists the latest schema, no per-file dict).

    */ static HudiScanRange buildMorRange(FileSlice fileSlice, Map partValues, String jniInstant, boolean forceJni, String basePath, String inputFormat, String serdeLib, - List columnNames, List columnTypes) { + List columnNames, List columnTypes, + Function schemaIdResolver) { Optional baseFileOpt = fileSlice.getBaseFile().toJavaOptional(); String filePath = baseFileOpt.map(BaseFile::getPath).orElse(""); long fileSize = baseFileOpt.map(BaseFile::getFileSize).orElse(0L); @@ -373,6 +406,12 @@ static HudiScanRange buildMorRange(FileSlice fileSlice, Map part .deltaLogs(logs) .columnNames(columnNames) .columnTypes(columnTypes); + } else if (schemaIdResolver != null) { + // Native no-log slice reads via the field-id native reader -> stamp its base file's schema version. + Long schemaId = schemaIdResolver.apply(filePath); + if (schemaId != null) { + builder.schemaId(schemaId); + } } return builder.build(); @@ -440,8 +479,9 @@ static Optional> incrementalRanges(IncrementalRelation String endTs = relation.getEndTs(); for (FileSlice fileSlice : relation.collectFileSlices()) { Map partValues = parsePartitionValues(fileSlice.getPartitionPath(), partitionFieldNames); + // @incr lists the LATEST schema (no per-file schema_id dict on the incremental path) -> null resolver. ranges.add(buildMorRange(fileSlice, partValues, endTs, forceJni, - basePath, inputFormat, serdeLib, columnNames, columnTypes)); + basePath, inputFormat, serdeLib, columnNames, columnTypes, null)); } return Optional.of(ranges); } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java index c9d644ff7b3b31..c18c2d866ae5ad 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanRange.java @@ -99,6 +99,12 @@ private HudiScanRange(Builder builder) { props.put("hudi.data_file_path", builder.dataFilePath); } props.put("hudi.data_file_length", String.valueOf(builder.dataFileLength)); + // Per-split native-reader schema version (mirror paimon.schema_id). Only carried when the provider + // resolved one for a native slice; populateRangeParams stamps THudiFileDesc.schema_id (field 12) from it + // ONLY on the native branch (never JNI). Absent -> BE BY_NAME. + if (builder.schemaId != null) { + props.put("hudi.schema_id", String.valueOf(builder.schemaId)); + } this.properties = Collections.unmodifiableMap(props); this.deltaLogs = builder.deltaLogs != null @@ -218,6 +224,13 @@ public void populateRangeParams(TTableFormatFileDesc formatDesc, } } else { rangeDesc.setFormatType(nativeFormatType(props)); + // Native field-id path only (paimon parity): the per-split schema version the native reader matches + // the base file's columns against. The JNI reader consumes no schema_id (it reads column_names/types + // @instant), so this is NEVER set on the JNI branch. Absent -> BE BY_NAME (no evolution). + String schemaId = props.get("hudi.schema_id"); + if (schemaId != null) { + fileDesc.setSchemaId(Long.parseLong(schemaId)); + } } formatDesc.setHudiParams(fileDesc); @@ -277,6 +290,8 @@ public static class Builder { private List columnNames; private List columnTypes; private boolean forceJni; + // Native-reader per-split schema version (nullable = not stamped; JNI slices never carry one). + private Long schemaId; public Builder path(String path) { this.path = path; @@ -358,6 +373,11 @@ public Builder forceJni(boolean forceJni) { return this; } + public Builder schemaId(long schemaId) { + this.schemaId = schemaId; + return this; + } + public HudiScanRange build() { return new HudiScanRange(this); } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java index 10ac57ca2508f2..84ab65f2296c1b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java @@ -28,12 +28,20 @@ import org.apache.doris.thrift.schema.external.TSchema; import org.apache.doris.thrift.schema.external.TStructField; +import org.apache.avro.Schema; +import org.apache.hudi.common.fs.FSUtils; +import org.apache.hudi.common.table.HoodieTableMetaClient; +import org.apache.hudi.common.table.TableSchemaResolver; +import org.apache.hudi.common.util.InternalSchemaCache; +import org.apache.hudi.common.util.Option; import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.internal.schema.Types; +import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; import org.apache.thrift.TDeserializer; import org.apache.thrift.TSerializer; import org.apache.thrift.protocol.TBinaryProtocol; +import java.io.File; import java.util.Base64; import java.util.List; import java.util.Locale; @@ -212,4 +220,53 @@ private static TFieldPtr fieldPtr(TField field) { ptr.setFieldPtr(field); return ptr; } + + // ========== mode-aware InternalSchema resolvers (shared by field-id / schema-id / dict steps) ========== + + /** The mode-aware table {@link InternalSchema} for the latest commit + whether schema-on-read evolution is on. */ + static final class ResolvedInternalSchema { + final InternalSchema internalSchema; + final boolean enableSchemaEvolution; + + ResolvedInternalSchema(InternalSchema internalSchema, boolean enableSchemaEvolution) { + this.internalSchema = internalSchema; + this.enableSchemaEvolution = enableSchemaEvolution; + } + } + + /** + * Resolve the mode-aware table {@link InternalSchema} for the LATEST commit. Mirror of legacy + * {@code HiveMetaStoreClientHelper.getHudiTableSchema}: when {@code hoodie.schema.on.read.enable} is on, the + * ids come from the commit-metadata {@link InternalSchema} (STABLE across renames) and evolution is flagged + * true; otherwise from {@code AvroInternalSchemaConverter.convert(latestAvro)} (positional ids, version 0) + * with evolution false. The no-arg {@code getTableInternalSchemaFromCommitMetadata()} pins the latest commit + * (steady-state / no time-travel pin). Shared by the field-id ({@code HudiConnectorMetadata}) and the + * per-file schema-id / dict scan paths so both agree on the id source. + */ + static ResolvedInternalSchema resolveTableInternalSchema(TableSchemaResolver schemaResolver, Schema latestAvro) { + Option fromCommit = schemaResolver.getTableInternalSchemaFromCommitMetadata(); + if (fromCommit.isPresent()) { + return new ResolvedInternalSchema(fromCommit.get(), true); + } + return new ResolvedInternalSchema(AvroInternalSchemaConverter.convert(latestAvro), false); + } + + /** + * Resolve the {@link InternalSchema} of the commit that WROTE a given base file — its {@code schemaId()} is + * the per-split {@code THudiFileDesc.schema_id} BE stamps native files with on the field-id path. Port of + * legacy {@code HudiScanNode.setHudiParams}: evolution on → {@code FSUtils.getCommitTime(fileName)} + * → {@code InternalSchemaCache.searchSchemaAndCache} (the real committed versionId); off → the base + * (non-evolution) InternalSchema ({@code convert(latestAvro)}, version {@code 0}). The base schema is passed + * in (resolved once per scan via {@link #resolveTableInternalSchema}) so a non-evolution scan pays no + * per-file metaClient touch. Runs on the TCCL-pinned scan thread; shared with the dict step (which reuses the + * returned InternalSchema to build the history entry for that version). + */ + static InternalSchema resolveFileInternalSchema(String filePath, boolean enableSchemaEvolution, + InternalSchema baseInternalSchema, HoodieTableMetaClient metaClient) { + if (!enableSchemaEvolution) { + return baseInternalSchema; + } + long commitTime = Long.parseLong(FSUtils.getCommitTime(new File(filePath).getName())); + return InternalSchemaCache.searchSchemaAndCache(commitTime, metaClient); + } } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java index 40e04bb93a96de..11dc67a1d8dc15 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java @@ -160,11 +160,13 @@ public void buildMorRangeStampsJniInstantAndMetadataForForceJniSlice() { Map partValues = HudiScanPlanProvider.parsePartitionValues( slice.getPartitionPath(), YEAR_MONTH); HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, partValues, END_TS, /*forceJni*/ true, - BASE_PATH, INPUT_FORMAT, SERDE, Arrays.asList("c1"), Arrays.asList("int")); + BASE_PATH, INPUT_FORMAT, SERDE, Arrays.asList("c1"), Arrays.asList("int"), p -> 7L); Assertions.assertEquals("jni", range.getFileFormat()); Assertions.assertEquals(END_TS, range.getProperties().get("hudi.instant_time")); Assertions.assertEquals(SERDE, range.getProperties().get("hudi.serde")); Assertions.assertEquals(BASE_PATH, range.getProperties().get("hudi.base_path")); + // C4c: a JNI slice NEVER carries schema_id even when the resolver returns one (native field-id path only). + Assertions.assertNull(range.getProperties().get("hudi.schema_id")); } @Test @@ -175,9 +177,11 @@ public void buildMorRangeDowngradesNoLogSliceToNativeWithoutForceJni() { "s3://b/t/year=2024/month=01/fileid-1_0_20240101000000.parquet"); HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, Collections.emptyMap(), END_TS, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, - Arrays.asList("c1"), Arrays.asList("int")); + Arrays.asList("c1"), Arrays.asList("int"), p -> 7L); Assertions.assertEquals("parquet", range.getFileFormat(), "a no-log slice without force_jni must downgrade to the native parquet reader"); + // C4c: a native downgraded slice carries the per-file schema_id from the resolver (native field-id path). + Assertions.assertEquals("7", range.getProperties().get("hudi.schema_id")); } // ── helpers ──────────────────────────────────────────────────────────────────────────────────────────── diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java index 9a4628233a8af0..30e359f1220ae4 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiScanRangeTest.java @@ -133,4 +133,64 @@ public void nativeOrcSliceExplicitlySetsFormatType() { Assertions.assertEquals(TFileFormatType.FORMAT_ORC, rangeDesc.getFormatType(), "a native orc slice must set FORMAT_ORC, not inherit the parquet node default"); } + + @Test + public void nativeSliceStampsSchemaId() { + // C4c: a native slice carrying a schema_id must stamp THudiFileDesc.schema_id (field 12) so BE's native + // field-id reader can match old files across schema evolution. MUTATION: skip the native-branch stamp -> + // isSetSchemaId false -> red. + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/base.parquet") + .fileFormat("parquet") + .fileSize(456L) + .schemaId(20240101000000000L) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertTrue(formatDesc.getHudiParams().isSetSchemaId()); + Assertions.assertEquals(20240101000000000L, formatDesc.getHudiParams().getSchemaId()); + } + + @Test + public void jniSliceNeverStampsSchemaId() { + // C4c: schema_id is a NATIVE-only field (the JNI merge reader reads column_names/types @instant and + // consumes no schema_id). Even if a schema_id is set on the Builder, a JNI (log-bearing) slice must NOT + // stamp it. MUTATION: stamp schema_id in the JNI branch -> isSetSchemaId true -> red. + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/file") + .fileFormat("jni") + .instantTime("20240101000000000") + .basePath("s3://bucket/t") + .dataFilePath("s3://bucket/t/base.parquet") + .deltaLogs(Arrays.asList("s3://bucket/t/.f.log.1_0")) + .schemaId(20240101000000000L) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertEquals(TFileFormatType.FORMAT_JNI, rangeDesc.getFormatType()); + Assertions.assertFalse(formatDesc.getHudiParams().isSetSchemaId()); + } + + @Test + public void nativeSliceWithoutSchemaIdLeavesItUnset() { + // A native slice with no resolved schema_id (non-evolution unresolved / BY_NAME baseline) must not stamp + // field 12. MUTATION: default schema_id to 0/-1 unconditionally -> isSetSchemaId true -> red. + HudiScanRange range = new HudiScanRange.Builder() + .path("s3://bucket/t/base.parquet") + .fileFormat("parquet") + .fileSize(456L) + .build(); + + TTableFormatFileDesc formatDesc = new TTableFormatFileDesc(); + TFileRangeDesc rangeDesc = new TFileRangeDesc(); + range.populateRangeParams(formatDesc, rangeDesc); + + Assertions.assertFalse(formatDesc.getHudiParams().isSetSchemaId()); + } } From 098f4ce7a7d41bea10b9a103c8d886e0fc5ab1f6 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 19:46:06 +0800 Subject: [PATCH 131/334] [doc](catalog) HMS cutover hudi HD-C4c DONE; next = C4d (scan-level schema dict, back-to-back) Per-round HANDOFF update: mark HD-C4c (per-split THudiFileDesc.schema_id, native branch only, dormant commit ac3222bc526) done in the current-status line and the hudi section; next = C4d (the base64 hudi.schema_evolution dict from getScanNodeProperties + populateScanLevelParams, reusing C4c's shared resolver for the referenced-version history entries). Records the two shared resolvers, the native-only stamping (paimon parity), the safe-prefix ordering (schema_id-without-dict -> BY_NAME, so C4d must follow), the 2-arg searchSchemaAndCache compile signature (vs the 3-arg default-m2 jar), and the honest dormancy. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 890c4ae609f3b4..79836b52cdd9b6 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）→ C4d（扫描级 `hudi.schema_evolution` 字典发射+`populateScanLevelParams` apply，C4c/C4d 背靠背落〔D1 共享 resolver + prefix-flip 安全〕）→ 完成后统一对抗复审 → Part 2 C5a/C5b。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级 `hudi.schema_evolution` 字典发射 `getScanNodeProperties`〔-1 目标条=requested columns + 每引用版本条=C4c resolver 复用〕+`populateScanLevelParams` apply，force_jni/MOR-realtime gate off；与 C4c 同为背靠背对，C4c 安全前缀已落）→ 完成后统一对抗复审 → Part 2 C5a/C5b。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -97,6 +97,7 @@ - **✅ HD-C4/C5 schema-evolution recon + 设计 + 签字（recon `wf_85bd47a0-0aa` 6 读者+完整性/拆解 2 critic〔iron-rule 读者 StructuredOutput 上限失败，gap 由 critic+亲手核对补〕；权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`，起步必读）= 完整对齐老实现（用户签字 B，非窄退化；memory `hudi-schema-evolution-full-parity-signoff`）**。**BE 核实**：原生机器已就位、**零 BE 改**（`be/src/format_v2/table/hudi_reader.cpp` 已读 `hudi_params.schema_id`+`can_map_by_history_schema`+`annotate_file_schema_from_history`，与 paimon 同）。**架构裁决**：**模板=paimon 多条历史(-1+每引用版本)+每-split schema_id**（非 iceberg 单条；`THudiFileDesc` 有 f12 schema_id 而 `TIcebergFileDesc` 无=load-bearing divergence）；**小写逐层=iceberg**（format_v2 大小写不敏感但同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader→`StructNode.children.at()` raw name→std::out_of_range→整库 SIGABRT）；**current_schema_id 保持 -1**（真 id 撞某 split id→v1 `current==split`→ConstNode 大小写敏感逐字=禁的半吊子退化；paimon+legacy 均 -1，BE `find_external_root_field` 认 -1 条目为 target overlay）；**field-id 源=mode-aware commit-metadata InternalSchema**（stable across renames；非 naive `convert(latest avro)` 否则 -1 条目 id≠per-file id→BY_FIELD_ID 失配）；**每-split schema_id 只盖原生分支**（JNI/MOR-realtime 不看 schema_id、读 `column_names/types`@instant→C5b 全靠 FE 供）；**零新 SPI**（3-arg `getTableSchema`+hive 委派已发 `HiveConnectorMetadata:1070-1078`、`populateScanLevelParams` no-op、`getScanNodeProperties`、`ConnectorColumn.withUniqueId`、`THudiFileDesc` f12、`TFileScanRangeParams` 25/26 全在；100% 连接器内部）。**关键结构事实**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:973` vs `:979`）→句柄 field-id 是 latest-keyed→时间旅行改名列须 **C5b 按 pinned schema 重解 -1 条目**（iceberg full-pinned/empty-requested 式），非靠句柄；**C4c 硬序在 C4d 前**（字典present 而 per-split schema_id 仍 -1 = 禁的 current==file==-1）。**§6 签字决策（对齐 legacy 字节忠实）**：历史集=被选中文件涉及版本的并集(非全量枚举、自洽=每发出 split id 必在史、=legacy per-referenced-file、C4c/C4d 共享 resolver 背靠背落)；非演进表 FOR TIME=latest-fallback(schema 从未变=字节等价)；scalar `TColumnType`=STRING placeholder(BE 只用作 nested-vs-scalar 判别，不 port legacy 全 avro→Doris 型表)；field-id 留句柄(=legacy `updateHudiColumnUniqueId`)。**用户签字两部分拆分（2026-07-09）**：**Part 1=C4a–C4d + 完成后统一对抗复审；Part 2=C5a–C5b。用户下个 session 起 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器：`InternalSchema/Types.Field→TSchema/TField`+base64 encode/apply 往返，逐层小写、STRING placeholder、id+name+is_optional 每层，wired 进空，完全可单测，零 fe-core import）**。**e2e 硬门（切换期，非静默）**：改名/调序/加/删列 COW+MOR 读(native BY_FIELD_ID engage)、混合大小写嵌套结构不崩、演进表 FOR TIME(COW 列表+MOR JNI 列表@instant)、非演进 FOR TIME 等价、@incr latest schema、异构 HMS vs 独立 hudi 目录同表比行（memory `hms-iceberg-delegation-needs-e2e`）。 - **✅ HD-C4a DONE（`d29bb52173d`，休眠，Part 1 首步）= 纯 `HudiSchemaUtils` 转换器**（新连接器内部类，`fe-connector-hudi`）。自足 `InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` 构建器 + base64 `encode`/`applySchemaEvolution` 往返 on 抛弃型 `TFileScanRangeParams`（f25 `current_schema_id` + f26 `history_schema_info`）。**移植 legacy `HudiUtils.getSchemaInfo(InternalSchema/List/Types.Field)` 两处有意偏离**（均由 mutation-tight 测钉）：①**逐层小写**（`Locale.ROOT`，镜像 `IcebergSchemaUtils.buildField`；legacy 不小写）——同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader，其 `StructNode` 按小写 Doris slot 名查，混大小写嵌套名→`std::out_of_range`→整库 SIGABRT；②**scalar `TColumnType`=STRING placeholder**（BE field-id 路只用 `type.type` 作 nested-vs-scalar 判别）——丢 legacy `fromAvroHudiTypeToDorisType` 全型表。`id`+`name`+`is_optional` 每层保真（legacy 平价）。`encode` serialize 错 fail-loud（catch `Exception|LinkageError` 防 thrift 类加载器裂）；`applySchemaEvolution` null/empty no-op、corrupt fail-loud（镜像 iceberg/paimon）。**模板=paimon（后续 C4c/C4d 复用 `buildSchemaInfo`+`encode` 发 -1 目标条 + 每引用版本历史 + 每-split schema_id），小写=iceberg 每层。wired 进空、零 fe-core import、零新连接器-api SPI。DORMANT**（hms 未进 `SPI_READY_TYPES`、hudi 句柄未分流）。`HudiSchemaUtilsTest` 11 全 same-loader（手搭混大小写嵌套 struct 子字段的 `InternalSchema` 断言每层小写/嵌套 struct·array·map 标签+每层 field-id/STRING scalar/InternalSchema-keyed 与显式 -1 schema id/base64 往返）。全绿（hudi 模块 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`）**。 - **✅ HD-C4b DONE（`d54dc2d3ac5`，休眠，Part 1 第二步）= field-id 上 `HudiColumnHandle`**（源自 mode-aware InternalSchema）。①`HudiColumnHandle` 加 `int fieldId`（ctor+`getFieldId()`+toString）——**不入 equals/hashCode**（身份保 name+typeName，镜像 `IcebergColumnHandle` 按名不折 id）。②`HudiConnectorMetadata.getSchemaFromMetaClient` 解析 mode-aware InternalSchema：no-arg `TableSchemaResolver.getTableInternalSchemaFromCommitMetadata()` 钉**最新 commit**（schema.on.read 开→commit-metadata InternalSchema〔改名跨版本**稳定 id**〕；关→`AvroInternalSchemaConverter.convert(latest avro)` 位置 id）——镜像 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`（steady-state/无时间旅行 pin，at-instant 变体留 C5a）；解析失败降级 columns 不带 id（UNSET→BE BY_NAME）而非丢整 schema。③`getColumnHandles` 穿 `col.getUniqueId()` 上 handle（镜像 iceberg）。**与 legacy 有意偏离（已文档化）**：legacy 列源+id 源同一 InternalSchema 位置 zip；连接器保 `getTableAvroSchema(true)` 独立列源（保住已发的含 meta 列 schema），故 `attachTopLevelFieldIds` **按名匹配**——对齐列集等价、且当 commit-metadata InternalSchema 缺某列（如 `_hoodie_*` meta）时该列留 UNSET→BY_NAME（正确，meta 列从不改名）。只顶层 id 上 handle；嵌套 field-id 由 InternalSchema 经 `HudiSchemaUtils` 直发（C4a）。**⚠残留**：`NOTE(C5a)` 此 metaClient touch 尚未 `HudiMetaClientExecutor.execute` 包裹——closing 既有 `getSchemaFromMetaClient` 未包裹 auth/TCCL gap 留 C5a（它重构此读）；evolution-mode commit-metadata id 源仅 e2e 可证（单测覆盖非演进 `convert(avro)` 路）。DORMANT（hudi 故意不在 `SPI_READY_TYPES`、句柄未分流→无 live 目录达 `getColumnHandles`）。`HudiColumnFieldIdTest` 4 same-loader（按名取 id/未命中 UNSET/handle 载 id/id 不入身份）。全绿（hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）**。 + - **✅ HD-C4c DONE（`ac3222bc526`，休眠，Part 1 第三步）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**。给每个原生 base 文件钉写它的那次 commit 的 InternalSchema 版本号（field 12），BE 原生 field-id 读取器据此在字典里查该文件的列并按 id 对齐当前表（改名安全）。**仅原生分支**：JNI 合并读取器读 `column_names/types`@instant、不消费 schema_id。①`HudiSchemaUtils` 两共享静态 resolver：`resolveTableInternalSchema`（mode-aware base：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 版本 0，+evolution 标志；C4b 的 `attachHudiFieldIds` 重构复用之=id 源单一真源）+`resolveFileInternalSchema`（每文件版本：evolution 开→`FSUtils.getCommitTime(fileName)`+`InternalSchemaCache.searchSchemaAndCache`；关→base 版本 0）——port legacy `HudiScanNode.setHudiParams`。②`HudiScanRange.Builder.schemaId`（可空）→ctor 仅设时放 `hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** `fileDesc.setSchemaId`（镜像 `PaimonScanRange`），JNI 臂永不；缺→BE BY_NAME。③`planScan` 解析一次 base schema、建每文件 schema_id resolver（base 文件路径→版本）；`collectCowSplits`（COW base 恒原生）+`buildMorRange` 原生无 log 片钉之；@incr 路传 null（@incr 列最新 schema、无每文件字典）。base-schema 未解析→null resolver→不钉；每文件解析失败→log 跳过该文件→均降 BY_NAME 不崩扫描。**⚠编译期坑**：`searchSchemaAndCache` 编译classpath 是 **2-arg** `(long, metaClient)`（与 legacy fe-core 同），非默认 m2 里 SELECTDB-0.14.1 jar 的 3-arg（javap 看错 jar）——已用 2-arg。**排序（关键）**：schema_id present 而字典缺→BE 不入 field-id 路→**BY_NAME=安全前缀**；字典步（C4d）须在其后，dict-present+schema_id-unset 才是禁的 `current==file==-1` v1-ConstNode 退化。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。schema_id **值 e2e**，只 stamping 机制 same-loader 测（`HudiScanRangeTest`+3：原生 set f12 / JNI 即便设了也不 set / 原生无 id 留 unset；`HudiIncrementalPlanScanTest` buildMorRange 断言原生 downgrade 载 resolver id、force-jni 片不载）。全绿（hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级字典发射，与 C4c 背靠背；C4c 安全前缀已落）**。 - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From fa04b27d032e21662f03aa4763a38794a8d4bc29 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 20:10:26 +0800 Subject: [PATCH 132/334] [feat](catalog) HMS cutover hudi HD-C4d: dormant scan-level schema-evolution dictionary (completes Part 1) Fourth and final steady-state step. Emits the native-reader schema dictionary (current_schema_id + history_schema_info) so BE matches file<->table columns BY FIELD ID across rename/reorder instead of BY_NAME (which reads NULL for a renamed column on its old files). Together with the per-split schema_id (HD-C4c) this completes the pair that engages BE's field-id path. - HudiSchemaUtils.buildSchemaEvolutionDict (pure, unit-tested): one -1 target entry + the history entries. The -1 target overlay (buildTargetSchema) is keyed off the REQUESTED columns so its top-level names == the BE scan slots (the CI-969249 StructNode invariant); each column's full nested structure + stable field id is looked up BY NAME in the mode-aware base InternalSchema; a requested column absent from that schema (e.g. a _hoodie_* meta column on a schema-evolved table) falls back to a scalar placeholder with the handle's field id; empty requested (count-only) -> all base fields. - HISTORY-SET = ALL committed schema versions (user-signed-off Option B), a robust SUPERSET of the referenced-file versions mirroring the paimon connector's all-listAllIds emission: self-consistent by construction (every native file's schema_id is present, so BE never fails "miss schema info"), with NO coupling between the dict and which files a scan selects (avoids the divergence/expense of re-enumerating files in a getScanNodeProperties that -- like paimon's -- has no access to planScan's splits). Evolution: all versions via FileBasedInternalSchemaStorageManager.getHistorySchemaStr + SerDeHelper.parseSchemas. Non-evolution: the single base convert()-schema (version 0). - HudiScanPlanProvider.getScanNodeProperties builds + emits the base64 hudi.schema_evolution prop, gated OFF under force_jni_scanner (every split then goes JNI and never consults the dict -- paimon-parity gate); best-effort (build failure logs + drops the prop -> BE BY_NAME, never a hard scan failure). populateScanLevelParams is overridden to copy fields 25/26 onto the real params via HudiSchemaUtils.applySchemaEvolution. Ordering: HD-C4c (per-split schema_id, the safe prefix) is already landed, so this dict never appears with schema_id unset (the forbidden current==file==-1 v1-ConstNode degrade). Zero fe-core import; zero new connector-api SPI. DORMANT (hudi is deliberately NOT in SPI_READY_TYPES; handles not diverted). BE engage + history-set completeness are flip-time e2e; the dict assembly + round-trip are same-loader tested. Tests: HudiSchemaUtilsTest +5 (target entry keyed off requested columns with nested field ids; scalar fallback for a column absent from the base schema; empty-requested -> all base fields; non-evolution dict = target + base version; evolution dict = target + all historical versions; all via encode/apply round-trip). Full hudi module 150/150 green; checkstyle 0; import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiScanPlanProvider.java | 42 +++++++ .../doris/connector/hudi/HudiSchemaUtils.java | 104 ++++++++++++++++++ .../connector/hudi/HudiSchemaUtilsTest.java | 97 ++++++++++++++++ 3 files changed, 243 insertions(+) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 50d432c34c4737..5ab6d27fccba93 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.thrift.TFileScanRangeParams; import org.apache.avro.Schema; import org.apache.hadoop.conf.Configuration; @@ -91,6 +92,11 @@ public class HudiScanPlanProvider implements ConnectorScanPlanProvider { // path as the paimon connector's FORCE_JNI_SCANNER. Default false, so normal reads are unaffected. private static final String FORCE_JNI_SCANNER = "force_jni_scanner"; + // Scan-node prop carrying the base64 native-reader schema-evolution dictionary (current_schema_id + + // history_schema_info). getScanNodeProperties builds it; populateScanLevelParams copies it onto the real + // TFileScanRangeParams. Mirrors the paimon connector's SCHEMA_EVOLUTION_PROP. + private static final String SCHEMA_EVOLUTION_PROP = "hudi.schema_evolution"; + private final Map properties; private final ConnectorContext context; @@ -295,9 +301,45 @@ public Map getScanNodeProperties( } } + // Emit the native-reader schema-evolution dictionary so BE matches file<->table columns BY FIELD ID + // across rename/reorder (else a renamed column reads NULL on its old files). Skipped under force_jni: + // every split then goes JNI and never consults the dict (paimon-parity gate). Best-effort: a build + // failure logs and drops the prop -> native reads fall back to BE BY_NAME (the safe baseline), never a + // hard scan failure. populateScanLevelParams copies it onto the real params. + if (!isForceJniScannerEnabled(session)) { + try { + HoodieTableMetaClient metaClient = buildMetaClient(buildHadoopConf(), hudiHandle.getBasePath()); + TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); + Schema latestAvro = schemaResolver.getTableAvroSchema(true); + HudiSchemaUtils.buildSchemaEvolutionProp(metaClient, schemaResolver, latestAvro, + castHudiColumns(columns)) + .ifPresent(v -> props.put(SCHEMA_EVOLUTION_PROP, v)); + } catch (Exception e) { + LOG.warn("Failed to build Hudi schema-evolution dict for {}.{}; native reads fall back to BY_NAME: {}", + hudiHandle.getDbName(), hudiHandle.getTableName(), e.getMessage()); + } + } + return props; } + /** The requested column handles as {@link HudiColumnHandle}s (this connector's own handle type). */ + private static List castHudiColumns(List columns) { + if (columns == null) { + return Collections.emptyList(); + } + List result = new ArrayList<>(columns.size()); + for (ConnectorColumnHandle handle : columns) { + result.add((HudiColumnHandle) handle); + } + return result; + } + + @Override + public void populateScanLevelParams(TFileScanRangeParams params, Map nodeProperties) { + HudiSchemaUtils.applySchemaEvolution(params, nodeProperties.get(SCHEMA_EVOLUTION_PROP)); + } + /** * Collect splits for COW (Copy on Write) tables. * COW tables only have base files — use native Parquet/ORC reader. diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java index 84ab65f2296c1b..fa8dab44fad5b2 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java @@ -37,14 +37,22 @@ import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.internal.schema.Types; import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; +import org.apache.hudi.internal.schema.io.FileBasedInternalSchemaStorageManager; +import org.apache.hudi.internal.schema.utils.SerDeHelper; import org.apache.thrift.TDeserializer; import org.apache.thrift.TSerializer; import org.apache.thrift.protocol.TBinaryProtocol; import java.io.File; +import java.util.ArrayList; import java.util.Base64; +import java.util.Collection; +import java.util.Collections; +import java.util.HashMap; import java.util.List; import java.util.Locale; +import java.util.Map; +import java.util.Optional; /** * Builds the native-reader schema dictionary ({@code current_schema_id} + {@code history_schema_info}) so BE @@ -269,4 +277,100 @@ static InternalSchema resolveFileInternalSchema(String filePath, boolean enableS long commitTime = Long.parseLong(FSUtils.getCommitTime(new File(filePath).getName())); return InternalSchemaCache.searchSchemaAndCache(commitTime, metaClient); } + + // ========== scan-level schema-evolution dictionary (current_schema_id + history_schema_info) ========== + + /** + * Build + serialize the native-reader schema dictionary for the scan-node prop. The {@code -1} target entry + * is keyed off the {@code requestedColumns} (its top-level names == BE scan slots by construction); the + * history entries cover every schema version any native file could carry. Touches the metaClient to resolve + * the mode-aware base schema and the historical versions; the pure dict assembly is + * {@link #buildSchemaEvolutionDict}. Runs on the TCCL-pinned scan thread. + */ + static Optional buildSchemaEvolutionProp(HoodieTableMetaClient metaClient, + TableSchemaResolver schemaResolver, Schema latestAvro, List requestedColumns) { + ResolvedInternalSchema resolved = resolveTableInternalSchema(schemaResolver, latestAvro); + Collection historical = resolved.enableSchemaEvolution + ? allHistoricalSchemas(metaClient) + : Collections.emptyList(); + return Optional.of(buildSchemaEvolutionDict(requestedColumns, resolved.internalSchema, + resolved.enableSchemaEvolution, historical)); + } + + /** + * Pure assembly of the schema dictionary (package-private for same-loader testing): one {@code -1} target + * entry (from the requested columns + base schema) + the history entries. HISTORY-SET = ALL committed + * schema versions (a robust SUPERSET of the referenced-file versions, mirroring the paimon connector's + * all-{@code listAllIds} emission): self-consistent by construction (every native file's {@code schema_id} + * is present, so BE never fails "miss schema info"), with no coupling between the dict and which files a + * given scan happens to select. For a non-evolution table there is no history file, so the single + * non-evolution InternalSchema (version {@code 0}) is emitted as the only history entry. + */ + static String buildSchemaEvolutionDict(List requestedColumns, InternalSchema baseSchema, + boolean enableSchemaEvolution, Collection historicalVersions) { + List history = new ArrayList<>(); + history.add(buildTargetSchema(requestedColumns, baseSchema)); + if (enableSchemaEvolution) { + for (InternalSchema version : historicalVersions) { + history.add(buildSchemaInfo(version)); + } + } else { + // Non-evolution: no history file; the base convert()-schema (version 0) is the only file version. + history.add(buildSchemaInfo(baseSchema)); + } + return encode(CURRENT_SCHEMA_ID, history); + } + + /** + * The {@code -1} target/current overlay, keyed off the requested (pruned) columns so its top-level names + * equal the BE scan slots (the CI-969249 StructNode invariant). Each requested column's full nested + * structure + stable field id is looked up BY NAME in {@code baseSchema}. A requested column absent from + * {@code baseSchema} (e.g. a {@code _hoodie_*} meta column on a schema-evolved table whose commit-metadata + * schema omits meta fields) falls back to a scalar placeholder carrying the handle's field id ({@code -1} = + * unset). Empty {@code requestedColumns} (count-only scan) falls back to all base top-level fields. + */ + static TSchema buildTargetSchema(List requestedColumns, InternalSchema baseSchema) { + TSchema tSchema = new TSchema(); + tSchema.setSchemaId(CURRENT_SCHEMA_ID); + TStructField root = new TStructField(); + if (requestedColumns == null || requestedColumns.isEmpty()) { + for (Types.Field field : baseSchema.getRecord().fields()) { + root.addToFields(fieldPtr(buildField(field))); + } + } else { + Map byName = new HashMap<>(); + for (Types.Field field : baseSchema.getRecord().fields()) { + byName.put(field.name().toLowerCase(Locale.ROOT), field); + } + for (HudiColumnHandle handle : requestedColumns) { + Types.Field field = byName.get(handle.getName().toLowerCase(Locale.ROOT)); + root.addToFields(fieldPtr(field != null + ? buildField(field) + : scalarField(handle.getName().toLowerCase(Locale.ROOT), handle.getFieldId()))); + } + } + tSchema.setRootField(root); + return tSchema; + } + + /** All committed InternalSchema versions of the table (empty for a non-evolution table). */ + private static Collection allHistoricalSchemas(HoodieTableMetaClient metaClient) { + String historyStr = new FileBasedInternalSchemaStorageManager(metaClient).getHistorySchemaStr(); + if (historyStr == null || historyStr.isEmpty()) { + return Collections.emptyList(); + } + return SerDeHelper.parseSchemas(historyStr).values(); + } + + /** A scalar-leaf {@link TField} (STRING placeholder) carrying a name + field id (for the target-entry fallback). */ + private static TField scalarField(String name, int id) { + TField tField = new TField(); + tField.setName(name); + tField.setId(id); + tField.setIsOptional(true); + TColumnType columnType = new TColumnType(); + columnType.setType(TPrimitiveType.STRING); + tField.setType(columnType); + return tField; + } } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java index 2ec39b6443f7fe..6078d7a6ae2721 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java @@ -248,4 +248,101 @@ public void applyFailsLoudOnCorruptProp() { Assertions.assertThrows(RuntimeException.class, () -> HudiSchemaUtils.applySchemaEvolution(params, "!!!not-base64-thrift!!!")); } + + // ========== C4d: scan-level dictionary (-1 target entry + history entries) ========== + + private static HudiColumnHandle handle(String name, int fieldId) { + return new HudiColumnHandle(name, "string", false, fieldId); + } + + private static InternalSchema internalSchemaWithId(long schemaId) { + return new InternalSchema(schemaId, Types.RecordType.get(Collections.singletonList( + Types.Field.get(1, false, "id", Types.IntType.get())))); + } + + @Test + public void targetEntryKeyedOffRequestedColumnsWithSchemaIds() { + // The -1 target entry's top-level names must be EXACTLY the requested (scan-slot) columns, in order, each + // carrying the stable field id + full nested structure looked up BY NAME in the base InternalSchema. This + // is the overlay BE stamps each table column's id from. MUTATION: emit all base columns instead of the + // requested subset -> extra "name"/"tags"/"props" appear -> red. + InternalSchema base = buildInternalSchema(); + TSchema target = HudiSchemaUtils.buildTargetSchema( + Arrays.asList(handle("id", 1), handle("Addr", 3)), base); + + Assertions.assertEquals(-1L, target.getSchemaId()); + Map top = topFields(target); + Assertions.assertEquals(Arrays.asList("id", "addr"), Arrays.asList(top.keySet().toArray())); + // "id" carries the base InternalSchema field id (1) + scalar placeholder type ... + Assertions.assertEquals(1, top.get("id").getId()); + Assertions.assertEquals(TPrimitiveType.STRING, top.get("id").getType().getType()); + // ... "Addr" (mixed-case request) is matched case-insensitively, lowercased, id 3, with its nested struct + // child "street" carried (id 4) — proves the target entry keeps nested field ids for BE nested matching. + TField addr = top.get("addr"); + Assertions.assertEquals(3, addr.getId()); + Assertions.assertEquals(TPrimitiveType.STRUCT, addr.getType().getType()); + Assertions.assertEquals(4, structChild(addr, "street").getId()); + } + + @Test + public void targetEntryFallsBackToScalarForColumnAbsentFromSchema() { + // A requested column not in the base InternalSchema (e.g. a _hoodie_* meta column on a schema-evolved + // table whose commit-metadata schema omits meta fields) must still appear in the -1 entry (it IS a BE scan + // slot) as a scalar placeholder carrying the handle's field id. MUTATION: drop it -> the -1 entry is + // missing a scan slot -> BE StructNode lookup miss -> red. + InternalSchema base = buildInternalSchema(); + TSchema target = HudiSchemaUtils.buildTargetSchema( + Arrays.asList(handle("id", 1), handle("_hoodie_commit_time", 77)), base); + + Map top = topFields(target); + Assertions.assertTrue(top.containsKey("_hoodie_commit_time")); + Assertions.assertEquals(77, top.get("_hoodie_commit_time").getId()); + Assertions.assertEquals(TPrimitiveType.STRING, top.get("_hoodie_commit_time").getType().getType()); + } + + @Test + public void targetEntryEmptyRequestedFallsBackToAllBaseFields() { + // A count-only scan (no projected columns) yields an empty requested list; the -1 entry then carries all + // base top-level fields (a valid superset). MUTATION: emit an empty root -> BE has no target overlay -> red. + TSchema target = HudiSchemaUtils.buildTargetSchema(Collections.emptyList(), buildInternalSchema()); + Assertions.assertEquals(5, target.getRootField().getFieldsSize()); + } + + @Test + public void dictNonEvolutionEmitsTargetPlusBaseVersion() { + // Non-evolution table: no schema history file, so the dict is the -1 target + the single base + // (convert()) version. MUTATION: skip the base entry -> a native file's schema_id (0) has no history + // entry -> BE "miss schema info" fail-loud -> red. + InternalSchema base = internalSchemaWithId(0L); + String encoded = HudiSchemaUtils.buildSchemaEvolutionDict( + Collections.singletonList(handle("id", 1)), base, false, Collections.emptyList()); + + TFileScanRangeParams params = new TFileScanRangeParams(); + HudiSchemaUtils.applySchemaEvolution(params, encoded); + Assertions.assertEquals(-1L, params.getCurrentSchemaId()); + Assertions.assertEquals(2, params.getHistorySchemaInfoSize()); + Assertions.assertEquals(-1L, params.getHistorySchemaInfo().get(0).getSchemaId()); + Assertions.assertEquals(0L, params.getHistorySchemaInfo().get(1).getSchemaId()); + } + + @Test + public void dictEvolutionEmitsTargetPlusAllHistoricalVersions() { + // Evolution table: the dict is the -1 target + ONE entry per committed schema version (all-historical = + // robust superset of the referenced-file versions). MUTATION: emit only the base version -> a native file + // written under an older version has no history entry -> BE "miss schema info" -> red. + InternalSchema base = internalSchemaWithId(2L); + String encoded = HudiSchemaUtils.buildSchemaEvolutionDict( + Collections.singletonList(handle("id", 1)), base, true, + Arrays.asList(internalSchemaWithId(0L), internalSchemaWithId(1L), internalSchemaWithId(2L))); + + TFileScanRangeParams params = new TFileScanRangeParams(); + HudiSchemaUtils.applySchemaEvolution(params, encoded); + Assertions.assertEquals(-1L, params.getCurrentSchemaId()); + // -1 target + versions 0/1/2 + Assertions.assertEquals(4, params.getHistorySchemaInfoSize()); + Assertions.assertEquals(-1L, params.getHistorySchemaInfo().get(0).getSchemaId()); + Assertions.assertEquals(0L, params.getHistorySchemaInfo().get(1).getSchemaId()); + Assertions.assertEquals(1L, params.getHistorySchemaInfo().get(2).getSchemaId()); + Assertions.assertEquals(2L, params.getHistorySchemaInfo().get(3).getSchemaId()); + } } From b5a8b9e687c175f246307e94ccdce4a449259f6e Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 20:11:55 +0800 Subject: [PATCH 133/334] [doc](catalog) HMS cutover hudi HD-C4d DONE; Part 1 (C4a-C4d) complete; next = consolidated adversarial review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Per-round HANDOFF update: mark HD-C4d (scan-level hudi.schema_evolution dictionary, dormant commit 617c44b049b) done; Part 1 steady-state read evolution (C4a converter -> C4b handle field-ids -> C4c per-split schema_id -> C4d scan-level dict) is implementation- complete. Records the user-signed-off Option B (all-historical versions, robust superset, paimon parity, no dict/split coupling), the requested-columns -1 target entry with the scalar fallback, the force_jni gate, and the safe-prefix ordering. Next per design §9 = a consolidated adversarial review of the whole steady-state part before Part 2 (C5a/C5b time-travel-over-evolved). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 79836b52cdd9b6..1d6f7ee452a906 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级 `hudi.schema_evolution` 字典发射 `getScanNodeProperties`〔-1 目标条=requested columns + 每引用版本条=C4c resolver 复用〕+`populateScanLevelParams` apply，force_jni/MOR-realtime gate off；与 C4c 同为背靠背对，C4c 安全前缀已落）→ 完成后统一对抗复审 → Part 2 C5a/C5b。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。下一步 = 对整块 Part 1 做统一对抗复审（设计 §9 签字要求，Part 2 前置）→ 之后 Part 2 = C5a（getTableSchema@instant）+ C5b（扫描侧@instant）时间旅行叠加演进。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -98,6 +98,7 @@ - **✅ HD-C4a DONE（`d29bb52173d`，休眠，Part 1 首步）= 纯 `HudiSchemaUtils` 转换器**（新连接器内部类，`fe-connector-hudi`）。自足 `InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` 构建器 + base64 `encode`/`applySchemaEvolution` 往返 on 抛弃型 `TFileScanRangeParams`（f25 `current_schema_id` + f26 `history_schema_info`）。**移植 legacy `HudiUtils.getSchemaInfo(InternalSchema/List/Types.Field)` 两处有意偏离**（均由 mutation-tight 测钉）：①**逐层小写**（`Locale.ROOT`，镜像 `IcebergSchemaUtils.buildField`；legacy 不小写）——同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader，其 `StructNode` 按小写 Doris slot 名查，混大小写嵌套名→`std::out_of_range`→整库 SIGABRT；②**scalar `TColumnType`=STRING placeholder**（BE field-id 路只用 `type.type` 作 nested-vs-scalar 判别）——丢 legacy `fromAvroHudiTypeToDorisType` 全型表。`id`+`name`+`is_optional` 每层保真（legacy 平价）。`encode` serialize 错 fail-loud（catch `Exception|LinkageError` 防 thrift 类加载器裂）；`applySchemaEvolution` null/empty no-op、corrupt fail-loud（镜像 iceberg/paimon）。**模板=paimon（后续 C4c/C4d 复用 `buildSchemaInfo`+`encode` 发 -1 目标条 + 每引用版本历史 + 每-split schema_id），小写=iceberg 每层。wired 进空、零 fe-core import、零新连接器-api SPI。DORMANT**（hms 未进 `SPI_READY_TYPES`、hudi 句柄未分流）。`HudiSchemaUtilsTest` 11 全 same-loader（手搭混大小写嵌套 struct 子字段的 `InternalSchema` 断言每层小写/嵌套 struct·array·map 标签+每层 field-id/STRING scalar/InternalSchema-keyed 与显式 -1 schema id/base64 往返）。全绿（hudi 模块 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`）**。 - **✅ HD-C4b DONE（`d54dc2d3ac5`，休眠，Part 1 第二步）= field-id 上 `HudiColumnHandle`**（源自 mode-aware InternalSchema）。①`HudiColumnHandle` 加 `int fieldId`（ctor+`getFieldId()`+toString）——**不入 equals/hashCode**（身份保 name+typeName，镜像 `IcebergColumnHandle` 按名不折 id）。②`HudiConnectorMetadata.getSchemaFromMetaClient` 解析 mode-aware InternalSchema：no-arg `TableSchemaResolver.getTableInternalSchemaFromCommitMetadata()` 钉**最新 commit**（schema.on.read 开→commit-metadata InternalSchema〔改名跨版本**稳定 id**〕；关→`AvroInternalSchemaConverter.convert(latest avro)` 位置 id）——镜像 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`（steady-state/无时间旅行 pin，at-instant 变体留 C5a）；解析失败降级 columns 不带 id（UNSET→BE BY_NAME）而非丢整 schema。③`getColumnHandles` 穿 `col.getUniqueId()` 上 handle（镜像 iceberg）。**与 legacy 有意偏离（已文档化）**：legacy 列源+id 源同一 InternalSchema 位置 zip；连接器保 `getTableAvroSchema(true)` 独立列源（保住已发的含 meta 列 schema），故 `attachTopLevelFieldIds` **按名匹配**——对齐列集等价、且当 commit-metadata InternalSchema 缺某列（如 `_hoodie_*` meta）时该列留 UNSET→BY_NAME（正确，meta 列从不改名）。只顶层 id 上 handle；嵌套 field-id 由 InternalSchema 经 `HudiSchemaUtils` 直发（C4a）。**⚠残留**：`NOTE(C5a)` 此 metaClient touch 尚未 `HudiMetaClientExecutor.execute` 包裹——closing 既有 `getSchemaFromMetaClient` 未包裹 auth/TCCL gap 留 C5a（它重构此读）；evolution-mode commit-metadata id 源仅 e2e 可证（单测覆盖非演进 `convert(avro)` 路）。DORMANT（hudi 故意不在 `SPI_READY_TYPES`、句柄未分流→无 live 目录达 `getColumnHandles`）。`HudiColumnFieldIdTest` 4 same-loader（按名取 id/未命中 UNSET/handle 载 id/id 不入身份）。全绿（hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）**。 - **✅ HD-C4c DONE（`ac3222bc526`，休眠，Part 1 第三步）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**。给每个原生 base 文件钉写它的那次 commit 的 InternalSchema 版本号（field 12），BE 原生 field-id 读取器据此在字典里查该文件的列并按 id 对齐当前表（改名安全）。**仅原生分支**：JNI 合并读取器读 `column_names/types`@instant、不消费 schema_id。①`HudiSchemaUtils` 两共享静态 resolver：`resolveTableInternalSchema`（mode-aware base：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 版本 0，+evolution 标志；C4b 的 `attachHudiFieldIds` 重构复用之=id 源单一真源）+`resolveFileInternalSchema`（每文件版本：evolution 开→`FSUtils.getCommitTime(fileName)`+`InternalSchemaCache.searchSchemaAndCache`；关→base 版本 0）——port legacy `HudiScanNode.setHudiParams`。②`HudiScanRange.Builder.schemaId`（可空）→ctor 仅设时放 `hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** `fileDesc.setSchemaId`（镜像 `PaimonScanRange`），JNI 臂永不；缺→BE BY_NAME。③`planScan` 解析一次 base schema、建每文件 schema_id resolver（base 文件路径→版本）；`collectCowSplits`（COW base 恒原生）+`buildMorRange` 原生无 log 片钉之；@incr 路传 null（@incr 列最新 schema、无每文件字典）。base-schema 未解析→null resolver→不钉；每文件解析失败→log 跳过该文件→均降 BY_NAME 不崩扫描。**⚠编译期坑**：`searchSchemaAndCache` 编译classpath 是 **2-arg** `(long, metaClient)`（与 legacy fe-core 同），非默认 m2 里 SELECTDB-0.14.1 jar 的 3-arg（javap 看错 jar）——已用 2-arg。**排序（关键）**：schema_id present 而字典缺→BE 不入 field-id 路→**BY_NAME=安全前缀**；字典步（C4d）须在其后，dict-present+schema_id-unset 才是禁的 `current==file==-1` v1-ConstNode 退化。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。schema_id **值 e2e**，只 stamping 机制 same-loader 测（`HudiScanRangeTest`+3：原生 set f12 / JNI 即便设了也不 set / 原生无 id 留 unset；`HudiIncrementalPlanScanTest` buildMorRange 断言原生 downgrade 载 resolver id、force-jni 片不载）。全绿（hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级字典发射，与 C4c 背靠背；C4c 安全前缀已落）**。 + - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；下一步 = 整块统一对抗复审（Part 2 前置）。** - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From 7b1241508c4455d63af8166571b3801eacfaa6ba Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 20:48:46 +0800 Subject: [PATCH 134/334] [fix](catalog) HMS cutover hudi Part 1 review fixes: meta-column BY_FIELD_ID gate + resilience + test-intent Fixes from the consolidated adversarial review of hudi schema-evolution Part 1 (C4a-C4d). 6 CONFIRMED findings (all dormant / flip-time; hudi is not in SPI_READY_TYPES): 1) [major, correctness] Meta-column NULL regression. BE's field-id mapping is PER-FILE, not per-column: once a native file carries a schema_id, EVERY projected column of that file is matched by id with NO per-column BY_NAME fallback. On a schema.on.read (evolution) table the connector exposes the _hoodie_* meta columns (getTableAvroSchema(true)) but the mode-aware commit-metadata InternalSchema omits them, so their handle field id is UNSET(-1); the -1 target overlay carried them with id=-1 and no history entry has them -> BE would read them const-NULL on SELECT * (verified end-to-end against be/src/format_v2 column_mapper.cpp). The repeated "UNSET -> BY_NAME" comments were false. FIX: HudiSchemaUtils.buildSchemaEvolutionProp now GATES the whole dict OFF when ANY projected column has an unresolved field id -> BE stays on BY_NAME for the entire scan (the safe baseline; data-only projections still get full field-id/rename matching). Comments in HudiColumnHandle / HudiConnectorMetadata / HudiSchemaUtils corrected. FLIP-TIME RESIDUAL (owed the e2e): full rename-correctness for SELECT* over an evolution table needs reserved meta-column ids injected into every entry (iceberg row-lineage pattern) or dropping meta exposure to mirror legacy -- deferred. 2) [minor, resilience] HudiScanPlanProvider.planScan had resolveTableInternalSchema as the last statement of the try that computes the JNI columnNames/columnTypes, whose catch wipes them to emptyList -- a schema-evolution hiccup would ship empty column lists to the MOR JNI reader instead of degrading to BY_NAME. FIX: resolve the InternalSchema in its OWN try/catch (null -> BY_NAME) without touching the already-computed JNI column lists. 3-6) [minor/nit, test-intent] Added a C4b test where the unmatched column is FIRST (kills a positional-zip regression the order-aligned fixtures missed); softened handleCarriesFieldId's javadoc (it pins only the ctor/getter; the getColumnHandles threading is flip-time e2e); added a same-loader test for resolveFileInternalSchema's non-evolution branch; replaced a tautological round-trip lowercase assertion with a nested-field-id survival assertion. Added a test pinning the new dict gate. Full hudi module 153/153 green; checkstyle 0; import gate clean. Still DORMANT. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiColumnHandle.java | 9 +++-- .../connector/hudi/HudiConnectorMetadata.java | 9 +++-- .../connector/hudi/HudiScanPlanProvider.java | 24 +++++++++--- .../doris/connector/hudi/HudiSchemaUtils.java | 27 ++++++++++++-- .../connector/hudi/HudiColumnFieldIdTest.java | 37 ++++++++++++++++++- .../connector/hudi/HudiSchemaUtilsTest.java | 32 ++++++++++++++-- 6 files changed, 117 insertions(+), 21 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java index 4a3a54e2e4266a..d350937a516acb 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiColumnHandle.java @@ -35,9 +35,12 @@ public class HudiColumnHandle implements ConnectorColumnHandle { private final boolean isPartitionKey; // Hudi InternalSchema field id (stable across renames), sourced from the mode-aware InternalSchema and // threaded here by HudiConnectorMetadata.getColumnHandles. ConnectorColumn.UNSET_UNIQUE_ID (-1) when no id - // was resolved (e.g. a _hoodie_* meta column absent from a commit-metadata schema -> BE falls back to - // BY_NAME). Deliberately NOT part of equals/hashCode: the handle's identity stays name+type (mirror - // IcebergColumnHandle, which keeps identity by name and does not fold the field id in). + // was resolved (e.g. a _hoodie_* meta column absent from a commit-metadata schema). BE's field-id mode is + // PER-FILE, not per-column, so an unresolved id CANNOT fall back BY_NAME on its own; instead the whole + // scan-level dict is gated OFF when any projected column is unresolved (see + // HudiSchemaUtils.buildSchemaEvolutionProp) -> BE stays on BY_NAME for the entire scan. Deliberately NOT part + // of equals/hashCode: the handle's identity stays name+type (mirror IcebergColumnHandle, which keeps identity + // by name and does not fold the field id in). private final int fieldId; public HudiColumnHandle(String name, String typeName, boolean isPartitionKey, int fieldId) { diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 03d29580ad8aea..08686e7f887be4 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -208,7 +208,8 @@ public Map getColumnHandles( boolean isPartKey = partKeyNames != null && partKeyNames.contains(col.getName()); // Thread the Hudi InternalSchema field id (set by getSchemaFromMetaClient) onto the handle so - // schema-evolution reads match old files BY FIELD ID (HD-C4b). UNSET (-1) -> BE BY_NAME. + // schema-evolution reads match old files BY FIELD ID (HD-C4b). UNSET (-1) when unresolved -> the + // scan-level dict is gated OFF for the whole scan (per-file BY_NAME), not per-column. handles.put(col.getName(), new HudiColumnHandle(col.getName(), col.getType().getTypeName(), isPartKey, col.getUniqueId())); @@ -723,8 +724,10 @@ private List attachHudiFieldIds(TableSchemaResolver schemaResol * legacy {@code HudiUtils.updateHudiColumnUniqueId} at the top level only: the handle carries the top-level * field id, while nested field ids for the BE schema dictionary come straight from the InternalSchema via * {@link HudiSchemaUtils}. A column with no matching InternalSchema field (e.g. a {@code _hoodie_*} meta - * column absent from a commit-metadata schema) keeps {@link ConnectorColumn#UNSET_UNIQUE_ID} -> BE BY_NAME - * (correct: meta columns are never renamed). Package-private + static for same-loader unit testing. + * column absent from a commit-metadata schema) keeps {@link ConnectorColumn#UNSET_UNIQUE_ID}; because BE's + * field-id mode is per-file (not per-column), that unresolved id gates the whole scan-level dict OFF -> + * BE BY_NAME for the entire scan (see {@code HudiSchemaUtils.buildSchemaEvolutionProp}), never a silent + * per-column drop. Package-private + static for same-loader unit testing. */ static List attachTopLevelFieldIds(List columns, InternalSchema internalSchema) { Map idByName = new HashMap<>(); diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 5ab6d27fccba93..21f418974fba5f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -160,22 +160,18 @@ public List planScan( // Resolve column names and types for JNI reader List columnNames; List columnTypes; - // The mode-aware table InternalSchema (+ evolution flag), resolved ONCE, drives the per-file - // THudiFileDesc.schema_id stamped on native slices for BE's field-id path. Null when unresolved -> - // no schema_id -> BE BY_NAME (the safe baseline). schema_id is dormant/inert until the dict is emitted. - HudiSchemaUtils.ResolvedInternalSchema resolvedSchema = null; + Schema avroSchema = null; try { TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); // include the 5 `_hoodie_*` meta columns (explicit `true` = legacy parity, in lockstep with // HudiConnectorMetadata.getSchemaFromMetaClient) so the JNI reader's column list matches the // exposed schema; byte-identical for the common populate.meta.fields=true table. - Schema avroSchema = schemaResolver.getTableAvroSchema(true); + avroSchema = schemaResolver.getTableAvroSchema(true); columnNames = avroSchema.getFields().stream() .map(Schema.Field::name).collect(Collectors.toList()); columnTypes = avroSchema.getFields().stream() .map(f -> HudiTypeMapping.toHiveTypeString(f.schema())) .collect(Collectors.toList()); - resolvedSchema = HudiSchemaUtils.resolveTableInternalSchema(schemaResolver, avroSchema); } catch (Exception e) { LOG.warn("Failed to resolve Hudi schema for JNI reader, JNI splits may fail: {}", e.getMessage()); @@ -183,6 +179,22 @@ public List planScan( columnTypes = Collections.emptyList(); } + // The mode-aware table InternalSchema (+ evolution flag), resolved ONCE, drives the per-file + // THudiFileDesc.schema_id stamped on native slices for BE's field-id path. Resolved in its OWN try/catch + // (NOT the JNI-column block above): a schema-evolution / commit-metadata hiccup must degrade to null -> + // no schema_id -> BE BY_NAME (the safe baseline), WITHOUT discarding the already-computed JNI columnNames/ + // columnTypes (which a plain MOR read needs). schema_id is dormant/inert until the dict is emitted. + HudiSchemaUtils.ResolvedInternalSchema resolvedSchema = null; + if (avroSchema != null) { + try { + resolvedSchema = HudiSchemaUtils.resolveTableInternalSchema( + new TableSchemaResolver(metaClient), avroSchema); + } catch (Exception e) { + LOG.warn("Failed to resolve Hudi InternalSchema for schema_id; native reads fall back to BY_NAME: {}", + e.getMessage()); + } + } + // Per-file native-reader schema_id resolver (base-file path -> version), or null to skip stamping // (base schema unresolved -> BY_NAME). A per-file resolution failure logs and returns null for that file // (BY_NAME) rather than failing the whole scan. Runs on this TCCL-pinned scan thread. diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java index fa8dab44fad5b2..66f3f4b1377577 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java @@ -289,6 +289,22 @@ static InternalSchema resolveFileInternalSchema(String filePath, boolean enableS */ static Optional buildSchemaEvolutionProp(HoodieTableMetaClient metaClient, TableSchemaResolver schemaResolver, Schema latestAvro, List requestedColumns) { + // Field-id matching is PER-FILE in BE, NOT per-column: once a native file carries a schema_id, EVERY + // projected column of that file is matched by id, with NO per-column BY_NAME fallback. So if ANY projected + // column has no resolved field id, emitting the dict would silently read that column as const-NULL. The + // case that hits this is a projected _hoodie_* meta column on a schema-on-read (evolution) table: the + // connector exposes the meta columns (getTableAvroSchema(true)) but the mode-aware commit-metadata + // InternalSchema omits them, so their handle field id stays UNSET (-1). Skip the dict entirely then -> BE + // stays on BY_NAME for the WHOLE scan (the safe baseline: only renamed columns read wrong, exactly as + // before this feature; meta columns read correctly by name). A data-only projection (all ids resolved) + // still gets full field-id / rename matching. FLIP-TIME RESIDUAL: full rename-correctness for SELECT* over + // an evolution table needs reserved meta-column field ids injected into every entry (iceberg row-lineage + // pattern) or dropping meta exposure to mirror legacy -- deferred, owed the flip e2e. + for (HudiColumnHandle handle : requestedColumns) { + if (handle.getFieldId() < 0) { + return Optional.empty(); + } + } ResolvedInternalSchema resolved = resolveTableInternalSchema(schemaResolver, latestAvro); Collection historical = resolved.enableSchemaEvolution ? allHistoricalSchemas(metaClient) @@ -324,10 +340,13 @@ static String buildSchemaEvolutionDict(List requestedColumns, /** * The {@code -1} target/current overlay, keyed off the requested (pruned) columns so its top-level names * equal the BE scan slots (the CI-969249 StructNode invariant). Each requested column's full nested - * structure + stable field id is looked up BY NAME in {@code baseSchema}. A requested column absent from - * {@code baseSchema} (e.g. a {@code _hoodie_*} meta column on a schema-evolved table whose commit-metadata - * schema omits meta fields) falls back to a scalar placeholder carrying the handle's field id ({@code -1} = - * unset). Empty {@code requestedColumns} (count-only scan) falls back to all base top-level fields. + * structure + stable field id is looked up BY NAME in {@code baseSchema}. Empty {@code requestedColumns} + * (count-only scan) falls back to all base top-level fields. + * + *

    The scalar-placeholder fallback (a requested column absent from {@code baseSchema}) is DEFENSIVE only: + * {@link #buildSchemaEvolutionProp} already gates the whole dict OFF when any projected column has an unset + * field id, so in production every column reaching here resolves in {@code baseSchema}. The fallback is kept + * so a direct/test call still produces a complete overlay rather than dropping a scan slot.

    */ static TSchema buildTargetSchema(List requestedColumns, InternalSchema baseSchema) { TSchema tSchema = new TSchema(); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java index d23f2395b88a0f..9a2a92f05835f6 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiColumnFieldIdTest.java @@ -123,10 +123,43 @@ public void unmatchedColumnKeepsUnsetId() { Assertions.assertNotEquals(ConnectorColumn.UNSET_UNIQUE_ID, attached.get(0).getUniqueId()); } + @Test + public void fieldIdsMatchedByNameNotPosition() { + // The load-bearing distinction of C4b: columns come from getTableAvroSchema(true) (which PREPENDS the 5 + // _hoodie_* meta columns) while ids come from an independent InternalSchema (data-only on an evolution + // table), so ids MUST be joined BY NAME, not zipped positionally like legacy. Here the UNMATCHED column is + // FIRST: columns = [_hoodie_commit_time, id, name] against a data-only InternalSchema [id, name]. + // MUTATION (regress to positional zip): _hoodie_commit_time would grab field[0]=id's id and id would grab + // field[1]=name's id -> every column shifts onto the wrong field id (silent BY_FIELD_ID corruption). + Schema metaFirst = parse("{\"type\":\"record\",\"name\":\"t\",\"fields\":[" + + "{\"name\":\"_hoodie_commit_time\",\"type\":[\"null\",\"string\"],\"default\":null}," + + "{\"name\":\"Id\",\"type\":[\"null\",\"int\"],\"default\":null}," + + "{\"name\":\"Name\",\"type\":\"string\"}]}"); + Schema dataOnly = parse("{\"type\":\"record\",\"name\":\"t\",\"fields\":[" + + "{\"name\":\"Id\",\"type\":[\"null\",\"int\"],\"default\":null}," + + "{\"name\":\"Name\",\"type\":\"string\"}]}"); + List columns = HudiConnectorMetadata.avroSchemaToColumns(metaFirst); + InternalSchema dataSchema = AvroInternalSchemaConverter.convert(dataOnly); + Map expected = expectedIds(dataSchema); + + List attached = HudiConnectorMetadata.attachTopLevelFieldIds(columns, dataSchema); + Map byName = new HashMap<>(); + for (ConnectorColumn col : attached) { + byName.put(col.getName(), col.getUniqueId()); + } + + // the unmatched meta column (FIRST) stays UNSET — a positional zip would give it "id"'s field id + Assertions.assertEquals(ConnectorColumn.UNSET_UNIQUE_ID, byName.get("_hoodie_commit_time").intValue()); + // the data columns keep THEIR OWN field id by name (not shifted by one position) + Assertions.assertEquals(expected.get("id"), byName.get("id")); + Assertions.assertEquals(expected.get("name"), byName.get("name")); + } + @Test public void handleCarriesFieldId() { - // getColumnHandles threads col.getUniqueId() onto the handle. MUTATION: hard-code UNSET / drop the arg - // -> getFieldId() != the sourced id -> red. + // Pins only the ctor + getFieldId() round-trip (the field id is carried, not dropped/reordered). The + // getColumnHandles threading of col.getUniqueId() onto the handle needs a live metaClient and is + // covered only by the flip-time e2e, not by this same-loader test. HudiColumnHandle handle = new HudiColumnHandle("c", "int", false, 7); Assertions.assertEquals(7, handle.getFieldId()); } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java index 6078d7a6ae2721..5375ad1d76c95b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java @@ -32,7 +32,6 @@ import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; -import java.util.Locale; import java.util.Map; /** @@ -224,8 +223,10 @@ public void encodeApplyRoundTripsThroughBase64() { Assertions.assertEquals(SCHEMA_ID, decoded.getSchemaId()); Map decodedTop = topFields(decoded); Assertions.assertEquals(5, decodedTop.size()); - Assertions.assertEquals("street", - structChild(decodedTop.get("addr"), "street").getName().toLowerCase(Locale.ROOT)); + // The nested struct child survives serialization with its lowercased name AND its field id (structChild's + // case-sensitive lookup fails if the round-trip dropped it or left it mixed-case; the id assertion proves + // the whole nested TField — not just the name — round-trips). + Assertions.assertEquals(4, structChild(decodedTop.get("addr"), "street").getId()); } @Test @@ -325,6 +326,31 @@ public void dictNonEvolutionEmitsTargetPlusBaseVersion() { Assertions.assertEquals(0L, params.getHistorySchemaInfo().get(1).getSchemaId()); } + @Test + public void dictGatedOffWhenAnyProjectedColumnUnresolved() { + // BE field-id matching is per-FILE, not per-column: a projected column with no field id (e.g. a _hoodie_* + // meta column on a schema-evolved table whose commit-metadata schema omits meta fields) cannot be BY_NAME'd + // on its own, so the WHOLE dict must be suppressed -> BE stays on BY_NAME for the scan (no silent + // const-NULL). The gate returns empty BEFORE touching the metaClient/schemaResolver, so the nulls here are + // never dereferenced. MUTATION: drop the gate -> the dict is built (NPEs on the null resolver here, and at + // runtime would emit a -1 target field with id=-1 that BE reads as const-NULL) -> not empty -> red. + Assertions.assertFalse(HudiSchemaUtils.buildSchemaEvolutionProp( + null, null, null, Arrays.asList(handle("id", 1), handle("_hoodie_commit_time", -1))).isPresent()); + } + + @Test + public void resolveFileSchemaNonEvolutionReturnsBaseWithoutMetaClient() { + // The COMMON (schema.on.read off) per-file schema_id source: the non-evolution branch returns the base + // schema (its schemaId, 0 for convert()) WITHOUT reading the file path or metaClient -> it must equal the + // version-0 history entry the dict emits (self-consistency, else BE "miss schema info"). Passing a null + // metaClient proves the branch never dereferences it. MUTATION: return null / a different schema -> red. + InternalSchema base = internalSchemaWithId(0L); + InternalSchema resolved = HudiSchemaUtils.resolveFileInternalSchema( + "s3://b/t/anyfile.parquet", false, base, null); + Assertions.assertSame(base, resolved); + Assertions.assertEquals(0L, resolved.schemaId()); + } + @Test public void dictEvolutionEmitsTargetPlusAllHistoricalVersions() { // Evolution table: the dict is the -1 target + ONE entry per committed schema version (all-historical = From 72223eecd0813d6f613f489b493540ce8ec1bc6c Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 20:49:40 +0800 Subject: [PATCH 135/334] [doc](catalog) HMS cutover hudi Part 1 review + fixes DONE; next = Part 2 (C5a/C5b time-travel-over-evolved) Per-round HANDOFF update: the consolidated adversarial review of steady-state read evolution (C4a-C4d) is done (wf_33882086-fbb; 6 dimensions + per-finding adversarial refute; 8 confirmed of 12 raw), and the fixes landed (ad8aa87bd2d). Records the major meta-column BY_FIELD_ID gate (BE field-id mode is per-file, so any unresolved projected column now gates the whole dict off -> BE BY_NAME for the scan, no silent const-NULL), the planScan resilience isolation, the test-intent hardening, and the owed flip-time e2e residual (full SELECT* rename-correctness on an evolution table needs reserved meta-column ids or dropping meta exposure). Part 1 is fully done; next work line = Part 2 (C5a getTableSchema@instant + C5b scan-side @instant). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 1d6f7ee452a906..12c3800f72aff9 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。下一步 = 对整块 Part 1 做统一对抗复审（设计 §9 签字要求，Part 2 前置）→ 之后 Part 2 = C5a（getTableSchema@instant）+ C5b（扫描侧@instant）时间旅行叠加演进。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。**下一步 = Part 2 = C5a（`getTableSchema`@instant，闭合 C5a 欠的 `HudiMetaClientExecutor` 包裹）+ C5b（扫描侧 MOR/JNI 列表@instant + `-1` 条按 pinned schema 重解）时间旅行叠加演进。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 From 03770821029dfaeceefcc8c3ad4356c62dd8ca6f Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 21:23:20 +0800 Subject: [PATCH 136/334] [feat](catalog) HMS cutover hudi HD-C5a: schema-at-instant getTableSchema override (Part 2 first step) Dormant. Post-flip a hudi-on-HMS table is served through the generic PluginDrivenScanNode path; C5a supplies the column list AS OF the pinned instant for a FOR TIME AS OF read under schema evolution, closing HD-C2 residual #1 (the SPI default previously returned the latest schema). HudiConnectorMetadata: - New 3-arg getTableSchema(session, handle, ConnectorMvccSnapshot) override keyed off HudiTableHandle.getQueryInstant() (the applySnapshot-stamped pin), NOT snapshot.getSchemaId() (which stays -1 for hudi). fe-core passes the post-applySnapshot handle (PluginDrivenMvccExternalTable), so the instant is already on it. A null instant (plain read, or a non-FOR-TIME pin such as @incr) resolves via the SAME shared build method with a null instant, so the latest and at-instant paths cannot drift. - getSchemaFromMetaClient(basePath) -> getSchemaFromMetaClient(basePath, queryInstant), now WRAPPED in HudiMetaClientExecutor.execute (plugin UGI doAs + TCCL pin), closing the previously-unwrapped auth/TCCL gap (getTableSchema can be called off the pinned scan thread). At-instant branch is byte-faithful to legacy HiveMetaStoreClientHelper.getHudiTableSchema + initHudiSchema: reloadActiveTimeline + getTableInternalSchemaFromCommitMetadata(instant); present (schema.on.read on) -> columnsFromInternalSchema (convert InternalSchema->avro, avroSchemaToColumns, attach stable field ids); absent (evolution off) -> latest fallback = byte-equivalent for a non-evolution table (design D3). - Uses HudiScanPlanProvider.buildMetaClient (byte-identical to the old inline build). Hive gateway 3-arg delegation already shipped; no hive change. Zero fe-core import, zero new connector-api SPI. DORMANT (hms not in SPI_READY_TYPES, hudi handles not yet diverted). The at-instant read itself is e2e; HudiSchemaAtInstantTest (3, same-loader) locks the instant-threading routing (2-arg latest, 3-arg no-pin -> latest, 3-arg pin -> at-instant, keying off the handle not the snapshot). 4-dimension adversarial review + refute: 0 confirmed defects. fe-connector-hudi 156 tests + checkstyle 0 + import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiConnectorMetadata.java | 120 ++++++++++++++---- .../hudi/HudiSchemaAtInstantTest.java | 114 +++++++++++++++++ 2 files changed, 210 insertions(+), 24 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaAtInstantTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 08686e7f887be4..4746324e33b670 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -46,8 +46,10 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hudi.common.table.HoodieTableMetaClient; import org.apache.hudi.common.table.TableSchemaResolver; +import org.apache.hudi.common.util.Option; import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.internal.schema.Types; +import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -267,12 +269,41 @@ public Optional> applyFilter( @Override public ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { + // Latest schema (no time-travel pin). Shares the single build path with the at-instant overload below + // (null instant = latest) so the two can never drift. + return buildTableSchema((HudiTableHandle) handle, null); + } + + /** + * Returns the schema AS OF the pinned instant for a {@code FOR TIME AS OF} read under schema evolution + * (HD-C5a), closing HD-C2 residual #1 (the SPI default previously returned the LATEST schema, ignoring the + * pin). Keys off {@link HudiTableHandle#getQueryInstant()} — the instant {@link #applySnapshot} stamps + * onto the handle — NOT {@code snapshot.getSchemaId()}, which stays {@code -1} for hudi (hudi pins by + * instant, not by a numeric schema id, so the generic schema-id carrier is inert here; fe-core passes the + * post-{@code applySnapshot} handle, so the instant is already on it). A {@code null} instant (a plain read, + * or a non-{@code FOR TIME} pin such as {@code @incr}, whose {@code loadSnapshot} branch lists the LATEST + * schema itself and never reaches here) resolves via the SAME shared build method with a null instant, so + * latest and at-instant cannot drift. Hive gateway delegation for this 3-arg overload already shipped + * ({@code HiveConnectorMetadata}), so no hive change is needed. + */ + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { HudiTableHandle hudiHandle = (HudiTableHandle) handle; + return buildTableSchema(hudiHandle, hudiHandle.getQueryInstant()); + } + + /** + * Single build path for {@link #getTableSchema}: reads the columns from the Hudi metaClient AS OF + * {@code queryInstant} ({@code null} = latest) and wraps them in a {@link ConnectorTableSchema}. Shared by + * the latest (2-arg) and at-instant (3-arg) entry points so they cannot diverge. + */ + private ConnectorTableSchema buildTableSchema(HudiTableHandle hudiHandle, String queryInstant) { String basePath = hudiHandle.getBasePath(); List columns; if (basePath != null && !basePath.isEmpty()) { - columns = getSchemaFromMetaClient(basePath); + columns = getSchemaFromMetaClient(basePath, queryInstant); } else { columns = getSchemaFromHms(hudiHandle.getDbName(), hudiHandle.getTableName()); } @@ -660,32 +691,74 @@ public TTableDescriptor buildTableDescriptor(ConnectorSession session, // ========== Internal helpers ========== /** - * Read schema from HoodieTableMetaClient's latest Avro schema. - * This is the authoritative schema for Hudi tables. + * Reads the columns from the Hudi metaClient AS OF {@code queryInstant} ({@code null} = the latest Avro + * schema, the authoritative schema for a plain read). The whole metaClient touch runs under + * {@link HudiMetaClientExecutor#execute} (plugin UGI {@code doAs} + TCCL pin) — HD-C5a closes the + * previously-unwrapped auth/TCCL gap, because {@code getTableSchema} can be called off the TCCL-pinned scan + * thread (e.g. catalog metadata load / MTMV refresh). + * + *

    At-instant ({@code queryInstant != null}, {@code FOR TIME AS OF}). Byte-faithful to legacy + * {@code HiveMetaStoreClientHelper.getHudiTableSchema} + {@code HMSExternalTable.initHudiSchema}: reload the + * active timeline, then {@code getTableInternalSchemaFromCommitMetadata(instant)}. When present + * ({@code hoodie.schema.on.read.enable} is ON) the columns/ids come from that AT-INSTANT + * {@link InternalSchema} (stable ids across renames), so a renamed column shows its historical name and BE + * matches its old files BY FIELD ID. When absent (evolution off) it falls through to the latest path — + * byte-equivalent for a non-evolution table whose schema never changed (legacy's latest-fallback, design + * decision D3).

    + * + *

    Package-private (not static) so a same-loader test can override it to assert the {@code queryInstant} + * is threaded from the handle without a live metaClient (the actual at-instant read is e2e).

    */ - private List getSchemaFromMetaClient(String basePath) { + List getSchemaFromMetaClient(String basePath, String queryInstant) { try { - Configuration conf = buildHadoopConf(); - HoodieTableMetaClient metaClient = HoodieTableMetaClient.builder() - .setConf(new org.apache.hudi.storage.hadoop.HadoopStorageConfiguration(conf)) - .setBasePath(basePath) - .build(); - TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); - // include the 5 `_hoodie_*` meta columns (byte-faithful to legacy getHudiTableSchema, which passes - // `true` unconditionally at HiveMetaStoreClientHelper). The no-arg overload only includes them when - // hoodie.populate.meta.fields defaults true, so the explicit `true` (a) restores legacy SELECT * - // / DESCRIBE parity even for a populate.meta.fields=false table and (b) guarantees `_hoodie_commit_time` - // is a visible, name-bindable output column for the synthetic incremental row filter to reference. - Schema avroSchema = schemaResolver.getTableAvroSchema(true); - List columns = avroSchemaToColumns(avroSchema); - return attachHudiFieldIds(schemaResolver, avroSchema, columns); + return metaClientExecutor.execute(() -> { + HoodieTableMetaClient metaClient = + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), basePath); + TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); + if (queryInstant != null) { + // Reload so a recently-committed instant's schema file is readable, not a stale cached one + // (legacy getHudiTableSchema reloads for exactly this reason). + metaClient.reloadActiveTimeline(); + Option atInstant = + schemaResolver.getTableInternalSchemaFromCommitMetadata(queryInstant); + if (atInstant.isPresent()) { + return columnsFromInternalSchema(atInstant.get()); + } + // schema.on.read off -> no commit-metadata InternalSchema -> latest fallback (D3): + // byte-equivalent for a non-evolution table (its schema never changed). + } + // Latest schema. Include the 5 `_hoodie_*` meta columns (byte-faithful to legacy + // getHudiTableSchema, which passes `true` unconditionally). The explicit `true` (a) restores + // legacy SELECT * / DESCRIBE parity even for a populate.meta.fields=false table and (b) keeps + // `_hoodie_commit_time` a visible, name-bindable column for the synthetic incremental row filter. + Schema avroSchema = schemaResolver.getTableAvroSchema(true); + List columns = avroSchemaToColumns(avroSchema); + return attachHudiFieldIds(schemaResolver, avroSchema, columns); + }); } catch (Exception e) { - LOG.warn("Failed to get schema from Hudi MetaClient for path '{}': {}", - basePath, e.getMessage()); + // Pass the throwable (not e.getMessage()) so the full cause chain + stack survives the + // HudiMetaClientExecutor.execute DorisConnectorException wrapper (whose fixed message would otherwise + // mask WHY this best-effort read degraded to an empty column list / BY_NAME). + LOG.warn("Failed to get schema from Hudi MetaClient for path '{}' (instant={})", + basePath, queryInstant, e); return Collections.emptyList(); } } + /** + * Builds the column list from an AT-INSTANT {@link InternalSchema} (schema-on-read time travel). Mirror of + * legacy {@code HMSExternalTable.initHudiSchema}: convert the InternalSchema to Avro to derive the column + * names/types AT the instant, then attach each top-level field id from the SAME InternalSchema (stable + * across renames). The record name passed to {@code convert} is cosmetic (only the ROOT record is named; the + * derived columns come from its fields), so a constant is used. Field-id resolution and lowercasing match + * the latest path exactly (shared {@link #attachTopLevelFieldIds}). + */ + private List columnsFromInternalSchema(InternalSchema internalSchema) { + Schema avroSchema = AvroInternalSchemaConverter.convert(internalSchema, "hudi_table"); + List columns = avroSchemaToColumns(avroSchema); + return attachTopLevelFieldIds(columns, internalSchema); + } + /** * Resolve the mode-aware InternalSchema for the LATEST commit and attach each column's top-level field id * (HD-C4b). Mirror of legacy {@code HiveMetaStoreClientHelper.getHudiTableSchema}: when {@code @@ -701,10 +774,9 @@ private List getSchemaFromMetaClient(String basePath) { * *

    Unlike legacy (which sources columns AND ids from the same InternalSchema and zips positionally), the * connector keeps its independent {@code getTableAvroSchema(true)} column source (preserving the shipped - * meta-column-inclusive schema), so ids are matched BY NAME in {@link #attachTopLevelFieldIds}. NOTE(C5a): - * this metaClient touch is not yet wrapped in {@code HudiMetaClientExecutor.execute} — closing the existing - * unwrapped {@code getSchemaFromMetaClient} auth/TCCL gap is deferred to the at-instant step (C5a), which - * restructures this whole read.

    + * meta-column-inclusive schema), so ids are matched BY NAME in {@link #attachTopLevelFieldIds}. This runs + * inside the {@link HudiMetaClientExecutor#execute} wrapper that {@link #getSchemaFromMetaClient} now + * establishes (HD-C5a closed the previously-unwrapped auth/TCCL gap).

    */ private List attachHudiFieldIds(TableSchemaResolver schemaResolver, Schema latestAvro, List columns) { diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaAtInstantTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaAtInstantTest.java new file mode 100644 index 00000000000000..09f3870377a712 --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaAtInstantTest.java @@ -0,0 +1,114 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorTableSchema; +import org.apache.doris.connector.api.ConnectorType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; + +/** + * Tests the routing of the schema-at-instant surface (HD-C5a): the 3-arg + * {@code getTableSchema(session, handle, snapshot)} override must resolve the schema AS OF the instant that + * {@code applySnapshot} stamped onto the handle for a {@code FOR TIME AS OF} read, while a plain read (or a + * non-{@code FOR TIME} pin) resolves the LATEST schema. Each assertion pins WHY the behavior matters: + *
      + *
    • the 2-arg entry point always resolves LATEST (no time-travel pin exists), so a plain read stays + * byte-identical to before this step;
      • + *
    • a handle WITHOUT a {@code queryInstant} threads a {@code null} instant = LATEST — the shared build + * path means a non-{@code FOR TIME} 3-arg call cannot drift from the 2-arg latest path;
      • + *
    • a handle WITH a {@code queryInstant} threads exactly that instant into the metaClient read — proving + * the override keys off {@link HudiTableHandle#getQueryInstant()} (the {@code applySnapshot}-stamped + * pin) and NOT {@code snapshot.getSchemaId()} (which stays {@code -1} for hudi; the snapshot is + * deliberately passed as {@code null} here so a mutant that keyed off it would surface latest, not the + * instant).
      • + *
    + * + *

    The actual at-instant metaClient read is e2e (a live schema-evolved Hudi table); this same-loader unit + * locks only the instant-threading routing by overriding the {@link HudiConnectorMetadata#getSchemaFromMetaClient} + * seam, so no live metaClient / plugin auth is needed.

    + */ +public class HudiSchemaAtInstantTest { + + /** + * Recording fake: overrides the metaClient schema seam to capture the {@code queryInstant} the build path + * threads down from the handle (no live metaClient). {@code null} = the LATEST path. + */ + private static final class RecordingMetadata extends HudiConnectorMetadata { + // Distinct from both null (= latest) and any real instant, so an un-invoked seam is detectable. + String lastInstant = "SEAM-NOT-CALLED"; + int calls; + + RecordingMetadata() { + super(null, Collections.emptyMap(), null); + } + + @Override + List getSchemaFromMetaClient(String basePath, String queryInstant) { + this.lastInstant = queryInstant; + this.calls++; + return Collections.singletonList( + new ConnectorColumn("c", ConnectorType.of("INT"), "", true, null)); + } + } + + private static HudiTableHandle handle(String queryInstant) { + HudiTableHandle.Builder b = new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE"); + if (queryInstant != null) { + b.queryInstant(queryInstant); + } + return b.build(); + } + + @Test + void twoArgGetTableSchemaAlwaysResolvesLatest() { + RecordingMetadata m = new RecordingMetadata(); + ConnectorTableSchema schema = m.getTableSchema(null, handle(null)); + Assertions.assertNull(m.lastInstant, "2-arg getTableSchema must resolve the LATEST schema (null instant)"); + Assertions.assertEquals(1, m.calls, "the metaClient seam must be reached exactly once"); + Assertions.assertEquals("t", schema.getTableName()); + Assertions.assertEquals("HUDI", schema.getTableFormatType()); + } + + @Test + void threeArgWithoutPinResolvesLatest() { + RecordingMetadata m = new RecordingMetadata(); + // No queryInstant on the handle (plain read / @incr pin): the shared build path threads a null instant. + m.getTableSchema(null, handle(null), null); + Assertions.assertNull(m.lastInstant, + "a handle without a queryInstant pin must resolve the LATEST schema, not an at-instant one"); + Assertions.assertEquals(1, m.calls); + } + + @Test + void threeArgThreadsPinnedInstantFromHandle() { + RecordingMetadata m = new RecordingMetadata(); + // FOR TIME AS OF: applySnapshot has stamped queryInstant onto the handle. getTableSchema must resolve AT + // that instant. The snapshot arg is null on purpose: keying off it (schemaId) instead of the handle + // would surface latest here, failing this assertion. + m.getTableSchema(null, handle("20240101120000"), null); + Assertions.assertEquals("20240101120000", m.lastInstant, + "the handle's queryInstant must be threaded to the metaClient schema read"); + Assertions.assertEquals(1, m.calls); + } +} From f1fd768f3715d660bf92915a178fe41f357decac Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 21:24:58 +0800 Subject: [PATCH 137/334] [doc](catalog) HMS cutover hudi HD-C5a DONE; next = C5b (scan-side at-instant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Part 2 first step (schema-at-instant getTableSchema override) done and dormant (af9a3c1a004); 4-dimension adversarial review 0 confirmed defects. Updated the top status line + the hudi ⭐ log with the C5a entry and its e2e residuals; pointed next = C5b. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 12c3800f72aff9..f96c4f10afddc3 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。**下一步 = Part 2 = C5a（`getTableSchema`@instant，闭合 C5a 欠的 `HudiMetaClientExecutor` 包裹）+ C5b（扫描侧 MOR/JNI 列表@instant + `-1` 条按 pinned schema 重解）时间旅行叠加演进。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**下一步 = C5b（扫描侧 MOR/JNI `column_names`/`column_types`@instant + 字典 `-1` 条按 pinned InternalSchema 重解〔句柄 id 是 latest-keyed，改名列须 iceberg full-pinned 式重解〕；`current_schema_id` 保 -1；off-scan-thread timeline touch 全进 `HudiMetaClientExecutor.execute`；deps C4c/C4d/C5a）。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -98,7 +98,8 @@ - **✅ HD-C4a DONE（`d29bb52173d`，休眠，Part 1 首步）= 纯 `HudiSchemaUtils` 转换器**（新连接器内部类，`fe-connector-hudi`）。自足 `InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` 构建器 + base64 `encode`/`applySchemaEvolution` 往返 on 抛弃型 `TFileScanRangeParams`（f25 `current_schema_id` + f26 `history_schema_info`）。**移植 legacy `HudiUtils.getSchemaInfo(InternalSchema/List/Types.Field)` 两处有意偏离**（均由 mutation-tight 测钉）：①**逐层小写**（`Locale.ROOT`，镜像 `IcebergSchemaUtils.buildField`；legacy 不小写）——同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader，其 `StructNode` 按小写 Doris slot 名查，混大小写嵌套名→`std::out_of_range`→整库 SIGABRT；②**scalar `TColumnType`=STRING placeholder**（BE field-id 路只用 `type.type` 作 nested-vs-scalar 判别）——丢 legacy `fromAvroHudiTypeToDorisType` 全型表。`id`+`name`+`is_optional` 每层保真（legacy 平价）。`encode` serialize 错 fail-loud（catch `Exception|LinkageError` 防 thrift 类加载器裂）；`applySchemaEvolution` null/empty no-op、corrupt fail-loud（镜像 iceberg/paimon）。**模板=paimon（后续 C4c/C4d 复用 `buildSchemaInfo`+`encode` 发 -1 目标条 + 每引用版本历史 + 每-split schema_id），小写=iceberg 每层。wired 进空、零 fe-core import、零新连接器-api SPI。DORMANT**（hms 未进 `SPI_READY_TYPES`、hudi 句柄未分流）。`HudiSchemaUtilsTest` 11 全 same-loader（手搭混大小写嵌套 struct 子字段的 `InternalSchema` 断言每层小写/嵌套 struct·array·map 标签+每层 field-id/STRING scalar/InternalSchema-keyed 与显式 -1 schema id/base64 往返）。全绿（hudi 模块 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`）**。 - **✅ HD-C4b DONE（`d54dc2d3ac5`，休眠，Part 1 第二步）= field-id 上 `HudiColumnHandle`**（源自 mode-aware InternalSchema）。①`HudiColumnHandle` 加 `int fieldId`（ctor+`getFieldId()`+toString）——**不入 equals/hashCode**（身份保 name+typeName，镜像 `IcebergColumnHandle` 按名不折 id）。②`HudiConnectorMetadata.getSchemaFromMetaClient` 解析 mode-aware InternalSchema：no-arg `TableSchemaResolver.getTableInternalSchemaFromCommitMetadata()` 钉**最新 commit**（schema.on.read 开→commit-metadata InternalSchema〔改名跨版本**稳定 id**〕；关→`AvroInternalSchemaConverter.convert(latest avro)` 位置 id）——镜像 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`（steady-state/无时间旅行 pin，at-instant 变体留 C5a）；解析失败降级 columns 不带 id（UNSET→BE BY_NAME）而非丢整 schema。③`getColumnHandles` 穿 `col.getUniqueId()` 上 handle（镜像 iceberg）。**与 legacy 有意偏离（已文档化）**：legacy 列源+id 源同一 InternalSchema 位置 zip；连接器保 `getTableAvroSchema(true)` 独立列源（保住已发的含 meta 列 schema），故 `attachTopLevelFieldIds` **按名匹配**——对齐列集等价、且当 commit-metadata InternalSchema 缺某列（如 `_hoodie_*` meta）时该列留 UNSET→BY_NAME（正确，meta 列从不改名）。只顶层 id 上 handle；嵌套 field-id 由 InternalSchema 经 `HudiSchemaUtils` 直发（C4a）。**⚠残留**：`NOTE(C5a)` 此 metaClient touch 尚未 `HudiMetaClientExecutor.execute` 包裹——closing 既有 `getSchemaFromMetaClient` 未包裹 auth/TCCL gap 留 C5a（它重构此读）；evolution-mode commit-metadata id 源仅 e2e 可证（单测覆盖非演进 `convert(avro)` 路）。DORMANT（hudi 故意不在 `SPI_READY_TYPES`、句柄未分流→无 live 目录达 `getColumnHandles`）。`HudiColumnFieldIdTest` 4 same-loader（按名取 id/未命中 UNSET/handle 载 id/id 不入身份）。全绿（hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）**。 - **✅ HD-C4c DONE（`ac3222bc526`，休眠，Part 1 第三步）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**。给每个原生 base 文件钉写它的那次 commit 的 InternalSchema 版本号（field 12），BE 原生 field-id 读取器据此在字典里查该文件的列并按 id 对齐当前表（改名安全）。**仅原生分支**：JNI 合并读取器读 `column_names/types`@instant、不消费 schema_id。①`HudiSchemaUtils` 两共享静态 resolver：`resolveTableInternalSchema`（mode-aware base：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 版本 0，+evolution 标志；C4b 的 `attachHudiFieldIds` 重构复用之=id 源单一真源）+`resolveFileInternalSchema`（每文件版本：evolution 开→`FSUtils.getCommitTime(fileName)`+`InternalSchemaCache.searchSchemaAndCache`；关→base 版本 0）——port legacy `HudiScanNode.setHudiParams`。②`HudiScanRange.Builder.schemaId`（可空）→ctor 仅设时放 `hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** `fileDesc.setSchemaId`（镜像 `PaimonScanRange`），JNI 臂永不；缺→BE BY_NAME。③`planScan` 解析一次 base schema、建每文件 schema_id resolver（base 文件路径→版本）；`collectCowSplits`（COW base 恒原生）+`buildMorRange` 原生无 log 片钉之；@incr 路传 null（@incr 列最新 schema、无每文件字典）。base-schema 未解析→null resolver→不钉；每文件解析失败→log 跳过该文件→均降 BY_NAME 不崩扫描。**⚠编译期坑**：`searchSchemaAndCache` 编译classpath 是 **2-arg** `(long, metaClient)`（与 legacy fe-core 同），非默认 m2 里 SELECTDB-0.14.1 jar 的 3-arg（javap 看错 jar）——已用 2-arg。**排序（关键）**：schema_id present 而字典缺→BE 不入 field-id 路→**BY_NAME=安全前缀**；字典步（C4d）须在其后，dict-present+schema_id-unset 才是禁的 `current==file==-1` v1-ConstNode 退化。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。schema_id **值 e2e**，只 stamping 机制 same-loader 测（`HudiScanRangeTest`+3：原生 set f12 / JNI 即便设了也不 set / 原生无 id 留 unset；`HudiIncrementalPlanScanTest` buildMorRange 断言原生 downgrade 载 resolver id、force-jni 片不载）。全绿（hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级字典发射，与 C4c 背靠背；C4c 安全前缀已落）**。 - - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；下一步 = 整块统一对抗复审（Part 2 前置）。** + - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；统一对抗复审 DONE（`wf_33882086-fbb`）+修 `ad8aa87bd2d`（见顶部 🎯）；下一步 = Part 2 C5a。** + - **✅ HD-C5a DONE（`af9a3c1a004`，休眠，Part 2 首步）= schema-at-instant 列表（3-arg `getTableSchema` 覆写）**。post-flip 通用路径下给 `FOR TIME AS OF` 演进读供**钉时刻**的列表，闭合 HD-C2 残留 #1（SPI 默认曾返最新、忽略 pin）。①新 3-arg `getTableSchema(session,handle,ConnectorMvccSnapshot)` 覆写 keyed off `HudiTableHandle.getQueryInstant()`（`applySnapshot` 钉的 pin），**非** `snapshot.getSchemaId()`（hudi 恒 -1；fe-core `PluginDrivenMvccExternalTable:375-376` 传 post-applySnapshot 句柄故 instant 已在其上）；null instant（plain / 非-FOR-TIME pin 如 @incr，其 loadSnapshot 分支自列最新 schema、根本不到此）经**同一** shared `buildTableSchema(handle,instant)` 走 latest→latest/at-instant 不发散。②`getSchemaFromMetaClient(basePath)`→`getSchemaFromMetaClient(basePath,queryInstant)`，**包进 `metaClientExecutor.execute`**（plugin UGI doAs+TCCL pin）闭合 C4b 欠的未包裹 auth/TCCL gap（getTableSchema 可跑在非 pin 的 scan 线程外，如 catalog metadata load）；at-instant 臂字节忠实 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`+`HMSExternalTable.initHudiSchema`：`reloadActiveTimeline()`+`getTableInternalSchemaFromCommitMetadata(instant)`→present（schema.on.read 开）经 `columnsFromInternalSchema`（`AvroInternalSchemaConverter.convert(InternalSchema,name)`→avro→`avroSchemaToColumns`→`attachTopLevelFieldIds` 稳定 id）→改名列显历史名、BE 按 field id 匹老文件；absent（演进关）落 latest fallback＝非演进表字节等价（D3）。③用 `HudiScanPlanProvider.buildMetaClient`（与旧 inline build 字节等价）；hive 3-arg 委派已发（`HiveConnectorMetadata:1071`）无须改；连接器编 unshaded hudi-common 故 `convert` 回 unshaded `org.apache.avro.Schema` 与 `avroSchemaToColumns` 相容。零 fe-core import、零新连接器-api SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、句柄未分流；live iceberg/paimon 不走 hudi getTableSchema）。at-instant **读**本身 e2e；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading（2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〔snapshot 传 null 证之〕）。**4 维对抗复审 `wf_703ea26e-49f`（legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent）+逐条 refute：0 确认缺陷**（1 nit＝wrapping 后 LOG.warn 只印 wrapper message refuted〔旧码亦只印 `e.getMessage()` 非本步引入〕；顺带把 `e.getMessage()`→传 throwable 补全 cause 链+栈）。全绿（hudi 156 测〔+3〕 + checkstyle 0 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进列表（改名/调序显历史名、非演进 latest 等价、meta 列集与 legacy 一致）只 e2e 可证（单测覆盖 routing）；schema-on-read commit-metadata InternalSchema 是否含 `_hoodie_*` meta 列＝随 legacy 定、e2e 核。**下一步 = C5b（扫描侧 MOR/JNI 列表@instant + 字典 `-1` 条按 pinned schema 重解）。** - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From e87a31cba1554e6395532481302cdbd9ad6d7cee Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 22:08:23 +0800 Subject: [PATCH 138/334] [feat](catalog) HMS cutover hudi HD-C5b: scan-side schema-at-instant (JNI col list + native -1 overlay) Part 2 (time-travel-over-evolved) final step. Under FOR TIME AS OF over a schema-on-read hudi table, both scan-side schema loci must resolve AT the pinned instant instead of latest, in lockstep: (i) planScan JNI (MOR-realtime) column_names/column_types -> the schema AS OF queryInstant (legacy HudiScanNode:222-224), via HudiSchemaUtils .resolveJniColumnSchema (pure choice chooseJniSchema). A plain read (queryInstant == null) short-circuits before any timeline reload -> the latest schema, byte-identical to before. Non-evolution (no commit-metadata InternalSchema at the instant) -> latest == at-instant (D3). (ii) getScanNodeProperties native -1 overlay -> re-resolved from the FULL pinned InternalSchema (buildSchemaEvolutionDictAtInstant, the empty-requested path over the pinned schema), NOT the requested column handles. The handles are latest-keyed (getColumnHandles runs before the MVCC pin), so a column renamed after the pin is absent from them under its pinned name; building the overlay from them would emit a SUBSET missing that scan slot, and BE's field-id reader (by_table_field_id StructNode.children.at) SIGABRTs on a scan slot absent from the overlay. The full pinned schema is a SUPERSET of the scan slots (BE looks each slot up by name and ignores extras) -> safe. current_schema_id stays -1; history is all committed versions (Option B). No meta column can reach the pinned path: the at-instant bound schema (HD-C5a, from the InternalSchema) has no _hoodie_* columns. The per-file schema_id resolver keeps its LATEST base (pin-independent, classifies each file's own written version). Zero fe-core, zero new connector-api SPI. DORMANT (hms not in SPI_READY_TYPES, hudi handles not diverted). 4-dim clean-room adversarial review (wf_ec6d47e3-37e: legacy-parity/at-instant, BE-contract/SIGABRT, no-regression/dormancy, test-intent) + adversarial refute: 0 production defects; 1 confirmed-minor test-comment honesty fix applied (atInstantOverlay test guards the pure full-pinned assembly, not the provider routing, which needs a live metaClient and is e2e-owed); 2 refuted. BE contract verified directly against table_schema_change_helper.{h,cpp} (superset-required) and format_v2/table_reader.cpp (superset-tolerant). Tests: HudiSchemaUtilsTest +3 (JNI no-pin no-op / chooseJniSchema pinned names / full-pinned overlay historical names). hudi module 159 tests, checkstyle 0, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiScanPlanProvider.java | 52 ++++++++++---- .../doris/connector/hudi/HudiSchemaUtils.java | 66 +++++++++++++++++ .../connector/hudi/HudiSchemaUtilsTest.java | 71 +++++++++++++++++++ 3 files changed, 177 insertions(+), 12 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 21f418974fba5f..720427a9d31c5c 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -44,6 +44,7 @@ import org.apache.hudi.common.table.view.FileSystemViewManager; import org.apache.hudi.common.table.view.HoodieTableFileSystemView; import org.apache.hudi.common.util.Option; +import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.metadata.HoodieTableMetadata; import org.apache.hudi.metadata.HoodieTableMetadataUtil; import org.apache.hudi.storage.StoragePath; @@ -77,9 +78,11 @@ * * * - *

    Scope: snapshot reads, {@code FOR TIME AS OF} time travel (a pinned {@code queryInstant}), and {@code @incr} - * incremental FILE selection (a resolved {@code (begin, end]} window). Incremental row-level filtering to the - * window (a synthetic {@code _hoodie_commit_time} predicate) and schema evolution are deferred to later steps.

    + *

    Scope: snapshot reads, {@code FOR TIME AS OF} time travel (a pinned {@code queryInstant}), {@code @incr} + * incremental FILE selection (a resolved {@code (begin, end]} window), and schema evolution (per-file {@code + * schema_id} + the scan-level {@code -1}/history dictionary for BE's field-id matching, resolved AT the pinned + * instant under time travel). Incremental row-level filtering to the window (a synthetic {@code + * _hoodie_commit_time} predicate) is an fe-core analysis-time step, not this provider's concern.

    */ public class HudiScanPlanProvider implements ConnectorScanPlanProvider { @@ -163,13 +166,20 @@ public List planScan( Schema avroSchema = null; try { TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); - // include the 5 `_hoodie_*` meta columns (explicit `true` = legacy parity, in lockstep with - // HudiConnectorMetadata.getSchemaFromMetaClient) so the JNI reader's column list matches the - // exposed schema; byte-identical for the common populate.meta.fields=true table. + // The LATEST table Avro schema (5 `_hoodie_*` meta columns included, explicit `true` = legacy + // parity): the base for the per-file schema_id resolver below, which classifies each file's OWN + // written version (independent of the query instant). avroSchema = schemaResolver.getTableAvroSchema(true); - columnNames = avroSchema.getFields().stream() + // HD-C5b: the JNI (MOR-realtime) reader's column list must be the schema AT the pinned instant for a + // FOR TIME AS OF read over a schema-on-read table (legacy HudiScanNode:222-224), kept in lockstep + // with the getScanNodeProperties native -1 overlay so a renamed column shows its historical name to + // the merge reader. A plain read (queryInstant == null) or a non-evolution table (no commit-metadata + // InternalSchema at the instant, D3) keeps the latest schema — byte-identical. + Schema jniSchema = HudiSchemaUtils.resolveJniColumnSchema( + schemaResolver, metaClient, avroSchema, hudiHandle.getQueryInstant()); + columnNames = jniSchema.getFields().stream() .map(Schema.Field::name).collect(Collectors.toList()); - columnTypes = avroSchema.getFields().stream() + columnTypes = jniSchema.getFields().stream() .map(f -> HudiTypeMapping.toHiveTypeString(f.schema())) .collect(Collectors.toList()); } catch (Exception e) { @@ -322,10 +332,28 @@ public Map getScanNodeProperties( try { HoodieTableMetaClient metaClient = buildMetaClient(buildHadoopConf(), hudiHandle.getBasePath()); TableSchemaResolver schemaResolver = new TableSchemaResolver(metaClient); - Schema latestAvro = schemaResolver.getTableAvroSchema(true); - HudiSchemaUtils.buildSchemaEvolutionProp(metaClient, schemaResolver, latestAvro, - castHudiColumns(columns)) - .ifPresent(v -> props.put(SCHEMA_EVOLUTION_PROP, v)); + // HD-C5b: FOR TIME AS OF over a schema-on-read table -> re-resolve the native -1 overlay from the + // FULL schema AT the pinned instant. The requested column HANDLES are latest-keyed + // (getColumnHandles runs before the MVCC pin), so a column renamed after the pin is absent from + // them under its pinned name; building the overlay from them would drop that BE scan slot (BE's + // field-id reader SIGABRTs on a scan slot missing from the overlay). A plain read (no pin) or a + // non-evolution FOR TIME AS OF (latest == at-instant, D3) uses the steady-state dict keyed off the + // requested columns. NOTE: no meta column can reach the pinned path — the at-instant bound schema + // (HD-C5a, from the InternalSchema) has no `_hoodie_*` columns, so the query cannot project one. + String pin = hudiHandle.getQueryInstant(); + Optional pinnedSchema = pin == null + ? Optional.empty() + : HudiSchemaUtils.resolveInternalSchemaAtInstant(schemaResolver, metaClient, pin); + Optional dict; + if (pinnedSchema.isPresent()) { + dict = Optional.of( + HudiSchemaUtils.buildSchemaEvolutionDictAtInstant(metaClient, pinnedSchema.get())); + } else { + Schema latestAvro = schemaResolver.getTableAvroSchema(true); + dict = HudiSchemaUtils.buildSchemaEvolutionProp(metaClient, schemaResolver, latestAvro, + castHudiColumns(columns)); + } + dict.ifPresent(v -> props.put(SCHEMA_EVOLUTION_PROP, v)); } catch (Exception e) { LOG.warn("Failed to build Hudi schema-evolution dict for {}.{}; native reads fall back to BY_NAME: {}", hudiHandle.getDbName(), hudiHandle.getTableName(), e.getMessage()); diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java index 66f3f4b1377577..a1665d154fbc70 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiSchemaUtils.java @@ -278,6 +278,54 @@ static InternalSchema resolveFileInternalSchema(String filePath, boolean enableS return InternalSchemaCache.searchSchemaAndCache(commitTime, metaClient); } + // ========== HD-C5b: table schema AT the pinned instant (FOR TIME AS OF over a schema-on-read table) ========== + + /** + * Resolve the table {@link InternalSchema} AS OF {@code queryInstant} (a {@code FOR TIME AS OF} pin), or + * {@link Optional#empty()} when the table is not schema-on-read (no commit-metadata {@link InternalSchema} at + * the instant) — the caller then uses the LATEST schema, which for a non-evolution table is byte-equivalent + * (its schema never changed; design decision D3). Mirror of legacy {@code + * HiveMetaStoreClientHelper.getHudiTableSchema} at-instant path + HD-C5a's {@code getSchemaFromMetaClient}: + * reload the active timeline first (so a just-committed instant's schema file is readable, not a stale cached + * one) then {@code getTableInternalSchemaFromCommitMetadata(instant)}. Runs on the caller's TCCL-pinned scan + * thread ({@code planScan} / {@code getScanNodeProperties}); the off-scan-thread {@code getTableSchema} + * at-instant read is wrapped by HD-C5a's {@code HudiMetaClientExecutor.execute} separately. + */ + static Optional resolveInternalSchemaAtInstant(TableSchemaResolver schemaResolver, + HoodieTableMetaClient metaClient, String queryInstant) { + metaClient.reloadActiveTimeline(); + Option atInstant = schemaResolver.getTableInternalSchemaFromCommitMetadata(queryInstant); + return atInstant.isPresent() ? Optional.of(atInstant.get()) : Optional.empty(); + } + + /** + * The Avro schema whose fields drive the JNI (MOR-realtime) reader's {@code column_names}/{@code column_types}: + * the schema AT {@code queryInstant} for a {@code FOR TIME AS OF} read over a schema-on-read table (legacy + * {@code HudiScanNode:222-224}, kept in lockstep with the native {@code -1} overlay), else {@code latestAvro} + * (a plain read {@code queryInstant == null}, or a non-evolution table whose schema never changed — D3). A + * {@code null} instant short-circuits BEFORE {@link #resolveInternalSchemaAtInstant}, so a plain read does not + * reload the timeline (byte-identical to before this step); the pure choice is {@link #chooseJniSchema}. + */ + static Schema resolveJniColumnSchema(TableSchemaResolver schemaResolver, HoodieTableMetaClient metaClient, + Schema latestAvro, String queryInstant) { + Optional pinned = queryInstant == null + ? Optional.empty() + : resolveInternalSchemaAtInstant(schemaResolver, metaClient, queryInstant); + return chooseJniSchema(latestAvro, pinned); + } + + /** + * Pure JNI-schema choice (package-private for same-loader testing): the at-instant schema converted back to + * Avro when a pinned {@link InternalSchema} is present, else {@code latestAvro} unchanged. Uses the same + * {@code AvroInternalSchemaConverter.convert(InternalSchema, name)} as HD-C5a's {@code + * columnsFromInternalSchema} (the record name is cosmetic — only the root record is named). + */ + static Schema chooseJniSchema(Schema latestAvro, Optional pinnedSchema) { + return pinnedSchema.isPresent() + ? AvroInternalSchemaConverter.convert(pinnedSchema.get(), "hudi_table") + : latestAvro; + } + // ========== scan-level schema-evolution dictionary (current_schema_id + history_schema_info) ========== /** @@ -313,6 +361,24 @@ static Optional buildSchemaEvolutionProp(HoodieTableMetaClient metaClien resolved.enableSchemaEvolution, historical)); } + /** + * The native-reader schema dictionary for a {@code FOR TIME AS OF} read over a schema-on-read table (HD-C5b): + * the {@code -1} target overlay is built from the FULL {@code pinnedSchema} (a SUPERSET of the pinned scan + * slots), NOT from the requested column handles. The handles are LATEST-keyed ({@code getColumnHandles} runs + * before the MVCC pin), so a column renamed after the pinned instant is absent from them under its pinned + * (historical) name; building the overlay from them would emit a SUBSET missing that scan slot, and BE's + * field-id reader ({@code by_table_field_id} StructNode) requires EVERY scan slot to be present in the {@code + * -1} overlay (a missing slot std::out_of_range / SIGABRTs; extra fields are looked up by name only, so a + * superset is safe). The history entries are ALL committed versions (Option B, identical to the steady-state + * dict). {@code pinnedSchema} being present already implies schema-on-read is on (its commit-metadata + * InternalSchema resolved), so evolution is {@code true}. Touches the metaClient to read the schema history; + * runs on the TCCL-pinned scan thread. + */ + static String buildSchemaEvolutionDictAtInstant(HoodieTableMetaClient metaClient, InternalSchema pinnedSchema) { + return buildSchemaEvolutionDict(Collections.emptyList(), pinnedSchema, true, + allHistoricalSchemas(metaClient)); + } + /** * Pure assembly of the schema dictionary (package-private for same-loader testing): one {@code -1} target * entry (from the requested columns + base schema) + the history entries. HISTORY-SET = ALL committed diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java index 5375ad1d76c95b..0c5111e11c215d 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaUtilsTest.java @@ -23,16 +23,20 @@ import org.apache.doris.thrift.schema.external.TFieldPtr; import org.apache.doris.thrift.schema.external.TSchema; +import org.apache.avro.Schema; import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.internal.schema.Types; +import org.apache.hudi.internal.schema.convert.AvroInternalSchemaConverter; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.Optional; /** * Same-loader unit tests for {@link HudiSchemaUtils} — the HD-C4a {@code InternalSchema -> thrift} converter that @@ -371,4 +375,71 @@ public void dictEvolutionEmitsTargetPlusAllHistoricalVersions() { Assertions.assertEquals(1L, params.getHistorySchemaInfo().get(2).getSchemaId()); Assertions.assertEquals(2L, params.getHistorySchemaInfo().get(3).getSchemaId()); } + + // ========== HD-C5b: scan-side schema AT the pinned instant (FOR TIME AS OF over an evolution table) ========== + + @Test + public void jniSchemaNoPinReturnsLatestWithoutResolving() { + // A plain read (queryInstant == null) must NOT reload the timeline / resolve at-instant -> byte-identical + // to before HD-C5b, and the JNI column list stays the latest schema. Passing a null schemaResolver + + // metaClient proves the pin branch is never entered. MUTATION: drop the null-instant short-circuit -> + // resolveInternalSchemaAtInstant NPEs on the null metaClient -> red. + Schema latest = AvroInternalSchemaConverter.convert(buildInternalSchema(), "hudi_table"); + Assertions.assertSame(latest, + HudiSchemaUtils.resolveJniColumnSchema(null, null, latest, null)); + } + + @Test + public void jniSchemaUsesPinnedInstantSchemaNames() { + // FOR TIME AS OF over a schema-on-read table: the JNI (MOR-realtime) reader's column list must be the + // schema AT the pin, so a column renamed AFTER the pin shows its HISTORICAL (pinned) name to the merge + // reader (legacy HudiScanNode:222-224). MUTATION: return latestAvro instead of converting the pinned + // InternalSchema -> the pinned field name is absent -> red. + InternalSchema pinned = new InternalSchema(3L, Types.RecordType.get(Collections.singletonList( + Types.Field.get(1, false, "oldname", Types.IntType.get())))); + // The SAME field is "newname" at latest — the JNI list must NOT use it under the pin. + Schema latest = AvroInternalSchemaConverter.convert(new InternalSchema(5L, Types.RecordType.get( + Collections.singletonList(Types.Field.get(1, false, "newname", Types.IntType.get())))), + "hudi_table"); + + Schema jni = HudiSchemaUtils.chooseJniSchema(latest, Optional.of(pinned)); + List names = new ArrayList<>(); + jni.getFields().forEach(f -> names.add(f.name())); + Assertions.assertEquals(Collections.singletonList("oldname"), names); + } + + @Test + public void atInstantOverlayCarriesFullPinnedSchemaWithHistoricalNames() { + // HD-C5b: the at-instant -1 overlay is built from the FULL pinned schema (the empty-requested path over + // the pinned InternalSchema), so every pinned field carries its PINNED (historical) name + stable id and + // the overlay is a SUPERSET of the scan slots — BE matches old files BY FIELD ID and looks each scan slot + // up BY NAME (a scan slot MISSING from the overlay would std::out_of_range/SIGABRT; extra fields are + // ignored). This test pins that PURE assembly with a rename-shaped fixture (field "oldname"@pin), asserting + // the overlay keys off the PINNED names, not latest. MUTATION: buildSchemaEvolutionDict dropping a pinned + // field from the empty-requested overlay, or emitting only the base version instead of all versions -> red. + // NOT covered here (e2e-owed, design §5 HD-C5b): the PROVIDER ROUTING that selects this full-pinned path + // when a FOR TIME AS OF pin is present (getScanNodeProperties `if (pinnedSchema.isPresent())` + + // resolveInternalSchemaAtInstant / buildSchemaEvolutionDictAtInstant) needs a LIVE metaClient; a mutation + // deleting that branch (a pinned read falling back to the latest-keyed steady-state dict) is only + // observable at flip-time e2e — no same-loader test can catch it. + InternalSchema pinned = new InternalSchema(3L, Types.RecordType.get(Arrays.asList( + Types.Field.get(1, false, "oldname", Types.IntType.get()), + Types.Field.get(2, true, "other", Types.StringType.get())))); + String encoded = HudiSchemaUtils.buildSchemaEvolutionDict( + Collections.emptyList(), pinned, true, + Arrays.asList(internalSchemaWithId(0L), internalSchemaWithId(3L))); + + TFileScanRangeParams params = new TFileScanRangeParams(); + HudiSchemaUtils.applySchemaEvolution(params, encoded); + Assertions.assertEquals(-1L, params.getCurrentSchemaId()); + TSchema overlay = params.getHistorySchemaInfo().get(0); + Assertions.assertEquals(-1L, overlay.getSchemaId()); + Map top = topFields(overlay); + // BOTH pinned fields present with their pinned names + stable ids (full-pinned superset). + Assertions.assertEquals(Arrays.asList("oldname", "other"), new ArrayList<>(top.keySet())); + Assertions.assertEquals(1, top.get("oldname").getId()); + Assertions.assertEquals(2, top.get("other").getId()); + // history = -1 overlay + the two committed versions (all-versions, Option B). + Assertions.assertEquals(3, params.getHistorySchemaInfoSize()); + } } From 521ea8b0497a5c60fca90ea86988106846f7ab3b Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 22:10:31 +0800 Subject: [PATCH 139/334] [doc](catalog) HMS cutover hudi HD-C5b DONE; Part 2 (C5a-C5b) + schema-evolution (DV-006) complete; next = Group D (HD-D1) Record HD-C5b (scan-side schema-at-instant, e0da87888e5): JNI col list + native -1 overlay resolved AT the pinned instant for FOR TIME AS OF over a schema-on-read table; full-pinned overlay (superset of scan slots) required because getColumnHandles runs before the MVCC pin (latest-keyed handles drop a renamed column's pinned name). BE contract verified directly (v1 by_table_field_id requires superset; format_v2 find_external_root_field tolerates it). 4-dim adversarial review wf_ec6d47e3-37e: 0 production defects, 1 minor test-comment honesty fix, 2 refuted. Next work line = Group D write-reject safety net (HD-D1), then HD-B2 getTableHandle divert HUDI (the armed pivot, last step). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index f96c4f10afddc3..e75f417bf3e047 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**下一步 = C5b（扫描侧 MOR/JNI `column_names`/`column_types`@instant + 字典 `-1` 条按 pinned InternalSchema 重解〔句柄 id 是 latest-keyed，改名列须 iceberg full-pinned 式重解〕；`current_schema_id` 保 -1；off-scan-thread timeline touch 全进 `HudiMetaClientExecutor.execute`；deps C4c/C4d/C5a）。** 之后 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流点亮（武装 pivot，最后一步，须 Group C+D 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**下一步 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C✅ + HD-D1 全完）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -100,6 +100,7 @@ - **✅ HD-C4c DONE（`ac3222bc526`，休眠，Part 1 第三步）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**。给每个原生 base 文件钉写它的那次 commit 的 InternalSchema 版本号（field 12），BE 原生 field-id 读取器据此在字典里查该文件的列并按 id 对齐当前表（改名安全）。**仅原生分支**：JNI 合并读取器读 `column_names/types`@instant、不消费 schema_id。①`HudiSchemaUtils` 两共享静态 resolver：`resolveTableInternalSchema`（mode-aware base：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 版本 0，+evolution 标志；C4b 的 `attachHudiFieldIds` 重构复用之=id 源单一真源）+`resolveFileInternalSchema`（每文件版本：evolution 开→`FSUtils.getCommitTime(fileName)`+`InternalSchemaCache.searchSchemaAndCache`；关→base 版本 0）——port legacy `HudiScanNode.setHudiParams`。②`HudiScanRange.Builder.schemaId`（可空）→ctor 仅设时放 `hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** `fileDesc.setSchemaId`（镜像 `PaimonScanRange`），JNI 臂永不；缺→BE BY_NAME。③`planScan` 解析一次 base schema、建每文件 schema_id resolver（base 文件路径→版本）；`collectCowSplits`（COW base 恒原生）+`buildMorRange` 原生无 log 片钉之；@incr 路传 null（@incr 列最新 schema、无每文件字典）。base-schema 未解析→null resolver→不钉；每文件解析失败→log 跳过该文件→均降 BY_NAME 不崩扫描。**⚠编译期坑**：`searchSchemaAndCache` 编译classpath 是 **2-arg** `(long, metaClient)`（与 legacy fe-core 同），非默认 m2 里 SELECTDB-0.14.1 jar 的 3-arg（javap 看错 jar）——已用 2-arg。**排序（关键）**：schema_id present 而字典缺→BE 不入 field-id 路→**BY_NAME=安全前缀**；字典步（C4d）须在其后，dict-present+schema_id-unset 才是禁的 `current==file==-1` v1-ConstNode 退化。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。schema_id **值 e2e**，只 stamping 机制 same-loader 测（`HudiScanRangeTest`+3：原生 set f12 / JNI 即便设了也不 set / 原生无 id 留 unset；`HudiIncrementalPlanScanTest` buildMorRange 断言原生 downgrade 载 resolver id、force-jni 片不载）。全绿（hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级字典发射，与 C4c 背靠背；C4c 安全前缀已落）**。 - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；统一对抗复审 DONE（`wf_33882086-fbb`）+修 `ad8aa87bd2d`（见顶部 🎯）；下一步 = Part 2 C5a。** - **✅ HD-C5a DONE（`af9a3c1a004`，休眠，Part 2 首步）= schema-at-instant 列表（3-arg `getTableSchema` 覆写）**。post-flip 通用路径下给 `FOR TIME AS OF` 演进读供**钉时刻**的列表，闭合 HD-C2 残留 #1（SPI 默认曾返最新、忽略 pin）。①新 3-arg `getTableSchema(session,handle,ConnectorMvccSnapshot)` 覆写 keyed off `HudiTableHandle.getQueryInstant()`（`applySnapshot` 钉的 pin），**非** `snapshot.getSchemaId()`（hudi 恒 -1；fe-core `PluginDrivenMvccExternalTable:375-376` 传 post-applySnapshot 句柄故 instant 已在其上）；null instant（plain / 非-FOR-TIME pin 如 @incr，其 loadSnapshot 分支自列最新 schema、根本不到此）经**同一** shared `buildTableSchema(handle,instant)` 走 latest→latest/at-instant 不发散。②`getSchemaFromMetaClient(basePath)`→`getSchemaFromMetaClient(basePath,queryInstant)`，**包进 `metaClientExecutor.execute`**（plugin UGI doAs+TCCL pin）闭合 C4b 欠的未包裹 auth/TCCL gap（getTableSchema 可跑在非 pin 的 scan 线程外，如 catalog metadata load）；at-instant 臂字节忠实 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`+`HMSExternalTable.initHudiSchema`：`reloadActiveTimeline()`+`getTableInternalSchemaFromCommitMetadata(instant)`→present（schema.on.read 开）经 `columnsFromInternalSchema`（`AvroInternalSchemaConverter.convert(InternalSchema,name)`→avro→`avroSchemaToColumns`→`attachTopLevelFieldIds` 稳定 id）→改名列显历史名、BE 按 field id 匹老文件；absent（演进关）落 latest fallback＝非演进表字节等价（D3）。③用 `HudiScanPlanProvider.buildMetaClient`（与旧 inline build 字节等价）；hive 3-arg 委派已发（`HiveConnectorMetadata:1071`）无须改；连接器编 unshaded hudi-common 故 `convert` 回 unshaded `org.apache.avro.Schema` 与 `avroSchemaToColumns` 相容。零 fe-core import、零新连接器-api SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、句柄未分流；live iceberg/paimon 不走 hudi getTableSchema）。at-instant **读**本身 e2e；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading（2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〔snapshot 传 null 证之〕）。**4 维对抗复审 `wf_703ea26e-49f`（legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent）+逐条 refute：0 确认缺陷**（1 nit＝wrapping 后 LOG.warn 只印 wrapper message refuted〔旧码亦只印 `e.getMessage()` 非本步引入〕；顺带把 `e.getMessage()`→传 throwable 补全 cause 链+栈）。全绿（hudi 156 测〔+3〕 + checkstyle 0 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进列表（改名/调序显历史名、非演进 latest 等价、meta 列集与 legacy 一致）只 e2e 可证（单测覆盖 routing）；schema-on-read commit-metadata InternalSchema 是否含 `_hoodie_*` meta 列＝随 legacy 定、e2e 核。**下一步 = C5b（扫描侧 MOR/JNI 列表@instant + 字典 `-1` 条按 pinned schema 重解）。** + - **✅ HD-C5b DONE（`e0da87888e5`，休眠，Part 2 收官）= 扫描侧 schema-at-instant（MOR/JNI 列表 + 原生 `-1` overlay @instant）**。`FOR TIME AS OF` over schema-on-read 表：两个扫描侧 schema locus 都须钉时刻、锁步。**(i) planScan JNI（MOR-realtime）`column_names`/`column_types`**＝pin 时刻的 schema（legacy `HudiScanNode:222-224` `getSchemaCacheValue(hmsTable,queryInstant)`），经新 `HudiSchemaUtils.resolveJniColumnSchema`（路由）→纯 `chooseJniSchema`（pinned present→`AvroInternalSchemaConverter.convert(pinnedIS,"hudi_table")`；else→latestAvro）。**plain read（`queryInstant==null`）短路在 `resolveInternalSchemaAtInstant` 之前→不 reload timeline＝字节不变**；非演进（instant 无 commit-metadata InternalSchema）→latest==at-instant（D3）。`avroSchema` 局部保 latest＝per-file schema_id resolver 的 base（分类每文件自身写入版本，pin-independent）。**(ii) getScanNodeProperties 原生 `-1` overlay** 从**完整 pinned InternalSchema** 重建（新 `buildSchemaEvolutionDictAtInstant`＝`buildSchemaEvolutionDict(emptyList, pinnedSchema, true, allHistoricalSchemas)`＝empty-requested 路径 over pinned schema），**非** latest-keyed 请求句柄。**为何 full-pinned**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:1534`＝latest-keyed 句柄图），FOR TIME 下 scan slots 按 pinned schema（C5a 3-arg，`PluginDrivenMvccExternalTable:376`）绑＝pinned 名；`buildColumnHandles` 用 slot(pinned 名) 查 latest 句柄图→改名列 miss→**从 `columns` 掉**；用 `columns` 建 overlay 会漏该 scan slot→BE 崩。**BE 契约亲手核对**（决定性）：v1 reader `by_table_field_id` 建 StructNode 每 overlay 字段一子、reader 按 **scan slot 名** `children_column_exists`＝`DCHECK(children.contains)`/`children.at`（`table_schema_change_helper.h:165-167`）→**scan slot 缺 overlay＝std::out_of_range/SIGABRT；overlay⊇scan slots＝必需**；format_v2 `find_external_root_field`（`table_reader.cpp:254-283`）按名查投影列、忽略多余＝**超集容忍**。⇒ full-pinned（全 pinned 字段）是 scan slots 超集＝双 reader 安全，用句柄建的子集会崩。`current_schema_id` 保 -1（须≠每 split id≥0）；史=全版本（Option B，C4d 同）；**无 meta 列可达 pin 路**（C5a schema-on-read 绑定源自 InternalSchema、无 `_hoodie_*`→查询绑不到 meta 列）。@incr（begin/endInstant 非 queryInstant）→pin==null→latest dict＝列最新 schema（§7.7）。零 fe-core、零新 SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、hudi 句柄未分流）。**4 维对抗复审 `wf_ec6d47e3-37e`（legacy-parity·at-instant / BE-contract·SIGABRT / no-regression·dormancy / test-intent）+逐条 refute：0 生产缺陷**（3 raised→1 survived〔confirmed-minor 测注释诚实性：`atInstantOverlayCarriesFullPinnedSchemaWithHistoricalNames` 测直调 `buildSchemaEvolutionDict(emptyList,…)`、守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝不可同加载器测、e2e-owed，已重写注释诚实标注〕→已修；2 refuted〔另一测覆盖提醒/设计文档措辞 nit〕）。`HudiSchemaUtilsTest`+3（JNI no-pin no-op〔null 短路不碰 metaClient〕/ chooseJniSchema pinned 历史名 / full-pinned overlay 历史名+全版本）。全绿（hudi 159 测 + checkstyle 0〔最长注释 116<130〕 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、非演进 FOR TIME latest 等价、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing**（`if(pinnedSchema.isPresent())` 分支 + `resolveInternalSchemaAtInstant` reload/resolve）**只 e2e 可证**（同加载器无 live metaClient；离线测覆盖纯 `chooseJniSchema`/full-pinned 组装）。**⇒ Part 2（C5a–C5b）+ HD-C4/C5 schema evolution（DV-006）全部收官。下一步 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步）。** - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md b/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md index 3e0ef043c0c9b6..eebe9809089f82 100644 --- a/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md +++ b/plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md @@ -70,7 +70,7 @@ Legacy flow (`HudiScanNode` + `HudiUtils` + `HudiSchemaCacheKey/Value` + `HMSExt - **HD-C4c — per-split `THudiFileDesc.schema_id` (native branch ONLY).** `HudiScanRange.Builder` gains `schemaId`; `populateRangeParams` sets `fileDesc.setSchemaId` ONLY in the native else-branch (`HudiScanRange.java:219-221`), NEVER the JNI branch (`:192-218`) — legacy/paimon parity. Provider computes per-file id in `collectCowSplits` and the native no-log MOR read-optimized slice that **downgrades to native** (`HudiScanRange:179-183` — NOT only COW): `FSUtils.getCommitTime(fileName) → InternalSchemaCache.searchSchemaAndCache` (evolution) else fallback InternalSchema id `0`. Runs on the TCCL-pinned scan thread. Factor a **shared static per-file→schemaId resolver** into `HudiSchemaUtils` that C4d reuses. *Test:* stamping is same-loader (`HudiScanRangeTest`: Builder→populateRangeParams asserts field 12 set on native, UNSET on JNI); the id value is e2e. **deps: shares resolver with C4d.** - **HD-C4d — scan-level dict emission + apply (steady-state / no-pin).** `getScanNodeProperties` builds a base64 `hudi.schema_evolution` prop = `current_schema_id(-1)` + the `-1` entry from the **requested** columns arg + one entry per **referenced-split** InternalSchema version (via the C4c shared resolver + C4a converter). OVERRIDE `populateScanLevelParams` to copy fields 25/26. Gate OFF for `force_jni`/MOR-realtime-only handles (paimon gate parity). *Test:* encode/decode round-trip same-loader; history-set completeness + BE engage are e2e. **deps: C4a; shares C4c resolver. HARD-ORDER after C4c** (a dormant prefix flipping with the dict present while per-split `schema_id` is unset(`-1`) is the forbidden `current==file==-1` v1-ConstNode degrade — land C4c+C4d back-to-back). - **HD-C5a — schema-at-instant column list (`getTableSchema` 3-arg override).** Override `ConnectorTableOps.getTableSchema(session,handle,snapshot)` on `HudiConnectorMetadata` (ABSENT today → SPI default returns latest = HD-C2 residual #1). Key off `HudiTableHandle.getQueryInstant()` (NOT `snapshot.getSchemaId()` which stays `-1` for hudi); null instant → delegate to the 2-arg latest path (shared build-path so latest/at-instant can't drift). Resolve InternalSchema AT the instant WRAPPED in `HudiMetaClientExecutor.execute` (also closes the existing unwrapped `getSchemaFromMetaClient` gap). Hive delegation already shipped ⇒ no hive change. *Test:* null-instant→latest delegation branch is same-loader; the at-instant read is e2e. **deps: C4b.** -- **HD-C5b — scan-side at-instant: MOR/JNI column list + `-1` entry at pinned instant.** (i) `planScan` JNI `column_names`/`column_types` re-derived from the queryInstant-scoped schema (replacing latest `getTableAvroSchema(true)`) — legacy `HudiScanNode:222-224`; keep BOTH schema loci in lockstep. (ii) The dict `-1` entry re-resolved against the **pinned** InternalSchema (iceberg full-pinned / empty-requested) because handle ids are latest-keyed — else a renamed column drops a BE slot. `current_schema_id` stays `-1`. All off-scan-thread timeline touches under `HudiMetaClientExecutor.execute`. *Test:* almost entirely e2e; FE-unit asserts JNI col list + `-1` entry are derived from the instant-resolved schema, not latest. **deps: C4c, C4d, C5a.** +- **HD-C5b — scan-side at-instant: MOR/JNI column list + `-1` entry at pinned instant.** (i) `planScan` JNI `column_names`/`column_types` re-derived from the queryInstant-scoped schema (replacing latest `getTableAvroSchema(true)`) — legacy `HudiScanNode:222-224`; keep BOTH schema loci in lockstep. (ii) The dict `-1` entry re-resolved against the **pinned** InternalSchema (iceberg full-pinned / empty-requested) because handle ids are latest-keyed — else a renamed column drops a BE slot. `current_schema_id` stays `-1`. All off-scan-thread timeline touches under `HudiMetaClientExecutor.execute`. *Test:* almost entirely e2e; FE-unit asserts that GIVEN an instant-resolved schema the JNI col list + `-1` overlay derive from IT not latest (the pure `chooseJniSchema` / full-pinned empty-requested assembly seams). The at-instant resolution + the provider pin-routing (`if (pinnedSchema.isPresent())`) themselves need a live metaClient, so a mutation that ignores the pin is only caught at flip-time e2e. **deps: C4c, C4d, C5a.** **STATUS: DONE** (`e0da87888e5`; 4-dim adversarial review `wf_ec6d47e3-37e` — 0 production defects, 1 confirmed-minor test-comment honesty fix applied, 1 nit). --- From a23e6b9ac90aa24a5089595d14cbe09c7185b3ba Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 22:52:35 +0800 Subject: [PATCH 140/334] [feat](catalog) HMS cutover hudi read-only write-reject safety net (Group D) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Once the hms cutover diverts a hudi-on-HMS table to a foreign hudi handle, the gateway's 3-way routing sends that handle to the HUDI sibling connector (never iceberg), so this connector answers all write/procedure/DDL/transaction calls. Hudi data is written by Spark/Flink; Doris only reads. Recon found the substantive rejection is ALREADY provided by HD-B1's routing + the connector-api SPI defaults, so this step is deliberately minimal: - Data writes (INSERT/OVERWRITE/DELETE/UPDATE/MERGE): getWritePlanProvider stays null -> supportedWriteOperations empty -> PhysicalPlanTranslator's admission gate throws a clean AnalysisException at PLANNING. Left as-is on purpose: the gate DERIVES from getWritePlanProvider, so throwing there would misclassify the error as an internal bug instead of the clean "does not support INSERT". - EXECUTE: getProcedureOps null -> rejected (like plain-hive). - DDL: SPI defaults throw "... not supported". User signed off STRICT reject of ALL DDL, including DROP TABLE / RENAME TABLE (a behavior change vs legacy HMS). Net-new = the explicit LAST-LINE DEFENSE: override HudiConnectorMetadata .beginTransaction(session) to throw an explicit "Hudi tables are read-only in this catalog" DorisConnectorException (replacing the generic "Transactions not supported" default). beginTransaction is execution-stage only (called after planning by the insert executor / rewrite driver), so for a hudi write it is dead code (rejected at planning first) — a safe, message-quality defensive net, never degrading the admission-gate classification. The per-handle overload's SPI default delegates to it. Proving test HudiReadOnlyWriteRejectTest locks the whole read-only surface: no writer + empty supportedWriteOperations, no procedures, beginTransaction (BOTH the no-arg and the production-routed per-handle overload) throws explicit read-only, and the FULL DDL mutation family (table drop/rename/truncate + the 16-method ALTER surface) throws. DORMANT (hms not in SPI_READY_TYPES, hudi handles not diverted). 3-dim focused adversarial review (wf_36fd1c6c-e58): 0 production defects; 2 surviving nits (both test-strength) fixed here — the per-handle beginTransaction overload and the full DDL family now locked. Refuted: DROP DATABASE FORCE cascade (pre- existing gateway code, legacy-parity, out of the per-table scope) and the by-design dormancy. hudi module 163 tests, checkstyle 0, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hudi/HudiConnectorMetadata.java | 26 ++++ .../hudi/HudiReadOnlyWriteRejectTest.java | 140 ++++++++++++++++++ 2 files changed, 166 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiReadOnlyWriteRejectTest.java diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 4746324e33b670..0d826faa268518 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -26,6 +26,7 @@ import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.handle.ConnectorTransaction; import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; import org.apache.doris.connector.api.pushdown.ConnectorAnd; @@ -319,6 +320,31 @@ public Map getProperties() { return properties; } + // ========== Read-only write-reject safety net ========== + + /** + * Hudi-on-HMS tables are READ-ONLY in this catalog (data is written by Spark/Flink; Doris only reads). A hudi + * table's write is already rejected UP FRONT by the engine's admission gate — {@code + * supportedWriteOperations(handle)} derives from {@link #getWritePlanProvider(ConnectorTableHandle)} which + * this connector leaves at the SPI default {@code null} → empty operation set → the {@code + * PhysicalPlanTranslator} INSERT/row-level-DML gates throw a clean {@code AnalysisException} before ever + * opening a transaction; {@code EXECUTE} is rejected by {@code getProcedureOps} → {@code null}; every DDL + * op throws the SPI default {@code "... not supported"}. This override is the explicit LAST-LINE DEFENSE: it + * replaces the generic {@code "Transactions not supported"} default with a hudi-specific read-only message so + * that any future write path reaching transaction-open (bypassing the gate) fails loud with the RIGHT reason + * rather than a confusing generic one. Overriding the no-arg form covers the per-handle overload too (its SPI + * default delegates here). It is deliberately NOT placed on {@code getWritePlanProvider}: the admission gate + * calls {@code getWritePlanProvider} to DECIDE, so throwing there would make the gate throw a {@code + * DorisConnectorException} the engine misclassifies as an internal error instead of the clean "does not + * support INSERT" — keep the provider {@code null} and reject explicitly only here (dormant until hms enters + * {@code SPI_READY_TYPES}). + */ + @Override + public ConnectorTransaction beginTransaction(ConnectorSession session) { + throw new DorisConnectorException( + "Hudi tables are read-only in this catalog; INSERT/UPDATE/DELETE/MERGE are not supported"); + } + // ========== ConnectorMvccOps (MTMV freshness) ========== /** diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiReadOnlyWriteRejectTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiReadOnlyWriteRejectTest.java new file mode 100644 index 00000000000000..a3b6d1753befda --- /dev/null +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiReadOnlyWriteRejectTest.java @@ -0,0 +1,140 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hudi; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.spi.ConnectorContext; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; + +/** + * Locks the READ-ONLY safety net for hudi-on-HMS tables: once a flipped hms gateway diverts a hudi table to a + * foreign handle, the gateway's 3-way routing sends that handle to THIS (the hudi sibling) connector, so this + * connector — not iceberg — answers every write / procedure / DDL / transaction call. Hudi data is written by + * Spark/Flink; Doris only reads. This test pins that the hudi connector rejects EVERY mutation fail-loud, so a + * future change cannot silently make a hudi table writable or DDL-mutable, and a hudi write is never + * mis-executed as an iceberg write. + * + *

    Dormant until hms enters {@code SPI_READY_TYPES} and hudi handles are diverted (no production path reaches + * this connector yet); this is a Rule-9 behavior lock. The correctness at flip is asserted end-to-end (a + * heterogeneous HMS catalog where hudi INSERT/DELETE/MERGE/EXECUTE all fail loud read-only). + */ +public class HudiReadOnlyWriteRejectTest { + + private static final ConnectorTableHandle HUDI_HANDLE = + new HudiTableHandle("db", "t", "s3://b/t", "COPY_ON_WRITE"); + + private static HudiConnector connector() { + return new HudiConnector(Collections.emptyMap(), new ConnectorContext() { + @Override + public String getCatalogName() { + return "test_catalog"; + } + + @Override + public long getCatalogId() { + return 1L; + } + }); + } + + /** The write/DDL methods throw before touching the client/executor, so null collaborators are sufficient. */ + private static HudiConnectorMetadata metadata() { + return new HudiConnectorMetadata(null, Collections.emptyMap(), null); + } + + @Test + public void noWriterSoDataWritesRejectedByAdmissionGate() { + // Read-only: the hudi connector supplies NO write plan provider, so supportedWriteOperations is empty and + // the engine's PhysicalPlanTranslator INSERT / row-level-DML gates throw a clean AnalysisException up + // front (before opening any transaction). Keeping the provider null (NOT throwing) is deliberate: the + // admission gate CALLS getWritePlanProvider to decide, so a throw there would make the gate raise a + // DorisConnectorException the engine misclassifies as an internal error instead of "does not support + // INSERT". MUTATION: return a write plan provider / a non-empty op set -> a hudi INSERT/DELETE would be + // admitted and mis-executed -> red. + HudiConnector connector = connector(); + Assertions.assertNull(connector.getWritePlanProvider(HUDI_HANDLE)); + Assertions.assertTrue(connector.supportedWriteOperations(HUDI_HANDLE).isEmpty()); + } + + @Test + public void noProceduresSoExecuteRejected() { + // No procedure ops -> EXECUTE on a hudi table is rejected ("does not support EXECUTE"), same as + // plain-hive. MUTATION: return a non-null ProcedureOps -> EXECUTE admitted -> red. + Assertions.assertNull(connector().getProcedureOps(HUDI_HANDLE)); + } + + @Test + public void beginTransactionThrowsExplicitReadOnly() { + // The explicit last-line defense: opening a write transaction on a hudi table fails loud with a + // HUDI-SPECIFIC read-only message rather than the generic "Transactions not supported" default, so any + // write path that reaches transaction-open (bypassing the admission gate) reports the right reason. + // Overriding the no-arg form covers the per-handle overload (its SPI default delegates to it). MUTATION: + // drop the override -> the message is the generic default, not read-only -> red. + HudiConnectorMetadata metadata = metadata(); + DorisConnectorException noArg = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata.beginTransaction(null)); + Assertions.assertTrue(noArg.getMessage().toLowerCase().contains("read-only"), + "expected an explicit hudi read-only message, got: " + noArg.getMessage()); + // The production-routed path is the PER-HANDLE overload (the gateway forwards + // siblingMetadata(session, handle).beginTransaction(session, handle)); its SPI default delegates to the + // no-arg override, so it too fails with the explicit read-only message. Lock BOTH so the delegation the + // real path relies on cannot silently break. + DorisConnectorException perHandle = Assertions.assertThrows(DorisConnectorException.class, + () -> metadata.beginTransaction(null, HUDI_HANDLE)); + Assertions.assertTrue(perHandle.getMessage().toLowerCase().contains("read-only"), + "the per-handle overload the gateway routes to must also be explicit read-only"); + } + + @Test + public void allDdlRejected() { + // Hudi tables are externally managed, so Doris rejects ALL DDL — catalog metadata ops (DROP TABLE / + // RENAME TABLE / TRUNCATE, a deliberate strict-read-only choice: a signed-off behavior change vs legacy + // HMS, which allowed them) AND the full ALTER family (add/drop/rename/modify/reorder column, create/drop + // branch/tag, add/drop/replace partition field). These are the SPI defaults; this test LOCKS the WHOLE + // mutation surface so a future override cannot silently make a hudi table DDL-mutable. MUTATION: implement + // any of these on the hudi metadata -> that one assertion goes red. + HudiConnectorMetadata m = metadata(); + // Table-level catalog DDL. + Assertions.assertThrows(DorisConnectorException.class, () -> m.dropTable(null, HUDI_HANDLE)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.renameTable(null, HUDI_HANDLE, "new_name")); + Assertions.assertThrows(DorisConnectorException.class, + () -> m.truncateTable(null, HUDI_HANDLE, Collections.emptyList())); + // Column ALTER. + Assertions.assertThrows(DorisConnectorException.class, () -> m.addColumn(null, HUDI_HANDLE, null, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.addColumns(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.dropColumn(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.renameColumn(null, HUDI_HANDLE, null, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.modifyColumn(null, HUDI_HANDLE, null, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.reorderColumns(null, HUDI_HANDLE, null)); + // Branch / tag / partition-field ALTER. + Assertions.assertThrows(DorisConnectorException.class, + () -> m.createOrReplaceBranch(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.createOrReplaceTag(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.dropBranch(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.dropTag(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.addPartitionField(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, () -> m.dropPartitionField(null, HUDI_HANDLE, null)); + Assertions.assertThrows(DorisConnectorException.class, + () -> m.replacePartitionField(null, HUDI_HANDLE, null)); + } +} From a1bcd6f27ec5a0b3f1e4e3de60ea881674044e8d Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 22:54:45 +0800 Subject: [PATCH 141/334] [doc](catalog) HMS cutover hudi HD-D1 DONE (read-only reject); Group A/C/D complete; next = HD-B2 (arming pivot) Record the read-only write-reject safety net (7ae9404af25): recon corrected the pre-HD-B1 plan (hudi routes to the hudi sibling not iceberg; SPI defaults already reject fail-loud). Signed-off Q1 (rely on the clean SPI-default rejection + a proving test + an explicit beginTransaction defense, NOT a getWritePlanProvider throw) and Q2 (strict: reject all DDL including DROP/RENAME TABLE). Review wf_36fd1c6c-e58: 0 production defects, 2 test-strength nits fixed. Group A + Group C + HD-D1 are all complete -> only HD-B2 (getTableHandle divert HUDI, the arming pivot, last connector step) remains before the hms flip. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 3 ++- .../tasks/hudi-on-hms-delegation-plan-2026-07-08.md | 10 ++++++++++ 2 files changed, 12 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index e75f417bf3e047..34fd6886a7d364 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**下一步 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C✅ + HD-D1 全完）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完 ⇒ 只余 HD-B2（getTableHandle 分流 HUDI，武装 pivot，最后一个连接器步）。下一步 = HD-B2。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -101,6 +101,7 @@ - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；统一对抗复审 DONE（`wf_33882086-fbb`）+修 `ad8aa87bd2d`（见顶部 🎯）；下一步 = Part 2 C5a。** - **✅ HD-C5a DONE（`af9a3c1a004`，休眠，Part 2 首步）= schema-at-instant 列表（3-arg `getTableSchema` 覆写）**。post-flip 通用路径下给 `FOR TIME AS OF` 演进读供**钉时刻**的列表，闭合 HD-C2 残留 #1（SPI 默认曾返最新、忽略 pin）。①新 3-arg `getTableSchema(session,handle,ConnectorMvccSnapshot)` 覆写 keyed off `HudiTableHandle.getQueryInstant()`（`applySnapshot` 钉的 pin），**非** `snapshot.getSchemaId()`（hudi 恒 -1；fe-core `PluginDrivenMvccExternalTable:375-376` 传 post-applySnapshot 句柄故 instant 已在其上）；null instant（plain / 非-FOR-TIME pin 如 @incr，其 loadSnapshot 分支自列最新 schema、根本不到此）经**同一** shared `buildTableSchema(handle,instant)` 走 latest→latest/at-instant 不发散。②`getSchemaFromMetaClient(basePath)`→`getSchemaFromMetaClient(basePath,queryInstant)`，**包进 `metaClientExecutor.execute`**（plugin UGI doAs+TCCL pin）闭合 C4b 欠的未包裹 auth/TCCL gap（getTableSchema 可跑在非 pin 的 scan 线程外，如 catalog metadata load）；at-instant 臂字节忠实 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`+`HMSExternalTable.initHudiSchema`：`reloadActiveTimeline()`+`getTableInternalSchemaFromCommitMetadata(instant)`→present（schema.on.read 开）经 `columnsFromInternalSchema`（`AvroInternalSchemaConverter.convert(InternalSchema,name)`→avro→`avroSchemaToColumns`→`attachTopLevelFieldIds` 稳定 id）→改名列显历史名、BE 按 field id 匹老文件；absent（演进关）落 latest fallback＝非演进表字节等价（D3）。③用 `HudiScanPlanProvider.buildMetaClient`（与旧 inline build 字节等价）；hive 3-arg 委派已发（`HiveConnectorMetadata:1071`）无须改；连接器编 unshaded hudi-common 故 `convert` 回 unshaded `org.apache.avro.Schema` 与 `avroSchemaToColumns` 相容。零 fe-core import、零新连接器-api SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、句柄未分流；live iceberg/paimon 不走 hudi getTableSchema）。at-instant **读**本身 e2e；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading（2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〔snapshot 传 null 证之〕）。**4 维对抗复审 `wf_703ea26e-49f`（legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent）+逐条 refute：0 确认缺陷**（1 nit＝wrapping 后 LOG.warn 只印 wrapper message refuted〔旧码亦只印 `e.getMessage()` 非本步引入〕；顺带把 `e.getMessage()`→传 throwable 补全 cause 链+栈）。全绿（hudi 156 测〔+3〕 + checkstyle 0 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进列表（改名/调序显历史名、非演进 latest 等价、meta 列集与 legacy 一致）只 e2e 可证（单测覆盖 routing）；schema-on-read commit-metadata InternalSchema 是否含 `_hoodie_*` meta 列＝随 legacy 定、e2e 核。**下一步 = C5b（扫描侧 MOR/JNI 列表@instant + 字典 `-1` 条按 pinned schema 重解）。** - **✅ HD-C5b DONE（`e0da87888e5`，休眠，Part 2 收官）= 扫描侧 schema-at-instant（MOR/JNI 列表 + 原生 `-1` overlay @instant）**。`FOR TIME AS OF` over schema-on-read 表：两个扫描侧 schema locus 都须钉时刻、锁步。**(i) planScan JNI（MOR-realtime）`column_names`/`column_types`**＝pin 时刻的 schema（legacy `HudiScanNode:222-224` `getSchemaCacheValue(hmsTable,queryInstant)`），经新 `HudiSchemaUtils.resolveJniColumnSchema`（路由）→纯 `chooseJniSchema`（pinned present→`AvroInternalSchemaConverter.convert(pinnedIS,"hudi_table")`；else→latestAvro）。**plain read（`queryInstant==null`）短路在 `resolveInternalSchemaAtInstant` 之前→不 reload timeline＝字节不变**；非演进（instant 无 commit-metadata InternalSchema）→latest==at-instant（D3）。`avroSchema` 局部保 latest＝per-file schema_id resolver 的 base（分类每文件自身写入版本，pin-independent）。**(ii) getScanNodeProperties 原生 `-1` overlay** 从**完整 pinned InternalSchema** 重建（新 `buildSchemaEvolutionDictAtInstant`＝`buildSchemaEvolutionDict(emptyList, pinnedSchema, true, allHistoricalSchemas)`＝empty-requested 路径 over pinned schema），**非** latest-keyed 请求句柄。**为何 full-pinned**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:1534`＝latest-keyed 句柄图），FOR TIME 下 scan slots 按 pinned schema（C5a 3-arg，`PluginDrivenMvccExternalTable:376`）绑＝pinned 名；`buildColumnHandles` 用 slot(pinned 名) 查 latest 句柄图→改名列 miss→**从 `columns` 掉**；用 `columns` 建 overlay 会漏该 scan slot→BE 崩。**BE 契约亲手核对**（决定性）：v1 reader `by_table_field_id` 建 StructNode 每 overlay 字段一子、reader 按 **scan slot 名** `children_column_exists`＝`DCHECK(children.contains)`/`children.at`（`table_schema_change_helper.h:165-167`）→**scan slot 缺 overlay＝std::out_of_range/SIGABRT；overlay⊇scan slots＝必需**；format_v2 `find_external_root_field`（`table_reader.cpp:254-283`）按名查投影列、忽略多余＝**超集容忍**。⇒ full-pinned（全 pinned 字段）是 scan slots 超集＝双 reader 安全，用句柄建的子集会崩。`current_schema_id` 保 -1（须≠每 split id≥0）；史=全版本（Option B，C4d 同）；**无 meta 列可达 pin 路**（C5a schema-on-read 绑定源自 InternalSchema、无 `_hoodie_*`→查询绑不到 meta 列）。@incr（begin/endInstant 非 queryInstant）→pin==null→latest dict＝列最新 schema（§7.7）。零 fe-core、零新 SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、hudi 句柄未分流）。**4 维对抗复审 `wf_ec6d47e3-37e`（legacy-parity·at-instant / BE-contract·SIGABRT / no-regression·dormancy / test-intent）+逐条 refute：0 生产缺陷**（3 raised→1 survived〔confirmed-minor 测注释诚实性：`atInstantOverlayCarriesFullPinnedSchemaWithHistoricalNames` 测直调 `buildSchemaEvolutionDict(emptyList,…)`、守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝不可同加载器测、e2e-owed，已重写注释诚实标注〕→已修；2 refuted〔另一测覆盖提醒/设计文档措辞 nit〕）。`HudiSchemaUtilsTest`+3（JNI no-pin no-op〔null 短路不碰 metaClient〕/ chooseJniSchema pinned 历史名 / full-pinned overlay 历史名+全版本）。全绿（hudi 159 测 + checkstyle 0〔最长注释 116<130〕 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、非演进 FOR TIME latest 等价、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing**（`if(pinnedSchema.isPresent())` 分支 + `resolveInternalSchemaAtInstant` reload/resolve）**只 e2e 可证**（同加载器无 live metaClient；离线测覆盖纯 `chooseJniSchema`/full-pinned 组装）。**⇒ Part 2（C5a–C5b）+ HD-C4/C5 schema evolution（DV-006）全部收官。下一步 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步）。** + - **✅ HD-D1 DONE（`7ae9404af25`，休眠，Group D 收官）= 只读写-拒安全网**（recon 对照真实代码校正规划：**规划 §HD-D1「让 getWritePlanProvider 抛显式只读」写于 HD-B1 之前、已过时**）。翻闸后 hudi 表→通用 `PluginDrivenExternalTable`+外来句柄→网关三路分流（`resolveSiblingOwner` peek）到 **hudi 兄弟连接器**（非 iceberg），故本连接器答所有写/过程/DDL/事务调用。**核心发现：拒绝主体已现成**——HD-B1 正确路由（hudi→hudi 兄弟、绝不 iceberg）+ connector-api SPI 默认：`getWritePlanProvider(handle)`=null→`supportedWriteOperations(handle)`=空→`PhysicalPlanTranslator:657/709` admission gate **规划期**抛干净 `AnalysisException`「does not support INSERT/row-level DML」；`getProcedureOps`=null→EXECUTE 拒；`ConnectorTableOps` 全 DDL 默认抛「… not supported」；`beginTransaction` 默认抛。**用户签字 Q1=方案 A**（保留干净拒绝+测试锁定+事务层防御，**不**让 getWritePlanProvider 抛——admission gate 由它派生，在其上抛→引擎误分类「maybe palo bug」而非干净「does not support INSERT」；已核对 fe-core 写路径顺序：gate 先于 getWritePlanProvider 先于 beginTransaction，且连接器 beginTransaction 只在**执行期**〔`PluginDrivenInsertExecutor`/`ConnectorRewriteDriver:130`〕被调、规划期已拒故对 hudi 写是死码）。**用户签字 Q2=方案乙**（严格拒绝**所有** DDL 含 `DROP TABLE`/`RENAME TABLE`——相对 legacy HMS〔经 `HiveMetadataOps` 支持删/改名 hudi 表 HMS 条目〕是**有意行为倒退**、hudi 完全外部管理）。**净新增（唯一代码改动）=override `HudiConnectorMetadata.beginTransaction(session)` 抛显式** `"Hudi tables are read-only in this catalog; INSERT/UPDATE/DELETE/MERGE are not supported"`（替换泛型默认；per-handle 重载 SPI 默认委派到它；纯消息质量末线兜底、不劣化 gate 分类）。`HudiReadOnlyWriteRejectTest`〔`new HudiConnectorMetadata(null,{},null)`＝write/DDL 抛前不 deref 故 null 够〕锁：无 writer+空写操作集〔=写不被误当 iceberg 执行〕/无过程/beginTransaction **双重载**〔no-arg + 生产路由的 per-handle〕均抛显式只读/全 DDL 变更族〔dropTable·renameTable·truncate + addColumn(s)·dropColumn·renameColumn·modifyColumn·reorderColumns·create/drop Branch|Tag·add/drop/replace PartitionField〕抛。**3 维对抗复审 `wf_36fd1c6c-e58`（reject-completeness·routing / classification·no-regression / test-intent）+逐条 refute：0 生产缺陷**；6 raised→2 survived nit（均测强化：per-handle beginTransaction 重载直测 + 全 DDL 族补锁，已修）；refuted：`DROP DATABASE … FORCE` 级联〔pre-existing 网关码、legacy 平价、超 per-table 范围〕/ 休眠〔HD-B2 未分流 HUDI＝设计延后〕/ beginTransaction 非可写性锁〔可写性由 `supportedWriteOperations` 空锁，已断言〕。全绿（hudi 163 测 + checkstyle 0 + import gate 净）。**⚠残留（翻闸/e2e，非静默）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名 `hmsClient.dropTable` 级联删 HMS 登记〔`HiveConnectorMetadata.dropDatabase` 无 per-source 分支、iceberg-on-HMS 同款、只删登记不删数据；翻闸若要 per-table 拒须网关加分流，S6 findings 已记〕；②写/DDL 只读端到端只在 HD-B2 分流 HUDI 后 live（e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud，memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 只余 HD-B2（getTableHandle 分流 HUDI，武装 pivot，最后一个连接器步，须 HD-B1✅+Group C✅+HD-D1✅ 全完＝现已满足）。下一步 = HD-B2。** - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 diff --git a/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md b/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md index ebf7bb58e9ab4f..c599af25c64d7b 100644 --- a/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md +++ b/plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md @@ -463,6 +463,16 @@ All Group C steps build a shared spine: a **pin field on `HudiTableHandle`** (e. ### Group D — Write-reject safety net (hudi is read-only) #### HD-D1 — Explicit read-only reject for hudi write / procedure / DDL +> **STATUS: DONE** (`7ae9404af25`, dormant). This "Lands" bullet was written BEFORE HD-B1 landed and is now +> partly stale: a hudi handle routes to the HUDI sibling (never iceberg), and the SPI defaults already reject all +> writes/procedures/DDL fail-loud. Signed-off deviations (user 2026-07-09): (Q1) do NOT make `getWritePlanProvider` +> throw — the admission gate `supportedWriteOperations` DERIVES from it, so a throw there misclassifies as an +> internal error; keep it `null` and reject explicitly only at `beginTransaction` (execution-stage, message-quality +> defense). (Q2) STRICT — reject ALL DDL including DROP/RENAME TABLE (a behavior change vs legacy HMS). Net code = +> one `beginTransaction` override + a `HudiReadOnlyWriteRejectTest` lock. Review `wf_36fd1c6c-e58`: 0 defects. +> Residual: `DROP DATABASE … FORCE` still name-cascades `hmsClient.dropTable` (legacy-parity, out of per-table +> scope). See HANDOFF ⭐ Hudi HD-D1 bullet. + - **Problem:** once HD-B2 diverts HUDI to a foreign handle, that handle is non-hive, so the gateway's per-handle `getWritePlanProvider(handle)` (`:142-146`), `getProcedureOps(handle)` (`:160-164`), and `supportedWriteOperations(handle)` (SPI default derives from `getWritePlanProvider(handle)`) — plus the From d93314b4008fc2ad30a72e5ce11a929602821cff Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 23:28:38 +0800 Subject: [PATCH 142/334] [feat](catalog) HMS cutover hudi HD-B2: getTableHandle HUDI divert (arming pivot, dormant) Add the HUDI arm to HiveConnectorMetadata.getTableHandle so a hudi-on-HMS table is served by the embedded hudi SIBLING connector, returning the sibling's OWN raw HudiTableHandle verbatim (never a HiveTableHandle stamped HUDI, which would CCE the moment the hudi scan/metadata path casts it). Mirrors the already-landed iceberg S4 divert exactly, reusing HD-A1's getOrCreateHudiSibling holder: - New NO_HUDI_SIBLING sentinel + hudiSiblingSupplier field + hudiSiblingMetadata(session) by-TYPE resolver (twins of the iceberg ones); the 6-arg ctor gains the hudi supplier (order: iceberg, hudi, ownerResolver); 3-arg convenience ctor passes NO_HUDI_SIBLING. - getTableHandle: `if (tableType == HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`, placed AFTER the iceberg arm (detector precedence: iceberg before hudi) and BEFORE the UNKNOWN fail-loud (the arm fires only on tableType==HUDI, so it cannot swallow the genuine-UNKNOWN throw). The foreign hudi handle is NEVER cast here. - HiveConnector.getMetadata wires this::getOrCreateHudiSibling as the hudi by-TYPE supplier. This is the pivot that lights up the 3-way guard-and-forward everywhere (HD-B1): a foreign hudi handle now starts flowing out of getTableHandle. Dormant until "hms" enters SPI_READY_TYPES (getTableHandle is never called for this connector today). Adversarial review (4 dims, per-finding refute): 0 production defects; 1 confirmed minor test-intent gap fixed -> HiveConnector.getMetadata's two same-typed Supplier args were transposable with no test driving the wiring. Extract a package-private newMetadata(HmsClient) seam (getMetadata delegates) so the wiring is unit-testable without getMetadata's eager ThriftHmsClient build; new HiveConnectorThreeWayRoutingTest guard pins each by-TYPE supplier to its own sibling (mutation-verified: transposition -> test fails). DivertTest rewritten with separate iceberg/hudi recording siblings: HUDI now diverts to the hudi sibling (assertSame), never iceberg; empty-passthrough proven to originate from the sibling; fail-loud symmetry; UNKNOWN-not-swallowed. hudi 240 tests + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 28 +++- .../connector/hive/HiveConnectorMetadata.java | 61 ++++++-- ...onnectorMetadataSiblingDelegationTest.java | 4 +- ...onnectorMetadataTableHandleDivertTest.java | 146 +++++++++++++----- .../HiveConnectorThreeWayRoutingTest.java | 30 ++++ 5 files changed, 205 insertions(+), 64 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index ac108df3325124..4967331355703b 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -106,14 +106,26 @@ public HiveConnector(Map properties, ConnectorContext context) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - // Two sibling seams for the metadata's guard-and-forward: - // - getOrCreateIcebergSibling (by-TYPE, force-build): only the getTableHandle ICEBERG divert uses it, to - // build+ask the iceberg sibling for a table detected as iceberg (no handle exists yet to route by). - // - resolveSiblingOwner (by-HANDLE, peek): every per-handle site routes a foreign handle to whichever - // ALREADY-BUILT sibling owns it. Passing the resolver (not a built sibling) keeps a pure-hive query - // from ever building/throwing a sibling. - return new HiveConnectorMetadata(getOrCreateClient(), properties, context, - this::getOrCreateIcebergSibling, this::resolveSiblingOwner); + return newMetadata(getOrCreateClient()); + } + + /** + * Builds the connector's metadata with its sibling seams wired in. Extracted (package-private) from + * {@link #getMetadata} so a unit test can assert the wiring WITHOUT {@link #getOrCreateClient()} building a + * real ThriftHmsClient (whose Hadoop stack is absent from connector unit tests). The seams: + *

      + *
    • getOrCreateIcebergSibling / getOrCreateHudiSibling (by-TYPE, force-build): only the getTableHandle + * ICEBERG / HUDI diverts use them, to build+ask the matching sibling for a table detected as iceberg/hudi + * (no handle exists yet to route by). These two args share the static type {@code Supplier}, + * so a transposition would compile clean — HiveConnectorThreeWayRoutingTest pins the pairing.
      • + *
    • resolveSiblingOwner (by-HANDLE, peek): every per-handle site routes a foreign handle to whichever + * ALREADY-BUILT sibling owns it. Passing the resolver (not a built sibling) keeps a pure-hive query from + * ever building/throwing a sibling.
      • + *
    + */ + HiveConnectorMetadata newMetadata(HmsClient client) { + return new HiveConnectorMetadata(client, properties, context, + this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwner); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 9f8e1b23e748ae..b1dc73526b8b21 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -173,6 +173,13 @@ public class HiveConnectorMetadata implements ConnectorMetadata { throw new DorisConnectorException("no iceberg sibling connector configured for this hive metadata"); }; + // The hudi analog of NO_ICEBERG_SIBLING installed by the 3-arg constructor (hive-only construction). Invoked + // only when a HUDI table is diverted BY TYPE — which such a construction never triggers — so it fails loud + // instead of NPEing. + private static final Supplier NO_HUDI_SIBLING = () -> { + throw new DorisConnectorException("no hudi sibling connector configured for this hive metadata"); + }; + // The by-handle owner resolver installed by the 3-arg constructor (hive-only construction). Invoked only when // a NON-hive handle reaches a per-handle guard-and-forward site — which such a construction never does — so it // fails loud instead of NPEing. @@ -194,6 +201,11 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // triggers it. Used ONLY through the parent-first Connector / ConnectorMetadata interfaces — its concrete // iceberg types are never cast here (cross-loader CCE). private final Supplier icebergSiblingSupplier; + // The hudi analog of icebergSiblingSupplier: supplies the embedded hudi SIBLING connector BY TYPE, for the + // getTableHandle HUDI divert only (a hudi-detected table has no handle yet to route by). Lazy via + // HiveConnector.getOrCreateHudiSibling; used ONLY through the parent-first Connector / ConnectorMetadata + // interfaces — its concrete hudi types are never cast here (cross-loader CCE). + private final Supplier hudiSiblingSupplier; // Resolves the embedded sibling connector that OWNS a foreign (non-hive) table handle, for the per-handle // guard-and-forward methods below. Backed by HiveConnector.resolveSiblingOwner (a 3-way ownsHandle dispatch // over the ALREADY-BUILT iceberg / hudi siblings). Used ONLY through the parent-first Connector / @@ -201,17 +213,19 @@ public class HiveConnectorMetadata implements ConnectorMetadata { private final Function siblingOwnerResolver; public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context) { - this(hmsClient, properties, context, NO_ICEBERG_SIBLING, NO_SIBLING_OWNER); + this(hmsClient, properties, context, NO_ICEBERG_SIBLING, NO_HUDI_SIBLING, NO_SIBLING_OWNER); } public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context, Supplier icebergSiblingSupplier, + Supplier hudiSiblingSupplier, Function siblingOwnerResolver) { this.hmsClient = hmsClient; this.properties = properties; this.context = context; this.typeMappingOptions = buildTypeMappingOptions(properties); this.icebergSiblingSupplier = icebergSiblingSupplier; + this.hudiSiblingSupplier = hudiSiblingSupplier; this.siblingOwnerResolver = siblingOwnerResolver; } @@ -222,11 +236,30 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties * live on the single (memoized) sibling connector, so this is cheap. The returned metadata and any handle it * produces are used ONLY through the parent-first SPI interfaces and MUST NOT be cast (the sibling's concrete * iceberg types would CCE across the loader split). + * + *

    Package-private (not private) so HiveConnectorThreeWayRoutingTest can assert that + * {@link HiveConnector#getMetadata} wires the iceberg by-TYPE supplier to THIS arm (the two same-typed + * supplier args are otherwise transposable at that sole production wiring point). */ - private ConnectorMetadata icebergSiblingMetadata(ConnectorSession session) { + ConnectorMetadata icebergSiblingMetadata(ConnectorSession session) { return icebergSiblingSupplier.get().getMetadata(session); } + /** + * The embedded hudi sibling's metadata resolved BY TYPE, for the getTableHandle HUDI divert only (a + * hudi-detected table has no handle yet, so the sibling is force-built and asked directly). Same lifecycle and + * casting contract as {@link #icebergSiblingMetadata}: obtained fresh per call, used ONLY through the + * parent-first SPI interfaces, and never cast (the sibling's concrete hudi types would CCE across the loader + * split). + * + *

    Package-private (not private) so HiveConnectorThreeWayRoutingTest can assert that + * {@link HiveConnector#getMetadata} wires the hudi by-TYPE supplier to THIS arm (see + * {@link #icebergSiblingMetadata}). + */ + ConnectorMetadata hudiSiblingMetadata(ConnectorSession session) { + return hudiSiblingSupplier.get().getMetadata(session); + } + /** * The OWNING sibling's metadata for a foreign (non-hive) table handle, resolved BY HANDLE (3-way ownsHandle * dispatch over the already-built iceberg / hudi siblings — see HiveConnector.resolveSiblingOwner). Every @@ -271,19 +304,23 @@ public Optional getTableHandle( } HmsTableInfo tableInfo = hmsClient.getTable(dbName, tableName); HiveTableType tableType = HiveTableFormatDetector.detect(tableInfo); - // Iceberg-on-HMS divert: an iceberg table registered in this HMS catalog is served by the embedded - // iceberg SIBLING connector, not by hive. Return the sibling's OWN table handle (the raw foreign iceberg - // handle) verbatim — NOT a HiveTableHandle stamped ICEBERG — so iceberg's scan/metadata path, which - // unconditionally casts the handle to its concrete IcebergTableHandle, succeeds. This is the pivot that - // activates the guard-and-forward overrides throughout this class: every gateway consumer discriminates by - // `instanceof HiveTableHandle` (the gateway's OWN hive-loader type) and forwards any non-hive handle to - // the sibling; the foreign iceberg handle is NEVER cast here (its concrete type is invisible across the - // classloader split). HUDI is intentionally NOT diverted — hudi-on-HMS delegation is a later substep, so a - // hudi table stays on the hive-handle path below (dormant, unchanged). Dormant overall until hms enters - // SPI_READY_TYPES: today getTableHandle is never called for this connector. + // Foreign-handle divert: an iceberg-on-HMS or hudi-on-HMS table registered in this HMS catalog is served by + // the embedded iceberg / hudi SIBLING connector, not by hive. Return the sibling's OWN table handle (the + // raw foreign iceberg/hudi handle) verbatim — NOT a HiveTableHandle stamped ICEBERG/HUDI — so the sibling's + // scan/metadata path, which unconditionally casts the handle to its concrete IcebergTableHandle / + // HudiTableHandle, succeeds. This is the pivot that activates the guard-and-forward overrides throughout + // this class: every gateway consumer discriminates by `instanceof HiveTableHandle` (the gateway's OWN + // hive-loader type) and forwards any non-hive handle to whichever sibling OWNS it (3-way ownsHandle + // dispatch); the foreign handle is NEVER cast here (its concrete type is invisible across the classloader + // split). Iceberg is checked before hudi, matching the detector's own precedence (a table carrying both + // resolves iceberg). Dormant overall until hms enters SPI_READY_TYPES: today getTableHandle is never called + // for this connector. if (tableType == HiveTableType.ICEBERG) { return icebergSiblingMetadata(session).getTableHandle(session, dbName, tableName); } + if (tableType == HiveTableType.HUDI) { + return hudiSiblingMetadata(session).getTableHandle(session, dbName, tableName); + } // Fail-loud parity with legacy HMSExternalTable.supportedHiveTable(), which threw on a null or // unrecognized input format instead of silently degrading (the old detector returned UNKNOWN). A view // short-circuits: legacy returns true for a view before the format check — a view has no data files so diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 27613586e55a18..7fa8147622a663 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -101,7 +101,7 @@ private static final class ForeignHandle implements ConnectorTableHandle { */ private HiveConnectorMetadata withSibling() { return new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), - SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector); + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector); } private HiveTableHandle hiveHandle() { @@ -315,7 +315,7 @@ public void beginTransactionForHiveHandleOpensHiveTxnAndNeverConsultsSibling() { // sibling. The selection must be symmetric — hive and iceberg write plans downcast to different types. ConnectorTransaction hiveTxn = new NoOpConnectorTransaction(70099L, "HIVE"); HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), - SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector) { + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, handle -> siblingConnector) { @Override public ConnectorTransaction beginTransaction(ConnectorSession session) { return hiveTxn; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java index 2872782d8f1d2c..a8996a9ca9c20c 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataTableHandleDivertTest.java @@ -34,23 +34,27 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.function.Function; /** - * Pins the HMS-cutover §4.4 S4 pivot: {@link HiveConnectorMetadata#getTableHandle} diverts an ICEBERG table to the - * embedded iceberg sibling connector — returning the sibling's OWN (foreign iceberg) handle verbatim — while HIVE - * and HUDI tables keep building a {@link HiveTableHandle} on the hive path and never touch the sibling. + * Pins the HMS-cutover foreign-handle diverts in {@link HiveConnectorMetadata#getTableHandle}: an ICEBERG table is + * diverted to the embedded iceberg sibling and a HUDI table (HD-B2, the arming pivot) to the embedded hudi sibling — + * each returning that sibling's OWN (foreign) handle verbatim — while a plain HIVE table keeps building a + * {@link HiveTableHandle} on the hive path and never touches either sibling. * - *

    WHY (Rule 9): S4 is the step that makes a foreign iceberg handle START flowing out of getTableHandle, which is - * what activates every {@code instanceof HiveTableHandle} guard-and-forward override in {@link HiveConnectorMetadata} - * (S3). The contract that must hold BEFORE the flip wires it:

    + *

    WHY (Rule 9): these diverts are what makes a foreign iceberg/hudi handle START flowing out of getTableHandle, + * which activates every {@code instanceof HiveTableHandle} guard-and-forward override in {@link HiveConnectorMetadata} + * (routed 3-way by the owning sibling). The contract that must hold BEFORE the flip wires it:

    *
      - *
    • An iceberg-on-HMS table must return the iceberg connector's handle unchanged (so iceberg's own - * unconditional {@code (IcebergTableHandle)} cast on the scan/metadata path succeeds) — NOT a HiveTableHandle - * stamped ICEBERG, which would CCE the moment iceberg cast it.
      • - *
    • HUDI is deliberately left on the hive-handle path (its delegation is a later substep); diverting it now - * would silently reroute hudi reads to iceberg.
      • - *
    • A pure-hive table must resolve entirely on the hive path, never building the sibling (a hive-only - * deployment has no iceberg plugin).
      • + *
    • An iceberg-on-HMS table must return the iceberg connector's handle unchanged, and a hudi-on-HMS table the + * hudi connector's handle unchanged — so each sibling's own unconditional concrete-handle cast on its + * scan/metadata path succeeds — NOT a HiveTableHandle stamped ICEBERG/HUDI, which would CCE the moment the + * sibling cast it.
      • + *
    • Routing must be BY TYPE and to the RIGHT sibling: a HUDI table must reach the hudi sibling, never the iceberg + * one (diverting it to iceberg would CCE / silently mis-read hudi data).
      • + *
    • A plain-hive table must resolve entirely on the hive path, never building either sibling (a hive-only + * deployment has no iceberg or hudi plugin).
      • + *
    • The HUDI arm must not swallow the genuine-UNKNOWN fail-loud (an unsupported non-view format still throws).
      • *
    * *

    Dormant until hms enters {@code SPI_READY_TYPES}: no production path calls getTableHandle for this connector @@ -60,80 +64,127 @@ public class HiveConnectorMetadataTableHandleDivertTest { private static final String PARQUET = "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; private static final String HUDI = "org.apache.hudi.hadoop.HoodieParquetInputFormat"; + private static final String UNKNOWN_FORMAT = "com.example.NotAHiveOrHudiOrIcebergInputFormat"; - /** The foreign (non-hive) handle the iceberg sibling's getTableHandle produces post-flip. */ + // getTableHandle routes BY TYPE (the two by-TYPE suppliers), NEVER via the by-handle owner resolver — so wire + // the owner resolver to fail loud if anything reaches for it. + private static final Function OWNER_RESOLVER_UNUSED = handle -> { + throw new AssertionError("getTableHandle must divert BY TYPE, never via the by-handle owner resolver"); + }; + + /** The foreign (non-hive) handle a sibling's getTableHandle produces post-flip. */ private static final class ForeignHandle implements ConnectorTableHandle { } - private final ForeignHandle siblingHandle = new ForeignHandle(); - private final RecordingSibling siblingConnector = new RecordingSibling(siblingHandle); + private final ForeignHandle icebergHandle = new ForeignHandle(); + private final ForeignHandle hudiHandle = new ForeignHandle(); + private final RecordingSibling icebergSibling = new RecordingSibling(icebergHandle); + private final RecordingSibling hudiSibling = new RecordingSibling(hudiHandle); @Test - public void icebergTableDivertsAndReturnsSiblingHandleUnmodified() { - HiveConnectorMetadata md = withSibling(icebergTable()); + public void icebergTableDivertsToIcebergSiblingNotHudi() { + HiveConnectorMetadata md = withSiblings(icebergTable()); Optional handle = md.getTableHandle(null, "db", "t"); Assertions.assertTrue(handle.isPresent(), "an existing iceberg-on-HMS table must resolve a handle"); - Assertions.assertSame(siblingHandle, handle.get(), + Assertions.assertSame(icebergHandle, handle.get(), "getTableHandle must return the iceberg sibling's OWN handle unmodified (a rewrap poisons the " + "downstream iceberg cast)"); Assertions.assertFalse(handle.get() instanceof HiveTableHandle, "an iceberg table must NOT be served a HiveTableHandle stamped ICEBERG"); - Assertions.assertEquals(1, siblingConnector.metadata.getTableHandleCalls, - "the divert must consult the sibling exactly once"); + Assertions.assertEquals(1, icebergSibling.metadata.getTableHandleCalls, + "the divert must consult the iceberg sibling exactly once"); + Assertions.assertEquals(0, hudiSibling.getMetadataCalls, + "an iceberg table must NEVER build or consult the hudi sibling"); } @Test public void icebergDivertPropagatesSiblingEmpty() { // The sibling is authoritative for iceberg existence: if its getTableHandle returns empty (e.g. the // iceberg catalog cannot load it), the gateway forwards that empty rather than fabricating a hive handle. - siblingConnector.metadata.returnHandle = null; - HiveConnectorMetadata md = withSibling(icebergTable()); + icebergSibling.metadata.returnHandle = null; + HiveConnectorMetadata md = withSiblings(icebergTable()); Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), "an empty from the sibling must pass through unchanged"); // Prove the empty was FORWARDED from the sibling (its getTableHandle was consulted), not short-circuited // to empty by the gateway — otherwise a broken `if (ICEBERG) return Optional.empty()` would pass this test. - Assertions.assertEquals(1, siblingConnector.metadata.getTableHandleCalls, + Assertions.assertEquals(1, icebergSibling.metadata.getTableHandleCalls, "the sibling is authoritative for iceberg existence: its getTableHandle must be the source of empty"); } + @Test + public void hudiTableDivertsToHudiSiblingNotIceberg() { + // HD-B2 arming pivot: a hudi-on-HMS table is diverted to the hudi sibling and returns the hudi sibling's + // OWN foreign handle verbatim — NOT a HiveTableHandle stamped HUDI, and NOT routed to the iceberg sibling. + HiveConnectorMetadata md = withSiblings(hiveTable(HUDI)); + + Optional handle = md.getTableHandle(null, "db", "t"); + + Assertions.assertTrue(handle.isPresent(), "an existing hudi-on-HMS table must resolve a handle"); + Assertions.assertSame(hudiHandle, handle.get(), + "getTableHandle must return the hudi sibling's OWN handle unmodified (a rewrap poisons the " + + "downstream hudi cast)"); + Assertions.assertFalse(handle.get() instanceof HiveTableHandle, + "a hudi table must NOT be served a HiveTableHandle stamped HUDI"); + Assertions.assertEquals(1, hudiSibling.metadata.getTableHandleCalls, + "the divert must consult the hudi sibling exactly once"); + Assertions.assertEquals(0, icebergSibling.getMetadataCalls, + "a hudi table must NEVER be diverted to the iceberg sibling"); + } + + @Test + public void hudiDivertPropagatesSiblingEmpty() { + // The hudi sibling is authoritative for hudi existence: an empty from it passes through, and it must be + // the SOURCE of that empty (its getTableHandle was consulted), not a gateway short-circuit. + hudiSibling.metadata.returnHandle = null; + HiveConnectorMetadata md = withSiblings(hiveTable(HUDI)); + + Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), + "an empty from the hudi sibling must pass through unchanged"); + Assertions.assertEquals(1, hudiSibling.metadata.getTableHandleCalls, + "the hudi sibling is authoritative for hudi existence: its getTableHandle must be the source of empty"); + } + @Test public void hiveTableBuildsHiveHandleWithoutConsultingSibling() { - HiveConnectorMetadata md = withSibling(hiveTable(PARQUET)); + HiveConnectorMetadata md = withSiblings(hiveTable(PARQUET)); Optional handle = md.getTableHandle(null, "db", "t"); Assertions.assertTrue(handle.get() instanceof HiveTableHandle, "a hive table resolves a HiveTableHandle"); Assertions.assertEquals(HiveTableType.HIVE, ((HiveTableHandle) handle.get()).getTableType()); - Assertions.assertEquals(0, siblingConnector.getMetadataCalls, + Assertions.assertEquals(0, icebergSibling.getMetadataCalls, "a hive table must never build or consult the iceberg sibling"); + Assertions.assertEquals(0, hudiSibling.getMetadataCalls, + "a hive table must never build or consult the hudi sibling"); } @Test - public void hudiTableStaysOnHiveHandlePathNotDiverted() { - // HUDI is NOT diverted in S4: it keeps building a HUDI-stamped HiveTableHandle (its own delegation is a - // later substep). Diverting it here would silently reroute hudi reads to the iceberg sibling. - HiveConnectorMetadata md = withSibling(hiveTable(HUDI)); - - Optional handle = md.getTableHandle(null, "db", "t"); + public void unknownNonViewTableStillFailsLoud() { + // The HUDI arm sits BEFORE the UNKNOWN fail-loud and fires only on tableType == HUDI, so a genuine-UNKNOWN + // non-view table is still rejected (not swallowed into a hudi divert). + HiveConnectorMetadata md = withSiblings(hiveTable(UNKNOWN_FORMAT)); - Assertions.assertTrue(handle.get() instanceof HiveTableHandle, "a hudi table stays on the hive path"); - Assertions.assertEquals(HiveTableType.HUDI, ((HiveTableHandle) handle.get()).getTableType()); - Assertions.assertEquals(0, siblingConnector.getMetadataCalls, - "a hudi table must NOT be diverted to the iceberg sibling"); + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"), + "an unsupported non-view input format must fail loud, not be diverted to a sibling"); + Assertions.assertEquals(0, hudiSibling.getMetadataCalls, + "the UNKNOWN fail-loud must not consult the hudi sibling"); + Assertions.assertEquals(0, icebergSibling.getMetadataCalls, + "the UNKNOWN fail-loud must not consult the iceberg sibling"); } @Test public void missingTableReturnsEmptyWithoutConsultingSibling() { HiveConnectorMetadata md = new HiveConnectorMetadata( new FakeHmsClient(icebergTable(), false), Collections.emptyMap(), new FakeConnectorContext(), - () -> siblingConnector, handle -> siblingConnector); + () -> icebergSibling, () -> hudiSibling, OWNER_RESOLVER_UNUSED); Assertions.assertFalse(md.getTableHandle(null, "db", "t").isPresent(), "a non-existent table short-circuits to empty before any format detection or divert"); - Assertions.assertEquals(0, siblingConnector.getMetadataCalls, "a missing table must not build the sibling"); + Assertions.assertEquals(0, icebergSibling.getMetadataCalls, "a missing table must not build the iceberg sibling"); + Assertions.assertEquals(0, hudiSibling.getMetadataCalls, "a missing table must not build the hudi sibling"); } @Test @@ -147,13 +198,24 @@ public void icebergTableFailsLoudWhenNoSiblingConfigured() { "an iceberg table with no sibling configured must fail loud"); } + @Test + public void hudiTableFailsLoudWhenNoSiblingConfigured() { + // Symmetric with iceberg: the 3-arg constructor installs a fail-loud hudi sibling supplier, so a hudi + // table must raise a clear error, not NPE, when the hudi plugin is unavailable. + HiveConnectorMetadata md = new HiveConnectorMetadata( + new FakeHmsClient(hiveTable(HUDI), true), Collections.emptyMap(), new FakeConnectorContext()); + + Assertions.assertThrows(DorisConnectorException.class, () -> md.getTableHandle(null, "db", "t"), + "a hudi table with no sibling configured must fail loud"); + } + // ===== helpers ===== - private HiveConnectorMetadata withSibling(HmsTableInfo tableInfo) { - // getTableHandle diverts iceberg BY TYPE (icebergSiblingSupplier); the by-handle owner resolver is unused - // in this suite, so it just returns the recording sibling. + private HiveConnectorMetadata withSiblings(HmsTableInfo tableInfo) { + // getTableHandle diverts iceberg/hudi BY TYPE (the two by-TYPE suppliers); the by-handle owner resolver is + // never used on this path, so it fails loud if anything reaches for it. return new HiveConnectorMetadata(new FakeHmsClient(tableInfo, true), Collections.emptyMap(), - new FakeConnectorContext(), () -> siblingConnector, handle -> siblingConnector); + new FakeConnectorContext(), () -> icebergSibling, () -> hudiSibling, OWNER_RESOLVER_UNUSED); } private static HmsTableInfo hiveTable(String inputFormat) { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java index f937d4e5f909f6..e127cac59229c2 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorThreeWayRoutingTest.java @@ -175,6 +175,36 @@ public void icebergHandleRoutesWithoutBuildingHudiSibling() { "routing an iceberg handle must NOT build (or require) the hudi sibling"); } + @Test + public void getMetadataWiresEachByTypeSupplierToItsOwnSibling() { + // The by-HANDLE routing above is one brain (resolveSiblingOwner). getTableHandle instead diverts BY TYPE + // (no handle exists yet), and HiveConnector.newMetadata (extracted from getMetadata) is the SOLE production + // point that wires those two by-TYPE suppliers: `this::getOrCreateIcebergSibling, + // this::getOrCreateHudiSibling`. They share the static type Supplier, so a transposition compiles + // clean and would silently send getTableHandle's ICEBERG arm to the hudi sibling (and vice versa) at flip. + // The per-handle DivertTest builds the metadata DIRECTLY with hand-labeled lambdas and cannot observe this + // order — guard it here, at the real wiring. newMetadata(null) exercises that wiring without getMetadata's + // eager ThriftHmsClient build (whose Hadoop stack is absent from unit tests); the null client is never + // dereferenced because only the by-TYPE sibling arms are driven. + RoutingSibling iceberg = new RoutingSibling("iceberg", IcebergLikeHandle.class); + RoutingSibling hudi = new RoutingSibling("hudi", HudiLikeHandle.class); + TwoSiblingContext ctx = new TwoSiblingContext(iceberg, hudi); + HiveConnector connector = new HiveConnector(props(), ctx); + + HiveConnectorMetadata md = connector.newMetadata(null); + + // The getTableHandle ICEBERG arm resolves BY TYPE via icebergSiblingMetadata, which force-builds the + // iceberg sibling (createSiblingConnector("iceberg")). A transposed getMetadata would build hudi here. + md.icebergSiblingMetadata(null); + Assertions.assertEquals(1, ctx.icebergBuilds, "the iceberg by-TYPE arm must force-build the iceberg sibling"); + Assertions.assertEquals(0, ctx.hudiBuilds, "the iceberg by-TYPE arm must NOT build the hudi sibling"); + + // Symmetrically the HUDI arm must resolve the hudi sibling, not rebuild iceberg. + md.hudiSiblingMetadata(null); + Assertions.assertEquals(1, ctx.hudiBuilds, "the hudi by-TYPE arm must force-build the hudi sibling"); + Assertions.assertEquals(1, ctx.icebergBuilds, "the hudi by-TYPE arm must not rebuild the iceberg sibling"); + } + private static Map props() { Map props = new HashMap<>(); props.put("hive.metastore.uris", METASTORE_URI); From c77a7483c6cf6efa8c86b77ee55476365a4c6785 Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 23:30:17 +0800 Subject: [PATCH 143/334] [doc](catalog) HMS cutover hudi HD-B2 DONE (getTableHandle HUDI divert, arming pivot) All dormant connector steps now complete: Group A/B/C/D + iceberg S0-S6/W1-W5 + write-chain WC1-WC4. Only the atomic flip itself remains (add "hms" to SPI_READY_TYPES + cyclic-dependency deletion units + flip e2e/hard gates), which needs a user packaging decision (single PR vs split PR + flip ordering, design doc D4 deferred to flip time). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 34fd6886a7d364..a2bfc926162393 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完 ⇒ 只余 HD-B2（getTableHandle 分流 HUDI，武装 pivot，最后一个连接器步）。下一步 = HD-B2。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸 e2e/硬门）。下一步 = 翻闸打包决策（设计文档 §2/§5/§7，须用户拍板单 PR vs 分 PR + 翻闸序）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 @@ -101,7 +101,7 @@ - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；统一对抗复审 DONE（`wf_33882086-fbb`）+修 `ad8aa87bd2d`（见顶部 🎯）；下一步 = Part 2 C5a。** - **✅ HD-C5a DONE（`af9a3c1a004`，休眠，Part 2 首步）= schema-at-instant 列表（3-arg `getTableSchema` 覆写）**。post-flip 通用路径下给 `FOR TIME AS OF` 演进读供**钉时刻**的列表，闭合 HD-C2 残留 #1（SPI 默认曾返最新、忽略 pin）。①新 3-arg `getTableSchema(session,handle,ConnectorMvccSnapshot)` 覆写 keyed off `HudiTableHandle.getQueryInstant()`（`applySnapshot` 钉的 pin），**非** `snapshot.getSchemaId()`（hudi 恒 -1；fe-core `PluginDrivenMvccExternalTable:375-376` 传 post-applySnapshot 句柄故 instant 已在其上）；null instant（plain / 非-FOR-TIME pin 如 @incr，其 loadSnapshot 分支自列最新 schema、根本不到此）经**同一** shared `buildTableSchema(handle,instant)` 走 latest→latest/at-instant 不发散。②`getSchemaFromMetaClient(basePath)`→`getSchemaFromMetaClient(basePath,queryInstant)`，**包进 `metaClientExecutor.execute`**（plugin UGI doAs+TCCL pin）闭合 C4b 欠的未包裹 auth/TCCL gap（getTableSchema 可跑在非 pin 的 scan 线程外，如 catalog metadata load）；at-instant 臂字节忠实 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`+`HMSExternalTable.initHudiSchema`：`reloadActiveTimeline()`+`getTableInternalSchemaFromCommitMetadata(instant)`→present（schema.on.read 开）经 `columnsFromInternalSchema`（`AvroInternalSchemaConverter.convert(InternalSchema,name)`→avro→`avroSchemaToColumns`→`attachTopLevelFieldIds` 稳定 id）→改名列显历史名、BE 按 field id 匹老文件；absent（演进关）落 latest fallback＝非演进表字节等价（D3）。③用 `HudiScanPlanProvider.buildMetaClient`（与旧 inline build 字节等价）；hive 3-arg 委派已发（`HiveConnectorMetadata:1071`）无须改；连接器编 unshaded hudi-common 故 `convert` 回 unshaded `org.apache.avro.Schema` 与 `avroSchemaToColumns` 相容。零 fe-core import、零新连接器-api SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、句柄未分流；live iceberg/paimon 不走 hudi getTableSchema）。at-instant **读**本身 e2e；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading（2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〔snapshot 传 null 证之〕）。**4 维对抗复审 `wf_703ea26e-49f`（legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent）+逐条 refute：0 确认缺陷**（1 nit＝wrapping 后 LOG.warn 只印 wrapper message refuted〔旧码亦只印 `e.getMessage()` 非本步引入〕；顺带把 `e.getMessage()`→传 throwable 补全 cause 链+栈）。全绿（hudi 156 测〔+3〕 + checkstyle 0 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进列表（改名/调序显历史名、非演进 latest 等价、meta 列集与 legacy 一致）只 e2e 可证（单测覆盖 routing）；schema-on-read commit-metadata InternalSchema 是否含 `_hoodie_*` meta 列＝随 legacy 定、e2e 核。**下一步 = C5b（扫描侧 MOR/JNI 列表@instant + 字典 `-1` 条按 pinned schema 重解）。** - **✅ HD-C5b DONE（`e0da87888e5`，休眠，Part 2 收官）= 扫描侧 schema-at-instant（MOR/JNI 列表 + 原生 `-1` overlay @instant）**。`FOR TIME AS OF` over schema-on-read 表：两个扫描侧 schema locus 都须钉时刻、锁步。**(i) planScan JNI（MOR-realtime）`column_names`/`column_types`**＝pin 时刻的 schema（legacy `HudiScanNode:222-224` `getSchemaCacheValue(hmsTable,queryInstant)`），经新 `HudiSchemaUtils.resolveJniColumnSchema`（路由）→纯 `chooseJniSchema`（pinned present→`AvroInternalSchemaConverter.convert(pinnedIS,"hudi_table")`；else→latestAvro）。**plain read（`queryInstant==null`）短路在 `resolveInternalSchemaAtInstant` 之前→不 reload timeline＝字节不变**；非演进（instant 无 commit-metadata InternalSchema）→latest==at-instant（D3）。`avroSchema` 局部保 latest＝per-file schema_id resolver 的 base（分类每文件自身写入版本，pin-independent）。**(ii) getScanNodeProperties 原生 `-1` overlay** 从**完整 pinned InternalSchema** 重建（新 `buildSchemaEvolutionDictAtInstant`＝`buildSchemaEvolutionDict(emptyList, pinnedSchema, true, allHistoricalSchemas)`＝empty-requested 路径 over pinned schema），**非** latest-keyed 请求句柄。**为何 full-pinned**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:1534`＝latest-keyed 句柄图），FOR TIME 下 scan slots 按 pinned schema（C5a 3-arg，`PluginDrivenMvccExternalTable:376`）绑＝pinned 名；`buildColumnHandles` 用 slot(pinned 名) 查 latest 句柄图→改名列 miss→**从 `columns` 掉**；用 `columns` 建 overlay 会漏该 scan slot→BE 崩。**BE 契约亲手核对**（决定性）：v1 reader `by_table_field_id` 建 StructNode 每 overlay 字段一子、reader 按 **scan slot 名** `children_column_exists`＝`DCHECK(children.contains)`/`children.at`（`table_schema_change_helper.h:165-167`）→**scan slot 缺 overlay＝std::out_of_range/SIGABRT；overlay⊇scan slots＝必需**；format_v2 `find_external_root_field`（`table_reader.cpp:254-283`）按名查投影列、忽略多余＝**超集容忍**。⇒ full-pinned（全 pinned 字段）是 scan slots 超集＝双 reader 安全，用句柄建的子集会崩。`current_schema_id` 保 -1（须≠每 split id≥0）；史=全版本（Option B，C4d 同）；**无 meta 列可达 pin 路**（C5a schema-on-read 绑定源自 InternalSchema、无 `_hoodie_*`→查询绑不到 meta 列）。@incr（begin/endInstant 非 queryInstant）→pin==null→latest dict＝列最新 schema（§7.7）。零 fe-core、零新 SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、hudi 句柄未分流）。**4 维对抗复审 `wf_ec6d47e3-37e`（legacy-parity·at-instant / BE-contract·SIGABRT / no-regression·dormancy / test-intent）+逐条 refute：0 生产缺陷**（3 raised→1 survived〔confirmed-minor 测注释诚实性：`atInstantOverlayCarriesFullPinnedSchemaWithHistoricalNames` 测直调 `buildSchemaEvolutionDict(emptyList,…)`、守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝不可同加载器测、e2e-owed，已重写注释诚实标注〕→已修；2 refuted〔另一测覆盖提醒/设计文档措辞 nit〕）。`HudiSchemaUtilsTest`+3（JNI no-pin no-op〔null 短路不碰 metaClient〕/ chooseJniSchema pinned 历史名 / full-pinned overlay 历史名+全版本）。全绿（hudi 159 测 + checkstyle 0〔最长注释 116<130〕 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、非演进 FOR TIME latest 等价、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing**（`if(pinnedSchema.isPresent())` 分支 + `resolveInternalSchemaAtInstant` reload/resolve）**只 e2e 可证**（同加载器无 live metaClient；离线测覆盖纯 `chooseJniSchema`/full-pinned 组装）。**⇒ Part 2（C5a–C5b）+ HD-C4/C5 schema evolution（DV-006）全部收官。下一步 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步）。** - - **✅ HD-D1 DONE（`7ae9404af25`，休眠，Group D 收官）= 只读写-拒安全网**（recon 对照真实代码校正规划：**规划 §HD-D1「让 getWritePlanProvider 抛显式只读」写于 HD-B1 之前、已过时**）。翻闸后 hudi 表→通用 `PluginDrivenExternalTable`+外来句柄→网关三路分流（`resolveSiblingOwner` peek）到 **hudi 兄弟连接器**（非 iceberg），故本连接器答所有写/过程/DDL/事务调用。**核心发现：拒绝主体已现成**——HD-B1 正确路由（hudi→hudi 兄弟、绝不 iceberg）+ connector-api SPI 默认：`getWritePlanProvider(handle)`=null→`supportedWriteOperations(handle)`=空→`PhysicalPlanTranslator:657/709` admission gate **规划期**抛干净 `AnalysisException`「does not support INSERT/row-level DML」；`getProcedureOps`=null→EXECUTE 拒；`ConnectorTableOps` 全 DDL 默认抛「… not supported」；`beginTransaction` 默认抛。**用户签字 Q1=方案 A**（保留干净拒绝+测试锁定+事务层防御，**不**让 getWritePlanProvider 抛——admission gate 由它派生，在其上抛→引擎误分类「maybe palo bug」而非干净「does not support INSERT」；已核对 fe-core 写路径顺序：gate 先于 getWritePlanProvider 先于 beginTransaction，且连接器 beginTransaction 只在**执行期**〔`PluginDrivenInsertExecutor`/`ConnectorRewriteDriver:130`〕被调、规划期已拒故对 hudi 写是死码）。**用户签字 Q2=方案乙**（严格拒绝**所有** DDL 含 `DROP TABLE`/`RENAME TABLE`——相对 legacy HMS〔经 `HiveMetadataOps` 支持删/改名 hudi 表 HMS 条目〕是**有意行为倒退**、hudi 完全外部管理）。**净新增（唯一代码改动）=override `HudiConnectorMetadata.beginTransaction(session)` 抛显式** `"Hudi tables are read-only in this catalog; INSERT/UPDATE/DELETE/MERGE are not supported"`（替换泛型默认；per-handle 重载 SPI 默认委派到它；纯消息质量末线兜底、不劣化 gate 分类）。`HudiReadOnlyWriteRejectTest`〔`new HudiConnectorMetadata(null,{},null)`＝write/DDL 抛前不 deref 故 null 够〕锁：无 writer+空写操作集〔=写不被误当 iceberg 执行〕/无过程/beginTransaction **双重载**〔no-arg + 生产路由的 per-handle〕均抛显式只读/全 DDL 变更族〔dropTable·renameTable·truncate + addColumn(s)·dropColumn·renameColumn·modifyColumn·reorderColumns·create/drop Branch|Tag·add/drop/replace PartitionField〕抛。**3 维对抗复审 `wf_36fd1c6c-e58`（reject-completeness·routing / classification·no-regression / test-intent）+逐条 refute：0 生产缺陷**；6 raised→2 survived nit（均测强化：per-handle beginTransaction 重载直测 + 全 DDL 族补锁，已修）；refuted：`DROP DATABASE … FORCE` 级联〔pre-existing 网关码、legacy 平价、超 per-table 范围〕/ 休眠〔HD-B2 未分流 HUDI＝设计延后〕/ beginTransaction 非可写性锁〔可写性由 `supportedWriteOperations` 空锁，已断言〕。全绿（hudi 163 测 + checkstyle 0 + import gate 净）。**⚠残留（翻闸/e2e，非静默）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名 `hmsClient.dropTable` 级联删 HMS 登记〔`HiveConnectorMetadata.dropDatabase` 无 per-source 分支、iceberg-on-HMS 同款、只删登记不删数据；翻闸若要 per-table 拒须网关加分流，S6 findings 已记〕；②写/DDL 只读端到端只在 HD-B2 分流 HUDI 后 live（e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud，memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 只余 HD-B2（getTableHandle 分流 HUDI，武装 pivot，最后一个连接器步，须 HD-B1✅+Group C✅+HD-D1✅ 全完＝现已满足）。下一步 = HD-B2。** + - **✅ HD-D1 DONE（`7ae9404af25`，休眠，Group D 收官）= 只读写-拒安全网**（recon 对照真实代码校正规划：**规划 §HD-D1「让 getWritePlanProvider 抛显式只读」写于 HD-B1 之前、已过时**）。翻闸后 hudi 表→通用 `PluginDrivenExternalTable`+外来句柄→网关三路分流（`resolveSiblingOwner` peek）到 **hudi 兄弟连接器**（非 iceberg），故本连接器答所有写/过程/DDL/事务调用。**核心发现：拒绝主体已现成**——HD-B1 正确路由（hudi→hudi 兄弟、绝不 iceberg）+ connector-api SPI 默认：`getWritePlanProvider(handle)`=null→`supportedWriteOperations(handle)`=空→`PhysicalPlanTranslator:657/709` admission gate **规划期**抛干净 `AnalysisException`「does not support INSERT/row-level DML」；`getProcedureOps`=null→EXECUTE 拒；`ConnectorTableOps` 全 DDL 默认抛「… not supported」；`beginTransaction` 默认抛。**用户签字 Q1=方案 A**（保留干净拒绝+测试锁定+事务层防御，**不**让 getWritePlanProvider 抛——admission gate 由它派生，在其上抛→引擎误分类「maybe palo bug」而非干净「does not support INSERT」；已核对 fe-core 写路径顺序：gate 先于 getWritePlanProvider 先于 beginTransaction，且连接器 beginTransaction 只在**执行期**〔`PluginDrivenInsertExecutor`/`ConnectorRewriteDriver:130`〕被调、规划期已拒故对 hudi 写是死码）。**用户签字 Q2=方案乙**（严格拒绝**所有** DDL 含 `DROP TABLE`/`RENAME TABLE`——相对 legacy HMS〔经 `HiveMetadataOps` 支持删/改名 hudi 表 HMS 条目〕是**有意行为倒退**、hudi 完全外部管理）。**净新增（唯一代码改动）=override `HudiConnectorMetadata.beginTransaction(session)` 抛显式** `"Hudi tables are read-only in this catalog; INSERT/UPDATE/DELETE/MERGE are not supported"`（替换泛型默认；per-handle 重载 SPI 默认委派到它；纯消息质量末线兜底、不劣化 gate 分类）。`HudiReadOnlyWriteRejectTest`〔`new HudiConnectorMetadata(null,{},null)`＝write/DDL 抛前不 deref 故 null 够〕锁：无 writer+空写操作集〔=写不被误当 iceberg 执行〕/无过程/beginTransaction **双重载**〔no-arg + 生产路由的 per-handle〕均抛显式只读/全 DDL 变更族〔dropTable·renameTable·truncate + addColumn(s)·dropColumn·renameColumn·modifyColumn·reorderColumns·create/drop Branch|Tag·add/drop/replace PartitionField〕抛。**3 维对抗复审 `wf_36fd1c6c-e58`（reject-completeness·routing / classification·no-regression / test-intent）+逐条 refute：0 生产缺陷**；6 raised→2 survived nit（均测强化：per-handle beginTransaction 重载直测 + 全 DDL 族补锁，已修）；refuted：`DROP DATABASE … FORCE` 级联〔pre-existing 网关码、legacy 平价、超 per-table 范围〕/ 休眠〔HD-B2 未分流 HUDI＝设计延后〕/ beginTransaction 非可写性锁〔可写性由 `supportedWriteOperations` 空锁，已断言〕。全绿（hudi 163 测 + checkstyle 0 + import gate 净）。**⚠残留（翻闸/e2e，非静默）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名 `hmsClient.dropTable` 级联删 HMS 登记〔`HiveConnectorMetadata.dropDatabase` 无 per-source 分支、iceberg-on-HMS 同款、只删登记不删数据；翻闸若要 per-table 拒须网关加分流，S6 findings 已记〕；②写/DDL 只读端到端只在 HD-B2 分流 HUDI 后 live（e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud，memory `hms-iceberg-delegation-needs-e2e`）。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，武装 pivot，最后一个连接器步）：getTableHandle 加 HUDI 臂分流到 hudi 兄弟（镜像 iceberg S4、复用 getOrCreateHudiSibling、外来句柄原样返/永不强转、iceberg 优先/UNKNOWN fail-loud 不吞）；4 维对抗复审 0 生产缺陷 + 1 minor 接线守缺口已修（抽 `newMetadata(HmsClient)` 缝 + ThreeWayRoutingTest 变异验证守）。详见上 🎯 段 HD-B2 条目。**⇒ 全部连接器休眠步收官 ⇒ 下一步 = 翻闸本身（原子加 `"hms"` + 删除单元 + 翻闸 e2e，须用户定打包/翻闸序）。** - 关键结论： - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 From dba003e18052a2bfcd7b6b2a0e534dab08c18c4b Mon Sep 17 00:00:00 2001 From: morningman Date: Thu, 9 Jul 2026 23:35:16 +0800 Subject: [PATCH 144/334] [doc](catalog) HMS cutover: record flip-planning as next step (user chose plan-first) User signed off (2026-07-09): next step = produce an up-to-date flip recon + design doc (plan-before-code), then decide D4 packaging (single vs split PR) + flip ordering. Records the 5 recon dimensions and the flip-time write-code items; recommends a fresh session for the flip-planning sub-phase (clean full context). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index a2bfc926162393..c6d0dc3dc299a8 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸 e2e/硬门）。下一步 = 翻闸打包决策（设计文档 §2/§5/§7，须用户拍板单 PR vs 分 PR + 翻闸序）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**⚠ 建议新 session 开工**（本 session 已完成 HD-B2 且上下文渐满；翻闸规划是大新子相，值得干净满上下文——memory `session-handoff-at-30pct-context`）。**下一步 = 翻闸 recon + 设计文档（先规划不写码）。** **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 From 733c55217b94056c45c36d019fcc59ff7cc98abb Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 05:17:33 +0800 Subject: [PATCH 145/334] [doc](catalog) HMS cutover: up-to-date execution plan + user decisions (plan-first) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Recon (wf_94feaccd-d60: 6 readers + 2 adversarial critics, HEAD-grounded) produced hms-cutover-execution-plan-2026-07-10.md, superseding the STATUS/phasing of the 2026-07-07 retype design (its flip-shape/decisions/hard-gate remain valid). Key correction (both critics, blocker): the flip is NOT one commit away. All connector dormant steps are done, but two fe-core subsystems the old plan assumed would land pre-flip are 0% built — HMS event-pipeline relocation (Model B) and connector-owned caching (D2). The atomic flip neutralizes 4 gates (event poller + 3 cache-routing) that go silently false; without those replacements event sync halts and cache routing collapses to ENGINE_DEFAULT. HEAD-verified: hms absent from SPI_READY_TYPES, no connector pollOnce/self-cache, 3 fe-core caches + GSON registerSubtype tags still legacy. Also: reconciled the stale 07-07 DONE-vs-TODO ledger + line numbers, surfaced the hudi-line flip items it predates (INC-5, hudi_meta TVF, hudi-scan chain deletion), refuted 2 false worries (SQL-cache "stale" = safe cache-off; plain-hive eager materialize = byte-parity). User decisions (2026-07-10): (1) build-first, no outage window; (2) delete legacy code LAST, don't fuss PR packaging, reach a working flip first (→ deletion-unblocking extractions move to the final delete phase); (3) keep hive auto-refresh parity, built into the D2 cache. Next = Phase 1 first step = connector-owned cache (D2). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- .../hms-cutover-execution-plan-2026-07-10.md | 184 ++++++++++++++++++ 2 files changed, 185 insertions(+), 1 deletion(-) create mode 100644 plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index c6d0dc3dc299a8..b6942cd0967600 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**⚠ 建议新 session 开工**（本 session 已完成 HD-B2 且上下文渐满；翻闸规划是大新子相，值得干净满上下文——memory `session-handoff-at-30pct-context`）。**下一步 = 翻闸 recon + 设计文档（先规划不写码）。** +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**下一步 = Phase 1 首步：连接器自持缓存（D2，含 max-partition-time；退休 fe-core 三份缓存 + 4 路由 instanceof 守卫；paimon/iceberg 自持缓存为模板）——先 recon+设计再实现。**（次为事件 Model B。） **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 diff --git a/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md b/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md new file mode 100644 index 00000000000000..ab0b8f73cf9629 --- /dev/null +++ b/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md @@ -0,0 +1,184 @@ +# HMS cutover — up-to-date execution plan (2026-07-10) + +> Produced by a HEAD-grounded recon (`wf_94feaccd-d60`: 6 dimension readers + 2 adversarial critics — atomic-set integrity + completeness) with lead-engineer verification of the load-bearing facts. **Supersedes the STATUS/phasing of `hms-cutover-retype-design-2026-07-07.md` §4/§5** (its §2 flip-shape, §3 heterogeneity crux, §6 D1–D6 decisions, and §7 hard-gate remain valid). The 07-07 doc was written *before* almost its entire §4 build-out landed; this doc reconciles it against HEAD (`db27455b4e1`). **Trust HEAD, not doc line numbers** — every line number below re-verified. + +--- + +## 0. TL;DR — the one thing that changed + +The 07-07 plan framed the cutover as **dormant build-out → one atomic flip → mechanical delete**. Two weeks of work went entirely into the **hudi delegation line** (HD-A0 … HD-D1 + the arming pivot HD-B2), so **all connector-side dormant steps are now DONE**. But the plan's §5 assumed two large **fe-core** subsystems (event-pipeline relocation + connector-owned cache) would also land dormant before the flip. **They were never started (0% at HEAD).** + +**⇒ The flip is NOT executable as one commit today.** The atomic flip commit *neutralizes* four gates (the HMS event poller + three cache-routing sites). Those gates going false is only *safe* if their replacements already exist. They don't. Executing the flip at HEAD would make **event sync silently halt forever** and **cache routing silently collapse to `ENGINE_DEFAULT`** — no crash, no log. Both critics independently ranked this the #1 finding (blocker). + +So the accurate remaining structure is **four** phases, and the real remaining *work* is Phase 1 (two new fe-core subsystems + a handful of seams/extractions), not the flip commit itself: + +| Phase | What | Status | +|---|---|---| +| **0. Connector dormant build-out** | Read-SPI, sibling delegation (iceberg S0–S6 + W1–W5 + WC1–WC4), whole hudi line (HD-A0…HD-D1 + HD-B2) | ✅ **DONE** | +| **1. Pre-flip fe-core build-out (dormant)** | **Event-pipeline Model B** + **connector-owned cache (D2)** + 3 coupling seams + deletion-unblocking extractions + W6 verify | ❌ **0% — the real remaining work** | +| **2. THE ATOMIC FLIP (one commit)** | `SPI_READY += hms` + 3 GSON remaps + `buildDbForInit` + dead-arm/INC-5/D5 removals + gate rewires + replay test | ❌ not started (Phase-1-gated) | +| **3. DELETE (mechanical PR)** | The legacy cyclic import unit (~90 classes) | ❌ last | +| **4. e2e / hard gates (R-002)** | Two non-unit harnesses + consolidated e2e matrix + 4 correctness-risk rows + clean-room + capability-twin | ❌ green-before-flip | + +### 0.1 Decisions recorded (user sign-off 2026-07-10) +- **Flip order = BUILD-FIRST (no outage).** Land the event-pipeline relocation + connector-owned cache dormant *before* the flip. No silent event-sync/cache-collapse window. (Trino-aligned: engine owns HA/replication, plugin owns fetch/parse.) +- **Auto-refresh = keep parity, built into the cache subsystem.** The hive connector surfaces a real max-partition modify time so SQL-dictionary / MV-on-hive-base auto-refresh matches today. `getNewestUpdateVersionOrTime` parity rides on D2. (Resolves §7 Q3.) +- **Priority = reach a WORKING FLIP (new code path live) first; DELETE legacy code LAST.** Do not fuss PR packaging now. **⇒ the deletion-unblocking extractions (§2.4) move OUT of Phase 1 and INTO Phase 3 (deletion)** — they are only needed to delete the legacy classes, which is deferred to the end; post-flip the legacy `HiveUtil`/`HiveSplit`/`IcebergUtils` classes still exist (dead-for-hms) so live consumers keep compiling. (Resolves §7 Q2 + supersedes §8.) +- **⇒ Phase 1 remaining work reduces to:** connector-owned cache (D2, with max-partition-time) → event-pipeline Model B → the 3 loud-break coupling seams → W6 verify. Then the atomic flip. Then (last) extractions + delete. + +--- + +## 1. Where we are (DONE ledger — reconciles the 07-07 §4 blockers) + +Every ⛔ blocker and build-out item the 07-07 doc §4/§4.2a/§6 listed is **landed and HEAD-verified**: + +- **§4.1 Kerberos authenticator** — done (`e63b03fb490` gateway; `561f777d5f9` hudi sibling). *Only the write-path half (W6) remains — see Phase 1.* +- **§4.2 read-side SPI** — all done: `partition_columns` emission, `listPartitions`/`listPartitionNames`, `getTableStatistics` (3-tier), `buildTableDescriptor` (`THiveTable`), `viewExists`/`getViewDefinition`/`dropView`, `getCapabilities`, file-format serde, `HiveTableFormatDetector` parity, per-table Top-N marker. +- **§4.2a per-column stats (D1=preserve)** — done (`cbf9526f776`). +- **§4.3 freshness-kind-aware `getTableSnapshot`** — done (`3d784673ca4`). *This resolves the 07-07 §3 "MAJOR consequence" (plain-hive stale-MV).* +- **§4.4 sibling-connector SPI + iceberg delegation** — done (S0–S6) + all write/procedure/DDL delegation (W1–W5) + write-chain (WC1–WC4). +- **§4.5 deletion-unblockers** — `BIND_BROKER_NAME` relocated (`BrokerProperties.BIND_BROKER_NAME_KEY`); `pluginCatalogTypeToEngine("hms")→ENGINE_HIVE`; LZO read-reject ported connector-side; read-ACID query-finish release (WC4). +- **Whole hudi line** — HD-A0…A5, HD-B1 (3-way routing), HD-C1 (MVCC/partitions), HD-C2 (FOR TIME AS OF), HD-C3 INC-1…4 (@incr), HD-C4/C5 (schema evolution), HD-D1 (read-only reject), **HD-B2 (getTableHandle diverts HUDI = the arming pivot).** +- **Capability preconditions for the flip** — `HiveConnector.getCapabilities` declares `SUPPORTS_MVCC_SNAPSHOT`+`VIEW`+`COLUMN_AUTO_ANALYZE` (dormant); `PluginDrivenExternalDatabase.buildTableInternal:53-60` picks `PluginDrivenMvccExternalTable` off that flag → a flipped hms table gets the right class **for free**. + +**⇒ The connector is flip-ready. fe-core is not.** + +--- + +## 2. Phase 1 — the real remaining work (dormant fe-core build-out) + +Everything here can and must land **dormant** (inert while hms is still legacy), each an independent reviewable commit, exactly like the connector steps. Ordered by size/criticality. + +### 2.1 ⛔ Event-pipeline relocation — Model B (the largest un-built blocker) +Design: `hms-event-pipeline-findings-2026-07-07.md` (Model B). **0% built** (grep `pollOnce`/`getNextNotification` across `fe-connector` = none; the poller still `instanceof HMSExternalCatalog` at `MetastoreEventsProcessor.java:116`). +- Thin fe-core role-aware `MasterDaemon` driver that probes each connector for an optional event-source capability (a **capability probe, not `instanceof`**) → `connector.pollOnce(lastSyncedEventId, isMaster)` returning a **neutral change-descriptor list** (must cover register/unregister/rename + **partition-NAME** granularity — closes the `invalidatePartition` values-vs-names gap). +- Plugin (`fe-connector-hms`) severs fetch+parse only (`getNextNotification`/`getCurrentNotificationEventId` on `HmsClient`, deserializers, event→descriptor mapping). No fe-core imports plugin-side. +- fe-core wraps `pollOnce` in an `onPluginClassLoader` pin (R-010 — today there is **no** pin in the poller path). +- Retain follower→master `REFRESH CATALOG` forward + HA/edit-log in fe-core. +- Keep **opcode 470** (`OP_ADD_META_ID_MAPPINGS`) + a neutral replay handler on disk (the replay-CCE fix already landed `a6dc782d816`); `ExternalMetaIdMgr` can otherwise be dropped for HMS. +- **New SPI:** `Connector.invalidatePartition(s)` (pairs with the descriptor granularity). + +### 2.2 ⛔ D2 — connector-owned scan-side cache (retire the fe-core caches) +Decision D2 (LOCKED 07-07) = connector-owned. **0% built.** All three fe-core caches present; `ExternalMetaCacheRouteResolver.java:66` still routes hms→HIVE+HUDI+ICEBERG caches. The hive connector must own scan-side caching (paimon/iceberg-native precedent) so that at the flip, routing collapsing to `ENGINE_DEFAULT` is *harmless*. Retire `HiveExternalMetaCache` / `HudiExternalMetaCache` / `IcebergExternalMetaCache` and the four routing `instanceof` gates (`ExternalMetaCacheRouteResolver:66`, `HiveExternalMetaCache:203/274`, `HudiExternalMetaCache:221`, `IcebergExternalMetaCache:234`) **with the flip set**. Also unblocks §2.6 below. + +### 2.3 Coupling seams that break LOUD at the flip (must be dormant-staged) +- **`partition_values()` TVF** — no PluginDriven arm today → post-flip throws `"Catalog of type 'hms' is not allowed"` and `MetadataGenerator:2088-2092` casts to `HMSExternalTable`. Add a PluginDriven catalog+table arm to the gate + a `partitionValuesMetadataResultForPluginTable` path (mirror the already-done `$partitions`). `PartitionValuesTableValuedFunction.java:113,130`. +- **`hudi_meta()` / TIMELINE TVF** (NEW — not in the 07-07 doc; the hudi line added it) — post-flip `!(instanceof HMSExternalTable)` → returns *"not a hudi table"*, and its body depends on the deletion-unit hudi cache. **Decide: connector-driven timeline path, or drop the TVF.** `MetadataGenerator.java:459-483`. +- **`canSample` / `SUPPORTS_SAMPLE_ANALYZE`** (optional) — no capability today → post-flip hive `ANALYZE … WITH SAMPLE` silently falls back to FULL (correct, slower). Add `SUPPORTS_SAMPLE_ANALYZE` + a PluginDriven `canSample` arm, **or** accept the degrade. `AnalysisManager.java:1484-1485`. + +### 2.4 Deletion-unblocking extractions (→ RE-SEQUENCED to Phase 3 per the 2026-07-10 decision) +> **Deferred to the deletion phase** (deletion is last). Post-flip the legacy classes still exist (dead-for-hms), so their live consumers keep compiling; these extractions are only needed at the moment of deletion. Listed here for completeness; do them in Phase 3, immediately before deleting the dirs. + +Three members of the deletion unit are consumed by **live surviving code** and must be relocated to a surviving util *before* the unit can be deleted: +- `HiveUtil.toPartitionValues` — consumed by **live** `PluginDrivenMvccExternalTable.java:296`. +- `HiveUtil.isLzoInputFormat` — consumed by **live** `BaseExternalTableDataSink.java:86`. +- `HiveSplit` branch in **live** `FileQueryScanNode.java:465` (remove the branch). +- **`IcebergUtils` 6-member extraction** to a new fe-core SDK-free util (`ICEBERG_ROW_ID_COL`, `ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL`, `ICEBERG_ROW_LINEAGE_MIN_VERSION`, `isIcebergRowLineageColumn(String)`, `isIcebergRowLineageColumn(Column)`, `getEffectiveIcebergFormatVersion` with its 3 iceberg string constants inlined). **The target util does not exist at HEAD.** Repoint `BindExpression`, `CreateTableInfo:1145-1171`, `IcebergMergeCommand`, `IcebergUpdateCommand`. + +### 2.5 W6 — write-path TCCL pin (verify-first, land dormant if needed) +`fe-connector-hive` has no `TcclPinningConnectorContext`; `HiveConnector.java:195` flags the write/DDL-thread pin as a separate flip-time concern. Iceberg-on-HMS *writes* run name-based reflection (`planWrite` TDataSink build, `getWritePartitioning`) on fe-core app-loader threads → classloader split-brain CCE risk. **Verify** whether the path is unpinned; if so add an fe-core write-path pin keyed off `provider.getClass().getClassLoader()`. Not dormant-testable → e2e-owed (Phase 4). + +### 2.6 `getNewestUpdateVersionOrTime` real max-partition-time (coupled to D2) +Flipped plain-hive returns constant `0` (names-only `listPartitions` → every `lastModifiedMillis` UNKNOWN `-1` → filtered → `orElse(0L)`), so a **hive-backed SQL dictionary / MV auto-refresh never sees a newer source version → silently stale**. Fix needs the connector to surface a real max-partition modify time cheaply — **which needs D2 first**. Alternatively, sign off an explicit temporary regression + e2e assertion. `PluginDrivenMvccExternalTable.java:713-714`. + +--- + +## 3. Phase 2 — THE ATOMIC FLIP (one commit, canonical union checklist) + +This is the "must-be-atomic set." All members ship in **one** commit (the GSON remaps + `SPI_READY` are mutually locked; the gates go false together). **This commit is only correct once Phase 1 has landed.** Line numbers HEAD-verified. + +1. **`SPI_READY_TYPES += "hms"`** + delete the dead `case "hms" → new HMSExternalCatalog` fallback. `CatalogFactory.java:55-56` (set) / `:139-141` (fallback). *(07-07 doc said :49-50 / :133-135 — stale.)* +2. **3 GSON remaps** — delete `registerSubtype` + add `registerCompatibleSubtype`: catalog→`PluginDrivenExternalCatalog` (`GsonUtils.java:366`), db→`PluginDrivenExternalDatabase` (`:447`), **table→`PluginDrivenMvccExternalTable`** (`:471`, target confirmed by the paimon `:489-490` / iceberg `:493-494` precedent). Mutually locked with (1) via `RuntimeTypeAdapterFactory:280` "labels must be unique" static-init throw **and** loss of write-registration (a live `HMSExternalCatalog` would crash image-write). Precedent: paimon `a26eaeff84c` did `SPI_READY`+GSON in one commit. +3. **`buildDbForInit` case HMS → `new PluginDrivenExternalDatabase`** (`ExternalCatalog.java:967-968`, mirror the ICEBERG arm). *Must-not-omit:* else a freshly-created (non-replayed) hms catalog silently builds legacy `HMSExternalDatabase→HMSExternalTable` under a PluginDriven catalog — heterogeneous legacy/plugin objects, no error. +4. **Remove dead `PhysicalPlanTranslator` arms + `DLAType` import** — the `instanceof HMSExternalTable` scan arm `:824-853` and `visitPhysicalHudiScan` `:895-943` become unreachable (PluginDriven arm `:814` wins) but compile-reference legacy scan nodes. Drop them + import `:65`. *(07-07 said :818-846/:925-928 — stale.)* +5. **INC-5 (NEW — 07-07 predates it):** remove the now-**stale** `visitPhysicalHudiScan` throws for FOR TIME AS OF / @incr (`:910`, `:914`) — HD-C2/C3 landed the connector support they reject → dead-on-arrival otherwise — and the legacy `CheckPolicy` hudi incremental arm (`CheckPolicy.java:94-99`, comment already says "deleted at the cutover"). Forward `tableSnapshot`/`scanParams` to `PluginDrivenScanNode`. +6. **D5 (LOCKED 07-07):** drop both `enable_query_hive_views` / `enable_query_iceberg_views` gates (serve any `isView()`), neutralize the iceberg-worded message + time-travel throw, deprecate both `@ConfField` to no-ops one release. `BindRelation.java:594` (dead hive arm), `:634-636` (iceberg gate+throw). **Visible iceberg behavior change — must ship in the flip, not leak early.** +7. **Rewire the 4 gates** — event poller (`MetastoreEventsProcessor:116`) → Model B driver; cache routing (`ExternalMetaCacheRouteResolver:66` + the 3 legacy-cache gates) → D2 connector cache. **Only safe because Phase 1 landed.** +8. **`HmsGsonCompatReplayTest`** (template: `Iceberg`/`PaimonGsonCompatReplayTest`) — pins the single-base-tag round-trip + the invariant that deserialized external-table objects never survive a replay (metaCache transient, `initialized` reset, rebuilt via `buildTableInternal`). + +--- + +## 4. Phase 3 — DELETE (mechanical, one PR, the cyclic unit) + +`datasource/hive` (52), `datasource/hudi` incl. `hudi/source/*` (15), `datasource/iceberg` (exactly 23) import each other cyclically (`HudiScanNode extends HiveScanNode:94`; `HudiUtils→HMSExternalTable`; iceberg scan/cache→hive; `HMSExternalTable→IcebergDlaTable/HudiDlaTable/HudiUtils/IcebergUtils`) → **cannot delete hive alone**. Connector build-out is **not** entangled (`fe-connector/**` has zero real `datasource.hive/.hudi/.iceberg` imports — the one grep hit is a vendored same-FQN copy). The full unit: + +- **datasource/hive (52):** catalog/db/table, DLA tables, HMS clients, write chain, cache, utils, `source/{HiveScanNode,HiveSplit}`. `event/*` (21) — **delete only after** the Model B relocation (it is the current event driver). +- **datasource/hudi + hudi/source/* (15)** — `HudiScanNode`, `HudiSplit`, `IncrementalRelation`+`{COW,MOR,Empty}IncrementalRelation`, `HudiUtils`, `HudiExternalMetaCache`, `HudiMvccSnapshot`, `HudiPartitionUtils`, schema/meta-client cache keys. *(07-07 doc never enumerated these — hudi-line addition.)* +- **datasource/iceberg (23)** — TIER-1 scan cluster (9) + TIER-2 metadata (14, incl. `IcebergUtils` after the 6-member extraction). +- **Nereids legacy chains** (die at the flip, compile-referenced until Phase-2 removals): `LogicalHiveTableSink`/`PhysicalHiveTableSink`/`HiveInsertExecutor`/`HiveTransactionManager`; `LogicalHudiScan`/`PhysicalHudiScan`/`LogicalHudiScanToPhysicalHudiScan` + wiring (`BindRelation:604`, `RuleSet:201,248`, `AggregateStrategies:750`, `StatsCalculator:849`, `RelationVisitor:98`). *(hudi-scan chain also undoc'd in 07-07.)* +- **statistics/HMSAnalysisTask**, `StatisticsUtil.getIcebergColumnStats`/`getHiveRowCount`/`getTotalSizeFromHMS` + `:1008` branch (NB: 07-07's "`StatisticsUtil.getHiveColumnStats`" is wrong — that's a **private** method inside `HMSExternalTable:892`), **systable/IcebergSysTable** (already a throwing dead-end). +- **`Env.hiveTransactionMgr`** field + init + accessors (`Env.java:568/865/1097-1102`) — sole consumer is legacy `HiveScanNode` (`:141/147/319`) → delete in the **same** PR as `HiveScanNode`. + +**Deletion ordering (topological):** (a) Phase-1 extractions (§2.4) land first; (b) Phase-2 flip removes all compile-references (dead arms, INC-5, hudi-scan wiring, `bindHiveTableSink`); (c) then the cyclic unit deletes mechanically; (d) `event/*` deletes only after Model B replaced the entrypoints. + +--- + +## 5. Phase 4 — e2e / hard gates (R-002 — do not silently skip, Rule 12) + +The 07-07 §7 gate is **materially stale** (predates the whole hudi line, names none of its rows, and predates the sibling-decomposition's two-loader fixture + W6). Two named non-unit harnesses — **neither exists yet**: + +- **Harness (i) — two-URLClassLoader routing fixture:** loads iceberg/hudi handles in a child-first loader and proves the gateway diverts each foreign handle to the correct sibling **with no CCE** across the loader split. Unit tests can't: test classpaths load everything on the app loader, so an illegal foreign-handle cast passes silently (the shipped `HiveConnectorThreeWayRoutingTest` is same-loader — proves only `ownsHandle` logic). +- **Harness (ii) — heterogeneous-HMS docker e2e:** ONE `type=hms` catalog holding plain-hive + iceberg-on-HMS + hudi-on-HMS, each served by the right provider, mixed queries in one session correct. + +**Consolidated e2e matrix** (every landed step's "⚠残留 e2e" note): transactional-hive read (full-ACID+insert-only in scope, full-ACID write hard-rejected); HMS event replay in sync post-flip; iceberg-on-HMS **INSERT/DELETE/MERGE/OVERWRITE/ALTER(14)/EXECUTE** == independent iceberg catalog (NET-NEW, signed 2026-07-08); MTMV freshness + time-travel over iceberg-on-HMS & hive base; hudi COW/MOR + hive-sync/non-hive-sync partition-value fidelity; hudi FOR TIME AS OF; hudi **@incr straddling-base-file** fixture; hudi schema-evolved reads (rename/reorder + **mixed-case nested struct must not SIGABRT**); hudi dropped-partition-set-at-instant; hudi MTMV refresh; hudi write/DDL/EXECUTE **fail-loud reject**; Kerberos-HMS smoke; `HmsGsonCompatReplayTest`. + +**4 rows are genuine CORRECTNESS risks the flip cannot ship without** (both critics ranked these above the docker run): +1. **hudi COW-@incr row-level `_hoodie_commit_time` filter** — INC-4 delivered the iron-rule-compliant fix (neutral `ConnectorExpression` reverse-converter + `CheckPolicy.collectConnectorSyntheticPredicates`, no fe-core source branch); the "possibly unfixable" §5.1 worry is **superseded**, but the straddling-base-file correctness is **e2e-owed**. +2. **W6 write-path TCCL pin** — unpinned name-reflection → CCE on iceberg-on-HMS write. +3. **WC4 read-ACID query-finish release** — a broken release leaks the metastore shared read lock. +4. **Kerberos-HMS smoke** — silent SIMPLE-auth degrade. + +**Human-review gates (§7, unchanged):** clean-room adversarial review over the flip+delete diffs; ENG-1 capability-twin audit over the coupling sites (now grown to include the hudi/write-delegation surface). + +--- + +## 6. Refuted worries — do NOT spend flip budget here + +Both critics verified these against HEAD and refuted them: +- **SQL-cache "stale-result correctness bug"** — **REFUTED (safe).** A flipped hive table reports `PLUGIN_EXTERNAL_TABLE`; `NereidsSqlCacheManager:441-443/480/506-507` returns `CHANGED_AND_INVALIDATE_CACHE` for it every time → cache **off**, never stale (same as paimon/iceberg-native). Deferrable perf item, **not** a flip gate. (Reconciled: dim-3 over-ranked it, dim-4 is correct.) +- **Plain-hive "eager `materializeLatest` vs lazy `EmptyMvccSnapshot`" regression** — **REFUTED.** Designed for byte-parity (`PluginDrivenMvccExternalTable:622` comment); the identical live path paimon/iceberg already run. e2e-owed only. +- **MTMV parity** — safe (`PluginDrivenMvccExternalTable` implements `MTMVRelatedTableIf`+`MTMVBaseTableIf`). +- **`IcebergExternalMetaCache:234` "silent-false gate"** — nuance: it would *throw* if reached, but post-flip it is **unreached** (downstream of the cache-route master). Retired with D2. + +--- + +## 7. Open decisions + +**Resolved 2026-07-10 (see §0.1):** Q1 pre-flip build vs outage → **build-first**. Q2 packaging → **don't fuss; delete last; reach working flip first.** Q3 `getNewestUpdateVersionOrTime` → **keep parity, built into D2 cache.** + +**Still open (I'll take the recommended default as I reach each, unless you say otherwise):** +4. **explicit-partition-spec INSERT on flipped hive** — reject fail-loud (legacy parity) vs gain static-partition-overwrite support. *Lean: reject fail-loud (parity).* Decide + e2e at the flip. +5. **`hudi_meta()` TVF** — connector-driven timeline path vs drop the TVF. *Lean: drop unless a consumer needs it.* +6. **`SUPPORTS_SAMPLE_ANALYZE`** — port `canSample` vs accept hive sample→FULL degrade. *Lean: port (cheap, keeps parity).* +7. **`DorisConnectorException` → user-error mapping** in `ConnectProcessor.handleQueryException` — add now vs accept catch-all. *Lean: add (cleaner for all plugin connectors).* + +*Settled / no decision needed:* hudi schema-evolution scope = **FULL parity** (user sign-off 2026-07-09, memory `hudi-schema-evolution-full-parity-signoff`) → the schema-evolution e2e rows are flip-blockers, not deferrable. `SUPPORTS_NESTED_COLUMN_PRUNE` = deliberately deferred (safe-off). SHOW CREATE TABLE generic form = accepted (D6-adjacent). TTL validation + file-format serde nuance = deferrable. + +--- + +## 8. Packaging (D4) — user steer 2026-07-10 + +User: *don't fuss PR packaging; the priority is to reach a working flip (new code path live); delete legacy code last.* So the working sequence is: + +- **Phase-1 build-out = independent dormant commits** (as every connector step so far), each inert while hms is legacy: **D2 cache (with max-partition-time) → Event Model B → the 3 loud-break coupling seams (§2.3) → W6 verify.** +- **Phase-2 = the atomic flip** → new code path live. This is the milestone the user cares about ("走新的代码逻辑即可"). +- **Phase-3 (LAST) = the deletion-unblocking extractions (§2.4) + delete the ~90-class cyclic unit.** Deferred to the end; the exact PR split (single vs two) is decided then, not now. + +Precedents kept for reference only: paimon `#64446` flip + `#64653/#64655` delete; iceberg `#64688` squash. Tracking: `apache/doris#65185`. + +--- + +## Appendix — line-number corrections vs the 07-07 doc (all re-verified at HEAD `db27455b4e1`) + +| Site | 07-07 doc | HEAD | +|---|---|---| +| `SPI_READY_TYPES` | CatalogFactory.java:49-50 | **:55-56** | +| dead `case "hms"` fallback | :133-135 | **:139-141** | +| GSON catalog/db/table remaps | :366 / :447 / :471 | **unchanged** | +| `buildDbForInit` case HMS | ExternalCatalog.java:966-968 | **:967-968** | +| PhysicalPlanTranslator scan arm | :818-846 | **:824-853** | +| PhysicalPlanTranslator hudi visitor / throws | :925-928 | **:895-943 / throws :910,:914** | +| `RuntimeTypeAdapterFactory` "labels must be unique" | fe-core :279 | **fe-common :280** | +| `CreateTableInfo` hms→engine | "missing → returns null" | **DONE: :943-947 → ENGINE_HIVE** | +| §4.1 Kerberos / §4.2 read-SPI / §4.3 freshness / §4.4 sibling / §4.5 unblockers | ⛔ TODO | **all DONE (see §1)** | +| §4.6 seams / §4.7 event / D2 cache | assumed phased before flip | **0% built (Phase 1)** | From f7e6b5e4a6b1244f81ef426923bdee805ce4982f Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 07:10:50 +0800 Subject: [PATCH 146/334] [doc](catalog) Hive connector cache: recon + design + sign-off (plan-first) Phase-1 first item (connector-owned scan-side cache). HEAD-grounded recon (wf_d19057ca-5bb: 8 readers + 3 adversarial critics). Authoritative design hive-connector-cache-step-design-2026-07-10.md. User signed off two decisions: metadata cache = CachingHmsClient decorator (Trino style); file-listing cache included this step. Decomposed into 6 dormant commits C-a..C-f. No code yet. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .claude/wf-d2-cache-recon.js | 266 ++++++++++++++ plan-doc/HANDOFF.md | 2 +- ...-connector-cache-step-design-2026-07-10.md | 334 ++++++++++++++++++ 3 files changed, 601 insertions(+), 1 deletion(-) create mode 100644 .claude/wf-d2-cache-recon.js create mode 100644 plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md diff --git a/.claude/wf-d2-cache-recon.js b/.claude/wf-d2-cache-recon.js new file mode 100644 index 00000000000000..a6f974839555d0 --- /dev/null +++ b/.claude/wf-d2-cache-recon.js @@ -0,0 +1,266 @@ +export const meta = { + name: 'd2-cache-recon', + description: 'HEAD-grounded recon for D2 connector-owned scan-side cache (retire fe-core Hive/Hudi/Iceberg caches + 4 routing gates + max-partition-time)', + phases: [ + { title: 'Recon', detail: '8 dimension readers over the cache subsystem, consumers, connector templates, max-time/invalidation, Trino' }, + { title: 'Critique', detail: '3 adversarial critics: completeness, flip-safety, scope-boundary' }, + ], +} + +const READER_SCHEMA = { + type: 'object', + additionalProperties: false, + properties: { + dimension: { type: 'string' }, + summary: { type: 'string', description: '4-8 sentence executive summary of what you found, in plain terms' }, + facts: { + type: 'array', + description: 'Code-grounded facts. Every one MUST cite file:line evidence verified at HEAD.', + items: { + type: 'object', + additionalProperties: false, + properties: { + claim: { type: 'string' }, + evidence: { type: 'string', description: 'file:line (repo-relative). Multiple allowed.' }, + implication: { type: 'string', description: 'why it matters for D2 / the flip' }, + }, + required: ['claim', 'evidence'], + }, + }, + flipImpact: { + type: 'array', + description: 'For this dimension: what silently breaks / changes / must be replaced at the atomic flip (when a type=hms catalog becomes a PluginDriven catalog and cache routing collapses to ENGINE_DEFAULT).', + items: { type: 'string' }, + }, + designOptions: { + type: 'array', + description: 'Concrete options/tradeoffs you surfaced for the design (do NOT pick one; list them with pros/cons).', + items: { type: 'string' }, + }, + openQuestions: { type: 'array', items: { type: 'string' } }, + filesRead: { type: 'array', items: { type: 'string' } }, + }, + required: ['dimension', 'summary', 'facts', 'flipImpact'], +} + +const CRITIC_SCHEMA = { + type: 'object', + additionalProperties: false, + properties: { + verdict: { type: 'string' }, + gaps: { + type: 'array', + items: { + type: 'object', + additionalProperties: false, + properties: { + area: { type: 'string' }, + whatsMissing: { type: 'string' }, + whyItMatters: { type: 'string' }, + severity: { type: 'string', enum: ['blocker', 'major', 'minor', 'nit'] }, + suggestedResolution: { type: 'string' }, + evidence: { type: 'string' }, + }, + required: ['area', 'whatsMissing', 'severity'], + }, + }, + confirmedRisks: { + type: 'array', + items: { + type: 'object', + additionalProperties: false, + properties: { + risk: { type: 'string' }, + severity: { type: 'string', enum: ['blocker', 'major', 'minor', 'nit'] }, + evidence: { type: 'string' }, + }, + required: ['risk', 'severity'], + }, + }, + refuted: { type: 'array', description: 'Worries you checked against HEAD and dismissed, with why.', items: { type: 'string' } }, + }, + required: ['verdict', 'gaps'], +} + +const CTX = ` +CONTEXT — you are reconnoitering for a DESIGN DOC (do not write code, do not implement). + +Project: Apache Doris catalog-SPI migration. Legacy \`type=hms\` catalogs (HMSExternalCatalog/HMSExternalTable, in fe-core \`datasource/hive|hudi|iceberg\`) are being migrated to a plugin-connector model (PluginDrivenExternalCatalog/PluginDrivenExternalTable holding an fe-connector-hive connector loaded in a child-first classloader). The migration ends in ONE ATOMIC FLIP that adds "hms" to SPI_READY_TYPES and remaps GSON subtypes so every hms catalog/db/table instantiates as PluginDriven* classes instead of the legacy classes. + +THE D2 TASK (what this recon informs): "connector-owned scan-side cache." Today fe-core owns three engine metadata caches — HiveExternalMetaCache, HudiExternalMetaCache, IcebergExternalMetaCache — routed by ExternalMetaCacheRouteResolver.addBuiltinRoutes(): a type=hms catalog routes to HIVE+HUDI+ICEBERG caches (ExternalMetaCacheRouteResolver.java:66-70). At the flip, the catalog is no longer \`instanceof HMSExternalCatalog\`, so routing falls through to ENGINE_DEFAULT (:72-73) and these three caches SILENTLY stop being fed/invalidated for hms — no crash, no log. Decision D2 (LOCKED): the hive connector must OWN its scan-side cache (mirroring how the paimon & iceberg-native connectors own theirs today), so that routing-collapse is harmless. Then retire the three fe-core caches + the four instanceof routing gates WITH the flip set. Coupled sub-item §2.6: flipped plain-hive \`getNewestUpdateVersionOrTime\` (PluginDrivenMvccExternalTable.java:713-714) returns constant 0 (names-only listPartitions → all lastModifiedMillis -1 → filtered → orElse(0)), so hive-backed SQL-dictionary / MV auto-refresh silently never sees a newer version. D2 must let the connector surface a real max-partition modify time cheaply. + +HARD ARCHITECTURE RULES (from project memory — respect them in what you flag): +- fe-core does NOT parse connector properties; storage parsing → fe-filesystem, meta parsing → fe-connector (both plugin-side). +- fe-core depends only on fe-connector-api/-spi (maven). Plugin connector classes load in a SEPARATE child-first classloader. A connector CANNOT import fe-core datasource classes. If you see "connector references HiveExternalMetaCache", determine whether it's a real import, a comment/string, or a VENDORED same-simple-name copy in a different package — this matters a lot. +- The generic fe-core SPI layer must stay connector-agnostic (no source-specific branches). +- Cross-classloader calls into plugins must pin TCCL to the connector classloader. + +DELIVERABLE: return the READER_SCHEMA object. Every fact needs file:line evidence you actually opened and verified at HEAD (branch catalog-spi-11-hive, HEAD commit d43ba31f3b3). Distinguish legacy deletion-unit code (dies at flip, needs NO replacement) from flip-survivor code (needs a replacement path). Surface design OPTIONS with tradeoffs; do not decide. +` + +phase('Recon') + +const readers = [ + { + label: 'dim-cache-machinery', + prompt: `${CTX} + +YOUR DIMENSION = the shared fe-core cache MACHINERY (not the engine-specific caches themselves). Read and map: +- fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaCacheMgr.java (the manager; public entry points prepareCatalog/invalidate*/removeCatalog; the CacheKey inner class; invalidateTableCache) +- fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCache.java (interface — already known: engine()/aliases()/initCatalog/entry/invalidate*/stats/close) +- fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/AbstractExternalMetaCache.java +- fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRegistry.java +- fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/ExternalMetaCacheRouteResolver.java (the routing — ENGINE_DEFAULT fallback at :72-73) +- fe/fe-core/src/main/java/org/apache/doris/connector/ExternalMetaCacheInvalidator.java +- fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/MetaCacheEntry.java / MetaCacheEntryStats.java +- fe/fe-core/src/main/java/org/apache/doris/datasource/doris/DorisExternalMetaCache.java (a peer engine cache — how a non-hive engine registers, for contrast) + +Answer: What is the engine→catalog→entry model? How do caches register with the registry and get resolved by engine name? What EXACTLY is ENGINE_DEFAULT — is there a registered "default" cache, or does registry.resolve("default") return null / throw? Trace what happens to every ExternalMetaCacheMgr public method for a flipped hms catalog (routing → ENGINE_DEFAULT). Which manager methods are called on the scan hot-path vs on REFRESH/DDL/event paths? Does the machinery itself survive the flip (it serves paimon/jdbc/doris too) or get retired? Note the CacheKey/registry stats surface.`, + }, + { + label: 'dim-hive-cache', + prompt: `${CTX} + +YOUR DIMENSION = HiveExternalMetaCache internals (the primary cache to relocate into the hive connector). +Read fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HiveExternalMetaCache.java IN FULL. Enumerate EVERY distinct cache it holds (schema cache, partition-value cache, partition cache, file cache, and any others — note the inner key classes PartitionValueCacheKey:849 / PartitionCacheKey:882 / FileCacheKey:942). For each cache: what is the key, what is the value, which loader populates it, and what remote calls (HMS thrift / filesystem listing) does the loader make. Document the four instanceof gates (:203, :209, :248, :274) — what does each gate protect and what does it do for a non-HMS catalog/table. Find how max-partition modify time / lastModifiedTime is computed here (relevant to §2.6 getNewestUpdateVersionOrTime). Note file-listing / partition-listing caching that is on the SCAN hot-path (this is the core of what the hive connector must own). Note the eager-init behavior (initCatalog). + +Then classify: which of these caches are SCAN-side (must be owned by the connector post-flip) vs which serve only legacy HMSExternalTable/Catalog paths that die at the flip. Surface options for WHERE the relocated cache lives (connector-internal vs the shared fe-connector-cache framework) with tradeoffs.`, + }, + { + label: 'dim-hudi-iceberg-cache', + prompt: `${CTX} + +YOUR DIMENSION = HudiExternalMetaCache + IcebergExternalMetaCache (the other two fe-core caches routed for hms), and whether they even need a replacement. +Read: +- fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiExternalMetaCache.java IN FULL (gate at :221 = \`!(dorisTable instanceof HMSExternalTable)\`). +- fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergExternalMetaCache.java IN FULL (gates at :213 = \`!(dorisTable instanceof MTMVRelatedTableIf)\`, :234 = \`catalog instanceof HMSExternalCatalog\`). +- fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/cache/IcebergManifestCacheLoader.java + +CRUCIAL NUANCE to resolve with evidence: post-flip, an iceberg-on-HMS table is served by DELEGATION to the sibling iceberg connector (which has its OWN native caches IcebergLatestSnapshotCache / IcebergManifestCache), and a hudi-on-HMS table is served by the hudi sibling connector. So does the fe-core IcebergExternalMetaCache / HudiExternalMetaCache still serve ANYTHING for a flipped hms catalog, or does it become fully dead (served by siblings)? Enumerate exactly what each caches, who feeds it, and whether the flip's sibling-delegation already covers it. Determine: can these two caches simply be RETIRED at the flip (no connector-owned replacement needed) because the sibling connectors' native caches cover it, or is there a residual fe-core-served path? Cite evidence. Note the execution plan's claim that IcebergExternalMetaCache:234 is "unreached post-flip" — verify.`, + }, + { + label: 'dim-consumers', + prompt: `${CTX} + +YOUR DIMENSION = classify EVERY consumer of the three fe-core caches into (a) LEGACY deletion-unit → dies at flip, needs no replacement; (b) FLIP-SURVIVOR → needs a replacement path post-flip; (c) fe-core plumbing that just reroutes. For each survivor, say what it needs and where it should get it after the flip. + +Known consumer files (grep of HiveExternalMetaCache|HudiExternalMetaCache|IcebergExternalMetaCache, non-test): +- fe-core LEGACY-suspect: datasource/hive/HMSExternalCatalog.java, HMSExternalTable.java, HiveDlaTable.java, source/HiveScanNode.java, datasource/hudi/HudiUtils.java, datasource/iceberg/source/IcebergScanNode.java, datasource/iceberg/IcebergUtils.java, statistics/HMSAnalysisTask.java, nereids/.../insert/HiveInsertExecutor.java, planner/HiveTableSink.java, datasource/hive/AcidUtil.java +- fe-core SURVIVOR-suspect: catalog/RefreshManager.java, datasource/CatalogMgr.java, datasource/FilePartitionUtils.java, datasource/hive/event/MetastoreEventsProcessor.java, tablefunction/MetadataGenerator.java, nereids/rules/expression/rules/SortedPartitionRanges.java, datasource/ExternalMetaCacheMgr.java +- CONNECTOR references (RESOLVE whether real import / comment / vendored copy): fe-connector-hive/.../HiveConnectorMetadata.java, fe-connector-hudi/.../HudiConnectorMetadata.java, fe-connector-iceberg/.../IcebergConnector.java + IcebergConnectorMetadata.java + IcebergLatestSnapshotCache.java + IcebergManifestCache.java + +For each file: open the actual reference line(s), state which cache/method it touches, and classify. Pay special attention to RefreshManager, CatalogMgr, FilePartitionUtils, MetadataGenerator, SortedPartitionRanges — these likely SURVIVE and drive REFRESH/invalidation/TVF; describe exactly how they invoke the cache and what the post-flip replacement must provide. For the CONNECTOR references: this is critical — a connector importing fe-core would break layering; determine the truth (package + import statement) and report it.`, + }, + { + label: 'dim-connector-template', + prompt: `${CTX} + +YOUR DIMENSION = the CONNECTOR-OWNED cache TEMPLATE (how paimon & iceberg-native own scan-side caches today — the pattern the hive connector must copy). +Read: +- fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheFactory.java + CacheSpec.java + MetaCacheEntry.java + MetaCacheEntryStats.java + package-info.java (the SHARED cache framework — note the memory: fe-core does NOT depend on this; it's a copied framework compiled into each plugin; Caffeine pinned to 2.9.3 lowest-common version). +- fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java + IcebergManifestCache.java + ManifestCacheValue.java + dlf/client/DLFCachedClientPool.java +- fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCache.java +- The IcebergConnector / PaimonConnector class to see where these caches are instantiated, held, and invalidated. + +Answer: How does a connector INSTANTIATE and HOLD a scan-side cache (per-connector field? via CacheFactory?)? How is it keyed/scoped (per-table, per-snapshot)? How is it INVALIDATED — is there an SPI the connector implements that fe-core's REFRESH TABLE / REFRESH CATALOG calls, or is it event/TTL only? CRITICALLY: how does fe-core's ExternalMetaCacheRouteResolver end up NOT routing to these connectors (paimon/iceberg-native catalogs get ENGINE_DEFAULT and it's harmless) — trace WHY it's harmless for them today, because that is exactly the state hive must reach. Note config (TTL, size, expire) — where does the connector read cache config from (catalog properties? Config?). This is the reference model for the hive connector cache; describe it precisely enough to copy.`, + }, + { + label: 'dim-hive-connector-state', + prompt: `${CTX} + +YOUR DIMENSION = the CURRENT state of caching (or absence) inside fe-connector-hive, and what scan-side metadata the hive connector fetches per query. +Read the fe-connector-hive metadata + scan path: +- fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java (does it cache anything? where does it get schema/partitions/files?) +- The HiveConnector class, the scan-plan provider, and the HMS client wrappers (HmsClient / ThriftHmsClient) in the hive/hms connector modules. +- fe/fe-connector/fe-connector-metastore-hms/** and fe-connector-metastore-spi/** if present (the shared HMS resolver). +Search the hive connector module for any existing Caffeine/cache usage, and for per-query getTableSchema / listPartitions / listPartitionNames / file-listing calls. + +Answer: Today (dormant, hms still legacy) does the hive connector cache ANY metadata, or does every getTableSchema/listPartitions/getTableStatistics call hit HMS thrift fresh? Enumerate the scan-side metadata reads the connector performs (schema, partition names, partition objects, file splits, column stats, table stats) and which are hot-path-per-query. Which of these does legacy HiveExternalMetaCache cache today that the connector currently re-fetches uncached? Note whether fe-connector-hive already depends on fe-connector-cache (check its pom.xml). Surface: what is the MINIMUM set of connector-owned caches needed at the flip for scan performance + correctness parity (schema? partition-name? partition-object? file-listing?), and note the TCCL/classloader considerations (filesystem listing runs plugin-side).`, + }, + { + label: 'dim-maxtime-invalidation', + prompt: `${CTX} + +YOUR DIMENSION = (§2.6) max-partition-time / getNewestUpdateVersionOrTime, PLUS the invalidation coupling between D2 and event-pipeline Model B. +Read: +- fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java around :713-714 (getNewestUpdateVersionOrTime returning 0 for flipped plain-hive) AND the getTableSnapshot / beginQuerySnapshot / freshness path (freshness-kind-aware, landed 3d784673ca4). +- How legacy computes newest-update-time: fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSExternalTable.java getNewestUpdateVersionOrTime + any max-partition-time logic, and how HiveExternalMetaCache serves it. +- fe/fe-core/src/main/java/org/apache/doris/dictionary/Dictionary.java + mtmv/MTMVRelatedTableIf.java (the CONSUMERS of getNewestUpdateVersionOrTime — SQL dictionary refresh + MV freshness). +- The invalidation path: ExternalMetaCacheInvalidator, ExternalMetaCacheMgr.invalidateTableCache / invalidateDb / invalidatePartitions, RefreshManager, and how REFRESH TABLE/CATALOG + the event pipeline currently invalidate the fe-core caches. +- The event pipeline: datasource/hive/event/MetastoreEventsProcessor.java (:116 instanceof HMSExternalCatalog) — note the planned "Connector.invalidatePartition(s)" SPI (§2.1 Model B) that pairs with D2. + +Answer: (1) Exactly how must the hive connector surface a cheap real max-partition modify time so flipped getNewestUpdateVersionOrTime matches legacy? What SPI shape is needed (ConnectorMvccPartitionView already exists — read fe-connector-api/.../mvcc/ConnectorMvccPartitionView.java and IcebergPartitionUtils.getNewestUpdateVersionOrTime for the iceberg precedent). (2) How is the connector-owned cache invalidated by REFRESH TABLE / REFRESH CATALOG and by metastore events — what invalidation SPI verbs does D2 need to expose, and how does that couple to event Model B (which is the NEXT task after D2)? Keep D2 and event-Model-B boundaries distinct but note the shared SPI. Surface options + tradeoffs for the max-time SPI and the invalidation SPI.`, + }, + { + label: 'dim-trino-reference', + prompt: `${CTX} + +YOUR DIMENSION = the Trino reference architecture for connector-owned Hive metadata caching, mapped onto this design. The user explicitly wants Trino consulted for architecture-level decisions. +From your knowledge of Trino (and any Trino-referencing comments in this codebase — grep for "Trino"/"trino" in fe-connector-hive and the metastore modules): +- How does Trino structure Hive metastore metadata caching? (CachingHiveMetastore: the per-plugin caching decorator wrapping the raw metastore client; cache config hive.metastore-cache-ttl / metastore-refresh-interval / metastore-cache-maximum-size; per-transaction vs shared caching; the SharedHiveMetastoreCache.) Where does caching live — engine or connector? How is it invalidated (flushMetadataCache procedure, per-table/per-partition flush)? +- How does Trino cache file listings / directory listings separately (the DirectoryLister / CachingDirectoryLister, hive.file-status-cache)? +- How does Trino avoid a central engine-side metadata cache (the SPI is stateless Metadata calls; the connector owns all caching)? + +Answer: Map each Trino mechanism to a concrete recommendation for the Doris hive connector D2 design: (1) a CachingHiveMetastore-style decorator around the HMS client inside fe-connector-hive; (2) a directory/file-listing cache; (3) invalidation verbs the connector exposes to fe-core REFRESH/events. Note where Doris ALREADY diverges (Doris fe-core historically centralized the cache in HiveExternalMetaCache; the D2 move re-aligns to Trino's connector-owned model). Flag any Trino mechanism that does NOT map cleanly (e.g. Doris's GSON edit-log replay, the SPI_READY flip model, the split file-listing across classloaders). Surface options where Trino offers more than one approach.`, + }, +] + +const readerResults = await parallel( + readers.map(r => () => agent(r.prompt, { label: r.label, phase: 'Recon', schema: READER_SCHEMA, effort: 'high' })) +) + +const valid = readerResults.filter(Boolean) +log(`Recon done: ${valid.length}/${readers.length} dimensions returned.`) + +// Build a compact digest of all reader findings for the critics. +function digest(list) { + return list.map(r => { + const facts = (r.facts || []).map(f => ` - ${f.claim} [${f.evidence}]${f.implication ? ' → ' + f.implication : ''}`).join('\n') + const flip = (r.flipImpact || []).map(x => ` - ${x}`).join('\n') + const opts = (r.designOptions || []).map(x => ` - ${x}`).join('\n') + const oq = (r.openQuestions || []).map(x => ` - ${x}`).join('\n') + return `### ${r.dimension}\nSUMMARY: ${r.summary}\nFACTS:\n${facts}\nFLIP-IMPACT:\n${flip}\nDESIGN-OPTIONS:\n${opts}\nOPEN-QUESTIONS:\n${oq}` + }).join('\n\n') +} +const allFindings = digest(valid) + +phase('Critique') + +const critics = [ + { + label: 'critic-completeness', + prompt: `${CTX} + +You are a COMPLETENESS critic. Below are the combined recon findings for the D2 connector-owned-cache design. Your job: find what is MISSING or unverified that would silently break post-flip if the connector-owned cache design omits it. Adversarially hunt for: (a) a consumer of the three fe-core caches not classified (legacy-die vs survivor); (b) a cached entry (schema/partition/file/stats/max-time) with no post-flip owner; (c) a REFRESH / DDL / event / TVF / dictionary / MV path that reads the cache and would collapse to ENGINE_DEFAULT silently; (d) a scan hot-path that becomes uncached (perf regression) at the flip; (e) an invalidation verb the connector cache won't receive. For each gap, cite the evidence file:line if you can and rank severity. Also RE-VERIFY any load-bearing claim you doubt by reasoning about the cited evidence. Return CRITIC_SCHEMA. + +COMBINED FINDINGS: +${allFindings}`, + }, + { + label: 'critic-flip-safety', + prompt: `${CTX} + +You are a FLIP-SAFETY & DORMANCY critic. Below are the combined recon findings. Your job: stress-test the core D2 premise — "the hive connector owns its scan-side cache, so at the flip, cache routing collapsing to ENGINE_DEFAULT is HARMLESS, and the 3 fe-core caches + 4 gates retire cleanly." Adversarially check: (1) Can the connector-owned cache land DORMANT (inert while hms is still legacy, exactly like every prior connector step) — or does anything force it to be co-committed with the flip? What must be dormant vs must ride the flip commit? (2) Is routing-collapse actually harmless, or is there a path where ENGINE_DEFAULT routing does something WRONG (not just no-op) — e.g. registry.resolve("default") throwing, or double-caching? (3) The iceberg/hudi fe-core caches: are they truly covered by sibling delegation post-flip, or is there a residual? (4) Ordering: does D2 have hidden dependencies on event Model B (the NEXT task) or vice versa — can D2 truly land first? (5) Any TCCL/classloader hazard when the connector-owned cache does filesystem listing or HMS calls. Rank severity, cite evidence, and list worries you REFUTED. Return CRITIC_SCHEMA. + +COMBINED FINDINGS: +${allFindings}`, + }, + { + label: 'critic-scope', + prompt: `${CTX} + +You are a SCOPE-BOUNDARY critic. Below are the combined recon findings. Your job: pin down the RIGHT scope for D2 so the design doc is neither bloated nor missing a flip-blocker. Adversarially decide, with evidence: (1) What is genuinely REQUIRED for a correct+performant flip (minimum viable connector cache set: which of schema/partition-name/partition-object/file-listing/stats/max-time) vs what can be DEFERRED post-flip as a perf item. (2) Does hudi/iceberg fe-core cache RETIREMENT belong inside the D2 commit-set or does it just ride the flip's deletion (since siblings cover them)? (3) Where is the clean boundary between D2 (cache) and event Model B (invalidation source) and §2.6 (max-time) — what shared SPI sits on the seam, and should max-time + invalidation SPI land in D2 or their own steps? (4) Is the shared fe-connector-cache framework the right home, or should the hive cache be connector-internal like PaimonLatestSnapshotCache? (5) How should D2 be decomposed into independent dormant commits (propose an ordered sub-step list, mirroring how the iceberg/hudi lines were decomposed). Cite evidence, rank, and note anything the plan §2.2/§2.6 got wrong vs HEAD. Return CRITIC_SCHEMA. + +COMBINED FINDINGS: +${allFindings}`, + }, +] + +const criticResults = await parallel( + critics.map(c => () => agent(c.prompt, { label: c.label, phase: 'Critique', schema: CRITIC_SCHEMA, effort: 'high' })) +) + +return { + readers: valid, + critics: criticResults.filter(Boolean), +} diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index b6942cd0967600..34200ae94ec103 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**下一步 = Phase 1 首步：连接器自持缓存（D2，含 max-partition-time；退休 fe-core 三份缓存 + 4 路由 instanceof 守卫；paimon/iceberg 自持缓存为模板）——先 recon+设计再实现。**（次为事件 Model B。） +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**下一步 = 实现 C-a（依次 C-b…C-f，每步独立休眠 commit + 单测 + checkstyle/import gate，完成后统一对抗复审再翻闸）。次子系统 = 事件 Model B。**（实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 diff --git a/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md b/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md new file mode 100644 index 00000000000000..a6b7bb600172a5 --- /dev/null +++ b/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md @@ -0,0 +1,334 @@ +# Hive connector-owned scan-side cache — step design (2026-07-10) + +> Phase-1 first item of the HMS cutover ("D2" in the execution plan). Produced by a HEAD-grounded +> recon (`wf_d19057ca-5bb`: 8 dimension readers + 3 adversarial critics — completeness / flip-safety / +> scope) with lead-engineer verification of every load-bearing fact. **Starting doc for the implementer.** +> Authoritative parent: `hms-cutover-execution-plan-2026-07-10.md` §2.2 + §2.6. HEAD `d43ba31f3b3`, +> branch `catalog-spi-11-hive`. + +--- + +## 0. TL;DR + +At the atomic flip a `type=hms` catalog stops being an `HMSExternalCatalog`, so +`ExternalMetaCacheRouteResolver.addBuiltinRoutes` (`:66-70`) no longer routes it to the fe-core +`HiveExternalMetaCache` / `HudiExternalMetaCache` / `IcebergExternalMetaCache`; routing falls through to +`ENGINE_DEFAULT` (`:72-73`). **This is not a crash** — `registry.resolve("default")` returns a real +`DefaultExternalMetaCache` (verified `ExternalMetaCacheRegistry:39-49`, `ExternalMetaCacheMgr:290-296`), +and schema caching *survives* because a flipped `PluginDrivenExternalTable` inherits +`getMetaCacheEngine()=="default"` (`ExternalTable:229-231`, no override) — the same `ENGINE_DEFAULT` +cache the schema feed and `invalidateTable` both target. **What silently dies is the engine-specific +metadata**: partition names, partition objects, and directory file listings — and the hive connector +**caches nothing today** (verified: no Caffeine/`MetaCacheEntry` anywhere in `fe-connector-hive`; +`HiveScanPlanProvider` header literally says "No file listing cache"). + +So **D2 exists to preserve legacy performance, not correctness**, plus to fix one coupled correctness +bug (§2.6). The whole step is **purely additive and dormant**: nothing is deleted, nothing changes for +paimon/iceberg/jdbc, and none of it goes live until `"hms"` enters `SPI_READY_TYPES` at the flip. + +**Deliverables:** (1) a connector-owned metastore + file-listing cache in `fe-connector-hive`, +(2) `HiveConnector.invalidateTable/invalidateAll` overrides (fe-core wiring already exists), (3) a cheap +real max-partition modify time so flipped `getNewestUpdateVersionOrTime` stops returning constant 0. + +--- + +## 1. Research note (what the recon established, HEAD-verified) + +### 1.1 The fe-core cache machinery — survives; only the hms *route* + the 3 engine caches are legacy +- `ExternalMetaCacheMgr` registers **five** engine caches: `default`, `hive`, `hudi`, `iceberg`, + `doris` (`:290-296`). It has two dispatch families: + - **instanceof-routed** (`prepareCatalog`/`invalidateCatalog`/`invalidateDb`/`invalidateTable`/ + `invalidatePartitions`/`removeCatalog`) → `ExternalMetaCacheRouteResolver.resolveCatalogCaches`. + This is the *only* place the `HMSExternalCatalog` gate lives; it collapses to `ENGINE_DEFAULT` at + the flip. + - **engine-name-routed** (`*ByEngine`, `hive()/hudi()/iceberg()/doris()`, `getSchemaCacheValue`) — + unaffected by the collapse. +- `ENGINE_DEFAULT` = `DefaultExternalMetaCache`, which registers **only a `schema` entry**. So the + collapse target is a working schema-only cache — no throw, no double-cache. **This is the exact state + native paimon/iceberg catalogs already run in today** (`PluginDrivenExternalCatalog` is not + `HMSExternalCatalog` → `ENGINE_DEFAULT`), which is why route-collapse is harmless for them and is + precisely the state hive must reach. +- **Do NOT retire** `AbstractExternalMetaCache` / `ExternalMetaCacheRegistry` / + `ExternalMetaCacheRouteResolver` / `MetaCacheEntry` / `ExternalMetaCacheMgr` / `DefaultExternalMetaCache` + — they serve `default`+`doris`+the `ExternalRowCountCache`+the catalog/db-listing cache for every + external catalog including PluginDriven. Only the **hms route branch** (`resolver:66-70` + its import) + and the three **engine-cache classes** are legacy. + +### 1.2 What `HiveExternalMetaCache` actually caches (the relocation surface) +Four entries (`HiveExternalMetaCache:119-161`): +| entry | key → value | loader hits | classification | +|---|---|---|---| +| `schema` | `SchemaCacheKey → SchemaCacheValue` | `externalCatalog.getSchema` | **redundant** post-flip — served by `ENGINE_DEFAULT` via `getMetaCacheEngine()=="default"`. No connector cache needed. | +| `partition_values` | `PartitionValueCacheKey → HivePartitionValues` | thrift `listPartitionNames` (`:283`) | **scan-side** (partition pruning). Connector must own. | +| `partition` | `PartitionCacheKey → HivePartition` | thrift `getPartition/getPartitions` (`:323/:367`) → inputFormat/location/**parameters** | **scan-side** + the **§2.6 max-modify-time source** (`transient_lastDdlTime` rides in `partition.getParameters()`, free in the thrift response). | +| `file` | `FileCacheKey → FileCacheValue` | **filesystem** `DirectoryLister.listFiles` (`:396`) | **scan hot-path**, the dominant per-scan cost. Connector must own. | + +The ACID `getFilesByTransaction` path (`:792-828`) is scan-side but **uncached** even in legacy +(`AcidUtil.getAcidState`); the connector already reimplements it (`HiveAcidUtil` wired at +`HiveScanPlanProvider:191`) — no cache-invalidation coupling to carry over. + +### 1.3 The hive connector today caches NOTHING +- No Caffeine/`LoadingCache`/`MetaCacheEntry` in `fe-connector-hive`, `-hms`, or `-metastore-hms`. + `ThriftHmsClient` "Cached" = connection pool only; every `getTable`/`listPartitionNames`/ + `getPartitions`/`getTableColumnStatistics` is a fresh RPC. +- `HiveScanPlanProvider` re-lists partitions **and** files on every scan (`:321/:326/:358/:361`); + `HiveConnectorMetadata.getTableHandle`+`getColumnHandles` each do a full `getTable` (`:302/:305/:422`). +- `fe-connector-hive/pom.xml` depends on neither `fe-connector-cache` nor Caffeine (unlike iceberg/paimon). + +### 1.4 The connector-owned cache template (paimon / iceberg-native) +- A connector holds its scan cache as **plain final fields on the per-catalog `Connector`**, built in + the ctor from **catalog properties** (not fe-core `Config`): `IcebergConnector.latestSnapshotCache` + +`manifestCache`, `PaimonConnector.latestSnapshotCache`. +- Each wrapper owns one `MetaCacheEntry` from the shared `fe-connector-cache` framework + (`CacheFactory`/`CacheSpec`/`MetaCacheEntry`; **Caffeine encapsulated** — never crosses to child code + as a type; Caffeine pinned **2.9.3**, self-bundled per plugin — hive is like paimon, no transitive + Caffeine). +- Config keys: `meta.cache...(enable|ttl-second|capacity)` via + `CacheSpec.fromProperties`; defaults hardcode legacy `Config` values (24h / capacity). fe-core does + **not** parse these (honors the no-property-parsing-in-fecore rule). +- **Invalidation is already wired connector-agnostically**: `REFRESH TABLE` → + `RefreshManager.refreshTableInternal` → `connector.invalidateTable(remoteDb, remoteTable)` + (`:247-249`); `REFRESH CATALOG` → `PluginDrivenExternalCatalog.onRefreshCache` → + `connector.invalidateAll()` (`:252-257`). Both are **no-op SPI defaults** (`Connector:306-316`) — hive + just overrides them. `ADD/MODIFY CATALOG` rebuilds the connector (caches dropped wholesale). TTL/LRU + evict. + +### 1.5 Trino reference (per the user's request to consult Trino) +Trino keeps **all** Hive caching inside the connector, in two separate layers: +- `CachingHiveMetastore` — a decorator wrapping the raw metastore client, caching + `getTable`/`getPartition`/`listPartitionNames`/`get*Statistics` under + `hive.metastore-cache-ttl` / `metastore-cache-maximum-size` (+ a strongly-consistent per-transaction + layer). Invalidated by a connector `flush_metadata_cache` procedure (per-table/per-partition). +- `CachingDirectoryLister` — a **separate** file-status cache (`hive.file-status-cache-*`). + +Doris historically did the opposite (central engine-side `HiveExternalMetaCache`). **D2 re-aligns Doris +to Trino's connector-owned, two-layer model.** The Doris-specific frictions Trino has no analog for: +(a) caches must be **transient** and never enter the GSON edit log (naturally satisfied — caches live on +the `Connector`, rebuilt on init/`REFRESH CATALOG`); (b) the atomic `SPI_READY` flip (the exact failure +D2 prevents); (c) file-listing crosses into the plugin's bundled Hadoop classloader (TCCL, §4.5). + +### 1.6 §2.6 — the one coupled correctness bug +`PluginDrivenMvccExternalTable.getNewestUpdateVersionOrTime` (`:699-715`) materializes the latest pin +and reads `nameToLastModifiedMillis`; hive's `listPartitions` is **names-only** (all `-1`) → filtered → +`orElse(0L)` → **constant 0**. It never checks `isLastModifiedFreshness` (unlike `getTableSnapshot` +`:616-635`). Consequence: a hive-backed **SQL dictionary** / **MV auto-refresh** never sees a newer +source version (`Dictionary.hasNewerSourceVersion:271-296` needs a monotone-increasing value) — silent, +no crash. The connector *already* has the right value cheaply: `getTableFreshness` (`:1032-1062`) maxes +`transient_lastDdlTime` over partitions (the same value legacy read from `HivePartition.parameters`) — +but it hits `hmsClient.getPartitions` **live** each call, so §2.6's fix must be **backed by the D2 +partition cache** or the periodic dictionary poll regresses to repeated uncached round-trips. + +--- + +## 2. Goals / Non-goals + +### Goals +1. The hive connector owns a scan-side cache (metastore metadata + directory listing) so that at the + flip, route-collapse to `ENGINE_DEFAULT` is **performance-neutral** vs legacy. +2. `HiveConnector.invalidateTable/invalidateAll` drop that cache (arms `REFRESH TABLE`/`REFRESH CATALOG`). +3. Fix §2.6: flipped `getNewestUpdateVersionOrTime` surfaces a real, cheap max-partition modify time, + cache-backed, restoring SQL-dictionary / MV auto-refresh over hive base tables. +4. Everything lands **dormant** (inert while hms is legacy; byte-neutral for all other connectors). + +### Non-goals (explicitly out of D2 — recorded so they are not silently dropped) +- **Deleting the fe-core caches + the 4 gates + the resolver hms branch.** Per the standing user + decision ("delete legacy last; reach a working flip first"), and because these have **zero live + readers post-flip** (all readers are legacy deletion-unit classes — `HiveScanNode`, `HMSExternalTable`, + `HiveDlaTable`, etc.), deletion rides the **final deletion phase**, not D2. D2 is purely additive; the + fe-core caches simply sit **unrouted** (harmless) after the flip until then. +- **Event-driven incremental invalidation (event Model B — the NEXT task).** Post-flip + `MetastoreEventsProcessor` is `instanceof HMSExternalCatalog`-gated (`:116`) → zero event sync for a + flipped catalog. D2 accepts interim staleness **bounded by TTL + explicit REFRESH**. Model B re-arms + the loop and adds `Connector.invalidatePartitions(db,table,names)`; it **depends on D2's cache**, not + vice-versa (verified: D2 lands first). D2 must **not** attempt partition-granular invalidation. +- **hudi-on-HMS caching regression.** The hudi sibling rebuilds `HoodieTableMetaClient` per query and has + no cache; post-flip it loses caching entirely (correctness fine, perf regression). This is a **separate + hudi-connector item** — but see §3 Option A, which would fix it for free. +- **Column statistics caching.** `getColumnStatistics` is uncached today but gated on `numRows>0` and off + the scan hot-path (planner fast-path only) → defer as a perf item. +- **The two TVF flip-breaks** (`partition_values()` `MetadataGenerator:2091`; `hudi_meta()` `:459`) and + the **`canSample`/`SUPPORTS_SAMPLE_ANALYZE`** stats-gate inconsistency (`AnalysisManager:1484`) — these + are live-SQL flip-survivors, but they belong to the **flip's coupling-seam set** (execution-plan §2.3), + not D2-cache authoring. Recon re-confirmed all three; they are tracked there. + +### Refuted worries (checked against HEAD — do NOT spend budget here) +- **`RefreshManager.refreshPartitions` CCE "blocker"** — REFUTED. **Verified: its sole caller is + `AlterPartitionEvent.java:123`** (event pipeline), which the `instanceof HMSExternalCatalog` event gate + never fires for a flipped catalog. It is Model-B / event-deletion territory, **not a D2 blocker.** + (Two critics initially mis-ranked this as a live-SQL blocker; the caller-trace refutes it.) +- **`CatalogMgr.add/dropExternalPartitions` silent no-op** — REFUTED as D2: callers are all event classes + (`AddPartitionEvent`/`DropPartitionEvent`/`AlterPartitionEvent`) → event-gated → Model B. +- **`replayRefreshTable` partition branch breaks** — REFUTED: post-flip the `instanceof HMSExternalCatalog` + guard is false → falls to `refreshTableInternal` which already fans out `connector.invalidateTable` for + PluginDriven. Graceful coarsening, no crash. +- **Schema cache incoherence at flip** — REFUTED (§1.1). +- **Connector references to fe-core `*ExternalMetaCache` are a layering violation** — REFUTED: + `grep 'import.*ExternalMetaCache'` over `fe-connector/` = zero; all mentions are javadoc. Deletion is + import-clean. +- **`ENGINE_DEFAULT` routing throws at the flip** — REFUTED (§1.1, it's a registered cache). + +--- + +## 3. ARCHITECTURE DECISION — where the metastore-metadata cache lives (needs sign-off) + +Both options put caching **inside the connector** (honoring the LOCKED D2 decision) and both need a +**separate file-listing cache** (§4.4) — Trino keeps these two layers separate too. They differ only in +how the *metastore-metadata* layer is built: + +**Option A — `CachingHmsClient` decorator (Trino `CachingHiveMetastore` style).** A +`CachingHmsClient implements HmsClient` in `fe-connector-hms`, wrapping `ThriftHmsClient`, caching +`getTable` / `listPartitionNames` / `getPartitions` / `getTableColumnStatistics` on +`MetaCacheEntry`s keyed by db/table (values are the connector's own `HmsTableInfo`/`HmsPartitionInfo` +DTOs). `HiveConnector` wraps its client once; `invalidateTable/invalidateAll` delegate to the decorator's +`flush(db,table)/flushAll()`. +- **Pros:** most Trino-faithful; **transparent** to all ~30 `hmsClient.*` call-sites in + `HiveConnectorMetadata` (no cache threading, low error surface); a **reusable class** — the hudi/iceberg + siblings each hold their own `HmsClient` from the same `fe-connector-hms` module (verified + `HudiConnector:68`), so later wrapping their clients with the same decorator **fixes the hudi-on-HMS + caching regression** with zero new machinery. +- **Cons:** caches raw metastore DTOs at RPC granularity (not the higher "resolved partition list" shape); + its own TTL/key config; forks slightly from the paimon/iceberg *shape* (though not the ownership model). + +**Option B — typed cache fields on `HiveConnector` (paimon/iceberg-native shape).** e.g. +`HivePartitionCache` (name+object; value carries max `transient_lastDdlTime` for §2.6) as final fields on +`HiveConnector`, mirroring `PaimonLatestSnapshotCache`; `HiveConnectorMetadata` reads through them. +- **Pros:** matches the existing Doris connector-owned precedent exactly (Rule 11); one obvious home; + aligns 1:1 with iceberg/paimon `invalidateTable/invalidateAll` lifecycle. +- **Cons:** hive-specific (does nothing for the hudi-on-HMS regression); must thread the cache through each + metadata method (more touch-points); caches at result granularity. + +**Recommendation: Option A.** Hive's `HmsClient` is exactly the thin thrift shape Trino decorates (unlike +paimon/iceberg's rich catalog clients), the decorator is transparent over the many call-sites, and its +reusability cleanly retires the hudi-on-HMS regression. It still honors "connector-owned" and keeps the +file-listing cache separate (Trino-aligned). **This is the primary decision for user sign-off (§8-Q1).** + +--- + +## 4. Design (assuming Option A; Option B differs only in §4.3 placement) + +### 4.1 Dependency + packaging (dormant, build-only) +Add `fe-connector-cache` + **Caffeine 2.9.3** to `fe-connector-hive/pom.xml` (paimon pattern — hive +carries no transitive Caffeine). Verify the built plugin zip carries **exactly one** Caffeine 2.9.3 +(memory `catalog-spi-connector-cache-framework-caffeine-coherence`: framework is child-loaded per plugin; +each consumer must self-bundle its lowest-common lib). + +### 4.2 Cache set (minimum viable for scan parity) +1. **Metastore-metadata cache** (Option A: inside `CachingHmsClient`): `getTable` (table meta incl. + `sd`+params → also feeds `getTableHandle`/`getColumnHandles`/`getTableStatistics`), `listPartitionNames` + (pruning), `getPartitions(names)` (per-partition location/inputFormat/**parameters** → also the §2.6 + max-`transient_lastDdlTime` source), `getTableColumnStatistics` (optional; gated, low priority). +2. **File-listing cache** (§4.4): directory `listStatus` results keyed by partition path. +3. **Schema: NOT cached by the connector** — `ENGINE_DEFAULT` covers it (§1.1). Connector `getTableSchema` + is hit only on a schema-cache miss. + +### 4.3 §2.6 — cheap max-partition modify time (dormant fe-core one-liner + connector backing) +- **fe-core** (`PluginDrivenMvccExternalTable.getNewestUpdateVersionOrTime`): add a last-modified branch + mirroring `getTableSnapshot` — when the query pin's `isLastModifiedFreshness()` is set, return + `queryTableFreshness().getTimestampMillis()`; else keep the exact existing path. + **Byte-neutrality constraint (memory `plugindriven-mvcc-table-is-live-not-dormant`):** paimon/iceberg + set `isLastModifiedFreshness()==false` → they keep the current RANGE/`orElse(0L)` path unchanged. The + new branch is dormant until a hive (last-modified) connector exists at the flip. Guard so paimon/iceberg + bytes **and** cost are unchanged. +- **connector**: `getTableFreshness`/`getPartitionFreshnessMillis` read the **D2 metastore-metadata cache** + (cached `getPartitions`), not a live RPC, so the periodic dictionary poll stays cheap. +- **Parity (verified):** legacy `getNewestUpdateVersionOrTime` returns 0 **only for a genuinely empty + partition set**; for an unpartitioned table it builds a partition from table params → table + `transient_lastDdlTime`. The connector's `getTableFreshness` already returns `lastDdlMillis(tableParams)` + for unpartitioned and `0` for empty-partition-set (`:1039-1046`) — **matching legacy**. Confirm at + implementation; if a divergence surfaces, surface it for sign-off (do not silently change behavior). +- **Monotonicity:** `Dictionary.hasNewerSourceVersion` THROWS on a *smaller* value than last seen. Confirm + `transient_lastDdlTime` is non-decreasing across partition rewrites (it is HMS-maintained epoch seconds). + +### 4.4 File-listing cache + TCCL +The directory listing runs plugin-side via Hadoop `FileSystem` reflection. **Adopt the template's +contextual + manual-miss + `autoRefresh=false` pattern** so the loader runs on the **caller (scan) +thread**, which `PluginDrivenScanNode.onPluginClassLoader` already TCCL-pins — the safest choice. If a +dedicated listing executor with background refresh is chosen for perf, **pin TCCL to the connector +classloader inside every loader** (mirror the existing save/set/restore at +`HiveConnectorMetadata:719-727`). This same file-listing cache should also back +`estimateDataSizeByListingFiles` (`:815`) so the periodic `ExternalRowCountCache` row-count refresh for a +no-stats plain-hive table is not an uncached re-listing (critic-confirmed 5th-cache coupling). +> Note: `fsCache` (fe-core `FileSystemCache`) is NOT part of D2 — the connector uses Hadoop's native +> `FileSystem` cache (scheme+authority+UGI keyed). Confirm `doAs`/UGI keying doesn't defeat it. + +### 4.5 Invalidation scope (coarse only) +- Override `HiveConnector.invalidateTable(db,table)` (drop that table's metastore + file entries) and + `invalidateAll()` (clear all). fe-core wiring already exists (§1.4) — no fe-core change. +- **REFRESH DB gap:** only `invalidateTable`+`invalidateAll` reach the connector; `REFRESH DB` on a + flipped catalog won't drop per-table entries for that db. **Recommendation: accept per-table/all + coverage** for D2 (REFRESH CATALOG covers it; a db-level verb is Model-B-adjacent). Documented, not asked. +- Partition-granular invalidation + event re-arming = **Model B**, out of scope. + +### 4.6 Dormancy proof +- `"hms"` is absent from `SPI_READY_TYPES` (`CatalogFactory:55-56`) → **no `HiveConnector` is built for + hms** until the flip; the cache fields/decorator exist but are never instantiated for a live catalog. +- `invalidateTable/invalidateAll` overrides fire only for a `PluginDrivenExternalCatalog` holding a + `HiveConnector` — none exist for hms pre-flip. +- The §2.6 fe-core branch only executes for `isLastModifiedFreshness()==true` — no live connector sets it. +- ⇒ Every sub-step is byte-neutral for paimon/iceberg/jdbc/doris and inert for hms until the flip. + +--- + +## 5. Decomposition — ordered dormant commits (mirrors the iceberg/hudi lines) + +| step | what | notes | +|---|---|---| +| **C-a** | pom: add `fe-connector-cache` + Caffeine 2.9.3 | build-only, dormant | +| **C-b** | metastore-metadata cache (Option A: `CachingHmsClient` in `-hms`; or Option B fields), from `meta.cache.hive.*` props, **unconsulted** | dormant field/decorator, not yet wired | +| **C-c** | route `getTable`/`listPartitionNames`/`getPartitions` through it **and** back `getTableFreshness`/`getPartitionFreshnessMillis` with it | dormant (connector not built for hms yet) | +| **C-d** | file-listing cache + route `HiveScanPlanProvider` listing + `estimateDataSizeByListingFiles` through it (TCCL) | separable, highest complexity | +| **C-e** | implement `HiveConnector.invalidateTable/invalidateAll` | arms REFRESH TABLE/CATALOG | +| **C-f** | §2.6 fe-core `getNewestUpdateVersionOrTime` last-modified branch (reads the now-cheap cached max time) | byte-neutral for paimon/iceberg | + +Each is an independent reviewable commit with its own tests, exactly like every prior connector step. +After all land, run a unified adversarial re-review (clean-room, per project practice) before the flip. + +--- + +## 6. Edge cases / risks +- **Byte-neutrality for live PluginDriven MVCC connectors** (§4.3) — the single sharpest correctness risk; + guard the §2.6 branch on `isLastModifiedFreshness()` and add a paimon/iceberg no-change test. +- **TCCL split-brain** on async listing threads (§4.4) — prefer the pinned-scan-thread loader. +- **Caffeine version split-brain** in the plugin zip (§4.1) — verify exactly one 2.9.3. +- **Interim event staleness** (§2 non-goals) — bounded by TTL + REFRESH until Model B; call it out, don't + hide it (Rule 12). +- **Cache must never enter the GSON edit log** (§1.5c) — keep all state on the `Connector`/decorator; no + `*Info`/handle carries a cache reference. + +## 7. Testing (dormant-testable) +- `CachingHmsClient` (or the typed cache): hit/miss, TTL/capacity from props, `flush(db,table)`/`flushAll`, + value identity — same-loader unit tests in `fe-connector-hive`/`-hms`. +- §2.6: a `PluginDrivenMvccExternalTableTest` asserting (a) `isLastModifiedFreshness` → freshness millis; + (b) paimon/iceberg (snapshot-id) path **unchanged** (byte/cost parity); (c) unpartitioned → tableDdl, + empty-partition-set → 0. +- Invalidation: `HiveConnector.invalidateTable/invalidateAll` drop the right entries. +- checkstyle 0 + import gate net, per project bar. +- **e2e-owed (flip / Phase 4):** cache hit under a real flipped hms catalog; REFRESH TABLE/CATALOG + end-to-end; dictionary/MV auto-refresh over a hive base sees new data; hudi-on-HMS perf (separate). + +--- + +## 8. Open decisions — RESOLVED (user sign-off 2026-07-10) + +- **Q1 — metastore-metadata cache placement → Option A: `CachingHmsClient` decorator (Trino-style).** + A `CachingHmsClient implements HmsClient` in `fe-connector-hms` wrapping `ThriftHmsClient`; transparent + to the ~30 call-sites; reusable so the hudi/iceberg siblings can later wrap their own clients (fixes the + hudi-on-HMS regression for free). §3. +- **Q2 — file-listing cache scope → include it in this step (step C-d).** Full legacy parity at the flip; + accept the TCCL/executor complexity (prefer the pinned-scan-thread loader, §4.4). Do NOT defer. +- **Settled without asking** (recorded, will proceed unless corrected): deletion of the fe-core caches → + final deletion phase, not D2; §2.6 → design for exact legacy parity, surface only if divergent; REFRESH + DB → accept per-table/all coverage; column-stats → defer; hudi-on-HMS perf → separate hudi item (or free + via Q1-Option-A). + +--- + +## 9. TODO (fill after Q1/Q2 sign-off) +- [ ] C-a pom dep + Caffeine 2.9.3 (+ verify single-copy in zip) +- [ ] C-b metastore-metadata cache (per Q1) from `meta.cache.hive.*`, unconsulted + tests +- [ ] C-c wire metadata reads + freshness probes through it + tests +- [ ] C-d file-listing cache (per Q2) + TCCL + row-count backing + tests +- [ ] C-e `invalidateTable/invalidateAll` overrides + tests +- [ ] C-f §2.6 fe-core last-modified branch (byte-neutral guard) + tests +- [ ] unified adversarial re-review before the flip +- [ ] update HANDOFF.md per step (per-phase discipline) From b0a287b2e5e2b98a5814e9fdfdf1ce0baed34ee2 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 07:33:40 +0800 Subject: [PATCH 147/334] [feat](catalog) Hive connector cache: add fe-connector-cache + Caffeine 2.9.3 dep (dormant, build-only) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit First step of the Hive connector-owned scan-side cache (the first Phase-1 item before the HMS cutover; authoritative design plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md, step C-a). Adds fe-connector-cache (the shared meta-cache framework: CacheSpec / MetaCacheEntry / CacheFactory) and Caffeine 2.9.3 to fe-connector-hive/pom.xml, mirroring fe-connector-paimon exactly: - fe-connector-cache declares Caffeine `provided` (compiles against, never bundles), so each consuming plugin must self-bundle its own Caffeine. - iceberg gets Caffeine 2.9.3 transitively from iceberg-core; hive (like paimon) has no transitive Caffeine, so it is declared here and pinned to 2.9.3 = the version fe-connector-cache is compiled against (lowest common version across consuming plugins) to avoid a plugin-zip version split-brain. Build-only and fully dormant: no source code, no new SPI, byte-neutral for paimon/iceberg/jdbc, and inert for hms (still absent from SPI_READY_TYPES, so no HiveConnector is ever built for a live catalog). The cache classes are not yet consulted anywhere — that begins at step C-b. Verified: - reactor build SUCCESS; hive plugin zip carries EXACTLY ONE caffeine jar (lib/caffeine-2.9.3.jar) plus lib/fe-connector-cache-*.jar, bundled child-first (no version split-brain despite 7 Caffeine versions in .m2). - hive module: 240 tests, 0 failures/errors/skipped (surefire actually executed, build cache disabled); checkstyle 0 violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- fe/fe-connector/fe-connector-hive/pom.xml | 24 +++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/pom.xml b/fe/fe-connector/fe-connector-hive/pom.xml index 3bc4646ca89de1..a55e1f9769a204 100644 --- a/fe/fe-connector/fe-connector-hive/pom.xml +++ b/fe/fe-connector/fe-connector-hive/pom.xml @@ -47,6 +47,30 @@ under the License. ${project.version} + + + ${project.groupId} + fe-connector-cache + ${project.version} + + + + + com.github.ben-manes.caffeine + caffeine + 2.9.3 + + ${project.groupId} From 81cc4e7d90dcd70163ebdea820e131192cf3826b Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 07:34:21 +0800 Subject: [PATCH 148/334] [doc](catalog) Hive connector cache: record C-a DONE, next = C-b (per-phase handoff) Marks the pom/Caffeine dependency step (C-a) complete in HANDOFF.md with its commit, verification results (single caffeine-2.9.3 in the zip, 240 hive tests green, checkstyle 0), and sets the next dormant step to C-b (the CachingHmsClient decorator field, unwired, per the signed-off Option A). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 34200ae94ec103..5bb433a2f41aa5 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**下一步 = 实现 C-a（依次 C-b…C-f，每步独立休眠 commit + 单测 + checkstyle/import gate，完成后统一对抗复审再翻闸）。次子系统 = 事件 Model B。**（实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**下一步 = C-b（`CachingHmsClient` 装饰器休眠字段，未接线，per Q1 方案 A）。** 依次 C-b…C-f 每步独立休眠 commit + 单测 + checkstyle/import gate，全部落地后统一对抗复审再翻闸。次子系统 = 事件 Model B。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 From 67e4fb93093c30c524f27e4eef20b90b00d980a0 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 08:30:21 +0800 Subject: [PATCH 149/334] [feat](catalog) Hive connector cache: CachingHmsClient decorator (dormant, unwired) Step C-b of the Hive connector-owned scan-side cache (Phase-1 item before the HMS cutover; authoritative design plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md, step C-b). Adds CachingHmsClient in fe-connector-hms: a transparent HmsClient decorator (Trino CachingHiveMetastore shape, signed-off Option A) that caches the four read methods on the shared fe-connector-cache MetaCacheEntry framework and delegates every other method (+ close()) verbatim: - getTable key (db, table) -> HmsTableInfo - listPartitionNames key (db, table, maxParts) -> partition names - getPartitions key (db, table, name-list) -> partition list - getTableColumnStatistics key (db, table, column-list) -> column stats getPartitions/column-stats use RPC-argument (whole-list, order-sensitive) keys by design (per-partition assembly would need Hive partition-name reconstruction that is unavailable here). Every key carries (db, table) so flush(db, table) drops one table (exact-key for the table cache, predicate invalidateIf for the three others) and flushAll() clears everything -- both ready for the later REFRESH wiring. Mirrors PaimonLatestSnapshotCache/IcebergLatestSnapshotCache: contextual-only + manual-miss load so a slow HMS RPC runs outside Caffeine's compute lock; Caffeine stays encapsulated. Defaults mirror legacy fe-core Config (TTL 86400s; caps 10000/10000/100000/10000), read from the connector's own catalog properties via meta.cache.hive..* (no fe-core Config). Caching all four methods (incl. column statistics) per user sign-off 2026-07-10, aligning with the design's four-method enumeration. pom: add fe-connector-cache (compile) so the decorator compiles in -hms, and Caffeine 2.9.3 (test scope) for the unit test's MetaCacheEntry. Caffeine stays `provided`/non-transitive in fe-connector-cache and is bundled by the hive plugin (C-a). This completes what C-a placed only on -hive (the code lives in -hms). Dormant: nothing constructs CachingHmsClient (HiveConnector still returns a raw ThriftHmsClient; "hms" is absent from SPI_READY_TYPES), so no live catalog ever instantiates it. Byte-neutral for paimon/iceberg/jdbc; the new transitive fe-connector-cache jar reaches the hudi plugin closure but is unused there in this step (hudi carries no Caffeine, needs one only if it later reuses the decorator). Verified: - fe-connector-hms: 68 tests, 0 failures/errors/skipped; checkstyle 0. - CachingHmsClientTest: 12 mutation-tight tests (hit/miss keyed by every dimension incl. db; per-entry enable/ttl knobs; flush per-table across all four caches; flushAll all four; verbatim pass-through; exception propagated and not cached; empty column-stats result cached; null delegate rejected). - fe-connector-hive + fe-connector-hudi regression: BUILD SUCCESS (transitive fe-connector-cache breaks neither); forbidden-import gate clean. - 4-lens clean-room adversarial review (wf_39f6458c-e37) + per-finding refutation: 0 production defects; 4 confirmed minor test-coverage gaps fixed here, 1 refuted nit (mutable list container by reference = accepted cache convention; javadoc clarified). Next: C-c wires the decorator into HiveConnector.createClient and routes the metadata reads + freshness probes through it. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- fe/fe-connector/fe-connector-hms/pom.xml | 25 + .../doris/connector/hms/CachingHmsClient.java | 431 ++++++++++++++++++ .../connector/hms/CachingHmsClientTest.java | 410 +++++++++++++++++ 3 files changed, 866 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java diff --git a/fe/fe-connector/fe-connector-hms/pom.xml b/fe/fe-connector/fe-connector-hms/pom.xml index 013a6945cef52c..eb7c4470b198b3 100644 --- a/fe/fe-connector/fe-connector-hms/pom.xml +++ b/fe/fe-connector/fe-connector-hms/pom.xml @@ -47,6 +47,18 @@ under the License. ${project.version} + + + ${project.groupId} + fe-connector-cache + ${project.version} + + org.apache.doris @@ -116,6 +128,19 @@ under the License. commons-lang test + + + + com.github.ben-manes.caffeine + caffeine + 2.9.3 + test + diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java new file mode 100644 index 00000000000000..ef62569aab0ad7 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -0,0 +1,431 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ForkJoinPool; +import java.util.function.Function; + +/** + * A caching {@link HmsClient} decorator: it wraps another {@code HmsClient} (in production the pooled + * {@link ThriftHmsClient}) and serves the three scan-hot-path read methods from a bounded, TTL-expiring + * cache, delegating every other method verbatim. + * + *

    Why this exists. Today the hive connector caches nothing — {@code getTable}, + * {@code listPartitionNames} and {@code getPartitions} are fresh Thrift RPCs on every scan. Legacy fe-core + * kept these in the engine-side {@code HiveExternalMetaCache}, which stops routing to a hive catalog once + * it becomes a plugin-driven ({@code SPI_READY}) catalog. This decorator re-homes that caching inside the + * connector (Trino {@code CachingHiveMetastore} shape), so the connector stays performance-neutral vs + * legacy after the cutover. Because the {@code HmsClient} is also held by the hudi/iceberg siblings from + * this same module, the decorator is reusable by them later.

    + * + *

    What it caches (4 methods), each on its own {@link MetaCacheEntry} configured from catalog + * properties {@code meta.cache.hive..(enable|ttl-second|capacity)} (defaults mirror the legacy + * fe-core {@code Config} values — the connector is {@code Config}-free):

    + *
      + *
    • {@code getTable} — keyed by {@code (db, table)} → {@link HmsTableInfo}.
      • + *
    • {@code listPartitionNames} — keyed by {@code (db, table, maxParts)} → partition-name list. Real + * callers pass the unbounded {@code maxParts}, so this is effectively one entry per table; keeping + * {@code maxParts} in the key keeps a bounded request from ever being served a fuller list.
      • + *
    • {@code getPartitions} — keyed by {@code (db, table, requested-name-list)} → partition list. This is + * RPC-argument granularity: a different requested set is a separate entry (no cross-request + * assembly). {@link HmsPartitionInfo} carries {@code transient_lastDdlTime} in its parameters, which + * a later step reads through this cache for the table max-modify-time.
      • + *
    • {@code getTableColumnStatistics} — keyed by {@code (db, table, requested-column-list)} → the + * (possibly sparse or empty) stats list. Same RPC-argument granularity; the empty-list "no stats" + * result is a legitimate cached value (only {@code null} loads are skipped). This is the planner + * column-stats fast path, off the scan hot path, so it caches at low priority but on the same + * machinery as the rest.
      • + *
    + * + *

    Pass-through. Every other read, plus every write / DDL / ACID method, is passed straight + * through to the delegate. A later invalidation step arms {@link #flush(String, String)} / + * {@link #flushAll()} onto {@code REFRESH TABLE} / {@code REFRESH CATALOG}. This decorator does NOT + * self-invalidate around writes — coarse REFRESH + TTL bound staleness.

    + * + *

    Cache-value safety. {@code HmsTableInfo} / {@code HmsPartitionInfo} / {@code HmsColumnStatistics} + * are immutable (all fields final, collections unmodifiable), so caching them by reference is safe. The + * three list-returning methods cache and return the delegate's outer {@code List} container by reference and + * do NOT defensively copy it — its elements are immutable but the container is shared, so callers must treat + * a returned collection as read-only (the codebase-wide metadata-cache convention). Null loads are never + * cached (the framework treats {@code null} as a miss), and a loader exception ({@link HmsClientException}) + * propagates to the caller and is not cached.

    + * + *

    Dormant. Nothing wraps a client with this decorator yet — {@code HiveConnector} still returns a + * raw {@code ThriftHmsClient}, and {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog + * builds a {@code HiveConnector} at all. Wiring the decorator into the client and the freshness probes is a + * later step; this class is fully unit-testable in isolation now.

    + */ +public class CachingHmsClient implements HmsClient { + + /** Engine token for the {@code meta.cache...*} property namespace. */ + static final String ENGINE = "hive"; + /** {@code meta.cache.hive.table.*} — cached {@link HmsTableInfo}. */ + static final String ENTRY_TABLE = "table"; + /** {@code meta.cache.hive.partition_names.*} — cached partition-name lists. */ + static final String ENTRY_PARTITION_NAMES = "partition_names"; + /** {@code meta.cache.hive.partition.*} — cached partition-object lists. */ + static final String ENTRY_PARTITION = "partition"; + /** {@code meta.cache.hive.column_stats.*} — cached column-statistics lists. */ + static final String ENTRY_COLUMN_STATS = "column_stats"; + + // Legacy fe-core Config values, mirrored locally (the connector never touches fe-core Config): + // TTL = Config.external_cache_expire_time_seconds_after_access (86400s = 24h), shared by all entries + // table cap = Config.max_external_schema_cache_num (per-table metadata sizing) + // names cap = Config.max_hive_partition_table_cache_num (per-table partition-name lists) + // part cap = Config.max_hive_partition_cache_num (partition objects) + // stats cap = Config.max_external_schema_cache_num (per-table, no legacy hive cache; reuse table sizing) + static final long DEFAULT_TTL_SECOND = 86400L; + static final long DEFAULT_TABLE_CAPACITY = 10000L; + static final long DEFAULT_PARTITION_NAMES_CAPACITY = 10000L; + static final long DEFAULT_PARTITION_CAPACITY = 100000L; + static final long DEFAULT_COLUMN_STATS_CAPACITY = 10000L; + + private final HmsClient delegate; + private final MetaCacheEntry tableCache; + private final MetaCacheEntry> partitionNamesCache; + private final MetaCacheEntry> partitionsCache; + private final MetaCacheEntry> columnStatsCache; + + public CachingHmsClient(HmsClient delegate, Map properties) { + this.delegate = Objects.requireNonNull(delegate, "delegate can not be null"); + Map props = properties == null ? Collections.emptyMap() : properties; + this.tableCache = newEntry("hive.table", props, ENTRY_TABLE, DEFAULT_TABLE_CAPACITY); + this.partitionNamesCache = + newEntry("hive.partition_names", props, ENTRY_PARTITION_NAMES, DEFAULT_PARTITION_NAMES_CAPACITY); + this.partitionsCache = newEntry("hive.partition", props, ENTRY_PARTITION, DEFAULT_PARTITION_CAPACITY); + this.columnStatsCache = + newEntry("hive.column_stats", props, ENTRY_COLUMN_STATS, DEFAULT_COLUMN_STATS_CAPACITY); + } + + private static MetaCacheEntry newEntry(String name, Map props, + String entry, long defaultCapacity) { + CacheSpec spec = CacheSpec.fromProperties(props, ENGINE, entry, + CacheSpec.of(true, DEFAULT_TTL_SECOND, defaultCapacity)); + // Contextual-only + manual-miss load so a slow HMS RPC runs outside Caffeine's sync compute lock + // (deduplicated by a striped lock instead), mirroring PaimonLatestSnapshotCache / IcebergLatestSnapshotCache. + return new MetaCacheEntry<>(name, null, spec, ForkJoinPool.commonPool(), false, true, 0L, true); + } + + // ========== Cached reads ========== + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + return tableCache.get(new TableKey(dbName, tableName), + key -> delegate.getTable(key.dbName, key.tableName)); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return partitionNamesCache.get(new PartitionNamesKey(dbName, tableName, maxParts), + key -> delegate.listPartitionNames(key.dbName, key.tableName, key.maxParts)); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + return partitionsCache.get(new PartitionsKey(dbName, tableName, partNames), + key -> delegate.getPartitions(key.dbName, key.tableName, key.partNames)); + } + + @Override + public List getTableColumnStatistics(String dbName, String tableName, + List columns) { + return columnStatsCache.get(new ColumnStatsKey(dbName, tableName, columns), + key -> delegate.getTableColumnStatistics(key.dbName, key.tableName, key.columns)); + } + + // ========== Coarse invalidation (wired onto REFRESH TABLE / REFRESH CATALOG in a later step) ========== + + /** Drop every cached entry for one table. Backs {@code REFRESH TABLE}. */ + public void flush(String dbName, String tableName) { + tableCache.invalidateKey(new TableKey(dbName, tableName)); + partitionNamesCache.invalidateIf(key -> key.matches(dbName, tableName)); + partitionsCache.invalidateIf(key -> key.matches(dbName, tableName)); + columnStatsCache.invalidateIf(key -> key.matches(dbName, tableName)); + } + + /** Drop the whole cache. Backs {@code REFRESH CATALOG}. */ + public void flushAll() { + tableCache.invalidateAll(); + partitionNamesCache.invalidateAll(); + partitionsCache.invalidateAll(); + columnStatsCache.invalidateAll(); + } + + // ========== Pass-through: everything else is delegated verbatim ========== + + @Override + public List listDatabases() { + return delegate.listDatabases(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + return delegate.getDatabase(dbName); + } + + @Override + public List listTables(String dbName) { + return delegate.listTables(dbName); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return delegate.tableExists(dbName, tableName); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return delegate.getDefaultColumnValues(dbName, tableName); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + return delegate.getPartition(dbName, tableName, values); + } + + @Override + public void createDatabase(HmsCreateDatabaseRequest request) { + delegate.createDatabase(request); + } + + @Override + public void dropDatabase(String dbName) { + delegate.dropDatabase(dbName); + } + + @Override + public void createTable(HmsCreateTableRequest request) { + delegate.createTable(request); + } + + @Override + public void dropTable(String dbName, String tableName) { + delegate.dropTable(dbName, tableName); + } + + @Override + public void truncateTable(String dbName, String tableName, List partitions) { + delegate.truncateTable(dbName, tableName, partitions); + } + + @Override + public void addPartitions(String dbName, String tableName, List partitions) { + delegate.addPartitions(dbName, tableName, partitions); + } + + @Override + public void updateTableStatistics(String dbName, String tableName, + Function update) { + delegate.updateTableStatistics(dbName, tableName, update); + } + + @Override + public void updatePartitionStatistics(String dbName, String tableName, String partitionName, + Function update) { + delegate.updatePartitionStatistics(dbName, tableName, partitionName, update); + } + + @Override + public boolean dropPartition(String dbName, String tableName, List partitionValues, + boolean deleteData) { + return delegate.dropPartition(dbName, tableName, partitionValues, deleteData); + } + + @Override + public boolean partitionExists(String dbName, String tableName, List partitionValues) { + return delegate.partitionExists(dbName, tableName, partitionValues); + } + + @Override + public long openTxn(String user) { + return delegate.openTxn(user); + } + + @Override + public void commitTxn(long txnId) { + delegate.commitTxn(txnId); + } + + @Override + public Map getValidWriteIds(String fullTableName, long currentTransactionId) { + return delegate.getValidWriteIds(fullTableName, currentTransactionId); + } + + @Override + public void acquireSharedLock(String queryId, long txnId, String user, String dbName, + String tableName, List partitionNames, long timeoutMs) { + delegate.acquireSharedLock(queryId, txnId, user, dbName, tableName, partitionNames, timeoutMs); + } + + @Override + public void close() throws IOException { + delegate.close(); + } + + // ========== Cache keys ========== + // All keys carry (db, table) so flush(db, table) can select every entry for one table. + + static final class TableKey { + private final String dbName; + private final String tableName; + + TableKey(String dbName, String tableName) { + this.dbName = dbName; + this.tableName = tableName; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof TableKey)) { + return false; + } + TableKey that = (TableKey) o; + return Objects.equals(dbName, that.dbName) && Objects.equals(tableName, that.tableName); + } + + @Override + public int hashCode() { + return Objects.hash(dbName, tableName); + } + } + + static final class PartitionNamesKey { + private final String dbName; + private final String tableName; + private final int maxParts; + + PartitionNamesKey(String dbName, String tableName, int maxParts) { + this.dbName = dbName; + this.tableName = tableName; + this.maxParts = maxParts; + } + + boolean matches(String db, String table) { + return Objects.equals(dbName, db) && Objects.equals(tableName, table); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof PartitionNamesKey)) { + return false; + } + PartitionNamesKey that = (PartitionNamesKey) o; + return maxParts == that.maxParts + && Objects.equals(dbName, that.dbName) + && Objects.equals(tableName, that.tableName); + } + + @Override + public int hashCode() { + return Objects.hash(dbName, tableName, maxParts); + } + } + + static final class PartitionsKey { + private final String dbName; + private final String tableName; + // Order-sensitive, defensively copied: same requested list (same order) → hit; a different set or + // order → separate entry. Simple and correct (no partition-name reconstruction from values). + private final List partNames; + + PartitionsKey(String dbName, String tableName, List partNames) { + this.dbName = dbName; + this.tableName = tableName; + this.partNames = partNames == null + ? Collections.emptyList() + : Collections.unmodifiableList(new ArrayList<>(partNames)); + } + + boolean matches(String db, String table) { + return Objects.equals(dbName, db) && Objects.equals(tableName, table); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof PartitionsKey)) { + return false; + } + PartitionsKey that = (PartitionsKey) o; + return Objects.equals(dbName, that.dbName) + && Objects.equals(tableName, that.tableName) + && Objects.equals(partNames, that.partNames); + } + + @Override + public int hashCode() { + return Objects.hash(dbName, tableName, partNames); + } + } + + static final class ColumnStatsKey { + private final String dbName; + private final String tableName; + // Order-sensitive, defensively copied (same as PartitionsKey): the value is exactly the (sparse or + // empty) stats list for this requested column set. + private final List columns; + + ColumnStatsKey(String dbName, String tableName, List columns) { + this.dbName = dbName; + this.tableName = tableName; + this.columns = columns == null + ? Collections.emptyList() + : Collections.unmodifiableList(new ArrayList<>(columns)); + } + + boolean matches(String db, String table) { + return Objects.equals(dbName, db) && Objects.equals(tableName, table); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ColumnStatsKey)) { + return false; + } + ColumnStatsKey that = (ColumnStatsKey) o; + return Objects.equals(dbName, that.dbName) + && Objects.equals(tableName, that.tableName) + && Objects.equals(columns, that.columns); + } + + @Override + public int hashCode() { + return Objects.hash(dbName, tableName, columns); + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java new file mode 100644 index 00000000000000..d27bcdd9f7efd7 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -0,0 +1,410 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Tests {@link CachingHmsClient}: the caching decorator over an {@link HmsClient}. + * + *

    WHY: at the HMS cutover a hive catalog stops routing to the engine-side {@code HiveExternalMetaCache}, + * so the connector must cache these reads itself or every scan regresses to fresh Thrift RPCs. These tests + * pin the behaviours that make that re-homed cache correct: (1) the four read methods actually cache (loader + * runs once per key), keyed exactly by their arguments — including the database dimension, so two databases + * never collide; (2) the per-entry {@code meta.cache.hive.*} knobs turn a cache off; (3) + * {@link CachingHmsClient#flush} / {@code flushAll} drop the right entries across all four caches (arming + * REFRESH) and {@code flush} is scoped to one table; and that other methods are a verbatim pass-through and + * a loader failure is neither swallowed nor cached.

    + */ +public class CachingHmsClientTest { + + private static Map props(String... kv) { + Map m = new HashMap<>(); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + // ---- getTable ---- + + @Test + public void getTableCachesByDbAndTable() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + HmsTableInfo first = cache.getTable("db", "t1"); + HmsTableInfo second = cache.getTable("db", "t1"); + // WHY: a hit must serve the cached instance without re-hitting the metastore. + Assertions.assertSame(first, second); + Assertions.assertEquals(1, delegate.getTableCalls); + + // WHY: a different table is a different key — must NOT serve t1's value. + HmsTableInfo other = cache.getTable("db", "t2"); + Assertions.assertNotSame(first, other); + Assertions.assertEquals(2, delegate.getTableCalls); + } + + @Test + public void cacheKeysAreScopedByDatabase() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Same table name, different database, across all four caches. WHY: the db dimension MUST be part of + // every key — otherwise "db2.t" would be served "db1.t"'s cached metadata (a cross-database mix-up). + HmsTableInfo t1 = cache.getTable("db1", "t"); + HmsTableInfo t2 = cache.getTable("db2", "t"); + Assertions.assertNotSame(t1, t2); + Assertions.assertEquals(2, delegate.getTableCalls); + + cache.listPartitionNames("db1", "t", -1); + cache.listPartitionNames("db2", "t", -1); + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + + cache.getPartitions("db1", "t", Arrays.asList("p=1")); + cache.getPartitions("db2", "t", Arrays.asList("p=1")); + Assertions.assertEquals(2, delegate.getPartitionsCalls); + + cache.getTableColumnStatistics("db1", "t", Arrays.asList("c1")); + cache.getTableColumnStatistics("db2", "t", Arrays.asList("c1")); + Assertions.assertEquals(2, delegate.getColumnStatsCalls); + } + + // ---- listPartitionNames ---- + + @Test + public void listPartitionNamesCachesByDbTableAndMaxParts() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + List a = cache.listPartitionNames("db", "t", -1); + List b = cache.listPartitionNames("db", "t", -1); + Assertions.assertSame(a, b); + Assertions.assertEquals(1, delegate.listPartitionNamesCalls); + + // WHY: maxParts is part of the key — a bounded request must never be served the unbounded list. + cache.listPartitionNames("db", "t", 10); + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + } + + // ---- getPartitions ---- + + @Test + public void getPartitionsCachesByRequestedNameList() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + List names = Arrays.asList("p=1", "p=2"); + List a = cache.getPartitions("db", "t", names); + List b = cache.getPartitions("db", "t", new ArrayList<>(names)); + // WHY: same requested set+order hits even if the caller passes a fresh list instance. + Assertions.assertSame(a, b); + Assertions.assertEquals(1, delegate.getPartitionsCalls); + + // WHY: a different requested set is a distinct entry (RPC-argument granularity). + cache.getPartitions("db", "t", Arrays.asList("p=1")); + Assertions.assertEquals(2, delegate.getPartitionsCalls); + + // WHY: the key is order-sensitive by design (no reorder assembly) — reversed order re-loads. + cache.getPartitions("db", "t", Arrays.asList("p=2", "p=1")); + Assertions.assertEquals(3, delegate.getPartitionsCalls); + } + + // ---- column statistics ---- + + @Test + public void columnStatisticsCacheByRequestedColumnList() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + List cols = Arrays.asList("c1", "c2"); + List a = cache.getTableColumnStatistics("db", "t", cols); + List b = cache.getTableColumnStatistics("db", "t", new ArrayList<>(cols)); + // WHY: same requested column set+order hits. + Assertions.assertSame(a, b); + Assertions.assertEquals(1, delegate.getColumnStatsCalls); + // WHY: the delegate's real stats must survive the cache, not the interface's empty-list default. + Assertions.assertEquals(1, a.size()); + Assertions.assertEquals("c1", a.get(0).getColumnName()); + + // WHY: a different requested column set is a distinct entry (RPC-argument granularity). + cache.getTableColumnStatistics("db", "t", Arrays.asList("c1")); + Assertions.assertEquals(2, delegate.getColumnStatsCalls); + } + + @Test + public void emptyColumnStatisticsResultIsCached() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // The fake returns an empty (no-stats) list for an empty column request. + cache.getTableColumnStatistics("db", "t", Collections.emptyList()); + cache.getTableColumnStatistics("db", "t", Collections.emptyList()); + // WHY: an empty "no stats" result is a real cached value (only null is treated as a miss) — it must + // NOT be re-fetched, or a table without column stats would hit HMS on every planner probe. + Assertions.assertEquals(1, delegate.getColumnStatsCalls); + } + + // ---- per-entry property knobs ---- + + @Test + public void perEntryPropertiesControlCaching() { + // table cache disabled via enable=false; partition_names disabled via ttl-second=0; partition left on. + Map properties = props( + "meta.cache.hive.table.enable", "false", + "meta.cache.hive.partition_names.ttl-second", "0"); + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, properties); + + cache.getTable("db", "t"); + cache.getTable("db", "t"); + // WHY: enable=false must bypass caching entirely — every call reloads. + Assertions.assertEquals(2, delegate.getTableCalls); + + cache.listPartitionNames("db", "t", -1); + cache.listPartitionNames("db", "t", -1); + // WHY: ttl-second=0 also disables the cache (a distinct knob from enable). + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + + cache.getPartitions("db", "t", Arrays.asList("p=1")); + cache.getPartitions("db", "t", Arrays.asList("p=1")); + // WHY: an unconfigured entry stays enabled by default — proves the knobs are read PER entry. + Assertions.assertEquals(1, delegate.getPartitionsCalls); + } + + // ---- flush(db, table) ---- + + @Test + public void flushDropsOnlyThatTablesEntries() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Populate ALL four caches for BOTH t1 and t2 (t2 must live in the three predicate-invalidated caches + // too, not just the table cache, so an over-broad flush of them is detectable). + cache.getTable("db", "t1"); + cache.listPartitionNames("db", "t1", -1); + cache.getPartitions("db", "t1", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db", "t1", Arrays.asList("c1")); + cache.getTable("db", "t2"); + cache.listPartitionNames("db", "t2", -1); + cache.getPartitions("db", "t2", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db", "t2", Arrays.asList("c1")); + Assertions.assertEquals(2, delegate.getTableCalls); + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + Assertions.assertEquals(2, delegate.getPartitionsCalls); + Assertions.assertEquals(2, delegate.getColumnStatsCalls); + + cache.flush("db", "t1"); + + // WHY: t1 must reload across all four caches after its flush. + cache.getTable("db", "t1"); + cache.listPartitionNames("db", "t1", -1); + cache.getPartitions("db", "t1", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db", "t1", Arrays.asList("c1")); + Assertions.assertEquals(3, delegate.getTableCalls); + Assertions.assertEquals(3, delegate.listPartitionNamesCalls); + Assertions.assertEquals(3, delegate.getPartitionsCalls); + Assertions.assertEquals(3, delegate.getColumnStatsCalls); + + // WHY: flush is scoped to ONE table — t2's entries in ALL four caches must survive (no reload). This + // pins the matches() per-table scoping of the three predicate caches, not just the table cache's + // exact-key invalidation: an over-broad flush that wiped every table would reload t2 here. + cache.getTable("db", "t2"); + cache.listPartitionNames("db", "t2", -1); + cache.getPartitions("db", "t2", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db", "t2", Arrays.asList("c1")); + Assertions.assertEquals(3, delegate.getTableCalls); + Assertions.assertEquals(3, delegate.listPartitionNamesCalls); + Assertions.assertEquals(3, delegate.getPartitionsCalls); + Assertions.assertEquals(3, delegate.getColumnStatsCalls); + } + + // ---- flushAll() ---- + + @Test + public void flushAllDropsEverything() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Populate all four caches so flushAll's independent invalidateAll() call on each is exercised. + cache.getTable("db", "t"); + cache.listPartitionNames("db", "t", -1); + cache.getPartitions("db", "t", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db", "t", Arrays.asList("c1")); + Assertions.assertEquals(1, delegate.getTableCalls); + Assertions.assertEquals(1, delegate.listPartitionNamesCalls); + Assertions.assertEquals(1, delegate.getPartitionsCalls); + Assertions.assertEquals(1, delegate.getColumnStatsCalls); + + cache.flushAll(); + + // WHY: flushAll drops ALL four caches — every one reloads (not just the table cache). + cache.getTable("db", "t"); + cache.listPartitionNames("db", "t", -1); + cache.getPartitions("db", "t", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db", "t", Arrays.asList("c1")); + Assertions.assertEquals(2, delegate.getTableCalls); + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + Assertions.assertEquals(2, delegate.getPartitionsCalls); + Assertions.assertEquals(2, delegate.getColumnStatsCalls); + } + + // ---- pass-through delegation ---- + + @Test + public void nonCachedMethodsDelegate() throws IOException { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + cache.listDatabases(); + Assertions.assertEquals(1, delegate.listDatabasesCalls); + + cache.dropTable("db", "t"); + Assertions.assertEquals(1, delegate.dropTableCalls); + + cache.close(); + Assertions.assertEquals(1, delegate.closeCalls); + } + + // ---- loader failures ---- + + @Test + public void loaderExceptionPropagatesAndIsNotCached() { + RecordingHmsClient delegate = new RecordingHmsClient(); + delegate.getTableError = new HmsClientException("boom"); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + HmsClientException e = Assertions.assertThrows(HmsClientException.class, + () -> cache.getTable("db", "t")); + Assertions.assertEquals("boom", e.getMessage()); + Assertions.assertEquals(1, delegate.getTableCalls); + + // WHY: a failed load must NOT be cached — after recovery, the next call reloads and succeeds. + delegate.getTableError = null; + HmsTableInfo ok = cache.getTable("db", "t"); + Assertions.assertNotNull(ok); + Assertions.assertEquals(2, delegate.getTableCalls); + } + + @Test + public void nullDelegateRejected() { + Assertions.assertThrows(NullPointerException.class, + () -> new CachingHmsClient(null, Collections.emptyMap())); + } + + /** + * A minimal {@link HmsClient} that counts calls and returns a fresh instance per call, so reference + * identity distinguishes a cache hit (same instance) from a reload (new instance). + */ + private static final class RecordingHmsClient implements HmsClient { + int getTableCalls; + int listPartitionNamesCalls; + int getPartitionsCalls; + int getColumnStatsCalls; + int listDatabasesCalls; + int dropTableCalls; + int closeCalls; + RuntimeException getTableError; + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + getTableCalls++; + if (getTableError != null) { + throw getTableError; + } + return HmsTableInfo.builder().dbName(dbName).tableName(tableName).build(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + listPartitionNamesCalls++; + return new ArrayList<>(Arrays.asList("p=1", "p=2")); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + getPartitionsCalls++; + List out = new ArrayList<>(); + for (int i = 0; i < partNames.size(); i++) { + out.add(new HmsPartitionInfo(Arrays.asList("v"), "loc", null, null, null, null)); + } + return out; + } + + @Override + public List getTableColumnStatistics(String dbName, String tableName, + List columns) { + getColumnStatsCalls++; + if (columns.isEmpty()) { + return Collections.emptyList(); + } + return new ArrayList<>(Arrays.asList(new HmsColumnStatistics("c1", 1L, 0L, 4.0))); + } + + @Override + public List listDatabases() { + listDatabasesCalls++; + return Collections.emptyList(); + } + + @Override + public void dropTable(String dbName, String tableName) { + dropTableCalls++; + } + + @Override + public void close() { + closeCalls++; + } + + // Unused abstract methods — trivial stubs. + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + return null; + } + + @Override + public List listTables(String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return false; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + return null; + } + } +} From a823790e77f0612193404bc1e15bfb4795002207 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 08:31:51 +0800 Subject: [PATCH 150/334] [doc](catalog) Hive connector cache: record C-b DONE, next = C-c (per-phase handoff) CachingHmsClient decorator (4fe55d88fab) landed dormant: transparent HmsClient decorator in fe-connector-hms caching getTable/listPartitionNames/getPartitions/ getTableColumnStatistics on the shared MetaCacheEntry framework, delegating all else; flush/flushAll ready for the later REFRESH wiring. Records the 4-lens clean-room adversarial review (0 production defects; 4 minor test-coverage gaps fixed, 1 refuted nit) and the user sign-off to cache column statistics too. Next = C-c: wire the decorator into HiveConnector.createClient and route the metadata reads + freshness probes through it (still dormant). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 5bb433a2f41aa5..ac2d1e85e603cc 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**下一步 = C-b（`CachingHmsClient` 装饰器休眠字段，未接线，per Q1 方案 A）。** 依次 C-b…C-f 每步独立休眠 commit + 单测 + checkstyle/import gate，全部落地后统一对抗复审再翻闸。次子系统 = 事件 Model B。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**✅ C-b DONE（`4fe55d88fab`，休眠）= `CachingHmsClient` 装饰器（方案 A，置 `fe-connector-hms`）**：`implements HmsClient` 透明包裹底层 client（生产=`ThriftHmsClient`），对 **4** 个只读方法各挂一 `MetaCacheEntry`（镜像 paimon/iceberg：contextual-only+manual-miss、`ForkJoinPool.commonPool()`、Caffeine 封装不外泄）——`getTable`(键 db,table)/`listPartitionNames`(键 db,table,maxParts)/`getPartitions`(键 db,table,请求名列表)/`getTableColumnStatistics`(键 db,table,请求列列表)；其余 20 方法+`close()` 逐字委派。`getPartitions`/列统计用 **RPC 参数(整列表、顺序敏感)键**（非 per-partition/per-column 装配——后者需重建 Hive 分区名字符串、此处无从取；Option A 已签字接受 RPC 粒度）。键全带 db,table 以便 `flush(db,table)` 按表清（tableCache 精确 invalidateKey，3 谓词缓存 invalidateIf(matches)）、`flushAll()` 清全部（为 C-e REFRESH TABLE/CATALOG 接线预备）。默认镜像 legacy fe-core Config（TTL 86400s；容量 table/names/partition/stats=10000/10000/100000/10000），经 `CacheSpec.fromProperties(props,"hive",,default)` 读 `meta.cache.hive..*`，连接器只读自己 props（不碰 fe-core Config）。**用户签字（2026-07-10）：第 4 类列统计一起缓存**（对齐 Q1 四方法枚举；"非目标"里"先不做"被 Q1 覆盖）。**pom**：`-hms` 加 `fe-connector-cache`(compile，让装饰器编译)+Caffeine 2.9.3(test scope，单测构造 MetaCacheEntry 需)；补 C-a"依赖只加在 -hive"的遗漏（类在 -hms）；Caffeine 仍 `provided`/非传递、由 -hive 插件打包。**残留（记账）**：`-hms` 亦被 `-hudi` 依赖→`fe-connector-cache` jar 传递进 hudi 闭包但 Caffeine provided 不进→hudi 若日后自包 client 复用须自带 Caffeine 2.9.3；C-b 里 hudi 不构造它→无 NoClassDefFound、仅休眠 jar。**休眠证**：grep 无任何处引用 `CachingHmsClient`、`SPI_READY_TYPES` 无 hms→永不实例化；`HiveConnector.createClient():379` 仍返裸 `ThriftHmsClient`（接线归 C-c）。**测**：`CachingHmsClientTest` 12 测（命中+各维含 **db 维**/props 三档开关/flush 按表 4 缓存隔离/flushAll 4 缓存全清/透传委派/异常传播不缓存/空列统计亦缓存/null delegate 拒）；hms 68 测 + checkstyle 0 + import gate 净；-hive/-hudi 回归 BUILD SUCCESS。**4 维 clean-room 对抗复审（`wf_39f6458c-e37`：委派完整/缓存正确/休眠打包/测试强度 + 逐条 refute）：生产码 0 缺陷**；5 findings→4 CONFIRMED（全 minor 测覆盖缺口，已补：db 维/flushAll 全缓存/flush 谓词缓存按表作用域/测类 javadoc 陈旧）+1 refuted（可变 list 容器按引返回=既有全库缓存惯例、休眠无变更方；顺带补主类 javadoc 说明容器共享须只读）。**下一步 = C-c**（`HiveConnector.createClient` 用 `CachingHmsClient` 包 `ThriftHmsClient`、元数据读走缓存 + 用缓存回填 `getTableFreshness`/`getPartitionFreshnessMillis`；仍休眠，连接器未为 hms 构建）。 依次 C-c…C-f 每步独立休眠 commit + 单测 + checkstyle/import gate，全部落地后统一对抗复审再翻闸。次子系统 = 事件 Model B。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 From ed8b8e7e5d937593f29162eb38436f77c294d45c Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:07:01 +0800 Subject: [PATCH 151/334] [feat](catalog) Hive connector cache: wire CachingHmsClient into the connector client (dormant) Make HiveConnector.createClient wrap the raw pooled ThriftHmsClient in the connector-owned CachingHmsClient (added, unwired, in the prior step) via a new package-private wrapWithCache seam. This single wrap makes every hmsClient.* read in HiveConnectorMetadata cache-backed transparently -- getTable / listPartitionNames / getPartitions / column-stats -- including the MTMV freshness probes getTableFreshness / getPartitionFreshnessMillis (both read hmsClient.getPartitions), so a hive-backed SQL dictionary / MV auto-refresh poll is served from the cache instead of a fresh Thrift RPC every tick. Rationale: at the hms cutover a hive catalog stops routing to the fe-core engine-side HiveExternalMetaCache (route collapses to the schema-only ENGINE_DEFAULT); without this connector-owned cache every scan would regress to uncached metastore RPCs. The decorator reads its per-entry knobs from the catalog's own meta.cache.hive.* properties (disabled/ttl<=0 => pass-through) and lives on the HmsClient, never on a handle or the GSON edit log. Dormant: "hms" is not in SPI_READY_TYPES, so no live catalog builds a HiveConnector; the wrap only runs in unit tests until the flip. Byte-neutral for paimon/iceberg/jdbc/hudi (only HiveConnector changed). The wrapWithCache seam mirrors the existing newMetadata(HmsClient) seam so a unit test can drive the wrap with a fake client without createClient building a real ThriftHmsClient (whose Hadoop stack is absent from connector tests). Tests: new HiveConnectorClientCacheTest (4) -- wrapWithCache returns a CachingHmsClient; repeated getTableFreshness / getPartitionFreshnessMillis are served from the cache (one RPC, not two); disabling meta.cache.hive.partition via props makes getPartitions reload while the still-enabled partition-name cache stays cached (per-entry props threading at the connector level). Full hive suite 244 green + checkstyle 0 + connector import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 28 ++- .../hive/HiveConnectorClientCacheTest.java | 228 ++++++++++++++++++ 2 files changed, 255 insertions(+), 1 deletion(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorClientCacheTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 4967331355703b..66b4d8c062c8e7 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -26,6 +26,7 @@ import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; +import org.apache.doris.connector.hms.CachingHmsClient; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.ThriftHmsClient; @@ -376,7 +377,32 @@ public T execute(Callable callable) throws Exception { } else { authAction = context::executeAuthenticated; } - return new ThriftHmsClient(config, authAction); + return wrapWithCache(new ThriftHmsClient(config, authAction)); + } + + /** + * Wraps the raw pooled metastore client in the connector-owned {@link CachingHmsClient} so this connector + * keeps caching its scan-side metastore reads AFTER the hms cutover. Once a hive catalog becomes plugin-driven + * the fe-core engine-side {@code HiveExternalMetaCache} stops routing to it, so without this wrap every + * {@code getTable} / {@code listPartitionNames} / {@code getPartitions} / column-stats read would regress to a + * fresh Thrift RPC on every scan. This single wrap makes ALL {@code hmsClient.*} reads in + * {@link HiveConnectorMetadata} cache-backed transparently — including the + * {@link HiveConnectorMetadata#getTableFreshness}/{@link HiveConnectorMetadata#getPartitionFreshnessMillis} + * MTMV probes (both read {@code hmsClient.getPartitions}), so the periodic SQL-dictionary / MV freshness poll + * stays cheap instead of hitting the metastore every tick. + * + *

    The decorator reads its per-entry knobs from this catalog's own {@code meta.cache.hive.*} properties; a + * disabled entry (or {@code ttl-second}/{@code capacity} <= 0) makes it a transparent pass-through. The + * cache lives on the {@code HmsClient} (rebuilt on connector init / {@code ADD}/{@code MODIFY CATALOG}), never + * on a handle or the GSON edit log. + * + *

    Extracted (package-private) so a unit test can verify the wrap + caching WITHOUT {@link #createClient()} + * building a real {@code ThriftHmsClient} (whose Hadoop stack is absent from connector unit tests) — mirrors + * {@link #newMetadata(HmsClient)}. Dormant: {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog + * builds a {@code HiveConnector} — this wrap only runs in unit tests until the flip. + */ + HmsClient wrapWithCache(HmsClient raw) { + return new CachingHmsClient(raw, properties); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorClientCacheTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorClientCacheTest.java new file mode 100644 index 00000000000000..518d5aaa499134 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorClientCacheTest.java @@ -0,0 +1,228 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.mvcc.ConnectorTableFreshness; +import org.apache.doris.connector.hms.CachingHmsClient; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.OptionalLong; + +/** + * Tests the C-c wiring: {@link HiveConnector} hands its {@link HiveConnectorMetadata} a caching metastore client, + * so every scan-side read (and the MTMV freshness probes) is served from the connector-owned cache after the hms + * cutover instead of a fresh Thrift RPC. + * + *

    Why this matters (Rule 9): when a hive catalog becomes plugin-driven at the flip, the fe-core + * engine-side {@code HiveExternalMetaCache} stops routing to it (it collapses to the schema-only + * {@code ENGINE_DEFAULT}). Without this connector-level wrap, {@code getTable} / {@code listPartitionNames} / + * {@code getPartitions} would each become an uncached RPC on every scan, and the periodic SQL-dictionary / MV + * freshness poll ({@link HiveConnectorMetadata#getTableFreshness} / + * {@link HiveConnectorMetadata#getPartitionFreshnessMillis}, both backed by {@code getPartitions}) would hit the + * metastore every tick. These tests pin that the connector's own {@code wrapWithCache} produces a + * {@link CachingHmsClient}, that reads through it are cache-backed end-to-end, and that the catalog's + * {@code meta.cache.hive.*} properties reach that cache (so it can be turned off). The decorator's internal + * caching correctness is covered separately by {@code CachingHmsClientTest} — this suite tests only the wiring. + * + *

    Dormant: {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog builds a {@link HiveConnector}; + * this exercises the wrap directly, as production {@code createClient} will at the flip. + */ +public class HiveConnectorClientCacheTest { + + private static final String METASTORE_URI = "thrift://host:9083"; + private static final List PART_KEYS = Arrays.asList("year", "month"); + private static final String PART_NAME = "year=2024/month=01"; + private static final String TRANSIENT_LAST_DDL_TIME = "transient_lastDdlTime"; + + private static Map props(String... kv) { + Map m = new HashMap<>(); + m.put("hive.metastore.uris", METASTORE_URI); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + private HiveTableHandle partitionedHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(PART_KEYS) + .build(); + } + + // ==================== the connector wraps its client in a caching decorator ==================== + + @Test + public void wrapWithCacheReturnsACachingDecorator() { + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + + HmsClient wrapped = connector.wrapWithCache(raw); + + // WHY: production createClient hands this wrapped client to HiveConnectorMetadata. If it returned the raw + // client (no wrap), every metadata read would be an uncached RPC after the flip. + Assertions.assertTrue(wrapped instanceof CachingHmsClient, + "the connector must wrap its metastore client in the caching decorator"); + Assertions.assertNotSame(raw, wrapped, "the wrapped client must not be the raw delegate"); + } + + // ==================== table freshness is served from the cache (§2.6 dictionary/MV poll stays cheap) ==== + + @Test + public void repeatedTableFreshnessHitsTheCacheNotTheMetastore() { + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + HiveConnectorMetadata md = metadataOver(connector.wrapWithCache(raw)); + + ConnectorTableFreshness first = md.getTableFreshness(null, partitionedHandle()).orElse(null); + ConnectorTableFreshness second = md.getTableFreshness(null, partitionedHandle()).orElse(null); + + Assertions.assertNotNull(first); + Assertions.assertNotNull(second); + Assertions.assertEquals(300_000L, first.getTimestampMillis(), + "freshness must still surface the real max transient_lastDdlTime (x1000)"); + // WHY: the second poll must be served from the cache — one listPartitionNames + one getPartitions RPC + // total, not two. This is exactly what keeps a hive-backed SQL dictionary's periodic version poll cheap. + Assertions.assertEquals(1, raw.listPartitionNamesCalls, + "a repeated table-freshness poll must not re-list partition names"); + Assertions.assertEquals(1, raw.getPartitionsCalls, + "a repeated table-freshness poll must not re-fetch partitions (served from the cache)"); + } + + @Test + public void repeatedPartitionFreshnessHitsTheCache() { + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + HiveConnectorMetadata md = metadataOver(connector.wrapWithCache(raw)); + + OptionalLong first = md.getPartitionFreshnessMillis(null, partitionedHandle(), PART_NAME); + OptionalLong second = md.getPartitionFreshnessMillis(null, partitionedHandle(), PART_NAME); + + Assertions.assertTrue(first.isPresent()); + Assertions.assertEquals(300_000L, first.getAsLong()); + Assertions.assertEquals(first.getAsLong(), second.getAsLong()); + // WHY: the same partition requested twice is one cache entry (RPC-argument granularity) — one round-trip. + Assertions.assertEquals(1, raw.getPartitionsCalls, + "a repeated per-partition freshness fetch for the same partition must be served from the cache"); + } + + // ==================== the catalog's meta.cache.hive.* props reach the connector-owned cache ============== + + @Test + public void disablingThePartitionCacheViaPropsMakesFreshnessReloadEachTime() { + // Disable ONLY the partition-object cache; leave the partition-name cache on. This proves the connector + // threads its own catalog properties into the decorator (so an operator can turn caching off) and that the + // knobs are read PER entry. + HiveConnector connector = + new HiveConnector(props("meta.cache.hive.partition.enable", "false"), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + HiveConnectorMetadata md = metadataOver(connector.wrapWithCache(raw)); + + md.getTableFreshness(null, partitionedHandle()); + md.getTableFreshness(null, partitionedHandle()); + + // WHY: with the partition cache disabled, getPartitions reloads every poll... + Assertions.assertEquals(2, raw.getPartitionsCalls, + "disabling meta.cache.hive.partition must make getPartitions reload on every freshness poll"); + // ...while the still-enabled partition-name cache is served once — proving the knob is per entry. + Assertions.assertEquals(1, raw.listPartitionNamesCalls, + "the still-enabled partition-name cache must not reload when only the partition cache is off"); + } + + private HiveConnectorMetadata metadataOver(HmsClient client) { + return new HiveConnectorMetadata(client, Collections.emptyMap(), new FakeConnectorContext()); + } + + /** + * A minimal {@link HmsClient} that counts the two freshness-backing calls and returns a single partition with + * a {@code transient_lastDdlTime}, so a cache hit (one call) is distinguishable from a reload (two calls). + */ + private static final class RecordingHmsClient implements HmsClient { + int getPartitionsCalls; + int listPartitionNamesCalls; + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + listPartitionNamesCalls++; + return new ArrayList<>(Collections.singletonList(PART_NAME)); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + getPartitionsCalls++; + List out = new ArrayList<>(); + for (String name : partNames) { + List values = HiveWriteUtils.toPartitionValues(name); + Map params = Collections.singletonMap(TRANSIENT_LAST_DDL_TIME, "300"); + out.add(new HmsPartitionInfo(values, "loc", "if", "of", "serde", params)); + } + return out; + } + + // Unused abstract methods — trivial stubs (never hit by the freshness path). + @Override + public List listDatabases() { + return Collections.emptyList(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + return null; + } + + @Override + public List listTables(String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return false; + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + return null; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + return null; + } + + @Override + public void close() { + } + } +} From 0e7c92356855bbb3bd7b432246e18b3e149f89e9 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:07:52 +0800 Subject: [PATCH 152/334] [doc](catalog) Hive connector cache: record C-c DONE, next = C-d (per-phase handoff) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- .../tasks/hive-connector-cache-step-design-2026-07-10.md | 8 +++++--- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index ac2d1e85e603cc..dcbb69e43b353a 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**✅ C-b DONE（`4fe55d88fab`，休眠）= `CachingHmsClient` 装饰器（方案 A，置 `fe-connector-hms`）**：`implements HmsClient` 透明包裹底层 client（生产=`ThriftHmsClient`），对 **4** 个只读方法各挂一 `MetaCacheEntry`（镜像 paimon/iceberg：contextual-only+manual-miss、`ForkJoinPool.commonPool()`、Caffeine 封装不外泄）——`getTable`(键 db,table)/`listPartitionNames`(键 db,table,maxParts)/`getPartitions`(键 db,table,请求名列表)/`getTableColumnStatistics`(键 db,table,请求列列表)；其余 20 方法+`close()` 逐字委派。`getPartitions`/列统计用 **RPC 参数(整列表、顺序敏感)键**（非 per-partition/per-column 装配——后者需重建 Hive 分区名字符串、此处无从取；Option A 已签字接受 RPC 粒度）。键全带 db,table 以便 `flush(db,table)` 按表清（tableCache 精确 invalidateKey，3 谓词缓存 invalidateIf(matches)）、`flushAll()` 清全部（为 C-e REFRESH TABLE/CATALOG 接线预备）。默认镜像 legacy fe-core Config（TTL 86400s；容量 table/names/partition/stats=10000/10000/100000/10000），经 `CacheSpec.fromProperties(props,"hive",,default)` 读 `meta.cache.hive..*`，连接器只读自己 props（不碰 fe-core Config）。**用户签字（2026-07-10）：第 4 类列统计一起缓存**（对齐 Q1 四方法枚举；"非目标"里"先不做"被 Q1 覆盖）。**pom**：`-hms` 加 `fe-connector-cache`(compile，让装饰器编译)+Caffeine 2.9.3(test scope，单测构造 MetaCacheEntry 需)；补 C-a"依赖只加在 -hive"的遗漏（类在 -hms）；Caffeine 仍 `provided`/非传递、由 -hive 插件打包。**残留（记账）**：`-hms` 亦被 `-hudi` 依赖→`fe-connector-cache` jar 传递进 hudi 闭包但 Caffeine provided 不进→hudi 若日后自包 client 复用须自带 Caffeine 2.9.3；C-b 里 hudi 不构造它→无 NoClassDefFound、仅休眠 jar。**休眠证**：grep 无任何处引用 `CachingHmsClient`、`SPI_READY_TYPES` 无 hms→永不实例化；`HiveConnector.createClient():379` 仍返裸 `ThriftHmsClient`（接线归 C-c）。**测**：`CachingHmsClientTest` 12 测（命中+各维含 **db 维**/props 三档开关/flush 按表 4 缓存隔离/flushAll 4 缓存全清/透传委派/异常传播不缓存/空列统计亦缓存/null delegate 拒）；hms 68 测 + checkstyle 0 + import gate 净；-hive/-hudi 回归 BUILD SUCCESS。**4 维 clean-room 对抗复审（`wf_39f6458c-e37`：委派完整/缓存正确/休眠打包/测试强度 + 逐条 refute）：生产码 0 缺陷**；5 findings→4 CONFIRMED（全 minor 测覆盖缺口，已补：db 维/flushAll 全缓存/flush 谓词缓存按表作用域/测类 javadoc 陈旧）+1 refuted（可变 list 容器按引返回=既有全库缓存惯例、休眠无变更方；顺带补主类 javadoc 说明容器共享须只读）。**下一步 = C-c**（`HiveConnector.createClient` 用 `CachingHmsClient` 包 `ThriftHmsClient`、元数据读走缓存 + 用缓存回填 `getTableFreshness`/`getPartitionFreshnessMillis`；仍休眠，连接器未为 hms 构建）。 依次 C-c…C-f 每步独立休眠 commit + 单测 + checkstyle/import gate，全部落地后统一对抗复审再翻闸。次子系统 = 事件 Model B。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**✅ C-b DONE（`4fe55d88fab`，休眠）= `CachingHmsClient` 装饰器（方案 A，置 `fe-connector-hms`）**：`implements HmsClient` 透明包裹底层 client（生产=`ThriftHmsClient`），对 **4** 个只读方法各挂一 `MetaCacheEntry`（镜像 paimon/iceberg：contextual-only+manual-miss、`ForkJoinPool.commonPool()`、Caffeine 封装不外泄）——`getTable`(键 db,table)/`listPartitionNames`(键 db,table,maxParts)/`getPartitions`(键 db,table,请求名列表)/`getTableColumnStatistics`(键 db,table,请求列列表)；其余 20 方法+`close()` 逐字委派。`getPartitions`/列统计用 **RPC 参数(整列表、顺序敏感)键**（非 per-partition/per-column 装配——后者需重建 Hive 分区名字符串、此处无从取；Option A 已签字接受 RPC 粒度）。键全带 db,table 以便 `flush(db,table)` 按表清（tableCache 精确 invalidateKey，3 谓词缓存 invalidateIf(matches)）、`flushAll()` 清全部（为 C-e REFRESH TABLE/CATALOG 接线预备）。默认镜像 legacy fe-core Config（TTL 86400s；容量 table/names/partition/stats=10000/10000/100000/10000），经 `CacheSpec.fromProperties(props,"hive",,default)` 读 `meta.cache.hive..*`，连接器只读自己 props（不碰 fe-core Config）。**用户签字（2026-07-10）：第 4 类列统计一起缓存**（对齐 Q1 四方法枚举；"非目标"里"先不做"被 Q1 覆盖）。**pom**：`-hms` 加 `fe-connector-cache`(compile，让装饰器编译)+Caffeine 2.9.3(test scope，单测构造 MetaCacheEntry 需)；补 C-a"依赖只加在 -hive"的遗漏（类在 -hms）；Caffeine 仍 `provided`/非传递、由 -hive 插件打包。**残留（记账）**：`-hms` 亦被 `-hudi` 依赖→`fe-connector-cache` jar 传递进 hudi 闭包但 Caffeine provided 不进→hudi 若日后自包 client 复用须自带 Caffeine 2.9.3；C-b 里 hudi 不构造它→无 NoClassDefFound、仅休眠 jar。**休眠证**：grep 无任何处引用 `CachingHmsClient`、`SPI_READY_TYPES` 无 hms→永不实例化；`HiveConnector.createClient():379` 仍返裸 `ThriftHmsClient`（接线归 C-c）。**测**：`CachingHmsClientTest` 12 测（命中+各维含 **db 维**/props 三档开关/flush 按表 4 缓存隔离/flushAll 4 缓存全清/透传委派/异常传播不缓存/空列统计亦缓存/null delegate 拒）；hms 68 测 + checkstyle 0 + import gate 净；-hive/-hudi 回归 BUILD SUCCESS。**4 维 clean-room 对抗复审（`wf_39f6458c-e37`：委派完整/缓存正确/休眠打包/测试强度 + 逐条 refute）：生产码 0 缺陷**；5 findings→4 CONFIRMED（全 minor 测覆盖缺口，已补：db 维/flushAll 全缓存/flush 谓词缓存按表作用域/测类 javadoc 陈旧）+1 refuted（可变 list 容器按引返回=既有全库缓存惯例、休眠无变更方；顺带补主类 javadoc 说明容器共享须只读）。**✅ C-c DONE（`7b05df6e55e`，休眠）= `CachingHmsClient` 接线进连接器 client**：`HiveConnector.createClient` 把裸 `ThriftHmsClient` 经新 package-private `wrapWithCache(raw)` seam 包成 `CachingHmsClient`（`new CachingHmsClient(raw, properties)`）；**透明装饰器**故此一处 wrap 令 `HiveConnectorMetadata` 全部 `hmsClient.*` 读自动走缓存——`getTable`(getTableHandle/getColumnHandles/getTableStatistics/view)、`listPartitionNames`、`getPartitions`（扫描 + **两条新鲜度探针**）、`getTableColumnStatistics`——**§2.6 达成**：`getTableFreshness`/`getPartitionFreshnessMillis` 都读 `hmsClient.getPartitions` 故周期性 SQL 字典/MV 刷新 poll 由缓存服务、不再每 tick 打 metastore。装饰器按 catalog 自身 `meta.cache.hive.*` props 配置（disable/ttl<=0→透传），缓存挂 `HmsClient`（连接器 init/`ADD`/`MODIFY CATALOG` 重建）、绝不入 handle/GSON edit log。`wrapWithCache` 镜像既有 `newMetadata(HmsClient)` seam 使单测能用 fake client 驱 wrap 而不触发 `createClient` 建真 `ThriftHmsClient`。**休眠**：hms 不在 `SPI_READY_TYPES`→无 live 目录建 `HiveConnector`，wrap 仅单测跑；只改 `HiveConnector.java`→paimon/iceberg/jdbc/hudi 字节不变。**测**：新 `HiveConnectorClientCacheTest`(4)——wrapWithCache 返 CachingHmsClient / 重复 getTableFreshness+getPartitionFreshnessMillis 由缓存服务（1 RPC 非 2）/ 关 `meta.cache.hive.partition` 使 getPartitions 每 poll 重载而仍开的 partition-name 缓存不重载（连接器级 per-entry props 穿透）；hive **244 测全绿**（240→244）+ checkstyle 0 + import gate 净（HiveVersionUtil vendored 跳过=已知误报）。**下一步 = C-d**（文件列表缓存：`HiveScanPlanProvider` 目录列举 + `estimateDataSizeByListingFiles` rowcount 回填走连接器自持文件缓存，TCCL 钉扫描线程 loader；最高复杂度、可分离步）。 C-d…C-f 每步独立休眠 commit + 单测 + checkstyle/import gate，全部落地后统一对抗复审再翻闸。次子系统 = 事件 Model B。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 diff --git a/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md b/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md index a6b7bb600172a5..f35cf13e79289c 100644 --- a/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md +++ b/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md @@ -324,9 +324,11 @@ After all land, run a unified adversarial re-review (clean-room, per project pra --- ## 9. TODO (fill after Q1/Q2 sign-off) -- [ ] C-a pom dep + Caffeine 2.9.3 (+ verify single-copy in zip) -- [ ] C-b metastore-metadata cache (per Q1) from `meta.cache.hive.*`, unconsulted + tests -- [ ] C-c wire metadata reads + freshness probes through it + tests +- [x] C-a pom dep + Caffeine 2.9.3 (+ verify single-copy in zip) — `f742651990d` +- [x] C-b metastore-metadata cache (per Q1) from `meta.cache.hive.*`, unconsulted + tests — `4fe55d88fab` +- [x] C-c wire metadata reads + freshness probes through it + tests — `7b05df6e55e` (wrapWithCache seam in + HiveConnector.createClient; transparent decorator => all HiveConnectorMetadata reads + both freshness + probes cache-backed by the single wrap; HiveConnectorClientCacheTest 4; hive 244 green) - [ ] C-d file-listing cache (per Q2) + TCCL + row-count backing + tests - [ ] C-e `invalidateTable/invalidateAll` overrides + tests - [ ] C-f §2.6 fe-core last-modified branch (byte-neutral guard) + tests From c2167b0641e6fc8b46f6a65b5284bac27d681718 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:28:38 +0800 Subject: [PATCH 153/334] [feat](catalog) Hive connector cache: connector-owned directory-listing cache (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add HiveFileListingCache -- the second, separate layer of the D2 scan-side cache (the metastore-metadata layer is CachingHmsClient), mirroring Trino's CachingDirectoryLister being kept separate from CachingHiveMetastore. It memoizes the (expensive) FileSystem.listStatus of each partition directory, keyed by (db, table, location), with the value a slim immutable HiveFileStatus {path,length,modificationTime} decoupled from Hadoop's FileStatus. It lives as a final field on the long-lived per-catalog HiveConnector (like readTxnManager) -- the ONLY object that outlives a query, since the scan provider and metadata are rebuilt per call -- and is injected as the SAME instance into both HiveScanPlanProvider (scan hot path) and HiveConnectorMetadata.estimateDataSizeByListingFiles (the periodic ExternalRowCountCache row-count refresh). So a scan warms the estimate and vice versa (§4.4 5th-cache coupling). TCCL: the entry is contextual-only + manual-miss + no auto-refresh, so the loader runs on the CALLER thread -- already pinned to the plugin classloader (the scan thread by PluginDrivenScanNode.onPluginClassLoader; the stats thread by estimateDataSizeByListingFiles itself). A background refresh thread would not inherit that pin and Hadoop's FileSystem reflection would fail. Correctness/parity: the real lister filters directories and _/.-prefixed hidden files (byte-parity with the pre-cache inline filters); a zero-length file is kept (the splitter skips it, the estimate adds 0). An I/O failure throws DorisConnectorException and is NEVER cached; the scan path catches it and skips the partition with a warning (pre-cache tolerance of a missing/unreadable partition dir), which also folds the former hard FileSystem.get failure into the same skip-with-warn path (documented; a broken storage config still fails loud via the estimate and the docker e2e). The estimate path lets it propagate and degrades to -1 (legacy all-or-nothing). Defaults mirror legacy Config: TTL 86400s, capacity max_external_file_cache_num (10000); read from meta.cache.hive.file.* (connector never touches fe-core Config). The ACID path is uncached (legacy parity), unchanged. splitFile is refactored to take primitives (path,size,modTime) so both the cached non-ACID path (HiveFileStatus) and the ACID path (FileStatus) share it. HiveConnectorMetadata gains a production 7-arg constructor that injects the shared cache; the existing 3-arg/6-arg convenience constructors build a private default (harmless -- direct-construction tests inject file sizes and never list), so no test-churn across ~20 metadata call-sites. Dormant: "hms" is not in SPI_READY_TYPES, so no live catalog builds a HiveConnector; byte-neutral for every other connector. invalidateTable/ invalidateAll on the cache arm REFRESH TABLE/CATALOG in the next step. Tests: new HiveFileListingCacheTest (9) -- per-location hit/miss, (db,table, location) key scoping, invalidateTable per-table scoping, invalidateAll, failure-propagated-not-cached, meta.cache.hive.file.enable=false pass-through, real-FS dir/hidden filter (@TempDir), and two integration tests proving the scan provider AND the row-count estimate are both served from the SAME cache (repeated scan/refresh lists each directory once). Full hive suite 253 green + checkstyle 0 + connector import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 12 +- .../connector/hive/HiveConnectorMetadata.java | 56 +-- .../connector/hive/HiveFileListingCache.java | 198 +++++++++++ .../doris/connector/hive/HiveFileStatus.java | 53 +++ .../connector/hive/HiveScanPlanProvider.java | 67 ++-- .../hive/HiveFileListingCacheTest.java | 319 ++++++++++++++++++ .../hive/HiveReadTransactionTest.java | 3 +- 7 files changed, 637 insertions(+), 71 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileStatus.java create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 66b4d8c062c8e7..953bc67889e468 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -84,6 +84,12 @@ public class HiveConnector implements Connector { // Plugin-owned and dormant until the read cutover wires its query-finish commit (see the manager). private final HiveReadTransactionManager readTxnManager = new HiveReadTransactionManager(); + // Connector-owned directory-listing cache, shared by the (per-call) scan provider and metadata. One per + // connector (like readTxnManager above) — the scan provider / metadata are rebuilt per query, so the cache + // must live on the long-lived connector to survive across scans. Built from catalog props; dormant until hms + // enters SPI_READY_TYPES. Its metastore-metadata sibling is the CachingHmsClient wrapping the HmsClient. + private final HiveFileListingCache fileListingCache; + // Embedded iceberg SIBLING connector: a flipped hms gateway delegates its iceberg-on-HMS tables to it. Built // once per gateway connector (lazily) in the iceberg plugin's OWN child-first classloader via // context.createSiblingConnector — never co-packaged into the hive zip (a second AWS SDK would poison S3 @@ -103,6 +109,7 @@ public class HiveConnector implements Connector { public HiveConnector(Map properties, ConnectorContext context) { this.properties = Collections.unmodifiableMap(properties); this.context = context; + this.fileListingCache = new HiveFileListingCache(this.properties); } @Override @@ -126,7 +133,8 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { */ HiveConnectorMetadata newMetadata(HmsClient client) { return new HiveConnectorMetadata(client, properties, context, - this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwner); + this::getOrCreateIcebergSibling, this::getOrCreateHudiSibling, this::resolveSiblingOwner, + fileListingCache); } /** @@ -158,7 +166,7 @@ private Connector resolveSiblingOwner(ConnectorTableHandle handle) { @Override public ConnectorScanPlanProvider getScanPlanProvider() { - return new HiveScanPlanProvider(getOrCreateClient(), properties, readTxnManager); + return new HiveScanPlanProvider(getOrCreateClient(), properties, readTxnManager, fileListingCache); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index b1dc73526b8b21..8be8336accf699 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -70,13 +70,9 @@ import com.google.gson.Gson; import com.google.gson.JsonObject; import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; -import java.io.IOException; import java.nio.charset.StandardCharsets; import java.util.ArrayList; import java.util.Base64; @@ -211,6 +207,11 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // over the ALREADY-BUILT iceberg / hudi siblings). Used ONLY through the parent-first Connector / // ConnectorMetadata interfaces — the owning sibling's concrete types are never cast here (cross-loader CCE). private final Function siblingOwnerResolver; + // Connector-owned directory-listing cache, shared with the scan provider so estimateDataSizeByListingFiles + // (the periodic ExternalRowCountCache refresh source) reuses listings a scan warmed and vice versa. Injected + // by HiveConnector.newMetadata as the SAME instance; the convenience constructors below build a private + // default (harmless for the direct-construction tests, which inject their file sizes and never list). + private final HiveFileListingCache fileListingCache; public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context) { this(hmsClient, properties, context, NO_ICEBERG_SIBLING, NO_HUDI_SIBLING, NO_SIBLING_OWNER); @@ -220,6 +221,15 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties Supplier icebergSiblingSupplier, Supplier hudiSiblingSupplier, Function siblingOwnerResolver) { + this(hmsClient, properties, context, icebergSiblingSupplier, hudiSiblingSupplier, siblingOwnerResolver, + new HiveFileListingCache(properties)); + } + + public HiveConnectorMetadata(HmsClient hmsClient, Map properties, ConnectorContext context, + Supplier icebergSiblingSupplier, + Supplier hudiSiblingSupplier, + Function siblingOwnerResolver, + HiveFileListingCache fileListingCache) { this.hmsClient = hmsClient; this.properties = properties; this.context = context; @@ -227,6 +237,7 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties this.icebergSiblingSupplier = icebergSiblingSupplier; this.hudiSiblingSupplier = hudiSiblingSupplier; this.siblingOwnerResolver = siblingOwnerResolver; + this.fileListingCache = fileListingCache; } /** @@ -721,7 +732,7 @@ public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTa Thread.currentThread().setContextClassLoader(getClass().getClassLoader()); Configuration conf = buildHadoopConf(); return estimateDataSize(hiveHandle, STATS_PARTITION_SAMPLE_SIZE, - location -> sumFileSizesUnder(location, conf)); + location -> sumCachedFileSizes(hiveHandle, location, conf)); } finally { Thread.currentThread().setContextClassLoader(previous); } @@ -806,31 +817,20 @@ private List resolvePartitionLocations(HiveTableHandle handle) { } /** - * Sums the sizes of the data files directly under {@code location} (non-recursive, skipping directories - * and {@code _}/{@code .}-prefixed hidden files — the same filter as {@link HiveScanPlanProvider}). A - * listing failure is surfaced as a {@link RuntimeException} so {@link #estimateDataSize} degrades the - * whole estimate to -1 (legacy's file-list estimate was all-or-nothing best-effort). + * Sums the sizes of the data files directly under {@code location}, served from the connector's shared + * {@link HiveFileListingCache} (which does the non-recursive {@code listStatus} and filters directories and + * {@code _}/{@code .}-prefixed hidden files — the same filter, and the same listing, the scan path uses). A + * listing failure propagates as a {@link DorisConnectorException} so {@link #estimateDataSize} degrades the + * whole estimate to -1 (legacy's file-list estimate was all-or-nothing best-effort). Routing through the + * cache keeps the periodic row-count refresh from re-listing directories a scan already cached. */ - private static long sumFileSizesUnder(String location, Configuration conf) { - try { - Path path = new Path(location); - FileSystem fs = FileSystem.get(path.toUri(), conf); - FileStatus[] statuses = fs.listStatus(path); - long sum = 0; - for (FileStatus status : statuses) { - if (status.isDirectory()) { - continue; - } - String fileName = status.getPath().getName(); - if (fileName.startsWith("_") || fileName.startsWith(".")) { - continue; - } - sum += status.getLen(); - } - return sum; - } catch (IOException e) { - throw new DorisConnectorException("Failed to list files under " + location, e); + private long sumCachedFileSizes(HiveTableHandle handle, String location, Configuration conf) { + long sum = 0; + for (HiveFileStatus file : fileListingCache.listDataFiles( + handle.getDbName(), handle.getTableName(), location, conf)) { + sum += file.getLength(); } + return sum; } /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java new file mode 100644 index 00000000000000..58d579df2bbc75 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -0,0 +1,198 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.FileStatus; +import org.apache.hadoop.fs.FileSystem; +import org.apache.hadoop.fs.Path; + +import java.io.IOException; +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ForkJoinPool; + +/** + * The hive connector's own directory-listing cache — the second, separate cache layer of the D2 scan-side cache + * (the metastore-metadata layer is {@link org.apache.doris.connector.hms.CachingHmsClient}). It memoizes the + * (expensive) {@code FileSystem.listStatus} of each partition directory, keyed by {@code (db, table, location)}. + * + *

    Why this exists. Legacy fe-core kept directory listings in the engine-side + * {@code HiveExternalMetaCache}'s {@code file} entry; once a hive catalog becomes plugin-driven that cache stops + * routing to it, so without this connector-owned cache every scan (and every periodic row-count refresh) would + * re-list every partition directory from the filesystem. Trino keeps the equivalent {@code CachingDirectoryLister} + * as a layer separate from its metastore cache — D2 mirrors that split.

    + * + *

    Who reads it. {@link HiveScanPlanProvider} (the scan hot path) and + * {@link HiveConnectorMetadata#estimateDataSizeByListingFiles} (the row-count estimate) — both go through the SAME + * instance, held as a {@code final} field on the per-catalog {@link HiveConnector} (the only object that outlives a + * single query; the scan provider / metadata are rebuilt per call). A scan therefore warms the estimate and vice + * versa.

    + * + *

    TCCL. The entry is contextual-only + manual-miss + no auto-refresh, so the loader runs synchronously on + * the CALLING thread — which is already pinned to the plugin classloader (the scan thread by + * {@code PluginDrivenScanNode.onPluginClassLoader}; the stats thread by {@code estimateDataSizeByListingFiles} + * itself). A background refresh thread would NOT inherit that pin and Hadoop's {@code FileSystem} reflection would + * fail to resolve its impl.

    + * + *

    Failures are not cached. The loader throws {@link DorisConnectorException} on any I/O error (never + * caching a transient failure, matching {@code MetaCacheEntry}'s null-is-a-miss / exception-propagates contract). + * The scan path catches it and skips the partition with a warning (the pre-cache tolerance of a missing/unreadable + * partition directory); the estimate path lets it propagate and degrades to {@code -1}.

    + * + *

    Dormant. {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog builds a + * {@link HiveConnector}, so this cache is never instantiated for a live catalog until the flip. Byte-neutral for + * every other connector.

    + */ +public class HiveFileListingCache { + + /** Engine token for the {@code meta.cache...*} property namespace. */ + static final String ENGINE = "hive"; + /** {@code meta.cache.hive.file.*} — cached directory listings. */ + static final String ENTRY_FILE = "file"; + + // Legacy fe-core Config values, mirrored locally (the connector never touches fe-core Config): + // TTL = Config.external_cache_expire_time_seconds_after_access (86400s = 24h) + // capacity = Config.max_external_file_cache_num (10000) + static final long DEFAULT_TTL_SECOND = 86400L; + static final long DEFAULT_FILE_CAPACITY = 10000L; + + /** + * The raw directory lister: the real Hadoop filesystem in production ({@link #listFromFileSystem}), a fake in + * unit tests. Injected so the cache's hit/miss/invalidation behaviour is testable without a live filesystem + * (mirrors {@code HiveConnectorMetadata.estimateDataSize} injecting its {@code ToLongFunction} size source). + */ + @FunctionalInterface + interface DirectoryLister { + List list(String location, Configuration conf); + } + + private final MetaCacheEntry> cache; + private final DirectoryLister lister; + + public HiveFileListingCache(Map properties) { + this(properties, HiveFileListingCache::listFromFileSystem); + } + + HiveFileListingCache(Map properties, DirectoryLister lister) { + Map props = properties == null ? Collections.emptyMap() : properties; + CacheSpec spec = CacheSpec.fromProperties(props, ENGINE, ENTRY_FILE, + CacheSpec.of(true, DEFAULT_TTL_SECOND, DEFAULT_FILE_CAPACITY)); + // Contextual-only + manual-miss so the slow listStatus runs on the caller (TCCL-pinned) thread outside + // Caffeine's sync compute lock, deduplicated by a striped lock — mirrors CachingHmsClient's entries. + this.cache = new MetaCacheEntry<>("hive.file", null, spec, ForkJoinPool.commonPool(), false, true, 0L, true); + this.lister = Objects.requireNonNull(lister, "lister can not be null"); + } + + /** + * Lists the data files directly under {@code location} (non-recursive; directories and {@code _}/{@code .} + * -prefixed hidden files removed), served from the cache. Keyed by {@code (db, table, location)} so + * {@link #invalidateTable} can drop exactly one table's entries. The loader runs on the calling thread; an I/O + * failure throws {@link DorisConnectorException} (and is NOT cached). The returned list is shared by reference + * (immutable elements) — callers must treat it as read-only, the codebase-wide metadata-cache convention. + */ + public List listDataFiles(String dbName, String tableName, String location, Configuration conf) { + return cache.get(new FileListingKey(dbName, tableName, location), + key -> lister.list(key.location, conf)); + } + + /** Drops every cached listing for one table. Backs {@code REFRESH TABLE}. */ + public void invalidateTable(String dbName, String tableName) { + cache.invalidateIf(key -> key.matches(dbName, tableName)); + } + + /** Drops the whole file-listing cache. Backs {@code REFRESH CATALOG}. */ + public void invalidateAll() { + cache.invalidateAll(); + } + + /** + * The production {@link DirectoryLister}: a real, non-recursive {@code FileSystem.listStatus}, filtering out + * directories and {@code _}/{@code .}-prefixed hidden files (byte-parity with the pre-cache filters in + * {@code HiveScanPlanProvider.listAndSplitFiles} and {@code HiveConnectorMetadata.sumFileSizesUnder}). A zero + * -length data file is kept (the scan splitter skips it; the size estimate adds 0) so both consumers keep + * their exact prior behaviour. Any {@link IOException} (unresolvable filesystem or unreadable directory) is + * wrapped in a {@link DorisConnectorException} so the entry is never cached and the caller decides how to + * degrade. + */ + static List listFromFileSystem(String location, Configuration conf) { + try { + Path path = new Path(location); + FileSystem fs = FileSystem.get(path.toUri(), conf); + FileStatus[] statuses = fs.listStatus(path); + List files = new ArrayList<>(statuses.length); + for (FileStatus status : statuses) { + if (status.isDirectory()) { + continue; + } + String fileName = status.getPath().getName(); + if (fileName.startsWith("_") || fileName.startsWith(".")) { + continue; + } + files.add(new HiveFileStatus(status.getPath().toString(), status.getLen(), + status.getModificationTime())); + } + return files; + } catch (IOException e) { + throw new DorisConnectorException("Failed to list files under " + location, e); + } + } + + /** Cache key: (db, table, location). db+table let {@link #invalidateTable} select one table's entries. */ + static final class FileListingKey { + private final String dbName; + private final String tableName; + private final String location; + + FileListingKey(String dbName, String tableName, String location) { + this.dbName = dbName; + this.tableName = tableName; + this.location = location; + } + + boolean matches(String db, String table) { + return Objects.equals(dbName, db) && Objects.equals(tableName, table); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof FileListingKey)) { + return false; + } + FileListingKey that = (FileListingKey) o; + return Objects.equals(dbName, that.dbName) + && Objects.equals(tableName, that.tableName) + && Objects.equals(location, that.location); + } + + @Override + public int hashCode() { + return Objects.hash(dbName, tableName, location); + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileStatus.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileStatus.java new file mode 100644 index 00000000000000..86b82ef895d2eb --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileStatus.java @@ -0,0 +1,53 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +/** + * An immutable, slim view of one data file in a partition directory: exactly the three fields the scan planner + * ({@link HiveScanPlanProvider#splitFile}) and the row-count estimate + * ({@link HiveConnectorMetadata#estimateDataSizeByListingFiles}) read from a Hadoop {@code FileStatus}. It is the + * value element cached by {@link HiveFileListingCache}, deliberately decoupled from the (heavier, Hadoop-internal) + * {@code FileStatus} so the cache holds only immutable, small values (the codebase-wide metadata-cache convention). + */ +public final class HiveFileStatus { + + private final String path; + private final long length; + private final long modificationTime; + + public HiveFileStatus(String path, long length, long modificationTime) { + this.path = path; + this.length = length; + this.modificationTime = modificationTime; + } + + /** The file's full path (e.g. {@code s3://wh/db/t/dt=1/000000_0}). */ + public String getPath() { + return path; + } + + /** The file length in bytes. */ + public long getLength() { + return length; + } + + /** The file's last-modification time in millis (BE uses it to detect a changed split). */ + public long getModificationTime() { + return modificationTime; + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index c48c4fb774daea..d854f2a41f1a15 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -84,12 +84,17 @@ public class HiveScanPlanProvider implements ConnectorScanPlanProvider { private final HmsClient hmsClient; private final Map catalogProperties; private final HiveReadTransactionManager readTxnManager; + // Connector-owned directory-listing cache (the SAME instance HiveConnector shares with HiveConnectorMetadata), + // so a repeated scan of the same partition directory is served from the cache instead of re-listing. Only the + // plain (non-ACID) path uses it; the ACID path lists via HiveAcidUtil and is uncached (legacy parity). + private final HiveFileListingCache fileListingCache; public HiveScanPlanProvider(HmsClient hmsClient, Map catalogProperties, - HiveReadTransactionManager readTxnManager) { + HiveReadTransactionManager readTxnManager, HiveFileListingCache fileListingCache) { this.hmsClient = hmsClient; this.catalogProperties = catalogProperties; this.readTxnManager = readTxnManager; + this.fileListingCache = fileListingCache; } @Override @@ -130,13 +135,8 @@ public List planScan( for (PartitionScanInfo partition : partitions) { HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; - try { - listAndSplitFiles(hadoopConf, partition, partFormat, - splittable, targetSplitSize, ranges); - } catch (IOException e) { - throw new DorisConnectorException( - "Failed to list files for partition: " + partition.location, e); - } + listAndSplitFiles(dbName, tableName, partition, partFormat, + splittable, targetSplitSize, hadoopConf, ranges); } } @@ -198,8 +198,9 @@ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, List encodedDeltas = isFullAcid ? encodeDeleteDeltas(state.getDeleteDeltas()) : null; for (FileStatus dataFile : state.getDataFiles()) { - splitFile(dataFile, partition, partFormat, splittable, targetSplitSize, - acidLocation, encodedDeltas, ranges); + splitFile(dataFile.getPath().toString(), dataFile.getLen(), + dataFile.getModificationTime(), partition, partFormat, splittable, + targetSplitSize, acidLocation, encodedDeltas, ranges); } } catch (IOException e) { throw new DorisConnectorException( @@ -348,33 +349,27 @@ private List convertPartitions( } /** - * Lists files in a partition directory and splits them into scan ranges. + * Lists the data files of a partition directory (through the connector's shared {@link HiveFileListingCache}, + * which filters directories and {@code _}/{@code .}-prefixed hidden files) and splits them into scan ranges. + * A listing failure is tolerated — the partition is skipped with a warning, the same resilience the pre-cache + * code gave a missing/unreadable partition directory (and it consolidates the earlier {@code FileSystem.get} + * failure into the same skip path; a bad storage config surfaces via the row-count estimate's fail-loud and + * the docker e2e). Failures are never cached (the cache loader throws). */ - private void listAndSplitFiles(Configuration conf, + private void listAndSplitFiles(String dbName, String tableName, PartitionScanInfo partition, HiveFileFormat fileFormat, - boolean splittable, long targetSplitSize, - List ranges) throws IOException { - Path partPath = new Path(partition.location); - FileSystem fs = FileSystem.get(partPath.toUri(), conf); - FileStatus[] statuses; + boolean splittable, long targetSplitSize, Configuration conf, + List ranges) { + List files; try { - statuses = fs.listStatus(partPath); - } catch (IOException e) { + files = fileListingCache.listDataFiles(dbName, tableName, partition.location, conf); + } catch (DorisConnectorException e) { LOG.warn("Cannot list files in partition: {}", partition.location, e); return; } - - for (FileStatus status : statuses) { - if (status.isDirectory()) { - // Skip directories (could be _temporary, etc.) - continue; - } - String fileName = status.getPath().getName(); - if (shouldSkipFile(fileName)) { - continue; - } - splitFile(status, partition, fileFormat, splittable, - targetSplitSize, null, null, ranges); + for (HiveFileStatus file : files) { + splitFile(file.getPath(), file.getLength(), file.getModificationTime(), + partition, fileFormat, splittable, targetSplitSize, null, null, ranges); } } @@ -384,14 +379,10 @@ private void listAndSplitFiles(Configuration conf, *

    When {@code acidPartitionLocation} is non-null the ranges carry ACID delete-delta info * (marking them {@code transactional_hive}); otherwise they are plain Hive ranges.

    */ - private void splitFile(FileStatus fileStatus, PartitionScanInfo partition, + private void splitFile(String filePath, long fileSize, long modTime, PartitionScanInfo partition, HiveFileFormat fileFormat, boolean splittable, long targetSplitSize, String acidPartitionLocation, List acidDeltas, List ranges) { - long fileSize = fileStatus.getLen(); - String filePath = fileStatus.getPath().toString(); - long modTime = fileStatus.getModificationTime(); - if (fileSize == 0) { return; } @@ -432,10 +423,6 @@ private static HiveScanRange.Builder newRangeBuilder(String filePath, long start return builder; } - private boolean shouldSkipFile(String fileName) { - return fileName.startsWith("_") || fileName.startsWith("."); - } - private long getTargetSplitSize(ConnectorSession session) { String splitSizeStr = session.getProperty( "file_split_size", String.class); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java new file mode 100644 index 00000000000000..513a9e07d90bce --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java @@ -0,0 +1,319 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.scan.ConnectorScanRange; +import org.apache.doris.connector.hms.HmsPartitionInfo; + +import org.apache.hadoop.conf.Configuration; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Files; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link HiveFileListingCache}: the connector-owned directory-listing cache (D2's second layer, separate + * from the CachingHmsClient metastore layer — Trino keeps CachingDirectoryLister separate too). + * + *

    WHY (Rule 9): after the hms cutover the fe-core engine-side {@code HiveExternalMetaCache} stops routing to a + * hive catalog, so without this connector-owned cache every scan (and every periodic row-count refresh) would + * re-list every partition directory from the filesystem. These tests pin the behaviours that make the re-homed + * cache correct: (1) a directory listing is cached keyed by {@code (db, table, location)} so it loads once; the db + * / table / location dimensions never collide; (2) {@code invalidateTable} drops exactly one table's entries and + * {@code invalidateAll} clears all (arming REFRESH); (3) an I/O failure propagates and is NOT cached; (4) the + * {@code meta.cache.hive.file.*} knobs turn it off; (5) the real filesystem lister filters directories and + * {@code _}/{@code .}-hidden files. Two integration tests prove BOTH consumers — the scan provider and the + * row-count estimate — are served from the SAME cache, so a repeated scan / refresh does not re-list.

    + * + *

    Dormant: {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog builds a connector; this + * exercises the cache directly.

    + */ +public class HiveFileListingCacheTest { + + private static final Configuration CONF = new Configuration(); + + private static Map props(String... kv) { + Map m = new HashMap<>(); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + // ==================== caching: hit / miss keyed by (db, table, location) ==================== + + @Test + public void listingIsCachedPerLocation() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + + List a = cache.listDataFiles("db", "t", "loc1", CONF); + List b = cache.listDataFiles("db", "t", "loc1", CONF); + // WHY: a hit must serve the cached listing without re-listing the filesystem. + Assertions.assertSame(a, b); + Assertions.assertEquals(1, lister.totalCalls); + + // WHY: a different directory is a different key — must re-list. + cache.listDataFiles("db", "t", "loc2", CONF); + Assertions.assertEquals(2, lister.totalCalls); + } + + @Test + public void keyIsScopedByDbTableAndLocation() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + + // Same location string, different db / table. WHY: (db, table) must be part of the key so invalidateTable + // can scope one table — and so two tables that happen to share a path never serve each other's listing. + cache.listDataFiles("db1", "t", "loc", CONF); + cache.listDataFiles("db2", "t", "loc", CONF); + cache.listDataFiles("db1", "t2", "loc", CONF); + Assertions.assertEquals(3, lister.totalCalls); + } + + // ==================== invalidation (arms REFRESH TABLE / REFRESH CATALOG) ==================== + + @Test + public void invalidateTableDropsOnlyThatTablesEntries() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + + cache.listDataFiles("db", "t1", "locA", CONF); + cache.listDataFiles("db", "t2", "locB", CONF); + Assertions.assertEquals(2, lister.totalCalls); + + cache.invalidateTable("db", "t1"); + + // WHY: t1 must re-list after its invalidation... + cache.listDataFiles("db", "t1", "locA", CONF); + Assertions.assertEquals(2, (int) lister.callsPerLocation.get("locA")); + // ...while t2's entry (a different table) must survive — invalidateTable is scoped by (db, table). + cache.listDataFiles("db", "t2", "locB", CONF); + Assertions.assertEquals(1, (int) lister.callsPerLocation.get("locB")); + } + + @Test + public void invalidateAllDropsEverything() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + + cache.listDataFiles("db", "t1", "locA", CONF); + cache.listDataFiles("db", "t2", "locB", CONF); + Assertions.assertEquals(2, lister.totalCalls); + + cache.invalidateAll(); + + // WHY: every entry must reload after invalidateAll (REFRESH CATALOG). + cache.listDataFiles("db", "t1", "locA", CONF); + cache.listDataFiles("db", "t2", "locB", CONF); + Assertions.assertEquals(4, lister.totalCalls); + } + + // ==================== failures are propagated, never cached ==================== + + @Test + public void listingFailureIsPropagatedAndNotCached() { + CountingLister lister = new CountingLister(); + lister.error = new DorisConnectorException("boom"); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> cache.listDataFiles("db", "t", "loc", CONF)); + Assertions.assertEquals("boom", e.getMessage()); + Assertions.assertEquals(1, lister.totalCalls); + + // WHY: a transient listing failure must NOT be cached — after recovery the next call re-lists and succeeds + // (otherwise a momentary storage blip would poison the listing for the whole TTL). + lister.error = null; + List ok = cache.listDataFiles("db", "t", "loc", CONF); + Assertions.assertEquals(1, ok.size()); + Assertions.assertEquals(2, lister.totalCalls); + } + + // ==================== per-entry property knob ==================== + + @Test + public void disablingViaPropsBypassesTheCache() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache( + props("meta.cache.hive.file.enable", "false"), lister); + + cache.listDataFiles("db", "t", "loc", CONF); + cache.listDataFiles("db", "t", "loc", CONF); + // WHY: meta.cache.hive.file.enable=false must bypass caching entirely — every call re-lists. + Assertions.assertEquals(2, lister.totalCalls); + } + + // ==================== the real filesystem lister: filters dirs + hidden files ==================== + + @Test + public void listFromFileSystemFiltersDirectoriesAndHiddenFiles(@TempDir java.nio.file.Path dir) throws Exception { + Files.write(dir.resolve("data1"), new byte[100]); + Files.write(dir.resolve("data2"), new byte[200]); + Files.write(dir.resolve("_SUCCESS"), new byte[3]); // hive success marker — must be skipped + Files.write(dir.resolve(".hidden"), new byte[3]); // dot-file — must be skipped + Files.createDirectory(dir.resolve("subdir")); // directory — must be skipped + + List files = HiveFileListingCache.listFromFileSystem(dir.toUri().toString(), CONF); + + // WHY: only the two real data files survive; the total size is exactly their sum. This pins the same + // dir/_-/.-filter the pre-cache scan and estimate paths applied inline. + Assertions.assertEquals(2, files.size(), "only the two data files must survive the dir/hidden filter"); + long totalSize = files.stream().mapToLong(HiveFileStatus::getLength).sum(); + Assertions.assertEquals(300L, totalSize); + for (HiveFileStatus f : files) { + String name = f.getPath().substring(f.getPath().lastIndexOf('/') + 1); + Assertions.assertTrue(name.equals("data1") || name.equals("data2"), "unexpected file: " + name); + } + } + + // ==================== integration: the scan provider is cache-backed ==================== + + @Test + public void scanProviderServesRepeatedScansFromTheCache() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + // hmsClient is null: with pruned partitions on the handle the scan never calls the metastore. + HiveScanPlanProvider provider = new HiveScanPlanProvider( + null, Collections.emptyMap(), new HiveReadTransactionManager(), cache); + + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat") + .serializationLib("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe") + .partitionKeyNames(Collections.singletonList("dt")) + .prunedPartitions(Arrays.asList( + new HmsPartitionInfo(Collections.singletonList("1"), "loc/dt=1", null, null, null, null), + new HmsPartitionInfo(Collections.singletonList("2"), "loc/dt=2", null, null, null, null))) + .build(); + + List first = provider.planScan( + new FakeSession(), handle, Collections.emptyList(), Optional.empty()); + List second = provider.planScan( + new FakeSession(), handle, Collections.emptyList(), Optional.empty()); + + // WHY: each partition directory is listed exactly once across the two scans — the second scan is served + // from the cache. Without the cache the file listing would be an uncached RPC/FS call on every scan. + Assertions.assertEquals(1, (int) lister.callsPerLocation.get("loc/dt=1")); + Assertions.assertEquals(1, (int) lister.callsPerLocation.get("loc/dt=2")); + // One 10-byte file per partition -> one range each; both scans plan the same ranges. + Assertions.assertEquals(2, first.size()); + Assertions.assertEquals(2, second.size()); + } + + // ==================== integration: the row-count estimate is cache-backed (7-arg wiring) ==================== + + @Test + public void estimateDataSizeIsServedFromTheCache() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); + // The production 7-arg constructor injects the connector's shared cache; the sibling seams are unused for + // a plain-hive handle, so dummy suppliers suffice. + HiveConnectorMetadata md = new HiveConnectorMetadata( + null, Collections.emptyMap(), new FakeConnectorContext(), + () -> null, () -> null, h -> null, cache); + + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .location("file:///wh/t") + .build(); + + long first = md.estimateDataSizeByListingFiles(null, handle); + long second = md.estimateDataSizeByListingFiles(null, handle); + + // WHY: the estimate returns the one 10-byte file's size, and the second (periodic ExternalRowCountCache + // refresh) call is served from the SAME cache — one listing, not two. This proves the row-count refresh + // reuses a listing a scan warmed (and vice versa), the §4.4 5th-cache coupling. + Assertions.assertEquals(10L, first); + Assertions.assertEquals(10L, second); + Assertions.assertEquals(1, lister.totalCalls); + } + + /** + * A {@link HiveFileListingCache.DirectoryLister} double: counts calls (total + per location) and returns a + * fresh single-file listing per call, so reference identity distinguishes a cache hit from a reload. Throws + * {@link #error} when set, to exercise the failure-not-cached path. + */ + private static final class CountingLister implements HiveFileListingCache.DirectoryLister { + final Map callsPerLocation = new HashMap<>(); + int totalCalls; + RuntimeException error; + + @Override + public List list(String location, Configuration conf) { + totalCalls++; + callsPerLocation.merge(location, 1, Integer::sum); + if (error != null) { + throw error; + } + return new ArrayList<>(Collections.singletonList(new HiveFileStatus(location + "/000000_0", 10L, 1L))); + } + } + + /** Minimal {@link ConnectorSession} for planScan (no split-size override, empty session properties). */ + private static final class FakeSession implements ConnectorSession { + @Override + public String getQueryId() { + return "q"; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return 0L; + } + + @Override + public String getCatalogName() { + return "hive_catalog"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java index 12a7c6aaba04ca..1650c282eb1d23 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java @@ -106,7 +106,8 @@ public void testScanProviderReleaseReadTransactionCommitsViaSharedManager() { // register and the provider's release MUST share the same per-connector manager (HiveConnector injects // one instance into every provider), so the release finds and commits the txn register opened. HiveReadTransactionManager mgr = new HiveReadTransactionManager(); - HiveScanPlanProvider provider = new HiveScanPlanProvider(client, new HashMap<>(), mgr); + HiveScanPlanProvider provider = new HiveScanPlanProvider(client, new HashMap<>(), mgr, + new HiveFileListingCache(new HashMap<>())); // A transactional-hive scan opened a read txn (as planAcidScan does via mgr.register), taking the shared // read lock. From 3cecb61e25020641c7e5ed41ba560456f2bfde31 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:36:44 +0800 Subject: [PATCH 154/334] [feat](catalog) Hive connector cache: arm REFRESH TABLE/CATALOG via invalidateTable/invalidateAll (dormant) Override Connector.invalidateTable(db,table) / invalidateAll() on HiveConnector so REFRESH TABLE / REFRESH CATALOG drop the connector-owned D2 caches. Each drops BOTH cache layers: the metastore-metadata cache (CachingHmsClient.flush / flushAll) AND the directory-listing cache (HiveFileListingCache.invalidateTable / invalidateAll) -- because a hive table's schema, partitions AND files are all mutable, so a REFRESH must re-read all of them (unlike iceberg, whose immutable manifests are kept across REFRESH TABLE). invalidateTable is scoped to one table via the (db,table) key both caches carry. The hooks read the hmsClient field WITHOUT building it (getOrCreateClient would force a real ThriftHmsClient just to flush an empty cache): a never-built client means no metastore cache exists, so a REFRESH on a never-scanned catalog is a cheap no-op on the metastore side; the file cache is a final field, always present. fe-core already routes REFRESH TABLE/CATALOG to connector.invalidateTable/invalidateAll for a plugin-driven catalog (no fe-core change). Dormant: "hms" is not in SPI_READY_TYPES. The metastore flush needs an observable CachingHmsClient, which a unit test can only build via the C-c wrapWithCache seam (the field is otherwise set only by getOrCreateClient building a real client), so the public hooks delegate to package-private invalidateTable(client,...) / invalidateAll(client) overloads; add a fileListingCacheForTest() accessor + HiveFileListingCache.size() (mirrors iceberg manifestCacheForTest()/size()). Tests: new HiveConnectorInvalidateTest (3) -- invalidateTable drops BOTH caches for that table ONLY (metastore entry re-fetches, file listing dropped; another table's entries survive in both); invalidateAll clears both fully; the public hooks are no-throw and clear the file cache WITHOUT building a metastore client (null-client path). Full hive suite 256 green + checkstyle 0 + import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 48 +++++ .../connector/hive/HiveFileListingCache.java | 7 + .../hive/HiveConnectorInvalidateTest.java | 194 ++++++++++++++++++ 3 files changed, 249 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 953bc67889e468..411a6a49759293 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -276,6 +276,54 @@ public Set getCapabilities() { ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT); } + /** + * REFRESH TABLE hook: drop this table's connector-owned scan caches. Clears BOTH cache layers for the table — + * the metastore-metadata entries ({@link CachingHmsClient#flush}: table meta, partition names, partition + * objects, column stats) AND the {@link HiveFileListingCache} directory listings — because a hive table's + * schema, partitions AND files are all mutable, so a REFRESH must re-read all of them (unlike iceberg, whose + * manifests are immutable and kept across REFRESH TABLE). fe-core already routes {@code REFRESH TABLE} to + * {@code connector.invalidateTable} for a plugin-driven catalog; dormant until hms enters SPI_READY_TYPES. + */ + @Override + public void invalidateTable(String dbName, String tableName) { + // Read the client field WITHOUT building it (getOrCreateClient would force a real ThriftHmsClient just to + // flush an empty cache): a never-built client means no metastore cache exists to flush. The file cache is + // a final field and always present. + invalidateTable(hmsClient, dbName, tableName); + } + + // Package-private seam: the metastore half needs an observable CachingHmsClient, which a unit test can build + // via wrapWithCache (the hmsClient field is otherwise only set by getOrCreateClient building a real client). + void invalidateTable(HmsClient client, String dbName, String tableName) { + if (client instanceof CachingHmsClient) { + ((CachingHmsClient) client).flush(dbName, tableName); + } + fileListingCache.invalidateTable(dbName, tableName); + } + + /** + * REFRESH CATALOG hook: drop ALL of this catalog's connector-owned scan caches — every metastore-metadata + * entry ({@link CachingHmsClient#flushAll}) and every directory listing. Same no-force-build read of the + * client as {@link #invalidateTable(String, String)}. Dormant until the flip. + */ + @Override + public void invalidateAll() { + invalidateAll(hmsClient); + } + + // Package-private seam (see invalidateTable above). + void invalidateAll(HmsClient client) { + if (client instanceof CachingHmsClient) { + ((CachingHmsClient) client).flushAll(); + } + fileListingCache.invalidateAll(); + } + + /** The connector-owned directory-listing cache, exposed for unit tests (mirrors iceberg manifestCacheForTest). */ + HiveFileListingCache fileListingCacheForTest() { + return fileListingCache; + } + private HmsClient getOrCreateClient() { if (hmsClient == null) { synchronized (this) { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java index 58d579df2bbc75..34d2ff80e0c85d 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -128,6 +128,13 @@ public void invalidateAll() { cache.invalidateAll(); } + /** Current number of cached directory listings — for unit tests only (mirrors iceberg manifestCache.size()). */ + long size() { + long[] count = {0L}; + cache.forEach((key, value) -> count[0]++); + return count[0]; + } + /** * The production {@link DirectoryLister}: a real, non-recursive {@code FileSystem.listStatus}, filtering out * directories and {@code _}/{@code .}-prefixed hidden files (byte-parity with the pre-cache filters in diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java new file mode 100644 index 00000000000000..b02f6d803579a8 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java @@ -0,0 +1,194 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.hms.CachingHmsClient; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.apache.hadoop.conf.Configuration; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.nio.file.Files; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** + * Tests {@link HiveConnector#invalidateTable}/{@link HiveConnector#invalidateAll} — the REFRESH TABLE / REFRESH + * CATALOG hooks that arm the connector-owned D2 caches. + * + *

    WHY (Rule 9): a flipped hive catalog's caches never expire on external change (event sync is off until the + * event Model B step), so {@code REFRESH TABLE} / {@code REFRESH CATALOG} are the user's explicit way to see new + * data. fe-core already routes them to {@code connector.invalidateTable} / {@code invalidateAll}; these tests pin + * that the hive connector then drops BOTH cache layers — the metastore-metadata cache ({@link CachingHmsClient}) + * AND the directory-listing cache ({@link HiveFileListingCache}) — because a hive table's schema, partitions and + * files are all mutable (unlike iceberg's immutable manifests). {@code invalidateTable} is scoped to one table; + * {@code invalidateAll} clears everything; and the public hooks never force-build a metastore client just to + * flush (a REFRESH on a never-scanned catalog must be a cheap no-op on the metastore side).

    + * + *

    Dormant: {@code "hms"} is not in {@code SPI_READY_TYPES}; this drives the hooks directly.

    + */ +public class HiveConnectorInvalidateTest { + + private static final Configuration CONF = new Configuration(); + + private static Map props() { + Map m = new HashMap<>(); + m.put("hive.metastore.uris", "thrift://host:9083"); + return m; + } + + @Test + public void invalidateTableFlushesBothCachesForThatTableOnly(@TempDir java.nio.file.Path dirA, + @TempDir java.nio.file.Path dirB) throws Exception { + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + CachingHmsClient cachingClient = (CachingHmsClient) connector.wrapWithCache(raw); + + // Metastore-metadata cache: populate t1 and t2 (each one RPC, then cached). + cachingClient.getTable("db", "t1"); + cachingClient.getTable("db", "t2"); + Assertions.assertEquals(2, raw.getTableCalls); + + // Directory-listing cache: populate t1 (dirA) and t2 (dirB) from real local dirs. + HiveFileListingCache fileCache = connector.fileListingCacheForTest(); + Files.write(dirA.resolve("f"), new byte[10]); + Files.write(dirB.resolve("f"), new byte[10]); + fileCache.listDataFiles("db", "t1", dirA.toUri().toString(), CONF); + fileCache.listDataFiles("db", "t2", dirB.toUri().toString(), CONF); + Assertions.assertEquals(2, fileCache.size()); + + connector.invalidateTable(cachingClient, "db", "t1"); + + // Metastore cache: t1 re-fetches; t2 (a different table) is still served from the cache. + cachingClient.getTable("db", "t1"); + Assertions.assertEquals(3, raw.getTableCalls, "REFRESH TABLE must drop t1's metastore entry"); + cachingClient.getTable("db", "t2"); + Assertions.assertEquals(3, raw.getTableCalls, "REFRESH TABLE must NOT drop another table's metastore entry"); + + // File cache: t1's listing dropped, t2's survives (invalidateTable is scoped by (db, table)). + Assertions.assertEquals(1, fileCache.size(), "REFRESH TABLE must drop only that table's directory listings"); + } + + @Test + public void invalidateAllFlushesBothCachesEntirely(@TempDir java.nio.file.Path dir) throws Exception { + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + CachingHmsClient cachingClient = (CachingHmsClient) connector.wrapWithCache(raw); + + cachingClient.getTable("db", "t1"); + Assertions.assertEquals(1, raw.getTableCalls); + HiveFileListingCache fileCache = connector.fileListingCacheForTest(); + Files.write(dir.resolve("f"), new byte[10]); + fileCache.listDataFiles("db", "t1", dir.toUri().toString(), CONF); + Assertions.assertEquals(1, fileCache.size()); + + connector.invalidateAll(cachingClient); + + // Both caches fully cleared: the metastore entry re-fetches and the file cache is empty. + cachingClient.getTable("db", "t1"); + Assertions.assertEquals(2, raw.getTableCalls, "REFRESH CATALOG must drop the metastore cache"); + Assertions.assertEquals(0, fileCache.size(), "REFRESH CATALOG must drop the directory-listing cache"); + } + + @Test + public void publicHooksAreNoThrowAndClearFileCacheWithoutBuildingAClient(@TempDir java.nio.file.Path dir) + throws Exception { + // A fresh connector never built its metastore client (hmsClient == null). The public hooks must not + // force-build one (a REFRESH on a never-scanned catalog is a cheap no-op on the metastore side), must not + // throw on the null client, and must still clear the file cache. + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + HiveFileListingCache fileCache = connector.fileListingCacheForTest(); + Files.write(dir.resolve("f"), new byte[10]); + + fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); + Assertions.assertEquals(1, fileCache.size()); + Assertions.assertDoesNotThrow(() -> connector.invalidateTable("db", "t")); + Assertions.assertEquals(0, fileCache.size(), "REFRESH TABLE clears the file cache even with no client built"); + + fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); + Assertions.assertEquals(1, fileCache.size()); + Assertions.assertDoesNotThrow(() -> connector.invalidateAll()); + Assertions.assertEquals(0, fileCache.size(), "REFRESH CATALOG clears the file cache even with no client built"); + } + + /** + * Minimal {@link HmsClient} that counts {@code getTable} calls and returns a fresh table info per call (so a + * cache hit — same instance — is distinguishable from a reload). Only the abstract read methods are stubbed; + * the write/txn methods are interface defaults. + */ + private static final class RecordingHmsClient implements HmsClient { + int getTableCalls; + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + getTableCalls++; + return HmsTableInfo.builder().dbName(dbName).tableName(tableName).build(); + } + + @Override + public List listDatabases() { + return Collections.emptyList(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + return null; + } + + @Override + public List listTables(String dbName) { + return Collections.emptyList(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + return false; + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + return Collections.emptyMap(); + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + return Collections.emptyList(); + } + + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + return Collections.emptyList(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + return null; + } + + @Override + public void close() { + } + } +} From 157b8994d3ef7dfdc7973e97b5cbc99d635901b5 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:45:33 +0800 Subject: [PATCH 155/334] [fix](catalog) Hive connector cache: fe-core getNewestUpdateVersionOrTime last-modified branch (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Fix the coupled §2.6 correctness bug: PluginDrivenMvccExternalTable. getNewestUpdateVersionOrTime materializes the latest pin and reduces nameToLastModifiedMillis to a max, but a hive connector's listPartitions is names-only (every per-partition millis is the -1 UNKNOWN sentinel) -> the stream filters them all out -> orElse(0) -> a CONSTANT 0. Since Dictionary.hasNewerSourceVersion needs a monotonically non-decreasing value, a SQL dictionary / MV over a hive base table would compare equal forever and NEVER auto-refresh (silent, no crash). Mirror getTableSnapshot: when the materialized pin flags last-modified freshness (a hive connector sets it on the query-begin pin), return the connector's whole-table freshness millis via queryTableFreshness() -> getTableFreshness (now cache-backed by the connector's D2 metastore cache, so the periodic dictionary poll stays cheap), else 0 (dropped catalog/table or a genuinely empty partition set -> parity legacy). Byte- and cost-neutral for snapshot-id connectors (paimon/iceberg): they leave isLastModifiedFreshness() false, so the new branch is a single boolean read and they take the EXACT pre-change max-over-partitions / range-view path with zero added metadata calls (memory plugindriven-mvcc-table-is-live-not-dormant). The degraded pin (dropped catalog / no handle) uses emptySnapshot() whose flag is false, so it too keeps the existing 0 path. Dormant: no live connector sets the flag until "hms" enters SPI_READY_TYPES. Tests: PluginDrivenMvccExternalTableTest +3 -- last-modified connector returns the whole-table freshness millis (deliberately distinct from the partition max so the max path would be red); empty freshness (dropped/empty) -> 0; snapshot-id connector keeps the max-partition path AND never fires the freshness probe (byte/cost-neutrality guard). fe-core PluginDrivenMvccExternalTableTest 56 green + checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../PluginDrivenMvccExternalTable.java | 12 +++++ .../PluginDrivenMvccExternalTableTest.java | 46 +++++++++++++++++++ 2 files changed, 58 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java index 870878a181b5e9..f6eae7b901ce0a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java @@ -700,6 +700,18 @@ public long getNewestUpdateVersionOrTime() { // Dictionary-update path: always probe LATEST (bypass any context pin), mirroring legacy // which passes empty/empty to force a fresh listing. PluginDrivenMvccSnapshot pin = materializeLatest(); + // Last-modified connector (e.g. hive): the whole-table newest-change signal is a modify TIMESTAMP + // (transient_lastDdlTime / the max partition modify time), NOT the partition listing — which for hive is + // names-only, so every nameToLastModifiedMillis below is -1, gets filtered, and collapses to a CONSTANT 0. + // That constant would make Dictionary.hasNewerSourceVersion compare equal forever, so a SQL dictionary / MV + // over a hive base table would NEVER auto-refresh. Mirror getTableSnapshot: when the pin flags last-modified + // freshness, return the connector's whole-table freshness millis (cache-backed getTableFreshness, so the + // periodic dictionary poll stays cheap), else 0 (dropped catalog/table or a genuinely empty partition set — + // parity legacy). A snapshot-id connector (paimon/iceberg) leaves the flag false and takes the EXACT + // pre-change path below: a single boolean read, zero added metadata calls — byte- and cost-neutral. + if (pin.getConnectorSnapshot().isLastModifiedFreshness()) { + return queryTableFreshness().map(ConnectorTableFreshness::getTimestampMillis).orElse(0L); + } if (pin.getPartitionType() != null) { // Range-view path: nameToLastModifiedMillis holds (non-monotonic) snapshot ids, NOT a usable // change marker. Use the connector-supplied newest-update-time, which IS monotonic (parity master diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java index 3195375f4d344e..53bb0c0c2da15c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java @@ -897,6 +897,52 @@ public void testGetNewestUpdateVersionOrTimeIgnoresUnknownAmongReal() throws Ana "the UNKNOWN sentinel must be filtered, leaving the max of the REAL values"); } + // ==================== getNewestUpdateVersionOrTime: last-modified (hive) freshness ==================== + + private static final long TS_TABLE_FRESH = 1_888_000_000_000L; // distinct from the partition maxes above + + @Test + public void testGetNewestUpdateVersionLastModifiedUsesTableFreshness() { + // A last-modified connector (hive) lists partitions names-only (all lastModifiedMillis == -1), so the + // legacy max-over-partitions path would collapse to a CONSTANT 0 and an MV / SQL dictionary over a hive + // base table would never auto-refresh. The pin flags last-modified freshness, so + // getNewestUpdateVersionOrTime must return the connector's whole-table freshness millis instead. + Fixture f = Fixture.partitioned(); + flagPinLastModified(f); + Mockito.when(f.metadata.getTableFreshness(Mockito.any(), Mockito.any())) + .thenReturn(Optional.of(new ConnectorTableFreshness("dt=2024-02-02", TS_TABLE_FRESH))); + + // MUTATION: taking the max-over-partitions path (ignoring the pin flag) would return the partition max + // TS_2024_02_02, not the freshness value TS_TABLE_FRESH -> red (the values are deliberately distinct). + Assertions.assertEquals(TS_TABLE_FRESH, f.table.getNewestUpdateVersionOrTime(), + "a last-modified connector must surface the whole-table freshness millis, not a constant 0"); + } + + @Test + public void testGetNewestUpdateVersionLastModifiedEmptyFreshnessReturnsZero() { + // A dropped catalog/table, or a genuinely empty partition set, makes getTableFreshness empty; fe-core must + // degrade to 0 (parity legacy getNewestUpdateVersionOrTime), NOT throw or leak a sentinel. + Fixture f = Fixture.partitioned(); + flagPinLastModified(f); + Mockito.when(f.metadata.getTableFreshness(Mockito.any(), Mockito.any())) + .thenReturn(Optional.empty()); + // MUTATION: mapping an empty freshness to anything but 0 (e.g. throwing, or leaking -1) makes this red. + Assertions.assertEquals(0L, f.table.getNewestUpdateVersionOrTime(), + "an empty whole-table freshness (dropped/empty) must reduce to the legacy 0"); + } + + @Test + public void testGetNewestUpdateVersionSnapshotIdConnectorSkipsFreshnessProbe() { + // Byte/cost-neutrality guard: a snapshot-id connector (paimon/iceberg) leaves the pin flag false, so + // getNewestUpdateVersionOrTime must take the EXACT pre-change max-over-partitions path and NEVER fire the + // freshness probe (an added metadata round-trip on the live dictionary poll). + Fixture f = Fixture.partitioned(); // build() pin has lastModifiedFreshness=false + Assertions.assertEquals(TS_2024_02_02, f.table.getNewestUpdateVersionOrTime(), + "a snapshot-id connector must keep the max-partition-modify path"); + // MUTATION: dropping the pin-flag gate (probing unconditionally) makes this verify red. + Mockito.verify(f.metadata, Mockito.never()).getTableFreshness(Mockito.any(), Mockito.any()); + } + @Test public void testIsPartitionColumnAllowNullTrue() { Assertions.assertTrue(Fixture.partitioned().table.isPartitionColumnAllowNull()); From 4aebaeedb0ea43e103f1f38ee4ff0f4c771cb373 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:47:03 +0800 Subject: [PATCH 156/334] [doc](catalog) Hive connector cache: record C-d/C-e/C-f DONE, next = unified adversarial review (per-phase handoff) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 2 +- ...ve-connector-cache-step-design-2026-07-10.md | 17 ++++++++++++----- 2 files changed, 13 insertions(+), 6 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index dcbb69e43b353a..81ffa019c8fba4 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🎯 当前状态（2026-07-09） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**✅ C-b DONE（`4fe55d88fab`，休眠）= `CachingHmsClient` 装饰器（方案 A，置 `fe-connector-hms`）**：`implements HmsClient` 透明包裹底层 client（生产=`ThriftHmsClient`），对 **4** 个只读方法各挂一 `MetaCacheEntry`（镜像 paimon/iceberg：contextual-only+manual-miss、`ForkJoinPool.commonPool()`、Caffeine 封装不外泄）——`getTable`(键 db,table)/`listPartitionNames`(键 db,table,maxParts)/`getPartitions`(键 db,table,请求名列表)/`getTableColumnStatistics`(键 db,table,请求列列表)；其余 20 方法+`close()` 逐字委派。`getPartitions`/列统计用 **RPC 参数(整列表、顺序敏感)键**（非 per-partition/per-column 装配——后者需重建 Hive 分区名字符串、此处无从取；Option A 已签字接受 RPC 粒度）。键全带 db,table 以便 `flush(db,table)` 按表清（tableCache 精确 invalidateKey，3 谓词缓存 invalidateIf(matches)）、`flushAll()` 清全部（为 C-e REFRESH TABLE/CATALOG 接线预备）。默认镜像 legacy fe-core Config（TTL 86400s；容量 table/names/partition/stats=10000/10000/100000/10000），经 `CacheSpec.fromProperties(props,"hive",,default)` 读 `meta.cache.hive..*`，连接器只读自己 props（不碰 fe-core Config）。**用户签字（2026-07-10）：第 4 类列统计一起缓存**（对齐 Q1 四方法枚举；"非目标"里"先不做"被 Q1 覆盖）。**pom**：`-hms` 加 `fe-connector-cache`(compile，让装饰器编译)+Caffeine 2.9.3(test scope，单测构造 MetaCacheEntry 需)；补 C-a"依赖只加在 -hive"的遗漏（类在 -hms）；Caffeine 仍 `provided`/非传递、由 -hive 插件打包。**残留（记账）**：`-hms` 亦被 `-hudi` 依赖→`fe-connector-cache` jar 传递进 hudi 闭包但 Caffeine provided 不进→hudi 若日后自包 client 复用须自带 Caffeine 2.9.3；C-b 里 hudi 不构造它→无 NoClassDefFound、仅休眠 jar。**休眠证**：grep 无任何处引用 `CachingHmsClient`、`SPI_READY_TYPES` 无 hms→永不实例化；`HiveConnector.createClient():379` 仍返裸 `ThriftHmsClient`（接线归 C-c）。**测**：`CachingHmsClientTest` 12 测（命中+各维含 **db 维**/props 三档开关/flush 按表 4 缓存隔离/flushAll 4 缓存全清/透传委派/异常传播不缓存/空列统计亦缓存/null delegate 拒）；hms 68 测 + checkstyle 0 + import gate 净；-hive/-hudi 回归 BUILD SUCCESS。**4 维 clean-room 对抗复审（`wf_39f6458c-e37`：委派完整/缓存正确/休眠打包/测试强度 + 逐条 refute）：生产码 0 缺陷**；5 findings→4 CONFIRMED（全 minor 测覆盖缺口，已补：db 维/flushAll 全缓存/flush 谓词缓存按表作用域/测类 javadoc 陈旧）+1 refuted（可变 list 容器按引返回=既有全库缓存惯例、休眠无变更方；顺带补主类 javadoc 说明容器共享须只读）。**✅ C-c DONE（`7b05df6e55e`，休眠）= `CachingHmsClient` 接线进连接器 client**：`HiveConnector.createClient` 把裸 `ThriftHmsClient` 经新 package-private `wrapWithCache(raw)` seam 包成 `CachingHmsClient`（`new CachingHmsClient(raw, properties)`）；**透明装饰器**故此一处 wrap 令 `HiveConnectorMetadata` 全部 `hmsClient.*` 读自动走缓存——`getTable`(getTableHandle/getColumnHandles/getTableStatistics/view)、`listPartitionNames`、`getPartitions`（扫描 + **两条新鲜度探针**）、`getTableColumnStatistics`——**§2.6 达成**：`getTableFreshness`/`getPartitionFreshnessMillis` 都读 `hmsClient.getPartitions` 故周期性 SQL 字典/MV 刷新 poll 由缓存服务、不再每 tick 打 metastore。装饰器按 catalog 自身 `meta.cache.hive.*` props 配置（disable/ttl<=0→透传），缓存挂 `HmsClient`（连接器 init/`ADD`/`MODIFY CATALOG` 重建）、绝不入 handle/GSON edit log。`wrapWithCache` 镜像既有 `newMetadata(HmsClient)` seam 使单测能用 fake client 驱 wrap 而不触发 `createClient` 建真 `ThriftHmsClient`。**休眠**：hms 不在 `SPI_READY_TYPES`→无 live 目录建 `HiveConnector`，wrap 仅单测跑；只改 `HiveConnector.java`→paimon/iceberg/jdbc/hudi 字节不变。**测**：新 `HiveConnectorClientCacheTest`(4)——wrapWithCache 返 CachingHmsClient / 重复 getTableFreshness+getPartitionFreshnessMillis 由缓存服务（1 RPC 非 2）/ 关 `meta.cache.hive.partition` 使 getPartitions 每 poll 重载而仍开的 partition-name 缓存不重载（连接器级 per-entry props 穿透）；hive **244 测全绿**（240→244）+ checkstyle 0 + import gate 净（HiveVersionUtil vendored 跳过=已知误报）。**下一步 = C-d**（文件列表缓存：`HiveScanPlanProvider` 目录列举 + `estimateDataSizeByListingFiles` rowcount 回填走连接器自持文件缓存，TCCL 钉扫描线程 loader；最高复杂度、可分离步）。 C-d…C-f 每步独立休眠 commit + 单测 + checkstyle/import gate，全部落地后统一对抗复审再翻闸。次子系统 = 事件 Model B。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） +**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**✅ C-b DONE（`4fe55d88fab`，休眠）= `CachingHmsClient` 装饰器（方案 A，置 `fe-connector-hms`）**：`implements HmsClient` 透明包裹底层 client（生产=`ThriftHmsClient`），对 **4** 个只读方法各挂一 `MetaCacheEntry`（镜像 paimon/iceberg：contextual-only+manual-miss、`ForkJoinPool.commonPool()`、Caffeine 封装不外泄）——`getTable`(键 db,table)/`listPartitionNames`(键 db,table,maxParts)/`getPartitions`(键 db,table,请求名列表)/`getTableColumnStatistics`(键 db,table,请求列列表)；其余 20 方法+`close()` 逐字委派。`getPartitions`/列统计用 **RPC 参数(整列表、顺序敏感)键**（非 per-partition/per-column 装配——后者需重建 Hive 分区名字符串、此处无从取；Option A 已签字接受 RPC 粒度）。键全带 db,table 以便 `flush(db,table)` 按表清（tableCache 精确 invalidateKey，3 谓词缓存 invalidateIf(matches)）、`flushAll()` 清全部（为 C-e REFRESH TABLE/CATALOG 接线预备）。默认镜像 legacy fe-core Config（TTL 86400s；容量 table/names/partition/stats=10000/10000/100000/10000），经 `CacheSpec.fromProperties(props,"hive",,default)` 读 `meta.cache.hive..*`，连接器只读自己 props（不碰 fe-core Config）。**用户签字（2026-07-10）：第 4 类列统计一起缓存**（对齐 Q1 四方法枚举；"非目标"里"先不做"被 Q1 覆盖）。**pom**：`-hms` 加 `fe-connector-cache`(compile，让装饰器编译)+Caffeine 2.9.3(test scope，单测构造 MetaCacheEntry 需)；补 C-a"依赖只加在 -hive"的遗漏（类在 -hms）；Caffeine 仍 `provided`/非传递、由 -hive 插件打包。**残留（记账）**：`-hms` 亦被 `-hudi` 依赖→`fe-connector-cache` jar 传递进 hudi 闭包但 Caffeine provided 不进→hudi 若日后自包 client 复用须自带 Caffeine 2.9.3；C-b 里 hudi 不构造它→无 NoClassDefFound、仅休眠 jar。**休眠证**：grep 无任何处引用 `CachingHmsClient`、`SPI_READY_TYPES` 无 hms→永不实例化；`HiveConnector.createClient():379` 仍返裸 `ThriftHmsClient`（接线归 C-c）。**测**：`CachingHmsClientTest` 12 测（命中+各维含 **db 维**/props 三档开关/flush 按表 4 缓存隔离/flushAll 4 缓存全清/透传委派/异常传播不缓存/空列统计亦缓存/null delegate 拒）；hms 68 测 + checkstyle 0 + import gate 净；-hive/-hudi 回归 BUILD SUCCESS。**4 维 clean-room 对抗复审（`wf_39f6458c-e37`：委派完整/缓存正确/休眠打包/测试强度 + 逐条 refute）：生产码 0 缺陷**；5 findings→4 CONFIRMED（全 minor 测覆盖缺口，已补：db 维/flushAll 全缓存/flush 谓词缓存按表作用域/测类 javadoc 陈旧）+1 refuted（可变 list 容器按引返回=既有全库缓存惯例、休眠无变更方；顺带补主类 javadoc 说明容器共享须只读）。**✅ C-c DONE（`7b05df6e55e`，休眠）= `CachingHmsClient` 接线进连接器 client**：`HiveConnector.createClient` 把裸 `ThriftHmsClient` 经新 package-private `wrapWithCache(raw)` seam 包成 `CachingHmsClient`（`new CachingHmsClient(raw, properties)`）；**透明装饰器**故此一处 wrap 令 `HiveConnectorMetadata` 全部 `hmsClient.*` 读自动走缓存——`getTable`(getTableHandle/getColumnHandles/getTableStatistics/view)、`listPartitionNames`、`getPartitions`（扫描 + **两条新鲜度探针**）、`getTableColumnStatistics`——**§2.6 达成**：`getTableFreshness`/`getPartitionFreshnessMillis` 都读 `hmsClient.getPartitions` 故周期性 SQL 字典/MV 刷新 poll 由缓存服务、不再每 tick 打 metastore。装饰器按 catalog 自身 `meta.cache.hive.*` props 配置（disable/ttl<=0→透传），缓存挂 `HmsClient`（连接器 init/`ADD`/`MODIFY CATALOG` 重建）、绝不入 handle/GSON edit log。`wrapWithCache` 镜像既有 `newMetadata(HmsClient)` seam 使单测能用 fake client 驱 wrap 而不触发 `createClient` 建真 `ThriftHmsClient`。**休眠**：hms 不在 `SPI_READY_TYPES`→无 live 目录建 `HiveConnector`，wrap 仅单测跑；只改 `HiveConnector.java`→paimon/iceberg/jdbc/hudi 字节不变。**测**：新 `HiveConnectorClientCacheTest`(4)——wrapWithCache 返 CachingHmsClient / 重复 getTableFreshness+getPartitionFreshnessMillis 由缓存服务（1 RPC 非 2）/ 关 `meta.cache.hive.partition` 使 getPartitions 每 poll 重载而仍开的 partition-name 缓存不重载（连接器级 per-entry props 穿透）；hive **244 测全绿**（240→244）+ checkstyle 0 + import gate 净（HiveVersionUtil vendored 跳过=已知误报）。**✅ C-d DONE（`7c0ee1ffb2a`，休眠）= 连接器自持文件列表缓存**：新 `HiveFileListingCache`（D2 第二层，独立于 CachingHmsClient 元数据层，对齐 Trino CachingDirectoryLister 独立于 CachingHiveMetastore），memoize 每分区目录的 `FileSystem.listStatus`，键 `(db,table,location)`，值 slim 不可变 `HiveFileStatus{path,length,modificationTime}`（脱 Hadoop FileStatus）。挂长命 `HiveConnector` final 字段（同 readTxnManager——扫描器/metadata 每次调用重建、缓存须活在连接器上），同一实例注入**扫描器**(`HiveScanPlanProvider`)与**行数估算**(`HiveConnectorMetadata.estimateDataSizeByListingFiles`)→扫描暖估算、反之亦然（§4.4 第 5 缓存耦合）。**TCCL**：contextual+manual-miss+无自动刷新→loader 在调用方线程跑（扫描线程由 `PluginDrivenScanNode.onPluginClassLoader` 钉、stats 线程由 estimateDataSizeByListingFiles 自钉），背景刷新线程不继承 pin 会崩 Hadoop 反射。**平价**：真 lister 滤目录+`_`/`.` 隐藏文件（字节对齐旧内联滤）、0 长文件保留（splitter 跳、估算加 0）；I/O 失败抛 `DorisConnectorException` 且**永不缓存**，扫描侧 catch→跳分区带 warn（旧对缺失分区目录的容忍，并把原 `FileSystem.get` 硬失败并入同一 skip 路径〔已记档；坏存储配置仍经估算 fail-loud + docker e2e〕），估算侧传播→-1（旧 all-or-nothing）。默认镜像旧 Config：TTL 86400s、容量 `max_external_file_cache_num`(10000)，读 `meta.cache.hive.file.*`。ACID 路径不缓存（旧平价）。`splitFile` 重构成收原语（path,size,modTime）供 cached 非-ACID 与 ACID FileStatus 共用；`HiveConnectorMetadata` 加生产 7-arg 构造器注入共享缓存、旧 3/6-arg 便利构造器自建私有默认（约 20 处测试 call-site **零改动**）。**测**：新 `HiveFileListingCacheTest`(9)——按 location 命中/miss、(db,table,location) 键隔离、invalidateTable 按表、invalidateAll、失败传播不缓存、`meta.cache.hive.file.enable=false` 透传、真 FS 目录/隐藏滤(@TempDir)、**两条集成测证扫描器 AND 行数估算都由同一缓存服务**（重复扫描/刷新每目录只列一次）；hive **253 测全绿**（244→253）+ checkstyle 0 + import gate 净。**✅ C-e DONE（`7bf90a7fe3c`，休眠）= REFRESH TABLE/CATALOG 接线**：`HiveConnector.invalidateTable(db,table)`/`invalidateAll()` override 各 drop **两层缓存**（`CachingHmsClient.flush/flushAll` 元数据层 + `HiveFileListingCache.invalidateTable/invalidateAll` 文件层）——因 hive 表 schema/分区/文件皆可变故 REFRESH 须全清（异于 iceberg 不可变 manifest 跨 REFRESH TABLE 保留）；invalidateTable 经两缓存共有的 (db,table) 键按表 scope。hooks **读 client 字段而不 force-build**（never-scanned 目录 REFRESH 元数据侧 cheap no-op；文件缓存 final 恒在）；fe-core 已路由 REFRESH→connector（零 fe-core 改）。metastore flush 需可观测 CachingHmsClient（只能经 C-c `wrapWithCache` seam 建）故公有 hooks 委派 package-private `invalidateTable(client,...)/invalidateAll(client)` 重载 + 加 `fileListingCacheForTest()`/`HiveFileListingCache.size()`（镜像 iceberg manifestCacheForTest/size）。**测**：新 `HiveConnectorInvalidateTest`(3)——invalidateTable 只 drop 该表两层（另表两层存活）、invalidateAll 全清两层、公有 hooks 无 client 时不 throw 且清文件缓存；hive **256 测全绿** + checkstyle 0 + import gate 净。**✅ C-f DONE（`12e0c9177c2`，休眠）= fe-core §2.6 末改时间分支**：`PluginDrivenMvccExternalTable.getNewestUpdateVersionOrTime` 加 last-modified 分支——pin `isLastModifiedFreshness()` 真（hive 在查询起始 pin 上设）→返 `queryTableFreshness().getTimestampMillis()`（cache-backed getTableFreshness），否则走原路径。**根因**：hive listPartitions names-only→每分区 millis=-1→过滤后 orElse(0) 恒 0→`Dictionary.hasNewerSourceVersion` 永判等→hive 基表字典/MV 永不自动刷新（静默）。镜像 getTableSnapshot；**对 snapshot-id 连接器(paimon/iceberg) 字节+成本双中立**（留 flag=false→单 boolean 读、零额外元数据往返、走原 max/range 路径），degraded pin 用 emptySnapshot()(flag=false) 亦保原 0 路径（memory `plugindriven-mvcc-table-is-live-not-dormant`）。**测**：`PluginDrivenMvccExternalTableTest`+3（last-modified 返 whole-table freshness millis〔刻意异于分区 max 使旧路径变红〕/空 freshness→0/snapshot-id 保 max 路径且**从不**探 getTableFreshness）=56 测全绿 + checkstyle 0。**⇒ 全部 6 个休眠 commit（C-a…C-f）落地，Hive 连接器自持缓存功能实现完整收官。下一步 = 统一 clean-room 对抗复审**（memory `clean-room-adversarial-review-pref`：多 agent 对抗、先独立判断后交叉核对历史结论；覆盖 6 commit：装饰器/文件缓存正确性·TCCL·休眠打包·字节中立·失效 scope·§2.6 单调性；随后才可翻闸〔hms 进 SPI_READY_TYPES〕）。次子系统 = 事件 Model B。**残留（e2e/翻闸，非静默）**：缓存命中/REFRESH/字典-MV 自动刷新/文件列表跨类加载器/hudi-on-HMS 复用 CachingHmsClient 均只翻闸后 live，须异构 HMS docker e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） **已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 diff --git a/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md b/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md index f35cf13e79289c..e3c1387977658b 100644 --- a/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md +++ b/plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md @@ -329,8 +329,15 @@ After all land, run a unified adversarial re-review (clean-room, per project pra - [x] C-c wire metadata reads + freshness probes through it + tests — `7b05df6e55e` (wrapWithCache seam in HiveConnector.createClient; transparent decorator => all HiveConnectorMetadata reads + both freshness probes cache-backed by the single wrap; HiveConnectorClientCacheTest 4; hive 244 green) -- [ ] C-d file-listing cache (per Q2) + TCCL + row-count backing + tests -- [ ] C-e `invalidateTable/invalidateAll` overrides + tests -- [ ] C-f §2.6 fe-core last-modified branch (byte-neutral guard) + tests -- [ ] unified adversarial re-review before the flip -- [ ] update HANDOFF.md per step (per-phase discipline) +- [x] C-d file-listing cache (per Q2) + TCCL + row-count backing + tests — `7c0ee1ffb2a` (HiveFileListingCache, + connector-owned + shared by scan provider AND estimate; (db,table,location) key; contextual+manual-miss + loader on the TCCL-pinned caller thread; dir/hidden filter parity; failure-not-cached; splitFile refactored + to primitives; HiveConnectorMetadata 7-arg production ctor; HiveFileListingCacheTest 9; hive 253 green) +- [x] C-e `invalidateTable/invalidateAll` overrides + tests — `7bf90a7fe3c` (drop BOTH layers per table / all; + no force-build of the client; package-private client-overloads + fileListingCacheForTest()/size() seams; + HiveConnectorInvalidateTest 3; hive 256 green) +- [x] C-f §2.6 fe-core last-modified branch (byte-neutral guard) + tests — `12e0c9177c2` (isLastModifiedFreshness + pin gate mirrors getTableSnapshot; snapshot-id connectors byte/cost-neutral; PluginDrivenMvccExternalTableTest + +3 = 56 green + checkstyle 0) +- [ ] unified adversarial re-review before the flip ← NEXT (all 6 dormant commits C-a…C-f landed) +- [x] update HANDOFF.md per step (per-phase discipline) From 00e8d015eadda6c69fbf72125cbb6fe17b1153ed Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 09:53:37 +0800 Subject: [PATCH 157/334] [doc](catalog) HANDOFF: slim to a lean rolling pointer doc (159KB->8.7KB) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The handoff had accumulated the full inline history of every landed step (P7 + §4.1-4.4 + iceberg/hudi delegation + the hive cache), which is already in git log and the tasks/ design docs. Rewrite it as a lean rolling doc that keeps only what the NEXT session needs: current position in the 4-phase flip plan, the just-finished Hive cache (6 commit hashes + one-line each), the immediate next step (unified adversarial review of those 6 commits), must-read pointers to the authoritative design docs, and the reused operating rules (branch/commit/build/test gotchas + fe-core iron rules + memory index). Progressive disclosure: deep detail lives in git log + tasks/*.md (all verified present); the handoff points to them instead of duplicating. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 169 ++++++++++---------------------------------- 1 file changed, 38 insertions(+), 131 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 81ffa019c8fba4..0a7437b7dccf36 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -1,155 +1,62 @@ # 🤝 Session Handoff -> 滚动文档：每次 session 结束**覆盖式更新**（不累积历史；历史见 `git log plan-doc/HANDOFF.md`）。协作规范：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md) -> **范围** = catalog-spi **主线** handoff。metastore/storage 抽取子线**已 CLOSED**（合入 #64446/#64653/#64655）——`metastore-storage-refactor/` 仅历史留存、勿读；metastore-spi 现状直接读代码 `fe/fe-connector/fe-connector-metastore-spi/`。 +> **滚动文档**：每次 session 结束**覆盖式更新**，**只保留下一个 session 必须的上下文**；完成的工作明细**不落这里**（在 `git log` + `tasks/` 设计文档里，见下「起步必读」）。协作规范：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)。 +> **范围** = catalog-spi **主线**（HMS 翻闸）。metastore/storage 抽取子线**已 CLOSED**（合入 #64446/#64653/#64655；`metastore-storage-refactor/` 仅历史留存、勿读，现状直接读代码 `fe/fe-connector/fe-connector-metastore-spi/`）。 --- -# 🎯 当前状态（2026-07-09） +# 🎯 当前状态（2026-07-10） -**▶ 当前工作线 = hudi flip-blocker（权威规划 `hudi-on-hms-delegation-plan-2026-07-08.md`，起步必读）**：✅HD-A0（`37b85348667`，DV-005 护栏）+ ✅HD-A1（`28d4781ccac`，hudi 兄弟持有/合成，镜像 iceberg S1/S2，休眠）已收官（详见文末 ⭐ Hudi 段）。**✅ Group A（HD-A0–A5）全部收官**（A2 Kerberos `561f777d5f9` / A3 分区值保真 `19f11063a79` / A4 force-JNI `7bc90edf4a3` / A5 BE 描述符+存储 `d14da8312b5`；+ 5 维对抗复审揪出并修好的 `populateRangeParams` 格式类型缺陷 `ce700db35e0`）。**✅ HD-B1 三路 foreign-handle 路由收官（`136b5ced333`，签字 Option 1 `Connector.ownsHandle` 中立 SPI + peek 路由，休眠；详见文末 ⭐ Hudi 段）。✅ HD-C1 = MVCC 快照 + 分区列举 + 新鲜度收官（`bbe6cfcd647`，休眠；hudi MTMV 钉真 instant〔有意提升非平价〕、listPartitions/Names/Values、hive-style 名转义保真、metaClient auth+TCCL 包装；6 维对抗复审揪出并修好 1 major〔单列斜杠值 name 撞车〕+2 测覆盖缺口；详见文末 ⭐ Hudi 段）。**✅ HD-C2 = FOR TIME AS OF 时间旅行收官（`cf4f0e1b756`，休眠；连接器 `resolveTimeTravel`+`applySnapshot`+`planScan` 钉查询时刻，byte-faithful legacy、宽松无校验、FOR VERSION 抛逐字拒；零 fe-core；4 维对抗复审 0 确认缺陷；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-1 = @incr 窗口解析脊柱收官（`9327261ec52`，休眠；连接器 `resolveTimeTravel(INCREMENTAL)`+`applySnapshot` 单一 locus 解析 (begin,end] 窗口、钉 `HudiTableHandle.begin/endInstant`；byte-faithful legacy COW/MOR、单时间线经 hudi-1.0.2 字节码证双类型平价、逐字 fail-loud；零 fe-core；4 维对抗复审+字节码核对 0 确认缺陷〔COW 时间线 blocker 被字节码驳倒〕；详见文末 ⭐ Hudi 段）。**✅ HD-C3 INC-2 = IncrementalRelation 家族移植收官（`d64a25f28fc`，休眠；用户签字方案 A 单一窗口权威=移植关系消费句柄已解析 (begin,end] 窗口只挑文件、不重解析→`MOR:92` 死码 sentinel bug 自然消失、文件集永不与 INC-4 行过滤 end 打架；archived-flag 起字节忠实 legacy、split re-home 成 `HudiScanRange`；INC-1 延后守卫作 `IncrementalRelation` 共享静态逐字文案；`resolveIncremental` END 轴按 `USE_TRANSITION_TIME` 策略走 completion-time；休眠=planScan 只读 queryInstant；5 维对抗复审 byte-parity/dormancy/guard 三镜头 0 发现+3 test/doc-honesty 已修+MOR:92/轴 crux 自证；详见文末 ⭐ Hudi 段）。✅ HD-C3 INC-3 = 增量 planScan 接线收官（`0fea5bc6ba4`，休眠；planScan 加 `beginInstant!=null` 支路按 metaClient 类型建 COW/MOR 关系挑文件、fallbackFullTableScan 降级到最新快照扫描、抽出 `buildMorRange` 钉 `getEndTs()`、原始 @incr 选项经 `hudi.incr-opt.` 前缀穿透进新 `HudiTableHandle.incrementalParams`；**用户签字 graceful**：force_jni+COW+增量不再抛 UnsupportedOperationException 崩溃、COW 增量恒原生；**RO-as-RT 被 metaClient.getTableType() 天然覆盖不移植**；行级 `_hoodie_commit_time` 过滤留 INC-4；4 维对抗复审 0 发现）。**✅ HD-C3 INC-4 = 中立 synthetic-predicate SPI 收官（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠；用户签字方案 A=通用 `ConnectorExpression` 非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f` 5 维〔row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing〕**0 发现**+1 minor 测 honesty 已修；详见文末 ⭐ Hudi 段）。**⚠ 交接旧「下一步 = INC-5」框架已纠正（HEAD 核实）**：INC-5 是**翻闸期(live)删除**（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）——这三处**今天仍为线上 legacy hudi 服务**（`CatalogFactory` 无 `hms`、`CheckPolicy.java:95` 注释「deleted at the cutover」、`datasource/hudi/source/*` 目录健在），**休眠期做不了、随原子翻闸一起删**，非独立下一步。**真正下一个休眠步 = HD-C4/C5 schema evolution（DV-006）**：**用户 2026-07-09 签字 B = 完整对齐老实现**（非接受窄退化；memory `hudi-schema-evolution-full-parity-signoff`）→ recon(`wf_85bd47a0-0aa`)+**权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`（起步必读）已签字**，拆 6 步（C4a 转换器→C4b 句柄 field-id→C4c 每-split schema_id→C4d 扫描字典→C5a getTableSchema@instant→C5b 扫描侧@instant）。**用户签字两部分拆分（2026-07-09）**：**Part 1 = C4a–C4d（稳态读演进）逐步休眠提交 + 完成后统一对抗复审**；**Part 2 = C5a–C5b（时间旅行叠加演进）**。**✅ Part 1 首步 C4a DONE（`d29bb52173d`，休眠）= 纯 `HudiSchemaUtils` 转换器**（`InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` + base64 encode/apply 往返 on `TFileScanRangeParams` f25/26；移植 legacy `HudiUtils.getSchemaInfo` 两处有意偏离〔逐层 `Locale.ROOT` 小写=镜像 iceberg、非 legacy verbatim；scalar `TColumnType`=STRING placeholder=丢 legacy 全型表〕、id+name+is_optional 每层；模板=paimon 多条历史、小写=iceberg 每层；wired 进空、零 fe-core、零新 SPI；`HudiSchemaUtilsTest` 11 mutation-tight 单测钉每层小写/嵌套 struct·array·map 标签/STRING scalar/id·optional/encode-apply 往返/null-noop/corrupt fail-loud；全绿 hudi 138 测 + checkstyle 0 + import gate 净）。**✅ C4b DONE（`d54dc2d3ac5`，休眠）= field-id 上 `HudiColumnHandle`**（`getSchemaFromMetaClient` 取 mode-aware InternalSchema〔no-arg `getTableInternalSchemaFromCommitMetadata()` 钉最新 commit：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 位置 id，镜像 legacy `getHudiTableSchema`〕、按名把顶层 field-id 挂 `ConnectorColumn.withUniqueId`、`getColumnHandles` 穿到 `HudiColumnHandle.fieldId`；**与 legacy 位置 zip 的有意偏离**=连接器保 `getTableAvroSchema(true)` 独立列源→按名匹配、未命中列〔如 `_hoodie_*`〕留 UNSET→BE BY_NAME；`fieldId` 不入 equals/hashCode〔身份保 name+type，镜像 iceberg〕；id 解析失败降级 columns 不带 id 而非丢整 schema；C5a 欠 `HudiMetaClientExecutor` 包裹；4 same-loader 测；全绿 hudi 142 测 + checkstyle 0 + import gate 净）。**✅ C4c DONE（`ac3222bc526`，休眠）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**（`HudiSchemaUtils` 加两共享 resolver：`resolveTableInternalSchema`〔mode-aware base，C4b 重构复用之为单一 id 源真源〕+`resolveFileInternalSchema`〔每文件版本：evolution 开→`FSUtils.getCommitTime`+`InternalSchemaCache.searchSchemaAndCache`〔2-arg=编译期真签名，非 javap 的 3-arg〕；关→base schema id 0〕，port legacy `HudiScanNode.setHudiParams`；`HudiScanRange.Builder.schemaId`〔可空〕→`hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** setSchemaId〔镜像 paimon，JNI 臂永不〕；`planScan` 解析一次 base schema 建每文件 resolver、`collectCowSplits`+`buildMorRange` 原生无 log 片钉之、@incr 传 null〔@incr 列最新 schema〕；base/每文件解析失败均降级 BY_NAME 不崩扫描。**排序**：schema_id present 而字典缺=BE 不入 field-id 路→BY_NAME=**安全前缀**，字典步须在其后〔dict-present+schema_id-unset 才是禁的 current==file==-1 退化〕。schema_id 值 e2e，只 stamping 机制〔原生 set f12、JNI 不 set〕same-loader 测；`HudiScanRangeTest`+3+`HudiIncrementalPlanScanTest` 断言；全绿 hudi 145 测 + checkstyle 0 + import gate 净）。**✅ C4d DONE（`617c44b049b`，休眠）= 扫描级 `hudi.schema_evolution` 字典发射（Part 1 收官）**（`HudiSchemaUtils.buildSchemaEvolutionDict`〔纯、可测〕=`-1` 目标条〔`buildTargetSchema` keyed off **requested columns**=BE scan slots，逐列按名在 base InternalSchema 查全嵌套+稳定 id、缺列〔如 evolution 表的 `_hoodie_*`〕退标量占位带 handle id、空 requested→全 base 字段〕+ 历史条；**用户签字 Option B=全部历史版本**〔非仅被引用，robust 超集、镜像 paimon all-`listAllIds`：每原生文件 schema_id 必在史→BE 永不 "miss schema info"，且字典与"扫哪些文件"零耦合、避开在无 splits 的 getScanNodeProperties 里重列文件的开销/发散〕：evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本、非 evolution 取 base convert 版本 0；`getScanNodeProperties` 建发 base64 prop〔force_jni gate off、best-effort 失败降 BY_NAME〕+ override `populateScanLevelParams` copy f25/26。排序：C4c〔schema_id 安全前缀〕已落故字典永不与 schema_id-unset 同现。`HudiSchemaUtilsTest`+5〔目标条 keyed off requested+嵌套 id/缺列标量退化/空 requested 全字段/非 evolution=目标+base/evolution=目标+全历史，皆 encode-apply 往返〕；全绿 hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d 稳态读演进）实现全部收官。✅ 统一对抗复审 DONE（`wf_33882086-fbb`，6 维读者+逐条对抗 refute；12 raw→8 CONFIRMED/4 refuted）→ 修复 `ad8aa87bd2d`**：①**[major] meta 列 NULL 回归修复**——BE field-id 是**逐文件非逐列**（文件带 schema_id 后整文件按 id 匹配、无逐列 BY_NAME 兜底），演进表投影 `_hoodie_*` meta 列（连接器暴露但 commit-metadata InternalSchema 无 meta→handle id UNSET）会被读成 const-NULL；`"UNSET→BY_NAME"` 注释是错的。**修=`buildSchemaEvolutionProp` 任一投影列 id 未解析即整份字典 gate OFF→BE 整扫描 BY_NAME（安全基线，data-only 投影仍全 field-id/改名匹配）**；②[minor] `planScan` 把 `resolveTableInternalSchema` 从 JNI 列 try 挪出独立 try/catch（演进解析失败降 BY_NAME、不再清空 JNI 列表挂 MOR 读）；③–⑥ [minor/nit] 测试加固（by-name vs 位置 zip 判别测〔未匹配列置首〕/软化 handleCarriesFieldId javadoc/补 resolveFileInternalSchema 非演进分支测/换掉 tautology 断言/加 gate 测）。全绿 hudi 153 测+checkstyle 0+import gate 净。**⚠翻闸残留（owed e2e，非静默）**：演进表 `SELECT *` 全改名正确性需给 meta 列注入保留 field-id 进每条字典（iceberg row-lineage 式）或不暴露 meta 对齐 legacy——延后，翻闸 e2e 决策。****✅ Part 2 首步 C5a DONE（`af9a3c1a004`，休眠）= schema-at-instant 列表**（3-arg `getTableSchema(session,handle,snapshot)` 覆写 keyed off `handle.getQueryInstant()`〔applySnapshot 钉的 pin，非 `snapshot.getSchemaId()`＝hudi 恒 -1；fe-core 传 post-applySnapshot 句柄〕、null instant 经**同一** shared build path 走 latest〔latest/at-instant 不发散〕；`getSchemaFromMetaClient` 包进 `metaClientExecutor.execute`〔plugin UGI doAs+TCCL pin，闭合 C4b 欠的 auth/TCCL gap〕、at-instant 臂字节忠实 legacy `getHudiTableSchema`+`initHudiSchema`〔reloadActiveTimeline+`getTableInternalSchemaFromCommitMetadata(instant)`→present=commit-metadata InternalSchema 稳定 id、absent=latest fallback〔D3 非演进字节等价〕〕；用 `HudiScanPlanProvider.buildMetaClient`〔与旧 inline build 字节等价〕、hive 3-arg 委派已发无须改；零 fe-core、零新 SPI；4 维对抗复审〔legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent〕+refute **0 确认缺陷**〔1 nit〔wrapping 后 LOG.warn 只印 wrapper message〕refuted＝旧码亦如此非本步引入，顺带把 `e.getMessage()`→传 throwable 补全 cause 链〕；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading〔2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〕；hudi 156 测+checkstyle 0+import gate 净）。**✅ C5b DONE（`e0da87888e5`，休眠）= 扫描侧 schema-at-instant**（FOR TIME AS OF over schema-on-read：①planScan JNI `column_names/types`@instant〔`HudiSchemaUtils.resolveJniColumnSchema`/纯 `chooseJniSchema`；plain read `queryInstant==null` 短路、不 reload timeline＝字节不变；非演进 latest==at-instant D3〕 ②getScanNodeProperties 原生 `-1` overlay 从**完整 pinned InternalSchema** 重建〔`buildSchemaEvolutionDictAtInstant`＝empty-requested over pinned；**非** latest-keyed 请求句柄——句柄 latest-keyed（`getColumnHandles` 在 MVCC pin 前跑），pin 后改名列的 pinned 名不在句柄里，用句柄建 overlay 会漏该 scan slot→BE `by_table_field_id` StructNode `children.at` std::out_of_range/SIGABRT；**full-pinned=scan slot 超集＝安全**，BE 按名查每槽、忽略多余字段〕；`current_schema_id` 保 -1；史=全版本 Option B；**无 meta 列可达 pin 路**（C5a 绑定 schema 源自 InternalSchema、无 `_hoodie_*`）；per-file schema_id resolver 仍用 latest base〔pin-independent〕；零 fe-core、零新 SPI）。**BE 契约亲手核对**：`be/src/format/table/table_schema_change_helper.{h,cpp}`（v1 reader `children_column_exists` DCHECK/`.at` = **超集必需**）vs `be/src/format_v2/table_reader.cpp`（`find_external_root_field` 按名查投影列 = **超集容忍**）。4 维对抗复审 `wf_ec6d47e3-37e`〔legacy-parity·at-instant/BE-contract·SIGABRT/no-regression·dormancy/test-intent〕+逐条 refute：**0 生产缺陷**，1 confirmed-minor 测注释诚实性已修〔`atInstantOverlay*` 测守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝e2e-owed〕，2 refuted。`HudiSchemaUtilsTest`+3〔JNI no-pin no-op / chooseJniSchema pinned 名 / full-pinned overlay 历史名〕；hudi 159 测+checkstyle 0+import gate 净。**⇒ Part 2（C5a–C5b）收官 ⇒ HD-C4/C5 schema evolution（DV-006）全部收官。⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing 只 e2e 可证**（离线测只覆盖纯 `chooseJniSchema`/full-pinned 组装，`if(pinnedSchema.isPresent())` 分支+`resolveInternalSchemaAtInstant` 需 live metaClient）。**✅ HD-D1 = 只读写-拒安全网收官（`7ae9404af25`，休眠；用户 2026-07-09 签字 Q1=保留干净拒绝+测试锁定+事务层防御、Q2=严格拒绝所有 DDL 含删表/改名）**：翻闸后 hudi 表→外来句柄→三路分流到 hudi 兄弟连接器（非 iceberg），拒绝主体已由 HD-B1 路由 + connector-api SPI 默认提供〔数据写：`getWritePlanProvider`=null→`supportedWriteOperations` 空→`PhysicalPlanTranslator` 规划期抛干净 `AnalysisException`；EXECUTE：`getProcedureOps`=null→拒；DDL：SPI 默认抛"not supported"〕。**故意保留 `getWritePlanProvider`=null 不改**〔admission gate 由它派生，在其上抛会被引擎误分类成内部 bug 而非干净"不支持 INSERT"〕；**净新增=显式末线防御**：override `HudiConnectorMetadata.beginTransaction(session)` 抛显式`"Hudi tables are read-only in this catalog…"`（替换泛型"Transactions not supported"默认；beginTransaction 只在执行期被调、规划期已拒故对 hudi 写是死码=纯消息质量兜底，不劣化分类；per-handle 重载经 SPI 默认委派到它）。`HudiReadOnlyWriteRejectTest` 锁全只读面〔无 writer+空写操作集/无过程/beginTransaction 双重载均抛显式只读/全 DDL 变更族〔删表·改名·truncate + 16 法 ALTER〕抛〕。3 维对抗复审 `wf_36fd1c6c-e58`：**0 生产缺陷**；2 survived nit（均测强化）已修〔per-handle beginTransaction 重载 + 全 DDL 族补锁〕；refuted：DROP DATABASE FORCE 级联〔pre-existing 网关码、legacy 平价、超出 per-table 范围＝翻闸残留〕+ 休眠（getTableHandle 未分流 HUDI＝HD-B2）。hudi 163 测+checkstyle 0+import gate 净。**⚠残留（翻闸/e2e）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名级联 `hmsClient.dropTable`〔与 iceberg-on-HMS 同款 legacy 平价、只删 HMS 登记不删数据、无 per-source 分支〕，翻闸时若要 per-table 拒绝须在网关 `dropDatabase` 加分流；②hudi 写/DDL 拒绝端到端只在 HD-B2 分流 HUDI 后 live（届时 e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud）。**⇒ Group A✅ + Group C✅ + HD-D1✅ 全完。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，休眠，武装 pivot，最后一个连接器步）**：`HiveConnectorMetadata.getTableHandle` 加 `if(tableType==HUDI) return hudiSiblingMetadata(session).getTableHandle(...)`（镜像 iceberg S4，复用 HD-A1 的 `getOrCreateHudiSibling`；返兄弟原始 `HudiTableHandle`、绝不盖 HiveTableHandle-HUDI〔否则 hudi 扫描/元数据路强转 CCE〕；置于 iceberg 臂后〔检测器 iceberg 优先〕+ UNKNOWN fail-loud 前〔只在 `tableType==HUDI` 触发、绝不吞真 UNKNOWN〕；外来 hudi 句柄永不强转）；新 `NO_HUDI_SIBLING` 哨兵 + `hudiSiblingSupplier` 字段 + `hudiSiblingMetadata` by-TYPE 解析器〔iceberg 孪生〕，6 参构造器加 hudi supplier〔序 iceberg,hudi,ownerResolver〕、3 参便利构造器传 `NO_HUDI_SIBLING`；`HiveConnector.getMetadata` 接 `this::getOrCreateHudiSibling`。点亮 HD-B1 三路 guard-and-forward（外来 hudi 句柄开始真流出 getTableHandle）。休眠至 hms 进 SPI_READY_TYPES。**4 维对抗复审 `wf_75e593cc-d0b`（routing/detector-agreement/dormancy-cce/test-intent）+逐条 refute：0 生产缺陷**；1 confirmed-minor 测缺口已修〔`getMetadata` 两同型 `Supplier` 实参可换序而无测守——抽 package-private `newMetadata(HmsClient)` 缝〔getMetadata 委派〕使接线可脱 Hadoop 客户端单测，`HiveConnectorThreeWayRoutingTest` 新守把每 by-TYPE supplier 钉到自家兄弟（**变异验证：换序即挂**）〕；1 refuted nit。DivertTest 重写〔独立 iceberg/hudi recording 兄弟：HUDI 分流到 hudi 兄弟 assertSame、绝不 iceberg、empty 透传证经兄弟、fail-loud 对称、UNKNOWN 不吞〕。hudi 240 测 + checkstyle 0 + import gate 净。**⚠残留（翻闸/e2e，非静默）**：hudi-on-HMS 委派全链（读〔COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved〕+ 写-拒 + 跨加载器 routing/CCE）只在翻闸〔hms 进 SPI_READY_TYPES〕后 live，须异构 HMS 目录 e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。**⇒ 全部连接器休眠步（Group A/B/C/D + iceberg S0–S6/W1–W5 + 写链 WC1–WC4）收官 ⇒ 只余「翻闸」本身（原子加 `"hms"` 进 SPI_READY_TYPES + 循环依赖删除单元 + 翻闸期须写码项〔W6 TCCL pin / D2 连接器缓存退休 / getNewestUpdateVersionOrTime 探针 / D5-D6 视图开关+SHOW CREATE / 异常映射 / dropDatabase force / 事件管道 / 系统表〕 + 翻闸 e2e/硬门）。**⭐ 用户已定（2026-07-09）：下一步 = 先出翻闸 recon + 设计文档（「先出规划不写码」，产出后用户再拍板 D4 打包〔单 PR vs 分 PR〕+ 翻闸序）**。翻闸设计**大部分已存**于权威设计 `hms-cutover-retype-design-2026-07-07.md` §2〔原子翻闸集〕/§5〔阶段序〕/§7〔硬门〕+ 两份 findings 的删除排序段——**须做的是"基于现已落地的全部休眠步，出一份 UP-TO-DATE 翻闸执行计划"**（勘察建议 5 维：①翻闸触发点+原子切换〔SPI_READY_TYPES/CatalogFactory case/GSON registerSubtype 三处/buildTableInternal MVCC 选类/PhysicalPlanTranslator 分派〕②删除单元+排序〔datasource/hive·hudi·iceberg HMS 支撑 ~23 类·HMSExternalCatalog/Table·visitPhysicalHudiScan 增量 throw·legacy CheckPolicy hudi 臂·bindHiveTableSink·Env.hiveTransactionMgr·HiveScanNode 等〕③翻闸期须写码项〔上列〕哪些硬前置 vs 可延后 ④e2e/硬门+两非单测 harness〔两加载器路由 fixture + 异构 HMS e2e，含 iceberg/hudi INSERT/DELETE/MERGE/ALTER/EXECUTE·COW/MOR·FOR TIME AS OF·@incr·schema-evolved·写-拒·dropped-partition-at-instant〕+ R-002 Kerberos-HMS 冒烟 ⑤D4 打包方案〔paimon 分 PR vs iceberg 合并 squash 两先例〕）。**✅ 翻闸 recon + 执行计划 DONE（2026-07-10，`wf_94feaccd-d60` 6 读者+2 对抗审查；权威文档 `hms-cutover-execution-plan-2026-07-10.md`，起步必读，SUPERSEDES 07-07 doc §4/§5 状态）**。**核心纠正（两审查一致 blocker）**：07-07 计划默认"翻闸前准备会做完、离切换一个提交"，但两周全投 hudi 支线→**两个较大 fe-core 子系统 0% 未建**：①HMS 事件同步插件化迁移（Model B）②连接器自持缓存（D2）。翻闸原子提交会让 4 个守卫（事件轮询 `MetastoreEventsProcessor:116` + 缓存路由 `ExternalMetaCacheRouteResolver:66` 等）**同时静默变 false**（事件同步永久停摆、缓存塌 ENGINE_DEFAULT，不报错不打日志）——故**翻闸今天不可作单提交**，两子系统是硬前置。已 HEAD 亲验（`SPI_READY_TYPES` 无 hms、连接器无 pollOnce/无自持缓存、3 份 fe-core 缓存+3 GSON registerSubtype 仍旧）。旧文档一批 ⛔TODO 其实早做完（Kerberos/读 SPI/新鲜度/兄弟委派/写链全 DONE）、行号漂移已修、hudi 支线带来新翻闸项（INC-5/hudi_meta TVF/hudi-scan 链删除）、**否决 2 虚惊**（SQL 缓存"过期结果"实为安全降级 cache-off；plain-hive eager-materialize 为字节平价，均 e2e-owed 非缺陷）。**用户签字（2026-07-10）**：①翻闸序=**先建两子系统再切换**（无中断窗口，Trino 对齐）②**删旧代码放最后、不纠结 PR 形态、优先"翻闸后走新代码逻辑"**（⇒删除前的抽取 §2.4 移到最后删除阶段）③hive 自动刷新=**随缓存子系统补齐达平价**（`getNewestUpdateVersionOrTime` 骑 D2）。**4 阶段结构**：①连接器休眠准备✅→②翻闸前 fe-core 建设（**真正剩余主活，0%**：D2 缓存〔带 max-partition-time〕→事件 Model B→3 个 loud-break 耦合缝〔partition_values TVF PluginDriven 臂 / hudi_meta TVF 决策 / 可选 SUPPORTS_SAMPLE_ANALYZE〕→W6 写路径 TCCL pin 验证）→③原子翻闸〔`SPI_READY+=hms`+3 GSON 重映射〔表→PluginDrivenMvccExternalTable〕+`buildDbForInit` HMS→PluginDrivenExternalDatabase+去 `PhysicalPlanTranslator` 死臂+DLAType import+INC-5 去 hudi 时间旅行/增量过时 throw+CheckPolicy hudi 臂+D5 视图开关去除+4 守卫改接新子系统+`HmsGsonCompatReplayTest`〕→④删除〔~90 类循环单元，最后做，含 §2.4 抽取 HiveUtil.toPartitionValues/isLzoInputFormat/HiveSplit 分支/IcebergUtils 6 成员〕→⑤e2e/硬门〔两非单测 harness=两 URLClassLoader 路由 fixture + 异构 HMS docker e2e；4 必验正确性风险=COW-@incr 行过滤〔INC-4 已给合规机制、straddling e2e-owed〕/W6/WC4 读锁/Kerberos 冒烟〕。**⚠ 建议新 session 开工 Phase 1**（本 session 已完成翻闸规划、上下文渐满；两子系统各自值得干净满上下文 recon→design→impl——memory `session-handoff-at-30pct-context`）。**✅ Phase 1 首步（Hive 连接器自持缓存）recon + 设计 + 签字 DONE（2026-07-10，`wf_d19057ca-5bb` 8 读者+3 对抗审查〔完整性/翻闸安全/范围〕，全 HEAD 核对；权威设计 `hive-connector-cache-step-design-2026-07-10.md`，起步必读）**。核心结论：翻闸后路由塌到 ENGINE_DEFAULT **不崩**（`registry.resolve("default")` 是真实注册的 schema-only 缓存），且 schema 经默认缓存**存活**（flipped 表 `getMetaCacheEngine()=="default"`，无 override）；真正丢的是**分区清单/分区对象/目录文件列表**三样扫描缓存，而 Hive 连接器**今天零缓存**（`HiveScanPlanProvider` 注释直言"无文件列表缓存"）。⇒ **本步=纯新增休眠**：保性能 + 修 §2.6（`getNewestUpdateVersionOrTime` 恒 0 致 hive 基表字典/MV 永不自动刷新），**不删旧缓存**（旧三缓存+4 守卫+resolver hms 臂零 live reader，删除放最后删除阶段）。对齐 **Trino**：连接器自持双层（`CachingHiveMetastore` 装饰器 + 独立 `CachingDirectoryLister`）。**用户签字两决策（2026-07-10）**：①元数据缓存=**装饰器方案**（`CachingHmsClient implements HmsClient` 置 fe-connector-hms，Trino 式，透明包 ~30 调用点、**可复用**给 hudi/iceberg 兄弟顺带修 hudi-on-HMS 翻闸丢缓存）②**文件列表缓存本步纳入**（完整对齐旧性能；TCCL 用钉住扫描线程的 loader）。**否决 2 虚惊**：`RefreshManager.refreshPartitions` CCE 非翻闸 blocker（**已核实**唯一调用者=事件类 `AlterPartitionEvent:123`，事件环 `instanceof HMSExternalCatalog` 门翻闸后对新目录不触发→属事件 Model B）；连接器"引用" fe-core 缓存=纯 javadoc（`import` grep 零，删除 import-clean）。**拆 6 步休眠 commit**：C-a pom（fe-connector-cache + Caffeine 2.9.3，构建期）→C-b `CachingHmsClient` 装饰器（未接线，休眠字段）→C-c 接线元数据读 + 新鲜度探针走缓存→C-d 文件列表缓存 + 接扫描/rowcount 列举（TCCL）→C-e `HiveConnector.invalidateTable/invalidateAll`（点亮 REFRESH TABLE/CATALOG）→C-f §2.6 fe-core `getNewestUpdateVersionOrTime` 末改时间分支（**对 paimon/iceberg 字节+成本双不变守卫**，memory `plugindriven-mvcc-table-is-live-not-dormant`）。**✅ C-a DONE（`f742651990d`，休眠·build-only）= pom 加 `fe-connector-cache` + Caffeine 2.9.3**（镜像 fe-connector-paimon：cache 框架把 Caffeine 声明 `provided`〔编译不打包〕→每消费插件须自带；hive 无传递 Caffeine〔iceberg 靠 iceberg-core 传递有、paimon 无〕故显式钉 **2.9.3**＝框架编译版＝各插件最低共有版，避 plugin-zip 版本分裂脑；**纯 build-only 零源码/零 SPI**，paimon/iceberg/jdbc 字节不变、hms 仍不在 `SPI_READY_TYPES` 故永不为 live catalog 实例化 HiveConnector；cache 类尚未被任何处消费＝从 C-b 起接线。构建实测：reactor BUILD SUCCESS + hive 插件 zip **仅一份** `lib/caffeine-2.9.3.jar` ＋ `lib/fe-connector-cache-*.jar` 子加载捆绑〔.m2 存 7 个 Caffeine 版本仍无分裂〕；hive **240 测全绿**〔surefire 真跑，禁 build-cache 复核〕+ checkstyle 0）。**✅ C-b DONE（`4fe55d88fab`，休眠）= `CachingHmsClient` 装饰器（方案 A，置 `fe-connector-hms`）**：`implements HmsClient` 透明包裹底层 client（生产=`ThriftHmsClient`），对 **4** 个只读方法各挂一 `MetaCacheEntry`（镜像 paimon/iceberg：contextual-only+manual-miss、`ForkJoinPool.commonPool()`、Caffeine 封装不外泄）——`getTable`(键 db,table)/`listPartitionNames`(键 db,table,maxParts)/`getPartitions`(键 db,table,请求名列表)/`getTableColumnStatistics`(键 db,table,请求列列表)；其余 20 方法+`close()` 逐字委派。`getPartitions`/列统计用 **RPC 参数(整列表、顺序敏感)键**（非 per-partition/per-column 装配——后者需重建 Hive 分区名字符串、此处无从取；Option A 已签字接受 RPC 粒度）。键全带 db,table 以便 `flush(db,table)` 按表清（tableCache 精确 invalidateKey，3 谓词缓存 invalidateIf(matches)）、`flushAll()` 清全部（为 C-e REFRESH TABLE/CATALOG 接线预备）。默认镜像 legacy fe-core Config（TTL 86400s；容量 table/names/partition/stats=10000/10000/100000/10000），经 `CacheSpec.fromProperties(props,"hive",,default)` 读 `meta.cache.hive..*`，连接器只读自己 props（不碰 fe-core Config）。**用户签字（2026-07-10）：第 4 类列统计一起缓存**（对齐 Q1 四方法枚举；"非目标"里"先不做"被 Q1 覆盖）。**pom**：`-hms` 加 `fe-connector-cache`(compile，让装饰器编译)+Caffeine 2.9.3(test scope，单测构造 MetaCacheEntry 需)；补 C-a"依赖只加在 -hive"的遗漏（类在 -hms）；Caffeine 仍 `provided`/非传递、由 -hive 插件打包。**残留（记账）**：`-hms` 亦被 `-hudi` 依赖→`fe-connector-cache` jar 传递进 hudi 闭包但 Caffeine provided 不进→hudi 若日后自包 client 复用须自带 Caffeine 2.9.3；C-b 里 hudi 不构造它→无 NoClassDefFound、仅休眠 jar。**休眠证**：grep 无任何处引用 `CachingHmsClient`、`SPI_READY_TYPES` 无 hms→永不实例化；`HiveConnector.createClient():379` 仍返裸 `ThriftHmsClient`（接线归 C-c）。**测**：`CachingHmsClientTest` 12 测（命中+各维含 **db 维**/props 三档开关/flush 按表 4 缓存隔离/flushAll 4 缓存全清/透传委派/异常传播不缓存/空列统计亦缓存/null delegate 拒）；hms 68 测 + checkstyle 0 + import gate 净；-hive/-hudi 回归 BUILD SUCCESS。**4 维 clean-room 对抗复审（`wf_39f6458c-e37`：委派完整/缓存正确/休眠打包/测试强度 + 逐条 refute）：生产码 0 缺陷**；5 findings→4 CONFIRMED（全 minor 测覆盖缺口，已补：db 维/flushAll 全缓存/flush 谓词缓存按表作用域/测类 javadoc 陈旧）+1 refuted（可变 list 容器按引返回=既有全库缓存惯例、休眠无变更方；顺带补主类 javadoc 说明容器共享须只读）。**✅ C-c DONE（`7b05df6e55e`，休眠）= `CachingHmsClient` 接线进连接器 client**：`HiveConnector.createClient` 把裸 `ThriftHmsClient` 经新 package-private `wrapWithCache(raw)` seam 包成 `CachingHmsClient`（`new CachingHmsClient(raw, properties)`）；**透明装饰器**故此一处 wrap 令 `HiveConnectorMetadata` 全部 `hmsClient.*` 读自动走缓存——`getTable`(getTableHandle/getColumnHandles/getTableStatistics/view)、`listPartitionNames`、`getPartitions`（扫描 + **两条新鲜度探针**）、`getTableColumnStatistics`——**§2.6 达成**：`getTableFreshness`/`getPartitionFreshnessMillis` 都读 `hmsClient.getPartitions` 故周期性 SQL 字典/MV 刷新 poll 由缓存服务、不再每 tick 打 metastore。装饰器按 catalog 自身 `meta.cache.hive.*` props 配置（disable/ttl<=0→透传），缓存挂 `HmsClient`（连接器 init/`ADD`/`MODIFY CATALOG` 重建）、绝不入 handle/GSON edit log。`wrapWithCache` 镜像既有 `newMetadata(HmsClient)` seam 使单测能用 fake client 驱 wrap 而不触发 `createClient` 建真 `ThriftHmsClient`。**休眠**：hms 不在 `SPI_READY_TYPES`→无 live 目录建 `HiveConnector`，wrap 仅单测跑；只改 `HiveConnector.java`→paimon/iceberg/jdbc/hudi 字节不变。**测**：新 `HiveConnectorClientCacheTest`(4)——wrapWithCache 返 CachingHmsClient / 重复 getTableFreshness+getPartitionFreshnessMillis 由缓存服务（1 RPC 非 2）/ 关 `meta.cache.hive.partition` 使 getPartitions 每 poll 重载而仍开的 partition-name 缓存不重载（连接器级 per-entry props 穿透）；hive **244 测全绿**（240→244）+ checkstyle 0 + import gate 净（HiveVersionUtil vendored 跳过=已知误报）。**✅ C-d DONE（`7c0ee1ffb2a`，休眠）= 连接器自持文件列表缓存**：新 `HiveFileListingCache`（D2 第二层，独立于 CachingHmsClient 元数据层，对齐 Trino CachingDirectoryLister 独立于 CachingHiveMetastore），memoize 每分区目录的 `FileSystem.listStatus`，键 `(db,table,location)`，值 slim 不可变 `HiveFileStatus{path,length,modificationTime}`（脱 Hadoop FileStatus）。挂长命 `HiveConnector` final 字段（同 readTxnManager——扫描器/metadata 每次调用重建、缓存须活在连接器上），同一实例注入**扫描器**(`HiveScanPlanProvider`)与**行数估算**(`HiveConnectorMetadata.estimateDataSizeByListingFiles`)→扫描暖估算、反之亦然（§4.4 第 5 缓存耦合）。**TCCL**：contextual+manual-miss+无自动刷新→loader 在调用方线程跑（扫描线程由 `PluginDrivenScanNode.onPluginClassLoader` 钉、stats 线程由 estimateDataSizeByListingFiles 自钉），背景刷新线程不继承 pin 会崩 Hadoop 反射。**平价**：真 lister 滤目录+`_`/`.` 隐藏文件（字节对齐旧内联滤）、0 长文件保留（splitter 跳、估算加 0）；I/O 失败抛 `DorisConnectorException` 且**永不缓存**，扫描侧 catch→跳分区带 warn（旧对缺失分区目录的容忍，并把原 `FileSystem.get` 硬失败并入同一 skip 路径〔已记档；坏存储配置仍经估算 fail-loud + docker e2e〕），估算侧传播→-1（旧 all-or-nothing）。默认镜像旧 Config：TTL 86400s、容量 `max_external_file_cache_num`(10000)，读 `meta.cache.hive.file.*`。ACID 路径不缓存（旧平价）。`splitFile` 重构成收原语（path,size,modTime）供 cached 非-ACID 与 ACID FileStatus 共用；`HiveConnectorMetadata` 加生产 7-arg 构造器注入共享缓存、旧 3/6-arg 便利构造器自建私有默认（约 20 处测试 call-site **零改动**）。**测**：新 `HiveFileListingCacheTest`(9)——按 location 命中/miss、(db,table,location) 键隔离、invalidateTable 按表、invalidateAll、失败传播不缓存、`meta.cache.hive.file.enable=false` 透传、真 FS 目录/隐藏滤(@TempDir)、**两条集成测证扫描器 AND 行数估算都由同一缓存服务**（重复扫描/刷新每目录只列一次）；hive **253 测全绿**（244→253）+ checkstyle 0 + import gate 净。**✅ C-e DONE（`7bf90a7fe3c`，休眠）= REFRESH TABLE/CATALOG 接线**：`HiveConnector.invalidateTable(db,table)`/`invalidateAll()` override 各 drop **两层缓存**（`CachingHmsClient.flush/flushAll` 元数据层 + `HiveFileListingCache.invalidateTable/invalidateAll` 文件层）——因 hive 表 schema/分区/文件皆可变故 REFRESH 须全清（异于 iceberg 不可变 manifest 跨 REFRESH TABLE 保留）；invalidateTable 经两缓存共有的 (db,table) 键按表 scope。hooks **读 client 字段而不 force-build**（never-scanned 目录 REFRESH 元数据侧 cheap no-op；文件缓存 final 恒在）；fe-core 已路由 REFRESH→connector（零 fe-core 改）。metastore flush 需可观测 CachingHmsClient（只能经 C-c `wrapWithCache` seam 建）故公有 hooks 委派 package-private `invalidateTable(client,...)/invalidateAll(client)` 重载 + 加 `fileListingCacheForTest()`/`HiveFileListingCache.size()`（镜像 iceberg manifestCacheForTest/size）。**测**：新 `HiveConnectorInvalidateTest`(3)——invalidateTable 只 drop 该表两层（另表两层存活）、invalidateAll 全清两层、公有 hooks 无 client 时不 throw 且清文件缓存；hive **256 测全绿** + checkstyle 0 + import gate 净。**✅ C-f DONE（`12e0c9177c2`，休眠）= fe-core §2.6 末改时间分支**：`PluginDrivenMvccExternalTable.getNewestUpdateVersionOrTime` 加 last-modified 分支——pin `isLastModifiedFreshness()` 真（hive 在查询起始 pin 上设）→返 `queryTableFreshness().getTimestampMillis()`（cache-backed getTableFreshness），否则走原路径。**根因**：hive listPartitions names-only→每分区 millis=-1→过滤后 orElse(0) 恒 0→`Dictionary.hasNewerSourceVersion` 永判等→hive 基表字典/MV 永不自动刷新（静默）。镜像 getTableSnapshot；**对 snapshot-id 连接器(paimon/iceberg) 字节+成本双中立**（留 flag=false→单 boolean 读、零额外元数据往返、走原 max/range 路径），degraded pin 用 emptySnapshot()(flag=false) 亦保原 0 路径（memory `plugindriven-mvcc-table-is-live-not-dormant`）。**测**：`PluginDrivenMvccExternalTableTest`+3（last-modified 返 whole-table freshness millis〔刻意异于分区 max 使旧路径变红〕/空 freshness→0/snapshot-id 保 max 路径且**从不**探 getTableFreshness）=56 测全绿 + checkstyle 0。**⇒ 全部 6 个休眠 commit（C-a…C-f）落地，Hive 连接器自持缓存功能实现完整收官。下一步 = 统一 clean-room 对抗复审**（memory `clean-room-adversarial-review-pref`：多 agent 对抗、先独立判断后交叉核对历史结论；覆盖 6 commit：装饰器/文件缓存正确性·TCCL·休眠打包·字节中立·失效 scope·§2.6 单调性；随后才可翻闸〔hms 进 SPI_READY_TYPES〕）。次子系统 = 事件 Model B。**残留（e2e/翻闸，非静默）**：缓存命中/REFRESH/字典-MV 自动刷新/文件列表跨类加载器/hudi-on-HMS 复用 CachingHmsClient 均只翻闸后 live，须异构 HMS docker e2e 断言（memory `hms-iceberg-delegation-needs-e2e`）。（后续实现宜新 session 满上下文起步，memory `session-handoff-at-30pct-context`。） +catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： -**已提交（休眠）**：P7.1（DDL 元数据）+ P7.3（`0b19506acfe`：hive/hms 写事务+提交器+写计划+读侧 ACID 生产者+读事务管理器）+ P7.4 子批 A（`0923077fe67`：`Connector.getScanPlanProvider(handle)` 按表选扫描器接缝）+ §4.1 Kerberos 认证器（`e63b03fb490`）+ §4.2 分区列发射（`32b9526f689`：`getTableSchema` 发射 `partition_columns` 属性 + string 分区列 widen 到 varchar(65533)）+ §4.2 表描述符（`164b25d9c42`：`buildTableDescriptor`→HIVE_TABLE）+ §4.2 按表能力接缝（`329853bf19c`：per-table `SUPPORTS_TOPN_LAZY_MATERIALIZE`，orc/parquet 门）+ §4.2 表行数统计三层全做（`eded05b05e6` 精确行数+数据字节 / `8c6d247bb34` 列文件估算 / `453a1926010` 复审修正）+ §4.2 视图 SPI（`6676bf365cf`：连接器侧 `viewExists`/`getViewDefinition`/`dropView` + 声明 `SUPPORTS_VIEW`；`HmsTableInfo`/`ThriftHmsClient` 补 view 文本字段）+ §4.2 file-format serde 修正（`c320df27712`：serde-authoritative 检测,修 JSON-当-CSV 读 + FORMAT_TEXT/CSV/JSON 区分 + OpenX 单列 + fail-loud;含 fe-core `mapFileFormatType` 拆 text/csv）+ §4.2 `listPartitions`/`listPartitionNames`（`68e9fc8835b`：连接器侧分区列举，names-only 热路径平价，`lastModifiedMillis`/统计留 -1）+ hms client -1=unbounded 修复（`3540d1327b6`：`ThriftHmsClient` `maxParts<=0` 原塌成 `Short.MAX_VALUE(32767)` 静默截断，改真 -1 无界，**顺带修好 hudi/hive-write 已有的同款截断**）+ §4.2 `getCapabilities` 连接器级（`96953cd8eec`：加 `SUPPORTS_COLUMN_AUTO_ANALYZE`+`SUPPORTS_METADATA_PRELOAD`，接 `SUPPORTS_VIEW`；MVCC/SHOW_CREATE/NESTED 明确**推迟非禁止**）+ §4.2 `HiveTableFormatDetector` 平价（`191449c7bd6`：补 3 个 LZO text 格式=对齐 legacy 7 格式集、`getTableHandle` 未知格式 fail-loud throw、view 短路）。**至此 §4.2 读侧连接器补齐全部完成**。+ §4.2a 列统计 fast-path SPI（`cbf9526f776`：新 `ConnectorColumnStatistics` DTO + `ConnectorStatisticsOps.getColumnStatistics`；hms 侧 `HmsColumnStatistics`+`ThriftHmsClient.getTableColumnStatistics`/`convertColumnStatistics`；hive `getColumnStatistics` 回传原始 ndv/nulls/avgColLen；fe-core `PluginDrivenExternalTable.getColumnStatistic` 覆写+`toColumnStatistic` 做 Doris 类型 size 运算；D1=保留免扫描列统计，byte-parity port of `getHiveColumnStats`/`setStatData`）+ §4.3 MVCC/MTMV 新鲜度（`3d784673ca4`：修 stale-MV 根因——`PluginDrivenMvccExternalTable.getTableSnapshot` 原写死 `MTMVSnapshotIdSnapshot`，hive 空 pin→常量 -1→MV 永判"已同步"永不刷新；改为按连接器放在**查询起始 pin** 上的 `lastModifiedFreshness` 标志分派：last-modified(hive)→`MTMVMaxTimestampSnapshot`/`MTMVTimestampSnapshot` 走按需 SPI（逐字复刻 legacy `HiveDlaTable`），snapshot-id(paimon/iceberg)→标志 false→**走改动前完全相同的 pin 路径、零额外元数据往返**；新 SPI `getTableFreshness`/`getPartitionFreshnessMillis` + DTO `ConnectorTableFreshness` + `ConnectorMvccSnapshot.lastModifiedFreshness`；hive `beginQuerySnapshot` 回传"空(-1)pin+last-modified 标志"、freshness 端口 `transient_lastDdlTime*1000`（解析全连接器侧）、声明 `SUPPORTS_MVCC_SNAPSHOT`；扫描热路径 `listPartitions` 仍 names-only、freshness 只在 MTMV 刷新按需取=两路径 legacy 平价）。**+ §4.4a 兄弟连接器 SPI 接缝**（`45dcca32fa8`：`ConnectorContext.createSiblingConnector(type,props)` 默认 return null + fe-core `DefaultConnectorContext` 覆写经 `ConnectorFactory.createConnector(...,this)` 建、iceberg/paimon 装饰器补委派）+ §4.4 S1 iceberg 属性合成纯函数（`d971410af62`）+ §4.4 S2 网关持有/懒建/close 转发兄弟（`119b4fca64b`）+ §4.4 S3 元数据 21 方法按句柄守卫转发外来 iceberg 句柄（`b80c614e5b4`）+ §4.4 S4 `getTableHandle` 对 iceberg 表返兄弟原始外来句柄、点亮 S3 全部守卫（`e7c28a39e69`）+ §4.4 S5 `getScanPlanProvider(handle)` 网关按句柄分流兄弟扫描器（`bb39802751b`：TCCL 白拿）+ §4.4 S6 按名方法/能力残留 = **零代码收官**（recon 证所有无句柄读方法 NO_DIVERT，只落 findings 文档）。**⇒ §4.4 读委派脊柱 S0–S6 全部收官**。**+ §4.4 写委派 W1（`e7a96f439e7`）：iceberg-on-HMS 的 14 个 ALTER-DDL（renameTable + 加/删/改/重排列、创/删 branch|tag、增/删/替换 partition-field）+ 2 个写校验（validateRowLevelDmlMode/validateStaticPartitionColumns）按句柄守卫转发兄弟（Shape B=复用 S3 模式、零新 SPI、零 fe-core；外来句柄永不强转；hive 分支字节不变=14 DDL 抛原 SPI 文案、2 校验静默 no-op〔Group-2 陷阱：校验误抛会新拒合法 plain-hive DML〕）。全绿（sibling-delegation +2 测〔16 方法 foreign 完整性锁 + hive throw/no-op + 零咨询兄弟〕，模块 209 测 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W2（`84890fe48d3`）：`Connector.getWritePlanProvider(handle)` per-handle 重载（默认委派无参、6 连接器不变）+ `HiveConnector` 按句柄分流兄弟 + fe-core 4 处 provider 取用点改 per-handle（复审揪出并修好 4 个 fe-core mock 测须 stub per-handle 重载的 blocker）。全绿（新 divert 测 4 + hive 213 + fe-core 触及套 + checkstyle 0 + import gate 净）。**+ §4.4 写委派 W3（`d1df2e0f7df`）：写能力判定 per-handle——**仅 4 个真差异**方法（`supportedWriteOperations`/`supportsWriteBranch`/`requiresPartitionHashWrite`/`requiresMaterializeStaticPartitionValues`；HEAD 核对 hive vs iceberg，另 3 个 requires* 两边同值=inert 留无参）加 per-handle 重载（从 `getWritePlanProvider(handle)` 派生、W2 已分流故无需 HiveConnector 覆写）；fe-core：`PhysicalPlanTranslator` 两 gate 先解析句柄、`PluginDrivenExternalTable` 加 `connectorSupported WriteOperations()`/`connectorSupportsWriteBranch()` 助手（解析句柄+句柄缺失降级 empty/false、绝不传 null 给网关）+ `requirePartitionHashOnWrite`/`materializeStaticPartitionValues` 改 per-handle，4 个 consumer（RowLevelDml/IcebergRowLevelDml/InsertInto/InsertOverwrite）走助手。对抗复审 0 blocker/major，修好被架空的降级测试+补 null-connector/句柄缺失覆盖。全绿（fe-core 触及套 incl. PluginDrivenExternalTableTest 28 + checkstyle 0〔api+fe-core〕）。**+ §4.4 写委派 W4（`5a41e4cc562`：事务按句柄）**：`ConnectorWriteOps.beginTransaction(session,handle)` per-handle 默认重载（委派无参、6 现存连接器继承默认→字节不变）+ `HiveConnectorMetadata` 守卫转发——foreign 句柄开兄弟 `IcebergConnectorTransaction`、hive 句柄开自有 `HiveConnectorTransaction`（**对称必需**，两写计划把 session-bound txn 强转不同具体类型）；fe-core 把已解析写目标句柄穿进**两个**开事务点（`PluginDrivenInsertExecutor` insert 漏斗经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()` + `ConnectorRewriteDriver` EXECUTE 已持句柄），**绝不给网关传 null**（fail-loud，否则普通写误路由兄弟）；侦察+4维对抗复审+verify（CCE 净、休眠/现存连接器平价、路由测 mutation-tight），2 测试覆盖缺口已补。**+ §4.4 写委派 W5（`8aee62bab20`：存储过程按句柄）**：`Connector.getProcedureOps(handle)` per-handle 默认重载（委派无参、6 连接器不变）+ `HiveConnector` 分流（hive/hudi 句柄→继承 null 无 procedure，外来 iceberg 句柄→兄弟，永不强转）；fe-core `ConnectorExecuteAction` 把 session+句柄解析上提到 `getProcedureOps` 空判之前、改调 per-handle 重载（网关无参恒 null，必须按句柄取）。**net-new**（旧拒 iceberg-on-HMS EXECUTE，已签字升级）；reorder 改失败文案序（坏表名先报"Table not found"再"does not support EXECUTE"，无测钉旧序）；顺带修好 `ConnectorExecuteAction` 类 javadoc 陈旧"never constructed pre-flip"（iceberg/paimon 已上线故 adapter 实时）。`ExecuteActionFactory.getSupportedActions` 留无参（无 live caller、handle-aware 变体延后）。4 维对抗复审（routing/precedence/CCE/completeness→逐条 verify）0 blocker/major/minor、1 nit（陈旧 javadoc，已修）。全绿（api 3 默认测 + hive 4 divert〔foreign/hive/hudi/plugin-absent〕 + fe-core per-handle 选择测；api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。**⇒ 至此 §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官，写执行捆绑仅余 W6（翻闸期 TCCL pin 验证，非休眠可测）。hive 尚未进 `SPI_READY_TYPES`，线上零路由。 +- **Phase 0 连接器休眠补齐** ✅ **DONE** — 读 SPI + iceberg/hudi 兄弟委派（S0–S6 / W1–W5 / WC1–WC4）+ 整条 hudi 线（HD-A0…HD-D1 + 武装 pivot HD-B2）。 +- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) → 事件管道 Model B → 3 个 loud-break 耦合缝 → W6 写路径 TCCL 验证。 +- **Phase 2 原子翻闸**（`SPI_READY += hms` + 3 处 GSON 重映射 + 死臂/INC-5/D5 删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门（R-002）**。 -**本 session 关键发现（重大计划校正，务必先读）**：对 iceberg-on-HMS 委派 + 事件管道两轮 code-grounded 侦察（`wf_24c2052f-198` / `wf_46c0c020-08f`）+ 亲手核对 HEAD 证实——**交接旧叙述把剩余工作拆成"翻闸前可独立落地的 6 子批（A–F）"，这个前提基本是错的**。真实结构性事实： +**✅ 本轮完成 = Phase 1 首项「Hive 连接器自持缓存(D2)」全部 6 个休眠 commit 落地**： +`f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` 装饰器) → `7b05df6e55e` C-c(接线进 client，元数据读+新鲜度探针走缓存) → `7c0ee1ffb2a` C-d(`HiveFileListingCache` 文件列表缓存，扫描+行数估算共享，TCCL 钉调用线程) → `7bf90a7fe3c` C-e(`invalidateTable/invalidateAll` 接 REFRESH TABLE/CATALOG，两层全清) → `12e0c9177c2` C-f(fe-core `getNewestUpdateVersionOrTime` last-modified 分支，修 hive 基表字典/MV 永不自动刷新)。 +设计 = **`tasks/hive-connector-cache-step-design-2026-07-10.md`**（TODO 全勾，**起步必读 #2**）。全休眠（hms 未进 `SPI_READY_TYPES`，线上零路由；paimon/iceberg/jdbc/hudi 字节不变；对齐 Trino 双层：`CachingHmsClient` 元数据层 + `HiveFileListingCache` 文件层）。 -- **`type=hms` 目录今天 100% 是旧类**：`CatalogFactory` case `"hms"` → `new HMSExternalCatalog`（`CatalogFactory.java:133-134`）；`HMSExternalTable extends ExternalTable`（非 `PluginDrivenExternalTable`）；`HMSExternalCatalog extends ExternalCatalog`（非 `PluginDrivenExternalCatalog`）。 -- **按表选扫描器接缝只在 `PluginDrivenScanNode` 内被调用**，而它只对 `instanceof PluginDrivenExternalTable` 的表生效（`PhysicalPlanTranslator.java:808`）；HMS 表走旧 `switch(dlaType)`→`IcebergScanNode` 分支（`:818-847`）。 -- **只有 `PluginDrivenExternalCatalog` 持有插件连接器**（`getConnector()`）；旧 `HMSExternalCatalog` 无连接器，用旧 `HMSCachedClient`。fe-core 只 maven 依赖 `fe-connector-api/-spi`，插件类经**独立 child-first 类加载器**加载。 -- **⇒ 结论**：iceberg/hudi 格式委派、事件管道搬迁，**都需要"HMS 目录先变成通用目录类（持有插件连接器）"才可达/可测**。它们不是翻闸前的独立步骤，而是**同一次翻闸的不同侧面**。翻闸前只能预置休眠脚手架或落"翻闸前减负"小修复。跨插件委派**不能靠加 maven 依赖/共包**（会出现第二份 AWS SDK，JVM 级毒化 S3，Paimon RC-3 有记录），须新增"网关拿到兄弟连接器"的中立 SPI。 +**⭐ 下一步 = 对这 6 个 commit 做统一 clean-room 对抗复审**（memory `clean-room-adversarial-review-pref`：多 agent 对抗、先独立判断后交叉核对历史结论）。重点镜头：装饰器/文件缓存正确性 · TCCL split-brain · 休眠打包 Caffeine 单版本(2.9.3) · paimon/iceberg **字节中立** · 失效 scope · §2.6 单调性 · **C-d 里 `FileSystem.get` 失败并入 skip-with-warn 的行为变化**（已记档，翻闸 e2e 复核）。复审通过后续 Phase 1 = **事件管道 Model B**（薄 fe-core 角色驱动 + 插件 `pollOnce` SPI，权威 = `tasks/hms-event-pipeline-findings-2026-07-07.md`）。 -**本 session 已落地（翻闸前唯一干净独立的减负项）**：`a6dc782d816` `[fix](catalog)` 消除编辑日志重放的 CCE 隐患——`ExternalMetaIdMgr.replayMetaIdMappingsLog` 原来强转 `(HMSExternalCatalog)` 只为取 id（日志本就带 catalogId），翻闸后会 CCE 使 FE 重放崩溃起不来；改为按 `catalogId(long)` 传，行为今天逐字不变，拆掉翻闸时必爆的雷。全绿（fe-core 编译 BUILD SUCCESS + `ExternalMetaIdMgrTest` 2 测过 + checkstyle 0）。 - -**两份侦察存档（起步必读，别重炒）**： -- `plan-doc/tasks/iceberg-on-hms-delegation-findings-2026-07-07.md` — 委派为何 flip-gated + 跨插件 handoff 机制（对照 Trino）+ 具体设计草图 + 删 23 类/按列统计(建议 DROP)/`IcebergUtils` 抽 6 个纯 helper 的推迟清单。 -- `plan-doc/tasks/hms-event-pipeline-findings-2026-07-07.md` — 事件管道 flip-gated + 主障碍（事件做**结构性变更**非仅失效，中立 `ConnectorMetaInvalidator` 缺 register/rename 动词 + 分区 NAME 粒度缺口）+ 推荐 Model B（薄 fe-core 角色驱动 + 插件 pollOnce SPI）+ `ExternalMetaIdMgr` 可对 HMS 弃用（但 opcode 470 须保重放）。 +**⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步（读 / 写-拒 / schema-evolution / 缓存 / 跨加载器委派）只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`）。**删除排序最硬约束**：`datasource/hive|hudi|iceberg/` 的 ~90 个 HMS 支撑类删不掉，直到翻闸把消费者切到连接器路径（详见 execution plan §2.4/§3 + `tasks/iceberg-on-hms-delegation-findings-2026-07-07.md`）。 --- -# 🚀 下个 session 任务 = **实现翻闸前的"休眠期"连接器 SPI 补齐（设计文档第 4/5 章 阶段 1）** - -**本 session 已完成（recon + 设计 + 签字）**：针对"HMS 目录/表 retype 到通用类"的聚焦 code-grounded 侦察（`wf_e0586006-60f`：8 维读者 + 完整性/排序两个 critic，全 HEAD 核对）→ 落地权威设计文档 **`hms-cutover-retype-design-2026-07-07.md`（起步必读 #1）**。核心结论：翻闸是**一次原子、整库同时生效**的切换（靠类身份 `instanceof PluginDriven*` 分派，非运行时开关），前置一大批**休眠期**连接器补齐，末尾一次**循环依赖单元**的机械删除。**已 HEAD 证实**：`HiveConnectorProvider.getType()=="hms"` 已存在；`createSiblingConnector` 全树 0 处（跨插件兄弟 SPI 真缺）；GSON `registerSubtype` HMS 三处（catalog:366/db:447/table:471）；`buildTableInternal` 按**目录级** `SUPPORTS_MVCC_SNAPSHOT` 选基类/Mvcc 子类；`PhysicalPlanTranslator` `:808` PluginDriven 臂先于 `:818` HMS 臂命中。 - -**已签字决策（2026-07-07，设计文档 §6）**： -- **D1 统计 = 保留免扫描列统计**（新增 `ConnectorStatisticsOps.getColumnStatistics` SPI + Hive/Iceberg 各实现；表级行数照旧）——**列统计 SPI 进入休眠期补齐清单（§4.2a）**。 -- **D2 缓存 = 连接器自持**（退休 fe-core 的 Hive/hudi/iceberg 元数据缓存，删 route/loader 的 `instanceof` 站点；随翻闸集落地）。 -- **D3 混合 Iceberg 表 = 翻闸前先做好跨插件委派，无倒退**（兄弟连接器 SPI §4.4 是硬前置，滑期则翻闸顺延，不走 fail-loud 拒绝）。 -- **D4 打包**（翻闸时再定）：倾向两 PR（可回滚翻闸 + 机械删除）。 - -**休眠期补齐进行中（设计文档 §5 阶段 1，每子步一个独立休眠 commit，线上零影响）**： -- **✅ §4.1 DONE（commit `e63b03fb490`）**：Hive 连接器插件侧 Kerberos 认证器（翻闸后注入 context 是 NOOP/SIMPLE，加密 HMS 会静默降级——修法=只替换给 `ThriftHmsClient` 的 `AuthAction` 走插件 UGI `doAs`，**不套 iceberg 的 TcclPinningConnectorContext**，因 `ThriftHmsClient.doAs` 已 pin SYSTEM 类加载器且 hadoop+fe-kerberos 是 child-first→插件认证器与插件 RPC 同一 UGI 副本）。**决策 C 签字**：新建中立 `fe-connector-metastore-hms` 模块（HMS 解析器原只在 iceberg/paimon 元数据模块各一份；中立 API/SPI 已存在）。全绿 5/5 + 0 checkstyle。详见 `hms-cutover-kerberos-auth-impl-notes-2026-07-07.md`。**残留**：写链 HDFS Kerberos（`HiveWritePlanProvider`/committer 直用 `context.executeAuthenticated`，仍 NOOP）留到写翻闸补；真 KDC 冒烟归 R-002 硬门。 -- **✅ §4.2 分区列发射 DONE（`32b9526f689`）**：连接器在 `getTableSchema` 发射跨连接器约定的 `partition_columns` 属性（**复用**既有约定、非新增字段——paimon `PaimonConnectorMetadata:313`/iceberg `:443`/maxcompute `:163` 三家均如此发射，已 HEAD 证实；设计文档原"新增一等字段、优于字符串 hack"那句已证伪并在设计文档 §4.2 更正），并把 `string` 分区列 widen 到 **varchar(65533)**（=`ScalarType.MAX_VARCHAR_LENGTH`，**非 65535**——旧代码注释 `HMSExternalTable:835` 与本设计原文都把宽度写错；仅 `PrimitiveType.STRING` 强转，非 string 分区列/普通 string 数据列不动）。全绿（6 新测 + 34 metadata 套 + checkstyle 0 + import gate 净）。 -- **✅ §4.2 表描述符 DONE（`164b25d9c42`）**：`buildTableDescriptor` 端口旧 `toThrift`→`TTableType.HIVE_TABLE`+`THiveTable`（否则 fe-core `PluginDrivenExternalTable.toThrift` 退化到通用 `SCHEMA_TABLE` 描述符→BE 建 SchemaTableDescriptor 而非 HiveTableDescriptor）；照 iceberg HIVE_TABLE 分支，无 handle 故 base+sys 表共用。全绿（schema 套 7 测 + checkstyle 0）。 -- **✅ §4.2 按表能力接缝 DONE（`329853bf19c`）**：**设计决策已签字**——两个"按文件格式启用"的加速能力（Top-N 延迟 / 嵌套列裁剪）走**按表接缝**忠实复刻（不 blanket-declare、不放弃；对齐 Trino 逐表委派）。机制 = 连接器在 `getTableSchema` 发射 FE 内部 schema-property marker `ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY`（CSV of `ConnectorCapability.name()`；缓存友好、无额外 HMS 往返、format 留连接器侧，复用 `partition_columns` 模式），fe-core `PluginDrivenExternalTable.supportsTopNLazyMaterialize/supportsNestedColumnPrune` 经新 `hasScanCapability` = 连接器级 Set **OR** 按表 marker；marker key 从 SHOW CREATE 属性剔除。**Top-N 已发射**（orc/parquet 非 view 非 iceberg/hudi-on-HMS，精确 input-format 类匹配 = 旧 `SUPPORTED_HIVE_TOPN_LAZY_FILE_FORMATS`）。iceberg/paimon/jdbc 零改（永不发 marker）。全绿（连接器 178 测 + fe-core 32 测 + checkstyle 0 + import gate）。 - - ⛔**NESTED_PRUNE 遗留（硬门，下一子步前置验证）**：同接缝，连接器**暂未发射** marker——`SlotTypeReplacer.java:678-680` 把嵌套访问路径重写成 `Column.getUniqueId()`，iceberg 靠自带 field-id（`withUniqueId`）正确，**hive 连接器现在没设 field-id**；对 hive 开启前须端到端验证 hive orc/parquet 的 uniqueId 语义（BE 按 id 匹配嵌套叶子）否则读 NULL。旧 hive nested-prune 走同一 `SlotTypeReplacer` 且能工作→查旧 hive column uniqueId 赋值 vs SPI 路径 `ConnectorColumnConverter.convertColumns`；验证通过则连接器发 `SUPPORTS_NESTED_COLUMN_PRUNE` marker（parquet/orc、非 hudi），否则记差异 fail-loud。 -- **✅ §4.2 `getTableStatistics` 三层全做 DONE**（`eded05b05e6`+`8c6d247bb34`+`453a1926010`；用户拍板"三层全做"，不推迟）。架构裁决：**连接器只回传原始事实、不做 Doris 类型运算**（不 import fe-type），`dataSize/rowWidth` 除法归 fe-core（它有类型信息且此逻辑 connector-agnostic，对现有 iceberg/paimon/jdbc 零影响——它们 dataSize 恒 -1）。三层 = ① 精确 `numRows`（→spark numRows，仅当 numRows 键在但≤0，忠实复刻旧不对称）② `totalSize`（→spark totalSize，标准键缺才回退）作 dataSize，fe-core 除**全列宽** ③ 列文件估算（新 SPI `estimateDataSizeByListingFiles`，只普通 hive、采样 30 分区放大、best-effort→-1、内部 pin TCCL 因统计线程 fe-core 不 pin），fe-core 除**非分区列宽**（旧 getRowCountFromFileList 语义）。**复审修正**（4 维对抗 + 逐条反驳核验 `wf_e0f3a369-77d`）：两层商截 0 须塌成 UNKNOWN（层2→穿透层3、层3→UNKNOWN），非报"空表 0 行"。**已签字裁决（本 session）**：采样数不再随运行时 `hive_stats_partition_sample_size` 变（连接器持默认 30；开关 `enable_get_row_count_from_file_list` 仍 fe-core 侧生效）；malformed `numRows` 防御式恢复 spark 值（比旧 throw 更稳，已文档化 deviation）；文件列举非递归顶层（与扫描侧一致）；精确 view-skip 待 view SPI（今 view 无可列位置→-1）。 -- **✅ §4.2 视图 SPI DONE（`6676bf365cf`）**（本轮侦察 `wf_ecb9c379-9bb`：4 维读者 + 完整性 critic，全 HEAD 核对）。连接器侧四件套：`HmsTableInfo`+`ThriftHmsClient.convertTable` 补 `viewOriginalText`/`viewExpandedText`（原被丢弃）；`viewExists`=**legacy `HMSExternalTable.isView` 谓词=view 文本存在**（非 tableType——base 表须返 false，否则 `PluginDrivenExternalCatalog.dropTable` **无条件** viewExists→dropView 会误路由 base 表 DROP）；`getViewDefinition`=忠实复刻 `getViewText`（expanded-first→跳 `/* Presto View */` sentinel→`viewOriginalText` base64/Gson 解 `originalSql`→任何失败回退 raw；dialect 占位不被读；列走 `buildColumns`，**fe-core 视图列只来自此处**——`initSchema` isView 早返、不调 getTableSchema）；`dropView`=复用 `HmsClient.dropTable`（hive 无独立 drop-view）、`DorisConnectorException` 归一化好让 fe-core 转 `DdlException`；`listViewNames` **保持 SPI 默认 empty**（hive `listTableNames`=getAllTables 已含视图，fe-core SHOW TABLES 合并是无去重 `addAll`，非空会双列；与 iceberg 相反）；`HiveConnector.getCapabilities`=仅 `SUPPORTS_VIEW`（其余连接器级能力各自子步、MVCC 故意不声明）。全绿（7 视图测 + hive 157 + hms 150 套无回归 + checkstyle 0 + import gate 净）。**留到翻闸的耦合（休眠期不暴露，勿现在动）见文末**。 -- **✅ §4.2 file-format serde 修正 DONE（`c320df27712`）**（侦察 `wf_470a2e51-373`：4 维 + critic，含 BE C++ 读取器核对）。修**两个**读正确性 bug:(1) `HiveFileFormat.detect` 原 inputFormat-first→标准 JSON 表(inputFormat=TextInputFormat)被当 CSV 读;(2) 默认 hive 文本表(LazySimpleSerDe)本该走 BE `FORMAT_TEXT` 读取器(懂 hive 集合/map 分隔、`\N` 空、转义),却被塌缩成 `FORMAT_CSV_PLAIN`(BE 确有两个不同读取器类)。修法=**两段式、文本类 serde-authoritative**(inputFormat substring 分 parquet/orc/text-file;text 类按**精确 FQCN** serde 分 json/csv/hive-text),新增 CSV 枚举;复刻 OpenX `read_hive_json_in_one_column`(首列 string→单列 CSV,否则 fail-loud;会话开关经 `getSessionProperties` 读,首列类型 plumb 进 `HiveTableHandle`);未知 inputFormat/serde **fail-loud throw**(用户签字,与线上一致)。**fe-core `PluginDrivenScanNode.mapFileFormatType` 拆 `"text"→FORMAT_TEXT`、`"csv"→FORMAT_CSV_PLAIN`**(中立既有 token、无 `if(engine)`、hive 是唯一发射者、休眠;须与连接器 csv 发射**同一 commit**否则 OpenCSV 回退)。`HiveTextProperties` 补 legacy hive-2 JsonSerDe + serde2 MultiDelimit FQCN。全绿(HiveFileFormatTest 真值表 + hive 153/hms 150 无回归 + fe-core 编译 + checkstyle 0 + import gate 净)。 -- **✅ §4.2 `listPartitions`/`listPartitionNames` DONE（`68e9fc8835b` + 前置修复 `3540d1327b6`）**（本轮侦察+3维对抗 review `wf_c25cc43a-43a`：legacy-parity/flip-consumer/code-edge 三镜头一致收敛，全 HEAD 核对）。**已签字决策（用户 2026-07-07）= `lastModifiedMillis` 填 -1，不付 per-partition 往返**：legacy 热分区裁剪路径（`HiveExternalMetaCache.loadPartitionValues`）只列**分区名**（无 `get_partitions_by_names` 往返），`transient_lastDdlTime` 只在 MTMV 刷新时取；`listPartitions` 跑在每次分区 hive 查询的 materialize 路径上，填 lastModified 会热路径倒退。故连接器 `listPartitions`/`listPartitionNames` 共用 `collectPartitionNames`（names-only，-1=全量），`lastModifiedMillis`/rowCount/size/fileCount 全留 UNKNOWN(-1)（对齐 maxcompute；hive 不声明 `SUPPORTS_PARTITION_STATS`→SHOW PARTITIONS 走 names-only 臂，-1 统计不外露）；filter 忽略（legacy 全量物化+FE 侧裁剪）；分区值经 `HiveWriteUtils.toPartitionValues`（byte-faithful port，含 unescape）解、按 remote 名 keying（`PluginDrivenExternalTable.getNameToPartitionItems` 按 remote 名读回）。**per-partition MTMV 新鲜度留到 §4.3 MVCC/MTMV 步**（届时像 legacy 只在刷新时取完整分区）。**前置修复（独立 commit `3540d1327b6`）**：`ThriftHmsClient.listPartitionNames` 原把 `maxParts<=0` 塌成 `Short.MAX_VALUE(32767)`→>32767 分区表**静默截断**（3 镜头一致标记的唯一真缺陷；且已默默毁掉 hudi/hive-write 传 -1 的"无截断"意图），改真 `(short)-1` 无界（正数透传、>32767 窄化成负 short 亦无界=保留 100000-callers 现状）。全绿（fe-connector-hive 160 + 7 新分区测 + fe-connector-hms 52 + 3 新 maxParts 测 + checkstyle 0 + import gate 净）。 - - ⚠**遗留（休眠期不动，翻闸/§4.3 处理）**：`getPartitionSnapshot` 对 hive 空 pin 返 `MTMVTimestampSnapshot(-1)`（两次 -1 相等→不会误判变更，但也不检测真实分区改动，须 §4.3 供真 millis）；`listPartitionValues`（`partition_values()` TVF）连接器**未实现**（无 fe-core 消费者，归 §4.6 TVF 门）。 -- **✅ §4.2 `getCapabilities` 连接器级 DONE（`96953cd8eec`）**（3 维对抗 review `wf_8813de68-7f9`：declared-preconditions/completeness-omissions/mvcc-heterogeneity，`fix-then-ship`，两 minor 已按建议改注释）。声明 = `SUPPORTS_VIEW` + `SUPPORTS_COLUMN_AUTO_ANALYZE`（legacy `StatisticsUtil.supportAutoAnalyze` 收 HMS dlaType HIVE/ICEBERG 进后台 FULL 自动分析）+ `SUPPORTS_METADATA_PRELOAD`（legacy `HMSExternalTable.supportsExternalMetadataPreload()`=true；只管元数据预热、不触发 `supportsLatestSnapshotPreload`——插件用 false 默认，hive 不做 snapshot 预热）。三者休眠期零消费（hive 非 PluginDriven）。**⚠ 纠正交接旧错**：`SUPPORTS_MVCC_SNAPSHOT` **不是"禁"而是"推迟到 §4.3 MVCC 步"**——设计文档 §3 明确混合 HMS 目录**必须**声明它（iceberg/hudi-on-HMS 是 MvccTable；GSON 单行把 `HMSExternalTable`→`PluginDrivenMvccExternalTable`，`buildTableInternal` 按**目录级** MVCC 能力选 Mvcc 子类，`PluginDrivenExternalDatabase:56` 已 HEAD 证实）；现在裸声明会让 plain-hive 拿到无新鲜度的空快照，故与 MVCC 机件（freshness-aware `getTableSnapshot`+空 pin+兄弟委派）同步落。同理 `SUPPORTS_SHOW_CREATE_DDL` 推迟（须连接器先发 `show.location` + 定通用 vs hive-specific 渲染）；`SUPPORTS_NESTED_COLUMN_PRUNE` 推迟（须 hive 嵌套 field-id，否则 BE 读 NULL 叶——与前"按表能力接缝"步的 ⛔ 硬门同源）。全绿（fe-connector-hive 164 + 4 新能力测 + checkstyle 0 + import gate 净）。 - - ⚠**翻闸期残留（记账，勿现在动）**：`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 → 翻闸时**也**会收 hudi-on-HMS 进自动分析（legacy 排除 HUDL），且**无按表 escape**（不像 TOPN 有 marker）——须在 iceberg/hudi 委派步按 handle 门控或显式接受。`supportsLatestSnapshotPreload` 对 iceberg/hudi-on-HMS legacy=true、插件默认 false → 翻闸时 iceberg-on-HMS 丢 snapshot 预热（纯性能、归 §4.3/§4.4）。 -- **✅ §4.2 `HiveTableFormatDetector` 平价 DONE（`191449c7bd6`）**（3 维对抗 review `wf_9a5db88a-956`：classification-parity/fail-loud-blast-radius/exception-catch-parity，verdict `ship`，全 HEAD 核对）。①补 3 个 LZO text 格式→连接器 `SUPPORTED_HIVE_INPUT_FORMATS` 与 legacy `SUPPORTED_HIVE_FILE_FORMATS` **逐条相等（7 格式）**；②`getTableHandle` 检出 UNKNOWN 且非视图→ **fail-loud** `DorisConnectorException`（"Unsupported hive input format"/"input format is null"，对齐 legacy `supportedHiveTable` throw，且 throw 时机=懒惰按表访问，SHOW TABLES 走 `listTableNames` 不受影响）；③视图短路（`isViewTable`=view 文本在=legacy 谓词，视图 handle 保 UNKNOWN 类型不被拒、由视图 SPI 服务）。detect() 仍是纯分类器（不 throw），供 TopN 门/detectFormatType 复用。全绿（fe-connector-hive 171 + 7 新测 + checkstyle 0 + import gate 净）。 - - ⚠**翻闸期残留（两处，均**非本步引入、iceberg 已同款**，记账留翻闸的 fe-core 读路径处理，勿现在改）**：(1) **异常分类**——查询不支持格式的 hive 表抛 `DorisConnectorException`(RuntimeException)，fe-core `ConnectProcessor.handleQueryException` 无对应臂→归类 `ERR_UNKNOWN_ERROR`("maybe palo bug") 而非 legacy 的 `ERR_NOT_SUPPORTED_YET`；须翻闸时在 fe-core 读路径把 `DorisConnectorException` 映射到合适用户错误（跨连接器通用，iceberg `IcebergTypeMapping` 已同款）。(2) **存在性检查 fail-loud**——`PluginDrivenExternalCatalog.tableExist`/`CREATE IF NOT EXISTS` 经 `getTableHandle().isPresent()` 判存在，对"已存在但不支持格式的非视图表"会 throw（legacy `client.tableExists` 只回布尔不查格式）；是 PluginDriven 通用契约（iceberg/paimon 同）、且是有意的 fail-loud 方向，翻闸时定夺是否要免格式检查的存在性路径。 -- **✅ §4.2a 列统计 fast-path SPI DONE（`cbf9526f776`）**（3 维对抗 review `wf_144b6d15-77f`：math-byte-parity/guards+flow/architecture，verdict `ship` 零确认缺陷，全 HEAD 核对）。架构=**连接器回传原始事实、fe-core 做 Doris 类型运算**（沿用 getTableStatistics 先例）：`convertColumnStatistics` 按 HMS variant 抽 ndv/nulls/avgColLen（string 才有 avgColLen，boolean/binary/timestamp→0/0 与 legacy 未处理分支同）；hive `getColumnStatistics` 仅 HIVE 表、需正 `numRows`（**无 spark 回退**，与表级 size 分支不对称、复刻 legacy）；fe-core `toColumnStatistic` 做 size math（string→`round(avgColLen*count)`、否则→`count*slotSize`；`avgSizeByte=dataSize/count`；min/max=builder -inf/+inf 默认）。guard 分歧仅**更安全**地倒向 empty→ANALYZE（负/畸形 numRows）。全绿（hms 4+hive 5+fe-core 4 新测；hms 56/hive 176 套 + checkstyle 0 + import gate 净）。 - - ⚠**本步范围=查询规划列统计 fast-path**；**两处后续**（非本步）：①分区级列统计（`getHivePartitionColumnStats`→`HMSAnalysisTask` 的 ANALYZE 元数据 fast-path，设计 §4.2a 标"revisit `ExternalAnalysisTask` 是否走新 SPI vs SQL-only FULL")；②iceberg 连接器 `getIcebergColumnStats` port（`enable_fetch_iceberg_stats` 门，属 iceberg 连接器/§4.4，非 hive）。 -- **✅ §4.3 MVCC/MTMV 新鲜度 DONE（`3d784673ca4`）**（侦察 `wf_c50495bc-cd9` 6维+critic + 对抗复审 `wf_5ffdf7fb-1dc` 3镜头+verify，全 HEAD 核对）。声明 `SUPPORTS_MVCC_SNAPSHOT`（getCapabilities 步推迟项）+ freshness-kind-aware `getTableSnapshot`/`getPartitionSnapshot` + hive 空 pin + 真 millis（收 listPartitions 步的 -1 遗留）。**关键设计裁决（复审揪出后改正）**：`PluginDrivenMvccExternalTable` **不是休眠**——paimon/iceberg(P5/P6 已上线、声明 MVCC)**实时**走它；故新鲜度探针**必须**用放在 pin 上的 `lastModifiedFreshness` 标志门控（fe-core 读它已持有的 pin，零额外解析），否则给 paimon/iceberg 的 MTMV 每分区加一次 `getTableHandle` 往返+新抛错面（透明改写时可炸）。两个 `verify(never)` 回归测试锁死"snapshot-id 连接器不触发探针"。 - - ⚠**范围=MTMV 表/分区快照新鲜度**；**遗留（翻闸前须补/记账，休眠期不动）**： - 1. **`getNewestUpdateVersionOrTime`（字典自动刷新探针）** 对 hive 仍走 LIST-path max(pin -1 过滤)→0 常量→字典不刷新；须同根因按需取 max 分区时间（未分区语义与 getTableSnapshot 略异=0 非 lastDdlTime），且探针可能高频→须连接器缓存(D2)后再落，故未并入本步。**翻闸前须补**。 - 2. **系统表**（`$partitions` TVF vs native + iceberg-on-HMS `$snapshots`）：用户 2026-07-08 拍板**推迟到整体切换**与跨插件委派一起做（本步只做 MVCC/新鲜度；plugin 系统表只出 native，服务 hive $partitions 须新 fe-core native/TVF 路由缝，无线上消费者故顺延）。 - 3. **flip-boundary 一次性 parity 微差（复审记账，均可接受、勿现在改）**：MTMVMaxTimestampSnapshot 的 name 用**远端**表名/分区键大小写（连接器自然值），legacy 用本地 Doris 名/列——仅 name-mapping/大小写异常的目录在翻闸后首刷一次性误刷新；`renderPartitionName` arity 不匹配 Math.min 静默（legacy checkState throw，HMS 不变式下不可达）；malformed `transient_lastDdlTime` throw=legacy 平价（非回归）；全分区无 modtime→(tableName,0) 比 legacy NPE 更安全；dropped-catalog 降级为 MTMVSnapshotIdSnapshot(-1)=既有降级哲学。 - 4. **vanished-partition** 已按复审改为 on-demand empty→fe-core throw "can not find partition"（parity），非返 0。 - 5. **iceberg/hudi-on-HMS 的 snapshot-id 新鲜度**靠 §4.4 兄弟委派（gateway `beginQuerySnapshot` 对 iceberg handle 委派→非 last-modified pin；per-handle pin 标志设计天然支持混合目录）。 -- **✅ §4.4a 兄弟连接器 SPI 接缝 DONE（`45dcca32fa8`）**（recon `wf_da155752-ebb` 6维+2 critic + 对抗复审 `wf_caef8bf7-ec2` 3镜头，全 HEAD 核对；分解权威文档 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`，起步必读）。新 `ConnectorContext.createSiblingConnector(String catalogType, Map props)` 默认 **`return null`**（对齐所有其它 SPI 默认的安全默认约定——**非** throw，更正设计 §4.4 的 UnsupportedOperationException）；fe-core `DefaultConnectorContext` 覆写 = `ConnectorFactory.createConnector(catalogType, props, this)`（传 `this` → 兄弟连接器复用本目录 id/auth/storage，其 hms-flavor HiveCatalog 名默认取 `context.getCatalogName()`，故共享 context 承重；fe-core 保持 connector-agnostic、零属性解析——props 由网关合成）；iceberg+paimon `TcclPinningConnectorContext` 装饰器补委派该新方法到 raw context（保持"穷尽 pass-through"契约为真——兄弟自套 TCCL/auth pin，须收未包裹 context 避免双重 pin）。唯一改引擎的一步、后面所有插件步的地基。全绿（spi 3 + fe-core 3 + iceberg 5 + paimon 4 测 + 4 模块 checkstyle 0 + import gate 净）。**⚠关键设计更正（更正设计 §4.4）**：设计的"网关包装句柄 `{format, delegate}`"方案**已证死**——`PluginDrivenScanNode` 选执行器(:209)与执行扫描(:966)用**同一句柄对象**、中间无拆包缝，`IcebergScanPlanProvider` 无条件强转 `(IcebergTableHandle)`(:319/364/468/1009)→包装句柄必 CCE；**改为** `getTableHandle` 对 iceberg 返**原始外来句柄**（iceberg 强转成功）、网关一律用 `instanceof HiveTableHandle`(自有 hive-loader 类型)→hive、否则转发兄弟，**永不强转外来句柄**，零新 SPI。 -- **✅ §4.4 S1 属性合成纯函数 DONE（`d971410af62`）**：`IcebergSiblingProperties.synthesize(gatewayCatalogProps)`=**整份目录属性逐字复制+注入字面量 `iceberg.catalog.type=hms`**（per-catalog、非 per-table，无需 HmsTableInfo；字面量硬编码因 iceberg 常量 child-first 不可见）。**copy-all（非选择性子集）经对抗核验安全**：与原生 `type=iceberg,iceberg.catalog.type=hms` 目录喂给 IcebergConnector 的**完全同形**（fe-core `createConnectorFromProperties` 也传整份目录属性）、iceberg `create()` 零属性校验→hive 侧 delta 键只被忽略不被误用（`warehouse` 无害=HMS flavor 不从它派生 fs.defaultFS、存量表用自存 location；唯一非缺陷=兄弟用 iceberg 默认缓存 TTL 与原生同）。不改输入、覆盖任何 stray flavor。全绿（4 测+checkstyle+import gate）。 -- **✅ §4.4 S2 兄弟持有/懒建/close 转发 DONE（`119b4fca64b`）**（对抗 review `wf_c17ad553-53a`：dormancy-parity/classloader-cce/concurrency-lifecycle/contract-test-intent 4 维 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 新增包级 `getOrCreateIcebergSibling()`：volatile 双检（复刻本文件 `getOrCreateClient()` 惯用法）经 `context.createSiblingConnector("iceberg", IcebergSiblingProperties.synthesize(properties))` 懒建；**单例 per-gateway**（非 per-table，因 IcebergConnector 持 per-catalog 缓存 latestSnapshot/manifest/deleteStash，per-op 会碎片化）；**仅以 parent-first `Connector` 接口持有、永不强转**（外来句柄类型 hive 加载器不可见，强转必 CCE）；插件缺失返 null→**fail-loud** `DorisConnectorException`（含目录名）且**失败不 memoize**（字段留 null 下次重试）。`close()` 追加转发兄弟并置 null（引擎只关主连接器，网关自持兄弟生命周期）、未建时 no-op。全绿（fe-connector-hive 194 测 + 6 新 `HiveConnectorSiblingTest` + checkstyle 0 + import gate 净；HEAD 证实 hms 不在 SPI_READY_TYPES=休眠成立）。⚠**复审记账（refuted，勿改）**：`close()` 在 `hmsClient.close()` 之后转发兄弟，若前者抛 IOException 理论会漏关兄弟——但唯一生产 `HmsClient`=`ThriftHmsClient` 的 `close()` 结构性吞异常（`clientPool.close()` 包在 catch 内）实际不抛，非真缺陷，保持与现有 close 惯用法一致。 -- **✅ §4.4 S3 元数据按句柄守卫转发 DONE（`b80c614e5b4`）**（先 code-grounded recon `wf_2676a99b-efa`：全 67 方法面枚举分类 + 完整性 critic 逐条 HEAD 核对；后对抗 review `wf_e393f9ec-b68`：hive-parity/forward-correctness/completeness-scope/wiring-dormancy 4 维 + verify，**0 findings**）。`HiveConnectorMetadata` 每 handle 方法 `if(!(handle instanceof HiveTableHandle)) return siblingMetadata(session).<同名>(…)`，否则跑原 hive body 字节不变；**外来句柄永不强转**。接线=`HiveConnector.getMetadata` 传 `this::getOrCreateIcebergSibling`（`Supplier`），`siblingMetadata(session)` **每次调用现取**（fe-core 亦 per-op 取 metadata，重状态在兄弟连接器上），3-arg 构造器装 fail-loud `NO_ICEBERG_SIBLING`（仅 hive-only 测用）。**21 方法两集**：①13 个 hive 已覆写方法加纯前缀守卫（getTableSchema[2]/getColumnHandles/get{Table,Column}Statistics/estimateDataSize/applyFilter/listPartitions[Names]/beginQuerySnapshot/get{Table,Partition}Freshness/dropTable/truncateTable）；②8 个 hive 缺席但 iceberg 有的新增守卫-转发覆写（schema-at-snapshot / getMvccPartitionView / resolveTimeTravel / applySnapshot / applyRewriteFileScope / applyTopnLazyMaterialization / listSupportedSysTables / getSysTableHandle）——不补则外来表拿 hive SPI 默认=**静默错答**（空 MVCC/时间旅行/系统表 或 未串快照 pin），hive 分支复刻 SPI 默认。`beginQuerySnapshot` 与两 freshness 同步分流（其 isLastModifiedFreshness 标志门控 freshness 是否被读）；handle-out（applyFilter/apply*/getSysTableHandle）**原样回传兄弟句柄**（rewrap 会毒化下游 planScan 强转）。**recon 校正**：`buildTableDescriptor` **无句柄→归 S6**（分解文档 §3 set-a 误列，已证）；`applyProjection/applyLimit/listPartitionValues` 两边皆未覆写=惰性、**故意不守卫**；ALTER-DDL 全家（addColumn/…/create-drop Branch|Tag / partition-field / renameTable）+ `validateRowLevelDmlMode`/`validateStaticPartitionColumns` 带句柄但 hive 未覆写、属**写切换**（另步，非本步）。全绿（fe-connector-hive 197 测 + 3 新 `HiveConnectorMetadataSiblingDelegationTest`〔含转发方法集完整性锁〕 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES 且 S4 未做→getTableHandle 仍返 HiveTableHandle→守卫今皆 no-op、休眠）。 -- **✅ §4.4 S4 `getTableHandle` iceberg 分流 DONE（`e7c28a39e69`）**（对抗复审 `wf_be91f274-b9b`：divert-correctness/classloader-cce/dormancy-parity/test-intent 4 镜头 + 逐条 verify，**生产码 0 缺陷**，全 HEAD 核对）。`HiveConnectorMetadata.getTableHandle` 在 `HiveTableFormatDetector.detect(tableInfo)==ICEBERG` 时**返 `siblingMetadata(session).getTableHandle(session,db,table)`（原始外来 iceberg 句柄）**、非盖 ICEBERG 的 HiveTableHandle——点亮 S3 全部 `instanceof HiveTableHandle` 守卫（外来句柄开始真流入）。分流放在 fail-loud UNKNOWN 检查**之前**（iceberg 早返，永不触 throw）；detect 内 iceberg(`table_type=ICEBERG`)先于 hudi/hive 判、故 iceberg 优先。**兄弟返 empty 原样透传**（表消失/iceberg 加载不了→不伪造 hive 句柄，兄弟对 iceberg 存在性权威）。**HUDI 不分流**（仍建 HUDI 章 HiveTableHandle、字节不变，hudi 委派属后续子步）；plain-hive/view 不动。网关自身 `hmsClient.tableExists+getTable`（detect 必需）→ 兄弟再自查一次=legacy 双载平价，非缺陷。新 `HiveConnectorMetadataTableHandleDivertTest`(6 测)钉合约：iceberg 返兄弟句柄`assertSame`且非 HiveTableHandle、empty 透传**证经兄弟转发**(getTableHandleCalls==1，复审 Rule-9 补强)、hive/hudi 零咨询兄弟、缺表短路 empty、无兄弟 fail-loud。全绿（fe-connector-hive 203 测〔+6 新〕 + hms 56 + checkstyle 0 + import gate 净；HEAD 证 hms 不在 SPI_READY_TYPES=休眠成立）。 -- **✅ §4.4 S5 扫描提供者网关分流 DONE（`bb39802751b`）**（对抗复审 `wf_6fc6999e-b80`：route-correctness/classloader-tccl/dormancy-parity/test-intent 4 镜头 + verify，**0 确认缺陷**，全 HEAD 核对）。`HiveConnector` 覆写 per-table 接缝 `getScanPlanProvider(ConnectorTableHandle handle)`——`if(handle instanceof HiveTableHandle) return getScanPlanProvider(); return getOrCreateIcebergSibling().getScanPlanProvider(handle);`（负判自有 hive-loader 类型=loader-safe；外来句柄**原样透传兄弟 per-handle overload、永不强转**）。返回的兄弟 provider 是 iceberg-loader 所建→`PluginDrivenScanNode.onPluginClassLoader`(:516)按 `provider.getClass().getClassLoader()` **自动 pin 扫描侧 TCCL 到 iceberg loader、白拿**（无须本步 pin）。`currentHandle` 构造器由已解析表句柄赋值(:172)、pushdown 变异但从不置 null→路由非 hive 句柄不会误分流。HUDI 章 HiveTableHandle 仍是 HiveTableHandle→留 hive 扫描路径（hudi 委派后续）。SPI 默认(`Connector.getScanPlanProvider(handle)`默认委派 no-arg)→本连接器改前对任意句柄都走 no-arg，改后非 hive 句柄才分流；hms 未 live 故零现存路径受影响。全绿（fe-connector-hive 207 测〔+4 新 `HiveConnectorScanProviderDivertTest`〕 + checkstyle 0 + import gate 净）。**测试隔离**：hive/hudi 例 stub no-arg `getScanPlanProvider()`（真 HmsClient 需 HiveConf、单测类路径无→`getScanNodeProperties` 套另测），只钉路由；foreign/fail-loud 例走真 override。 -- **✅ §4.4 S6 按名方法/能力残留 = 零代码收官（recon `wf_6ac0e047-768`：4 维读者 + 完整性/正确性 critic 逐条 HEAD 核对；权威结论 = `hms-s6-name-based-divert-findings-2026-07-08.md`，起步必读）。校正分解文档 §3 S6 草案**——所有无句柄**读**方法**均 NO_DIVERT**，S6 无连接器代码改动，只落 findings 文档 + 本残留清单。理由：`buildTableDescriptor` 字节等价（兄弟恒 hms-flavor→同 `HIVE_TABLE`+`THiveTable`；legacy iceberg-on-HMS 亦发 `HIVE_TABLE`；分流只白加一次 `getTable`）；视图族（`viewExists`/`getViewDefinition`/`dropView`）legacy 平价（iceberg 表 `viewExists`=false 留 getTableHandle 分流路径；iceberg-on-HMS 视图存 hive 方言 SQL 于 HMS view text、走 hive 视图路=legacy 平价）**且无句柄不可判别**（`detect` 对视图返 UNKNOWN——iceberg 视图 `table_type='iceberg-view'`≠`'ICEBERG'`）；`listViewNames` 保持不覆写（分流会双列视图=回归）；`getTableComment` 不覆写=SPI 默认 `""`=legacy `getComment()` 恒 `""` 精确平价（分流会显示 legacy 从不显示的 comment=回归）；db/目录级方法（listDatabaseNames/databaseExists/listTableNames/getProperties/create[Database]/getPrimaryKeys/getDatabase）format-agnostic。**⇒ 至此 §4.4 读委派脊柱（S0–S6）全部收官**。 - - ⚠**翻闸残留（findings 文档已详列，休眠期不动）**：①`dropDatabase` force 级联对 iceberg 表裸 `hmsClient.dropTable`（=legacy 平价、无 purge，只泄漏 iceberg 文件）→**写/DDL 委派步**；②`createTable` 引擎路由（新表无可 detect）→写/DDL 步；③`beginTransaction`/`getWritePlanProvider` **无 per-handle overload**→写委派步（新 SPI overload + ALTER-DDL/写校验，见分解文档 §6）；④`SUPPORTS_COLUMN_AUTO_ANALYZE` 连接器级 over-admit hudi（已注于 `HiveConnector.java:139-144`，cost-only、无 per-handle escape）；⑤`supportsLatestSnapshotPreload` under-admit iceberg/hudi（fe-core TableIf 默认 false vs legacy true，latency-only、非 enum-backed）；⑥**first-class iceberg-on-HMS 视图 = 翻闸显式非目标**（legacy 从不经 HMS 目录服务 iceberg ViewCatalog 视图；要做属新特性）。 -- **§4.4 写委派权威分解 = `hms-write-delegation-decomposition-2026-07-08.md`（起步必读，recon `wf_f508ac0e-8ec` 7 维 + 完整性/分解 critic）。拆成 W1–W6 六个休眠小步，三种委派形态**：Shape A=写/存储过程 getter 加 per-handle 重载（W2/W3/W5）；Shape B=ConnectorMetadata 句柄守卫转发（W1，零新 SPI，**已 DONE**）；Shape C=事务按句柄选择（W4，唯一无先例的新形态）。**签字决策（用户 2026-07-08）：iceberg-on-HMS 升级为完整 native-iceberg 写/DELETE/MERGE/EXECUTE/ALTER 能力**（旧行为全拒=有意升级，非字节平价；对齐 Trino 按句柄委派）。**e2e 硬要求（用户 2026-07-08，memory `hms-iceberg-delegation-needs-e2e`）：每项新增 iceberg 能力（写/ALTER/procedure）最终都须补 e2e 回归（休眠单测只证路由、证不了跨加载器 cast/commit/worker-pool）；实现完成后择机统一补。** -- **✅ W2 DONE（`84890fe48d3`）**：`Connector` 加 `getWritePlanProvider(ConnectorTableHandle)` 默认重载（默认委派无参版=对齐 `getScanPlanProvider(handle)` 先例，对 6 个现存连接器行为不变）；`HiveConnector` 覆写按 `instanceof HiveTableHandle` 分流兄弟（外来句柄永不强转，hudi 保 HiveTableHandle 留 hive）；fe-core 4 处 provider 取用点把已解析的 `getTableHandle` 上提、改调 per-handle 重载（`PhysicalPlanTranslator` DELETE/MERGE+INSERT、`PhysicalIcebergMergeSink`、`PluginDrivenExternalTable.fetchSyntheticWriteColumns`；connector-agnostic 合铁律）。**对抗复审揪出 1 blocker**：4 个 fe-core mock 测只 stub 无参 `getWritePlanProvider()`，Mockito mock 不跑 default→per-handle 返 null→测挂（memory `catalog-spi-fe-core-test-infra`），已补 stub per-handle 重载（照 scan seam）；1 minor=`fetchSyntheticWriteColumns` reorder 对只读连接器（show-hidden 门）多解析一次 handle，注释已改准。全绿（新 `HiveConnectorWriteProviderDivertTest` 4 测；hive 213 + fe-core 触及套 42 测 + checkstyle 0〔api/hive/fe-core〕 + import gate 净）。 -- **✅ W4 DONE（`5a41e4cc562`）= 事务按句柄（Shape C，唯一无先例的新形态）**：`ConnectorWriteOps` 加 `beginTransaction(session, handle)` 默认重载（委派无参版=对齐 `getWritePlanProvider(handle)` 先例；6 现存连接器继承默认→各自无参 `beginTransaction`，字节不变）；`HiveConnectorMetadata` 覆写守卫转发 foreign 句柄→兄弟（得 `IcebergConnectorTransaction`）、hive 句柄→落连接器自有 `HiveConnectorTransaction`——**必须对称**（hive/iceberg 写计划把 session-bound txn 分别强转不同具体类型，"总转发"/"总 hive"都炸另一侧）。fe-core 把已解析写目标句柄穿进**两个**开事务点（否则 EXECUTE 路径留活 CCE）：`PluginDrivenInsertExecutor`（INSERT/OVERWRITE/DELETE/MERGE 单一漏斗）经新 fail-loud `PluginDrivenExternalTable.resolveWriteTargetHandle()`、`ConnectorRewriteDriver`（EXECUTE，已持句柄）；connector-agnostic，**绝不给网关传 null 句柄**（fail-loud，故不会把普通写误路由兄弟）。侦察(`wf_a6e430b0-3e9`)+4维对抗复审(`wf_6af16e19-0cc`)+verify（CCE 净、休眠/现存连接器字节平价、路由测 mutation-tight）；2 确认发现均测试覆盖缺口、已补（`PluginDrivenExternalTableTest` fail-loud 体测→`orElse(null)` 降级现挂；`PluginDrivenInsertExecutorTest` 句柄穿透 assertSame→null 二参 mutant 现挂）。全绿（fe-connector-hive 7〔+2 divert〕+ fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30〔+2〕/ ConnectorRewriteDriverTest 3 + checkstyle 0〔api/hive/fe-core〕+ import gate 净）。**W4 复审记账（residual，休眠期不动）**：①**once-per-statement `getTableHandle` 往返**——每笔 plugin 写开事务时新增一次解析（paimon/iceberg 缓存命中、jdbc/maxcompute 未缓存），有意接受、非 per-row；②fail-loud orElseThrow 开事务时重查——并发 DROP/瞬时元数据错稍扩失败窗（paimon getTableHandle 吞异常成 empty→通用文案盖根因），非正确性回退；③W6 写路径 TCCL pin=**开事务不需**（`IcebergConnectorTransaction` 构造只赋值、SDK/反射推迟到 beginWrite/commit），commit 侧 pin 归 W6；④hudi-on-HMS 保 HiveTableHandle→hive txn（未变差，后续 hudi 委派步）；⑤e2e owed（休眠单测只证路由）。 -- **✅ W5 DONE（`8aee62bab20`）= 存储过程按句柄**（详见上 🎯 段 W5 条目）。`Connector.getProcedureOps(handle)` per-handle 默认 + `HiveConnector` 分流（hive/hudi→继承 null、foreign→兄弟永不强转）+ fe-core `ConnectorExecuteAction` reorder（句柄解析上提、改调 per-handle）。net-new 升级、reorder 无测钉旧序、顺带修陈旧 javadoc。4 维对抗复审 0 blocker/major/minor + 1 nit（已修）。**⇒ §4.4 写/procedure/DDL 委派休眠脊柱 W1–W5 全部收官。** - - ⚠**W5 复审记账（休眠期不动）**：①**paimon/无-procedure 连接器 EXECUTE 现多一次 `getTableHandle` 往返**（reorder 先解析句柄再判 null-ops；happy-path iceberg 无新增往返，只错误路径先报 "Table not found"）——有意接受、非正确性回退；②`ExecuteActionFactory.getSupportedActions` 留无参 getter（SHOW 发现，无 live caller；翻闸对 hms 网关会返 hive 连接器级空——handle-aware 变体延后，无消费者故无害）；③e2e owed（休眠单测只证路由；随其它 iceberg-on-HMS 能力翻闸期统一补，memory `hms-iceberg-delegation-needs-e2e`）。 -- **✅ 写链收尾（§4.5）休眠四步 WC1–WC4 全部收官（本 session：WC2/WC3/WC4；WC1 上 session）。权威拆解 = `hms-write-chain-decomposition-2026-07-08.md`**。核心 = 三类休眠 taxonomy（**A**=翻闸前不可达、**B**=值恒等、**C**=活跑靠 no-op SPI 默认——**C 类须补现存连接器无回归测**，非仅休眠 hive 单测）+ 风险升序 4 步： - - **✅ WC1 DONE（`6c01805109c`）= 搬 `BIND_BROKER_NAME` → `BrokerProperties.BIND_BROKER_NAME_KEY`**（B 值恒等：两字面量均 `"broker.name"`；`private→public` 成单一真源、generic base 不再依赖 HMS 子类；`.get()` 保**大小写敏感、不 harmonize**；删 HMS 副本〔HEAD 证单引用〕）。全绿（BrokerPropertiesTest 6〔+1 常量真源 pin〕 + HiveScanNodeTest 6 + checkstyle 0）。 - - **✅ WC2 DONE（`650658b4ce8`）= engine-map 三 case（A 不可达、极低风险，一 commit）**：`CreateTableInfo.pluginCatalogTypeToEngine` 加 `case "hms"→ENGINE_HIVE`（CREATE 引擎；镜像 legacy paddingEngineName/checkEngineWithCatalog 的 `instanceof HMSExternalCatalog` 臂；`hive != iceberg` 故 `getEffectiveIcebergFormatVersion` 臂对 hms 保持 inert，已加非干扰锁测）+ `PluginDrivenExternalTable.getEngine()` 加 `case "hms"→HMS_EXTERNAL_TABLE.toEngineName()`（== `"hms"`；**DISPLAY 引擎，legacy 显示 "hms" 非 "hive"，与 CREATE 引擎两回事**；落 "Plugin" 会退化 SHOW TABLE STATUS/information_schema.tables）、`getEngineTableTypeName()` 加 `case "hms"→HMS_EXTERNAL_TABLE.name()`（守 pluginCatalogTypeToEngine sync 不变量）。休眠成立已 HEAD 证实（`SPI_READY_TYPES` 无 `"hms"`、CatalogFactory `case "hms"` 仍建 legacy HMSExternalCatalog→无 PluginDriven 目录 getType()=="hms"→三 case 翻闸前不可达）。休眠单测：`CreateTableInfoEngineCatalogTest` +5（no-ENGINE pad hive / CTAS pad hive / 错引擎拒 / hive 过 / iceberg-format 非干扰锁）、`PluginDrivenExternalTableEngineTest` +2（engine `"hms"` + type name `HMS_EXTERNAL_TABLE`）。全绿（fe-core BUILD SUCCESS + 31 测 0 fail + checkstyle 0；纯 fe-core 无连接器改动故 import gate N/A）。 - - **✅ WC3 DONE（`ab3b0a63ed1`）= 显式分区 INSERT 拒绝（item 2a，C 活跑）**：新 `ConnectorWriteOps.validateWritePartitionNames`（宽松 no-op 默认，镜像 `validateStaticPartitionColumns` 模式）+ `HiveConnectorMetadata` 覆写（foreign→兄弟**无论空非空**都转发〔兄弟接受 `PARTITION(names)`＝独立 iceberg 目录平价，空早返是 hive-only〕、hive+非空→抛**逐字** legacy 文案 `"Not support insert with partition spec in hive catalog."`、hive+空→静默；外来句柄永不强转）+ fe-core `BindSink.checkConnectorWritePartitionNames`（镜像 `checkConnectorStaticPartitions`：空守卫→解析句柄→SPI→`DorisConnectorException`→`AnalysisException`），`bindConnectorTableSink` 用 **`sink.getPartitions()`**（动态分区名列表，**非** `staticPartitionKeyValues`）调用；静态 `PARTITION(col='val')` 与裸 `INSERT...SELECT` 不受影响。**C 活跑但空守卫使裸 INSERT 对现存连接器字节不变、无连接器覆写该 SPI（皆继承 no-op）→ iceberg/paimon/mc 的 `PARTITION(names)` 仍静默接受如旧**；hive 抛拒休眠至 hms 进 SPI_READY_TYPES；翻闸删 `BindSink:667-669`（随 `bindHiveTableSink` 整体）是残留非本步。**覆盖策略=镜像 peer 静态分区缝（连接器单测+e2e，`bindConnectorTableSink` 无 fe-core 单测先例，已核实 HEAD 无同类测）**：`test_hive_write_type.groovy:250` e2e 文案逐字对齐（两路径都跑）+ `HiveConnectorMetadataSiblingDelegationTest` +1（hive 非空抛/空静默/不咨询兄弟）+ `EXPECTED_WRITE_METHODS` Rule-9 完整性锁加 `validateWritePartitionNames`（foreign 空亦转发）。4 维对抗复审（route/message-parity/live-no-regression+dormancy/field-keying+binding-completeness）+ refute 复核 **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + 兄弟测 8/8 + checkstyle 0〔api/hive/core〕 + import gate 净）。 - - **✅ WC4 DONE（`7aeac97c8e8`）= 读-ACID 查询结束提交接线（item 1，C 活跑、最高风险、翻闸硬前置）**：新 `ConnectorScanPlanProvider.releaseReadTransaction(queryId)`（no-op 默认，对齐 getDeleteFiles/classifyColumn 先例）+ `HiveScanPlanProvider` 覆写→`readTxnManager.deregister`（与 register 共用 HiveConnector 注入的**同一** per-connector 管理器→键匹配；commit 恰一次、幂等、吞 commit 异常=best-effort）+ fe-core `PluginDrivenScanNode.getSplits` 在 scanProvider 空判后**无条件**注册 query-finish 回调（`QeProcessorImpl.registerQueryFinishCallback`，**置于 pruned-to-zero 短路与 planScan 之前**故 planScan 开事务后抛仍有 release 兜底）+ 抽出 `buildReadTransactionReleaseCallback`（public static，回调经 `onPluginClassLoader` pin 插件 loader——回调在 StmtExecutor 线程〔app loader TCCL〕跑、commitTxn 按名解析 metastore/thrift 类故 pin 载荷；抽出以便脱离 getSplits 单测，镜像 `applyMvccSnapshotPin`）。**queryId 单串不变量**：`connectorSession.getQueryId()`（注册键+release 键，只取一次）==`DebugUtil.printId`（runAndClear 键）==连接器 txnMap 键。**C 活跑但仅 hive 覆写 SPI**（已 grep 证 iceberg/paimon/mc/jdbc/es 皆继承 no-op→回调对它们 inert：pin TCCL+调 no-op，永不抛，runAndClear 查询结束清除）；hive release 休眠至 hms 进 SPI_READY_TYPES；legacy `HiveScanNode`+`Env.hiveTransactionMgr` 未动（其删除+live-ACID e2e 读=翻闸残留）；foreign iceberg-on-HMS 句柄路由到兄弟 provider 的 no-op release（未开 hive 读事务）。测试：`HiveReadTransactionTest` +1（provider release 经共享管理器恰一次 commit+幂等+吞异常）、新 `PluginDrivenScanNodeReadTxnReleaseTest`（回调 release+对 distinct sentinel TCCL 证 pin 到 provider loader 且事后还原）；现存 scan-node 套无影响（无一驱动 getSplits 端到端）。5 维对抗复审（placement/dominance·TCCL·queryId-identity+registry·live-no-regression+dormancy·connector-release/idempotency）+ refute **0 确认发现**。全绿（api+hive+core BUILD SUCCESS + HiveReadTransactionTest 4/4 + fe-core 16 个 scan-node 类 78/78 + checkstyle 0〔api/hive/core〕 + import gate 净）。**⇒ 至此 §4.5 写链休眠四步 WC1–WC4 全部收官；写链只余翻闸期删除残留（R1–R4）。** - - **翻闸期残留（勿现在建，见分解 §3 R1–R4）**：删 `bindHiveTableSink`（含两处旧写前检查=一次删）、删旧 `HiveScanNode` + `Env.hiveTransactionMgr`（item 3，gated on WC4）、删 `HiveTableSink`、翻闸线本身；**item 2b（LZO 拒）无休眠工作**（已在连接器、只余翻闸删）。**W6 写路径 TCCL pin + e2e 回归亦翻闸期。** -- **⭐ Hudi 已升格为一等翻闸前工作线（硬 flip-blocker，非仅"写-拒"脚注）。权威规划 = `hudi-on-hms-delegation-plan-2026-07-08.md`（起步必读；recon `wf_ca1b3400-652`：4 维读者 + 完整性/排序 critic，全 HEAD 核对）。已签字（2026-07-08）：①hudi-on-HMS 走兄弟委派（镜像 iceberg）②增量读/时间旅行/MVCC 快照切换前必须补齐、不接受回退 ③先出规划不写码（本 doc 即交付）。** - - **✅ HD-A0 DONE（`37b85348667`，doc-only）= DV-005 护栏**：修正 `HudiConnectorProvider`/`HudiConnector` javadoc + `CatalogFactory.SPI_READY_TYPES` 注释——`"hudi"` 只是**兄弟类型串**（`createSiblingConnector("hudi",…)` 的查找键，绕过 `SPI_READY_TYPES`），**非**用户可见目录类型、无 `HudiExternalCatalog`；永不加进 `SPI_READY_TYPES`/工厂 case（否则重造 DV-005）。checkstyle 0（hudi+fe-core）。 - - **✅ HD-A1 DONE（`28d4781ccac`）= hudi 兄弟持有 + 属性合成（镜像 iceberg S1/S2，休眠）**：新 `HudiSiblingProperties.synthesize`=网关整份属性**逐字防御性拷贝**（与 iceberg 不同**不注入任何 flavor 键**——hudi 无 `iceberg.catalog.type` 类比，`HudiConnector.createClient` 直读 `hive.metastore.uris`/`uri`+raw `hadoop.*`/`fs.*`/`dfs.*`）+ `HiveConnector` 加 `HUDI_CONNECTOR_TYPE` 常量/`volatile hudiSibling` 字段/`getOrCreateHudiSibling()`（volatile 双检懒建、缺插件 fail-loud 且不 memoize、仅以 parent-first `Connector` 持有永不强转）/`close()` 转发兄弟。休眠（无生产引用，待 HD-B2 getTableHandle 分流点亮）。全绿（fe-connector-hive 230 测〔+3 `HudiSiblingPropertiesTest` +6 hudi holder 测含 `closeForwardsToBothSiblingsIndependently` 回归锁〕+ checkstyle 0 + import gate 净）；4 维对抗复审（holder-parity/close-lifecycle/dormancy-synthesis/test-intent）+ verify **0 发现**。 - - **✅ HD-A2 DONE（`561f777d5f9`）= hudi 兄弟 Kerberos plugin-auth**：端口 `HiveConnector.buildPluginAuthenticator`/`buildHadoopConf`/createClient auth-action（+pom 加 `fe-connector-metastore-hms`＝带 HMS Kerberos 解析器 + 传递 fe-kerberos，插件 zip runtime dependencySet 自动 child-first 打包）；走插件 UGI doAs 否则翻闸后共享网关 NOOP context 静默把 Kerberized HMS 降级 SIMPLE；**hudi 兄弟自有加载器故须自持认证器**（共享 hive 网关的会跨加载器裂 UGI 副本）。5 测（镜像 hive，KDC-free 惰性登录）。 - - **✅ HD-A3 DONE（`19f11063a79`）= 分区值解析保真 FIX**：`parsePartitionValues` 逐字端口 legacy `HudiPartitionUtils.parsePartitionValues`（positional 映射 + 单列整路回退 + `>1` 列 fail-loud + 内联 `unescapePathName`＝Hive FileUtils 字节复刻、无 hive-common 依赖），修非-hive-style 分区（`2024/01`）静默读 NULL 分区列 + 未 unescape 的真 bug；**范围仅值解析（unpruned 最基本读）**；pruned-path HMS-hive-style-vs-Hudi-positional 源一致性（须 `useHiveSyncPartition`）归 **HD-C1 partition-listing 步**（亦翻闸前、无翻闸回退，已记账非静默跳）。8 测。 - - **✅ HD-A4 DONE（`7bc90edf4a3` + 复审修 `ce700db35e0`）= force_jni_scanner + COW/MOR 检测硬化**：读 `force_jni_scanner` session 标志（镜像 paimon），planScan `useNativeCowPath = isCow && !forceJni`（force-JNI 把 COW 也推 JNI 路＝legacy `canUseNativeReader`）+ collectMorSplits 无-log native 降级加 `!forceJni` 守卫；**isCow 改由权威 `metaClient.getTableType()` 定**（非 substring 检测，UNKNOWN 不再静默误选路）。**5 维对抗复审揪出真缺陷（本 session 修）**：`populateRangeParams` 未显式 `setFormatType`、只靠单一 node 默认 → ①MOR 无-log native slice 当 `FORMAT_JNI` 空 THudiFileDesc 发 BE（blocker）②COW+force_jni 当 native（major，架空开关）③COW-ORC 当 parquet——已**镜像 `PaimonScanRange` 逐 range 显式设格式**修好（旧测用 `fileFormat("jni")` 假形状掩盖＝Rule-9 masking，已补 planner 真形状 native-parquet/orc 测）。6+4 测。 - - **✅ HD-A5 DONE（`d14da8312b5`）= BE 描述符 + BE-canonical 存储**：`buildTableDescriptor`→`TTableType.HIVE_TABLE`+`THiveTable`（镜像 hive；否则 SPI 默认 null 退化 SCHEMA_TABLE 描述符）；getScanNodeProperties 存储 = `context.getBackendStorageProperties()`（BE-canonical AWS_*，BE native FILE_S3 只认 canonical，否则私桶 403）**＋** hadoop passthrough（fs.* 给 Hudi JNI reader）＝legacy getLocationProperties 双 merge；context 穿进 provider（新 2-arg ctor）。残留（记账）：仅-Doris-alias 的 S3 上 JNI(MOR) 读缺 hadoop-normalized `fs.s3a.*`（须 hadoop-format hook，后续细化）；插件 zip 须带 JNI reader 传递依赖（be-java-ext 共享 classpath，翻闸期验）。3 测。 - - **✅ HD-B1 DONE（`136b5ced333`）= 三路 foreign-handle 路由（签字 Option 1 `Connector.ownsHandle` 中立 SPI）**：新 SPI `Connector.ownsHandle(handle)` 默认 false（**fe-core 永不调用**，引擎保格式无关）+ iceberg/hudi 各覆写自加载器内 `instanceof`（外来句柄类型跨插件加载器不可见，网关自己 instanceof 不到）；`HiveConnector.resolveSiblingOwner` **peek 已建好的兄弟 volatile 字段**（`icebergSibling`/`hudiSibling`，**不 force-build**）三路路由外来句柄→归属兄弟、否则 fail-loud 带目录名；3 个连接器级 `get*Provider(handle)` + `HiveConnectorMetadata` **~38 个按句柄转发点**改走它（`siblingMetadata(session, handle)`=`siblingOwnerResolver.apply(handle).getMetadata(session)`），**getTableHandle ICEBERG 按类型 divert 保 force-build**（helper 更名 `icebergSiblingMetadata(session)`，因无句柄可路由、须建 iceberg 兄弟）。**用户签字 peek（非计划字面 force-build）**：归属兄弟必被 getTableHandle 先 force-build（外来句柄只源于此），故路由只 peek 已建好的兄弟——一个只有 hudi 表、没装 iceberg 插件的目录仍能路由 hudi（**避免 force-build 引入的"hudi 查询要 iceberg 插件"回退**），且 iceberg 臂字节等价。休眠（hms 未进 SPI_READY_TYPES；**外来句柄永不强转**）。4 维对抗复审（iceberg 等价/peek 前提严密/路由正确+CCE/完整性+测试强度）+ 逐条 refute **0 blocker/major/minor、2 nit（均测试收紧：delegation 套把 by-type supplier 改 fail-loud 以锁死"按句柄站点走 peek resolver 非 supplier"、修陈旧覆盖交叉引用，已修）**。全绿（api `ownsHandle` 默认测 + iceberg/hudi `ownsHandle` 覆写测 + `HiveConnectorThreeWayRoutingTest` 6 测〔含 peek 双向不建对方兄弟 = no-regression 皇冠证据〕；hive 236 + hudi 58 + iceberg 15 + api + checkstyle 4 模块 0 + import gate 净）。 - - **✅ HD-C1 DONE（`bbe6cfcd647`）= MVCC 快照 + 分区列举 + 新鲜度（权威设计 `hudi-mvcc-partition-step-design-2026-07-09.md`；6 维对抗复审 `wf_c9cea94a-96d` + refute verify）**：镜像 paimon——只 override `beginQuerySnapshot`+`listPartitions`/`listPartitionNames`/`listPartitionValues`，**不** override `getMvccPartitionView`/`getTableFreshness`/`getPartitionFreshnessMillis`（flag=false 下死代码，加 SPI-默认守卫测）。`beginQuerySnapshot` 钉最新 completed instant 为 snapshotId（空 timeline→0L）、**不设 lastModifiedFreshness**→整表 `MTMVSnapshotIdSnapshot(instant)`/分区 `MTMVTimestampSnapshot(instant)`。**hudi MTMV 新鲜度改为真 instant = 有意提升非平价**（旧 `HudiDlaTable` 恒 `MTMVTimestampSnapshot(0L)` 桩从不刷新；memory `hudi-mtmv-freshness-real-instant`）。分区名 = **hive-style `col=escape(val)/...`**（新增 `escapePathName`=Hive makePartName 端口、`unescapePathName` 逆）——复审揪出并修好的 **1 major**：单列斜杠值（`TimestampBasedKeyGenerator` yyyy/MM/dd→路径 `2024/01/02`）不转义 `/` 会被 `HiveUtil.toPartitionValues` 截成 `2024` 且多分区撞车；转义 `%2F` 后 round-trip 保真。分区源 useHiveSyncPartition-aware（HMS names / hudi metadata listing，空 HMS 回退）、每分区 `lastModifiedMillis`=instant。**R4**：`HudiConnector` 注入 `HudiMetaClientExecutor`（plugin UGI doAs + TCCL pin 包 metaClient 触碰）。shared statics（`buildMetaClient`/`latestCompletedInstant`/`listAllPartitionPaths`）保 MVCC pin 与扫描同 instant + 单份 getAllPartitionPaths；`resolvePartitions` 复用后者（字节不变）；`parsePartitionValues`（HD-A3）字节不变。复审 refute 掉 R2 applyFilter 项（pre-existing 休眠扫描路径、非本 diff、按签字设计推迟到翻闸 e2e）。全绿（fe-connector-hudi 76 测〔`HudiConnectorPartitionListingTest` 18〕+ checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e）**：R2 prune-to-zero（非 hive-sync 分区谓词查询完整性，翻闸 e2e 验，届时定 `ignorePartitionPruneShortCircuit`）；`getNewestUpdateVersionOrTime` 字典探针 hudi 走 max(instant)〔已天然覆盖，无回退〕；e2e（COW/MOR × hive-sync/非、MTMV 刷新、Kerberos）随其它 iceberg/hudi-on-HMS 能力统一补。 - - **✅ HD-C2 DONE（`cf4f0e1b756`）= FOR TIME AS OF 时间旅行（权威设计 `hudi-time-travel-step-design-2026-07-09.md`；5 读者 recon `wf_e0b62364-d20`+对账 critic；4 维对抗复审 `wf_5fd05048-a97`〔no-regression/legacy-parity/pin-plumbing/test-strength〕+refute verify，**0 确认缺陷**）**：`HudiTableHandle` 加 `queryInstant` 钉字段（pin 脊柱，HD-C3 复用）；`resolveTimeTravel`：TIMESTAMP→归一化剥离 `[-: ]` byte-faithful legacy（无 session-TZ、无 epoch-millis、**无时间线校验、永不报错**——**纠正规划文档**照搬 paimon 的「校验+notFound」，那会把老路径返空的查询变报错=回退）、经内部 MVCC 属性钉；SNAPSHOT_ID/VERSION_REF→**抛**逐字 legacy 文案（含反引号）「Hudi does not support FOR VERSION AS OF, please use FOR TIME AS OF」（抛而非返空，否则 fe-core 显示领域不对的「can't find snapshot」）；其它 kind→empty（SPI 默认，INCREMENTAL 归 HD-C3）。`applySnapshot`：属性在→`toBuilder().queryInstant()` 钉（**保 prunedPartitionPaths**，applyFilter 先于 applySnapshot）；空属性(query-begin 最新 pin)/null→句柄不变=普通读字节不变。`planScan` 单一 instant 收敛点读 `handle.queryInstant`（驱动 COW/MOR before-or-on 选文件 + MOR-JNI `THudiFileDesc.instantTime` 一致）。**零 fe-core**（pin 走通用 MVCC 缝：loadSnapshot→resolveTimeTravel→ConnectorMvccSnapshot 属性→applyMvccSnapshotPin→applySnapshot→句柄→planScan；版本感知 `getSnapshotFromContext` 保 FOR TIME 查询取 TT pin、普通读取最新 pin）。全绿（`HudiTimeTravelTest` 9 + hudi 85/85 + checkstyle 0 + import gate 净）。**⚠残留（翻闸前/e2e，已文档化非静默）**：①schema-at-instant（schema-on-read 演进表 FOR TIME 读最新 schema；非演进/非 schema-on-read 平价）归 HD-C4；②**partition-SET-at-instant**（复审揪出：FOR TIME 的分区**集合**仍取最新→pin 后被 DROP 的分区可能静默漏行，取决于 Hudi `getAllPartitionPaths` 是否 tombstone；ADD 的分区无害）——翻闸前补/验（dropped-partition fixture）；③planScan 钉时刻只 e2e 可证（离线测覆盖 routing/归一/句柄穿透）。 - - **✅ HD-C3 INC-1 DONE（`9327261ec52`，休眠）= @incr 窗口解析脊柱**（recon `wf_3398c7fb-bfb` 4 读者+parity critic〔scope critic StructuredOutput 上限失败，已用我亲手字节码核对补〕 + 对抗复审 `wf_13bea565-a47` 4 维〔各 finding 逐条 refute〕）。连接器侧扩展 HD-C2 脊柱：`resolveTimeTravel(INCREMENTAL)` **单一 locus** `resolveIncremental` 解析 (begin,end]〔begin 必填逐字 fail-loud、`"earliest"`→`"000"`、end 缺省→最新 completed instant、`"latest"`→最新〔测**已解析 end 值**=legacy COW 形，天然规避 `MOR:92` 死码 bug〕、空时间线→`(000,000]` 免 begin 检查〕→ **NON-EMPTY property-only pin**（snapshotId/schemaId inert：fe-core INCREMENTAL `loadSnapshot` 分支自列 latest 分区+latest schema、只读窗口属性）；`applySnapshot` 钉 `begin/endInstant`（与 FOR TIME `queryInstant` 载体互斥、保 prunedPartitionPaths）。新 `HudiTableHandle.begin/endInstant` + `HudiScanPlanProvider.latestCompletedInstantTime(String)`（从 `latestCompletedInstant` 字节等价抽出）。**复审关键裁决**：①**blocker（COW 时间线 byte-parity）被 hudi-1.0.2 字节码驳倒**——单 `getCommitsAndCompactionTimeline()` 按类型解析到 legacy 逐类型时间线（COW→`getCommitAndReplaceTimeline`={commit,replacecommit,clustering}=legacy COW 的 `metaClient.getCommitTimeline()`〔该 metaClient 方法**非** commit-only，委派给 getCommitAndReplaceTimeline〕；MOR→`getWriteTimeline`），review+adversarial-verify 都把 metaClient 方法与 `BaseHoodieTimeline.getCommitTimeline()` 时间线方法混淆了；②major/minor/nit 均**按设计延后**（见下）。`HudiIncrementalTest` +11、fe-connector-hudi 96 测 + checkstyle 0 + import gate 净。**⚠延后（已文档化 fail-loud、非静默丢，INC-2/4 须接）**：(a) **`populateMetaFields()` fail-loud→INC-2**（"meta fields disabled" 拒，结构上属 relation 构造器=INC-2 移植；INC-1 窗口解析不查，INC-2 移植时在 planScan 强制、同为查询规划期失败；INC-2 须字节忠实移植+e2e 加 meta-fields-disabled fixture）；(b) **hollow-commit `USE_TRANSITION_TIME`（completionTime）+ hollow 抛→INC-2/INC-3**（默认 FAIL 策略用 requestedTime，非默认策略绑 relation 文件选择）；(c) **原生 `hoodie.datasource.read.*.instanttime` 窗口键→INC-4**（INC-1 读标准 `@incr` 别名 beginTime/endTime，是否也收原生 hoodie 键归 INC-4）。 - - **✅ HD-C3 INC-2 DONE（`d64a25f28fc`，休眠）= IncrementalRelation 家族移植**（HEAD-grounded recon `wf_50a59f91-14e` 3 读者〔窗口协调/边界/API+可测性〕+完整性 critic + 5 维对抗复审 `wf_b247a285-d64`〔byte-parity/dormancy-iron-rule/guard-correctness 三镜头 **0 发现**；test-strength 3 test/doc-honesty finding 已修；mor92-and-axis 镜头 API stall 死→crux 亲手 grep+读自证〕）。COW/MOR/Empty + `IncrementalRelation` 接口移植进 `fe-connector-hudi`（连接器内部、package-private）。**用户签字方案 A（单一窗口权威，见本 session 决策）**：移植关系**消费句柄已解析的 (begin,end] 窗口只挑文件、自己不再解析**（构造器删窗口解析 prologue）→ 死码 `MORIncrementalRelation:92` sentinel bug **自然消失**、文件集永不与 INC-4 `_hoodie_commit_time` 行过滤的 end 打架。split 型 re-home `spi.Split`/`HudiSplit`→`HudiScanRange`（丢 `LocationPath`/`TableFormatType`/`HudiPartitionUtils` fe-core 型）；COW `collectSplits`→`List`（path + length/fileSize 取 `HoodieWriteStat.getFileSizeInBytes()`、format 走 `detectFileFormat`、分区值走连接器 Map `parsePartitionValues` on `metaClient.getTableConfig().getPartitionFields()`=legacy COW:212 平价，**非** HMS handle keys）；MOR `collectFileSlices`→`List` 逐字；archived-flag 起字节忠实 legacy，仅删窗口 prologue。INC-1 延后守卫作 `IncrementalRelation` 共享静态、逐字文案 re-type 成 `DorisConnectorException`：`checkIncrementalMetaFields`（meta-fields-disabled 首守卫、非空时间线 only 靠上游 Empty 路由）/`checkStateTransitionTimeFullTableScan`/`decideArchivalFullTableScan`/`checkNotFullTableScan`。**`resolveIncremental` END 轴按策略**：`hoodie.read.timeline.holes.resolution.policy=USE_TRANSITION_TIME` 下 default/`latest` end 走 completion-time 轴（新 `latestCompletedInstantTime(mc, useCompletionTime)` 重载：getCompletionTime vs requestedTime）→ 单一 handle-stamped end 与关系 `findInstantsInRangeByCompletionTime` 选片同轴、无末批漏读；默认策略字节不变、休眠（planScan 只读 `queryInstant` 从不读 begin/endInstant）。`getHoodieParams`/`getStartTs` 丢弃（FE-filter 模型下已死）。全绿（`HudiIncrementalRelationTest` 13 + `HudiIncrementalTest` 12 + 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。**⚠ e2e-only（休眠单测只覆盖纯守卫+Empty；构造器 timeline/metadata/fs 选片 + completion-time 轴须 e2e，设计 §5 已加 meta-fields-disabled + USE_TRANSITION_TIME fixture）**。**下一步 = INC-4（中立 synthetic-predicate SPI + fe-core 分析期规则 + 反向 ConnectorExpression→Nereids 转换器 + 暴露 5 可见 `_hoodie_*` 元数据列）**。 - - **✅ HD-C3 INC-3 DONE（`0fea5bc6ba4`，休眠）= 增量 planScan 接线**（对抗复审 `wf_99c5b718-c11` 4 维〔snapshot byte-parity / incremental-correctness / dormancy-transport / test-intent〕+ 逐条 adversarial-verify **0 发现**）。`HudiScanPlanProvider.planScan` 加 `getBeginInstant()!=null` 增量支路（置于 schema 解析后、fsView 构建前，纯早返；非增量读〔含 FOR TIME AS OF〕字节不变）：`buildIncrementalRelation` 按 `isCow`(=`metaClient.getTableType()==COPY_ON_WRITE`，权威) 建 COW/MOR 关系（IOException 重型为 `DorisConnectorException`）；`incrementalRanges` **纯静态**路由（fake 关系可离线测）——`fallbackFullTableScan()`→`Optional.empty()`→降级到最新快照扫描（legacy `getSplits:470`，读 latest；两载体互斥故 @incr 句柄无 queryInstant）；COW→`collectSplits()` 原生；MOR→`collectFileSlices()` 逐片经**抽出的** `buildMorRange`（从 `collectMorSplits` 内联 lambda 提取、快照路径字节不变）转 JNI range 钉 `getEndTs()`（窗口 END=BE 合并时刻，非 latest）、分区值按 `metaClient.getTableConfig().getPartitionFields()` 逐片解析（=legacy `HudiScanNode:391-393`，**非** HMS handle 键）。**RO-as-RT 翻转 = 被 metaClient.getTableType() 天然覆盖、不移植、不需 serde 参数上句柄**（旧翻转只为补偿 legacy 按 hive inputFormat 把 MOR `_ro` 门面误判 COW；连接器读 hoodie.properties 直接得 MERGE_ON_READ）。**用户签字方案 B（graceful，2026-07-09）**：`force_jni+COW+增量` 旧代码 `collectFileSlices()` on COW relation→`UnsupportedOperationException` 崩溃，改按**关系类型**路由→COW 增量恒原生（忽略 force_jni），不崩溃；MOR 仍经 buildMorRange 尊重 force_jni（无 log 片留 JNI）。**参数穿透**：`resolveIncremental` 把原始 @incr 选项（glob/fallback/hollow-policy）复制进 MVCC 快照属性（前缀 `hudi.incr-opt.`、跳 null 值），`applySnapshot` 前缀剥离重建进**新** `HudiTableHandle.incrementalParams`（**BE 安全**：快照属性只经 `PluginDrivenScanNode.applyMvccSnapshotPin`→`applySnapshot` 消费、从不入 BE thrift〔亲手 + 复审双证〕；begin/end 载体走 `hudi.incremental-*` 不同命名空间不混入；`toBuilder` 复制）；planScan 由 `IncrementalRelation.hollowCommitHandling(optParams)`=`valueOf(getOrDefault(policyKey,"FAIL"))` 派生策略，与 `resolveIncremental` END 轴 `INCR_HOLLOW_POLICY_KEY` 同源（文件集与窗口 END 同轴）。**行级过滤（`_hoodie_commit_time` 谓词）= INC-4；本步只挑 FILE，裸 @incr 会多读直到 INC-4（休眠无影响）**。全绿（`HudiIncrementalPlanScanTest` 7〔fake IncrementalRelation + 手造 FileSlice：fallback 降级/COW 路由不碰 collectFileSlices/COW 忽略 force_jni/MOR 钉 endTs+逐片分区值/buildMorRange JNI+native 降级〕+ `HudiIncrementalTest` 15〔+3 参数穿透+命名空间隔离+非增量 pin 空 incrementalParams〕+ `HudiIncrementalRelationTest` 16〔+3 hollowCommitHandling FAIL 默认/USE_TRANSITION_TIME/bogus 抛〕+ 全 hudi 模块 0 fail + checkstyle 0 + import gate 净）。 - - **✅ HD-C3 INC-4 DONE（`c692f4fe763`〔连接器 SPI+hudi+hive+可见元数据列〕+`537afcb37f6`〔fe-core 反向转换器〕+`81b20559f45`〔fe-core CheckPolicy 中立臂+表接线〕+复审修 `48bc2b207cd`，休眠）= 中立 synthetic-predicate SPI（行级 `_hoodie_commit_time` 窗口过滤）**（**用户签字方案 A**=通用 `ConnectorExpression` 而非专用 struct；6 维对抗复审 `wf_2e2ac37c-d3f`：row-correctness/reverse-converter/rule-dormancy/iron-rule-gateway/schema-plumbing 5 维 **0 发现** + test-intent 1 minor〔meta-col 测 javadoc overclaim + isVisible 空断言〕已修）。**问题=COW base 文件在窗口内被重写会带出窗口外旧行，只挑文件不够、须行级 `_hoodie_commit_time > begin AND <= end`；legacy 靠 source-specific fe-core 注入（`CheckPolicy`+`LogicalHudiScan.generateIncrementalExpression`），翻闸后死码+违铁律，须挪进中立 SPI（对齐 Trino 残余谓词）**。①新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,ConnectorMvccSnapshot)` 默认空——**纠正设计硬伤**：原签名收 fe-core `analysis.TableScanParams`，连接器-api 不依赖 fe-core+import gate 禁→编译不过，改收连接器-api `ConnectorMvccSnapshot`；hudi override 从**同一** `resolveIncremental` 盖的 `HUDI_INCREMENTAL_BEGIN/END_PROPERTY` 读窗口（**单一窗口权威**，与 applySnapshot 文件选择同源、永不发散）发 `> begin`(GT)+`<= end`(LE) 两 flat STRING conjunct（词典序 instant 比较，byte-faithful legacy）；非增量 pin（plain/FOR TIME）返空。②**HiveConnectorMetadata 委派 override**（**recon 揪出漏项**：无它→foreign hudi 句柄继承空默认→静默 over-read）转发兄弟，镜像 resolveTimeTravel/applySnapshot、外来句柄永不强转、hive 句柄返空。③fe-core 新 `ConnectorExpressionToNereidsConverter`（**TOTAL+FAIL-LOUD**，与 forward 转换器 drop-on-unknown **相反**：漏 conjunct=under-filter 漏行；仅 {ConnectorAnd, GT/LE/EQ/LT/GE, ColumnRef 按名绑, STRING literal}，And(size≥2) 守卫+单 conjunct 解包，其余/未解析列/非 STRING literal 抛 `AnalysisException`）。④`CheckPolicy` **中立臂**（keys `LogicalFileScan`+`instanceof PluginDrivenExternalTable`+`scanParams.isPresent()`——**零 source-specific**，legacy `LogicalHudiScan` 臂在前先命中不重叠；从 `StatementContext.getSnapshot` 取分析期已解析快照→`PluginDrivenExternalTable.getSyntheticScanPredicates(MvccSnapshot)`〔连接器/句柄/快照解包 plumbing，镜像 resolveWriteTargetHandle〕→反向转换绑扫描输出 slot→注入 `LogicalFilter`）。**关键=注入必在分析层非扫描节点**（扫描节点无法为未投影列造 slot；窗口经 `ConnectorMvccSnapshot` 而非 scan-node 才盖的 handle→analysis-time 可达）。⑤`getTableAvroSchema()→getTableAvroSchema(true)`（连接器 schema+扫描侧 lockstep）=legacy 平价、common(populate.meta.fields=true)表字节不变、robust 暴露 `_hoodie_commit_time` 可绑列。**iceberg/paimon 返空默认→计划字节不变；DORMANT**（hms 未进 SPI_READY_TYPES + hudi getTableHandle 分流=HD-B2 后续步；live iceberg/paimon @branch/@tag 命中臂但 SPI 空→字节不变，多一次 same-statement 缓存命中 getTableHandle）。全绿（hudi `HudiIncrementalTest` 18+`HudiSchemaParityTest` 4 / hive `HiveConnectorMetadataSiblingDelegationTest` 8〔+divert+完整性锁+hive 空〕/ fe-core `ConnectorExpressionToNereidsConverterTest` 8+`PluginDrivenExternalTableTest` 33+`CheckRowPolicyTest` 6 无回归 + api/hive/hudi/fe-core checkstyle 0 + import gate 净）。**下一步 = INC-5 翻闸接线**。**⚠残留（e2e/翻闸，非静默）**：①`getTableAvroSchema(true)` 对 populate.meta.fields=false 表的 SELECT*/DESCRIBE 元数据列暴露只 e2e 可证（单测只覆盖 `avroSchemaToColumns` 名/型/可见保真=绑定前提，非 live getTableAvroSchema）；②行级过滤正确性（straddling-base-file carry-forward）+ 兄弟委派跨加载器只 e2e 可证；③原生 `hoodie.datasource.read.*.instanttime` 窗口键是否收（INC-1 延后项）=INC-5/e2e 决策；④INC-5 翻闸删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`。 - - **HD-C3 recon + 设计 + 签字（`99417d3762e`，权威设计 `hudi-incremental-step-design-2026-07-09.md`，起步必读）**。两轮 code-grounded 侦察（`wf_4b001028-a0c` BE 行过滤判定 + `wf_05d58d84-143` FE-only 可行性，均逐条 HEAD/BE 源码核对 + 我亲手复核）。**§5.1 landmine 已解且判定 = FE-only、零 BE 改**（**纠正交接旧说「本 checkout 无 BE 树」——BE 树在本 checkout，已实测读 `be/src/format*/…/hudi_*reader.*` + Java `HadoopHudiJniScanner`**）：BE 两读取路径都不自窗口过滤（native 读整 base 文件、JNI Java 扫描器丢弃 hoodie.* 增量参数），但**都已应用扫描 conjunct**；旧行级正确性靠 `CheckPolicy:83-88`+`LogicalHudiScan.generateIncrementalExpression` 注入的 `_hoodie_commit_time` 通用谓词（plugin 路径 `instanceof LogicalHudiScan`/`HMSExternalTable` 皆不命中→死，铁律禁 fe-core 复刻）。**签字方案（用户 2026-07-09）= 中立通用 SPI**：连接器经新 `ConnectorMetadata.getSyntheticScanPredicates(session,handle,scanParams)`（默认空、iceberg/paimon 返空→计划字节不变）供谓词；fe-core **分析期**中立规则（换掉/旁替 `CheckPolicy:83-88` 的 source-specific 臂）把它作 `LogicalFilter` 注入、引用连接器暴露的 `_hoodie_commit_time` 列强制物化，BE 用**已有** conjunct 机制执行——零 BE 改、零 fe-core source-specific 分支（对齐 Trino 未强制谓词模型 + 本迁移「挪进中立 SPI」套路）。**注入必须在逻辑(分析)层非物理扫描节点**（扫描节点无法为未投影列造 slot；`PhysicalPlanTranslator.java:896-898` 只循环 `scan.getOutput()`；窗口在分析期已知——scanParams 于 `BindRelation.java:654`、snapshot 于 `StatementContext.loadSnapshots`）。**签字 D-C3-1（用户 2026-07-09）= 5 个 `_hoodie_*` 元数据列全可见（对齐 legacy `SELECT *`，无倒退基准）**，顺带使 `_hoodie_commit_time` 天然可物化。**5 步休眠拆解**（设计 §4）：**✅ INC-1 DONE（`9327261ec52`）** 句柄 pin+`resolveTimeTravel(INCREMENTAL)`/`applySnapshot`（见上 INC-1 条目）→ **✅ INC-2 DONE（`d64a25f28fc`）** 移植 COW/MOR/Empty IncrementalRelation（方案 A 单一窗口权威、split re-home `HudiScanRange`、`MORIncrementalRelation:92` 因不重解析而自然消失；**+ INC-1 延后的 `populateMetaFields()` fail-loud + hollow-commit END-轴处理**）→ **✅ INC-3 DONE（`0fea5bc6ba4`）** 增量 `planScan`（COW/MOR 按 metaClient 类型路由〔**RO-as-RT 被 metaClient 覆盖、不移植**〕/fallback 降级到最新快照/`buildMorRange` 钉 endTs/原始 @incr 选项经 `hudi.incr-opt.` 穿透句柄/**force_jni+COW graceful 不崩溃**；**不发** hoodie.* 到 BE）→ **✅ INC-4 DONE（`c692f4fe763`/`537afcb37f6`/`81b20559f45`+复审修 `48bc2b207cd`）** 中立 SPI（**收 `ConnectorMvccSnapshot` 非 `TableScanParams`**——设计原签名编译不过已纠正）+ fe-core 反向 `ConnectorExpression→Nereids` 转换器（~100 LOC 实测，非设计估的 150-250；TOTAL+fail-loud）+ CheckPolicy 中立臂（keys `PluginDrivenExternalTable`+scanParams，注入分析层）+ 可见元数据列（`getTableAvroSchema(true)`）〔原生 hoodie.* 窗口键决策=延后 INC-5/e2e〕→ **下一步 INC-5** 翻闸接线（删 `visitPhysicalHudiScan` 增量 throw + legacy `CheckPolicy` hudi 臂 + `datasource/hudi/source/*`）。**再后 = HD-C4/C5 schema evolution（DV-006，用户须定范围）→ Group D HD-D1 写-拒安全网 → HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步，须 HD-B1✅ + Group C + HD-D1 全完）**。C2/C3 共享 `HudiTableHandle` pin 脊柱=建一次；C1/C3 共享 ported `HudiPartitionUtils`/`HudiUtils` 片段=port 一次。 - - **✅ HD-C4/C5 schema-evolution recon + 设计 + 签字（recon `wf_85bd47a0-0aa` 6 读者+完整性/拆解 2 critic〔iron-rule 读者 StructuredOutput 上限失败，gap 由 critic+亲手核对补〕；权威设计 `hudi-schema-evolution-step-design-2026-07-09.md`，起步必读）= 完整对齐老实现（用户签字 B，非窄退化；memory `hudi-schema-evolution-full-parity-signoff`）**。**BE 核实**：原生机器已就位、**零 BE 改**（`be/src/format_v2/table/hudi_reader.cpp` 已读 `hudi_params.schema_id`+`can_map_by_history_schema`+`annotate_file_schema_from_history`，与 paimon 同）。**架构裁决**：**模板=paimon 多条历史(-1+每引用版本)+每-split schema_id**（非 iceberg 单条；`THudiFileDesc` 有 f12 schema_id 而 `TIcebergFileDesc` 无=load-bearing divergence）；**小写逐层=iceberg**（format_v2 大小写不敏感但同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader→`StructNode.children.at()` raw name→std::out_of_range→整库 SIGABRT）；**current_schema_id 保持 -1**（真 id 撞某 split id→v1 `current==split`→ConstNode 大小写敏感逐字=禁的半吊子退化；paimon+legacy 均 -1，BE `find_external_root_field` 认 -1 条目为 target overlay）；**field-id 源=mode-aware commit-metadata InternalSchema**（stable across renames；非 naive `convert(latest avro)` 否则 -1 条目 id≠per-file id→BY_FIELD_ID 失配）；**每-split schema_id 只盖原生分支**（JNI/MOR-realtime 不看 schema_id、读 `column_names/types`@instant→C5b 全靠 FE 供）；**零新 SPI**（3-arg `getTableSchema`+hive 委派已发 `HiveConnectorMetadata:1070-1078`、`populateScanLevelParams` no-op、`getScanNodeProperties`、`ConnectorColumn.withUniqueId`、`THudiFileDesc` f12、`TFileScanRangeParams` 25/26 全在；100% 连接器内部）。**关键结构事实**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:973` vs `:979`）→句柄 field-id 是 latest-keyed→时间旅行改名列须 **C5b 按 pinned schema 重解 -1 条目**（iceberg full-pinned/empty-requested 式），非靠句柄；**C4c 硬序在 C4d 前**（字典present 而 per-split schema_id 仍 -1 = 禁的 current==file==-1）。**§6 签字决策（对齐 legacy 字节忠实）**：历史集=被选中文件涉及版本的并集(非全量枚举、自洽=每发出 split id 必在史、=legacy per-referenced-file、C4c/C4d 共享 resolver 背靠背落)；非演进表 FOR TIME=latest-fallback(schema 从未变=字节等价)；scalar `TColumnType`=STRING placeholder(BE 只用作 nested-vs-scalar 判别，不 port legacy 全 avro→Doris 型表)；field-id 留句柄(=legacy `updateHudiColumnUniqueId`)。**用户签字两部分拆分（2026-07-09）**：**Part 1=C4a–C4d + 完成后统一对抗复审；Part 2=C5a–C5b。用户下个 session 起 Part 1（首步 C4a=纯 `HudiSchemaUtils` 转换器：`InternalSchema/Types.Field→TSchema/TField`+base64 encode/apply 往返，逐层小写、STRING placeholder、id+name+is_optional 每层，wired 进空，完全可单测，零 fe-core import）**。**e2e 硬门（切换期，非静默）**：改名/调序/加/删列 COW+MOR 读(native BY_FIELD_ID engage)、混合大小写嵌套结构不崩、演进表 FOR TIME(COW 列表+MOR JNI 列表@instant)、非演进 FOR TIME 等价、@incr latest schema、异构 HMS vs 独立 hudi 目录同表比行（memory `hms-iceberg-delegation-needs-e2e`）。 - - **✅ HD-C4a DONE（`d29bb52173d`，休眠，Part 1 首步）= 纯 `HudiSchemaUtils` 转换器**（新连接器内部类，`fe-connector-hudi`）。自足 `InternalSchema`/`Types.Field`→`TSchema/TStructField/TField` 构建器 + base64 `encode`/`applySchemaEvolution` 往返 on 抛弃型 `TFileScanRangeParams`（f25 `current_schema_id` + f26 `history_schema_info`）。**移植 legacy `HudiUtils.getSchemaInfo(InternalSchema/List/Types.Field)` 两处有意偏离**（均由 mutation-tight 测钉）：①**逐层小写**（`Locale.ROOT`，镜像 `IcebergSchemaUtils.buildField`；legacy 不小写）——同 `history_schema_info` thrift 也喂 v1 `format/table` hudi reader，其 `StructNode` 按小写 Doris slot 名查，混大小写嵌套名→`std::out_of_range`→整库 SIGABRT；②**scalar `TColumnType`=STRING placeholder**（BE field-id 路只用 `type.type` 作 nested-vs-scalar 判别）——丢 legacy `fromAvroHudiTypeToDorisType` 全型表。`id`+`name`+`is_optional` 每层保真（legacy 平价）。`encode` serialize 错 fail-loud（catch `Exception|LinkageError` 防 thrift 类加载器裂）；`applySchemaEvolution` null/empty no-op、corrupt fail-loud（镜像 iceberg/paimon）。**模板=paimon（后续 C4c/C4d 复用 `buildSchemaInfo`+`encode` 发 -1 目标条 + 每引用版本历史 + 每-split schema_id），小写=iceberg 每层。wired 进空、零 fe-core import、零新连接器-api SPI。DORMANT**（hms 未进 `SPI_READY_TYPES`、hudi 句柄未分流）。`HudiSchemaUtilsTest` 11 全 same-loader（手搭混大小写嵌套 struct 子字段的 `InternalSchema` 断言每层小写/嵌套 struct·array·map 标签+每层 field-id/STRING scalar/InternalSchema-keyed 与显式 -1 schema id/base64 往返）。全绿（hudi 模块 138 测 + checkstyle 0 + import gate 净）。**下一步 = C4b（field-id 上 `HudiColumnHandle`）**。 - - **✅ HD-C4b DONE（`d54dc2d3ac5`，休眠，Part 1 第二步）= field-id 上 `HudiColumnHandle`**（源自 mode-aware InternalSchema）。①`HudiColumnHandle` 加 `int fieldId`（ctor+`getFieldId()`+toString）——**不入 equals/hashCode**（身份保 name+typeName，镜像 `IcebergColumnHandle` 按名不折 id）。②`HudiConnectorMetadata.getSchemaFromMetaClient` 解析 mode-aware InternalSchema：no-arg `TableSchemaResolver.getTableInternalSchemaFromCommitMetadata()` 钉**最新 commit**（schema.on.read 开→commit-metadata InternalSchema〔改名跨版本**稳定 id**〕；关→`AvroInternalSchemaConverter.convert(latest avro)` 位置 id）——镜像 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`（steady-state/无时间旅行 pin，at-instant 变体留 C5a）；解析失败降级 columns 不带 id（UNSET→BE BY_NAME）而非丢整 schema。③`getColumnHandles` 穿 `col.getUniqueId()` 上 handle（镜像 iceberg）。**与 legacy 有意偏离（已文档化）**：legacy 列源+id 源同一 InternalSchema 位置 zip；连接器保 `getTableAvroSchema(true)` 独立列源（保住已发的含 meta 列 schema），故 `attachTopLevelFieldIds` **按名匹配**——对齐列集等价、且当 commit-metadata InternalSchema 缺某列（如 `_hoodie_*` meta）时该列留 UNSET→BY_NAME（正确，meta 列从不改名）。只顶层 id 上 handle；嵌套 field-id 由 InternalSchema 经 `HudiSchemaUtils` 直发（C4a）。**⚠残留**：`NOTE(C5a)` 此 metaClient touch 尚未 `HudiMetaClientExecutor.execute` 包裹——closing 既有 `getSchemaFromMetaClient` 未包裹 auth/TCCL gap 留 C5a（它重构此读）；evolution-mode commit-metadata id 源仅 e2e 可证（单测覆盖非演进 `convert(avro)` 路）。DORMANT（hudi 故意不在 `SPI_READY_TYPES`、句柄未分流→无 live 目录达 `getColumnHandles`）。`HudiColumnFieldIdTest` 4 same-loader（按名取 id/未命中 UNSET/handle 载 id/id 不入身份）。全绿（hudi 142 测 + checkstyle 0 + import gate 净）。**下一步 = C4c（每-split `THudiFileDesc.schema_id` 仅原生分支）**。 - - **✅ HD-C4c DONE（`ac3222bc526`，休眠，Part 1 第三步）= 每-split `THudiFileDesc.schema_id`（仅原生分支）**。给每个原生 base 文件钉写它的那次 commit 的 InternalSchema 版本号（field 12），BE 原生 field-id 读取器据此在字典里查该文件的列并按 id 对齐当前表（改名安全）。**仅原生分支**：JNI 合并读取器读 `column_names/types`@instant、不消费 schema_id。①`HudiSchemaUtils` 两共享静态 resolver：`resolveTableInternalSchema`（mode-aware base：schema.on.read 开→commit-metadata 稳定 id、关→`convert(latest avro)` 版本 0，+evolution 标志；C4b 的 `attachHudiFieldIds` 重构复用之=id 源单一真源）+`resolveFileInternalSchema`（每文件版本：evolution 开→`FSUtils.getCommitTime(fileName)`+`InternalSchemaCache.searchSchemaAndCache`；关→base 版本 0）——port legacy `HudiScanNode.setHudiParams`。②`HudiScanRange.Builder.schemaId`（可空）→ctor 仅设时放 `hudi.schema_id` prop→`populateRangeParams` **仅原生 else 臂** `fileDesc.setSchemaId`（镜像 `PaimonScanRange`），JNI 臂永不；缺→BE BY_NAME。③`planScan` 解析一次 base schema、建每文件 schema_id resolver（base 文件路径→版本）；`collectCowSplits`（COW base 恒原生）+`buildMorRange` 原生无 log 片钉之；@incr 路传 null（@incr 列最新 schema、无每文件字典）。base-schema 未解析→null resolver→不钉；每文件解析失败→log 跳过该文件→均降 BY_NAME 不崩扫描。**⚠编译期坑**：`searchSchemaAndCache` 编译classpath 是 **2-arg** `(long, metaClient)`（与 legacy fe-core 同），非默认 m2 里 SELECTDB-0.14.1 jar 的 3-arg（javap 看错 jar）——已用 2-arg。**排序（关键）**：schema_id present 而字典缺→BE 不入 field-id 路→**BY_NAME=安全前缀**；字典步（C4d）须在其后，dict-present+schema_id-unset 才是禁的 `current==file==-1` v1-ConstNode 退化。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。schema_id **值 e2e**，只 stamping 机制 same-loader 测（`HudiScanRangeTest`+3：原生 set f12 / JNI 即便设了也不 set / 原生无 id 留 unset；`HudiIncrementalPlanScanTest` buildMorRange 断言原生 downgrade 载 resolver id、force-jni 片不载）。全绿（hudi 145 测 + checkstyle 0 + import gate 净）。**下一步 = C4d（扫描级字典发射，与 C4c 背靠背；C4c 安全前缀已落）**。 - - **✅ HD-C4d DONE（`617c44b049b`，休眠，Part 1 收官）= 扫描级 `hudi.schema_evolution` 字典发射**。发原生读取器结构字典（`current_schema_id`+`history_schema_info`），BE 据此按字段 ID 对齐 file↔table（否则改名列在老文件读 NULL）；与 C4c 每-split schema_id 成对，一起点亮 BE field-id 路。①`HudiSchemaUtils.buildSchemaEvolutionDict`（纯、可测）=一条 `-1` 目标条 + 历史条。`-1` 目标条（`buildTargetSchema`）**keyed off requested columns**（顶层名==BE scan slots=CI-969249 StructNode 不变量）：逐列按名在 mode-aware base InternalSchema 查全嵌套结构+稳定 field-id；requested 列不在该 schema（如演进表投影 `_hoodie_*` meta、其 commit-metadata schema 无 meta）→退标量占位带 handle field-id；空 requested（count-only）→全 base 顶层字段。②**历史集=全部已提交版本（用户签字 Option B）**——被引用文件版本的 robust 超集，镜像 paimon all-`listAllIds`：自洽（每原生文件 schema_id 必在史→BE 永不 fail-loud "miss schema info"），且字典与"某次扫描选哪些文件"**零耦合**（避开在 getScanNodeProperties〔与 paimon 同、拿不到 planScan 的 splits〕里重列文件的开销+两处发散风险）。evolution 经 `FileBasedInternalSchemaStorageManager.getHistorySchemaStr`+`SerDeHelper.parseSchemas` 取全版本；非 evolution 取单条 base convert 版本（0）。③`HudiScanPlanProvider.getScanNodeProperties` 建发 base64 `hudi.schema_evolution` prop，**force_jni gate off**（全 JNI 时 BE 不查字典=paimon 门控平价；hudi 无 per-handle force_jni，仅 session `isForceJniScannerEnabled`）；best-effort（build 失败 log+丢 prop→BE BY_NAME、绝不硬挂扫描）；override `populateScanLevelParams` 经 `HudiSchemaUtils.applySchemaEvolution` copy f25/26。**排序**：C4c（schema_id 安全前缀）已落→字典永不与 schema_id-unset 同现（禁的 `current==file==-1` 退化）。DORMANT（hudi 不在 `SPI_READY_TYPES`、句柄未分流）。BE engage+历史集完整性=e2e；字典组装+往返 same-loader 测（`HudiSchemaUtilsTest`+5）。全绿（hudi 150 测 + checkstyle 0 + import gate 净）。**⇒ Part 1（C4a–C4d）实现收官；统一对抗复审 DONE（`wf_33882086-fbb`）+修 `ad8aa87bd2d`（见顶部 🎯）；下一步 = Part 2 C5a。** - - **✅ HD-C5a DONE（`af9a3c1a004`，休眠，Part 2 首步）= schema-at-instant 列表（3-arg `getTableSchema` 覆写）**。post-flip 通用路径下给 `FOR TIME AS OF` 演进读供**钉时刻**的列表，闭合 HD-C2 残留 #1（SPI 默认曾返最新、忽略 pin）。①新 3-arg `getTableSchema(session,handle,ConnectorMvccSnapshot)` 覆写 keyed off `HudiTableHandle.getQueryInstant()`（`applySnapshot` 钉的 pin），**非** `snapshot.getSchemaId()`（hudi 恒 -1；fe-core `PluginDrivenMvccExternalTable:375-376` 传 post-applySnapshot 句柄故 instant 已在其上）；null instant（plain / 非-FOR-TIME pin 如 @incr，其 loadSnapshot 分支自列最新 schema、根本不到此）经**同一** shared `buildTableSchema(handle,instant)` 走 latest→latest/at-instant 不发散。②`getSchemaFromMetaClient(basePath)`→`getSchemaFromMetaClient(basePath,queryInstant)`，**包进 `metaClientExecutor.execute`**（plugin UGI doAs+TCCL pin）闭合 C4b 欠的未包裹 auth/TCCL gap（getTableSchema 可跑在非 pin 的 scan 线程外，如 catalog metadata load）；at-instant 臂字节忠实 legacy `HiveMetaStoreClientHelper.getHudiTableSchema`+`HMSExternalTable.initHudiSchema`：`reloadActiveTimeline()`+`getTableInternalSchemaFromCommitMetadata(instant)`→present（schema.on.read 开）经 `columnsFromInternalSchema`（`AvroInternalSchemaConverter.convert(InternalSchema,name)`→avro→`avroSchemaToColumns`→`attachTopLevelFieldIds` 稳定 id）→改名列显历史名、BE 按 field id 匹老文件；absent（演进关）落 latest fallback＝非演进表字节等价（D3）。③用 `HudiScanPlanProvider.buildMetaClient`（与旧 inline build 字节等价）；hive 3-arg 委派已发（`HiveConnectorMetadata:1071`）无须改；连接器编 unshaded hudi-common 故 `convert` 回 unshaded `org.apache.avro.Schema` 与 `avroSchemaToColumns` 相容。零 fe-core import、零新连接器-api SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、句柄未分流；live iceberg/paimon 不走 hudi getTableSchema）。at-instant **读**本身 e2e；`HudiSchemaAtInstantTest` 3 same-loader 钉 instant-threading（2-arg=latest / 3-arg 无 pin=latest / 3-arg 有 pin=at-instant，keyed off 句柄非快照〔snapshot 传 null 证之〕）。**4 维对抗复审 `wf_703ea26e-49f`（legacy-parity/no-regression-dormancy/routing-handle-vs-snapshot/exec-shading-exceptions/test-intent）+逐条 refute：0 确认缺陷**（1 nit＝wrapping 后 LOG.warn 只印 wrapper message refuted〔旧码亦只印 `e.getMessage()` 非本步引入〕；顺带把 `e.getMessage()`→传 throwable 补全 cause 链+栈）。全绿（hudi 156 测〔+3〕 + checkstyle 0 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进列表（改名/调序显历史名、非演进 latest 等价、meta 列集与 legacy 一致）只 e2e 可证（单测覆盖 routing）；schema-on-read commit-metadata InternalSchema 是否含 `_hoodie_*` meta 列＝随 legacy 定、e2e 核。**下一步 = C5b（扫描侧 MOR/JNI 列表@instant + 字典 `-1` 条按 pinned schema 重解）。** - - **✅ HD-C5b DONE（`e0da87888e5`，休眠，Part 2 收官）= 扫描侧 schema-at-instant（MOR/JNI 列表 + 原生 `-1` overlay @instant）**。`FOR TIME AS OF` over schema-on-read 表：两个扫描侧 schema locus 都须钉时刻、锁步。**(i) planScan JNI（MOR-realtime）`column_names`/`column_types`**＝pin 时刻的 schema（legacy `HudiScanNode:222-224` `getSchemaCacheValue(hmsTable,queryInstant)`），经新 `HudiSchemaUtils.resolveJniColumnSchema`（路由）→纯 `chooseJniSchema`（pinned present→`AvroInternalSchemaConverter.convert(pinnedIS,"hudi_table")`；else→latestAvro）。**plain read（`queryInstant==null`）短路在 `resolveInternalSchemaAtInstant` 之前→不 reload timeline＝字节不变**；非演进（instant 无 commit-metadata InternalSchema）→latest==at-instant（D3）。`avroSchema` 局部保 latest＝per-file schema_id resolver 的 base（分类每文件自身写入版本，pin-independent）。**(ii) getScanNodeProperties 原生 `-1` overlay** 从**完整 pinned InternalSchema** 重建（新 `buildSchemaEvolutionDictAtInstant`＝`buildSchemaEvolutionDict(emptyList, pinnedSchema, true, allHistoricalSchemas)`＝empty-requested 路径 over pinned schema），**非** latest-keyed 请求句柄。**为何 full-pinned**：`getColumnHandles` 在 MVCC pin 前跑（`PluginDrivenScanNode:1534`＝latest-keyed 句柄图），FOR TIME 下 scan slots 按 pinned schema（C5a 3-arg，`PluginDrivenMvccExternalTable:376`）绑＝pinned 名；`buildColumnHandles` 用 slot(pinned 名) 查 latest 句柄图→改名列 miss→**从 `columns` 掉**；用 `columns` 建 overlay 会漏该 scan slot→BE 崩。**BE 契约亲手核对**（决定性）：v1 reader `by_table_field_id` 建 StructNode 每 overlay 字段一子、reader 按 **scan slot 名** `children_column_exists`＝`DCHECK(children.contains)`/`children.at`（`table_schema_change_helper.h:165-167`）→**scan slot 缺 overlay＝std::out_of_range/SIGABRT；overlay⊇scan slots＝必需**；format_v2 `find_external_root_field`（`table_reader.cpp:254-283`）按名查投影列、忽略多余＝**超集容忍**。⇒ full-pinned（全 pinned 字段）是 scan slots 超集＝双 reader 安全，用句柄建的子集会崩。`current_schema_id` 保 -1（须≠每 split id≥0）；史=全版本（Option B，C4d 同）；**无 meta 列可达 pin 路**（C5a schema-on-read 绑定源自 InternalSchema、无 `_hoodie_*`→查询绑不到 meta 列）。@incr（begin/endInstant 非 queryInstant）→pin==null→latest dict＝列最新 schema（§7.7）。零 fe-core、零新 SPI。DORMANT（hms 不在 `SPI_READY_TYPES`、hudi 句柄未分流）。**4 维对抗复审 `wf_ec6d47e3-37e`（legacy-parity·at-instant / BE-contract·SIGABRT / no-regression·dormancy / test-intent）+逐条 refute：0 生产缺陷**（3 raised→1 survived〔confirmed-minor 测注释诚实性：`atInstantOverlayCarriesFullPinnedSchemaWithHistoricalNames` 测直调 `buildSchemaEvolutionDict(emptyList,…)`、守**纯** full-pinned 组装非 provider 路由——路由须 live metaClient＝不可同加载器测、e2e-owed，已重写注释诚实标注〕→已修；2 refuted〔另一测覆盖提醒/设计文档措辞 nit〕）。`HudiSchemaUtilsTest`+3（JNI no-pin no-op〔null 短路不碰 metaClient〕/ chooseJniSchema pinned 历史名 / full-pinned overlay 历史名+全版本）。全绿（hudi 159 测 + checkstyle 0〔最长注释 116<130〕 + import gate 净）。**⚠残留（e2e/翻闸，非静默）**：at-instant 演进读正确性（改名 COW native BY_FIELD_ID + MOR JNI 列表@instant、非演进 FOR TIME latest 等价、dropped-partition-set-at-instant〔HD-C2 残留②〕）+ **provider pin-routing**（`if(pinnedSchema.isPresent())` 分支 + `resolveInternalSchemaAtInstant` reload/resolve）**只 e2e 可证**（同加载器无 live metaClient；离线测覆盖纯 `chooseJniSchema`/full-pinned 组装）。**⇒ Part 2（C5a–C5b）+ HD-C4/C5 schema evolution（DV-006）全部收官。下一步 = Group D 写-拒安全网（HD-D1）→ HD-B2 getTableHandle 分流 HUDI（武装 pivot，最后一步）。** - - **✅ HD-D1 DONE（`7ae9404af25`，休眠，Group D 收官）= 只读写-拒安全网**（recon 对照真实代码校正规划：**规划 §HD-D1「让 getWritePlanProvider 抛显式只读」写于 HD-B1 之前、已过时**）。翻闸后 hudi 表→通用 `PluginDrivenExternalTable`+外来句柄→网关三路分流（`resolveSiblingOwner` peek）到 **hudi 兄弟连接器**（非 iceberg），故本连接器答所有写/过程/DDL/事务调用。**核心发现：拒绝主体已现成**——HD-B1 正确路由（hudi→hudi 兄弟、绝不 iceberg）+ connector-api SPI 默认：`getWritePlanProvider(handle)`=null→`supportedWriteOperations(handle)`=空→`PhysicalPlanTranslator:657/709` admission gate **规划期**抛干净 `AnalysisException`「does not support INSERT/row-level DML」；`getProcedureOps`=null→EXECUTE 拒；`ConnectorTableOps` 全 DDL 默认抛「… not supported」；`beginTransaction` 默认抛。**用户签字 Q1=方案 A**（保留干净拒绝+测试锁定+事务层防御，**不**让 getWritePlanProvider 抛——admission gate 由它派生，在其上抛→引擎误分类「maybe palo bug」而非干净「does not support INSERT」；已核对 fe-core 写路径顺序：gate 先于 getWritePlanProvider 先于 beginTransaction，且连接器 beginTransaction 只在**执行期**〔`PluginDrivenInsertExecutor`/`ConnectorRewriteDriver:130`〕被调、规划期已拒故对 hudi 写是死码）。**用户签字 Q2=方案乙**（严格拒绝**所有** DDL 含 `DROP TABLE`/`RENAME TABLE`——相对 legacy HMS〔经 `HiveMetadataOps` 支持删/改名 hudi 表 HMS 条目〕是**有意行为倒退**、hudi 完全外部管理）。**净新增（唯一代码改动）=override `HudiConnectorMetadata.beginTransaction(session)` 抛显式** `"Hudi tables are read-only in this catalog; INSERT/UPDATE/DELETE/MERGE are not supported"`（替换泛型默认；per-handle 重载 SPI 默认委派到它；纯消息质量末线兜底、不劣化 gate 分类）。`HudiReadOnlyWriteRejectTest`〔`new HudiConnectorMetadata(null,{},null)`＝write/DDL 抛前不 deref 故 null 够〕锁：无 writer+空写操作集〔=写不被误当 iceberg 执行〕/无过程/beginTransaction **双重载**〔no-arg + 生产路由的 per-handle〕均抛显式只读/全 DDL 变更族〔dropTable·renameTable·truncate + addColumn(s)·dropColumn·renameColumn·modifyColumn·reorderColumns·create/drop Branch|Tag·add/drop/replace PartitionField〕抛。**3 维对抗复审 `wf_36fd1c6c-e58`（reject-completeness·routing / classification·no-regression / test-intent）+逐条 refute：0 生产缺陷**；6 raised→2 survived nit（均测强化：per-handle beginTransaction 重载直测 + 全 DDL 族补锁，已修）；refuted：`DROP DATABASE … FORCE` 级联〔pre-existing 网关码、legacy 平价、超 per-table 范围〕/ 休眠〔HD-B2 未分流 HUDI＝设计延后〕/ beginTransaction 非可写性锁〔可写性由 `supportedWriteOperations` 空锁，已断言〕。全绿（hudi 163 测 + checkstyle 0 + import gate 净）。**⚠残留（翻闸/e2e，非静默）**：①`DROP DATABASE … FORCE` 对含 hudi 表的库仍按名 `hmsClient.dropTable` 级联删 HMS 登记〔`HiveConnectorMetadata.dropDatabase` 无 per-source 分支、iceberg-on-HMS 同款、只删登记不删数据；翻闸若要 per-table 拒须网关加分流，S6 findings 已记〕；②写/DDL 只读端到端只在 HD-B2 分流 HUDI 后 live（e2e 断言异构 HMS 目录 hudi INSERT/DELETE/MERGE/EXECUTE/ALTER 全 fail-loud，memory `hms-iceberg-delegation-needs-e2e`）。**✅ HD-B2 = getTableHandle 分流 HUDI 收官（`5b1409f9f7c`，武装 pivot，最后一个连接器步）：getTableHandle 加 HUDI 臂分流到 hudi 兄弟（镜像 iceberg S4、复用 getOrCreateHudiSibling、外来句柄原样返/永不强转、iceberg 优先/UNKNOWN fail-loud 不吞）；4 维对抗复审 0 生产缺陷 + 1 minor 接线守缺口已修（抽 `newMetadata(HmsClient)` 缝 + ThreeWayRoutingTest 变异验证守）。详见上 🎯 段 HD-B2 条目。**⇒ 全部连接器休眠步收官 ⇒ 下一步 = 翻闸本身（原子加 `"hms"` + 删除单元 + 翻闸 e2e，须用户定打包/翻闸序）。** - - 关键结论： - - **翻闸是原子的——加 `"hms"` 一行同时切 hive+iceberg-on-HMS+hudi-on-HMS**，故 hudi 无回退补齐**阻塞整个翻闸**（iceberg 脊柱做完还不够）。 - - **PhysicalHudiScan 难点已解**：翻闸后 hudi-on-HMS 是通用 `PluginDrivenExternalTable`→`LogicalFileScan`→通用 `PluginDrivenScanNode`（与 iceberg 同路径）；`visitPhysicalHudiScan` 的 plugin 分支翻闸后**死代码**。增量/时间旅行经**已存在**的 `resolveTimeTravel`/`applySnapshot`/`ConnectorTimeTravelSpec.Kind.INCREMENTAL` 缝重落连接器侧（paimon 已实现=模板）——**无需新 fe-core SPI**，纯连接器实现。**禁**给 `BindRelation`/`CheckPolicy` 加 `dlaType==HUDI`/`instanceof` 臂（铁律）。 - - **DV-005 模型错配已解**：`HudiConnectorProvider.getType()="hudi"` 保留但**只作兄弟类型串**（`createSiblingConnector("hudi",…)` 绕过 `SPI_READY_TYPES`）；**永不**把 `hudi` 加进 `SPI_READY_TYPES`/工厂 case（会重造 DV-005）。 - - **有序子步（HD-A0..HD-D1，4 组；✅Group A（HD-A0–A5）全部收官，见上）**：A=hudi 连接器变合格兄弟（含 **HD-A3 分区值解析保真=快照路径静默 NULL 分区列的真 bug**、Kerberos、BE 描述符/存储属性、force_jni）；B=网关委派（**HD-B1=唯一需签字新决策**：两兄弟后二元 `instanceof HiveTableHandle else iceberg` 失效，须三路由——**推荐 Option 1 `Connector.ownsHandle(handle)` 中立 SPI 默认**〔各兄弟自判、同加载器可单测〕vs Option 2 类加载器身份〔零 SPI 但不可同加载器单测〕；**会改动已落的 iceberg 活脊柱**，须保 iceberg 臂字节等价 + HD-B2=getTableHandle 分流 HUDI，武装 pivot、**最后一步**）；C=无回退补齐（HD-C1 MVCC/listPartitions、HD-C2 时间旅行、**HD-C3 增量读=最大风险**、HD-C4/C5 schema evolution/DV-006）；D=只读写-拒安全网（HD-D1）。 - - **✅ 最大风险 HD-C3（§5.1）已解（本 session）**：COW 增量行级 `_hoodie_commit_time > begin AND <= end` 谓词的正确性落点已判定 = **FE-only 中立 SPI、零 BE 改**（详见上 HD-C3 条目 + 设计文档）。侦察坐实 BE 两读取路径不自窗口过滤但**已应用扫描 conjunct**，故连接器供中立 SPI 谓词、fe-core 分析期注入 `LogicalFilter` 引用可见 `_hoodie_commit_time` 列即可，BE 无改。**旧交接「本 checkout 无 BE 树/可能无 fe-core 解」已证伪**（BE 树在、已读 reader 源码）。straddling-base-file fixture 仍是翻闸 e2e 硬门（离线单测证 routing/窗口解析/句柄穿透，行正确性只 e2e 可证）。paimon **非**先例（其 incremental 是 snapshot-delta 非行过滤）。 - - **翻闸硬门 B1–B6**（见 plan §4.2）+ 两非单测 harness（两加载器路由 fixture + 异构 HMS e2e，含 COW/MOR 快照·FOR TIME AS OF·@incr·schema-evolved·非 hive-style/转义分区·写-拒）。**schema 缺失 SIGABRT 方向**（§5.2）已由 critic 对 BE 核对**降级为窄回退非崩溃**（emit-nothing=安全基线走 BE 小写 name-match），但 BE 树不在本 checkout、须落 HD-C4 前复核。 -- **其余翻闸前休眠候选**：**§4.3 遗留1 `getNewestUpdateVersionOrTime`**（字典自动刷新探针，**被 D2 连接器缓存阻塞**）。 - - ⚠**W3 复审记账（休眠期不动，翻闸/后续验证）**：①**inert 3 方法留无参**靠"hive==iceberg 恰好相等"——若将来某 provider 改这 3 个值，网关会给 iceberg-on-HMS 错答（须届时补 per-handle 或加断言守恒）；②`connectorSupported*` 助手在 live iceberg 路径（`isRowIdInjectionTarget` 等，非休眠）**新增一次 `getTableHandle` 往返**（正常缓存命中=字节等价；句柄缺失降级 false，`materializeStaticPartitionValues` 降级会把静态分区字面量 NULL 填而非 materialize——仅并发 drop 边缘、查询本会失败，acceptable-by-design；须确认这些点 getTableHandle 恒缓存服务）。 -- **翻闸残留（写委派期不建，见分解文档 §3）**：`hms`→`SPI_READY_TYPES`（翻闸线）；**hudi-on-HMS 写平价（hudi flip-blocker）**——hudi 保 HiveTableHandle→W2 后误路由 hive 写 provider、`HiveConnectorTransaction` 未必拒非事务 hudi，须显式拒或 hudi 委派步补；hive ALTER-DDL/RENAME port；`createTable(format=iceberg)` 引擎路由 + `pluginCatalogTypeToEngine` 无 `hms`；`dropDatabase` force purge；§4.5 写链（read-ACID query-finish commit、`BindSink` 显式分区 INSERT 拒未 port 进插件、`Env.hiveTransactionMgr`、`BIND_BROKER_NAME`）。**§4.3 遗留1** = `getNewestUpdateVersionOrTime`（字典自动刷新探针，hive 走 max 分区时间；探针高频→须连接器缓存 D2 后落）。**CCE 边界同加载器单测测不出**（需两加载器 fixture 或翻闸重部署冒烟）。**翻闸集（原子）/删除单元/硬门见设计文档 §2/§5/§7，勿在翻闸前动**。 - - ⚠**视图翻闸期遗留（休眠期不暴露、勿现在改 fe-core，翻闸/耦合中和阶段统一处理；决策见设计文档 §6 D5/D6，已签字）**：(1) **D5=去掉查询期开关（对齐 Trino）**：翻闸时 `BindRelation.java:634` plugin-view 臂**不再**gate 任何 `enable_query_*_views`——`isView()` 即服务；两个老开关退化为一版 no-op（`@ConfField` 保留一版防升级报"未知配置"）；顺带中和 iceberg 时间旅行文案 + `:626-632` 陈旧注释。⚠**须与翻闸捆绑**：这会**同时**去掉 iceberg 视图的开关（iceberg 已上线、现受 `enable_query_iceberg_views` 管），是可见行为变更，别提前泄漏。铁律 OK（无 `if(engine)`/无 OR 两开关）。(2) **D6=接受解码后新输出**：SHOW CREATE VIEW 走 plugin 路径出 Presto-**解码**后的 `getViewText`（比 legacy raw 编码可读），翻闸时把 pin 了 raw 字节的 hive-view SHOW CREATE 基线改成解码文本。另 `$partitions` sys-table 是 **TVF-routed**（`PartitionsSysTable extends TvfSysTable`），SPI sys-table 面只支持 native——留 fe-core vs 建 native 归 §4.3 系统表子步。 - 3. ~~`listPartitions`~~ **✅ DONE**（见上 §4.2 DONE 条目；`68e9fc8835b`+`3540d1327b6`；决策=接受 -1）。 - 4. ~~`getCapabilities` 连接器级~~ **✅ DONE**（见上 §4.2 DONE 条目；`96953cd8eec`；声明 VIEW+AUTO_ANALYZE+METADATA_PRELOAD；**MVCC/SHOW_CREATE/NESTED 推迟非禁**——旧写"禁 MVCC(hive 非 MVCC)"已纠正，设计 §3 要求混合目录声明 MVCC）。`SUPPORTS_PASSTHROUGH_QUERY`（无 query() TVF）/`SUPPORTS_PARTITION_STATS`（SHOW PARTITIONS 只列名）确不声明。 - 5. ~~`HiveTableFormatDetector` 平价~~ **✅ DONE**（见上 §4.2 DONE 条目；`191449c7bd6`；3 LZO 格式 + getTableHandle fail-loud + view 短路）。 -- 后续子步：**§4.4 兄弟 SPI + 网关委派（下一步，D3 硬前置）**、§4.5 读-ACID 收尾 + 写前检查 + `BIND_BROKER_NAME` 搬家 + engine-map。（§4.2a 列统计 fast-path 已 DONE；§4.3 MVCC/MTMV 新鲜度已 DONE，其 `getNewestUpdateVersionOrTime` 字典探针 + 系统表两处遗留见上 §4.3 条目。）**翻闸集（原子）/删除单元/硬门 见设计文档 §2/§5/§7，勿在翻闸前动**。 - -## 开场要点（承接） -1. **先读设计文档 `hms-cutover-retype-design-2026-07-07.md`（权威计划）+ 两份 findings + 本文顶部 🎯 段**。剩余 HMS 迁移 = 一次原子翻闸 + 前置休眠补齐 + 末尾循环删除。 -2. **已提交勿回炒**：P7.1 / P7.3(`0b19506acfe`) / 按表 scan seam(`0923077fe67`) 全休眠；replay-CCE fix(`a6dc782d816`) 已合；设计文档 commit `5bfc55f6d59`。 -3. **纪律**：设计已签字 → **现在进入实现**：每子步独立休眠 commit（fresh recompile 杜绝 stale `.class`）→ 更新本 HANDOFF。上下文超 30% 找干净节点交接。**path-whitelist `git add`，严禁 `-A`**。铁律见 🧠 起步必读 #3（fe-core 不加 `if(format)`/`instanceof HMSExternal*`/`switch(dlaType)`；不解析属性；跨插件 pin TCCL）。 -4. **硬门 = ACID/事件/异构集成测试 + Kerberos-HMS 冒烟**（R-002 最大风险，需 live 路径，勿静默跳过——Rule 12）。full-ACID **写**继续硬拒；full-ACID + insert-only **读**在范围（已落地插件侧）。 +# 🧠 起步必读（读文档，别炒 git log 历史） + +1. **权威翻闸计划** = `tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。**SUPERSEDES** 07-07 doc §4/§5 的状态。 +2. **本步（Hive 缓存）设计** = `tasks/hive-connector-cache-step-design-2026-07-10.md`（TODO 全勾）。 +3. **补充权威**：`tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + 能力孪生 + 已签字 §6 D1–D6 决策，仍有效）；`tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 S0–S6 + CCE/TCCL 硬约束）；两份 findings（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）；hudi = `tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `tasks/hudi-schema-evolution-step-design-2026-07-09.md`（均 DONE）。 +4. **样板**：`tasks/P5-paimon-migration.md`（翻闸+删 legacy 全流程）、`tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 +5. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 --- # 📦 分支 / Commit 须知 -- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi` @ `8b391c7459d`）。PR base = `branch-catalog-spi`，**squash 合并**。**打包/复审策略（翻闸前/后、单 PR vs 分 PR）= 翻闸阶段的开放决策**（paimon 分 PR vs iceberg 合并 squash 两先例）。 -- **公开 tracking issue = apache/doris#65185**；P7 PR 应引用它。进度按已合入 `branch-catalog-spi` PR 口径。 -- **⚠️ path-whitelist `git add`，严禁 `git add -A`**（工作树有历史遗留 scratch：`*.bak`·`regression-test/conf/regression-conf.groovy` 明文 key·`.audit-scratch/`·`conf.cmy/`·`META-INF/`·`docker/...`·`plan-doc/reviews/P5-*`·`.claude/`·`failed-cases.out`——均**非本线程产物，勿混入任何 commit**）。 -- commit message：`[feat](catalog) …` / `[fix](catalog) …` / `[doc](catalog) …` 范式 + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每阶段/每条 fix = 独立 commit**；HANDOFF + 设计/findings 文档单独 commit（与 code 分开）。 +- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi`）。PR base = `branch-catalog-spi`，**squash 合并**。打包/复审策略（单 PR vs 分 PR）= 翻闸阶段的开放决策。 +- **公开 tracking issue = apache/doris#65185**（进度按已合入 `branch-catalog-spi` PR 口径）。 +- **⚠️ path-whitelist `git add`，严禁 `git add -A`**：工作树有大量历史遗留 scratch（`*.bak` / `regression-test/conf/regression-conf.groovy` 明文 key / `.audit-scratch/` / `conf.cmy/` / `META-INF/` / `docker/...` / `plan-doc/reviews/P5-*` / `.claude/` / `failed-cases.out`——**非本线程产物，勿混入任何 commit**）。 +- commit message：`[feat|fix|doc](catalog) …` + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每子步/每条 fix = 独立 commit**；HANDOFF + 设计文档单独 commit（与 code 分开）。上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 -# ⚙️ 操作须知（复用） +# ⚙️ 操作须知（构建/测试，复用） -- maven：`-f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl : **-am** -DfailIfNoTests=false -Dmaven.build.cache.enabled=false`（漏 `-am`→假错 `${revision}`）。跑单测加 `-Dtest=`。**checkstyle 别加 `-am`**：`mvn -pl : checkstyle:check`。artifactId：`fe-core` / `fe-connector-api` / `fe-connector-hive` / `fe-connector-hms` / `fe-connector-iceberg` / `fe-connector-hudi`。 -- **⚠️ bash 工具默认 timeout 120s**：fe-core 全量编译 ~1.5–2min → **务必**把 `timeout` 调到 ~580000ms。**后台/管道 exit 不可信**——读 LOG 的 `BUILD SUCCESS` 行 + surefire `Tests run=/Failures=/Errors=`。改代码后 commit 前 fresh recompile 杜绝 stale `.class`。 -- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`mockStatic(Env.class)` 是本仓惯用法，191 处；stub `Env.getCurrentEnv()`→自定义 `TestingEnv extends Env{super(true)}` 覆写 getter，见 `ExternalMetaCacheRouteResolverTest` / 本轮 `ExternalMetaIdMgrTest`）。⚠️ `Mockito.mock(接口)` **不跑 default 方法**（返 null）。checkstyle **禁 static import**、**扫 test 源**。 -- 连接器禁 import fe-core：`bash tools/check-connector-imports.sh`（**repo 根**，非 `fe/`）。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory）。 -- **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 紧**（~82% used）；**勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 +- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-connector/ **-am** test -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**；.m2 里有坏 `${revision}` 目录）。单测加 `-Dtest=`。禁 build-cache 才真跑 surefire。 +- **checkstyle 单独跑**：`mvn -o ... -pl fe-connector/ -am checkstyle:check`（module `test` 阶段已含 checkstyle，BUILD SUCCESS 即过；仍建议显式确认「0 violations」——Rule 12）。 +- **import gate**（连接器禁 import fe-core）：`bash tools/check-connector-imports.sh`（**repo 根**跑）。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 +- **⚠️ bash 工具默认 timeout 120s**：fe-core 全量编译 ~2–3min、`-am` 更久 → `timeout` 调到 ~580000+ 或后台跑；**后台/管道 exit 不可信**——读 LOG 的 `BUILD SUCCESS` + surefire `Tests run=/Failures=/Errors=`（memory `doris-build-verify-gotchas`）。 +- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`Mockito.mock(接口)` **不跑 default 方法**返 null；`mockStatic(Env)` 是本仓惯用法）。checkstyle **禁 static import**、**扫 test 源**（memory `catalog-spi-fe-core-test-infra`）。 +- **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 盘紧（~82%）；勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 -# 🧠 起步必读 +# 🔒 铁律（fe-core 约束，翻闸靠"表类=通用类 + 网关按句柄委派"） -1. **权威计划 = `hms-cutover-retype-design-2026-07-07.md`**（原子翻闸模型 + 休眠补齐清单 §4 + 能力孪生图 + 阶段序 §5 + 已签字决策 §6 + 硬门 §7；全 recon 明细在 `tool-results/w0bg9i509.output`）。**§4.4 兄弟连接器/委派的权威分解 = `hms-cutover-sibling-connector-decomposition-2026-07-08.md`**（S0–S6 序 + 更正设计 §4.4 包装句柄为死方案 + CCE/TCCL/生命周期硬约束；§4.4a=S0 已 DONE）。再读**两份 findings 文档**（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）+ 本文顶部 🎯/🚀。 -2. **样板**：`P5-paimon-migration.md`（翻闸+删 legacy 全流程）；`P6-iceberg-migration.md` + `P6.6-iceberg-flip-blockers-tasklist.md`（净室复审 + 能力孪生审计 + GSON replay 范式）。委派/缝模板 = A 的设计文档 `P7.4-scan-provider-per-table-seam-design.md`。特性地图 = `P7-cutover-scope-map-2026-07-06.md`（排序前提已校正）。 -3. **铁律**：fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/`switch(dlaType)`/引擎名判别（翻闸靠"表类=通用类 + 网关按句柄委派"，不靠在 `PhysicalPlanTranslator` 加分支）；fe-core 不解析属性（memory `catalog-spi-no-property-parsing-in-fecore`）；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）；跨插件/跨边界 pin TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha`，事件轮询后台线程 R-010 亦需）；history_schema_info nested 名 lowercase（memory）。 -4. **memory 相关项**：`handoff-discipline-per-phase`、`clean-room-adversarial-review-pref`、`ask-user-explain-in-chinese-first`、`session-handoff-at-30pct-context`、`doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`、`catalog-spi-tracking-issue`。 - ---- +- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点保持 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。 +- fe-core **不解析属性**（storage→fe-filesystem、meta→fe-connector，均插件侧；memory `catalog-spi-no-property-parsing-in-fecore`）。 +- 跨插件/跨边界**须 pin TCCL** 到连接器 classloader（扫描线程 / 写-DDL 引擎线程 / iceberg 内部 worker 池 / 事件轮询后台线程；memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 +- `history_schema_info` 嵌套字段名逐层 lowercase（否则 BE SIGABRT；memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 +- `PluginDrivenMvccExternalTable` 是 paimon/iceberg **实时**基类：改其共享方法须对二者字节+成本双不变（memory `plugindriven-mvcc-table-is-live-not-dormant`）。 -## 背景：跨连接器删除排序（翻闸最硬约束） +# 🗂 memory 相关项 -`datasource/hive/` **删不掉**，直到非-hive 消费者全 retype 到 generic：`datasource/hudi/HudiUtils`/`HudiScanNode`(extends `HiveScanNode`)/`HudiExternalMetaCache`；`datasource/iceberg/source/IcebergHMSSource`、`statistics/HMSAnalysisTask`、`statistics/util/StatisticsUtil.getIcebergColumnStats`、`datasource/systable/IcebergSysTable`。P6 #64688 删的是原生 iceberg，但 iceberg-on-HMS 仍走 fe-core，故 `datasource/iceberg/` 还保活 ~23 个 HMS-iceberg 支撑类（两 tier，见委派 findings 文档）——翻闸把它们切到连接器路径后才能删。同理 `datasource/hudi/`、`datasource/hive/`。整条 catalog-SPI 阶段链已合入 upstream `branch-catalog-spi`：P0 #63582 · P1 #63641 · P2 #64096 · P3 #64143 · P4 #64300 · P5 #64446+#64653 · P3b #64655 · P6 #64688。 +`handoff-discipline-per-phase` · `clean-room-adversarial-review-pref` · `ask-user-explain-in-chinese-first` · `session-handoff-at-30pct-context` · `doris-build-verify-gotchas` · `catalog-spi-fe-core-test-infra` · `catalog-spi-tracking-issue` · `hms-iceberg-delegation-needs-e2e` · `catalog-spi-connector-cache-framework-caffeine-coherence` · `memory-keep-only-general-or-requested`。 From 4d08eec6412528e73b2d740e4c641d0651b710d7 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 10:36:19 +0800 Subject: [PATCH 158/334] [fix](catalog) Hive connector cache: restore fail-loud on FileSystem.get, keep per-directory skip (dormant) Clean-room re-review finding (fix): the directory-listing cache folded TWO distinct failure modes into one skip-with-warn path, silently coarsening a fail-loud storage error into a wrong result. Before the listing cache, HiveScanPlanProvider.listAndSplitFiles ran FileSystem.get OUTSIDE its inner try: a FileSystem.get failure (unknown scheme, bad credentials/endpoint) propagated as IOException and the caller re-threw it as DorisConnectorException = the query failed loud; only a listStatus failure (this partition dir missing/unreadable) was skip-with-warn. The cache's listFromFileSystem wrapped BOTH in one try -> DorisConnectorException, and the scan caught DorisConnectorException and skipped. Since all partitions of a table share one scheme+authority, a FileSystem.get failure is SYSTEMIC (fails for every partition) -> the scan silently returned an empty/partial result with no error where legacy aborted the query. The documented mitigation ("still fails loud via the row-count estimate") does not hold: estimateDataSize catches RuntimeException and returns -1 (never throws to the user), and for a table that already has HMS stats the estimate path is not even invoked. Restore the legacy distinction by blast radius: - FileSystem.get failure -> plain DorisConnectorException (SYSTEMIC, fail loud). The scan path no longer catches this, so it propagates and aborts the query. - listStatus failure -> new HiveDirectoryListingException extends DorisConnectorException (LOCAL, skippable). listAndSplitFiles catches ONLY this subtype and skips the one partition with a warning (pre-cache parity). The estimate path still degrades to -1 on either (legacy best-effort parity). Neither failure is cached (MetaCacheEntry never caches a failed load). Dormant: "hms" is not in SPI_READY_TYPES, so no live catalog builds a HiveConnector; byte-neutral for every other connector. Tests: HiveFileListingCacheTest +6 (9 -> 15) -- listFromFileSystem throws a plain (loud) DorisConnectorException on an unresolvable scheme and NOT the skippable subtype, and HiveDirectoryListingException on a missing directory; the SAME two failures driven through the cache keep their type across the manual-miss boundary AND leave NO cache entry (kills a "catch -> return empty" mutation that would poison the listing for the TTL); the scan SKIPS only the failed partition and still plans the healthy one (per-partition tolerance), and FAILS LOUD (propagates) on a systemic DorisConnectorException. Full connector suite green + checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../hive/HiveDirectoryListingException.java | 41 ++++++ .../connector/hive/HiveFileListingCache.java | 38 ++++-- .../connector/hive/HiveScanPlanProvider.java | 12 +- .../hive/HiveFileListingCacheTest.java | 121 +++++++++++++++++- 4 files changed, 196 insertions(+), 16 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveDirectoryListingException.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveDirectoryListingException.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveDirectoryListingException.java new file mode 100644 index 00000000000000..158e11ecd8e284 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveDirectoryListingException.java @@ -0,0 +1,41 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.DorisConnectorException; + +/** + * Thrown when listing ONE partition directory fails ({@code FileSystem.listStatus} raised an + * {@code IOException}: a missing / unreadable / transiently-failing directory). This is a local + * failure that the scan path tolerates by skipping that partition with a warning — the pre-cache resilience + * (legacy {@code HiveScanPlanProvider.listAndSplitFiles} caught the {@code listStatus} {@code IOException} + * and skipped the partition). + * + *

    It is deliberately a distinct subtype of {@link DorisConnectorException} so the scan path can catch + * only this (skip) while letting a plain {@link DorisConnectorException} — used for a systemic + * filesystem-resolution failure ({@code FileSystem.get}: unknown scheme, bad credentials/endpoint, which + * affects every partition of the table) — propagate and fail the query loud, exactly as legacy did before the + * listing cache folded the two failure modes together. Never cached (the cache loader throws, and + * {@code MetaCacheEntry} never caches a failed load).

    + */ +public class HiveDirectoryListingException extends DorisConnectorException { + + public HiveDirectoryListingException(String message, Throwable cause) { + super(message, cause); + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java index 34d2ff80e0c85d..78ebc3566a48aa 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -57,10 +57,15 @@ * itself). A background refresh thread would NOT inherit that pin and Hadoop's {@code FileSystem} reflection would * fail to resolve its impl.

    * - *

    Failures are not cached. The loader throws {@link DorisConnectorException} on any I/O error (never - * caching a transient failure, matching {@code MetaCacheEntry}'s null-is-a-miss / exception-propagates contract). - * The scan path catches it and skips the partition with a warning (the pre-cache tolerance of a missing/unreadable - * partition directory); the estimate path lets it propagate and degrades to {@code -1}.

    + *

    Failures are not cached, and are split by blast radius. The loader never caches a failed load (matching + * {@code MetaCacheEntry}'s null-is-a-miss / exception-propagates contract). A SYSTEMIC filesystem-resolution failure + * ({@code FileSystem.get}: unknown scheme, bad credentials/endpoint — it fails for every partition of the table) is + * thrown as a plain {@link DorisConnectorException} and the scan path lets it propagate to fail the query loud. A + * LOCAL per-directory failure ({@code listStatus}: this partition missing/unreadable) is thrown as + * {@link HiveDirectoryListingException} and the scan path skips just that partition with a warning (the pre-cache + * tolerance). The estimate path degrades to {@code -1} on either. This keeps a broken storage config from silently + * returning an empty scan (legacy failed {@code FileSystem.get} loud) while preserving one-bad-partition + * resilience.

    * *

    Dormant. {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog builds a * {@link HiveConnector}, so this cache is never instantiated for a live catalog until the flip. Byte-neutral for @@ -140,14 +145,24 @@ long size() { * directories and {@code _}/{@code .}-prefixed hidden files (byte-parity with the pre-cache filters in * {@code HiveScanPlanProvider.listAndSplitFiles} and {@code HiveConnectorMetadata.sumFileSizesUnder}). A zero * -length data file is kept (the scan splitter skips it; the size estimate adds 0) so both consumers keep - * their exact prior behaviour. Any {@link IOException} (unresolvable filesystem or unreadable directory) is - * wrapped in a {@link DorisConnectorException} so the entry is never cached and the caller decides how to - * degrade. + * their exact prior behaviour. A {@code FileSystem.get} failure (unresolvable filesystem — unknown scheme, bad + * credentials/endpoint) is SYSTEMIC and wrapped in a plain {@link DorisConnectorException} so the scan fails + * loud; a {@code listStatus} failure (this partition directory missing/unreadable) is LOCAL and wrapped in + * {@link HiveDirectoryListingException} so the scan skips just that partition. Neither is ever cached. */ static List listFromFileSystem(String location, Configuration conf) { + Path path = new Path(location); + FileSystem fs; + try { + fs = FileSystem.get(path.toUri(), conf); + } catch (IOException e) { + // Filesystem resolution/initialization failed (unknown scheme, bad credentials/endpoint): a SYSTEMIC + // storage-config error affecting every partition of the table. Fail loud with a plain + // DorisConnectorException (the scan path does NOT skip this), matching the pre-cache behavior where a + // FileSystem.get failure aborted the query instead of silently returning an empty scan. + throw new DorisConnectorException("Failed to resolve filesystem for " + location, e); + } try { - Path path = new Path(location); - FileSystem fs = FileSystem.get(path.toUri(), conf); FileStatus[] statuses = fs.listStatus(path); List files = new ArrayList<>(statuses.length); for (FileStatus status : statuses) { @@ -163,7 +178,10 @@ static List listFromFileSystem(String location, Configuration co } return files; } catch (IOException e) { - throw new DorisConnectorException("Failed to list files under " + location, e); + // Listing THIS partition directory failed (missing / unreadable / transient): a LOCAL failure the scan + // path tolerates by skipping the partition with a warning (pre-cache parity). Distinct exception type so + // only this is skipped, while the systemic FileSystem.get failure above stays loud. + throw new HiveDirectoryListingException("Failed to list files under " + location, e); } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index d854f2a41f1a15..8c302e3a45539a 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -351,10 +351,12 @@ private List convertPartitions( /** * Lists the data files of a partition directory (through the connector's shared {@link HiveFileListingCache}, * which filters directories and {@code _}/{@code .}-prefixed hidden files) and splits them into scan ranges. - * A listing failure is tolerated — the partition is skipped with a warning, the same resilience the pre-cache - * code gave a missing/unreadable partition directory (and it consolidates the earlier {@code FileSystem.get} - * failure into the same skip path; a bad storage config surfaces via the row-count estimate's fail-loud and - * the docker e2e). Failures are never cached (the cache loader throws). + * A LOCAL per-directory listing failure ({@link HiveDirectoryListingException}) is tolerated — the partition is + * skipped with a warning, the same resilience the pre-cache code gave a missing/unreadable partition directory. + * A SYSTEMIC filesystem-resolution failure (a plain {@link DorisConnectorException} from {@code FileSystem.get}, + * which affects every partition) is NOT caught here: it propagates to fail the query loud, matching the pre-cache + * behavior where a {@code FileSystem.get} failure aborted the query instead of silently returning an empty scan. + * Failures are never cached (the cache loader throws). */ private void listAndSplitFiles(String dbName, String tableName, PartitionScanInfo partition, HiveFileFormat fileFormat, @@ -363,7 +365,7 @@ private void listAndSplitFiles(String dbName, String tableName, List files; try { files = fileListingCache.listDataFiles(dbName, tableName, partition.location, conf); - } catch (DorisConnectorException e) { + } catch (HiveDirectoryListingException e) { LOG.warn("Cannot list files in partition: {}", partition.location, e); return; } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java index 513a9e07d90bce..b78213fea2478c 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java @@ -28,6 +28,7 @@ import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; +import java.io.IOException; import java.nio.file.Files; import java.util.ArrayList; import java.util.Arrays; @@ -193,6 +194,122 @@ public void listFromFileSystemFiltersDirectoriesAndHiddenFiles(@TempDir java.nio } } + // ==================== failure split: systemic (FileSystem.get) is loud, local (listStatus) is skippable ====== + + @Test + public void listFromFileSystemFailsLoudWhenFilesystemCannotBeResolved() { + // An unknown scheme makes FileSystem.get fail (no FileSystem for scheme) — a SYSTEMIC storage-config error + // that affects every partition of the table. It must throw a plain DorisConnectorException (which the scan + // path does NOT skip), and NOT the skippable HiveDirectoryListingException. + // WHY (Rule 9): if this were the skippable subtype, a misconfigured storage would silently return an empty + // scan instead of failing the query loud — the exact regression this fix prevents. + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> HiveFileListingCache.listFromFileSystem("nosuchscheme://host/path", CONF)); + Assertions.assertFalse(e instanceof HiveDirectoryListingException, + "a filesystem-resolution failure must be loud (plain DorisConnectorException), not skippable"); + } + + @Test + public void listFromFileSystemIsSkippableWhenDirectoryMissing(@TempDir java.nio.file.Path dir) { + // A resolvable filesystem (file://) but a non-existent directory makes listStatus fail — a LOCAL failure of + // one partition directory. It must throw the skippable HiveDirectoryListingException so the scan skips just + // that partition (pre-cache tolerance of a missing/unreadable partition dir). + String missing = dir.resolve("does-not-exist").toUri().toString(); + Assertions.assertThrows(HiveDirectoryListingException.class, + () -> HiveFileListingCache.listFromFileSystem(missing, CONF)); + } + + @Test + public void getFailureThroughCacheFailsLoudAndIsNotCached() { + // Drives the REAL production lister THROUGH the cache lookup with an unresolvable scheme, so + // FileSystem.get genuinely fails. WHY (Rule 9): pins that (1) a systemic storage-config failure propagates + // from listDataFiles as the loud plain DorisConnectorException — MetaCacheEntry's manual-miss load rethrows + // RuntimeException unwrapped, so the type survives the cache boundary — and NOT the skippable subtype; and + // (2) the failure leaves NO cache entry. (2) kills the mutation "catch -> return emptyList" in the loader, + // which would cache a poisoned empty listing and silently turn every later scan into 0 rows for the TTL. + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap()); + + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> cache.listDataFiles("db", "t", "nosuchfs://host/path", CONF)); + Assertions.assertFalse(e instanceof HiveDirectoryListingException, + "a filesystem-resolution failure through the cache must be the loud type, not the skippable one"); + Assertions.assertEquals(0L, cache.size(), "a failed load must never leave a cache entry"); + } + + @Test + public void listStatusFailureThroughCacheIsSkippableAndNotCached(@TempDir java.nio.file.Path dir) { + // Drives the REAL production lister THROUGH the cache with a resolvable filesystem (file://) but a missing + // directory, so listStatus genuinely fails. WHY (Rule 9): pins that the LOCAL per-directory failure keeps + // its distinct skippable type (HiveDirectoryListingException, exactly what the scan's per-partition skip + // catches) across the cache boundary, and leaves no cache entry either. + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap()); + String missing = dir.resolve("does-not-exist").toUri().toString(); + + Assertions.assertThrows(HiveDirectoryListingException.class, + () -> cache.listDataFiles("db", "t", missing, CONF)); + Assertions.assertEquals(0L, cache.size(), "a failed load must never leave a cache entry"); + } + + @Test + public void scanSkipsOnlyTheFailedPartitionAndPlansTheRest() { + // A LOCAL per-directory listing failure (HiveDirectoryListingException) on ONE partition among several + // must be tolerated PER PARTITION: the failed one is skipped with a warning, every other partition still + // plans its splits — the pre-cache resilience. WHY (Rule 9): if the scan aborted on the subtype, or the + // skip were scan-wide instead of per-partition, one bad directory would lose the whole table. + CountingLister lister = new CountingLister(); + lister.error = new HiveDirectoryListingException("dir gone", new IOException("boom")); + lister.errorLocation = "loc/dt=1"; + HiveScanPlanProvider provider = new HiveScanPlanProvider(null, Collections.emptyMap(), + new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); + + List ranges = provider.planScan( + new FakeSession(), twoPartitionHandle(), Collections.emptyList(), + Optional.empty()); + + // Both partitions were attempted; only the healthy one produced its (one-file, one-range) split. + Assertions.assertEquals(1, (int) lister.callsPerLocation.get("loc/dt=1")); + Assertions.assertEquals(1, (int) lister.callsPerLocation.get("loc/dt=2")); + Assertions.assertEquals(1, ranges.size(), + "the failed partition is skipped, the healthy partition must still plan its split"); + } + + @Test + public void scanFailsLoudOnSystemicFilesystemFailure() { + // A SYSTEMIC filesystem-resolution failure (plain DorisConnectorException, affects every partition) must + // fail the query loud, NOT be swallowed into a silent empty scan. + // MUTATION: if listAndSplitFiles caught the base DorisConnectorException (the pre-fix behavior) this would + // return an empty range list instead of throwing -> red. + CountingLister lister = new CountingLister(); + lister.error = new DorisConnectorException("bad storage config"); + HiveScanPlanProvider provider = new HiveScanPlanProvider(null, Collections.emptyMap(), + new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); + + Assertions.assertThrows(DorisConnectorException.class, () -> provider.planScan( + new FakeSession(), singlePartitionHandle(), Collections.emptyList(), + Optional.empty())); + } + + private static HiveTableHandle singlePartitionHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat") + .serializationLib("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe") + .partitionKeyNames(Collections.singletonList("dt")) + .prunedPartitions(Collections.singletonList( + new HmsPartitionInfo(Collections.singletonList("1"), "loc/dt=1", null, null, null, null))) + .build(); + } + + private static HiveTableHandle twoPartitionHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat") + .serializationLib("org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe") + .partitionKeyNames(Collections.singletonList("dt")) + .prunedPartitions(Arrays.asList( + new HmsPartitionInfo(Collections.singletonList("1"), "loc/dt=1", null, null, null, null), + new HmsPartitionInfo(Collections.singletonList("2"), "loc/dt=2", null, null, null, null))) + .build(); + } + // ==================== integration: the scan provider is cache-backed ==================== @Test @@ -262,12 +379,14 @@ private static final class CountingLister implements HiveFileListingCache.Direct final Map callsPerLocation = new HashMap<>(); int totalCalls; RuntimeException error; + // When set, error is thrown only for this location (a single bad partition among healthy ones). + String errorLocation; @Override public List list(String location, Configuration conf) { totalCalls++; callsPerLocation.merge(location, 1, Integer::sum); - if (error != null) { + if (error != null && (errorLocation == null || errorLocation.equals(location))) { throw error; } return new ArrayList<>(Collections.singletonList(new HiveFileStatus(location + "/000000_0", 10L, 1L))); From 7bb5c13fa925dd7d3e85fd26abb1818868667b01 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 10:50:28 +0800 Subject: [PATCH 159/334] [feat](catalog) Hive connector cache: arm REFRESH DATABASE via connector.invalidateDb (dormant) Clean-room re-review finding (fix): REFRESH DATABASE never reached the connector-owned D2 caches, a parity regression at the flip. fe-core's RefreshManager.refreshDbInternal only did db.resetMetaToUninitialized() (the engine-side meta), with no connector hook -- unlike REFRESH TABLE (refreshTableInternal -> connector.invalidateTable) and REFRESH CATALOG (onRefreshCache -> connector.invalidateAll). So after the hms cutover a REFRESH DATABASE would refresh the schema but leave the connector serving stale partition/file listings up to the 24h TTL. Legacy REFRESH DATABASE dropped the engine-side caches for the whole db, so this is a user-visible parity loss. Add a generic Connector.invalidateDb(dbName) SPI (no-op default, exactly like invalidateTable/invalidateAll -- connector-agnostic, no source-specific code in fe-core) and wire refreshDbInternal to call it for a PluginDrivenExternalCatalog, keyed by the REMOTE db name (mirrors the refreshTableInternal branch verbatim). HiveConnector.invalidateDb drops BOTH cache layers for every table in the db: - CachingHmsClient.flushDb(db): invalidateIf(matchesDb) on all four metastore entries (table / partition_names / partition / column_stats). Adds a matchesDb(db) predicate to each key (TableKey previously had none). - HiveFileListingCache.invalidateDb(db): invalidateIf(matchesDb) on the (db,table,location) key. Same no-force-build read of the hmsClient field as invalidateTable (a REFRESH on a never-scanned catalog is a cheap no-op on the metastore side; the file cache is a final field, always present). Dormant: "hms" is not in SPI_READY_TYPES. Tests: - CachingHmsClientTest +1: flushDb drops EVERY table of the db (t1 AND t2) across all four caches (pins matchesDb, not per-table matches), while another db survives. - HiveConnectorInvalidateTest +1 (and the no-force-build hook test extended): invalidateDb drops both layers for every db1 table, db2 survives; the public invalidateDb hook clears the file cache WITHOUT building a metastore client. The one-line fe-core wiring mirrors the existing (untested-at-RefreshManager) refreshTableInternal connector branch; end-to-end REFRESH DATABASE is owed to the heterogeneous-HMS docker e2e at the flip. Full connector suite 263 green + checkstyle 0; fe-core compiles + RefreshManager checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/api/Connector.java | 8 ++++ .../doris/connector/hive/HiveConnector.java | 20 ++++++++++ .../connector/hive/HiveFileListingCache.java | 14 ++++++- .../hive/HiveConnectorInvalidateTest.java | 40 +++++++++++++++++++ .../doris/connector/hms/CachingHmsClient.java | 27 ++++++++++++- .../connector/hms/CachingHmsClientTest.java | 37 +++++++++++++++++ .../apache/doris/catalog/RefreshManager.java | 7 ++++ 7 files changed, 151 insertions(+), 2 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index 9dfbaf5c36e760..98a7c743539fbc 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -315,6 +315,14 @@ default void invalidateTable(String dbName, String tableName) { default void invalidateAll() { } + /** + * Invalidates the connector-side caches for every table in one database. Called by the engine on + * {@code REFRESH DATABASE}. The name is the REMOTE db name (as seen by the connector). Default no-op for + * connectors that cache nothing. + */ + default void invalidateDb(String dbName) { + } + /** * Optional per-connector override of the catalog's schema-cache TTL (in seconds), consulted generically by * the engine when sizing the schema meta-cache. Semantics match {@code schema.cache.ttl-second}: diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 411a6a49759293..7a8bf11754f876 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -301,6 +301,26 @@ void invalidateTable(HmsClient client, String dbName, String tableName) { fileListingCache.invalidateTable(dbName, tableName); } + /** + * REFRESH DATABASE hook: drop the connector-owned scan caches for EVERY table in this database — both cache + * layers ({@link CachingHmsClient#flushDb} metastore-metadata + {@link HiveFileListingCache#invalidateDb} + * directory listings). Same no-force-build read of the client as {@link #invalidateTable(String, String)}. + * fe-core routes {@code REFRESH DATABASE} to {@code connector.invalidateDb} for a plugin-driven catalog; + * dormant until hms enters SPI_READY_TYPES. + */ + @Override + public void invalidateDb(String dbName) { + invalidateDb(hmsClient, dbName); + } + + // Package-private seam (see invalidateTable above). + void invalidateDb(HmsClient client, String dbName) { + if (client instanceof CachingHmsClient) { + ((CachingHmsClient) client).flushDb(dbName); + } + fileListingCache.invalidateDb(dbName); + } + /** * REFRESH CATALOG hook: drop ALL of this catalog's connector-owned scan caches — every metastore-metadata * entry ({@link CachingHmsClient#flushAll}) and every directory listing. Same no-force-build read of the diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java index 78ebc3566a48aa..c716edd130e2a9 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -128,6 +128,11 @@ public void invalidateTable(String dbName, String tableName) { cache.invalidateIf(key -> key.matches(dbName, tableName)); } + /** Drops every cached listing for one database (all its tables). Backs {@code REFRESH DATABASE}. */ + public void invalidateDb(String dbName) { + cache.invalidateIf(key -> key.matchesDb(dbName)); + } + /** Drops the whole file-listing cache. Backs {@code REFRESH CATALOG}. */ public void invalidateAll() { cache.invalidateAll(); @@ -185,7 +190,10 @@ static List listFromFileSystem(String location, Configuration co } } - /** Cache key: (db, table, location). db+table let {@link #invalidateTable} select one table's entries. */ + /** + * Cache key: (db, table, location). db+table let {@link #invalidateTable} select one table's entries; + * db alone lets {@link #invalidateDb} select one database's entries. + */ static final class FileListingKey { private final String dbName; private final String tableName; @@ -201,6 +209,10 @@ boolean matches(String db, String table) { return Objects.equals(dbName, db) && Objects.equals(tableName, table); } + boolean matchesDb(String db) { + return Objects.equals(dbName, db); + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java index b02f6d803579a8..7beaf2f56ad712 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java @@ -91,6 +91,41 @@ public void invalidateTableFlushesBothCachesForThatTableOnly(@TempDir java.nio.f Assertions.assertEquals(1, fileCache.size(), "REFRESH TABLE must drop only that table's directory listings"); } + @Test + public void invalidateDbFlushesBothCachesForThatDbOnly(@TempDir java.nio.file.Path dirA, + @TempDir java.nio.file.Path dirB) throws Exception { + HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); + RecordingHmsClient raw = new RecordingHmsClient(); + CachingHmsClient cachingClient = (CachingHmsClient) connector.wrapWithCache(raw); + + // Metastore cache: TWO tables in db1 (t1, t2) and one in db2. REFRESH DATABASE db1 must drop every db1 + // table, not just one, while db2 survives. + cachingClient.getTable("db1", "t1"); + cachingClient.getTable("db1", "t2"); + cachingClient.getTable("db2", "t1"); + Assertions.assertEquals(3, raw.getTableCalls); + + // Directory-listing cache: db1.t1 (dirA) and db2.t1 (dirB) from real local dirs. + HiveFileListingCache fileCache = connector.fileListingCacheForTest(); + Files.write(dirA.resolve("f"), new byte[10]); + Files.write(dirB.resolve("f"), new byte[10]); + fileCache.listDataFiles("db1", "t1", dirA.toUri().toString(), CONF); + fileCache.listDataFiles("db2", "t1", dirB.toUri().toString(), CONF); + Assertions.assertEquals(2, fileCache.size()); + + connector.invalidateDb(cachingClient, "db1"); + + // Metastore cache: every db1 table re-fetches (t1 AND t2); db2 (another database) is still cached. + cachingClient.getTable("db1", "t1"); + cachingClient.getTable("db1", "t2"); + Assertions.assertEquals(5, raw.getTableCalls, "REFRESH DATABASE must drop every db1 table's metastore entry"); + cachingClient.getTable("db2", "t1"); + Assertions.assertEquals(5, raw.getTableCalls, "REFRESH DATABASE must NOT drop another database's entries"); + + // File cache: db1's listing dropped, db2's survives (invalidateDb is scoped by db). + Assertions.assertEquals(1, fileCache.size(), "REFRESH DATABASE must drop only that db's directory listings"); + } + @Test public void invalidateAllFlushesBothCachesEntirely(@TempDir java.nio.file.Path dir) throws Exception { HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); @@ -127,6 +162,11 @@ public void publicHooksAreNoThrowAndClearFileCacheWithoutBuildingAClient(@TempDi Assertions.assertDoesNotThrow(() -> connector.invalidateTable("db", "t")); Assertions.assertEquals(0, fileCache.size(), "REFRESH TABLE clears the file cache even with no client built"); + fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); + Assertions.assertEquals(1, fileCache.size()); + Assertions.assertDoesNotThrow(() -> connector.invalidateDb("db")); + Assertions.assertEquals(0, fileCache.size(), "REFRESH DATABASE clears the file cache with no client built"); + fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); Assertions.assertEquals(1, fileCache.size()); Assertions.assertDoesNotThrow(() -> connector.invalidateAll()); diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index ef62569aab0ad7..9bb5b15d71f3d0 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -63,7 +63,8 @@ * *

    Pass-through. Every other read, plus every write / DDL / ACID method, is passed straight * through to the delegate. A later invalidation step arms {@link #flush(String, String)} / - * {@link #flushAll()} onto {@code REFRESH TABLE} / {@code REFRESH CATALOG}. This decorator does NOT + * {@link #flushDb(String)} / {@link #flushAll()} onto {@code REFRESH TABLE} / {@code REFRESH DATABASE} / + * {@code REFRESH CATALOG}. This decorator does NOT * self-invalidate around writes — coarse REFRESH + TTL bound staleness.

    * *

    Cache-value safety. {@code HmsTableInfo} / {@code HmsPartitionInfo} / {@code HmsColumnStatistics} @@ -167,6 +168,14 @@ public void flush(String dbName, String tableName) { columnStatsCache.invalidateIf(key -> key.matches(dbName, tableName)); } + /** Drop every cached entry for one database (all its tables). Backs {@code REFRESH DATABASE}. */ + public void flushDb(String dbName) { + tableCache.invalidateIf(key -> key.matchesDb(dbName)); + partitionNamesCache.invalidateIf(key -> key.matchesDb(dbName)); + partitionsCache.invalidateIf(key -> key.matchesDb(dbName)); + columnStatsCache.invalidateIf(key -> key.matchesDb(dbName)); + } + /** Drop the whole cache. Backs {@code REFRESH CATALOG}. */ public void flushAll() { tableCache.invalidateAll(); @@ -298,6 +307,10 @@ static final class TableKey { this.tableName = tableName; } + boolean matchesDb(String db) { + return Objects.equals(dbName, db); + } + @Override public boolean equals(Object o) { if (this == o) { @@ -331,6 +344,10 @@ boolean matches(String db, String table) { return Objects.equals(dbName, db) && Objects.equals(tableName, table); } + boolean matchesDb(String db) { + return Objects.equals(dbName, db); + } + @Override public boolean equals(Object o) { if (this == o) { @@ -370,6 +387,10 @@ boolean matches(String db, String table) { return Objects.equals(dbName, db) && Objects.equals(tableName, table); } + boolean matchesDb(String db) { + return Objects.equals(dbName, db); + } + @Override public boolean equals(Object o) { if (this == o) { @@ -409,6 +430,10 @@ boolean matches(String db, String table) { return Objects.equals(dbName, db) && Objects.equals(tableName, table); } + boolean matchesDb(String db) { + return Objects.equals(dbName, db); + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java index d27bcdd9f7efd7..d2346392824d18 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -243,6 +243,43 @@ public void flushDropsOnlyThatTablesEntries() { Assertions.assertEquals(3, delegate.getColumnStatsCalls); } + // ---- flushDb() ---- + + @Test + public void flushDbDropsOnlyThatDatabasesEntries() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Populate all four caches for db1.t1, plus db1.t2 (a SECOND table in the same db) and db2.t1 (a table in + // ANOTHER db). flushDb("db1") must drop EVERY db1 table (t1 AND t2) across all four caches, while db2 lives. + cache.getTable("db1", "t1"); + cache.listPartitionNames("db1", "t1", -1); + cache.getPartitions("db1", "t1", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db1", "t1", Arrays.asList("c1")); + cache.getTable("db1", "t2"); + cache.getTable("db2", "t1"); + Assertions.assertEquals(3, delegate.getTableCalls); + + cache.flushDb("db1"); + + // WHY: every db1 table reloads across all four caches — this pins the matchesDb() db scoping (not the + // per-table matches()): t2 reloading proves the whole database was dropped, not just one table. + cache.getTable("db1", "t1"); + cache.listPartitionNames("db1", "t1", -1); + cache.getPartitions("db1", "t1", Arrays.asList("p=1")); + cache.getTableColumnStatistics("db1", "t1", Arrays.asList("c1")); + cache.getTable("db1", "t2"); + Assertions.assertEquals(5, delegate.getTableCalls, "flushDb must drop EVERY table in the database (t1 and t2)"); + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + Assertions.assertEquals(2, delegate.getPartitionsCalls); + Assertions.assertEquals(2, delegate.getColumnStatsCalls); + + // WHY: flushDb is scoped to ONE database — db2's entry must survive (no reload). An over-broad flushDb that + // wiped every db would reload db2 here -> red. + cache.getTable("db2", "t1"); + Assertions.assertEquals(5, delegate.getTableCalls, "flushDb must NOT drop another database's entries"); + } + // ---- flushAll() ---- @Test diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java index ca049fb4736406..c2fe59a7ae6352 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java @@ -118,6 +118,13 @@ public void replayRefreshDb(ExternalObjectLog log) { private void refreshDbInternal(ExternalDatabase db) { db.resetMetaToUninitialized(); + // Also drop any connector-side caches for every table in this db (e.g. hive's metastore + directory-listing + // caches) so a subsequent read reflects the latest external state — otherwise REFRESH DATABASE would reset + // only the engine-side meta and leave the connector serving stale partition/file listings up to its TTL. + // Connector-agnostic (generic SPI no-op default); keyed by the REMOTE db name. Mirrors refreshTableInternal. + if (db.getCatalog() instanceof PluginDrivenExternalCatalog) { + ((PluginDrivenExternalCatalog) db.getCatalog()).getConnector().invalidateDb(db.getRemoteName()); + } LOG.info("refresh database {} in catalog {}", db.getFullName(), db.getCatalog().getName()); } From 7b28f4e97a8d8a2609ddf9b4bdd7524bd7a36797 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 10:52:12 +0800 Subject: [PATCH 160/334] [doc](catalog) Hive connector cache: clean-room re-review DONE + 2 fixes landed; next = event pipeline Model B MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the unified clean-room adversarial re-review of the 6 dormant D2 commits (9 blind dimension reviewers -> adversarial per-finding verification -> cross- check vs the signed-off design conclusions): 7 findings raised, 2 confirmed, 5 refuted to nit/clean. Full report in plan-doc/reviews/. The two confirmed findings were fixed (user chose fix-now for both): - fda344e6022 restore fail-loud on FileSystem.get (systemic) vs skip-with-warn on listStatus (local) -- undoes a silent-empty-result coarsening. - cdc837563a7 arm REFRESH DATABASE via a new generic Connector.invalidateDb SPI wired into RefreshManager.refreshDbInternal. Confirmed-clean by the review (not just untested): the §2.6 last-modified branch (monotonicity == legacy parity; no NPE surface; dormant), the CachingHmsClient decorator pass-through completeness, TCCL on the caller thread only, and the single-Caffeine packaging. One low-severity nit left open for user sign-off: mark fe-connector-cache optional in fe-connector-hms so the unused cache jar stops flowing into the hudi zip. HANDOFF: D2 first Phase-1 item DONE (cache + review + fixes); next Phase-1 item = event pipeline Model B (depends on the D2 cache). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 10 +- ...ector-cache-cleanroom-review-2026-07-10.md | 117 ++++++++++++++++++ 2 files changed, 122 insertions(+), 5 deletions(-) create mode 100644 plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 0a7437b7dccf36..b9391f8daf814d 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -1,7 +1,7 @@ # 🤝 Session Handoff > **滚动文档**：每次 session 结束**覆盖式更新**，**只保留下一个 session 必须的上下文**；完成的工作明细**不落这里**（在 `git log` + `tasks/` 设计文档里，见下「起步必读」）。协作规范：[AGENT-PLAYBOOK.md](./AGENT-PLAYBOOK.md)。 -> **范围** = catalog-spi **主线**（HMS 翻闸）。metastore/storage 抽取子线**已 CLOSED**（合入 #64446/#64653/#64655；`metastore-storage-refactor/` 仅历史留存、勿读，现状直接读代码 `fe/fe-connector/fe-connector-metastore-spi/`）。 +> **范围** = catalog-spi **主线**（HMS 翻闸）。 --- @@ -13,11 +13,11 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms - **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) → 事件管道 Model B → 3 个 loud-break 耦合缝 → W6 写路径 TCCL 验证。 - **Phase 2 原子翻闸**（`SPI_READY += hms` + 3 处 GSON 重映射 + 死臂/INC-5/D5 删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门（R-002）**。 -**✅ 本轮完成 = Phase 1 首项「Hive 连接器自持缓存(D2)」全部 6 个休眠 commit 落地**： -`f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` 装饰器) → `7b05df6e55e` C-c(接线进 client，元数据读+新鲜度探针走缓存) → `7c0ee1ffb2a` C-d(`HiveFileListingCache` 文件列表缓存，扫描+行数估算共享，TCCL 钉调用线程) → `7bf90a7fe3c` C-e(`invalidateTable/invalidateAll` 接 REFRESH TABLE/CATALOG，两层全清) → `12e0c9177c2` C-f(fe-core `getNewestUpdateVersionOrTime` last-modified 分支，修 hive 基表字典/MV 永不自动刷新)。 -设计 = **`tasks/hive-connector-cache-step-design-2026-07-10.md`**（TODO 全勾，**起步必读 #2**）。全休眠（hms 未进 `SPI_READY_TYPES`，线上零路由；paimon/iceberg/jdbc/hudi 字节不变；对齐 Trino 双层：`CachingHmsClient` 元数据层 + `HiveFileListingCache` 文件层）。 +**✅ 本轮完成 = Phase 1 首项「Hive 连接器自持缓存(D2)」6 个休眠 commit + 净室对抗复审 + 2 处复审修复**： +D2 = `f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` 装饰器) → `7b05df6e55e` C-c(接线进 client) → `7c0ee1ffb2a` C-d(`HiveFileListingCache`) → `7bf90a7fe3c` C-e(`invalidateTable/invalidateAll`) → `12e0c9177c2` C-f(fe-core `getNewestUpdateVersionOrTime` last-modified 分支)。设计 = **`tasks/hive-connector-cache-step-design-2026-07-10.md`**。全休眠（hms 未进 `SPI_READY_TYPES`；paimon/iceberg/jdbc/hudi 字节不变；对齐 Trino 双层）。 +**净室对抗复审**（9 维独立盲评 → 逐条对抗验证 → 交叉核对历史结论）= 7 疑点 2 坐实、5 误报/干净；报告 = **`plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md`**。两处坐实（用户拍板即修，均已落地）：`fda344e6022`（恢复 `FileSystem.get` 系统性失败大声报错、仅单目录失败 skip——修静默空结果回归）+ `cdc837563a7`（新增通用 `Connector.invalidateDb` SPI + 接 `RefreshManager.refreshDbInternal`，让 REFRESH DATABASE 清连接器两层缓存）。复审确认干净项：§2.6 单调性=与旧实现平价（删最新分区致 max 下降在新旧都抛，非新回归）、无空指针面、装饰器 pass-through 完整、TCCL 只在调用线程、Caffeine 单版本。**未做的低危建议**：`fe-connector-hms/pom.xml` 把 `fe-connector-cache` 标 `optional`（免那个空 Caffeine jar 流进 hudi 包，现无害但去掉未来隐患）——待用户点头。 -**⭐ 下一步 = 对这 6 个 commit 做统一 clean-room 对抗复审**（memory `clean-room-adversarial-review-pref`：多 agent 对抗、先独立判断后交叉核对历史结论）。重点镜头：装饰器/文件缓存正确性 · TCCL split-brain · 休眠打包 Caffeine 单版本(2.9.3) · paimon/iceberg **字节中立** · 失效 scope · §2.6 单调性 · **C-d 里 `FileSystem.get` 失败并入 skip-with-warn 的行为变化**（已记档，翻闸 e2e 复核）。复审通过后续 Phase 1 = **事件管道 Model B**（薄 fe-core 角色驱动 + 插件 `pollOnce` SPI，权威 = `tasks/hms-event-pipeline-findings-2026-07-07.md`）。 +**⭐ 下一步 = Phase 1 第二项「事件管道 Model B」**（薄 fe-core 角色驱动 + 插件 `pollOnce` SPI，翻闸后重新武装 `MetastoreEventsProcessor` 的增量失效，权威 = `tasks/hms-event-pipeline-findings-2026-07-07.md`）。它**依赖** D2 缓存（在其之上加 `Connector.invalidatePartitions(db,table,names)`）。起步先读 findings + 对照 HEAD。 **⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步（读 / 写-拒 / schema-evolution / 缓存 / 跨加载器委派）只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`）。**删除排序最硬约束**：`datasource/hive|hudi|iceberg/` 的 ~90 个 HMS 支撑类删不掉，直到翻闸把消费者切到连接器路径（详见 execution plan §2.4/§3 + `tasks/iceberg-on-hms-delegation-findings-2026-07-07.md`）。 diff --git a/plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md b/plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md new file mode 100644 index 00000000000000..51575be71118ba --- /dev/null +++ b/plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md @@ -0,0 +1,117 @@ +# Hive connector-owned cache — clean-room adversarial re-review (2026-07-10) + +> Scope: the 6 dormant commits `f742651990d`(C-a) `4fe55d88fab`(C-b) `7b05df6e55e`(C-c) +> `7c0ee1ffb2a`(C-d) `7bf90a7fe3c`(C-e) `12e0c9177c2`(C-f) — the Hive connector scan-side cache. +> Method: 9 independent clean-room reviewers (blind to the design rationale) → adversarial refutation of +> every finding → cross-check vs the signed-off design conclusions. Workflow `wf_124cb0a7-6bb`. +> Result: 7 findings raised, **2 survived** as confirmed, 5 refuted to nit/clean. + +## Verdict + +**Sound to carry into the cutover, with one recommended fix.** No blocker, no dormancy/byte-neutrality +breakage that affects a live connector, no TCCL or classloader hazard. The two survivors are both +behavior *coarsenings* that were consciously accepted in the design doc — but the review shows one of +them (FileSystem.get) rests on a **mitigation that does not actually hold**, so it is worth re-opening. + +## Resolution (2026-07-10, user-approved) + +Both confirmed findings were fixed (user chose fix-now for both): +- **Finding #1 (FileSystem.get)** → `fda344e6022` — restored the legacy blast-radius distinction: a + systemic `FileSystem.get` failure fails loud (plain `DorisConnectorException`, not skipped); a local + `listStatus` failure stays skip-with-warn (new `HiveDirectoryListingException`). Tests +6. +- **Finding #2 (REFRESH DATABASE)** → `cdc837563a7` — added a generic `Connector.invalidateDb(db)` SPI + (no-op default) and wired `RefreshManager.refreshDbInternal` to it; `HiveConnector.invalidateDb` drops + both cache layers (`CachingHmsClient.flushDb` + `HiveFileListingCache.invalidateDb`) for every table in + the db. Tests +2. **This upgrades finding #2 from "accept-deferred" to fixed.** +- **hudi byte-neutrality nit** → recommended (not yet applied): mark `fe-connector-cache` + `true` in `fe-connector-hms/pom.xml` so the unused, Caffeine-less cache jar stops + flowing into the hudi plugin zip. Awaiting user go-ahead (benign now; removes a future NoClassDefFound + trap the day hudi reuses the caching client). + +## Confirmed findings (survived adversarial verification) + +### 1. FileSystem.get failure now silently skips partitions (was fail-loud) — recommend fix +- **Where:** `HiveFileListingCache.listFromFileSystem` wraps *both* `FileSystem.get` and `listStatus` in + one `try → DorisConnectorException`; `HiveScanPlanProvider.listAndSplitFiles` catches that and + skips-the-partition-with-a-warning. +- **Legacy (pre-cache):** `FileSystem.get` ran *outside* the inner try → its `IOException` propagated and + the caller re-threw it as `DorisConnectorException` = **fail loud**. Only `listStatus` failures were + skipped-with-warn. So legacy drew a deliberate line: *storage-init failure = loud; one unreadable + partition dir = tolerate.* +- **Now:** both are skipped. A `FileSystem.get` failure is **systemic** (all partitions of a table share + one scheme+authority → it fails for all or none), so a broken storage config makes the scan **silently + return empty/partial results with no error**, where legacy failed the query loudly. +- **The documented mitigation does not hold.** The design doc accepts this because "a broken storage + config still fails loud via the row-count estimate." Verified false: `estimateDataSize` + (`HiveConnectorMetadata.java:771`) **catches `RuntimeException` and returns -1** — it degrades silently, + never throws to the user; and for a table that already has HMS stats the estimate path is not even + invoked. So nothing surfaces the error. +- **Verify verdict:** CONFIRMED, regression-vs-legacy = true. Severity medium (silent wrong *results*, not + a crash; narrow trigger = storage misconfig). +- **Suggested fix:** restore the legacy distinction — let a `FileSystem.get` (FS-resolution) failure + propagate loud while a per-directory `listStatus` failure stays skip-with-warn. (Options in the handoff.) + +### 2. REFRESH DATABASE does not drop the connector-owned caches — accept-deferred (documented) +- **Where:** `RefreshManager.refreshDbInternal → ExternalDatabase.resetMetaToUninitialized → + ExtMetaCacheMgr.invalidateDb`. There is no `Connector.invalidateDb` SPI, so for a flipped hive catalog + `REFRESH DATABASE` drops the fe-core schema cache but **not** the connector's partition/file caches. +- Legacy `REFRESH DATABASE` dropped the engine-side caches for the whole db; post-flip it won't → a user + who runs `REFRESH DATABASE` expecting fresh data across the db still sees stale partition/file listings + until TTL / `REFRESH TABLE` / `REFRESH CATALOG`. +- **Verify verdict:** CONFIRMED, regression-vs-legacy = true. Severity medium. +- **Design status:** explicitly signed off as "accept per-table/all coverage; a db-level verb is + Model-B-adjacent" (§4.5). A `Connector.invalidateDb` SPI + fe-core wiring is genuinely event-Model-B + territory, not this step. **Recommend: keep deferred, but document loudly as a known post-flip gap** + (covered in the interim by `REFRESH CATALOG`). + +## Refuted / clean (what the review actively verified as correct) + +- **§2.6 fe-core last-modified branch (C-f) — CLEAN.** The two sharp risks were examined with line-level + evidence and resolved as **legacy parity**, not new bugs: + - *Monotonicity on partition drop:* a decreasing max `transient_lastDdlTime` makes + `Dictionary.hasNewerSourceVersion` throw in **both** the new and the legacy + `HMSExternalTable.getNewestUpdateVersionOrTime` (`:1060`, `max(HivePartition::getLastModifiedTime)`) — + `lastDdlMillis` is byte-parity with it. Pre-existing property of HMS, the flipped value **equals** + legacy. (Worth listing as a known-issue for the flip e2e, not a regression.) + - *NPE surface:* `pin.getConnectorSnapshot()` is provably never null — `materializeLatest` builds a + non-null connector snapshot on every path (`beginQuerySnapshot.orElseGet(emptySnapshot)`, both degraded + branches use `emptySnapshot()`, range-view passes it through). + - *Dormancy:* the only `lastModifiedFreshness(true)` in the whole tree is `HiveConnectorMetadata:1014`; + paimon/iceberg/empty pins leave it false; SPI_READY excludes hms/hive → no live pin is flagged. The new + line is one boolean read for live connectors; the pre-change max/range paths are byte-unchanged. +- **CachingHmsClient decorator (C-b) — CLEAN.** All 25 SPI methods enumerated: exactly the 4 scan reads + cached, 21 pass-through; no write/DDL/txn cached; every must-be-fresh read (`tableExists`, + `getPartition`-by-values, `getValidWriteIds`, `partitionExists`, `list*`) is pass-through. Key + equals/hashCode correct; list-key order-sensitivity is a non-issue because every call-site passes + deterministic HMS-ordered or singleton lists. `flush(db,table)` reaches all 4 key types; returned list + containers are only ever read-only-iterated by consumers (traced) → shared-reference caching can't + corrupt. `getTableColumnStatistics` is newly cached (legacy did a raw RPC) but under the same TTL + REFRESH + invalidation as the existing fe-core `StatisticsCache` → no new staleness window. +- **TCCL / classloader — CLEAN.** The listing loader runs on the calling (TCCL-pinned) thread; with + `autoRefresh=false` the `ForkJoinPool.commonPool()` refresh executor is never used to run a loader off + the pinned thread. Cache holds only JDK/plugin types → no cross-loader ClassCast. +- **Caffeine single-version packaging (C-a) — CLEAN.** Mirrors the paimon pattern; hive self-bundles exactly + one Caffeine 2.9.3 child-first; no leak onto fe-core. +- **New tests are non-vacuous** — the three "untested tolerance" worries were refuted (the guards + degrade + paths are in fact covered / present at HEAD); the freshness/neutrality tests distinguish hit-vs-reload via + call-count and a `verify(never())` probe-gate. + +## Minor / cleanliness item (low, optional hardening) + +- **hudi plugin zip gains an unused, Caffeine-less `fe-connector-cache` jar** because C-b makes + `fe-connector-cache` a compile-scope dep of `fe-connector-hms` and hudi bundles transitive deps. + **Benign now** (JVM class-loading is lazy; hudi has zero `connector.cache.*` refs → never linked, no + NoClassDefFoundError). But it deviates from the "byte-neutral for …hudi" wording in C-c's message and + plants a latent trap for the day hudi reuses the caching client (the design *anticipates* this reuse). + **Cheap fix:** mark `fe-connector-cache` `true` in `fe-connector-hms/pom.xml` + (hms compiles against it; hive already declares it directly; it stops flowing into the hudi zip). + +## e2e debt to assert at the flip (heterogeneous-HMS docker) + +1. A misconfigured storage on a hive scan **must surface an error**, not an empty result (this is exactly + finding #1 — if we adopt the fix, assert it fails loud; if we don't, assert the behavior consciously). +2. `REFRESH TABLE` / `REFRESH CATALOG` end-to-end drop both cache layers and show new data; document that + `REFRESH DATABASE` does **not** (finding #2). +3. A hive-backed SQL dictionary / MV **auto-refreshes** after the base table changes (validates the §2.6 + fix end-to-end), and note the pre-existing "drop-newest-partition lowers the max" monotonicity property. +4. Cache hit under a real flipped hms catalog (metastore + file listing); hudi-on-HMS caching (separate item). From 57c447b456c1d56947146305cfc9fa1d71aac4e8 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 10:53:35 +0800 Subject: [PATCH 161/334] =?UTF-8?q?[doc](catalog)=20Hive=20connector=20cac?= =?UTF-8?q?he:=20second=20clean-room=20review=20=E2=80=94=20delta=20findin?= =?UTF-8?q?gs=20+=20live=20paimon/iceberg=20drop+recreate=20hole?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second independent adversarial review of the 6 dormant cache commits (14 raised / 12 survived / 2 refuted). Agrees with review #1 on everything it surfaced (FileSystem.get fail-loud, REFRESH DATABASE gap, C-f clean, TCCL clean, packaging clean) and adds three confirmed deltas: - HIGH: invalidateTable/invalidateAll/invalidateDb never forward to the iceberg sibling's latestSnapshotCache (close() does) -> REFRESH cannot unpin iceberg-on-HMS staleness (unbounded under access-based expiry). - HIGH: Doris-initiated DROP/CREATE TABLE + DROP DATABASE never invalidate the connector caches -> drop+recreate serves the dropped table's schema/location up to 24h (Trino self-invalidates). - MEDIUM: partition-cache capacity 100000 now counts request-LISTS (no weigher), not partitions as legacy did -> heap growth far beyond legacy's bound. Plus a targeted follow-up finding: the DROP/CREATE hole is LIVE today for paimon/iceberg latestSnapshotCache (+ PaimonSchemaAtMemo narrow time-travel variant). Open decisions and e2e debt additions recorded in the doc. Co-Authored-By: Claude Fable 5 Claude-Session: e95529ec-63df-4dcf-b1a0-e9c39514be8d --- ...ctor-cache-cleanroom-review2-2026-07-10.md | 153 ++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md diff --git a/plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md b/plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md new file mode 100644 index 00000000000000..8c32fcd53e3d61 --- /dev/null +++ b/plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md @@ -0,0 +1,153 @@ +# Hive connector-owned cache — SECOND independent clean-room review (2026-07-10) + +> A second, fully independent clean-room run over the same 6 dormant commits +> (`f742651990d` `4fe55d88fab` `7b05df6e55e` `7c0ee1ffb2a` `7bf90a7fe3c` `12e0c9177c2`), same method as +> [hive-connector-cache-cleanroom-review-2026-07-10.md](./hive-connector-cache-cleanroom-review-2026-07-10.md) +> (review #1): 9 blind dimension reviewers → adversarial refutation per finding → cross-check vs the +> signed-off design doc. Different session, different workflow run (`wf_390c0bfc-ca5`; full per-agent +> results in that session's workflow journal). Totals: **14 raised, 12 survived, 2 refuted**. +> +> **Why this doc exists**: the two runs AGREE on everything review #1 surfaced, but this run surfaced +> **three additional confirmed defects** review #1 missed, plus (via a follow-up targeted investigation) +> a **live bug in paimon/iceberg today** with the same shape as one of them. Only the deltas are detailed +> here; agreements are summarized. + +## 1. Agreement with review #1 (independently re-derived) + +- **FileSystem.get fail-loud** — found by both runs (this run: CONFIRMED high ×2 reviewers, incl. proof the + "surfaces via the estimate" mitigation is false on three counts). Fixed by `fda344e6022` (restore + fail-loud on `FileSystem.get`, keep per-directory skip via `HiveDirectoryListingException`); this + session's follow-up verified the fix satisfies the contract and folded in 3 hardening tests + (fail-not-cached through the real loader, skippable-vs-loud type split, multi-partition skip scope). +- **REFRESH DATABASE gap** — found by both; review #1 recommended keep-deferred+document, the follow-up + session chose to fix it (`Connector.invalidateDb` SPI + `RefreshManager.refreshDbInternal` wiring + + `flushDb`/`invalidateDb` in both cache layers). NOTE: the new `invalidateDb` hook has the same sibling + gap as §2.1 below. +- **fe-core last-modified branch (C-f) — CLEAN in both runs**: monotonicity-decrease on partition drop is + legacy parity (legacy `HMSExternalTable.getNewestUpdateVersionOrTime` computes the same decreasable max; + `Dictionary.hasNewerSourceVersion` throw is a pre-existing property, list as flip-e2e known-issue); + `pin.getConnectorSnapshot()` provably never null; dormant for paimon/iceberg (only + `HiveConnectorMetadata` sets `lastModifiedFreshness(true)`). +- **TCCL/classloader — CLEAN** (manual-miss loader runs on the calling pinned thread; `commonPool` + refresh executor never runs a loader with `autoRefresh=false`; only JDK/plugin types cached). +- **Packaging/byte-neutrality — CLEAN** (this run's 2 refuted findings were here: the + `fe-connector-cache → fe-connector-hudi` transitive compile edge is harmless; the unrelocated Caffeine + copy inside `hive-catalog-shade` is CRC-identical to caffeine-2.9.3, single effective version). +- **Decorator method set — CLEAN** (exactly the 4 scan reads cached; writes/DDL/txn and must-be-fresh + reads pass through). + +## 2. DELTA: confirmed defects review #1 did not surface + +### 2.1 REFRESH is never forwarded to the iceberg sibling's snapshot cache (HIGH, latent-until-flip) + +- **Where**: `HiveConnector.invalidateTable`/`invalidateAll` (and the new `invalidateDb`) flush only + `CachingHmsClient` + `HiveFileListingCache`. `close()` DOES forward to the built siblings + (`HiveConnector.java:551+`, fields `icebergSibling:99` / `hudiSibling:107`); the invalidate hooks do not. + fe-core routes REFRESH only to the catalog's PRIMARY connector, so nothing can ever reach + `IcebergConnector.latestSnapshotCache` for iceberg-on-HMS tables behind the flipped gateway. +- **Failure scenario**: iceberg-on-HMS table externally updated → `latestSnapshotCache` (86400s + **access-based** expiry) keeps serving the old snapshot; continuous querying keeps the entry alive + forever; user runs `REFRESH TABLE`/`REFRESH CATALOG`/(new)`REFRESH DATABASE` → no effect. Staleness is + **unbounded**, breaking the signed-off acceptance "staleness bounded by TTL + explicit REFRESH", and a + parity regression (legacy REFRESH dropped the iceberg engine cache via `ExternalMetaCacheRouteResolver`). + Also weakens the follower-replay "graceful coarsening" argument (replay hits the same non-forwarding hook). +- **Fix (small, symmetric with `close()`)**: forward all three invalidate hooks to the ALREADY-BUILT + siblings (volatile field read, no force-build). Hudi sibling has no snapshot cache today — forwarding is + a harmless no-op but keeps the contract uniform. + +### 2.2 Doris-initiated DROP TABLE / CREATE TABLE / DROP DATABASE never invalidate the connector caches (HIGH, latent-until-flip) + +- **Where**: `PluginDrivenExternalCatalog.dropTable` (ends at `metadata.dropTable` + editlog + + `unregisterTable`) and `createTable` (ends at `resetMetaCacheNames`) never call + `connector.invalidateTable`; `unregisterTable`/`unregisterDatabase` reach only fe-core engine caches + (`ExtMetaCacheMgr`), never the connector. Only `RefreshManager.refreshTableInternal` (REFRESH TABLE; + INSERT/TRUNCATE/ALTER route here) and `onRefreshCache` (REFRESH CATALOG) reach the connector hooks. + The decorator deliberately does not self-invalidate around writes (javadoc: "coarse REFRESH + TTL"). +- **Failure scenario (post-flip)**: `DROP TABLE t; CREATE TABLE t (new schema/location);` (common in + ETL/tests) → next `SELECT` rebuilds the fe-core table via `getTableHandle` → + `CachingHmsClient.getTable(db,t)` cache HIT returns the **dropped** table's `HmsTableInfo` + (schema/location) → query planned against the wrong schema, reads the old location; CTAS write planning + likewise. Up to 24h unless an explicit REFRESH intervenes. File-listing entries collide too when the + recreated table reuses the same location. The §2 staleness acceptance covers EXTERNAL HMS changes, not + Doris's own DDL — no sign-off covers this. Trino's `CachingHiveMetastore` (the signed-off model) + self-invalidates on these mutations; legacy invalidated on every `unregisterTable`. +- **Fix options** (decision recorded in §4): + - **(i) plugin-side, Trino-faithful (recommended for the dormant hive line)**: `CachingHmsClient` + self-invalidates (`flush(db,table)` after createTable/dropTable/truncateTable/add-dropPartition/ + update*Statistics; `flushDb` after dropDatabase) + `HiveConnectorMetadata.dropTable/createTable/ + dropDatabase` drop the matching `HiveFileListingCache` entries. Fully dormant, zero fe-core change. + - **(ii) fe-core-side, generic**: `PluginDrivenExternalCatalog.dropTable/createTable/dropDb` call + `connector.invalidateTable/invalidateDb`. Also fixes the LIVE paimon/iceberg hole (§3) in one shot, + but touches live paths → needs paimon/iceberg behavior-neutrality argument + its own review. + - Both is fine too ((i) now in the dormant line, (ii) as the separate live-bug fix). + +### 2.3 Partition-cache capacity semantics changed: 100000 now counts request-LISTS, not partitions (MEDIUM) + +- **Where**: `CachingHmsClient` `DEFAULT_PARTITION_CAPACITY = 100000` claims to mirror legacy + `Config.max_hive_partition_cache_num`, but legacy keyed per-partition (100000 partition OBJECTS; + `HiveExternalMetaCache.java:121`), while `partitionsCache` keys the full requested-name-list and stores + the whole `List` as ONE entry — `CacheFactory` has `maximumSize` only, no weigher. + Overlapping requests duplicate partition objects across entries (full-list scans + each distinct pruned + subset via `applyFilter` + MTMV per-partition singletons via `getPartitionFreshnessMillis`). +- **Failure scenario**: 10k–100k-partition tables + dashboard-style sliding predicates → each distinct + predicate window caches another multi-thousand-object list, each size-1 to Caffeine, 24h TTL → FE heap + grows far beyond legacy's bound; OOM reachable under a workload legacy handled. +- **Fix options**: (a) weigher summing list sizes (Trino uses a weigher here) — requires adding optional + weigher support to the shared `fe-connector-cache` framework (bundled into paimon/iceberg zips → breaks + this series' "paimon/iceberg byte-identical" claim; needs behavior-neutrality tests + explicit OK); + (b) much smaller list-entry default with an honest comment (no framework change); (c) document-only. + **User decision needed** (§4). + +### 2.4 Test-adequacy deltas (LOW, production code correct at HEAD) + +- Empirically proven mutation-survivable at review time (full module suite stayed green under the + mutation): the `listFromFileSystem` IOException fold (catch→emptyList would CACHE a poisoned empty + listing) — **closed** by the tests folded into `fda344e6022`. +- Still open: (a) nothing pins that `createClient` actually wraps with `CachingHmsClient` + (removing `wrapWithCache` from the production call-site survives the suite; closable via the + `newMetadata` seam); (b) the PUBLIC `invalidateTable/invalidateAll` hooks are never driven with a BUILT + metastore client (dropping the metastore flush from them survives the suite; the seam tests cover the + internals only). Cheap insurance for the two one-line surfaces REFRESH depends on. + +## 3. LIVE BUG (today, not dormant): paimon/iceberg drop+recreate serves a stale snapshot pin + +Targeted follow-up investigation (this session), same shape as §2.2 but for the LIVE plugin connectors: + +- `IcebergLatestSnapshotCache` — key = `TableIdentifier.of(db, table)` (plain names), value = + `(snapshotId, schemaId)`, 86400s access-based TTL, maxSize 1000. **HOLE**: Doris-initiated + `DROP TABLE`+`CREATE TABLE` same name never invalidates (`IcebergConnectorMetadata.dropTable/createTable` + only call `catalogOps`; the fe-core DDL path never reaches `connector.invalidate*`) → next query's + `beginQuerySnapshot` pins the DROPPED table's snapshot/schema against the new table. +- `PaimonLatestSnapshotCache` — same shape, same **HOLE** (key `Identifier.create(db, table)`, value + snapshotId, 86400s access-based). +- `PaimonSchemaAtMemo` — **narrow HOLE** (time-travel only): keyed `(db, table, sysTable, branch, + schemaId)`, correctness rests on "schemaId content is write-once", violated by drop+recreate (fresh + table reuses schemaId 0). NOT cleared even by `invalidateTable`/`invalidateAll` — only by connector + rebuild (REFRESH CATALOG). +- NOT affected: `IcebergManifestCache` (path-keyed, unique paths), `IcebergRewritableDeleteStash` + (queryId-keyed), fe-core schema cache (cleared on drop via `unregisterTable`). +- Mitigations today: bounded by the 24h access TTL (but continuous querying keeps it alive), REFRESH + TABLE/CATALOG clears the snapshot caches (NOT the schema-at memo), `ttl-second<=0` catalogs immune. +- **Recommended handling**: separate fix line (NOT folded into the dormant hive series): fe-core + `PluginDrivenExternalCatalog.dropTable/createTable/dropDb` → `connector.invalidateTable/invalidateDb` + (option (ii) above), plus make paimon `invalidateTable/invalidateAll` also clear `PaimonSchemaAtMemo`. + Touches live behavior → own commits + own adversarial review + regression coverage. + +## 4. Open decisions (need user sign-off) + +1. §2.2 fix locus: plugin-side (i), fe-core-side (ii), or both. (Recommended: (i) for the dormant line, + (ii) as the live-bug fix.) +2. §2.3 capacity fix: weigher in shared framework (Trino-faithful, touches paimon/iceberg-bundled + framework bytes) vs smaller default vs document-only. +3. §3 live bug: fix now as its own line vs defer with documented risk. + +## 5. e2e debt additions (heterogeneous-HMS docker, on top of review #1's list) + +- Drop+recreate freshness: `DROP TABLE`+`CREATE TABLE` same name → immediate query sees the new + schema/location with no REFRESH (hive post-flip; iceberg/paimon live once §3 is fixed). +- REFRESH reaches the iceberg sibling: externally mutate an iceberg-on-HMS table → `REFRESH TABLE` / + `REFRESH DATABASE` / `REFRESH CATALOG` each unpin the snapshot (and via follower replay). +- Broken storage config fails loud (post-`fda344e6022` contract): bogus scheme/credentials → query errors, + never 0-rows-as-success. +- An end-to-end read that provably transits `CachingHmsClient` via the real `createClient` (second query + hits cache, HMS call count flat) — closes §2.4(a) at the e2e level. From 04d0ed45cd6bb81bf22380ca78367568dbf0acbb Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 10:58:23 +0800 Subject: [PATCH 162/334] [fix](catalog) Hive connector cache: forward REFRESH invalidation to built siblings (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second clean-room review, delta finding (HIGH, latent-until-flip): fe-core routes REFRESH TABLE/DATABASE/CATALOG only to a catalog's PRIMARY connector, and the gateway's invalidateTable/invalidateDb/invalidateAll flushed only its own two cache layers — so nothing could ever drop the iceberg sibling's latestSnapshotCache for iceberg-on-HMS tables. That pin uses ACCESS-based expiry: a continuously-queried stale entry never dies, making staleness UNBOUNDED and breaking the signed-off acceptance "staleness bounded by TTL + explicit REFRESH" (legacy REFRESH did drop the iceberg engine cache via ExternalMetaCacheRouteResolver). It also weakened the follower replayRefreshTable "graceful coarsening": replay hits the same non-forwarding hooks. Fix mirrors close(): a forEachBuiltSibling helper does plain volatile reads of the iceberg/hudi sibling fields and forwards each hook to the ALREADY-BUILT siblings only — never force-builds one (a never-built sibling has no cache to drop; building just to flush could fail-loud spuriously when the sibling plugin is absent). Hudi sibling has no snapshot cache today, forwarding is a harmless no-op that keeps the contract uniform. Tests (HiveConnectorSiblingTest): forwarding of all three hooks to BOTH built siblings with the right arguments, and a no-force-build guard (buildCount stays 0 when hooks fire on a gateway whose siblings were never built). Dormant: hms is absent from SPI_READY_TYPES; nothing builds a HiveConnector for a live catalog. fe-connector-hive module: Tests run: 265, Failures: 0, Errors: 0; checkstyle 0 violations. See plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md §2.1. Co-Authored-By: Claude Fable 5 Claude-Session: e95529ec-63df-4dcf-b1a0-e9c39514be8d --- .../doris/connector/hive/HiveConnector.java | 27 +++++++ .../hive/HiveConnectorSiblingTest.java | 71 ++++++++++++++++++- 2 files changed, 97 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 7a8bf11754f876..ad5753fd181682 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -48,6 +48,7 @@ import java.util.Optional; import java.util.Set; import java.util.concurrent.Callable; +import java.util.function.Consumer; /** * Hive connector implementation. Manages the lifecycle of @@ -283,6 +284,7 @@ public Set getCapabilities() { * schema, partitions AND files are all mutable, so a REFRESH must re-read all of them (unlike iceberg, whose * manifests are immutable and kept across REFRESH TABLE). fe-core already routes {@code REFRESH TABLE} to * {@code connector.invalidateTable} for a plugin-driven catalog; dormant until hms enters SPI_READY_TYPES. + * Also forwarded to the already-built embedded siblings — see {@link #forEachBuiltSibling}. */ @Override public void invalidateTable(String dbName, String tableName) { @@ -290,6 +292,7 @@ public void invalidateTable(String dbName, String tableName) { // flush an empty cache): a never-built client means no metastore cache exists to flush. The file cache is // a final field and always present. invalidateTable(hmsClient, dbName, tableName); + forEachBuiltSibling(sibling -> sibling.invalidateTable(dbName, tableName)); } // Package-private seam: the metastore half needs an observable CachingHmsClient, which a unit test can build @@ -307,10 +310,12 @@ void invalidateTable(HmsClient client, String dbName, String tableName) { * directory listings). Same no-force-build read of the client as {@link #invalidateTable(String, String)}. * fe-core routes {@code REFRESH DATABASE} to {@code connector.invalidateDb} for a plugin-driven catalog; * dormant until hms enters SPI_READY_TYPES. + * Also forwarded to the already-built embedded siblings — see {@link #forEachBuiltSibling}. */ @Override public void invalidateDb(String dbName) { invalidateDb(hmsClient, dbName); + forEachBuiltSibling(sibling -> sibling.invalidateDb(dbName)); } // Package-private seam (see invalidateTable above). @@ -325,10 +330,12 @@ void invalidateDb(HmsClient client, String dbName) { * REFRESH CATALOG hook: drop ALL of this catalog's connector-owned scan caches — every metastore-metadata * entry ({@link CachingHmsClient#flushAll}) and every directory listing. Same no-force-build read of the * client as {@link #invalidateTable(String, String)}. Dormant until the flip. + * Also forwarded to the already-built embedded siblings — see {@link #forEachBuiltSibling}. */ @Override public void invalidateAll() { invalidateAll(hmsClient); + forEachBuiltSibling(Connector::invalidateAll); } // Package-private seam (see invalidateTable above). @@ -339,6 +346,26 @@ void invalidateAll(HmsClient client) { fileListingCache.invalidateAll(); } + /** + * Runs one invalidation action on each ALREADY-BUILT embedded sibling (iceberg, hudi). fe-core routes + * {@code REFRESH TABLE/DATABASE/CATALOG} only to a catalog's PRIMARY connector, so the gateway must + * propagate invalidation to the sibling-owned caches — e.g. the iceberg sibling's latest-snapshot pin, + * whose ACCESS-based expiry keeps a continuously-queried stale entry alive indefinitely, making an explicit + * REFRESH the only way to unpin an iceberg-on-HMS table's staleness. Mirrors {@link #close()}: plain + * volatile reads, never force-builds (a never-built sibling has no cache to drop, and building one just to + * flush it could fail-loud spuriously on a catalog whose sibling plugin is absent). + */ + private void forEachBuiltSibling(Consumer action) { + Connector iceberg = icebergSibling; + if (iceberg != null) { + action.accept(iceberg); + } + Connector hudi = hudiSibling; + if (hudi != null) { + action.accept(hudi); + } + } + /** The connector-owned directory-listing cache, exposed for unit tests (mirrors iceberg manifestCacheForTest). */ HiveFileListingCache fileListingCacheForTest() { return fileListingCache; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java index cb6cc64b5e66b5..38f7b8bac79a45 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorSiblingTest.java @@ -250,9 +250,63 @@ public Connector createSiblingConnector(String catalogType, Map } } - /** A bare {@link Connector} stand-in for the cross-loader iceberg sibling; only close-counting matters here. */ + @Test + public void refreshHooksForwardToBothBuiltSiblings() { + // WHY: fe-core routes REFRESH TABLE/DATABASE/CATALOG only to a catalog's PRIMARY connector. If the + // gateway did not forward its invalidate hooks, the iceberg sibling's latest-snapshot pin could NEVER + // be dropped by an explicit REFRESH — and its access-based expiry keeps a continuously-queried stale + // entry alive indefinitely, so staleness would be unbounded (breaks "bounded by TTL + REFRESH"). + FakeSibling icebergSibling = new FakeSibling(); + FakeSibling hudiSibling = new FakeSibling(); + FakeConnectorContext context = new FakeConnectorContext() { + @Override + public Connector createSiblingConnector(String catalogType, Map properties) { + return "hudi".equals(catalogType) ? hudiSibling : icebergSibling; + } + }; + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + connector.getOrCreateIcebergSibling(); + connector.getOrCreateHudiSibling(); + + connector.invalidateTable("db1", "t1"); + connector.invalidateDb("db2"); + connector.invalidateAll(); + + for (FakeSibling sibling : new FakeSibling[] {icebergSibling, hudiSibling}) { + Assertions.assertEquals("db1.t1", sibling.lastInvalidatedTable, + "invalidateTable must forward the (db, table) pair to each built sibling"); + Assertions.assertEquals("db2", sibling.lastInvalidatedDb, + "invalidateDb must forward the db to each built sibling"); + Assertions.assertEquals(1, sibling.invalidateAllCount, + "invalidateAll must forward to each built sibling exactly once"); + } + } + + @Test + public void refreshHooksNeverForceBuildSiblings() { + // WHY: a REFRESH on a pure-hive catalog must not construct sibling connectors — a never-built sibling + // has no cache to drop, and building one just to flush it would fail-loud spuriously whenever the + // sibling plugin is not installed (REFRESH would start throwing on plain hive catalogs). + RecordingSiblingContext context = new RecordingSiblingContext(new FakeSibling()); + HiveConnector connector = new HiveConnector(new HashMap<>(), context); + + connector.invalidateTable("db", "t"); + connector.invalidateDb("db"); + connector.invalidateAll(); + + Assertions.assertEquals(0, context.buildCount, + "invalidate hooks must only forward to ALREADY-BUILT siblings, never force-build one"); + } + + /** + * A bare {@link Connector} stand-in for the cross-loader iceberg/hudi sibling; records lifecycle + * ({@code close}) and invalidation forwarding. + */ private static final class FakeSibling implements Connector { int closeCount; + int invalidateAllCount; + String lastInvalidatedTable; + String lastInvalidatedDb; @Override public ConnectorMetadata getMetadata(ConnectorSession session) { @@ -263,5 +317,20 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { public void close() { closeCount++; } + + @Override + public void invalidateTable(String dbName, String tableName) { + lastInvalidatedTable = dbName + "." + tableName; + } + + @Override + public void invalidateDb(String dbName) { + lastInvalidatedDb = dbName; + } + + @Override + public void invalidateAll() { + invalidateAllCount++; + } } } From 8ca40220e2a8df64b8579f6fb3f92d2ad2d77afe Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 10:59:22 +0800 Subject: [PATCH 163/334] =?UTF-8?q?[doc](catalog)=20HANDOFF:=20record=20se?= =?UTF-8?q?cond=20clean-room=20review=20deltas=20=E2=80=94=201=20fixed,=20?= =?UTF-8?q?3=20user=20decisions=20pending=20before=20Model=20B?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: e95529ec-63df-4dcf-b1a0-e9c39514be8d --- plan-doc/HANDOFF.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index b9391f8daf814d..f4db99c17f2638 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -17,7 +17,14 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms D2 = `f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` 装饰器) → `7b05df6e55e` C-c(接线进 client) → `7c0ee1ffb2a` C-d(`HiveFileListingCache`) → `7bf90a7fe3c` C-e(`invalidateTable/invalidateAll`) → `12e0c9177c2` C-f(fe-core `getNewestUpdateVersionOrTime` last-modified 分支)。设计 = **`tasks/hive-connector-cache-step-design-2026-07-10.md`**。全休眠（hms 未进 `SPI_READY_TYPES`；paimon/iceberg/jdbc/hudi 字节不变；对齐 Trino 双层）。 **净室对抗复审**（9 维独立盲评 → 逐条对抗验证 → 交叉核对历史结论）= 7 疑点 2 坐实、5 误报/干净；报告 = **`plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md`**。两处坐实（用户拍板即修，均已落地）：`fda344e6022`（恢复 `FileSystem.get` 系统性失败大声报错、仅单目录失败 skip——修静默空结果回归）+ `cdc837563a7`（新增通用 `Connector.invalidateDb` SPI + 接 `RefreshManager.refreshDbInternal`，让 REFRESH DATABASE 清连接器两层缓存）。复审确认干净项：§2.6 单调性=与旧实现平价（删最新分区致 max 下降在新旧都抛，非新回归）、无空指针面、装饰器 pass-through 完整、TCCL 只在调用线程、Caffeine 单版本。**未做的低危建议**：`fe-connector-hms/pom.xml` 把 `fe-connector-cache` 标 `optional`（免那个空 Caffeine jar 流进 hudi 包，现无害但去掉未来隐患）——待用户点头。 -**⭐ 下一步 = Phase 1 第二项「事件管道 Model B」**（薄 fe-core 角色驱动 + 插件 `pollOnce` SPI，翻闸后重新武装 `MetastoreEventsProcessor` 的增量失效，权威 = `tasks/hms-event-pipeline-findings-2026-07-07.md`）。它**依赖** D2 缓存（在其之上加 `Connector.invalidatePartitions(db,table,names)`）。起步先读 findings + 对照 HEAD。 +**⚠ 第二次独立净室复审**（另一 session 同日同法复跑，14 疑点 12 坐实；差量报告 = **`plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md`**，与首审一致处不重复）**多抓 3 个坐实缺陷 + 1 个线上活性 bug**： +- ✅ 已修 `30ecd930319`：REFRESH 三钩子不转发给已构建的 iceberg/hudi 兄弟连接器（iceberg 快照 pin 是 access 过期、持续查询下永不失效 → REFRESH 是唯一逃生口却够不着 → staleness 无界）。修法镜像 `close()`。 +- ❌ **未修-待拍板 #1**（review2 §2.2，HIGH）：Doris 自发 `DROP TABLE`/`CREATE TABLE`/`DROP DATABASE` 不清连接器缓存 → 同名删建后按旧表 schema/location 出错误结果最长 24h（Trino 装饰器写后自失效；旧实现 unregisterTable 即清）。修法选项：插件侧自失效（休眠、推荐）vs fe-core DDL 路径接 `connector.invalidateTable/Db`（顺带修下条 live bug）vs 两者都做。 +- ❌ **未修-待拍板 #2**（review2 §2.3，MEDIUM）：分区缓存容量 100000 的**单位悄变**——旧=分区对象数，新=请求列表条数（无 weigher，重叠请求重复驻留）→ 高分区表+滑动谓词可致堆增长远超旧界。修法选项：共享 `fe-connector-cache` 框架加可选 weigher（Trino 同款，但破坏本系列「paimon/iceberg 字节不变」声明）vs 缩小默认值 vs 只记档。 +- ❌ **未修-待拍板 #3**（review2 §3，**线上活性 bug 非休眠**）：paimon/iceberg `latestSnapshotCache`（+`PaimonSchemaAtMemo` 窄化变体）今天就有同形状的删建重名洞（fe-core DDL 路径从不 invalidate、key=纯表名）。修=独立线（动 live 行为，须自带复审+回归），或记档延后。 +- 测试加固欠账（LOW，代码正确仅测试盲区）：`createClient` 真包裹无 pin、公开 invalidate 钩子未用已构建 client 驱动（review2 §2.4）。 + +**⭐ 下一步 = 先让用户拍板上面 3 项 → 落对应修复 → 再进 Phase 1 第二项「事件管道 Model B」**（薄 fe-core 角色驱动 + 插件 `pollOnce` SPI，翻闸后重新武装 `MetastoreEventsProcessor` 的增量失效，权威 = `tasks/hms-event-pipeline-findings-2026-07-07.md`）。它**依赖** D2 缓存（在其之上加 `Connector.invalidatePartitions(db,table,names)`）。起步先读 findings + 对照 HEAD。**⚠ 并行 session 风险**：本轮曾有两个 session 同时在此工作树工作（一个 amend 卷入另一个的未提交测试）；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 **⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步（读 / 写-拒 / schema-evolution / 缓存 / 跨加载器委派）只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`）。**删除排序最硬约束**：`datasource/hive|hudi|iceberg/` 的 ~90 个 HMS 支撑类删不掉，直到翻闸把消费者切到连接器路径（详见 execution plan §2.4/§3 + `tasks/iceberg-on-hms-delegation-findings-2026-07-07.md`）。 From 2318ade5b39a04ba7f23d1dffb96236066f56025 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 11:32:13 +0800 Subject: [PATCH 164/334] [doc](catalog) Hive event pipeline: Model B step design + user sign-off A (eager rebuild) Authority design for relocating the HMS metastore-event pipeline out of fe-core into the hive plugin (dormant, build-first). Grounded on a HEAD recon (wf_0c686fb4-d9d: 6-dim + completeness critic) + lead verification. Records: architecture (thin fe-core role-aware driver + plugin pollOnce returning neutral change-descriptors; new Connector.getEventSource / invalidatePartition SPI; TCCL pin around pollOnce); user sign-off A = structural table/db events use EAGER in-memory rebuild (parity), which also fixes the latent registerDatabase NotImplementedException on a flipped catalog; the 6-commit dormant sequence E-a..E-f; the flip-time rewire + e2e-owed list. Confirmed the connector partition cache is pull-through so an invalidate-only SPI expresses ADD_PARTITION. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- ...e-event-pipeline-step-design-2026-07-10.md | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md diff --git a/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md b/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md new file mode 100644 index 00000000000000..23eb7f41d91991 --- /dev/null +++ b/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md @@ -0,0 +1,94 @@ +# Hive 连接器 metastore-event 管道下沉 — 本步设计稿（2026-07-10） + +> 权威设计 = 本文件。基于一轮 HEAD-grounded 多维侦察（`wf_0c686fb4-d9d`：6 维盲评 + 完整性对抗 critic）+ lead 独立核对最载重事实 + 用户签字决策。**行号信 HEAD 不信文档**（每处实现前在 HEAD 复核）。 +> 上位：`tasks/hms-cutover-execution-plan-2026-07-10.md` §2.1（本步）。样板：`tasks/hive-connector-cache-step-design-2026-07-10.md`（上一步"连接器自持缓存"，本步在其之上加分区失效）。 + +--- + +## 0. 一句话 & 用户签字 + +把 HMS 通知事件管道从 fe-core 下沉到 hive 插件：fe-core 保留**连接器无关、角色感知**的薄驱动（主/从 + 编辑日志 + 游标 + 转发），插件只接管**抓取 + 解析**，通过新 SPI 回传**中立变更描述**。全休眠落地（hms 仍是 legacy 时完全惰性；paimon/iceberg/jdbc/hudi 字节不变）。对齐 Trino：引擎持有 HA/复制，插件持有取数/解析。 + +- **签字 A（2026-07-10，本步）** = 结构性事件（建/删/改名 库和表）用**即时重建（平价）**：事件到达即在内存挂库/表壳，从节点即时可见；不采"作废+惰性重建"。理由：即时重建成本低（`PluginDrivenExternalDatabase.buildTableInternal` 只 `new` 一个懒壳，不访问 metastore），零可观察行为变化，且顺带修一个通用目录上"未实现即抛异常"的隐患（`registerDatabase`）。 +- **默认（无需签字，本步一并落地）**： + - 分区事件在连接器侧只能做到**整表级**缓存失效（D2 的分区缓存按"整批名字列表"和目录 location 建键，无单名谓词）。因连接器缓存是**拉取式**（失效后下次 `listPartitionNames` 自动重列，见 §2 实证），正确性不受影响，仅略粗；**不为此重构刚落地的 D2 缓存**。描述符仍带分区名，供编辑日志与未来精确化。 + - 能力探测用**可空 provider 访问器** `Connector.getEventSource()`（非 `instanceof`、非布尔能力位）——pollOnce 带参且返数据，形状贴近 `getScanPlanProvider/getProcedureOps`，且把事件 API 从 99% 无此能力的连接器基座上移开。 + - **编辑日志留 fe-core**：驱动从描述符自建 `MetaIdMappingsLog` 并写盘；插件绝不触碰 `EditLog`。 + - **合并/去重（mergeEvents）放插件侧**：其依赖 HMS 类型的分组语义（`TableKey`/`canBeBatched`/跨事件 `removePartition`），pollOnce 返回**已合并**的描述符。 + - 每步 opt-in 开关（`isHmsEventsIncrementalSyncEnabled`）随连接器：`HiveConnector.getEventSource()` 在该目录未启用增量同步时返回 null（连接器读自己的 `HmsProperties`，fe-core 不解析属性）。 + +--- + +## 1. 关键实证（HEAD 核对，决定设计） + +1. **拉取式缓存 ⇒ 失效即可表达"新增分区"**。`CachingHmsClient.listPartitionNames`（:143-145）= `partitionNamesCache.get(key, key -> delegate.listPartitionNames(...))`：失效后下次 `get` miss → 重跑 loader → 新 Thrift RPC → 新分区出现。故**只需失效、无需 eager 插名**即可表达 `ADD_PARTITION`。`flush(db,table)`（:164-168）按 `invalidateIf(key.matches(db,table))` 丢该表全部条目 ⇒ 分区失效退化整表级，但拉取式保证正确。 +2. **游标写盘在工厂、仅 master**：`MetastoreEventFactory.getMetastoreEvents → logMetaIdMappings`（:86-102）构 `MetaIdMappingsLog(fromHmsEvent=true, lastSyncedEventId)` + 每事件 `transferToMetaIdMappings()`，`replayMetaIdMappingsLog`（本地）+ `editLog.logMetaIdMappingsLog`（复制）。**注意**：编辑日志写入当前**埋在 Model B 要下沉的 parse/merge 步里** — 拆分时必须把它**回提到 fe-core 驱动**，否则 master→follower 游标传播悄然中断（follower 永卡 masterLastSyncedEventId==-1）。 +3. **游标传播回放**：`ExternalMetaIdMgr.replayMetaIdMappingsLog`（:109-127）已 catalogId-keyed（无 HMS cast，a6dc782d816），`fromHmsEvent` 分支调 `getMetastoreEventsProcessor().updateMasterLastSyncedEventId(catalogId, lastSyncedEventId)`。id 映射仅喂**死 getter**（`getDbId/getTblId/getPartitionId` 零生产调用）；**活用途只剩游标**。opcode `OP_ADD_META_ID_MAPPINGS=470`（`OperationType:397`，回放 `EditLog:1414`，写 `EditLog:2580`）+ 中立 GSON 回放**必须存活**（旧 journal 兼容）。 +4. **fe-core mutator 现状**（8 个事件调用点；只被事件路径调用，泛化无旁支）： + - 已通用（翻转即用）：`CatalogMgr.unregisterExternalTable`（:679，走 `ExternalDatabase.unregisterTable`）；`RefreshManager.refreshExternalTableFromEvent/refreshTableInternal`（:224/:245，已含 D2 的 `PluginDrivenExternalCatalog.getConnector().invalidateTable` 钩子 :254-257）；`replayRefreshTable` rename 分支（:192-198）。 + - **多余 cast**（删即可，无行为变化）：`unregisterExternalDatabase`（:769 → 用通用 `ExternalCatalog.unregisterDatabase`，:1193 是唯一实现）；`refreshPartitions` 末尾 `setUpdateTime` cast（:297 → `ExternalTable.setUpdateTime` 通用 :311）。 + - **真阻塞**：`registerExternalDatabaseFromEvent`（:772）→ `catalog.registerDatabase`，通用 `ExternalCatalog.registerDatabase`（:1203）体是 `throw NotImplementedException`，只 `HMSExternalCatalog`（:205-215）override；`PluginDrivenExternalCatalog` **未 override** ⇒ 翻转后建/改名库事件**运行时抛异常**（编译通过、prod 才炸）。override 体（`buildDbForInit`+`metaCache.updateCache`+`Util.genIdByName`）全通用 ⇒ **上提到 `PluginDrivenExternalCatalog`**。 + - `registerExternalTableFromEvent`（:723）硬 cast `(HMSExternalCatalog)`(:742)+`(HMSExternalDatabase)`(:751) ⇒ 翻转 CCE。`buildTableForInit` 是 `ExternalDatabase:260` 通用 `T`，`PluginDrivenExternalDatabase.buildTableInternal`(:44-61) 返回正确子类（只 `new` 懒壳，**不访问 metastore**）⇒ 传通用 catalog/db、返回捕为 `ExternalTable`。 + - **分区缓存耦合**：`addExternalPartitions`（:792，`(HMSExternalTable)` :822 + `getPartitionColumnTypes` + `ExtMetaCacheMgr.hive(id).addPartitionsCache`）/ `dropExternalPartitions`（:835，`(HMSExternalTable)` :861 + `dropPartitionsCache`）/ `RefreshManager.refreshPartitions`（:263，`invalidatePartitionCache`）/ `replayRefreshTable` 分区分支（:200-221，gated `instanceof HMSExternalCatalog` + `refreshAffectedPartitionsCache((HMSExternalTable)...)`）—— 全戳 **D2 正在退休的 fe-core hive 缓存**。翻转后既 CCE 又丢失效 ⇒ 加 `PluginDrivenExternalCatalog` 分支路由到新 `Connector.invalidatePartition`。 +5. **SPI 现状**：`Connector` 已有 `invalidateTable/invalidateAll/invalidateDb`（:311/315/323，D2，活）；**无** `invalidatePartition`、**无** pollOnce/事件方法。`ConnectorCapability` 无事件位。旧 `ConnectorMetaInvalidator`（5 动词，`invalidatePartition` 带 VALUES 退化整表）**仍死**（零生产调用）——**不复用它**，走 D2 活 SPI 族。插件 `HmsClient` **无** `getNextNotification/getCurrentNotificationEventId`（旧 fe-core `HMSCachedClient:74-78` 是模板；底层 vendored `HiveMetaStoreClient:3012/3043` 已实现 Thrift）。 +6. **TCCL/R-010**：poller 路径**当前零 pin**（跨 `event/` grep 无 classloader）。样板 = `PluginDrivenScanNode.onPluginClassLoader`（:517-525，pin 到 `provider.getClass().getClassLoader()`）。驱动是 `MasterDaemon` 后台线程（不继承调用方 pin），须**显式**在每次 `pollOnce` 外 try/finally pin 到 `getEventSource().getClass().getClassLoader()`（=插件加载器），覆盖 RPC + JSON/GZIP 反序列化。`ThriftHmsClient.doAs` 只 pin SYSTEM 加载器且即刻还原，**不够**。 +7. **无连接器现在轮询事件**（iceberg/paimon-native 仅手动 REFRESH）⇒ 本步是全新插件+SPI 管道，非镜像。 + +--- + +## 2. 目标架构（Model B，全休眠） + +### 2.1 新 SPI（fe-connector-api / -spi，全 default，其余连接器零感知） +- **`MetastoreChangeDescriptor`**（中立 POJO，仅原语/枚举/名字，**无任何 HMS 类型**——R-010 依赖此）： + - `op`：`REGISTER_DATABASE / UNREGISTER_DATABASE / RENAME_DATABASE / REGISTER_TABLE / UNREGISTER_TABLE / RENAME_TABLE(含视图重建，after 可==before) / REFRESH_TABLE / INVALIDATE_PARTITIONS(kind∈ADD/DROP/REFRESH)` + - 载荷：`dbName, tableName`（remote），`dbNameAfter/tableNameAfter`（rename），`partitionNames: List`（**规范名 `col=val/...`，非 values**），`updateTime`（ms），`eventId`（long） +- **`ConnectorEventSource`**：`long getCurrentEventId()` + `EventPollResult pollOnce(EventPollRequest req)` + - `EventPollRequest{ long lastSyncedEventId, boolean isMaster, long masterUpperBound }`（follower 用 upperBound 上界；batchSize 连接器自读配置） + - `EventPollResult{ long newCursor, List descriptors, boolean needsFullRefresh }`（覆盖 master 首拉/`REPL_EVENTS_MISSING`、follower 首拉的"推进游标+无事件+需全刷"分支） +- **`Connector.getEventSource()`** → `ConnectorEventSource`（default `null`；探测=非空） +- **`Connector.invalidatePartition(String dbName, String tableName, List partitionNames)`**（default no-op，名字键；D2 活族新成员） + +### 2.2 插件侧（fe-connector-hms / -hive；只接管抓取+解析） +- `HmsClient` 加 `getCurrentNotificationEventId()` + `getNextNotification(lastEventId, maxEvents, filter)`（default 抛/空，避免测试替身破裂）；`ThriftHmsClient` 委派 vendored `HiveMetaStoreClient`；`CachingHmsClient` 透传。 +- 反序列化器（`GzipJSONMessageDeserializer` + JSON 选择器）+ 事件→描述符映射 + `mergeEvents` 合并 **移入插件**。 +- `HmsEventSource implements ConnectorEventSource`：master 分支 `getCurrentEventId` 比较+拉取；follower 分支按 upperBound 拉取；解析→合并→中立描述符；`REPL_EVENTS_MISSING` 检测→`needsFullRefresh=true`。 +- `HiveConnector.getEventSource()`：增量同步启用时返回 `HmsEventSource`，否则 null（读自身 `HmsProperties`，try/catch→null 保"未初始化即跳过"语义）。覆盖**整个 HMS 目录所有格式**（hive+iceberg-on-HMS+hudi-on-HMS，名字键，无 DLAType）。 +- `HiveConnector.invalidatePartition(...)`：镜像 `invalidateTable` 三段式 —— `CachingHmsClient.flush`（整表级）+ `HiveFileListingCache.invalidateTable`（location 键无法按名精确，退整表）+ `forEachBuiltSibling` 转发。 + +### 2.3 fe-core 侧 +- **mutator 泛化（§1.4，全向后兼容 legacy，休眠安全）**：删 `registerExternalTableFromEvent` 两处 cast；`registerDatabase` 上提 `PluginDrivenExternalCatalog`；删 `unregisterExternalDatabase` 多余 cast；`addExternalPartitions/dropExternalPartitions/refreshPartitions/replayRefreshTable 分区分支`加 `if(PluginDrivenExternalCatalog) connector.invalidatePartition(...)` 分支（与现有 `instanceof HMSExternalTable` 分支互斥；pre-flip 走 HMS 分支不变）。 +- **新驱动 `MetastoreEventSyncDriver extends MasterDaemon`**（**存活包** `datasource/`，非将删的 `datasource/hive/event/`）： + - 迭代 `getCatalogIds()`；`catalog instanceof PluginDrivenExternalCatalog && getConnector().getEventSource()!=null` 才处理（能力探测，非 `instanceof HMS*`）。 + - 自持两 `Map`（lastSynced + masterLastSynced）；`Env.isMaster()` 分角色。 + - `onPluginClassLoader` pin 外包 `pollOnce`；据 `EventPollResult`：`needsFullRefresh` → master `replayRefreshCatalog` / follower 转发 `REFRESH CATALOG`；否则应用描述符（走泛化 mutator + `connector.invalidatePartition`）；master 从描述符自建 `MetaIdMappingsLog`（op→ADD/DELETE×objType×names，**修 drop-partition 误标 DATABASE 的旧 bug 为 PARTITION**，喂 `nextMetaId` 保编辑日志形状）+ `replayMetaIdMappingsLog` + `editLog.logMetaIdMappingsLog`；存 `newCursor`。 + - 构造+启动**休眠**（`Env` 内，仿老 poller）：pre-flip 无匹配目录 → 惰性、不写编辑日志、游标空。 + +### 2.4 翻转时接线（Phase 2，非本步；本步只留清单） +- `MetastoreEventsProcessor:116` 老 gate 去除；`ExternalMetaIdMgr.replayMetaIdMappingsLog` 的游标回放**改指新驱动**（或双写，二选一，翻转时定）；启用新驱动对 flipped-hms 生效。老 poller + `datasource/hive/event/` 整目录 **Phase 3 删**（新驱动替换入口后）。 + +--- + +## 3. 提交序列（各休眠、独立、可复审） + +| # | 提交 | 内容 | 休眠性 | +|---|---|---|---| +| E-a | feat SPI | `MetastoreChangeDescriptor`(+op enum) / `ConnectorEventSource` / `EventPollRequest/Result` / `Connector.getEventSource()` default null / `Connector.invalidatePartition` default no-op | 全 default，零连接器感知 | +| E-b | feat 插件取数 | `HmsClient.getCurrentNotificationEventId/getNextNotification` + `ThriftHmsClient` 委派 + `CachingHmsClient` 透传 | 无调用者 | +| E-c | feat 插件事件源 | 反序列化器+event→descriptor+merge 移入插件；`HmsEventSource`；`HiveConnector.getEventSource/invalidatePartition` | hms 未 SPI_READY | +| E-d | feat fe-core mutator 泛化 | de-cast + `registerDatabase` 上提 + 分区 PluginDriven 分支（互斥、加性） | PluginDriven 分支 pre-flip 不可达 | +| E-e | feat fe-core 驱动 | `MetastoreEventSyncDriver` + Env 构造/启动 | pre-flip 无匹配目录 | +| E-f | test | 描述符映射、`registerDatabase` on PluginDriven 不抛、驱动能力探测跳过非事件连接器等休眠单测 | — | + +设计稿 + HANDOFF 单独 commit（与 code 分开）。每步 `mvn -pl -am test` + `check-connector-imports.sh` + checkstyle。 + +--- + +## 4. e2e 欠账（翻转后，勿静默丢，Rule 12） +异构 HMS docker：CREATE/DROP/ALTER(rename)/INSERT + ADD/DROP/ALTER PARTITION 事件到达后 FE 元数据同步正确（master + follower 双跑）；跨类加载器 pin 无 CCE（child-first fixture，单测同加载器测不出 R-010）；`HmsGsonCompatReplayTest` 旁的 opcode-470 旧 journal 回放；heterogeneous 事件覆盖 iceberg-on-HMS/hudi-on-HMS。 + +## 5. 铁律核对 +- fe-core 无新增 `instanceof HMS*`/`switch(dlaType)`（新驱动用 `PluginDrivenExternalCatalog`+能力探测）✓ +- fe-core 不解析属性（enable 开关随连接器）✓ +- 跨边界 pin TCCL（驱动线程包 pollOnce）✓ +- `history_schema_info` 逐层小写：本步不涉 schema 字典 ✓ +- `PluginDrivenMvccExternalTable` 字节+成本双不变：本步不改其共享方法（只经 `buildTableInternal` 新建，走既有路径）✓ From a5515a657e78493181a669f28e6baa94f6148ad4 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 11:36:28 +0800 Subject: [PATCH 165/334] [feat](catalog) Hive event pipeline: neutral event SPI types (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit E-a of the metastore-event relocation. Adds the connector-neutral vocabulary the engine's future role-aware driver will use, all as additive SPI defaults so every existing connector is byte-unaffected: - MetastoreChangeDescriptor (+ Op enum) — one neutral metadata change (register/unregister/rename db+table, refresh table, add/drop/refresh partitions by canonical NAME). No source-native type crosses it. - ConnectorEventSource — getCurrentEventId() + pollOnce(request); the plugin owns fetch+parse only. - EventPollRequest / EventPollResult — the engine-owned cursor + role in, the new cursor + descriptors + needsFullRefresh out. - Connector.getEventSource() default null (capability-probe getter, not instanceof) + Connector.invalidatePartition(db,table,names) default no-op (name-keyed, joins the D2 invalidate* family). Dormant: no producer/consumer yet (E-c wires the hive plugin, E-e the fe-core driver). fe-connector-api compiles; checkstyle 0; import gate ok. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/connector/api/Connector.java | 25 +++ .../api/event/ConnectorEventSource.java | 58 ++++++ .../connector/api/event/EventPollRequest.java | 66 +++++++ .../connector/api/event/EventPollResult.java | 89 +++++++++ .../api/event/MetastoreChangeDescriptor.java | 170 ++++++++++++++++++ 5 files changed, 408 insertions(+) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollResult.java create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java index 98a7c743539fbc..e1bdc83c088324 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/Connector.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.api; +import org.apache.doris.connector.api.event.ConnectorEventSource; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; @@ -323,6 +324,30 @@ default void invalidateAll() { default void invalidateDb(String dbName) { } + /** + * Invalidates the connector-side caches for specific partitions of a table so a subsequent read + * reflects the latest external state. Driven by the engine's metastore-event sync when partitions + * are added/dropped/altered. The names are the REMOTE db/table names and canonical partition names + * ({@code "col=val/.../colN=valN"}); an empty/whole-table drop is expressed by + * {@link #invalidateTable(String, String)}. A connector whose partition cache cannot target a single + * name may degrade to invalidating the whole table's partition caches (correctness-safe when the + * cache re-lists on miss). Default no-op for connectors that cache nothing. + */ + default void invalidatePartition(String dbName, String tableName, List partitionNames) { + } + + /** + * Returns this connector's incremental metadata-change source, or {@code null} if it has none. + * A capability-probe getter (mirrors {@link #getScanPlanProvider()} / {@link #getProcedureOps()}): + * the engine's single, connector-agnostic, role-aware event driver iterates catalogs and calls + * {@link ConnectorEventSource#pollOnce} only on connectors that expose a source, never via + * {@code instanceof}. The default returns {@code null}, so every connector without a metastore-event + * feed is unaffected. + */ + default ConnectorEventSource getEventSource() { + return null; + } + /** * Optional per-connector override of the catalog's schema-cache TTL (in seconds), consulted generically by * the engine when sizing the schema meta-cache. Semantics match {@code schema.cache.ttl-second}: diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java new file mode 100644 index 00000000000000..1955ef681ed11e --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java @@ -0,0 +1,58 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.event; + +/** + * A connector's incremental-metadata-change source: it fetches the underlying metastore's native + * notification events and parses them into connector-neutral {@link MetastoreChangeDescriptor}s. + * + *

    Surfaced via {@link org.apache.doris.connector.api.Connector#getEventSource()} (a + * capability-probe getter, NOT an {@code instanceof} — a connector without an event source returns + * {@code null}). The engine runs one connector-agnostic, role-aware background driver that iterates + * catalogs, calls {@link #pollOnce} on those whose connector exposes a source, and applies the + * returned descriptors to its own object graph and caches.

    + * + *

    Engine/connector split (Trino-aligned: engine owns HA/replication, plugin owns fetch/parse). + * The connector owns ONLY the metastore RPC and the message deserialization/merge behind {@link + * #getCurrentEventId()} and {@link #pollOnce}. The engine owns everything stateful and replicated: the + * per-catalog cursor (passed in via {@link EventPollRequest}, stored from {@link EventPollResult}), + * the master/follower role, the edit-log write of the synced cursor, and the follower→master + * {@code REFRESH CATALOG} forward. The connector is stateless with respect to the cursor.

    + * + *

    Classloader. The engine calls these methods under a context-classloader pin to this + * source's own plugin classloader, so the notification RPC and the JSON/GZIP deserialization inside + * resolve the plugin's bundled metastore classes. Implementations therefore need no pin of their own + * for the fetch/parse path.

    + */ +public interface ConnectorEventSource { + + /** + * The metastore's current (latest) notification event id. Used by the master to cheaply decide + * whether there is anything new to pull before calling {@link #pollOnce}. Returns {@code -1} if the + * id cannot be read this cycle. + */ + long getCurrentEventId(); + + /** + * Fetch and parse one batch of notification events for a catalog, returning connector-neutral + * descriptors plus the new cursor and an optional full-refresh signal. Never returns source-native + * types. See {@link EventPollRequest} / {@link EventPollResult} for the role/cursor/refresh + * semantics. The batch size is the connector's own concern (read from its catalog properties). + */ + EventPollResult pollOnce(EventPollRequest request); +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java new file mode 100644 index 00000000000000..4662c308b22e51 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.event; + +/** + * The engine-supplied input to one {@link ConnectorEventSource#pollOnce} call. The engine owns the + * cursor and the role; the connector owns the fetch strategy those imply. + * + *
      + *
    • {@link #getLastSyncedEventId()} — the last event id this FE has already applied for the + * catalog (the engine's per-catalog cursor; {@code -1} means "never synced" → the connector + * should signal a full refresh rather than replay from 0).
      • + *
    • {@link #isMaster()} — whether this FE is the master. The master fetches the metastore's + * current event id and pulls new events directly; a follower pulls only up to + * {@link #getMasterUpperBound()} (what the master has already committed and replicated), so it + * never reads past the master's high-water mark.
      • + *
    • {@link #getMasterUpperBound()} — the master's committed event id, learned by the follower via + * edit-log replay; ignored on the master. {@code -1} means "not yet learned" → a follower + * does nothing this cycle.
      • + *
    + */ +public final class EventPollRequest { + + private final long lastSyncedEventId; + private final boolean isMaster; + private final long masterUpperBound; + + public EventPollRequest(long lastSyncedEventId, boolean isMaster, long masterUpperBound) { + this.lastSyncedEventId = lastSyncedEventId; + this.isMaster = isMaster; + this.masterUpperBound = masterUpperBound; + } + + public long getLastSyncedEventId() { + return lastSyncedEventId; + } + + public boolean isMaster() { + return isMaster; + } + + public long getMasterUpperBound() { + return masterUpperBound; + } + + @Override + public String toString() { + return "EventPollRequest{lastSyncedEventId=" + lastSyncedEventId + ", isMaster=" + isMaster + + ", masterUpperBound=" + masterUpperBound + '}'; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollResult.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollResult.java new file mode 100644 index 00000000000000..9673ddc3fe44bc --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollResult.java @@ -0,0 +1,89 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.event; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; + +/** + * The connector's answer to one {@link ConnectorEventSource#pollOnce} call. + * + *
      + *
    • {@link #getNewCursor()} — the event id the engine should store as its new per-catalog cursor. + * This is NOT always {@code lastSyncedEventId + descriptors.size()}: on a first sync or an + * events-gap recovery the connector advances the cursor to the metastore's current id WITHOUT + * emitting events (paired with {@link #isNeedsFullRefresh()}); on an empty poll it is the + * unchanged cursor.
      • + *
    • {@link #getDescriptors()} — the already-merged, connector-neutral changes to apply, in order. + * Empty when nothing changed or when a full refresh is requested instead.
      • + *
    • {@link #isNeedsFullRefresh()} — the connector could not produce an incremental delta (first + * sync, or the metastore reported its notification log was trimmed past the cursor). The engine + * responds per role: the master invalidates the whole catalog; a follower forwards + * {@code REFRESH CATALOG} to the master. The connector owns detecting the source-specific gap + * condition; the engine owns the HA response.
      • + *
    + */ +public final class EventPollResult { + + private final long newCursor; + private final List descriptors; + private final boolean needsFullRefresh; + + public EventPollResult(long newCursor, List descriptors, + boolean needsFullRefresh) { + this.newCursor = newCursor; + this.descriptors = descriptors == null + ? Collections.emptyList() + : Collections.unmodifiableList(new ArrayList<>(descriptors)); + this.needsFullRefresh = needsFullRefresh; + } + + /** A result carrying incremental changes; the cursor advances to {@code newCursor}. */ + public static EventPollResult ofChanges(long newCursor, List descriptors) { + return new EventPollResult(newCursor, descriptors, false); + } + + /** A result requesting a full catalog refresh and seeding the cursor to {@code newCursor}. */ + public static EventPollResult ofFullRefresh(long newCursor) { + return new EventPollResult(newCursor, Collections.emptyList(), true); + } + + /** A no-op result: nothing changed, keep the cursor at {@code cursor}. */ + public static EventPollResult ofNothing(long cursor) { + return new EventPollResult(cursor, Collections.emptyList(), false); + } + + public long getNewCursor() { + return newCursor; + } + + public List getDescriptors() { + return descriptors; + } + + public boolean isNeedsFullRefresh() { + return needsFullRefresh; + } + + @Override + public String toString() { + return "EventPollResult{newCursor=" + newCursor + ", descriptors=" + descriptors.size() + + ", needsFullRefresh=" + needsFullRefresh + '}'; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java new file mode 100644 index 00000000000000..1910dd25aaecb4 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java @@ -0,0 +1,170 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.event; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * A single connector-neutral description of one metadata change the engine must apply after a + * metastore poll. It is the vocabulary {@link ConnectorEventSource#pollOnce} returns: the plugin + * fetches and parses the source's native notification events (Hive {@code NotificationEvent} + + * JSON/GZIP messages, iceberg/hudi-on-HMS included) and maps each to one of these, so the engine can + * update its catalog→db→table object graph and caches WITHOUT ever seeing a source-native type. + * + *

    Type-neutrality is load-bearing. Every field is a primitive, a {@code String}, or a + * {@code List} — no Hive/thrift class ever crosses this boundary. The engine calls the plugin + * under a classloader pin; if a plugin-loaded object leaked into a field, a field access on the engine + * side would fail across the loader split. Keep it neutral.

    + * + *

    Engine/connector split. The connector owns fetch+parse and emits these descriptors + * (already merged/deduplicated). The engine owns the structural application — register/unregister/ + * rename in {@code CatalogMgr}, cache invalidation via {@code Connector.invalidateTable/Db/Partition}, + * the master's edit-log write, and follower/master role handling. Trino-aligned: the plugin surfaces + * what changed; the engine decides how its own metadata reacts.

    + * + *

    Partition granularity is by canonical partition NAME ({@code "col=val/.../colN=valN"}), never by + * raw column values — the caches are name-keyed, so carrying names avoids the values→whole-table + * degrade.

    + */ +public final class MetastoreChangeDescriptor { + + /** + * The kind of change. Maps 1:1 to what the engine does when applying the descriptor. A source + * event that changes nothing the engine tracks (e.g. a properties-only ALTER DATABASE, or an + * unsupported event type) produces NO descriptor rather than a no-op {@code Op}. + */ + public enum Op { + /** A new database exists; register it into the catalog. */ + REGISTER_DATABASE, + /** A database was dropped; unregister it from the catalog. */ + UNREGISTER_DATABASE, + /** A database was renamed; {@link #getDbName()} → {@link #getDbNameAfter()}. */ + RENAME_DATABASE, + /** A new table exists; register it into its database. */ + REGISTER_TABLE, + /** A table was dropped; unregister it from its database. */ + UNREGISTER_TABLE, + /** + * A table was renamed OR a view was recreated; {@link #getTableName()} → + * {@link #getTableNameAfter()} (the after-name equals the before-name for a view recreate, + * which rebuilds the table object so a changed view definition takes effect). + */ + RENAME_TABLE, + /** A table changed in place (insert / plain alter); drop its caches, keep it registered. */ + REFRESH_TABLE, + /** Partitions were added; invalidate the table's partition caches so a re-list picks them up. */ + ADD_PARTITIONS, + /** Partitions were dropped; invalidate the table's partition caches. */ + DROP_PARTITIONS, + /** Partitions changed in place; invalidate the named partitions' caches. */ + REFRESH_PARTITIONS + } + + private final Op op; + private final String dbName; + private final String tableName; + private final String dbNameAfter; + private final String tableNameAfter; + private final List partitionNames; + private final long updateTime; + private final long eventId; + + private MetastoreChangeDescriptor(Op op, String dbName, String tableName, String dbNameAfter, + String tableNameAfter, List partitionNames, long updateTime, long eventId) { + this.op = Objects.requireNonNull(op, "op"); + this.dbName = dbName; + this.tableName = tableName; + this.dbNameAfter = dbNameAfter; + this.tableNameAfter = tableNameAfter; + this.partitionNames = partitionNames == null + ? Collections.emptyList() + : Collections.unmodifiableList(new ArrayList<>(partitionNames)); + this.updateTime = updateTime; + this.eventId = eventId; + } + + /** A database-level change ({@code REGISTER_/UNREGISTER_/RENAME_DATABASE}). */ + public static MetastoreChangeDescriptor forDatabase(Op op, String dbName, String dbNameAfter, + long eventId, long updateTime) { + return new MetastoreChangeDescriptor(op, dbName, null, dbNameAfter, null, null, updateTime, eventId); + } + + /** A table-level change ({@code REGISTER_/UNREGISTER_/RENAME_TABLE}, {@code REFRESH_TABLE}). */ + public static MetastoreChangeDescriptor forTable(Op op, String dbName, String tableName, + String tableNameAfter, long eventId, long updateTime) { + return new MetastoreChangeDescriptor(op, dbName, tableName, null, tableNameAfter, null, + updateTime, eventId); + } + + /** A partition-level change ({@code ADD_/DROP_/REFRESH_PARTITIONS}). */ + public static MetastoreChangeDescriptor forPartitions(Op op, String dbName, String tableName, + List partitionNames, long eventId, long updateTime) { + return new MetastoreChangeDescriptor(op, dbName, tableName, null, null, partitionNames, + updateTime, eventId); + } + + public Op getOp() { + return op; + } + + /** The remote database name (as the connector sees it). */ + public String getDbName() { + return dbName; + } + + /** The remote table name, or {@code null} for database-level ops. */ + public String getTableName() { + return tableName; + } + + /** The new database name for {@link Op#RENAME_DATABASE}, else {@code null}. */ + public String getDbNameAfter() { + return dbNameAfter; + } + + /** The new table name for {@link Op#RENAME_TABLE} (may equal {@link #getTableName()}), else {@code null}. */ + public String getTableNameAfter() { + return tableNameAfter; + } + + /** Canonical partition names for partition ops (empty for non-partition ops). */ + public List getPartitionNames() { + return partitionNames; + } + + /** The source update time in epoch millis, used as the object's freshness signal. */ + public long getUpdateTime() { + return updateTime; + } + + /** The source notification event id this descriptor was derived from. */ + public long getEventId() { + return eventId; + } + + @Override + public String toString() { + return "MetastoreChangeDescriptor{op=" + op + ", db=" + dbName + ", table=" + tableName + + ", dbAfter=" + dbNameAfter + ", tableAfter=" + tableNameAfter + + ", partitions=" + partitionNames + ", updateTime=" + updateTime + + ", eventId=" + eventId + '}'; + } +} From 861ce2e212040bbad106eaae83d5a705408a8b25 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 11:41:36 +0800 Subject: [PATCH 166/334] [feat](catalog) Hive event pipeline: plugin notification fetch on HmsClient (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit E-b of the metastore-event relocation. Severs the FETCH half plugin-side: the notification RPCs the fe-core poller does today via HMSCachedClient move onto the plugin's SPI-clean HmsClient, so the future event source never reaches back into fe-core for them. - HmsNotificationEvent — SPI-clean DTO (eventId/type/db/table/message/ format/time), the neutral projection of Hive's NotificationEvent so no thrift type crosses HmsClient. - HmsClient: default (throwing) getCurrentNotificationEventId() + getNextNotification(lastEventId, maxEvents) — Phase-5 block, defaults so read-only test doubles keep compiling. - ThriftHmsClient overrides both, delegating to the vendored HiveMetaStoreClient via execute(...); the REPL_EVENTS_MISSING sentinel survives in the wrapped HmsClientException for the caller's full-refresh fallback. - CachingHmsClient passes both through un-cached (notifications are never cached). Dormant: no caller yet (E-c's HmsEventSource consumes these). fe-connector-hms compiles; checkstyle 0; import gate ok. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hms/CachingHmsClient.java | 10 +++ .../apache/doris/connector/hms/HmsClient.java | 33 +++++++ .../connector/hms/HmsNotificationEvent.java | 90 +++++++++++++++++++ .../doris/connector/hms/ThriftHmsClient.java | 28 ++++++ 4 files changed, 161 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsNotificationEvent.java diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index 9bb5b15d71f3d0..b89e9a60454cf5 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -290,6 +290,16 @@ public void acquireSharedLock(String queryId, long txnId, String user, String db delegate.acquireSharedLock(queryId, txnId, user, dbName, tableName, partitionNames, timeoutMs); } + @Override + public long getCurrentNotificationEventId() { + return delegate.getCurrentNotificationEventId(); + } + + @Override + public List getNextNotification(long lastEventId, int maxEvents) { + return delegate.getNextNotification(lastEventId, maxEvents); + } + @Override public void close() throws IOException { delegate.close(); diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java index c453cf147ef3d8..4810305d7d7db2 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java @@ -349,4 +349,37 @@ default void acquireSharedLock(String queryId, long txnId, String user, String d String tableName, List partitionNames, long timeoutMs) { throw new UnsupportedOperationException("acquireSharedLock is not supported by this client"); } + + // ========== Phase 5: Metastore notification events (incremental metadata sync) ========== + // + // Optional: the incremental-metadata event feed. Only the production {@link ThriftHmsClient} + // (and the {@link CachingHmsClient} pass-through) implement these; read-only test doubles and + // connectors without an event feed keep the throwing defaults. + + /** + * The metastore's current (latest) notification event id, or {@code -1} if unavailable. Used to + * cheaply decide whether there is anything new to pull. + * + * @throws HmsClientException if the operation fails + */ + default long getCurrentNotificationEventId() { + throw new UnsupportedOperationException( + "getCurrentNotificationEventId is not supported by this client"); + } + + /** + * Fetch the next batch of notification events after {@code lastEventId} (exclusive), up to + * {@code maxEvents}, as SPI-clean DTOs. + * + * @param lastEventId the last event id already consumed + * @param maxEvents maximum number of events to return + * @return the events in ascending id order (empty when none) + * @throws HmsClientException if the operation fails; when the metastore has trimmed its + * notification log past {@code lastEventId} the message carries the + * {@code REPL_EVENTS_MISSING_IN_METASTORE} sentinel so the caller can fall back to a + * full refresh + */ + default List getNextNotification(long lastEventId, int maxEvents) { + throw new UnsupportedOperationException("getNextNotification is not supported by this client"); + } } diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsNotificationEvent.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsNotificationEvent.java new file mode 100644 index 00000000000000..ae8b01d7986289 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsNotificationEvent.java @@ -0,0 +1,90 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +/** + * One HMS notification event as a connector-SPI DTO — the SPI-clean projection of Hive's + * {@code NotificationEvent} that {@link HmsClient#getNextNotification} returns, so no Hive thrift type + * crosses the client interface. It carries exactly the fields the plugin's event parser needs to build + * a neutral change descriptor: the event id + type + affected db/table, and the raw message payload + + * its format so the JSON/GZIP message deserializers can extract the details (partition names, renamed + * objects, etc.). + */ +public final class HmsNotificationEvent { + + private final long eventId; + private final String eventType; + private final String dbName; + private final String tableName; + private final String message; + private final String messageFormat; + private final long eventTime; + + public HmsNotificationEvent(long eventId, String eventType, String dbName, String tableName, + String message, String messageFormat, long eventTime) { + this.eventId = eventId; + this.eventType = eventType; + this.dbName = dbName; + this.tableName = tableName; + this.message = message; + this.messageFormat = messageFormat; + this.eventTime = eventTime; + } + + /** The event's unique incremental id. */ + public long getEventId() { + return eventId; + } + + /** The event type name (e.g. {@code CREATE_TABLE}, {@code ADD_PARTITION}). */ + public String getEventType() { + return eventType; + } + + /** The affected database name (may be {@code null} for some event types). */ + public String getDbName() { + return dbName; + } + + /** The affected table name (may be {@code null} for database-level events). */ + public String getTableName() { + return tableName; + } + + /** The raw message payload, to be parsed by the message deserializer for this event's format. */ + public String getMessage() { + return message; + } + + /** The message format (e.g. {@code json-0.2}, {@code gzip(json-0.2)}); selects the deserializer. */ + public String getMessageFormat() { + return messageFormat; + } + + /** The event time in epoch seconds (as recorded by the metastore). */ + public long getEventTime() { + return eventTime; + } + + @Override + public String toString() { + return "HmsNotificationEvent{eventId=" + eventId + ", eventType=" + eventType + + ", db=" + dbName + ", table=" + tableName + ", messageFormat=" + messageFormat + + ", eventTime=" + eventTime + '}'; + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 31660e7acd3fd2..e0e03f9d1699c5 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -39,6 +39,7 @@ import org.apache.hadoop.hive.metastore.RetryingMetaStoreClient; import org.apache.hadoop.hive.metastore.api.ColumnStatisticsData; import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj; +import org.apache.hadoop.hive.metastore.api.CurrentNotificationEventId; import org.apache.hadoop.hive.metastore.api.DataOperationType; import org.apache.hadoop.hive.metastore.api.Database; import org.apache.hadoop.hive.metastore.api.DateColumnStatsData; @@ -52,6 +53,8 @@ import org.apache.hadoop.hive.metastore.api.LongColumnStatsData; import org.apache.hadoop.hive.metastore.api.MetaException; import org.apache.hadoop.hive.metastore.api.NoSuchObjectException; +import org.apache.hadoop.hive.metastore.api.NotificationEvent; +import org.apache.hadoop.hive.metastore.api.NotificationEventResponse; import org.apache.hadoop.hive.metastore.api.Partition; import org.apache.hadoop.hive.metastore.api.SQLDefaultConstraint; import org.apache.hadoop.hive.metastore.api.StorageDescriptor; @@ -536,6 +539,31 @@ public void acquireSharedLock(String queryId, long txnId, String user, String db } } + @Override + public long getCurrentNotificationEventId() { + return execute(client -> { + CurrentNotificationEventId id = client.getCurrentNotificationEventId(); + return id == null ? -1L : id.getEventId(); + }); + } + + @Override + public List getNextNotification(long lastEventId, int maxEvents) { + return execute(client -> { + NotificationEventResponse response = + client.getNextNotification(lastEventId, maxEvents, null); + List events = new ArrayList<>(); + if (response != null && response.getEvents() != null) { + for (NotificationEvent event : response.getEvents()) { + events.add(new HmsNotificationEvent(event.getEventId(), event.getEventType(), + event.getDbName(), event.getTableName(), event.getMessage(), + event.getMessageFormat(), event.getEventTime())); + } + } + return events; + }); + } + private static List createLockComponentsForRead(String dbName, String tableName, List partitionNames) { List components = new ArrayList<>( From 0e6ac6ae704889b43a59a65a9d2c443a1e7fc2c2 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 11:55:39 +0800 Subject: [PATCH 167/334] [feat](catalog) Hive event pipeline: plugin event source + HiveConnector wiring (dormant) E-c of the metastore-event relocation. Severs the PARSE half plugin-side and wires the hive connector's event source, so a flipped catalog will poll+parse HMS notifications entirely inside the plugin. - HmsEventParser: faithful event->neutral-descriptor mapping replacing the fe-core datasource.hive.event.* process() bodies. Preserves the legacy semantics exactly (table-name lowercasing, rename vs view-recreate, alter-partition rename = drop+add, canonical partition names, gzip message decompression via inline base64+gzip). No CatalogMgr/EditLog. - HmsEventSource implements ConnectorEventSource: role-aware fetch (master reads current id + pulls; follower is bounded by the master's committed upper bound), REPL_EVENTS_MISSING -> needsFullRefresh, cursor in/out via the request/result. Stateless w.r.t. the cursor (engine-owned). - MetastoreChangeDescriptor.forTableRename: table rename may cross dbs, so carry before+after db/table (also used for view-recreate, after==before). - HiveConnector.getEventSource(): per-catalog opt-in read from the connector's own props (hive.enable_hms_events_incremental_sync; fe-core parses nothing), null when disabled/uninitialized. invalidatePartition(): degrades to a table-level flush (name-list/location-keyed caches can't target one name; correctness-safe via pull-through) + sibling forward. - HiveConnectorProperties: the two per-catalog keys + getBoolean helper. Dormant: hms not in SPI_READY_TYPES, no PluginDriven hms catalog polls yet (E-e adds the fe-core driver). Merge/batch deliberately omitted (a batch-size optimization; order-preserving apply over pull-through caches is correct without it). fe-connector-hive/-hms compile; checkstyle 0; import gate ok. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../api/event/MetastoreChangeDescriptor.java | 12 + .../doris/connector/hive/HiveConnector.java | 48 ++++ .../hive/HiveConnectorProperties.java | 23 ++ .../connector/hms/event/HmsEventParser.java | 242 ++++++++++++++++++ .../connector/hms/event/HmsEventSource.java | 134 ++++++++++ 5 files changed, 459 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java index 1910dd25aaecb4..a491c1fe58cd46 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/MetastoreChangeDescriptor.java @@ -114,6 +114,18 @@ public static MetastoreChangeDescriptor forTable(Op op, String dbName, String ta updateTime, eventId); } + /** + * A {@link Op#RENAME_TABLE} change (also used for view-recreate, where the after-db/table equal the + * before-db/table). Carries both the before ({@code dbName}/{@code tableName}) and after + * ({@code dbNameAfter}/{@code tableNameAfter}) identities, since a Hive table rename may move the + * table across databases. + */ + public static MetastoreChangeDescriptor forTableRename(String dbName, String tableName, + String dbNameAfter, String tableNameAfter, long eventId, long updateTime) { + return new MetastoreChangeDescriptor(Op.RENAME_TABLE, dbName, tableName, dbNameAfter, + tableNameAfter, null, updateTime, eventId); + } + /** A partition-level change ({@code ADD_/DROP_/REFRESH_PARTITIONS}). */ public static MetastoreChangeDescriptor forPartitions(Op op, String dbName, String tableName, List partitionNames, long eventId, long updateTime) { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index ad5753fd181682..30e80405dd94ff 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; +import org.apache.doris.connector.api.event.ConnectorEventSource; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.procedure.ConnectorProcedureOps; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; @@ -30,6 +31,7 @@ import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.ThriftHmsClient; +import org.apache.doris.connector.hms.event.HmsEventSource; import org.apache.doris.connector.metastore.HmsMetaStoreProperties; import org.apache.doris.connector.metastore.spi.MetaStoreProviders; import org.apache.doris.connector.spi.ConnectorContext; @@ -44,6 +46,7 @@ import java.io.IOException; import java.util.Collections; import java.util.EnumSet; +import java.util.List; import java.util.Map; import java.util.Optional; import java.util.Set; @@ -346,6 +349,51 @@ void invalidateAll(HmsClient client) { fileListingCache.invalidateAll(); } + /** + * Invalidates a table's caches on a partition add/drop/alter. The connector's partition caches are + * keyed by name-LIST ({@link CachingHmsClient}) and directory LOCATION ({@link HiveFileListingCache}), + * neither of which can target a single partition name — so this degrades to a table-level flush, which + * is correctness-safe because both caches re-list on the next miss. The names are still carried on the + * SPI for future precision. Also forwarded to the already-built embedded siblings. + */ + @Override + public void invalidatePartition(String dbName, String tableName, List partitionNames) { + invalidatePartition(hmsClient, dbName, tableName); + forEachBuiltSibling(sibling -> sibling.invalidatePartition(dbName, tableName, partitionNames)); + } + + // Package-private seam (mirrors invalidateTable): the metastore half needs an observable CachingHmsClient. + void invalidatePartition(HmsClient client, String dbName, String tableName) { + if (client instanceof CachingHmsClient) { + ((CachingHmsClient) client).flush(dbName, tableName); + } + fileListingCache.invalidateTable(dbName, tableName); + } + + /** + * The catalog's incremental metadata-change source, or {@code null} when this catalog does not opt into + * HMS notification-event sync. The per-catalog opt-in is the connector's concern (fe-core does not parse + * hive properties): a source is returned only when the + * {@code hive.enable_hms_events_incremental_sync} property is set, and the engine's role-aware event + * driver skips connectors whose source is null. A malformed / not-yet-initialized property reads as + * disabled, mirroring the legacy poller's skip-on-throw. + */ + @Override + public ConnectorEventSource getEventSource() { + try { + if (!HiveConnectorProperties.getBoolean(properties, + HiveConnectorProperties.ENABLE_HMS_EVENTS_INCREMENTAL_SYNC, false)) { + return null; + } + } catch (RuntimeException e) { + return null; + } + int batchSize = HiveConnectorProperties.getInt(properties, + HiveConnectorProperties.HMS_EVENTS_BATCH_SIZE_PER_RPC, + HiveConnectorProperties.DEFAULT_HMS_EVENTS_BATCH_SIZE); + return new HmsEventSource(getOrCreateClient(), batchSize); + } + /** * Runs one invalidation action on each ALREADY-BUILT embedded sibling (iceberg, hudi). fe-core routes * {@code REFRESH TABLE/DATABASE/CATALOG} only to a catalog's PRIMARY connector, so the gateway must diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java index 0ab3b22e91f002..07c47a7c90c4ab 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java @@ -93,6 +93,18 @@ private HiveConnectorProperties() { */ public static final String BUCKET_ALGO_RANDOM = "doris_random"; + // ===== Metastore incremental event sync (per-catalog opt-in) ===== + + /** Whether this catalog polls HMS notification events for incremental metadata refresh. */ + public static final String ENABLE_HMS_EVENTS_INCREMENTAL_SYNC = + "hive.enable_hms_events_incremental_sync"; + + /** Max notification events fetched per RPC when incremental event sync is enabled. */ + public static final String HMS_EVENTS_BATCH_SIZE_PER_RPC = "hive.hms_events_batch_size_per_rpc"; + + /** Default batch size, matching the engine's legacy {@code hms_events_batch_size_per_rpc} default. */ + public static final int DEFAULT_HMS_EVENTS_BATCH_SIZE = 500; + /** * Parse an integer property with a default value. */ @@ -107,4 +119,15 @@ public static int getInt(Map props, String key, int defaultVal) return defaultVal; } } + + /** + * Parse a boolean property with a default value. + */ + public static boolean getBoolean(Map props, String key, boolean defaultVal) { + String value = props.get(key); + if (value == null || value.isEmpty()) { + return defaultVal; + } + return Boolean.parseBoolean(value.trim()); + } } diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java new file mode 100644 index 00000000000000..31224920f9bcf1 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java @@ -0,0 +1,242 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms.event; + +import org.apache.doris.connector.api.event.MetastoreChangeDescriptor; +import org.apache.doris.connector.api.event.MetastoreChangeDescriptor.Op; +import org.apache.doris.connector.hms.HmsNotificationEvent; + +import org.apache.commons.io.IOUtils; +import org.apache.hadoop.hive.common.FileUtils; +import org.apache.hadoop.hive.metastore.api.Database; +import org.apache.hadoop.hive.metastore.api.FieldSchema; +import org.apache.hadoop.hive.metastore.api.Partition; +import org.apache.hadoop.hive.metastore.api.Table; +import org.apache.hadoop.hive.metastore.messaging.AddPartitionMessage; +import org.apache.hadoop.hive.metastore.messaging.AlterPartitionMessage; +import org.apache.hadoop.hive.metastore.messaging.CreateTableMessage; +import org.apache.hadoop.hive.metastore.messaging.DropPartitionMessage; +import org.apache.hadoop.hive.metastore.messaging.json.JSONAlterDatabaseMessage; +import org.apache.hadoop.hive.metastore.messaging.json.JSONAlterTableMessage; +import org.apache.hadoop.hive.metastore.messaging.json.JSONDropTableMessage; +import org.apache.hadoop.hive.metastore.messaging.json.JSONMessageDeserializer; + +import java.io.ByteArrayInputStream; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.Base64; +import java.util.Collections; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.stream.Collectors; +import java.util.zip.GZIPInputStream; + +/** + * Parses one HMS {@link HmsNotificationEvent} into connector-neutral {@link MetastoreChangeDescriptor}s. + * + *

    This is the plugin-side half of the metastore-event relocation: it replaces the fe-core + * {@code datasource.hive.event.*} classes' {@code process()} bodies, but instead of mutating the + * engine's object graph it emits neutral descriptors the engine applies. It preserves the legacy + * semantics faithfully (table-name lowercasing, rename vs view-recreate, alter-partition rename = + * drop+add, canonical partition names), so a flipped catalog behaves as the legacy poller did.

    + * + *

    The GZIP message format (some HDP/CDH Hive versions) base64+gzip-wraps the JSON payload; we + * decompress it before handing the plain {@link JSONMessageDeserializer} the body, avoiding a bespoke + * deserializer subclass.

    + */ +public final class HmsEventParser { + + private static final JSONMessageDeserializer JSON = new JSONMessageDeserializer(); + private static final String GZIP_FORMAT_PREFIX = "gzip"; + + private HmsEventParser() { + } + + /** + * Map one notification event to zero or more descriptors (empty for unsupported / no-op events, + * e.g. a properties-only ALTER DATABASE or a transaction event). + */ + public static List parse(HmsNotificationEvent event) { + try { + return doParse(event); + } catch (Exception e) { + throw new RuntimeException("failed to parse HMS notification event " + event, e); + } + } + + private static List doParse(HmsNotificationEvent event) throws Exception { + String type = event.getEventType(); + if (type == null) { + return Collections.emptyList(); + } + String dbName = lower(event.getDbName()); + String tblName = event.getTableName(); + long eventId = event.getEventId(); + // Hive records event time in seconds; the engine tracks freshness in millis. + long updateTime = event.getEventTime() * 1000L; + String body = prepareBody(event.getMessageFormat(), event.getMessage()); + + switch (type) { + case "CREATE_TABLE": { + CreateTableMessage message = JSON.getCreateTableMessage(body); + String table = message.getTableObj().getTableName().toLowerCase(Locale.ROOT); + return one(MetastoreChangeDescriptor.forTable( + Op.REGISTER_TABLE, dbName, table, null, eventId, updateTime)); + } + case "DROP_TABLE": { + JSONDropTableMessage message = (JSONDropTableMessage) JSON.getDropTableMessage(body); + return one(MetastoreChangeDescriptor.forTable( + Op.UNREGISTER_TABLE, dbName, message.getTable(), null, eventId, updateTime)); + } + case "ALTER_TABLE": { + JSONAlterTableMessage message = (JSONAlterTableMessage) JSON.getAlterTableMessage(body); + Table after = message.getTableObjAfter(); + Table before = message.getTableObjBefore(); + String afterTable = after.getTableName().toLowerCase(Locale.ROOT); + boolean isRename = !before.getDbName().equalsIgnoreCase(after.getDbName()) + || !before.getTableName().equalsIgnoreCase(afterTable); + boolean isView = before.isSetViewExpandedText() || before.isSetViewOriginalText(); + if (isRename || isView) { + // rename (possibly across dbs) or view-recreate (same name) => unregister+register + return one(MetastoreChangeDescriptor.forTableRename( + before.getDbName(), before.getTableName(), + after.getDbName(), afterTable, eventId, updateTime)); + } + return one(MetastoreChangeDescriptor.forTable( + Op.REFRESH_TABLE, before.getDbName(), before.getTableName(), null, + eventId, updateTime)); + } + case "CREATE_DATABASE": + return one(MetastoreChangeDescriptor.forDatabase( + Op.REGISTER_DATABASE, dbName, null, eventId, updateTime)); + case "DROP_DATABASE": + return one(MetastoreChangeDescriptor.forDatabase( + Op.UNREGISTER_DATABASE, dbName, null, eventId, updateTime)); + case "ALTER_DATABASE": { + JSONAlterDatabaseMessage message = + (JSONAlterDatabaseMessage) JSON.getAlterDatabaseMessage(body); + Database before = message.getDbObjBefore(); + Database after = message.getDbObjAfter(); + if (before.getName().equalsIgnoreCase(after.getName())) { + // properties-only change: nothing the engine must react to + return Collections.emptyList(); + } + return one(MetastoreChangeDescriptor.forDatabase( + Op.RENAME_DATABASE, before.getName(), after.getName(), eventId, updateTime)); + } + case "ADD_PARTITION": { + AddPartitionMessage message = JSON.getAddPartitionMessage(body); + Table table = message.getTableObj(); + List names = new ArrayList<>(); + List colNames = partitionColNames(table); + for (Partition partition : message.getPartitionObjs()) { + names.add(FileUtils.makePartName(colNames, partition.getValues())); + } + return one(MetastoreChangeDescriptor.forPartitions( + Op.ADD_PARTITIONS, dbName, table.getTableName(), names, eventId, updateTime)); + } + case "DROP_PARTITION": { + DropPartitionMessage message = JSON.getDropPartitionMessage(body); + Table table = message.getTableObj(); + List names = new ArrayList<>(); + List colNames = partitionColNames(table); + for (Map partition : message.getPartitions()) { + names.add(rawPartName(partition, colNames)); + } + return one(MetastoreChangeDescriptor.forPartitions( + Op.DROP_PARTITIONS, dbName, table.getTableName(), names, eventId, updateTime)); + } + case "ALTER_PARTITION": { + AlterPartitionMessage message = JSON.getAlterPartitionMessage(body); + Table table = message.getTableObj(); + List colNames = partitionColNames(table); + String before = FileUtils.makePartName(colNames, message.getPtnObjBefore().getValues()); + String after = FileUtils.makePartName(colNames, message.getPtnObjAfter().getValues()); + if (!before.equalsIgnoreCase(after)) { + // partition rename = drop old + add new (on the event's db/table) + List out = new ArrayList<>(2); + out.add(MetastoreChangeDescriptor.forPartitions( + Op.DROP_PARTITIONS, dbName, tblName, Collections.singletonList(before), + eventId, updateTime)); + out.add(MetastoreChangeDescriptor.forPartitions( + Op.ADD_PARTITIONS, dbName, tblName, Collections.singletonList(after), + eventId, updateTime)); + return out; + } + return one(MetastoreChangeDescriptor.forPartitions( + Op.REFRESH_PARTITIONS, dbName, table.getTableName(), + Collections.singletonList(after), eventId, updateTime)); + } + case "INSERT": + // non-partitioned insert: no partition event fires, just drop the table's caches + return one(MetastoreChangeDescriptor.forTable( + Op.REFRESH_TABLE, dbName, tblName, null, eventId, updateTime)); + default: + // ALTER_PARTITIONS / INSERT_PARTITIONS / ALLOC_WRITE_ID / COMMIT_TXN / ... => ignored + return Collections.emptyList(); + } + } + + private static List partitionColNames(Table table) { + return table.getPartitionKeys().stream().map(FieldSchema::getName).collect(Collectors.toList()); + } + + // Raw "col=val/col2=val2" name (mirrors the legacy DropPartition path, which does no escaping). + private static String rawPartName(Map part, List colNames) { + if (part.isEmpty() || colNames.size() != part.size()) { + return ""; + } + StringBuilder name = new StringBuilder(); + int i = 0; + for (String colName : colNames) { + if (i++ > 0) { + name.append('/'); + } + name.append(colName).append('=').append(part.get(colName)); + } + return name.toString(); + } + + private static String lower(String s) { + return (s == null || s.isEmpty()) ? s : s.toLowerCase(Locale.ROOT); + } + + private static List one(MetastoreChangeDescriptor descriptor) { + return Collections.singletonList(descriptor); + } + + private static String prepareBody(String format, String message) { + if (format != null && format.startsWith(GZIP_FORMAT_PREFIX)) { + return deCompress(message); + } + return message; + } + + private static String deCompress(String messageBody) { + try { + byte[] decodedBytes = Base64.getDecoder().decode(messageBody.getBytes(StandardCharsets.UTF_8)); + try (ByteArrayInputStream in = new ByteArrayInputStream(decodedBytes); + GZIPInputStream is = new GZIPInputStream(in)) { + return new String(IOUtils.toByteArray(is), StandardCharsets.UTF_8); + } + } catch (Exception e) { + throw new RuntimeException("cannot decode the gzip notification message", e); + } + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java new file mode 100644 index 00000000000000..74ba211b1da843 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java @@ -0,0 +1,134 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms.event; + +import org.apache.doris.connector.api.event.ConnectorEventSource; +import org.apache.doris.connector.api.event.EventPollRequest; +import org.apache.doris.connector.api.event.EventPollResult; +import org.apache.doris.connector.api.event.MetastoreChangeDescriptor; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsClientException; +import org.apache.doris.connector.hms.HmsNotificationEvent; + +import org.apache.hadoop.hive.metastore.HiveMetaStoreClient; + +import java.util.ArrayList; +import java.util.List; + +/** + * The hive connector's {@link ConnectorEventSource}: fetches HMS notification events and parses them + * into neutral descriptors. Stateless with respect to the cursor — the engine passes the cursor in and + * stores the returned one, so the same source instance serves both master (fetch-directly) and follower + * (bounded-by-master) roles. + * + *

    Mirrors the legacy fe-core {@code MetastoreEventsProcessor} fetch decisions exactly: the master + * reads the metastore's current id to decide whether to pull; a follower pulls only up to the master's + * committed high-water mark; a first sync or a trimmed notification log (the + * {@code REPL_EVENTS_MISSING_IN_METASTORE} sentinel) yields a full-refresh signal instead of events.

    + */ +public class HmsEventSource implements ConnectorEventSource { + + private final HmsClient client; + private final int batchSize; + + public HmsEventSource(HmsClient client, int batchSize) { + this.client = client; + this.batchSize = batchSize; + } + + @Override + public long getCurrentEventId() { + return client.getCurrentNotificationEventId(); + } + + @Override + public EventPollResult pollOnce(EventPollRequest request) { + if (request.isMaster()) { + return pollForMaster(request.getLastSyncedEventId()); + } + return pollForFollower(request.getLastSyncedEventId(), request.getMasterUpperBound()); + } + + private EventPollResult pollForMaster(long lastSyncedEventId) { + long currentEventId = client.getCurrentNotificationEventId(); + if (lastSyncedEventId < 0) { + // first pull: seed the cursor to now and rebuild via a full refresh (no events to replay) + return EventPollResult.ofFullRefresh(currentEventId); + } + if (currentEventId == lastSyncedEventId) { + return EventPollResult.ofNothing(lastSyncedEventId); + } + try { + return toResult(client.getNextNotification(lastSyncedEventId, batchSize), lastSyncedEventId); + } catch (HmsClientException e) { + if (isEventsMissing(e)) { + // the metastore trimmed its log past our cursor; jump to now and full-refresh + return EventPollResult.ofFullRefresh(currentEventId); + } + throw e; + } + } + + private EventPollResult pollForFollower(long lastSyncedEventId, long masterUpperBound) { + // -1 => the master's cursor has not been learned yet (via edit-log replay); do nothing + if (masterUpperBound == -1L || lastSyncedEventId == masterUpperBound) { + return EventPollResult.ofNothing(lastSyncedEventId); + } + if (lastSyncedEventId < 0) { + // first pull: seed to the master's committed id and full-refresh (the engine forwards + // REFRESH CATALOG to the master for a follower) + return EventPollResult.ofFullRefresh(masterUpperBound); + } + // never read past what the master has already committed and replicated + int maxEvents = (int) Math.min(masterUpperBound - lastSyncedEventId, batchSize); + try { + return toResult(client.getNextNotification(lastSyncedEventId, maxEvents), lastSyncedEventId); + } catch (HmsClientException e) { + if (isEventsMissing(e)) { + return EventPollResult.ofFullRefresh(masterUpperBound); + } + throw e; + } + } + + private EventPollResult toResult(List events, long lastSyncedEventId) { + if (events.isEmpty()) { + return EventPollResult.ofNothing(lastSyncedEventId); + } + List descriptors = new ArrayList<>(); + for (HmsNotificationEvent event : events) { + descriptors.addAll(HmsEventParser.parse(event)); + } + long newCursor = events.get(events.size() - 1).getEventId(); + return EventPollResult.ofChanges(newCursor, descriptors); + } + + private static boolean isEventsMissing(HmsClientException e) { + // ThriftHmsClient.execute wraps the vendored client's + // IllegalStateException(REPL_EVENTS_MISSING_IN_METASTORE) into an HmsClientException; the + // sentinel survives in the message chain. + for (Throwable t = e; t != null; t = t.getCause()) { + String message = t.getMessage(); + if (message != null + && message.contains(HiveMetaStoreClient.REPL_EVENTS_MISSING_IN_METASTORE)) { + return true; + } + } + return false; + } +} From cedee1d73cd97b683144f992e0aa4dae5484ee37 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 12:00:24 +0800 Subject: [PATCH 168/334] [doc](catalog) HANDOFF: Hive event pipeline E-a/E-b/E-c landed (dormant); E-d/E-e/E-f + review remain MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records this round's progress on Phase 1 item 2 (event-pipeline Model B): the neutral event SPI (E-a), plugin notification fetch (E-b), and plugin event source + HiveConnector wiring (E-c) landed dormant and verified. fe-core mutator generalization (E-d), the connector-agnostic driver (E-e), dormant tests (E-f) + clean-room review remain — precise remaining edits listed. The 3 pending D2 review decisions are preserved and noted as decoupled from the event pipeline (they refine the D2 cache; the event pipeline only builds on the already-landed D2 invalidate SPI). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index f4db99c17f2638..df5fb59eccbe00 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -24,7 +24,16 @@ D2 = `f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` - ❌ **未修-待拍板 #3**（review2 §3，**线上活性 bug 非休眠**）：paimon/iceberg `latestSnapshotCache`（+`PaimonSchemaAtMemo` 窄化变体）今天就有同形状的删建重名洞（fe-core DDL 路径从不 invalidate、key=纯表名）。修=独立线（动 live 行为，须自带复审+回归），或记档延后。 - 测试加固欠账（LOW，代码正确仅测试盲区）：`createClient` 真包裹无 pin、公开 invalidate 钩子未用已构建 client 驱动（review2 §2.4）。 -**⭐ 下一步 = 先让用户拍板上面 3 项 → 落对应修复 → 再进 Phase 1 第二项「事件管道 Model B」**（薄 fe-core 角色驱动 + 插件 `pollOnce` SPI，翻闸后重新武装 `MetastoreEventsProcessor` 的增量失效，权威 = `tasks/hms-event-pipeline-findings-2026-07-07.md`）。它**依赖** D2 缓存（在其之上加 `Connector.invalidatePartitions(db,table,names)`）。起步先读 findings + 对照 HEAD。**⚠ 并行 session 风险**：本轮曾有两个 session 同时在此工作树工作（一个 amend 卷入另一个的未提交测试）；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 +**⭐ 状态（本轮，2026-07-10 晚）= Phase 1 第二项「事件管道 Model B」的 SPI + 插件侧已落地（E-a/E-b/E-c，休眠、独立可复审）；fe-core 侧（E-d/E-e/E-f）+ 净室复审待续。上面 3 项 D2 复审决策仍待用户拍板——与事件管道解耦（那 3 项是 D2 缓存精化：DDL 路径失效 / 分区缓存容量单位 / paimon-iceberg live 删建洞；事件管道只依赖已落地的 D2 invalidate SPI），可并行处理。** +- 权威设计 = **`tasks/hive-event-pipeline-step-design-2026-07-10.md`**（**起步必读**；含用户签字 A=结构性事件即时重建平价 + 6 步序列 E-a…E-f + 侦察 `wf_0c686fb4-d9d` 6 维+critic）。**SUPERSEDES** `hms-event-pipeline-findings-2026-07-07.md`（那份是旧侦察，行号已漂；设计稿是当前权威）。 +- 已落地（全休眠：hms 未 SPI_READY；paimon/iceberg/jdbc/hudi 字节不变；各步 test-compile BUILD SUCCESS + checkstyle 0 + import gate ok）：`0214f0405c1` **E-a** 中立事件 SPI（`MetastoreChangeDescriptor`+op / `ConnectorEventSource` / `EventPollRequest|Result` / `Connector.getEventSource()` default null / `Connector.invalidatePartition` default no-op）→ `b13ed79e32d` **E-b** 插件取数（`HmsClient.getCurrentNotificationEventId/getNextNotification` default 抛 + `ThriftHmsClient` 委派 vendored client + `CachingHmsClient` 透传 + `HmsNotificationEvent` DTO）→ `902546d79c5` **E-c** 插件事件源（`HmsEventParser` 忠实映射事件→中立描述符 + `HmsEventSource` 角色感知 fetch + `HiveConnector.getEventSource/invalidatePartition` + 2 个 per-catalog 属性键）。 +- **剩余（权威=设计稿 §2.3/§3，行号信 HEAD 不信文档）**： + - **E-d fe-core mutator 泛化（休眠、向后兼容 legacy）**：`CatalogMgr.registerExternalTableFromEvent` 去两处 HMS cast（`buildTableForInit` 通用；⚠ 注意 `ExternalDatabase.buildTableForInit`+`registerTable` 泛型捕获，先读签名，可能用 raw 或 typed helper）；`registerDatabase` 上提 `PluginDrivenExternalCatalog`（通用体 `buildDbForInit`+`metaCache.updateCache`+`genIdByName`，见 `HMSExternalCatalog:205`）；`unregisterExternalDatabase` 去多余 cast；`addExternalPartitions/dropExternalPartitions`+`RefreshManager.refreshPartitions`+`replayRefreshTable 分区分支` 加 `if(PluginDrivenExternalCatalog) getConnector().invalidatePartition(...)` 分支（与现有 `instanceof HMSExternalTable` 分支互斥，pre-flip 走 HMS 分支不变）。 + - **E-e fe-core 连接器无关驱动**：新 `MasterDaemon`（**存活包** `datasource/`，非将删的 `hive/event/`），能力探测 `PluginDrivenExternalCatalog + getConnector().getEventSource()!=null`；两 cursor map；`Env.isMaster()` 分角色；`onPluginClassLoader` pin 包 `pollOnce`；应用描述符（泛化 mutator + `invalidatePartition`）；master 从描述符建 `MetaIdMappingsLog`（op→ADD/DELETE×objType×names，**修 drop-partition 误标 DATABASE→PARTITION**）+ 写编辑日志；`needsFullRefresh` → master `replayRefreshCatalog` / follower 转发 `REFRESH CATALOG`。构造+启动休眠（Env 内，pre-flip 无匹配目录）。 + - **E-f 休眠单测** + **净室对抗复审**（多 agent，先独立判断后交叉核对历史结论）。 +- **翻闸时接线（Phase 2，非本步）**：去 `MetastoreEventsProcessor:116` gate；`ExternalMetaIdMgr.replayMetaIdMappingsLog` 游标回放改指新驱动。 +- **⚠ 遗留（设计稿已标注）**：连接器 enable 开关默认 false；若部署靠 fe-core 全局 Config `enable_hms_events_incremental_sync=true`（无 per-catalog 属性）开启增量同步，翻闸后须补 per-catalog `hive.enable_hms_events_incremental_sync` 属性（flip/e2e 迁移项）。合并/去重（mergeEvents）本步有意省略（批大小优化；拉取式缓存+顺序应用即正确）。 +- **⚠ 并行 session 风险**：曾有两个 session 同时在此工作树（一个 amend 卷入另一个未提交测试；本轮 HANDOFF 也被并行 session 更新过）；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 **⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步（读 / 写-拒 / schema-evolution / 缓存 / 跨加载器委派）只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`）。**删除排序最硬约束**：`datasource/hive|hudi|iceberg/` 的 ~90 个 HMS 支撑类删不掉，直到翻闸把消费者切到连接器路径（详见 execution plan §2.4/§3 + `tasks/iceberg-on-hms-delegation-findings-2026-07-07.md`）。 From b524e81fd7983ac2b5ecf5756af4803d4fcf738f Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 12:14:17 +0800 Subject: [PATCH 169/334] [feat](catalog) Hive event pipeline: generalize fe-core event mutators (dormant) E-d of the metastore-event relocation. Makes the fe-core mutators the event apply-path calls work on a flipped (PluginDriven) catalog, while staying byte-identical for a legacy HMS catalog (the live pre-flip path). These are the only callers of these mutators, so the change is surgical. - registerExternalTableFromEvent: drop the (HMSExternalCatalog)/(HMSExternalDatabase) casts; buildTableForInit is generic on ExternalDatabase and dispatches to the catalog's own table type, so it builds+registers the right table on both. - PluginDrivenExternalCatalog.registerDatabase: pulled up from HMSExternalCatalog (generic body: buildDbForInit + metaCache + name-derived id) so a create/rename- database event no longer throws NotImplementedException on a flipped catalog. registerExternalDatabaseFromEvent / unregisterExternalDatabase now call it via ExternalCatalog (the base still throws => fail-loud for catalogs that can't). - addExternalPartitions / dropExternalPartitions / RefreshManager.refreshPartitions: add a guarded `else if (catalog instanceof PluginDrivenExternalCatalog)` branch routing to the new connector.invalidatePartition (name-keyed; pull-through cache re-lists), mutually exclusive with the existing HMSExternalTable branch that stays unchanged for legacy. Drop the gratuitous (HMSExternalTable) setUpdateTime cast. replayRefreshTable's partition branch needs no change: post-flip its instanceof-HMSExternalCatalog guard is false => it falls to the full refreshTableInternal, whose D2 connector.invalidateTable hook already flushes the whole table's connector cache (== the table-level degrade invalidatePartition does), so it is correctness-safe with no CCE. Dormant: the PluginDriven branches are unreachable until the flip. fe-core compiles (-am); checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../apache/doris/catalog/RefreshManager.java | 16 +++-- .../apache/doris/datasource/CatalogMgr.java | 72 +++++++++++-------- .../PluginDrivenExternalCatalog.java | 17 +++++ 3 files changed, 72 insertions(+), 33 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java index c2fe59a7ae6352..bec60a78c918fb 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java @@ -290,11 +290,19 @@ public void refreshPartitions(String catalogName, String dbName, String tableNam } ExternalTable externalTable = (ExternalTable) table; - HiveExternalMetaCache cache = Env.getCurrentEnv().getExtMetaCacheMgr().hive(externalTable.getCatalog().getId()); - for (String partitionName : partitionNames) { - cache.invalidatePartitionCache(externalTable, partitionName); + if (externalTable.getCatalog() instanceof PluginDrivenExternalCatalog) { + // Flipped: the connector owns the partition cache (pull-through); invalidate by name. The fe-core + // hive cache below is retired for a flipped catalog. Mirrors refreshTableInternal's connector hook. + ((PluginDrivenExternalCatalog) externalTable.getCatalog()).getConnector().invalidatePartition( + ((ExternalDatabase) db).getRemoteName(), externalTable.getRemoteName(), partitionNames); + } else { + HiveExternalMetaCache cache = + Env.getCurrentEnv().getExtMetaCacheMgr().hive(externalTable.getCatalog().getId()); + for (String partitionName : partitionNames) { + cache.invalidatePartitionCache(externalTable, partitionName); + } } - ((HMSExternalTable) table).setUpdateTime(updateTime); + externalTable.setUpdateTime(updateTime); } public void addToRefreshMap(long catalogId, Integer[] sec) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java index 83e778ac2f0ced..013bc9fa5f1223 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogMgr.java @@ -38,8 +38,6 @@ import org.apache.doris.common.util.DatasourcePrintableMap; import org.apache.doris.common.util.TimeUtils; import org.apache.doris.common.util.Util; -import org.apache.doris.datasource.hive.HMSExternalCatalog; -import org.apache.doris.datasource.hive.HMSExternalDatabase; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.hive.HiveExternalMetaCache; import org.apache.doris.datasource.mvcc.MvccUtil; @@ -738,9 +736,7 @@ public void registerExternalTableFromEvent(String dbName, String tableName, return; } - long tblId; - HMSExternalCatalog hmsCatalog = (HMSExternalCatalog) catalog; - tblId = Util.genIdByName(catalogName, dbName, tableName); + long tblId = Util.genIdByName(catalogName, dbName, tableName); // -1L means it will be dropped later, ignore if (tblId == ExternalMetaIdMgr.META_ID_FOR_NOT_EXISTS) { return; @@ -748,8 +744,12 @@ public void registerExternalTableFromEvent(String dbName, String tableName, db.writeLock(); try { - HMSExternalTable namedTable = ((HMSExternalDatabase) db) - .buildTableForInit(tableName, tableName, tblId, hmsCatalog, (HMSExternalDatabase) db, false); + // buildTableForInit is generic on ExternalDatabase and dispatches to the catalog's own table + // type (HMSExternalTable for a legacy catalog, PluginDrivenMvccExternalTable/PluginDrivenExternalTable + // for a flipped one), so the event path builds+registers the right table on both without an HMS cast. + ExternalDatabase externalDb = (ExternalDatabase) db; + ExternalTable namedTable = externalDb.buildTableForInit(tableName, tableName, tblId, + (ExternalCatalog) catalog, externalDb, false); namedTable.setUpdateTime(updateTime); db.registerTable(namedTable); } finally { @@ -766,7 +766,7 @@ public void unregisterExternalDatabase(String dbName, String catalogName) if (!(catalog instanceof ExternalCatalog)) { throw new DdlException("Only support drop ExternalCatalog databases"); } - ((HMSExternalCatalog) catalog).unregisterDatabase(dbName); + ((ExternalCatalog) catalog).unregisterDatabase(dbName); } public void registerExternalDatabaseFromEvent(String dbName, String catalogName) @@ -779,14 +779,15 @@ public void registerExternalDatabaseFromEvent(String dbName, String catalogName) throw new DdlException("Only support create ExternalCatalog databases"); } - HMSExternalCatalog hmsCatalog = (HMSExternalCatalog) catalog; long dbId = Util.genIdByName(catalogName, dbName); // -1L means it will be dropped later, ignore if (dbId == ExternalMetaIdMgr.META_ID_FOR_NOT_EXISTS) { return; } - hmsCatalog.registerDatabase(dbId, dbName); + // registerDatabase is overridden by HMSExternalCatalog (legacy) and PluginDrivenExternalCatalog + // (flipped); the generic ExternalCatalog base throws (fail-loud for catalogs that cannot register). + ((ExternalCatalog) catalog).registerDatabase(dbId, dbName); } public void addExternalPartitions(String catalogName, String dbName, String tableName, @@ -814,22 +815,27 @@ public void addExternalPartitions(String catalogName, String dbName, String tabl } return; } - if (!(table instanceof HMSExternalTable)) { - LOG.warn("only support HMSTable"); - return; - } - - HMSExternalTable hmsTable = (HMSExternalTable) table; - List partitionColumnTypes; - try { - partitionColumnTypes = hmsTable.getPartitionColumnTypes(MvccUtil.getSnapshotFromContext(hmsTable)); - } catch (NotSupportedException e) { - LOG.warn("Ignore not supported hms table, message: {} ", e.getMessage()); - return; + if (table instanceof HMSExternalTable) { + HMSExternalTable hmsTable = (HMSExternalTable) table; + List partitionColumnTypes; + try { + partitionColumnTypes = hmsTable.getPartitionColumnTypes(MvccUtil.getSnapshotFromContext(hmsTable)); + } catch (NotSupportedException e) { + LOG.warn("Ignore not supported hms table, message: {} ", e.getMessage()); + return; + } + HiveExternalMetaCache cache = Env.getCurrentEnv().getExtMetaCacheMgr().hive(catalog.getId()); + cache.addPartitionsCache(hmsTable.getOrBuildNameMapping(), partitionNames, partitionColumnTypes); + hmsTable.setUpdateTime(updateTime); + } else if (catalog instanceof PluginDrivenExternalCatalog) { + // Flipped: the connector owns the partition cache (pull-through), so invalidating by name is + // enough for the added partitions to show up on the next listing. Mirrors refreshTableInternal's + // connector hook; the fe-core hive cache above is retired for a flipped catalog. + ((PluginDrivenExternalCatalog) catalog).getConnector().invalidatePartition( + ((ExternalDatabase) db).getRemoteName(), ((ExternalTable) table).getRemoteName(), + partitionNames); + ((ExternalTable) table).setUpdateTime(updateTime); } - HiveExternalMetaCache cache = Env.getCurrentEnv().getExtMetaCacheMgr().hive(catalog.getId()); - cache.addPartitionsCache(hmsTable.getOrBuildNameMapping(), partitionNames, partitionColumnTypes); - hmsTable.setUpdateTime(updateTime); } public void dropExternalPartitions(String catalogName, String dbName, String tableName, @@ -858,10 +864,18 @@ public void dropExternalPartitions(String catalogName, String dbName, String tab return; } - HMSExternalTable hmsTable = (HMSExternalTable) table; - Env.getCurrentEnv().getExtMetaCacheMgr().hive(catalog.getId()) - .dropPartitionsCache(hmsTable, partitionNames, true); - hmsTable.setUpdateTime(updateTime); + if (table instanceof HMSExternalTable) { + HMSExternalTable hmsTable = (HMSExternalTable) table; + Env.getCurrentEnv().getExtMetaCacheMgr().hive(catalog.getId()) + .dropPartitionsCache(hmsTable, partitionNames, true); + hmsTable.setUpdateTime(updateTime); + } else if (catalog instanceof PluginDrivenExternalCatalog) { + // Flipped: the connector owns the partition cache (pull-through); invalidate by name. + ((PluginDrivenExternalCatalog) catalog).getConnector().invalidatePartition( + ((ExternalDatabase) db).getRemoteName(), ((ExternalTable) table).getRemoteName(), + partitionNames); + ((ExternalTable) table).setUpdateTime(updateTime); + } } public void registerCatalogRefreshListener(Env env) { diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index c57ee69da972cb..cf41447ab0cc8b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -31,6 +31,7 @@ import org.apache.doris.common.ErrorCode; import org.apache.doris.common.ErrorReport; import org.apache.doris.common.UserException; +import org.apache.doris.common.util.Util; import org.apache.doris.connector.ConnectorFactory; import org.apache.doris.connector.ConnectorSessionBuilder; import org.apache.doris.connector.DefaultConnectorContext; @@ -317,6 +318,22 @@ public Connector getConnector() { return connector; } + /** + * Registers a newly-observed database into this catalog, driven by the metastore-event sync's + * REGISTER_DATABASE change (via {@code CatalogMgr.registerExternalDatabaseFromEvent}). Pulled up from + * {@code HMSExternalCatalog} so a flipped (generic) catalog no longer throws + * {@code NotImplementedException} on a create/rename-database event. The body is fully generic + * (buildDbForInit + metaCache, name-derived id) and mirrors the legacy HMS implementation. + */ + @Override + public void registerDatabase(long dbId, String dbName) { + ExternalDatabase db = buildDbForInit(dbName, null, dbId, logType, false); + if (isInitialized()) { + metaCache.updateCache(db.getRemoteName(), db.getFullName(), db, + Util.genIdByName(name, db.getFullName())); + } + } + /** * FIX-4: let the connector's own cache knob also govern the schema cache (restoring the legacy single-knob * semantics — e.g. paimon's {@code meta.cache.paimon.table.ttl-second} sized the whole table cache, schema From 6b8f1181202edf84e11c6583711e1282c20af4c7 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 12:28:42 +0800 Subject: [PATCH 170/334] [feat](catalog) Hive event pipeline: connector-agnostic fe-core event driver (dormant) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit E-e of the metastore-event relocation — the fe-core half. A new connector-agnostic, role-aware MasterDaemon (in the surviving datasource package, not the doomed hive/event dir) that replaces the legacy MetastoreEventsProcessor's source-specific machinery: - Iterates catalogs and probes each for an event source via capability (PluginDrivenExternalCatalog + getConnector().getEventSource() != null), never instanceof HMSExternalCatalog. Skips non-initialized catalogs so an idle non-hive plugin catalog is never force-initialized. - Owns the cursor (two catalogId->eventId maps, non-persisted), the role split (Env.isMaster()), the edit-log cursor write, and the follower->master REFRESH CATALOG forward — all HA/replication stays fe-core. - Wraps pollOnce in an onPluginClassLoader pin to the event source's plugin classloader (covers the RPC + JSON/GZIP deserialize; R-010), mirroring PluginDrivenScanNode. - Applies neutral descriptors through the E-d generalized mutators (register/ unregister/rename db+table, refresh table, add/drop/refresh partitions), faithfully preserving the legacy guards (rename-to-existing skip, view recreate always proceeds). On apply failure rolls the cursor back to just before the failing event and does NOT advance followers. - Writes a cursor-only MetaIdMappingsLog (the id-mapping payload the legacy path wrote is vestigial — its getters have no production reader — so only the synced cursor, the log's live effect, is written; opcode 470 + neutral GSON format unchanged). - updateMasterLastSyncedEventId(catalogId,eventId): the flip repoints the MetaIdMappingsLog replay handler here to feed the follower cursor. Constructed + started dormant in Env alongside the legacy poller (inert pre-flip: no PluginDriven hms catalog exposes an event source yet). fe-core compiles (-am); checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../java/org/apache/doris/catalog/Env.java | 11 + .../datasource/MetastoreEventSyncDriver.java | 312 ++++++++++++++++++ 2 files changed, 323 insertions(+) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java index 60dc28dda930bb..3c1e6066890645 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java @@ -101,6 +101,7 @@ import org.apache.doris.datasource.ExternalMetaCacheMgr; import org.apache.doris.datasource.ExternalMetaIdMgr; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.MetastoreEventSyncDriver; import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.PluginDrivenSysExternalTable; import org.apache.doris.datasource.SplitSourceManager; @@ -407,6 +408,9 @@ public class Env { private CooldownConfHandler cooldownConfHandler; private ExternalMetaIdMgr externalMetaIdMgr; private MetastoreEventsProcessor metastoreEventsProcessor; + // Connector-agnostic incremental metastore-event sync driver (Model B). Dormant until an HMS catalog is + // served by a PluginDrivenExternalCatalog whose connector exposes an event source. + private MetastoreEventSyncDriver metastoreEventSyncDriver; private JobManager, ?> jobManager; private LabelProcessor labelProcessor; @@ -762,6 +766,7 @@ public Env(boolean isCheckpointCatalog) { } this.externalMetaIdMgr = new ExternalMetaIdMgr(); this.metastoreEventsProcessor = new MetastoreEventsProcessor(); + this.metastoreEventSyncDriver = new MetastoreEventSyncDriver(); this.jobManager = new JobManager<>(); this.labelProcessor = new LabelProcessor(); this.transientTaskManager = new TransientTaskManager(); @@ -1042,6 +1047,10 @@ public MetastoreEventsProcessor getMetastoreEventsProcessor() { return metastoreEventsProcessor; } + public MetastoreEventSyncDriver getMetastoreEventSyncDriver() { + return metastoreEventSyncDriver; + } + public KeyManagerStore getKeyManagerStore() { return keyManagerStore; } @@ -2078,6 +2087,8 @@ protected void startNonMasterDaemonThreads() { feDiskUpdater.start(); metastoreEventsProcessor.start(); + // Dormant pre-flip: only drives PluginDrivenExternalCatalogs whose connector exposes an event source. + metastoreEventSyncDriver.start(); dnsCache.start(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java new file mode 100644 index 00000000000000..5e42a8a4ceb09c --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java @@ -0,0 +1,312 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.analysis.RedirectStatus; +import org.apache.doris.analysis.UserIdentity; +import org.apache.doris.catalog.Env; +import org.apache.doris.common.Config; +import org.apache.doris.common.util.MasterDaemon; +import org.apache.doris.connector.api.event.ConnectorEventSource; +import org.apache.doris.connector.api.event.EventPollRequest; +import org.apache.doris.connector.api.event.EventPollResult; +import org.apache.doris.connector.api.event.MetastoreChangeDescriptor; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.MasterOpExecutor; +import org.apache.doris.qe.OriginStatement; + +import com.google.common.collect.Maps; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; + +import java.util.List; +import java.util.Map; +import java.util.function.Supplier; + +/** + * The connector-agnostic, role-aware driver of incremental metastore-event sync. It is the fe-core half + * of the metastore-event relocation: it iterates catalogs, asks each connector that exposes a + * {@link ConnectorEventSource} for a batch of neutral {@link MetastoreChangeDescriptor}s, and applies + * them to the engine's catalog→db→table object graph and caches — the plugin never touches + * {@code CatalogMgr}, {@code EditLog}, or the HA state. + * + *

    Engine/connector split (Trino-aligned). The engine owns everything stateful and replicated: + * the per-catalog cursor, the master/follower role, the edit-log write of the synced cursor, and the + * follower→master {@code REFRESH CATALOG} forward. The connector owns only the metastore fetch + + * message parse behind {@code pollOnce}. This mirrors the legacy {@code MetastoreEventsProcessor} role + * logic exactly, but the source-specific work is now behind the SPI and the type gate + * ({@code instanceof HMSExternalCatalog}) is replaced by a capability probe ({@code getEventSource() != null}). + * + *

    Dormant until the flip. Only a {@link PluginDrivenExternalCatalog} whose connector exposes an + * event source is driven; pre-flip no such catalog exists, so this daemon is inert. At the flip the legacy + * poller's gate goes false and this driver takes over, and the {@code MetaIdMappingsLog} replay handler is + * repointed to feed THIS driver's follower cursor (see {@link #updateMasterLastSyncedEventId}). + * + *

    Classloader. {@code pollOnce} runs under a context-classloader pin to the event source's own + * plugin classloader (covering the notification RPC and the JSON/GZIP deserialization), mirroring + * {@code PluginDrivenScanNode.onPluginClassLoader}; the daemon thread does not inherit any pin. + */ +public class MetastoreEventSyncDriver extends MasterDaemon { + private static final Logger LOG = LogManager.getLogger(MetastoreEventSyncDriver.class); + + // This FE's per-catalog synced cursor. Not persisted (rebuilt as -1 on restart); single-threaded (the + // daemon thread), so a plain HashMap is fine. + private final Map lastSyncedEventIdMap = Maps.newHashMap(); + // The master's committed high-water mark per catalog, learned on followers via edit-log replay. A + // follower never fetches past it. Only meaningful on followers. + private final Map masterLastSyncedEventIdMap = Maps.newHashMap(); + + private boolean isRunning; + + public MetastoreEventSyncDriver() { + super(MetastoreEventSyncDriver.class.getName(), Config.hms_events_polling_interval_ms); + this.isRunning = false; + } + + @Override + protected void runAfterCatalogReady() { + if (isRunning) { + LOG.warn("Last metastore-event sync task not finished, ignore current task."); + return; + } + isRunning = true; + try { + realRun(); + } catch (Exception ex) { + LOG.warn("Metastore-event sync task failed", ex); + } + isRunning = false; + } + + private void realRun() { + List catalogIds = Env.getCurrentEnv().getCatalogMgr().getCatalogIds(); + for (Long catalogId : catalogIds) { + CatalogIf catalog = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogId); + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + continue; + } + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + // Only probe already-initialized catalogs — do not force-init an idle catalog just to ask + // whether it has an event source (the legacy poller never touched non-HMS catalogs; an idle + // catalog has no warm cache to keep fresh and re-reads on its first access anyway). + if (!pluginCatalog.isInitialized()) { + continue; + } + ConnectorEventSource eventSource; + try { + eventSource = pluginCatalog.getConnector().getEventSource(); + } catch (RuntimeException e) { + // uninitialized / unavailable connector this cycle => skip (mirrors the legacy skip-on-throw) + continue; + } + if (eventSource == null) { + continue; + } + try { + syncCatalog(pluginCatalog, eventSource); + } catch (Exception e) { + LOG.warn("Failed to sync metastore events for catalog [{}]", pluginCatalog.getName(), e); + } + } + } + + private void syncCatalog(PluginDrivenExternalCatalog catalog, ConnectorEventSource eventSource) + throws Exception { + long catalogId = catalog.getId(); + boolean isMaster = Env.getCurrentEnv().isMaster(); + long lastSyncedEventId = lastSyncedEventIdMap.getOrDefault(catalogId, -1L); + long masterUpperBound = masterLastSyncedEventIdMap.getOrDefault(catalogId, -1L); + + EventPollRequest request = new EventPollRequest(lastSyncedEventId, isMaster, masterUpperBound); + EventPollResult result = onPluginClassLoader(eventSource, () -> eventSource.pollOnce(request)); + + if (result.isNeedsFullRefresh()) { + // first sync or an events-gap: the master invalidates the whole catalog locally; a follower + // forwards REFRESH CATALOG to the master. Then seed the cursor to the connector's current id. + if (isMaster) { + refreshCatalogForMaster(catalog); + } else { + refreshCatalogForSlave(catalog); + } + commitCursor(catalogId, result.getNewCursor(), isMaster); + return; + } + + List descriptors = result.getDescriptors(); + if (descriptors.isEmpty()) { + // nothing to apply; still advance the cursor if it moved (e.g. a batch of ignored events) + if (result.getNewCursor() != lastSyncedEventId) { + commitCursor(catalogId, result.getNewCursor(), isMaster); + } + return; + } + + // Apply in order; on failure the cursor is rolled back to just before the failing descriptor and the + // exception propagates (caught + logged in realRun), so the next cycle re-fetches from there and the + // edit-log cursor below is NOT written (followers do not jump past a failed apply). + applyDescriptors(catalog, descriptors); + commitCursor(catalogId, result.getNewCursor(), isMaster); + } + + // Stores the local cursor and, on the master, replicates it to followers via the edit-log. + private void commitCursor(long catalogId, long newCursor, boolean isMaster) { + lastSyncedEventIdMap.put(catalogId, newCursor); + if (isMaster) { + writeSyncedCursorLog(catalogId, newCursor); + } + } + + private void applyDescriptors(PluginDrivenExternalCatalog catalog, + List descriptors) { + for (MetastoreChangeDescriptor descriptor : descriptors) { + try { + applyOne(catalog, descriptor); + } catch (Exception e) { + lastSyncedEventIdMap.put(catalog.getId(), descriptor.getEventId() - 1); + throw new RuntimeException( + "Failed to apply metastore change " + descriptor + " on catalog " + + catalog.getName(), e); + } + } + } + + // Applies one neutral descriptor via the engine's own (connector-agnostic) mutators — the same ones the + // legacy event.process() bodies called, now generalized to work on a flipped catalog. + private void applyOne(PluginDrivenExternalCatalog catalog, MetastoreChangeDescriptor descriptor) + throws Exception { + String catalogName = catalog.getName(); + CatalogMgr catalogMgr = Env.getCurrentEnv().getCatalogMgr(); + switch (descriptor.getOp()) { + case REGISTER_DATABASE: + catalogMgr.registerExternalDatabaseFromEvent(descriptor.getDbName(), catalogName); + break; + case UNREGISTER_DATABASE: + catalogMgr.unregisterExternalDatabase(descriptor.getDbName(), catalogName); + break; + case RENAME_DATABASE: + // legacy AlterDatabaseEvent.processRename: skip when the after-db already exists locally + if (catalog.getDbNullable(descriptor.getDbNameAfter()) == null) { + catalogMgr.unregisterExternalDatabase(descriptor.getDbName(), catalogName); + catalogMgr.registerExternalDatabaseFromEvent(descriptor.getDbNameAfter(), catalogName); + } + break; + case REGISTER_TABLE: + catalogMgr.registerExternalTableFromEvent(descriptor.getDbName(), descriptor.getTableName(), + catalogName, descriptor.getUpdateTime(), true); + break; + case UNREGISTER_TABLE: + catalogMgr.unregisterExternalTable(descriptor.getDbName(), descriptor.getTableName(), + catalogName, true); + break; + case RENAME_TABLE: + applyRenameTable(catalog, descriptor); + break; + case REFRESH_TABLE: + Env.getCurrentEnv().getRefreshManager().refreshExternalTableFromEvent(catalogName, + descriptor.getDbName(), descriptor.getTableName(), descriptor.getUpdateTime()); + break; + case ADD_PARTITIONS: + catalogMgr.addExternalPartitions(catalogName, descriptor.getDbName(), + descriptor.getTableName(), descriptor.getPartitionNames(), + descriptor.getUpdateTime(), true); + break; + case DROP_PARTITIONS: + catalogMgr.dropExternalPartitions(catalogName, descriptor.getDbName(), + descriptor.getTableName(), descriptor.getPartitionNames(), + descriptor.getUpdateTime(), true); + break; + case REFRESH_PARTITIONS: + Env.getCurrentEnv().getRefreshManager().refreshPartitions(catalogName, + descriptor.getDbName(), descriptor.getTableName(), + descriptor.getPartitionNames(), descriptor.getUpdateTime(), true); + break; + default: + break; + } + } + + private void applyRenameTable(PluginDrivenExternalCatalog catalog, MetastoreChangeDescriptor descriptor) + throws Exception { + String catalogName = catalog.getName(); + CatalogMgr catalogMgr = Env.getCurrentEnv().getCatalogMgr(); + // legacy AlterTableEvent: a rename to a DIFFERENT key that already exists locally is skipped + // (processRename guard); a view recreate (after == before) always proceeds (processRecreateTable). + boolean sameKey = descriptor.getDbName().equalsIgnoreCase(descriptor.getDbNameAfter()) + && descriptor.getTableName().equalsIgnoreCase(descriptor.getTableNameAfter()); + if (!sameKey && catalogMgr.externalTableExistInLocal(descriptor.getDbNameAfter(), + descriptor.getTableNameAfter(), catalogName)) { + return; + } + catalogMgr.unregisterExternalTable(descriptor.getDbName(), descriptor.getTableName(), + catalogName, true); + catalogMgr.registerExternalTableFromEvent(descriptor.getDbNameAfter(), + descriptor.getTableNameAfter(), catalogName, descriptor.getUpdateTime(), true); + } + + // Writes the synced-event-id cursor to the edit-log so followers advance to it (the log's only live + // purpose; the id-mapping payload the legacy path also wrote is vestigial — its getters have no + // production reader — so a cursor-only log is written). Opcode + neutral GSON format are unchanged. + private void writeSyncedCursorLog(long catalogId, long cursor) { + MetaIdMappingsLog log = new MetaIdMappingsLog(); + log.setCatalogId(catalogId); + log.setFromHmsEvent(true); + log.setLastSyncedEventId(cursor); + Env.getCurrentEnv().getExternalMetaIdMgr().replayMetaIdMappingsLog(log); + Env.getCurrentEnv().getEditLog().logMetaIdMappingsLog(log); + } + + private void refreshCatalogForMaster(CatalogIf catalog) { + CatalogLog log = new CatalogLog(); + log.setCatalogId(catalog.getId()); + log.setInvalidCache(true); + Env.getCurrentEnv().getRefreshManager().replayRefreshCatalog(log); + } + + private void refreshCatalogForSlave(CatalogIf catalog) throws Exception { + // A follower cannot refresh a catalog locally (that mutation must originate on the master); forward + // REFRESH CATALOG to the master, which replicates the result back. + String sql = "REFRESH CATALOG " + catalog.getName(); + OriginStatement originStmt = new OriginStatement(sql, 0); + ConnectContext ctx = new ConnectContext(); + ctx.setCurrentUserIdentity(UserIdentity.ROOT); + ctx.setEnv(Env.getCurrentEnv()); + MasterOpExecutor masterOpExecutor = new MasterOpExecutor(originStmt, ctx, + RedirectStatus.FORWARD_WITH_SYNC, false); + masterOpExecutor.execute(); + } + + /** + * Advances a follower's known master-committed cursor for a catalog. Wired from the + * {@code MetaIdMappingsLog} edit-log replay at the flip (mirrors the legacy + * {@code MetastoreEventsProcessor.updateMasterLastSyncedEventId}); keyed by catalog id only, so it never + * casts the catalog to a source-specific type. + */ + public void updateMasterLastSyncedEventId(long catalogId, long eventId) { + masterLastSyncedEventIdMap.put(catalogId, eventId); + } + + private static T onPluginClassLoader(ConnectorEventSource eventSource, Supplier body) { + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(eventSource.getClass().getClassLoader()); + return body.get(); + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } +} From 94acc88d6fa3091150d694c84df54fa6b33b8f71 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 12:54:38 +0800 Subject: [PATCH 171/334] [fix](catalog) Hive event pipeline: restore self-heal on a poison event (review) Clean-room review (wf_0d49c409-a86) confirmed the driver dropped the legacy poller's self-heal, so a deterministically-failing (poison) event or a descriptor whose apply throws would wedge the whole catalog's incremental sync forever (and freeze every follower waiting on the replicated cursor) until an FE restart. Legacy MetastoreEventsProcessor.realRun reacted to any processing exception with onRefreshCache(true) + reset-cursor-to-(-1), so the next cycle first-pulled a clean full refresh past the poison event. Restore it, split cleanly across the boundary: - HmsEventSource: a transient FETCH error (metastore blip, non-REPL HmsClientException) now retries the same cursor in place (log + ofNothing) instead of propagating, so an outage does not trigger a reset/invalidation storm. A deterministic PARSE error is a RuntimeException (not HmsClientException) and still propagates to the engine's self-heal. - MetastoreEventSyncDriver: realRun's per-catalog catch now resets the cursor to -1 (self-heal) so the next cycle first-pulls a full refresh, jumping past the poison event; applyDescriptors no longer rolls the cursor back one event (which caused the infinite retry). Also corrects the isInitialized()-gate comment to state the real limitation it carries (a flipped catalog never accessed on the master but queried on a follower does not seed its cursor) and that closing it without force-initing the byte-inert non-event catalogs is owed at the flip. Dormant pre-flip (no effect until an hms catalog is PluginDriven with an event source). fe-core + fe-connector-hms compile (-am); checkstyle 0; gate ok. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hms/event/HmsEventSource.java | 13 +++++++-- .../datasource/MetastoreEventSyncDriver.java | 27 +++++++++++++------ 2 files changed, 30 insertions(+), 10 deletions(-) diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java index 74ba211b1da843..856c58e3713ddf 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventSource.java @@ -26,6 +26,8 @@ import org.apache.doris.connector.hms.HmsNotificationEvent; import org.apache.hadoop.hive.metastore.HiveMetaStoreClient; +import org.apache.logging.log4j.LogManager; +import org.apache.logging.log4j.Logger; import java.util.ArrayList; import java.util.List; @@ -42,6 +44,7 @@ * {@code REPL_EVENTS_MISSING_IN_METASTORE} sentinel) yields a full-refresh signal instead of events.

    */ public class HmsEventSource implements ConnectorEventSource { + private static final Logger LOG = LogManager.getLogger(HmsEventSource.class); private final HmsClient client; private final int batchSize; @@ -80,7 +83,11 @@ private EventPollResult pollForMaster(long lastSyncedEventId) { // the metastore trimmed its log past our cursor; jump to now and full-refresh return EventPollResult.ofFullRefresh(currentEventId); } - throw e; + // transient fetch error (metastore blip): retry the same cursor next cycle without resetting or + // invalidating. A deterministic PARSE error is a RuntimeException, not an HmsClientException, so it + // propagates to the engine's self-heal instead of being swallowed here. + LOG.warn("Failed to fetch HMS notifications from event id {}; will retry", lastSyncedEventId, e); + return EventPollResult.ofNothing(lastSyncedEventId); } } @@ -102,7 +109,9 @@ private EventPollResult pollForFollower(long lastSyncedEventId, long masterUpper if (isEventsMissing(e)) { return EventPollResult.ofFullRefresh(masterUpperBound); } - throw e; + // transient fetch error: retry the same cursor next cycle (see pollForMaster). + LOG.warn("Failed to fetch HMS notifications from event id {}; will retry", lastSyncedEventId, e); + return EventPollResult.ofNothing(lastSyncedEventId); } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java index 5e42a8a4ceb09c..8729f144a2fc4b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java @@ -101,9 +101,13 @@ private void realRun() { continue; } PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; - // Only probe already-initialized catalogs — do not force-init an idle catalog just to ask - // whether it has an event source (the legacy poller never touched non-HMS catalogs; an idle - // catalog has no warm cache to keep fresh and re-reads on its first access anyway). + // Probe only already-initialized catalogs: calling getConnector() force-initializes, and we must + // NOT force-init the idle paimon/iceberg/jdbc/hudi PluginDriven catalogs that already exist (they + // stay byte-inert pre-flip). LIMITATION vs the legacy poller (which force-initialized every HMS + // catalog on the master via getHmsProperties): a flipped HMS catalog that is NEVER accessed on the + // master but IS queried on a follower never seeds its cursor, so that follower stops receiving + // incremental updates. Closing this without force-initing non-event catalogs needs a flip-time + // hook that initializes flipped event-source catalogs on the master (owed at the flip). if (!pluginCatalog.isInitialized()) { continue; } @@ -120,7 +124,15 @@ private void realRun() { try { syncCatalog(pluginCatalog, eventSource); } catch (Exception e) { - LOG.warn("Failed to sync metastore events for catalog [{}]", pluginCatalog.getName(), e); + // Self-heal (mirrors the legacy poller's onRefreshCache(true) + reset-to-(-1)): reset the + // cursor so the next cycle first-pulls -> full refresh, jumping past a deterministically-failing + // (poison) event/descriptor instead of retrying it forever and wedging the catalog's sync (and, + // on the master, freezing every follower that waits on the replicated cursor). Transient FETCH + // errors do not reach here — HmsEventSource retries them in place (ofNothing) — so this reset + // fires only on a deterministic parse/apply failure. + lastSyncedEventIdMap.put(catalogId, -1L); + LOG.warn("Failed to sync metastore events for catalog [{}]; reset cursor for a full re-sync", + pluginCatalog.getName(), e); } } } @@ -156,9 +168,9 @@ private void syncCatalog(PluginDrivenExternalCatalog catalog, ConnectorEventSour return; } - // Apply in order; on failure the cursor is rolled back to just before the failing descriptor and the - // exception propagates (caught + logged in realRun), so the next cycle re-fetches from there and the - // edit-log cursor below is NOT written (followers do not jump past a failed apply). + // Apply in order; on failure the exception propagates and realRun's catch resets the cursor to -1 + // (self-heal), so the edit-log cursor below is NOT written (followers do not jump past a failed apply) + // and the next cycle first-pulls a clean full refresh instead of retrying the poison descriptor. applyDescriptors(catalog, descriptors); commitCursor(catalogId, result.getNewCursor(), isMaster); } @@ -177,7 +189,6 @@ private void applyDescriptors(PluginDrivenExternalCatalog catalog, try { applyOne(catalog, descriptor); } catch (Exception e) { - lastSyncedEventIdMap.put(catalog.getId(), descriptor.getEventId() - 1); throw new RuntimeException( "Failed to apply metastore change " + descriptor + " on catalog " + catalog.getName(), e); From fa1f86f37d88a8a951d3884ef0902b59d7a7d90f Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 12:54:51 +0800 Subject: [PATCH 172/334] [fix](catalog) Hive event pipeline: decompress the message body lazily (review) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Clean-room review (wf_0d49c409-a86) confirmed HmsEventParser eagerly base64+gzip-decompressed the message for EVERY event before the type switch, including CREATE/DROP DATABASE, INSERT, and ignored/unsupported (e.g. high-frequency ACID txn) events that never read the body — wasted CPU per ignored event on a gzip-format metastore, and a corrupt body on an otherwise-ignored event would throw. Legacy deserialized lazily only inside the ctor of a supported event that actually needs the body. Gate prepareBody() on needsBody(type): only the seven payload-reading event types decompress; all others build their descriptor from the base event fields alone. (The wedge risk this widened is separately closed by the self-heal fix.) Dormant pre-flip. fe-connector-hms compiles; checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hms/event/HmsEventParser.java | 24 ++++++++++++++++++- 1 file changed, 23 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java index 31224920f9bcf1..a407bd62092a24 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/event/HmsEventParser.java @@ -90,7 +90,11 @@ private static List doParse(HmsNotificationEvent even long eventId = event.getEventId(); // Hive records event time in seconds; the engine tracks freshness in millis. long updateTime = event.getEventTime() * 1000L; - String body = prepareBody(event.getMessageFormat(), event.getMessage()); + // Decompress/parse the message body ONLY for event types that actually read it — mirrors the legacy + // events, which deserialize lazily inside the supported-event ctors. So a db-level / INSERT / ignored + // (e.g. high-frequency ACID txn) event never touches the payload: no wasted gzip work, and a corrupt + // body on an event Doris ignores can never throw. + String body = needsBody(type) ? prepareBody(event.getMessageFormat(), event.getMessage()) : null; switch (type) { case "CREATE_TABLE": { @@ -221,6 +225,24 @@ private static List one(MetastoreChangeDescriptor des return Collections.singletonList(descriptor); } + // The event types whose descriptor mapping reads the (possibly gzip) message payload; all others + // (CREATE/DROP DATABASE, INSERT, and ignored/unsupported types) build their descriptor from the base + // event fields alone and never decompress. + private static boolean needsBody(String type) { + switch (type) { + case "CREATE_TABLE": + case "DROP_TABLE": + case "ALTER_TABLE": + case "ALTER_DATABASE": + case "ADD_PARTITION": + case "DROP_PARTITION": + case "ALTER_PARTITION": + return true; + default: + return false; + } + } + private static String prepareBody(String format, String message) { if (format != null && format.startsWith(GZIP_FORMAT_PREFIX)) { return deCompress(message); From cc2b0945436fe4d75cb584cab64c766a31f986a7 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 12:58:20 +0800 Subject: [PATCH 173/334] [test](catalog) Hive event pipeline: dormant parser coverage (E-f) Unit-covers HmsEventParser's neutral (body-free) event mappings and the lazy-decompress guarantee: CREATE/DROP DATABASE -> register/unregister, INSERT -> refresh-table (+ db-name lowercasing, seconds->millis eventTime), unsupported/null type -> no descriptor, and that a body-free event never decompresses its message (a gzip-tagged un-decompressible or null body must not throw). The body-parsing table/partition paths need captured Hive JSON/GZIP fixtures and are covered by the heterogeneous-HMS e2e matrix owed at the flip. 6 tests green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../hms/event/HmsEventParserTest.java | 96 +++++++++++++++++++ 1 file changed, 96 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/event/HmsEventParserTest.java diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/event/HmsEventParserTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/event/HmsEventParserTest.java new file mode 100644 index 00000000000000..78d44033a47173 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/event/HmsEventParserTest.java @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms.event; + +import org.apache.doris.connector.api.event.MetastoreChangeDescriptor; +import org.apache.doris.connector.api.event.MetastoreChangeDescriptor.Op; +import org.apache.doris.connector.hms.HmsNotificationEvent; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.List; + +/** + * Dormant unit coverage of the neutral (body-free) event mappings and the lazy-decompress guarantee. + * The body-parsing table/partition paths (which need captured Hive JSON/GZIP message fixtures) are + * covered by the heterogeneous-HMS e2e matrix owed at the flip. + */ +public class HmsEventParserTest { + + private static HmsNotificationEvent event(long id, String type, String db, String table, + String message, String format, long timeSec) { + return new HmsNotificationEvent(id, type, db, table, message, format, timeSec); + } + + @Test + public void createDatabaseMapsToRegisterDb() { + List out = HmsEventParser.parse( + event(7L, "CREATE_DATABASE", "MyDb", null, null, "json-2.0", 100L)); + Assertions.assertEquals(1, out.size()); + MetastoreChangeDescriptor d = out.get(0); + Assertions.assertEquals(Op.REGISTER_DATABASE, d.getOp()); + // db name is lowercased, mirroring the legacy base MetastoreEvent + Assertions.assertEquals("mydb", d.getDbName()); + Assertions.assertEquals(7L, d.getEventId()); + // event time (seconds) is surfaced as millis + Assertions.assertEquals(100L * 1000L, d.getUpdateTime()); + } + + @Test + public void dropDatabaseMapsToUnregisterDb() { + List out = HmsEventParser.parse( + event(8L, "DROP_DATABASE", "db1", null, null, "json-2.0", 0L)); + Assertions.assertEquals(1, out.size()); + Assertions.assertEquals(Op.UNREGISTER_DATABASE, out.get(0).getOp()); + Assertions.assertEquals("db1", out.get(0).getDbName()); + } + + @Test + public void insertMapsToRefreshTable() { + List out = HmsEventParser.parse( + event(9L, "INSERT", "db1", "t1", null, "json-2.0", 0L)); + Assertions.assertEquals(1, out.size()); + Assertions.assertEquals(Op.REFRESH_TABLE, out.get(0).getOp()); + Assertions.assertEquals("db1", out.get(0).getDbName()); + Assertions.assertEquals("t1", out.get(0).getTableName()); + } + + @Test + public void unsupportedTypeProducesNoDescriptor() { + Assertions.assertTrue(HmsEventParser.parse( + event(10L, "COMMIT_TXN", "db1", null, null, "json-2.0", 0L)).isEmpty()); + } + + @Test + public void nullEventTypeIsIgnored() { + Assertions.assertTrue(HmsEventParser.parse( + event(13L, null, "db1", null, null, "json-2.0", 0L)).isEmpty()); + } + + @Test + public void bodyFreeEventNeverDecompressesTheMessage() { + // A db-level / ignored event must not touch the payload: even a gzip-tagged, un-decompressible body + // (or a null body) cannot throw, because prepareBody is gated on needsBody(type). This is the + // regression the lazy-decompress fix closed. + Assertions.assertDoesNotThrow(() -> HmsEventParser.parse( + event(11L, "CREATE_DATABASE", "db1", null, "not-valid-gzip", "gzip(json-2.0)", 0L))); + Assertions.assertDoesNotThrow(() -> HmsEventParser.parse( + event(12L, "COMMIT_TXN", "db1", null, null, "gzip(json-2.0)", 0L))); + } +} From b1a218fe58fe7675e0a90dffdfb0eff368531b1d Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 13:00:57 +0800 Subject: [PATCH 174/334] [doc](catalog) Hive event pipeline DONE: record landing + clean-room review + fixes; HANDOFF -> Phase 1 remaining MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Event-pipeline Model B (Phase 1 item 2) is complete: E-a..E-f landed dormant, clean-room adversarial review (wf_0d49c409-a86) done, 2 confirmed regressions fixed (self-heal poison-event, lazy gzip decompress), 1 kept-with-flip-owed (isInitialized gate). Design doc gets a landing+review record (§6) + the new flip-time item (master must init flipped event-source catalogs). HANDOFF rolls forward: next = the 3 loud-break coupling seams + W6 write-path TCCL verify, then the atomic flip; the 3 pending D2 review decisions still stand. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 16 +++++++--------- ...hive-event-pipeline-step-design-2026-07-10.md | 14 ++++++++++++++ 2 files changed, 21 insertions(+), 9 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index df5fb59eccbe00..224925edfbce52 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -24,15 +24,13 @@ D2 = `f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` - ❌ **未修-待拍板 #3**（review2 §3，**线上活性 bug 非休眠**）：paimon/iceberg `latestSnapshotCache`（+`PaimonSchemaAtMemo` 窄化变体）今天就有同形状的删建重名洞（fe-core DDL 路径从不 invalidate、key=纯表名）。修=独立线（动 live 行为，须自带复审+回归），或记档延后。 - 测试加固欠账（LOW，代码正确仅测试盲区）：`createClient` 真包裹无 pin、公开 invalidate 钩子未用已构建 client 驱动（review2 §2.4）。 -**⭐ 状态（本轮，2026-07-10 晚）= Phase 1 第二项「事件管道 Model B」的 SPI + 插件侧已落地（E-a/E-b/E-c，休眠、独立可复审）；fe-core 侧（E-d/E-e/E-f）+ 净室复审待续。上面 3 项 D2 复审决策仍待用户拍板——与事件管道解耦（那 3 项是 D2 缓存精化：DDL 路径失效 / 分区缓存容量单位 / paimon-iceberg live 删建洞；事件管道只依赖已落地的 D2 invalidate SPI），可并行处理。** -- 权威设计 = **`tasks/hive-event-pipeline-step-design-2026-07-10.md`**（**起步必读**；含用户签字 A=结构性事件即时重建平价 + 6 步序列 E-a…E-f + 侦察 `wf_0c686fb4-d9d` 6 维+critic）。**SUPERSEDES** `hms-event-pipeline-findings-2026-07-07.md`（那份是旧侦察，行号已漂；设计稿是当前权威）。 -- 已落地（全休眠：hms 未 SPI_READY；paimon/iceberg/jdbc/hudi 字节不变；各步 test-compile BUILD SUCCESS + checkstyle 0 + import gate ok）：`0214f0405c1` **E-a** 中立事件 SPI（`MetastoreChangeDescriptor`+op / `ConnectorEventSource` / `EventPollRequest|Result` / `Connector.getEventSource()` default null / `Connector.invalidatePartition` default no-op）→ `b13ed79e32d` **E-b** 插件取数（`HmsClient.getCurrentNotificationEventId/getNextNotification` default 抛 + `ThriftHmsClient` 委派 vendored client + `CachingHmsClient` 透传 + `HmsNotificationEvent` DTO）→ `902546d79c5` **E-c** 插件事件源（`HmsEventParser` 忠实映射事件→中立描述符 + `HmsEventSource` 角色感知 fetch + `HiveConnector.getEventSource/invalidatePartition` + 2 个 per-catalog 属性键）。 -- **剩余（权威=设计稿 §2.3/§3，行号信 HEAD 不信文档）**： - - **E-d fe-core mutator 泛化（休眠、向后兼容 legacy）**：`CatalogMgr.registerExternalTableFromEvent` 去两处 HMS cast（`buildTableForInit` 通用；⚠ 注意 `ExternalDatabase.buildTableForInit`+`registerTable` 泛型捕获，先读签名，可能用 raw 或 typed helper）；`registerDatabase` 上提 `PluginDrivenExternalCatalog`（通用体 `buildDbForInit`+`metaCache.updateCache`+`genIdByName`，见 `HMSExternalCatalog:205`）；`unregisterExternalDatabase` 去多余 cast；`addExternalPartitions/dropExternalPartitions`+`RefreshManager.refreshPartitions`+`replayRefreshTable 分区分支` 加 `if(PluginDrivenExternalCatalog) getConnector().invalidatePartition(...)` 分支（与现有 `instanceof HMSExternalTable` 分支互斥，pre-flip 走 HMS 分支不变）。 - - **E-e fe-core 连接器无关驱动**：新 `MasterDaemon`（**存活包** `datasource/`，非将删的 `hive/event/`），能力探测 `PluginDrivenExternalCatalog + getConnector().getEventSource()!=null`；两 cursor map；`Env.isMaster()` 分角色；`onPluginClassLoader` pin 包 `pollOnce`；应用描述符（泛化 mutator + `invalidatePartition`）；master 从描述符建 `MetaIdMappingsLog`（op→ADD/DELETE×objType×names，**修 drop-partition 误标 DATABASE→PARTITION**）+ 写编辑日志；`needsFullRefresh` → master `replayRefreshCatalog` / follower 转发 `REFRESH CATALOG`。构造+启动休眠（Env 内，pre-flip 无匹配目录）。 - - **E-f 休眠单测** + **净室对抗复审**（多 agent，先独立判断后交叉核对历史结论）。 -- **翻闸时接线（Phase 2，非本步）**：去 `MetastoreEventsProcessor:116` gate；`ExternalMetaIdMgr.replayMetaIdMappingsLog` 游标回放改指新驱动。 -- **⚠ 遗留（设计稿已标注）**：连接器 enable 开关默认 false；若部署靠 fe-core 全局 Config `enable_hms_events_incremental_sync=true`（无 per-catalog 属性）开启增量同步，翻闸后须补 per-catalog `hive.enable_hms_events_incremental_sync` 属性（flip/e2e 迁移项）。合并/去重（mergeEvents）本步有意省略（批大小优化；拉取式缓存+顺序应用即正确）。 +**⭐ 状态（本轮，2026-07-10 晚）= Phase 1 第二项「事件管道 Model B」全部落地 + 净室对抗复审 + 2 处复审修复，✅ DONE。** 上面 3 项 D2 复审决策仍待用户拍板（与事件管道解耦，可并行）。 +- 权威设计 = **`tasks/hive-event-pipeline-step-design-2026-07-10.md`**（**起步必读**；含用户签字 A=结构性事件即时重建平价 + 6 步序列 + 落地/复审记录 §6 + 侦察 `wf_0c686fb4-d9d` / 复审 `wf_0d49c409-a86`）。**SUPERSEDES** `hms-event-pipeline-findings-2026-07-07.md`（旧侦察，行号已漂）。 +- 已落地（全休眠：hms 未 SPI_READY；paimon/iceberg/jdbc/hudi 字节不变；各步 test-compile SUCCESS + checkstyle 0 + import gate ok）：`0214f04` **E-a** 中立事件 SPI（`MetastoreChangeDescriptor`/`ConnectorEventSource`/`getEventSource`/`invalidatePartition`）→ `b13ed79` **E-b** 插件取数（`HmsClient` 通知方法 + `HmsNotificationEvent`）→ `902546d` **E-c** 插件事件源（`HmsEventParser` 忠实映射 + `HmsEventSource` 角色感知 + `HiveConnector` 接线）→ `3552554` **E-d** fe-core mutator 泛化（de-cast + `registerDatabase` 上提 PluginDriven + 分区 PluginDriven 分支）→ `6a96820` **E-e** fe-core 驱动（`MetastoreEventSyncDriver` + Env 接线）→ `9113c51` **E-f** 休眠 parser 单测（6 绿）。 +- **净室复审**（`wf_0d49c409-a86`，6 维 refute-by-default）14 疑点、4 坐实=**3 真回归（均休眠 pre-flip、须翻闸前修）**：**① 自愈丢失(poison event 死锁)** 已修 `fb21498`（`HmsEventSource` 瞬时 fetch 错误原地重试`ofNothing`、驱动 `realRun` catch 把游标归 -1 → 下轮 first-pull 全刷跳过毒事件；`applyDescriptors` 不再回退一格）；**② eager gzip 解压** 已修 `134907b`（`prepareBody` 按 `needsBody(type)` 惰性）；**③ `isInitialized()` gate**（主从只读分离下漏 seed 游标 → 从节点静默停更）**保留**（去掉会 force-init 空闲 paimon/iceberg/jdbc 破字节不变）+ 改诚实注释 + 列翻闸项。10 条误报（大小写canonical enum / 空分区 / enable-gate 翻闸自owed 等）已 refute。 +- **⭐ 下一步 = Phase 1 剩余项**（权威=execution plan §2.3 + 设计稿）：**3 个 loud-break 耦合缝**（`partition_values()` TVF / `hudi_meta()` TIMELINE TVF / `canSample`+`SUPPORTS_SAMPLE_ANALYZE`，均加休眠 arm，翻闸后否则 loud-break）→ **W6 写路径 TCCL 验证**（`fe-connector-hive` 无 `TcclPinningConnectorContext`，iceberg-on-HMS 写按名反射有 CCE 风险；verify-first）。之后 → **Phase 2 原子翻闸**。 +- **翻闸时接线（Phase 2，非本步，勿丢）**：① 去 `MetastoreEventsProcessor:116` gate + `ExternalMetaIdMgr.replayMetaIdMappingsLog` 游标回放**改指新驱动 `getMetastoreEventSyncDriver().updateMasterLastSyncedEventId`**；② **主节点初始化 flipped 事件源目录**（复审 finding #2，否则主从只读分离静默停更；设计稿 §2.4）；③ enable 开关：靠 fe-core 全局 Config 开启的部署须补 per-catalog `hive.enable_hms_events_incremental_sync`，或翻闸时 fe-core 把 resolved 值注入属性图。 +- **⚠ 复审确认 e2e 欠账（补 execution plan §4）**：毒事件自愈断言、主从只读分离下 flipped 目录游标传播、gzip/plain 各事件类型表/分区体解析忠实度（单测只覆盖 body-free 中立路径）。合并/去重(mergeEvents)本步有意省略（批优化；拉取式+顺序应用即正确）。 - **⚠ 并行 session 风险**：曾有两个 session 同时在此工作树（一个 amend 卷入另一个未提交测试；本轮 HANDOFF 也被并行 session 更新过）；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 **⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步（读 / 写-拒 / schema-evolution / 缓存 / 跨加载器委派）只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`）。**删除排序最硬约束**：`datasource/hive|hudi|iceberg/` 的 ~90 个 HMS 支撑类删不掉，直到翻闸把消费者切到连接器路径（详见 execution plan §2.4/§3 + `tasks/iceberg-on-hms-delegation-findings-2026-07-07.md`）。 diff --git a/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md b/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md index 23eb7f41d91991..3dee87b09063f4 100644 --- a/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md +++ b/plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md @@ -65,6 +65,7 @@ ### 2.4 翻转时接线（Phase 2，非本步；本步只留清单） - `MetastoreEventsProcessor:116` 老 gate 去除；`ExternalMetaIdMgr.replayMetaIdMappingsLog` 的游标回放**改指新驱动**（或双写，二选一，翻转时定）；启用新驱动对 flipped-hms 生效。老 poller + `datasource/hive/event/` 整目录 **Phase 3 删**（新驱动替换入口后）。 +- **（复审新增，finding #2）主节点须初始化 flipped 事件源目录**：新驱动只处理已初始化的目录（避免 force-init 空闲的 paimon/iceberg/jdbc，保 pre-flip 字节不变）。但老 poller 曾在主节点强制初始化每个 HMS 目录。故翻转时须有一处让主节点**初始化 flipped 事件源目录**（否则"主从只读分离、某目录仅被从节点查询"时，主不 seed 游标 → 该从节点静默停更）。若不做则须签字接受该 HA 退化 + e2e 断言。 --- @@ -92,3 +93,16 @@ - 跨边界 pin TCCL（驱动线程包 pollOnce）✓ - `history_schema_info` 逐层小写：本步不涉 schema 字典 ✓ - `PluginDrivenMvccExternalTable` 字节+成本双不变：本步不改其共享方法（只经 `buildTableInternal` 新建，走既有路径）✓ + +--- + +## 6. 落地 + 净室复审记录（2026-07-10 晚，DONE） +6 步全部落地（休眠、独立提交、各 test-compile SUCCESS + checkstyle 0 + import gate ok）： +E-a `0214f04`（SPI）→ E-b `b13ed79`（插件取数）→ E-c `902546d`（插件事件源）→ E-d `3552554`（fe-core mutator 泛化）→ E-e `6a96820`（fe-core 驱动）→ E-f `9113c51`（休眠 parser 单测，6 绿）。 + +**净室对抗复审**（`wf_0d49c409-a86`：6 维盲评 → 逐条 refute-by-default 对抗验证）= 14 疑点、4 坐实（其余 10 条含大小写/空分区/enable-gate 等经验证为误报或休眠+翻闸自owed）。4 坐实收敛为 3 个真回归（均休眠 pre-flip、须翻闸前修）： +- **自愈丢失（poison event 死锁，#1≡#3，#4 的根因）** → 已修 `fb21498`：老 poller 遇处理异常做 `onRefreshCache(true)+游标归 -1` 自愈跳过毒事件；新驱动之前无限重试。修法=`HmsEventSource` 把**瞬时 fetch 错误**原地重试(`ofNothing`，不 reset/不invalidate)，只让**确定性 parse/apply 错误**上抛；驱动 `realRun` catch 把游标归 -1 → 下轮 first-pull 全刷跳过毒事件；`applyDescriptors` 不再回退一格。 +- **eager gzip 解压（#4）** → 已修 `134907b`：`prepareBody` 按 `needsBody(type)` 惰性——db/insert/ignored 事件不解压（省 CPU + 损坏体不抛；死锁本身已由自愈兜底）。 +- **`isInitialized()` gate（#2，未改代码，留翻闸项）**：跳过主节点空闲目录会漏 seed 游标 → 从节点静默停更；但去掉 gate 会 force-init 空闲 paimon/iceberg/jdbc（破坏 pre-flip 字节不变）——**两难**。裁决=保 gate（字节不变优先）+ 改诚实注释 + 列为翻闸项（§2.4 复审新增：翻转时主节点初始化 flipped 事件源目录）。 + +**⚠ 复审确认的 e2e 欠账（补 §4）**：毒事件自愈（构造确定性抛错的 descriptor，断言下轮全刷自愈非死锁）；主从只读分离下 flipped 目录游标传播（断言从节点不停更）；gzip/plain 各事件类型的表/分区体解析忠实度（本步单测只覆盖 body-free 中立路径）。 From b226e4110f276d7be1f01f6eae15f7ecda66d088 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 13:44:25 +0800 Subject: [PATCH 175/334] [doc](catalog) Hive coupling-seams step design: recon + 3 parity decisions + TODO Phase-1 remaining after D2 cache + event pipeline: the loud/silent-break coupling seams that the atomic hms flip would otherwise break. Distills the HEAD-grounded recon (wf_dfe1cb86-df4) and records the user's three FULL-PARITY decisions: keep hudi_meta (connector-driven), full-port sampled ANALYZE, and a per-table auto-analyze gate excluding hudi-on-HMS. W6 = verified false gap. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- ...e-coupling-seams-step-design-2026-07-10.md | 149 ++++++++++++++++++ 1 file changed, 149 insertions(+) create mode 100644 plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md diff --git a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md new file mode 100644 index 00000000000000..e81d87c48c8fea --- /dev/null +++ b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md @@ -0,0 +1,149 @@ +# Hive coupling-seams step — design (2026-07-10) + +> Phase-1 remaining fe-core build-out after D2 cache + event-pipeline landed. Authoritative plan = +> `hms-cutover-execution-plan-2026-07-10.md` §2.3 (loud-break seams) + §2.5 (W6). This doc distills the +> HEAD-grounded recon (`wf_dfe1cb86-df4`: 4 seam readers + completeness critic) and records the user's +> three parity decisions (2026-07-10). **Trust HEAD, not these line numbers — re-verify on edit.** +> Every seam ships as an INDEPENDENT dormant commit (inert while hms is legacy), same discipline as the +> connector steps. Clean-room adversarial review at the end. + +## Why these exist + +At the atomic flip a hms table becomes a `PluginDrivenMvccExternalTable` (type `PLUGIN_EXTERNAL_TABLE`), +not an `HMSExternalTable`. fe-core sites that `instanceof HMSExternalTable` / cast / gate on +`HMSExternalCatalog` break — loud (throw) or silent (wrong result / degrade). We stage the fixes dormant +now so the flip commit only has to swap gates, never grow behavior. + +## User decisions (2026-07-10) — all chose FULL PARITY + +1. **Timeline TVF (`hudi_meta`)** → **KEEP, rework connector-driven.** (Recon overturned the plan's DROP + lean: 4 p2 hudi suites consume it — `test_hudi_meta` asserts it directly; `test_hudi_incremental` / + `test_hudi_partition_prune` / `test_hudi_timetravel` use it as a commit-timestamp source.) +2. **Sampled ANALYZE (`ANALYZE … WITH SAMPLE`) on hive** → **FULL PORT** so hive keeps working (today it + works via `HMSAnalysisTask.doSample`; a no-op flip would make it throw `DdlException`, unlike a silent + FULL fallback the plan assumed). Chose parity over the recon's "accept degrade" lean. +3. **Background column auto-analyze eligibility** → **per-table gate excluding hudi-on-HMS**, exactly like + legacy (`dlaType HIVE || ICEBERG`, HUDI excluded). Mirrors the Top-N / nested-prune per-table pattern. + +W6 (write-path TCCL) = **verified false gap, no code** (pin already lives on the iceberg sibling's +`TcclPinningConnectorContext`, `IcebergConnector.java:174`, threaded through hive's per-handle delegation). +Only owes an e2e. Optional: soften the over-cautious comment at `HiveConnector.java:206-208`. + +--- + +## Seam 1 — `partition_values()` TVF (loud break; LIVE for paimon/iceberg once landed) + +**Break:** `PartitionValuesTableValuedFunction.analyzeAndGetTable` gate (`:113`) throws +`"Catalog of type 'hms' is not allowed in ShowPartitionsStmt"` for a `PluginDrivenExternalCatalog`; +downstream casts to `HMSExternalTable` (`:130-133`, `:170`) would CCE; `MetadataGenerator` +`partitionValuesMetadataResult` switch (`:2090`) has only `HMS_EXTERNAL_TABLE`. + +**Fix = mirror the already-done `$partitions` TVF** (`PartitionsTableValuedFunction` gate `:172-176`, +allowed-types `:184-186`, plugin arm `:201-209`; `MetadataGenerator.dealPluginDrivenCatalog`). Edits: +- (A) gate `:113` → add `|| catalog instanceof PluginDrivenExternalCatalog`. +- (B) `getTableOrMetaException` `:124-125` → add `TableType.PLUGIN_EXTERNAL_TABLE`. +- (C) `:130-136` → add a `PluginDrivenExternalTable` arm doing `isPartitionedTable()` (no HMS cast); + keep the HMS arm. +- (D) `getTableColumns` `:170` → hoist to base `((ExternalTable) table).getPartitionColumns( + MvccUtil.getSnapshotFromContext(table))` (`ExternalTable.getPartitionColumns(Optional)` + `:468`) — resolves for both legacy HMS and plugin without a source cast, no branch. +- (E) `MetadataGenerator` `:2090` → add `case PLUGIN_EXTERNAL_TABLE -> + partitionValuesMetadataResultForPluginTable(table, colNames)`; new method feeds the EXISTING TCell + type-switch (`:2144-2181`). +- **Values source (Opt B, chosen):** add SPI method `PluginDrivenExternalTable.getNameToPartitionValues( + Optional) : Map>` (name → per-column values in partition-column + order), refactor the extraction loop already in `getNameToPartitionItems` (`:753-764`) so both share it. + Keeps `MetadataGenerator` symmetric with the HMS arm (`getHivePartitionValues().getNameToPartitionValues()`). + +**Byte-parity:** the new arm must map `null` / `TablePartitionValues.HIVE_DEFAULT_PARTITION` → NULL TCell +(HMS path `:2140`) and preserve partition-column ORDER. For paimon/iceberg this is a NEW capability (no +parity target, just correctness); for hive it is e2e-owed post-flip. + +**Not dormant:** paimon/iceberg are already `PluginDrivenExternalCatalog`, so edits A/C/E go LIVE for them +at merge — this is a deliberate expansion consistent with `$partitions` (which already did it). ⇒ +**unit/regression-testable NOW** against paimon/iceberg (partitioned table returns rows; unpartitioned +throws "not a partitioned table"). Iron rules: dispatch on `PluginDrivenExternalCatalog`/base +`ExternalTable`, never `instanceof HMSExternal*`; no property parsing (values come from connector +`listPartitions`). + +## Seam 2 — `hudi_meta()` / TIMELINE TVF (silent break + delete-time compile break) → KEEP connector-driven + +**Break:** `MetadataGenerator.hudiMetadataResult` gate `:459` `!(dorisTable instanceof HMSExternalTable)` +→ post-flip returns `"The specified table is not a hudi table"`; body casts (`:463`) and reaches +`ExternalMetaCacheMgr.hudi(...).getHoodieTableMetaClient(...).getActiveTimeline()` (deletion-unit classes) ++ imports `org.apache.hudi` timeline classes into fe-core (`:128-129`, used only here `:469,:473`). + +**Fix (KEEP):** add a connector-neutral metadata-rows SPI (mirror `ConnectorProcedureResult` row-return), +e.g. `ConnectorCapability.SUPPORTS_METADATA_TABLE` + a connector method returning neutral rows; implement +in `HudiConnector` (timeline data already connector-side: `HudiMetaClientExecutor` / +`HoodieTableMetaClient.getActiveTimeline().getInstants()`). Rewrite `hudiMetadataResult` to gate on the +generic plugin/capability type and delegate; drop the `HMSExternalTable` cast and the `org.apache.hudi` +imports from fe-core. Dormant: while legacy no hms table is `PluginDriven`, so the old HMS arm still +serves; at the flip the plugin arm activates. Iron rules: timeline iteration + parsing stays in +`HudiConnector` (fe-core sheds `org.apache.hudi`); pin TCCL on the delegated read (bundled hudi +reflection). Parity target = the 4 p2 suites' rows (`timestamp/action/state/state_transition_time`) — +e2e-owed (enableHudiTest). **The old body is removed at the delete step regardless.** + +## Seam 3 — `ANALYZE … WITH SAMPLE` on hive → FULL PORT (3 coordinated pieces) + +**Break:** flipped hive table is `PluginDrivenMvccExternalTable` → `AnalysisManager.canSample` (`:1480`, +HMS arm `:1484-1485` casts + `getDlaType()==HIVE`) returns false → `buildAndAssignJob` (`:224`) throws +`DdlException("… doesn't support sample analyze.")`. Today hive WITH SAMPLE WORKS (via +`HMSAnalysisTask.doSample` `:218-270` + `getSampleInfo` `:344-379` reading `getChunkSizes` `:972-981`). +A naïve `canSample=true` only converts the clean build-time error into a runtime +`NotImplementedException` from `ExternalAnalysisTask.doSample` (`:119`) / `ExternalTable.getChunkSizes` +(`:420`). Also `AnalyzeTableCommand.isSamplingPartition` (`:315`, `:322`) degrades (critic) — port too. + +**Fix (port), all connector-agnostic:** +1. `ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE` (new). Hive emits it PER-TABLE (marker path) for its + plain-hive tables only — legacy gated `dlaType==HIVE`, so iceberg-on-HMS / hudi-on-HMS excluded; iceberg/ + paimon-native withhold it (keep their current reject → cross-connector unchanged). +2. `AnalysisManager.canSample` arm: `table instanceof PluginDrivenExternalTable && + ((PluginDrivenExternalTable) table).supportsSampleAnalyze()` where `supportsSampleAnalyze()` uses the + existing `hasScanCapability`/`PER_TABLE_CAPABILITIES_KEY` path (mirror `supportsTopNLazyMaterialize`). + Same treatment for `isSamplingPartition`. +3. `PluginDrivenExternalTable.createAnalysisTask` → return a sample-capable task (port + `HMSAnalysisTask.doSample`+`getSampleInfo`+`needLimit`); `PluginDrivenExternalTable.getChunkSizes` + override → real per-file byte sizes via a new `Connector` chunk-sizes SPI (connector supplies raw + byte lengths like `HMSExternalTable.getChunkSizes`; Doris-type slot-width math stays fe-core). + +**Not dormant-unit-testable end-to-end** (issues real sampling SQL) → e2e-owed on heterogeneous HMS. +Iron rules: capability must be per-table (a connector-wide flag would source-branch by proxy and admit +iceberg/hudi-on-HMS); connector returns raw facts, fe-core does type math. + +## Seam 4 — background column auto-analyze eligibility → per-table gate (exclude hudi-on-HMS) + +**Break (silent, documented residual):** post-flip `StatisticsUtil.supportAutoAnalyze` (`:989`; dead HMS +arm `:1008-1011` = `HIVE||ICEBERG`) resolves via `PluginDrivenExternalTable.supportsColumnAutoAnalyze()` +(`:223-230`) which reads CONNECTOR-WIDE `SUPPORTS_COLUMN_AUTO_ANALYZE` (`HiveConnector.getCapabilities` +`:278`). Connector-wide can't express the legacy per-dlaType gate → declaring it admits hudi-on-HMS +(legacy excluded) = silent expansion; withholding it drops plain-hive = silent degrade. Comment +`HiveConnector.java:247-249` already flags it as "residual … gate per-handle or explicitly accept". + +**Fix (per-table, chosen):** change `supportsColumnAutoAnalyze()` to resolve via `hasScanCapability( +SUPPORTS_COLUMN_AUTO_ANALYZE)` (additive: native iceberg/paimon still declare it connector-wide → +unchanged); REMOVE it from `HiveConnector`'s connector-wide `EnumSet` and instead emit the per-table +marker for hive-type + iceberg-on-HMS tables, NOT hudi-on-HMS. Iron rules: connector decides per-table by +emitting the marker; fe-core never inspects dlaType/format. Byte-parity for iceberg/paimon-native = they +keep the connector-wide flag → unchanged. e2e-owed: hudi-on-HMS NOT auto-analyzed, hive/iceberg-on-HMS are. + +--- + +## TODO (each = independent dormant commit; re-verify line #s at edit) + +- [ ] **S1** `partition_values` plugin arm (edits A–E + `getNameToPartitionValues` SPI) + unit/regression + test against paimon/iceberg (live). checkstyle + import-gate. +- [ ] **S4** auto-analyze per-table gate (`supportsColumnAutoAnalyze` → `hasScanCapability`; hive emits + per-table marker, drops connector-wide flag). Smallest; do early. +- [ ] **S2** `hudi_meta` connector-driven (neutral metadata-rows SPI + `HudiConnector` impl + rewrite + `hudiMetadataResult`, shed `org.apache.hudi` from fe-core). Dormant unit test for the plugin arm. +- [ ] **S3** sample-analyze full port (capability + per-table `supportsSampleAnalyze`/`isSamplingPartition` + arms + sample-capable plugin task + chunk-sizes SPI + hive marker/impl). +- [ ] (optional) soften `HiveConnector.java:206-208` comment (W6 doc-only). +- [ ] clean-room adversarial review over all seam commits; fix confirmed findings. +- [ ] update HANDOFF + this doc's checkboxes; record e2e-owed rows into execution-plan §4. + +## e2e-owed (Phase 4, do-not-drop) +partition_values over heterogeneous HMS == legacy hive rows; hudi_meta timeline rows == the 4 p2 suites; +hive ANALYZE WITH SAMPLE FULL-vs-SAMPLE stat assertions; auto-analyze admits hive/iceberg-on-HMS but not +hudi-on-HMS; W6 iceberg-on-HMS write no-CCE on bundled-AWS S3. From ba10118454affd7edf15ffb20c03cd008138a50c Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 14:03:08 +0800 Subject: [PATCH 176/334] [feat](catalog) Hive coupling-seams S1: partition_values() plugin-driven arm (dormant) The partition_values() TVF hard-cast to HMSExternalTable and gated on HMSExternalCatalog, so a flipped hms table (PluginDrivenMvccExternalTable) would throw "Catalog of type 'hms' is not allowed" at plan time and CCE at BE fetch. Mirror the already-done $partitions TVF to keep fe-core connector-agnostic: - PartitionValuesTableValuedFunction.analyzeAndGetTable: admit PluginDrivenExternalCatalog in the gate + PLUGIN_EXTERNAL_TABLE in the allowed types; add a PluginDrivenExternalTable arm (isPartitionedTable, no HMS cast); hoist getTableColumns to the common ExternalTable.getPartitionColumns(snapshot). - PluginDrivenExternalTable.getNameToPartitionValues(snapshot): new SPI method sourcing name->per-column-values from the connector's listPartitions. Kept standalone (not folded into the live getNameToPartitionItems) so the paimon/ iceberg pruning path stays byte- and cost-identical. - MetadataGenerator: add a PLUGIN_EXTERNAL_TABLE arm to partitionValuesMetadataResult; refactor the HMS and plugin paths onto a shared partitionValuesRows builder so the typed-TCell rendering (incl. HIVE_DEFAULT_PARTITION/null -> SQL NULL) is identical; drop the now-unused HiveExternalMetaCache import. Additive for paimon/iceberg (a previously-throwing TVF now works); dormant for hive until the flip. fe-core compiles, 0 checkstyle violations, import gate clean. Functional parity (paimon/iceberg live rows; hive post-flip == legacy) is e2e-owed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../datasource/PluginDrivenExternalTable.java | 41 +++++++++++++++++++ .../tablefunction/MetadataGenerator.java | 30 +++++++++++--- .../PartitionValuesTableValuedFunction.java | 32 +++++++++++---- 3 files changed, 88 insertions(+), 15 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 55d42cf23c28db..d8c07f5a28cb9b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -778,6 +778,47 @@ public Map getNameToPartitionItems(Optional return nameToPartitionItem; } + /** + * Partition display-name -> per-column partition values (in partition-column order), sourced from the + * connector's {@code listPartitions} in one round-trip (no FE-side partition-value cache, per the + * cutover's connector-owned caching). Values are extracted by the cached remote names; a value absent + * from the connector's raw map is left {@code null} (the partition_values() TVF renders it as SQL NULL, + * mirroring the legacy HMS {@code HivePartitionValues.getNameToPartitionValues()}). Empty for a + * non-partitioned table or an unresolved handle. Partition order preserved via a {@link LinkedHashMap}. + * + *

    Deliberately separate from {@link #getNameToPartitionItems} (not a shared helper) so that live + * path stays byte- and cost-identical for paimon/iceberg — a shared name-keyed map would collapse the + * pathological duplicate-partition-name case that the parallel-list build there tolerates. + */ + public Map> getNameToPartitionValues(Optional snapshot) { + if (getPartitionColumns().isEmpty()) { + return Collections.emptyMap(); + } + List remoteNames = getSchemaCacheValue() + .map(value -> ((PluginDrivenSchemaCacheValue) value).getPartitionColumnRemoteNames()) + .orElse(Collections.emptyList()); + + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + if (!handleOpt.isPresent()) { + return Collections.emptyMap(); + } + List partitions = + metadata.listPartitions(session, handleOpt.get(), Optional.empty()); + Map> nameToValues = Maps.newLinkedHashMap(); + for (ConnectorPartitionInfo partition : partitions) { + List values = new ArrayList<>(remoteNames.size()); + for (String remoteName : remoteNames) { + values.add(partition.getPartitionValues().get(remoteName)); + } + nameToValues.put(partition.getPartitionName(), values); + } + return nameToValues; + } + @Override public long getCachedRowCount() { // Do NOT call makeSureInitialized() here. diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java index fe0caf25f45351..270d23e463cf76 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java @@ -67,11 +67,12 @@ import org.apache.doris.datasource.ExternalTable; import org.apache.doris.datasource.InternalCatalog; import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.TablePartitionValues; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalTable; -import org.apache.doris.datasource.hive.HiveExternalMetaCache; import org.apache.doris.datasource.metacache.MetaCacheEntryStats; +import org.apache.doris.datasource.mvcc.MvccSnapshot; import org.apache.doris.datasource.mvcc.MvccUtil; import org.apache.doris.job.common.JobType; import org.apache.doris.job.extensions.insert.streaming.AbstractStreamingTask; @@ -2092,6 +2093,10 @@ private static TFetchSchemaTableDataResult partitionValuesMetadataResult(TMetada dataBatch = partitionValuesMetadataResultForHmsTable((HMSExternalTable) table, params.getColumnsName()); break; + case PLUGIN_EXTERNAL_TABLE: + dataBatch = partitionValuesMetadataResultForPluginTable((PluginDrivenExternalTable) table, + params.getColumnsName()); + break; default: return errorResult("not support table type " + tableType.name()); } @@ -2107,7 +2112,23 @@ private static TFetchSchemaTableDataResult partitionValuesMetadataResult(TMetada private static List partitionValuesMetadataResultForHmsTable(HMSExternalTable tbl, List colNames) throws AnalysisException { - List partitionCols = tbl.getPartitionColumns(); + Map> valuesMap = tbl.getHivePartitionValues( + MvccUtil.getSnapshotFromContext(tbl)).getNameToPartitionValues(); + return partitionValuesRows(tbl.getPartitionColumns(), colNames, valuesMap, tbl.getName()); + } + + // A flipped hms table (and paimon/iceberg) is a PluginDrivenExternalTable, not an HMSExternalTable; the + // partition values come from the connector's listPartitions via the generic SPI, then feed the same row + // builder as the HMS path (identical typed-TCell rendering, including HIVE_DEFAULT_PARTITION -> NULL). + private static List partitionValuesMetadataResultForPluginTable(PluginDrivenExternalTable tbl, + List colNames) throws AnalysisException { + Optional snapshot = MvccUtil.getSnapshotFromContext(tbl); + Map> valuesMap = tbl.getNameToPartitionValues(snapshot); + return partitionValuesRows(tbl.getPartitionColumns(snapshot), colNames, valuesMap, tbl.getName()); + } + + private static List partitionValuesRows(List partitionCols, List colNames, + Map> valuesMap, String tableName) throws AnalysisException { List colIdxs = Lists.newArrayList(); List types = Lists.newArrayList(); for (String colName : colNames) { @@ -2120,12 +2141,9 @@ private static List partitionValuesMetadataResultForHmsTable(HMSExternalTa } if (colIdxs.size() != colNames.size()) { throw new AnalysisException( - "column " + colNames + " does not match partition columns of table " + tbl.getName()); + "column " + colNames + " does not match partition columns of table " + tableName); } - HiveExternalMetaCache.HivePartitionValues hivePartitionValues = tbl.getHivePartitionValues( - MvccUtil.getSnapshotFromContext(tbl)); - Map> valuesMap = hivePartitionValues.getNameToPartitionValues(); List dataBatch = Lists.newArrayList(); for (Map.Entry> entry : valuesMap.entrySet()) { TRow trow = new TRow(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java index 12e27c1c7f9402..d0668b55cea1cc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/PartitionValuesTableValuedFunction.java @@ -25,8 +25,12 @@ import org.apache.doris.common.ErrorCode; import org.apache.doris.common.MetaNotFoundException; import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.hive.HMSExternalTable; +import org.apache.doris.datasource.mvcc.MvccUtil; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.exceptions.AnalysisException; import org.apache.doris.qe.ConnectContext; @@ -110,7 +114,8 @@ public static TableIf analyzeAndGetTable(String catalogName, String dbName, Stri throw new AnalysisException("can not find catalog: " + catalogName); } // disallow unsupported catalog - if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog)) { + if (!(catalog.isInternalCatalog() || catalog instanceof HMSExternalCatalog + || catalog instanceof PluginDrivenExternalCatalog)) { throw new AnalysisException(String.format("Catalog of type '%s' is not allowed in ShowPartitionsStmt", catalog.getType())); } @@ -122,19 +127,27 @@ public static TableIf analyzeAndGetTable(String catalogName, String dbName, Stri TableIf table; try { table = db.get().getTableOrMetaException(tableName, TableType.OLAP, - TableType.HMS_EXTERNAL_TABLE, TableType.MAX_COMPUTE_EXTERNAL_TABLE); + TableType.HMS_EXTERNAL_TABLE, TableType.MAX_COMPUTE_EXTERNAL_TABLE, + TableType.PLUGIN_EXTERNAL_TABLE); } catch (MetaNotFoundException e) { throw new AnalysisException(e.getMessage(), e); } - if (!(table instanceof HMSExternalTable)) { - throw new AnalysisException("Currently only support hive table's partition values meta table"); + // A flipped hms table is a PluginDrivenExternalTable, not an HMSExternalTable; both are served + // via their common partition SPI below, mirroring the $partitions TVF (PartitionsTableValuedFunction). + if (table instanceof HMSExternalTable) { + if (!((HMSExternalTable) table).isPartitionedTable()) { + throw new AnalysisException("Table " + tableName + " is not a partitioned table"); + } + return table; } - HMSExternalTable hmsTable = (HMSExternalTable) table; - if (!hmsTable.isPartitionedTable()) { - throw new AnalysisException("Table " + tableName + " is not a partitioned table"); + if (table instanceof PluginDrivenExternalTable) { + if (!((PluginDrivenExternalTable) table).isPartitionedTable()) { + throw new AnalysisException("Table " + tableName + " is not a partitioned table"); + } + return table; } - return table; + throw new AnalysisException("Currently only support hive table's partition values meta table"); } @Override @@ -167,7 +180,8 @@ public List getTableColumns() throws AnalysisException { Preconditions.checkNotNull(table); // TODO: support other type of sys tables if (schema == null) { - List partitionColumns = ((HMSExternalTable) table).getPartitionColumns(); + List partitionColumns = ((ExternalTable) table).getPartitionColumns( + MvccUtil.getSnapshotFromContext(table)); schema = Lists.newArrayList(); for (Column column : partitionColumns) { schema.add(new Column(column)); From d0291e501bcafb64530baedb14db895b351e1dbb Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 14:06:29 +0800 Subject: [PATCH 177/334] [doc](catalog) Hive coupling-seams: S1 landed; HANDOFF -> S4/S2/S3 + review Record S1 (partition_values plugin arm, 166515cdc88) done in the step design; rewrite HANDOFF for the next session: W6 verified false-gap, the 3 parity decisions locked, and the remaining seams (S4 auto-analyze per-table gate with its delegation-capability investigation, S2 hudi_meta connector-driven, S3 sample-analyze full port) + clean-room review. Also flags the 3 still-open D2 cache-review decisions as decoupled/pending. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 84 +++++++++---------- ...e-coupling-seams-step-design-2026-07-10.md | 13 ++- 2 files changed, 51 insertions(+), 46 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 224925edfbce52..ea743b5c7bf7ff 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -9,68 +9,66 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： -- **Phase 0 连接器休眠补齐** ✅ **DONE** — 读 SPI + iceberg/hudi 兄弟委派（S0–S6 / W1–W5 / WC1–WC4）+ 整条 hudi 线（HD-A0…HD-D1 + 武装 pivot HD-B2）。 -- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) → 事件管道 Model B → 3 个 loud-break 耦合缝 → W6 写路径 TCCL 验证。 -- **Phase 2 原子翻闸**（`SPI_READY += hms` + 3 处 GSON 重映射 + 死臂/INC-5/D5 删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门（R-002）**。 - -**✅ 本轮完成 = Phase 1 首项「Hive 连接器自持缓存(D2)」6 个休眠 commit + 净室对抗复审 + 2 处复审修复**： -D2 = `f742651990d` C-a(pom+Caffeine) → `4fe55d88fab` C-b(`CachingHmsClient` 装饰器) → `7b05df6e55e` C-c(接线进 client) → `7c0ee1ffb2a` C-d(`HiveFileListingCache`) → `7bf90a7fe3c` C-e(`invalidateTable/invalidateAll`) → `12e0c9177c2` C-f(fe-core `getNewestUpdateVersionOrTime` last-modified 分支)。设计 = **`tasks/hive-connector-cache-step-design-2026-07-10.md`**。全休眠（hms 未进 `SPI_READY_TYPES`；paimon/iceberg/jdbc/hudi 字节不变；对齐 Trino 双层）。 -**净室对抗复审**（9 维独立盲评 → 逐条对抗验证 → 交叉核对历史结论）= 7 疑点 2 坐实、5 误报/干净；报告 = **`plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md`**。两处坐实（用户拍板即修，均已落地）：`fda344e6022`（恢复 `FileSystem.get` 系统性失败大声报错、仅单目录失败 skip——修静默空结果回归）+ `cdc837563a7`（新增通用 `Connector.invalidateDb` SPI + 接 `RefreshManager.refreshDbInternal`，让 REFRESH DATABASE 清连接器两层缓存）。复审确认干净项：§2.6 单调性=与旧实现平价（删最新分区致 max 下降在新旧都抛，非新回归）、无空指针面、装饰器 pass-through 完整、TCCL 只在调用线程、Caffeine 单版本。**未做的低危建议**：`fe-connector-hms/pom.xml` 把 `fe-connector-cache` 标 `optional`（免那个空 Caffeine jar 流进 hudi 包，现无害但去掉未来隐患）——待用户点头。 - -**⚠ 第二次独立净室复审**（另一 session 同日同法复跑，14 疑点 12 坐实；差量报告 = **`plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md`**，与首审一致处不重复）**多抓 3 个坐实缺陷 + 1 个线上活性 bug**： -- ✅ 已修 `30ecd930319`：REFRESH 三钩子不转发给已构建的 iceberg/hudi 兄弟连接器（iceberg 快照 pin 是 access 过期、持续查询下永不失效 → REFRESH 是唯一逃生口却够不着 → staleness 无界）。修法镜像 `close()`。 -- ❌ **未修-待拍板 #1**（review2 §2.2，HIGH）：Doris 自发 `DROP TABLE`/`CREATE TABLE`/`DROP DATABASE` 不清连接器缓存 → 同名删建后按旧表 schema/location 出错误结果最长 24h（Trino 装饰器写后自失效；旧实现 unregisterTable 即清）。修法选项：插件侧自失效（休眠、推荐）vs fe-core DDL 路径接 `connector.invalidateTable/Db`（顺带修下条 live bug）vs 两者都做。 -- ❌ **未修-待拍板 #2**（review2 §2.3，MEDIUM）：分区缓存容量 100000 的**单位悄变**——旧=分区对象数，新=请求列表条数（无 weigher，重叠请求重复驻留）→ 高分区表+滑动谓词可致堆增长远超旧界。修法选项：共享 `fe-connector-cache` 框架加可选 weigher（Trino 同款，但破坏本系列「paimon/iceberg 字节不变」声明）vs 缩小默认值 vs 只记档。 -- ❌ **未修-待拍板 #3**（review2 §3，**线上活性 bug 非休眠**）：paimon/iceberg `latestSnapshotCache`（+`PaimonSchemaAtMemo` 窄化变体）今天就有同形状的删建重名洞（fe-core DDL 路径从不 invalidate、key=纯表名）。修=独立线（动 live 行为，须自带复审+回归），或记档延后。 -- 测试加固欠账（LOW，代码正确仅测试盲区）：`createClient` 真包裹无 pin、公开 invalidate 钩子未用已构建 client 驱动（review2 §2.4）。 - -**⭐ 状态（本轮，2026-07-10 晚）= Phase 1 第二项「事件管道 Model B」全部落地 + 净室对抗复审 + 2 处复审修复，✅ DONE。** 上面 3 项 D2 复审决策仍待用户拍板（与事件管道解耦，可并行）。 -- 权威设计 = **`tasks/hive-event-pipeline-step-design-2026-07-10.md`**（**起步必读**；含用户签字 A=结构性事件即时重建平价 + 6 步序列 + 落地/复审记录 §6 + 侦察 `wf_0c686fb4-d9d` / 复审 `wf_0d49c409-a86`）。**SUPERSEDES** `hms-event-pipeline-findings-2026-07-07.md`（旧侦察，行号已漂）。 -- 已落地（全休眠：hms 未 SPI_READY；paimon/iceberg/jdbc/hudi 字节不变；各步 test-compile SUCCESS + checkstyle 0 + import gate ok）：`0214f04` **E-a** 中立事件 SPI（`MetastoreChangeDescriptor`/`ConnectorEventSource`/`getEventSource`/`invalidatePartition`）→ `b13ed79` **E-b** 插件取数（`HmsClient` 通知方法 + `HmsNotificationEvent`）→ `902546d` **E-c** 插件事件源（`HmsEventParser` 忠实映射 + `HmsEventSource` 角色感知 + `HiveConnector` 接线）→ `3552554` **E-d** fe-core mutator 泛化（de-cast + `registerDatabase` 上提 PluginDriven + 分区 PluginDriven 分支）→ `6a96820` **E-e** fe-core 驱动（`MetastoreEventSyncDriver` + Env 接线）→ `9113c51` **E-f** 休眠 parser 单测（6 绿）。 -- **净室复审**（`wf_0d49c409-a86`，6 维 refute-by-default）14 疑点、4 坐实=**3 真回归（均休眠 pre-flip、须翻闸前修）**：**① 自愈丢失(poison event 死锁)** 已修 `fb21498`（`HmsEventSource` 瞬时 fetch 错误原地重试`ofNothing`、驱动 `realRun` catch 把游标归 -1 → 下轮 first-pull 全刷跳过毒事件；`applyDescriptors` 不再回退一格）；**② eager gzip 解压** 已修 `134907b`（`prepareBody` 按 `needsBody(type)` 惰性）；**③ `isInitialized()` gate**（主从只读分离下漏 seed 游标 → 从节点静默停更）**保留**（去掉会 force-init 空闲 paimon/iceberg/jdbc 破字节不变）+ 改诚实注释 + 列翻闸项。10 条误报（大小写canonical enum / 空分区 / enable-gate 翻闸自owed 等）已 refute。 -- **⭐ 下一步 = Phase 1 剩余项**（权威=execution plan §2.3 + 设计稿）：**3 个 loud-break 耦合缝**（`partition_values()` TVF / `hudi_meta()` TIMELINE TVF / `canSample`+`SUPPORTS_SAMPLE_ANALYZE`，均加休眠 arm，翻闸后否则 loud-break）→ **W6 写路径 TCCL 验证**（`fe-connector-hive` 无 `TcclPinningConnectorContext`，iceberg-on-HMS 写按名反射有 CCE 风险；verify-first）。之后 → **Phase 2 原子翻闸**。 -- **翻闸时接线（Phase 2，非本步，勿丢）**：① 去 `MetastoreEventsProcessor:116` gate + `ExternalMetaIdMgr.replayMetaIdMappingsLog` 游标回放**改指新驱动 `getMetastoreEventSyncDriver().updateMasterLastSyncedEventId`**；② **主节点初始化 flipped 事件源目录**（复审 finding #2，否则主从只读分离静默停更；设计稿 §2.4）；③ enable 开关：靠 fe-core 全局 Config 开启的部署须补 per-catalog `hive.enable_hms_events_incremental_sync`，或翻闸时 fe-core 把 resolved 值注入属性图。 -- **⚠ 复审确认 e2e 欠账（补 execution plan §4）**：毒事件自愈断言、主从只读分离下 flipped 目录游标传播、gzip/plain 各事件类型表/分区体解析忠实度（单测只覆盖 body-free 中立路径）。合并/去重(mergeEvents)本步有意省略（批优化；拉取式+顺序应用即正确）。 -- **⚠ 并行 session 风险**：曾有两个 session 同时在此工作树（一个 amend 卷入另一个未提交测试；本轮 HANDOFF 也被并行 session 更新过）；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 - -**⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步（读 / 写-拒 / schema-evolution / 缓存 / 跨加载器委派）只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`）。**删除排序最硬约束**：`datasource/hive|hudi|iceberg/` 的 ~90 个 HMS 支撑类删不掉，直到翻闸把消费者切到连接器路径（详见 execution plan §2.4/§3 + `tasks/iceberg-on-hms-delegation-findings-2026-07-07.md`）。 +- **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 +- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → **3 个 loud-break 耦合缝 + W6（本步，进行中）** → 剩余。 +- **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 + +**⭐ 本轮（2026-07-10 晚）= Phase 1 第三项「翻闸耦合缝」开工：HEAD 对齐侦察 + 3 处用户决策(全选逐位一致) + 第一缝落地。** +权威设计 = **`tasks/hive-coupling-seams-step-design-2026-07-10.md`**（**起步必读**；含 4 缝 HEAD 锚点 + 修法 + 用户决策 + TODO + e2e 欠账）。侦察 = `wf_dfe1cb86-df4`（4 缝读者 + 完备性审查；完备性审查确认 4 缝之外无未护 loud break，另抓出 auto-analyze 静默扩大缝）。 + +- **W6 写路径 TCCL = 已核实为假警报，无需改代码**：iceberg-on-HMS 写委派穿过 iceberg 兄弟已绑定的 `TcclPinningConnectorContext`（`IcebergConnector.java:174`，线程无关）。只欠 e2e。可选：软化 `HiveConnector.java:206-208` 过度谨慎注释（纯文档）。 +- **用户 3 处决策（2026-07-10，全选 FULL PARITY）**：① 时间线函数 `hudi_meta` = **保留(改造连接器驱动)**（侦察推翻"删"倾向——4 个 p2 hudi 套件在用）；② Hive `ANALYZE … WITH SAMPLE` = **完整移植**（今天可用，不移植翻闸后会 `DdlException`）；③ 后台列自动统计 = **按表细分排除 hudi-on-HMS**（复刻旧 `HIVE||ICEBERG` 门）。 +- **✅ 已落地：S1「partition_values() 通用插件臂」** commit `166515cdc88`（3 个 fe-core 文件；镜像 `$partitions`：闸门放行 `PluginDrivenExternalCatalog`+`PLUGIN_EXTERNAL_TABLE`、加 `PluginDrivenExternalTable` 臂无 HMS 强转、`getTableColumns` 上提基类；新增 SPI `PluginDrivenExternalTable.getNameToPartitionValues`（**独立方法**、不动 live `getNameToPartitionItems` 保 paimon/iceberg 字节+成本不变）；`MetadataGenerator` 加 `PLUGIN_EXTERNAL_TABLE` 臂 + HMS/插件共用 `partitionValuesRows` 构建器保 `HIVE_DEFAULT_PARTITION`→NULL）。**对 paimon/iceberg 是加法(此前抛错的 TVF 现可用)**、对 hive 休眠至翻闸。fe-core BUILD SUCCESS + 0 checkstyle + import-gate 干净。功能校验 e2e-owed。 + +**⭐ 下一步 = 本步剩余 3 缝 + 收尾（权威=`hive-coupling-seams-step-design-2026-07-10.md` TODO）：** +1. **S4 auto-analyze 按表门**（最小，但**先侦察**）：`PluginDrivenExternalTable.supportsColumnAutoAnalyze()` 由连接器级 `getCapabilities().contains(SUPPORTS_COLUMN_AUTO_ANALYZE)` 改为 `hasScanCapability(...)`（对 native iceberg/paimon 无变——它们仍连接器级声明）；hive 连接器从 `getCapabilities` 的 EnumSet 去掉该 flag，改在 `HiveConnectorMetadata.getTableSchema` 按表发标记（`PER_TABLE_CAPABILITIES_KEY`，precedent `:402-414` 的 TOPN 标记）。**隐藏深度须先查清**：翻闸后 iceberg-on-HMS / hudi-on-HMS 表的 `SUPPORTS_COLUMN_AUTO_ANALYZE` 是从 **hive 连接器**解析还是从**委派的兄弟连接器**解析？旧门 admit `dlaType==ICEBERG`。若 iceberg-on-HMS 已经从 iceberg 兄弟(连接器级声明)拿到，则 hive 只需对 plain-hive 发标记、对 hudi 不发，"排除 hudi"就落到"hudi 兄弟是否声明该 flag"。**动码前先在真实代码里把这个解析路径查清**（`HiveConnector` 的 3-way 路由 / 兄弟委派）。 +2. **S2 hudi_meta 连接器驱动**（保留）：加中立"元数据行"SPI（镜像 `ConnectorProcedureResult` 行返回）+ `HudiConnector` 实现（时间线数据已在连接器侧：`HudiMetaClientExecutor`/`getActiveTimeline().getInstants()`）；重写 `MetadataGenerator.hudiMetadataResult`（gate 改通用插件/能力型、去 `HMSExternalTable` 强转、fe-core 摆脱 `org.apache.hudi` timeline import `:128-129`）。休眠。委派须 pin TCCL。parity = 4 个 p2 套件行(e2e)。旧 body 无论如何删除步移除。 +3. **S3 sample-analyze 完整移植**（最大）：新 `ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE`（hive **按表**发给 plain-hive 表——旧 `dlaType==HIVE`，排除 iceberg/hudi-on-HMS；native iceberg/paimon 不发保持现拒绝）+ `AnalysisManager.canSample`/`AnalyzeTableCommand.isSamplingPartition` 加 `PluginDrivenExternalTable.supportsSampleAnalyze()` 臂（走 `hasScanCapability`）+ `PluginDrivenExternalTable.createAnalysisTask` 返回可抽样任务(移植 `HMSAnalysisTask.doSample`+`getSampleInfo`+`needLimit`) + `getChunkSizes` override(经新 `Connector` chunk-sizes SPI 拿原始字节长度，type-math 留 fe-core)。非休眠单测端到端(发真抽样 SQL)→ e2e-owed。铁律：能力**按表**(连接器级会误 admit iceberg/hudi-on-HMS)。 +4. **收尾**：（可选）软化 W6 注释；**净室对抗复审**跨全部 seam commit（memory `clean-room-adversarial-review-pref`）；把 e2e 欠账登记进 execution-plan §4；勾选设计文档 TODO；更新本 HANDOFF。 + +**⚠ 与本步解耦、仍待用户拍板的 3 处 D2 缓存复审项**（上一步遗留，见旧 HANDOFF/review2 报告；本轮未动）：① 自发 DROP/CREATE TABLE 不清连接器缓存(HIGH)；② 分区缓存容量单位悄变(MEDIUM)；③ paimon/iceberg `latestSnapshotCache` 现网同形删建洞(线上活性 bug)。下个 session 可择机连同上面 seam 一起或单独找用户定。 + +**⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步(读/写-拒/schema-evolution/缓存/跨加载器委派/本步 4 缝)只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + `hive-coupling-seams-step-design` e2e-owed 段 + memory `hms-iceberg-delegation-needs-e2e`）。删除排序最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 + +**⚠ 并行 session 风险**：曾有两个 session 同工作树互相 amend；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 --- # 🧠 起步必读（读文档，别炒 git log 历史） -1. **权威翻闸计划** = `tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。**SUPERSEDES** 07-07 doc §4/§5 的状态。 -2. **本步（Hive 缓存）设计** = `tasks/hive-connector-cache-step-design-2026-07-10.md`（TODO 全勾）。 -3. **补充权威**：`tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + 能力孪生 + 已签字 §6 D1–D6 决策，仍有效）；`tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 S0–S6 + CCE/TCCL 硬约束）；两份 findings（`iceberg-on-hms-delegation-findings-2026-07-07.md` + `hms-event-pipeline-findings-2026-07-07.md`）；hudi = `tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `tasks/hudi-schema-evolution-step-design-2026-07-09.md`（均 DONE）。 -4. **样板**：`tasks/P5-paimon-migration.md`（翻闸+删 legacy 全流程）、`tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 -5. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 +1. **权威翻闸计划** = `tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。 +2. **本步（翻闸耦合缝）设计** = `tasks/hive-coupling-seams-step-design-2026-07-10.md`（4 缝 + 3 决策 + TODO；S1 已勾）。**起步先读，再对照 HEAD 核行号**。 +3. 已完成的前两项 Phase-1 步：`tasks/hive-connector-cache-step-design-2026-07-10.md`（D2 缓存）、`tasks/hive-event-pipeline-step-design-2026-07-10.md`（事件管道）。 +4. **补充权威**：`tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + §6 D1–D6 决策）；`tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 + CCE/TCCL 硬约束）；hudi = `tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `tasks/hudi-schema-evolution-step-design-2026-07-09.md`。 +5. **样板**：`tasks/P5-paimon-migration.md`、`tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 +6. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 --- # 📦 分支 / Commit 须知 -- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi`）。PR base = `branch-catalog-spi`，**squash 合并**。打包/复审策略（单 PR vs 分 PR）= 翻闸阶段的开放决策。 +- **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi`）。PR base = `branch-catalog-spi`，**squash 合并**。 - **公开 tracking issue = apache/doris#65185**（进度按已合入 `branch-catalog-spi` PR 口径）。 -- **⚠️ path-whitelist `git add`，严禁 `git add -A`**：工作树有大量历史遗留 scratch（`*.bak` / `regression-test/conf/regression-conf.groovy` 明文 key / `.audit-scratch/` / `conf.cmy/` / `META-INF/` / `docker/...` / `plan-doc/reviews/P5-*` / `.claude/` / `failed-cases.out`——**非本线程产物，勿混入任何 commit**）。 +- **⚠️ path-whitelist `git add`，严禁 `git add -A`**：工作树有大量历史遗留 scratch（`*.bak` / `regression-conf.groovy` 明文 key / `.audit-scratch/` / `conf.cmy/` / `META-INF/` / `docker/...` / `plan-doc/reviews/P5-*` / `.claude/` / `failed-cases.out`——**非本线程产物，勿混入任何 commit**）。 - commit message：`[feat|fix|doc](catalog) …` + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每子步/每条 fix = 独立 commit**；HANDOFF + 设计文档单独 commit（与 code 分开）。上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 # ⚙️ 操作须知（构建/测试，复用） -- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-connector/ **-am** test -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**；.m2 里有坏 `${revision}` 目录）。单测加 `-Dtest=`。禁 build-cache 才真跑 surefire。 -- **checkstyle 单独跑**：`mvn -o ... -pl fe-connector/ -am checkstyle:check`（module `test` 阶段已含 checkstyle，BUILD SUCCESS 即过；仍建议显式确认「0 violations」——Rule 12）。 -- **import gate**（连接器禁 import fe-core）：`bash tools/check-connector-imports.sh`（**repo 根**跑）。**HMS `HiveVersionUtil` 命中 = 误报非违规**（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 -- **⚠️ bash 工具默认 timeout 120s**：fe-core 全量编译 ~2–3min、`-am` 更久 → `timeout` 调到 ~580000+ 或后台跑；**后台/管道 exit 不可信**——读 LOG 的 `BUILD SUCCESS` + surefire `Tests run=/Failures=/Errors=`（memory `doris-build-verify-gotchas`）。 -- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`Mockito.mock(接口)` **不跑 default 方法**返 null；`mockStatic(Env)` 是本仓惯用法）。checkstyle **禁 static import**、**扫 test 源**（memory `catalog-spi-fe-core-test-infra`）。 -- **cwd 会被 harness 重置** → 一律绝对路径。**⚠️ `/mnt/disk1` 盘紧（~82%）；勿用 worktree 隔离编译 agent**（复制整仓，盘不够）。 +- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**）。连接器：`-pl fe-connector/ -am`。单测加 `-Dtest=`。 +- **验证信 LOG 不信 exit**：`-q` + `| tail` 会吞掉 `BUILD SUCCESS` 且管道 exit=tail 的(不可信)；改**重定向到文件**跑（不加 `-q`），再 grep `BUILD SUCCESS`/`[ERROR].*\.java:`/checkstyle `You have N violations`（memory `doris-build-verify-gotchas`）。test-compile 的 `validate` 阶段已跑 checkstyle（本轮见 fe-core `0 violations`）。 +- **build 中 ANTLR `mismatched input '->'`/`super::` 噪音 = 非致命**（某 codegen/工具 Java 语法解析器不识 lambda/方法引用；本轮 BUILD SUCCESS 照出）。 +- **import gate**：`bash tools/check-connector-imports.sh`（repo 根跑）。HMS `HiveVersionUtil` 命中 = 误报（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 +- **⚠️ bash 默认 timeout 120s**：fe-core 全量编译 ~3min+ → 后台跑 + 读 LOG。**⚠️ `/mnt/disk1` 盘紧(~82%)；勿用 worktree 隔离编译 agent**。cwd 会被重置 → 绝对路径。 +- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`mock(接口)` 不跑 default 方法返 null；`mockStatic(Env)` 惯用）。checkstyle 禁 static import、扫 test 源（memory `catalog-spi-fe-core-test-infra`）。 -# 🔒 铁律（fe-core 约束，翻闸靠"表类=通用类 + 网关按句柄委派"） +# 🔒 铁律（fe-core 约束） -- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点保持 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。 +- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。能力按表判要用 `hasScanCapability`/`PER_TABLE_CAPABILITIES_KEY`（连接器发标记，fe-core 不看格式/dlaType）。 - fe-core **不解析属性**（storage→fe-filesystem、meta→fe-connector，均插件侧；memory `catalog-spi-no-property-parsing-in-fecore`）。 -- 跨插件/跨边界**须 pin TCCL** 到连接器 classloader（扫描线程 / 写-DDL 引擎线程 / iceberg 内部 worker 池 / 事件轮询后台线程；memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 +- 跨插件/跨边界**须 pin TCCL** 到连接器 classloader（扫描线程/写-DDL 引擎线程/iceberg worker 池/事件轮询后台线程；memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 - `history_schema_info` 嵌套字段名逐层 lowercase（否则 BE SIGABRT；memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 -- `PluginDrivenMvccExternalTable` 是 paimon/iceberg **实时**基类：改其共享方法须对二者字节+成本双不变（memory `plugindriven-mvcc-table-is-live-not-dormant`）。 +- `PluginDrivenMvccExternalTable`/`PluginDrivenExternalTable` 是 paimon/iceberg **实时**基类：改其共享方法须对二者字节+成本双不变（本轮 S1 即因此把 `getNameToPartitionValues` 做成独立方法、不动 `getNameToPartitionItems`；memory `plugindriven-mvcc-table-is-live-not-dormant`）。 # 🗂 memory 相关项 -`handoff-discipline-per-phase` · `clean-room-adversarial-review-pref` · `ask-user-explain-in-chinese-first` · `session-handoff-at-30pct-context` · `doris-build-verify-gotchas` · `catalog-spi-fe-core-test-infra` · `catalog-spi-tracking-issue` · `hms-iceberg-delegation-needs-e2e` · `catalog-spi-connector-cache-framework-caffeine-coherence` · `memory-keep-only-general-or-requested`。 +`handoff-discipline-per-phase` · `clean-room-adversarial-review-pref` · `ask-user-explain-in-chinese-first` · `session-handoff-at-30pct-context` · `doris-build-verify-gotchas` · `catalog-spi-fe-core-test-infra` · `catalog-spi-plugindriven-no-source-specific-code` · `plugindriven-mvcc-table-is-live-not-dormant` · `catalog-spi-tracking-issue` · `hms-iceberg-delegation-needs-e2e` · `concurrent-sessions-shared-worktree-hazard` · `memory-keep-only-general-or-requested`。 diff --git a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md index e81d87c48c8fea..cffe20df6c2d80 100644 --- a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md +++ b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md @@ -131,10 +131,17 @@ keep the connector-wide flag → unchanged. e2e-owed: hudi-on-HMS NOT auto-analy ## TODO (each = independent dormant commit; re-verify line #s at edit) -- [ ] **S1** `partition_values` plugin arm (edits A–E + `getNameToPartitionValues` SPI) + unit/regression - test against paimon/iceberg (live). checkstyle + import-gate. +- [x] **S1** `partition_values` plugin arm (edits A–E + `getNameToPartitionValues` SPI). ✅ commit + `166515cdc88`. fe-core BUILD SUCCESS + 0 checkstyle + import-gate clean. Functional test (paimon/ + iceberg live rows; hive post-flip == legacy) = e2e-owed. **Additive for paimon/iceberg**, dormant hive. - [ ] **S4** auto-analyze per-table gate (`supportsColumnAutoAnalyze` → `hasScanCapability`; hive emits - per-table marker, drops connector-wide flag). Smallest; do early. + per-table marker, drops connector-wide flag). ⚠ **INVESTIGATE FIRST** (hidden depth): post-flip do + iceberg-on-HMS / hudi-on-HMS tables resolve `SUPPORTS_COLUMN_AUTO_ANALYZE` from the HIVE connector or + from their delegated sibling connector? Legacy admitted `dlaType==ICEBERG` too. If iceberg-on-HMS + already gets it via the iceberg sibling's connector-wide flag, then hive only needs to emit the marker + for plain-hive (and NOT hudi), and the "exclude hudi" fix reduces to: does the hudi sibling declare + it? Nail this before editing. Per-table marker precedent = `HiveConnectorMetadata.java:402-414` + (`SUPPORTS_TOPN_LAZY_MATERIALIZE`). - [ ] **S2** `hudi_meta` connector-driven (neutral metadata-rows SPI + `HudiConnector` impl + rewrite `hudiMetadataResult`, shed `org.apache.hudi` from fe-core). Dormant unit test for the plugin arm. - [ ] **S3** sample-analyze full port (capability + per-table `supportsSampleAnalyze`/`isSamplingPartition` From 56a48bcdd949d55967a9dee803042a6c74b92338 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 15:19:06 +0800 Subject: [PATCH 178/334] [feat](catalog) Hive coupling-seams S4: auto-analyze per-table gate + sibling-cap reflection (dormant) Background column auto-analyze eligibility was resolved connector-wide: HiveConnector declared SUPPORTS_COLUMN_AUTO_ANALYZE, so at the flip EVERY hms table (plain-hive, iceberg-on-HMS, hudi-on-HMS) would be admitted -- over-admitting hudi-on-HMS, which legacy StatisticsUtil.supportAutoAnalyze excluded (it admitted only dlaType HIVE || ICEBERG). Recon confirmed a flipped table always resolves capability from the CATALOG (HIVE) connector, never the delegated iceberg/hudi sibling, so simply dropping the hive flag would also silently regress iceberg-on-HMS. Make the gate per-table (connector-agnostic), the way Top-N lazy / nested-column prune already are: - PluginDrivenExternalTable.supportsColumnAutoAnalyze() -> hasScanCapability( SUPPORTS_COLUMN_AUTO_ANALYZE). Byte/cost-invariant for native iceberg/paimon: hasScanCapability checks the connector-wide set FIRST, which still contains the flag for them, so it short-circuits before touching the per-table marker. - HiveConnector.getCapabilities(): drop SUPPORTS_COLUMN_AUTO_ANALYZE from the connector-wide EnumSet (this is what de-admits hudi-on-HMS). - HiveConnectorMetadata.getTableSchema plain-hive branch: emit the marker per-table for every plain-hive data table (any format, view excluded) = legacy dlaType==HIVE. - HiveConnectorMetadata.getTableSchema delegation branch (Option C, user-chosen full- parity): reflect the owning sibling connector's connector-wide capability set onto the delegated (iceberg/hudi-on-HMS) schema as a per-table marker, so a scan capability the sibling declares connector-wide survives delegation (fe-core's hasScanCapability never reads the sibling directly). Mirrors Trino table-redirection, where the redirected-to connector's capabilities govern the table. This restores iceberg-on-HMS auto-analyze AND closes the same-root-cause Top-N / nested-column-prune loss for iceberg-on-HMS in one place; hudi-on-HMS (sibling declares neither) is correctly withheld. Only hasScanCapability consumers read the marker, so reflecting non-scan capabilities (view / show-create / mvcc) is inert. Dormant until hms enters SPI_READY_TYPES (StatisticsUtil.supportAutoAnalyze keeps its live HMSExternalTable arm). Tests: invert HiveConnectorCapabilitiesTest (auto-analyze is now per-table, not connector-wide); add per-table-marker + delegation-reflection coverage to PluginDrivenExternalTableTest / HiveConnectorMetadataSchemaTest / HiveConnectorMetadataSiblingDelegationTest. fe-core + fe-connector-hive BUILD SUCCESS, 0 checkstyle, import gate clean, all four suites green. Post-flip parity is e2e-owed (auto-analyze admits hive + iceberg-on-HMS, excludes hudi-on-HMS). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/connector/hive/HiveConnector.java | 22 +++--- .../connector/hive/HiveConnectorMetadata.java | 70 ++++++++++++++++++- .../hive/HiveConnectorCapabilitiesTest.java | 11 +-- .../hive/HiveConnectorMetadataSchemaTest.java | 54 ++++++++++++-- ...onnectorMetadataSiblingDelegationTest.java | 60 ++++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 20 +++--- .../PluginDrivenExternalTableTest.java | 13 ++++ 7 files changed, 217 insertions(+), 33 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 30e80405dd94ff..9966f390c77024 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -241,12 +241,6 @@ public Set getCapabilities() { // consults viewExists, routes a view DROP to dropView, and merges listViewNames into SHOW TABLES — // hive returns an EMPTY listViewNames (its listTableNames already includes views), so the merge is a // no-op and each view is listed once (legacy parity). - // - SUPPORTS_COLUMN_AUTO_ANALYZE: legacy StatisticsUtil.supportAutoAnalyze admitted HMS tables of - // dlaType HIVE and ICEBERG (NOT hudi) into the background per-column FULL auto-analyze (sample is - // unimplemented for external SQL-driven tables); this capability replaces that instanceof- - // HMSExternalTable arm. It is connector-wide, so at the flip it also admits hudi-on-HMS tables that - // legacy excluded — a residual for the iceberg/hudi delegation substep to gate per-handle or - // explicitly accept (there is no per-table escape for it today, unlike Top-N). // - SUPPORTS_METADATA_PRELOAD: legacy HMSExternalTable.supportsExternalMetadataPreload() returned true; // the capability replaces the legacy engine-name "jdbc" gate. Opt-in via enable_preload_external_ // metadata (default off), a pure lock-latency optimization with no correctness effect. @@ -269,13 +263,19 @@ public Set getCapabilities() { // PARTITIONS lists names only. // - SUPPORTS_TOPN_LAZY_MATERIALIZE: a per-table marker emitted in getTableSchema (orc/parquet only), // never a connector-wide flag. - // - SUPPORTS_NESTED_COLUMN_PRUNE: NOT emitted yet — a genuine deferral like MVCC. Legacy parquet/orc - // pruned nested columns, but the connector must first carry stable nested field-ids down its column - // tree (SlotTypeReplacer rewrites nested access to field-ids; without them BE reads NULL leaves). - // Restore it as a per-table marker once hive field-ids are verified. + // - SUPPORTS_COLUMN_AUTO_ANALYZE: legacy StatisticsUtil.supportAutoAnalyze admitted HMS tables of dlaType + // HIVE and ICEBERG (NOT hudi) into background per-column auto-analyze. A connector-wide flag here would + // over-admit hudi-on-HMS (which legacy excluded), so it is NOT declared connector-wide: getTableSchema + // emits it per-table for every plain-hive table, and the sibling-delegation branch reflects the iceberg + // sibling's connector-wide capability set onto an iceberg-on-HMS table's schema (so it survives the + // delegation) — hudi-on-HMS, whose connector declares neither, is thereby correctly withheld. + // - SUPPORTS_NESTED_COLUMN_PRUNE: NOT emitted for plain-hive yet — a genuine deferral like MVCC. Legacy + // parquet/orc pruned nested columns, but the connector must first carry stable nested field-ids down its + // column tree (SlotTypeReplacer rewrites nested access to field-ids; without them BE reads NULL leaves). + // Restore it as a per-table marker once hive field-ids are verified. (An iceberg-on-HMS table DOES get + // it, via the sibling-capability reflection above, because the iceberg sibling carries field-ids.) return EnumSet.of( ConnectorCapability.SUPPORTS_VIEW, - ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE, ConnectorCapability.SUPPORTS_METADATA_PRELOAD, ConnectorCapability.SUPPORTS_MVCC_SNAPSHOT); } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 8be8336accf699..dc9bf629324e4f 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -79,6 +79,7 @@ import java.util.Collections; import java.util.HashMap; import java.util.LinkedHashMap; +import java.util.LinkedHashSet; import java.util.List; import java.util.Locale; import java.util.Map; @@ -369,7 +370,18 @@ public Optional getTableHandle( public ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { if (!(handle instanceof HiveTableHandle)) { - return siblingMetadata(session, handle).getTableSchema(session, handle); + // An iceberg/hudi-on-HMS table's schema is built by the embedded sibling connector, but fe-core's + // PluginDrivenExternalTable.hasScanCapability only ever reads the CATALOG connector (this HIVE + // connector), never the sibling — so a per-table scan capability the sibling declares connector-wide + // (auto-analyze / Top-N lazy / nested-column prune) would be lost for the embedded table. Reflect the + // owning sibling's connector-wide capability set onto the delegated schema as a per-table marker so it + // survives delegation (mirrors Trino table-redirection, where the redirected-to connector's + // capabilities govern the table). Only hasScanCapability consumers read the marker, so a capability + // that is not per-table-refinable (view / show-create / mvcc) is inert here. Resolve the owner ONCE + // (getMetadata is not free) and reuse it for the schema build and the capability read. + Connector owner = siblingOwnerResolver.apply(handle); + ConnectorTableSchema siblingSchema = owner.getMetadata(session).getTableSchema(session, handle); + return reflectSiblingScanCapabilities(owner, siblingSchema); } HiveTableHandle hiveHandle = (HiveTableHandle) handle; String dbName = hiveHandle.getDbName(); @@ -405,6 +417,15 @@ public ConnectorTableSchema getTableSchema( // cannot express for a heterogeneous hive catalog; emit it per-table so fe-core enables the optimization // only for eligible tables and never for text/csv/json/view/hudi. List perTableCapabilities = new ArrayList<>(); + // Legacy StatisticsUtil.supportAutoAnalyze admitted EVERY plain-hive (dlaType==HIVE) table into background + // per-column auto-analyze regardless of file format. Emit it per-table for every plain-hive data table (any + // format, view excluded) so fe-core's hasScanCapability admits them WITHOUT a connector-wide flag (which + // would also admit hudi-on-HMS, which legacy excluded). This branch is reached only for a HiveTableHandle; + // an iceberg-on-HMS table is served by the delegation branch above (which reflects the iceberg sibling's + // own auto-analyze capability), and a hudi-on-HMS table's connector declares neither. + if (supportsHiveColumnAutoAnalyze(tableInfo)) { + perTableCapabilities.add(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE.name()); + } if (supportsHiveTopNLazyMaterialize(tableInfo)) { perTableCapabilities.add(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name()); } @@ -416,6 +437,41 @@ public ConnectorTableSchema getTableSchema( return new ConnectorTableSchema(tableName, allColumns, formatType, tableProperties); } + /** + * Reflects the owning sibling connector's connector-wide capability set onto a delegated (iceberg/hudi-on-HMS) + * table's schema as a per-table {@link ConnectorTableSchema#PER_TABLE_CAPABILITIES_KEY} marker, merged with any + * marker the sibling already emitted. fe-core's {@code PluginDrivenExternalTable.hasScanCapability} resolves a + * per-table scan capability from the CATALOG (hive) connector-wide set OR this marker and NEVER consults the + * sibling connector directly, so without this reflection an iceberg-on-HMS table would silently lose every scan + * capability the iceberg sibling declares connector-wide (auto-analyze / Top-N lazy / nested-column prune). + * Returns the sibling schema unchanged when the sibling declares no capabilities (e.g. a hudi sibling that + * declares none). Only per-table-refinable capabilities are ever consulted from the marker, so reflecting the + * whole set (including non-scan capabilities) is inert for the rest. + */ + private ConnectorTableSchema reflectSiblingScanCapabilities(Connector owner, ConnectorTableSchema siblingSchema) { + Set ownerCaps = owner.getCapabilities(); + if (ownerCaps.isEmpty()) { + return siblingSchema; + } + LinkedHashSet caps = new LinkedHashSet<>(); + String existing = siblingSchema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); + if (existing != null && !existing.isEmpty()) { + for (String name : existing.split(",")) { + String trimmed = name.trim(); + if (!trimmed.isEmpty()) { + caps.add(trimmed); + } + } + } + for (ConnectorCapability cap : ownerCaps) { + caps.add(cap.name()); + } + Map props = new HashMap<>(siblingSchema.getProperties()); + props.put(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, String.join(",", caps)); + return new ConnectorTableSchema(siblingSchema.getTableName(), siblingSchema.getColumns(), + siblingSchema.getTableFormatType(), props); + } + @Override public Map getProperties() { return properties; @@ -1810,6 +1866,18 @@ private boolean supportsHiveTopNLazyMaterialize(HmsTableInfo tableInfo) { return MAPRED_PARQUET_INPUT_FORMAT.equals(inputFormat) || ORC_INPUT_FORMAT.equals(inputFormat); } + /** + * Whether {@code tableInfo} is a plain-hive data table (any file format) eligible for background per-column + * auto-analyze, replicating legacy {@code StatisticsUtil.supportAutoAnalyze}'s {@code dlaType==HIVE} gate. + * Unlike {@link #supportsHiveTopNLazyMaterialize} there is NO orc/parquet restriction (legacy analyzed any hive + * format); a view is excluded (nothing to analyze) and an iceberg/hudi-on-HMS table is excluded here + * ({@code detect() != HIVE}) — iceberg-on-HMS instead inherits the capability from its sibling via + * {@link #reflectSiblingScanCapabilities}, and hudi-on-HMS is withheld. + */ + private boolean supportsHiveColumnAutoAnalyze(HmsTableInfo tableInfo) { + return !isView(tableInfo) && HiveTableFormatDetector.detect(tableInfo) == HiveTableType.HIVE; + } + /** Whether the HMS table is a view (tableType VIRTUAL_VIEW), mirroring legacy {@code HMSExternalTable.isView}. */ private static boolean isView(HmsTableInfo tableInfo) { return VIRTUAL_VIEW_TABLE_TYPE.equalsIgnoreCase(tableInfo.getTableType()); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java index d0c16b0a9818f4..f5546610ddd0bc 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java @@ -45,9 +45,6 @@ public void declaresLegacyParityCapabilities() { // Legacy resolved isView() from the remote view text; the plugin view path is gated on this. Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_VIEW), "SUPPORTS_VIEW: legacy hive views are queryable/droppable/listed"); - // Legacy StatisticsUtil.supportAutoAnalyze admitted HMS dlaType HIVE into background FULL auto-analyze. - Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE), - "SUPPORTS_COLUMN_AUTO_ANALYZE: legacy admitted HMS HIVE tables into auto-analyze"); // Legacy HMSExternalTable.supportsExternalMetadataPreload() returned true. Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_METADATA_PRELOAD), "SUPPORTS_METADATA_PRELOAD: legacy HMS tables were preload-eligible"); @@ -75,7 +72,7 @@ public void withholdsCapabilitiesWithoutSupportingSpi() { } @Test - public void topNAndNestedPruneAreNotConnectorWide() { + public void perTableScanCapabilitiesAreNotConnectorWide() { Set caps = capabilities(); // Both are per-table markers emitted in getTableSchema (orc/parquet only), never connector-wide flags, // otherwise a text/json/csv/view/hudi table would be wrongly eligible. @@ -83,5 +80,11 @@ public void topNAndNestedPruneAreNotConnectorWide() { "Top-N lazy is a per-table marker, not connector-wide"); Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE), "nested-column-prune is a per-table marker, not connector-wide"); + // SUPPORTS_COLUMN_AUTO_ANALYZE is likewise per-table (getTableSchema emits it for every plain-hive table). + // A connector-wide flag would over-admit hudi-on-HMS, which legacy StatisticsUtil.supportAutoAnalyze + // excluded (it admitted only dlaType HIVE || ICEBERG). MUTATION: re-declaring it connector-wide silently + // re-admits hudi-on-HMS -> red here. + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE), + "auto-analyze is a per-table marker (excludes hudi-on-HMS), not connector-wide"); } } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index a05fb3381345cb..4c609f2afe7b79 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.hive; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; @@ -92,6 +93,20 @@ private static String perTableCapabilities(ConnectorTableSchema schema) { return schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); } + /** Membership test over the CSV marker (order-independent, robust as more per-table capabilities are added). */ + private static boolean hasCapability(ConnectorTableSchema schema, ConnectorCapability capability) { + String csv = perTableCapabilities(schema); + if (csv == null) { + return false; + } + for (String name : csv.split(",")) { + if (name.trim().equals(capability.name())) { + return true; + } + } + return false; + } + @Test public void testPartitionColumnsPropertyEmittedWithRawNamesInOrder() { ConnectorTableSchema schema = schemaOf(partitionedTable().build()); @@ -147,17 +162,42 @@ public void testTopNLazyCapabilityMarkerEmittedForParquetAndOrc() { // The connector-wide SUPPORTS_TOPN_LAZY_MATERIALIZE cannot express that for a heterogeneous hive catalog, so // the connector emits it per-table; fe-core (PluginDrivenExternalTable.supportsTopNLazyMaterialize) enables the // optimization only for tables carrying this marker. MUTATION: not emitting it -> orc/parquet hive tables lose - // Top-N lazy materialization. - Assertions.assertEquals("SUPPORTS_TOPN_LAZY_MATERIALIZE", - perTableCapabilities(schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT).build()))); - Assertions.assertEquals("SUPPORTS_TOPN_LAZY_MATERIALIZE", - perTableCapabilities(schemaOf(unpartitionedTable(ORC_INPUT_FORMAT).build()))); + // Top-N lazy materialization. Membership (not exact CSV) because the same marker also carries auto-analyze. + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE)); + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(ORC_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE)); } @Test public void testTopNLazyCapabilityMarkerAbsentForText() { - // A text hive table is not Top-N-lazy eligible in legacy; emitting the marker would over-enable it. - Assertions.assertNull(perTableCapabilities(schemaOf(unpartitionedTable(TEXT_INPUT_FORMAT).build()))); + // A text hive table is not Top-N-lazy eligible in legacy; emitting the Top-N marker would over-enable it. + // (Auto-analyze IS emitted for it — legacy analyzed any hive format — asserted separately below.) + Assertions.assertFalse(hasCapability(schemaOf(unpartitionedTable(TEXT_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE)); + } + + @Test + public void testColumnAutoAnalyzeMarkerEmittedForEveryPlainHiveFormat() { + // WHY: legacy StatisticsUtil.supportAutoAnalyze admitted EVERY plain-hive (dlaType==HIVE) table into + // background per-column auto-analyze regardless of file format. Emitting it per-table (not connector-wide) + // is what lets fe-core exclude hudi-on-HMS (which legacy excluded) while admitting plain-hive. Unlike Top-N, + // it has NO orc/parquet restriction. MUTATION: gating it on input format -> text/csv/json hive tables + // silently drop out of auto-analyze. + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE)); + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(ORC_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE)); + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(TEXT_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE)); + } + + @Test + public void testColumnAutoAnalyzeMarkerAbsentForView() { + // A view has nothing to analyze; excluded like Top-N (isView guard before the format check). + ConnectorTableSchema schema = schemaOf( + unpartitionedTable(PARQUET_INPUT_FORMAT).tableType("VIRTUAL_VIEW").build()); + Assertions.assertFalse(hasCapability(schema, ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE)); } @Test diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 7fa8147622a663..233f47d10aef58 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorColumnStatistics; import org.apache.doris.connector.api.ConnectorMetadata; @@ -53,6 +54,7 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.EnumSet; import java.util.List; import java.util.Map; import java.util.Optional; @@ -329,6 +331,45 @@ public ConnectorTransaction beginTransaction(ConnectorSession session) { Assertions.assertTrue(siblingMetadata.calls.isEmpty(), "the sibling must not be forwarded a hive handle"); } + @Test + public void foreignHandleSchemaReflectsSiblingScanCapabilitiesAsPerTableMarker() { + // Option C: fe-core's PluginDrivenExternalTable.hasScanCapability reads only the CATALOG (hive) connector, + // never the embedded sibling — so the hive gateway must reflect the sibling's connector-wide scan + // capabilities onto the delegated schema as a per-table marker, or an iceberg-on-HMS table silently loses + // auto-analyze / Top-N lazy / nested-column prune (all of which the iceberg sibling declares connector-wide). + // MUTATION: dropping the reflection -> the returned schema carries no marker -> the embedded table drops the + // capabilities post-flip -> red here. + Set siblingCaps = EnumSet.of( + ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE, + ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE, + ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE); + HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, + handle -> new CapabilityDeclaringSiblingConnector(siblingCaps)); + + ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); + String csv = schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); + Assertions.assertNotNull(csv, "the delegated schema must carry the reflected per-table capability marker"); + List names = Arrays.asList(csv.split(",")); + Assertions.assertTrue(names.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE.name()), + "auto-analyze must survive the delegation as a per-table marker"); + Assertions.assertTrue(names.contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name()), + "Top-N lazy must survive the delegation as a per-table marker"); + Assertions.assertTrue(names.contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE.name()), + "nested-column prune must survive the delegation as a per-table marker"); + } + + @Test + public void foreignHandleSchemaUnchangedWhenSiblingDeclaresNoCapabilities() { + // A hudi sibling declares no scan capabilities -> no marker is stamped -> the embedded table stays out of + // auto-analyze etc. (hudi-on-HMS parity: legacy StatisticsUtil.supportAutoAnalyze excluded it). MUTATION: + // unconditionally stamping a marker -> hudi-on-HMS would be wrongly admitted -> red here. + HiveConnectorMetadata md = withSibling(); // RecordingSiblingConnector declares no capabilities + ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); + Assertions.assertNull(schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY), + "no marker when the sibling declares no capabilities"); + } + private static void assertThrowsMessage(Executable exec, String expectedMessage) { DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, exec); Assertions.assertEquals(expectedMessage, e.getMessage(), @@ -351,6 +392,25 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { } } + /** A sibling {@link Connector} declaring a fixed capability set; its metadata returns a marker-less schema. */ + private static final class CapabilityDeclaringSiblingConnector implements Connector { + private final Set caps; + + CapabilityDeclaringSiblingConnector(Set caps) { + this.caps = caps; + } + + @Override + public Set getCapabilities() { + return caps; + } + + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return new RecordingSiblingMetadata(); + } + } + /** Records each forwarded method name and returns distinguishable sentinels. */ private static final class RecordingSiblingMetadata implements ConnectorMetadata { static final ConnectorTableHandle SIBLING_HANDLE = new ForeignHandle(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index d8c07f5a28cb9b..afd2839b4bebae 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -215,18 +215,18 @@ public boolean connectorSupportsWriteBranch() { } /** - * Returns whether the underlying connector's tables support background per-column auto-analyze. - * The statistics auto-collector consults this (in place of the legacy {@code instanceof - * IcebergExternalTable} whitelist) to admit a flipped plugin table into the auto-analyze framework - * and to force FULL analyze (sample analyze is unimplemented for external SQL-driven tables). + * Returns whether THIS table supports background per-column auto-analyze. The statistics auto-collector + * consults this (in place of the legacy {@code instanceof IcebergExternalTable} whitelist) to admit a flipped + * plugin table into the auto-analyze framework. Resolved per-table via {@link #hasScanCapability} (not the + * connector-wide set alone) so a heterogeneous hive catalog can express the legacy + * {@code StatisticsUtil.supportAutoAnalyze} gate of {@code dlaType HIVE || ICEBERG} but NOT {@code HUDI}: a + * uniform-format connector (native iceberg/paimon) still declares it connector-wide, while hive emits it + * per-table for its plain-hive tables and reflects the iceberg sibling's connector-wide set onto an + * iceberg-on-HMS table's delegated schema — so hudi-on-HMS, whose connector declares neither, is correctly + * withheld. Mirrors {@link #supportsTopNLazyMaterialize} / {@link #supportsNestedColumnPrune}. */ public boolean supportsColumnAutoAnalyze() { - if (!(catalog instanceof PluginDrivenExternalCatalog)) { - return false; - } - Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); - return connector != null - && connector.getCapabilities().contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE); + return hasScanCapability(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE); } /** diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index 945a269350122a..bfacdfb599e743 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -379,6 +379,19 @@ public void supportsColumnAutoAnalyzeReflectsConnectorCapability() { EnumSet.of(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE)).supportsTopNLazyMaterialize()); Assertions.assertFalse(pluginTableWithCapabilities( EnumSet.noneOf(ConnectorCapability.class)).supportsColumnAutoAnalyze()); + // Auto-analyze is now resolved per-table (like Top-N / nested-prune): a heterogeneous hive catalog emits it + // via the connector.per-table-capabilities marker for its plain-hive tables (and reflects the iceberg + // sibling's set onto an iceberg-on-HMS table) even when the CATALOG connector-wide set lacks it. MUTATION: + // reverting supportsColumnAutoAnalyze() to a connector-wide-only read ignores this marker, so a flipped + // plain-hive / iceberg-on-HMS table silently drops out of auto-analyze -> red here. + Map autoAnalyzeMarker = Collections.singletonMap( + ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, + ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE.name()); + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), autoAnalyzeMarker).supportsColumnAutoAnalyze()); + // The marker is capability-specific: an auto-analyze marker must NOT enable Top-N. + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), autoAnalyzeMarker).supportsTopNLazyMaterialize()); } @Test From 285ecb4a2a5cae7e0ab22bfb29fcec6c4f5b5f3f Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 15:37:32 +0800 Subject: [PATCH 179/334] [feat](catalog) Hive coupling-seams S2: hudi_meta() connector-driven (dormant) + fe-core sheds org.apache.hudi MetadataGenerator.hudiMetadataResult gated on `instanceof HMSExternalTable` and cast to it, then reached org.apache.hudi timeline classes inline -- so post-flip a hudi table (PluginDrivenExternalTable) would return "not a hudi table", and fe-core's MetadataGenerator imported org.apache.hudi just for this method. The 4 p2 hudi_meta suites consume the TVF, so KEEP it, connector-driven (user decision, full parity): - New engine-neutral SPI ConnectorMetadata.getMetadataTableRows(session, handle, kind) -> List> (default empty; the TVF owns the fixed 4-column schema, so the connector returns only String cell values -- lighter than a schema-carrying result type). - ConnectorCapability.SUPPORTS_METADATA_TABLE; HudiConnector declares it connector-wide (every hudi table has a commit timeline). - HudiConnectorMetadata.getMetadataTableRows: iterate the FULL active timeline (all states, not the completed-only helper) under the existing metaClientExecutor (plugin auth + TCCL pin), mapping each instant to (requestedTime, action, state, completionTime) -- byte-parity with the former fe-core inline loop; fe-core adds no pin of its own. - PluginDrivenExternalTable.supportsMetadataTable() = hasScanCapability(SUPPORTS_METADATA_TABLE) and getMetadataTableRows(kind) -- NEW standalone methods (byte/cost-invariant for paimon/ iceberg, which never declare the capability and so never reach them). - MetadataGenerator.hudiMetadataResult rewritten to a dual-arm switch(table.getType()) (mirrors partitionValuesMetadataResult): HMS_EXTERNAL_TABLE (active legacy) sources rows from the relocated HudiExternalMetaCache.getTimelineRows (delete-unit class that already imports org.apache.hudi), PLUGIN_EXTERNAL_TABLE (dormant) delegates to the connector via the SPI. The two org.apache.hudi imports are removed from MetadataGenerator. Dormant until the hudi catalog is flipped (hudi tables stay HMSExternalTable under type='hms', so the HMS arm keeps serving the p2 suites today). Tests: HudiConnector declares the capability; PluginDrivenExternalTable.supportsMetadataTable resolves connector-wide + per-table marker. fe-core + fe-connector-hudi BUILD SUCCESS, 0 checkstyle, import gate clean. Timeline row parity is e2e-owed (enableHudiTest: the 4 p2 suites' timestamp/action/state/state_transition_time rows). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/api/ConnectorCapability.java | 11 +++- .../connector/api/ConnectorMetadata.java | 16 ++++++ .../doris/connector/hudi/HudiConnector.java | 14 +++++ .../connector/hudi/HudiConnectorMetadata.java | 29 +++++++++++ .../hudi/HudiConnectorOwnsHandleTest.java | 14 +++++ .../datasource/PluginDrivenExternalTable.java | 38 ++++++++++++++ .../hudi/HudiExternalMetaCache.java | 20 +++++++ .../tablefunction/MetadataGenerator.java | 52 +++++++++++-------- .../PluginDrivenExternalTableTest.java | 19 +++++++ 9 files changed, 190 insertions(+), 23 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java index 427d31584675f1..d266a7e0821363 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java @@ -137,5 +137,14 @@ public enum ConnectorCapability { * authenticate with a single static catalog identity (every non-REST iceberg flavor, JDBC, ES, ...) must * NOT declare it. Declared by the iceberg connector only when configured {@code iceberg.rest.session=user}.

    */ - SUPPORTS_USER_SESSION + SUPPORTS_USER_SESSION, + /** + * Indicates the connector exposes a metadata table (e.g. the hudi commit timeline) whose rows are read via + * {@link ConnectorMetadata#getMetadataTableRows}. + * + *

    The {@code hudi_meta()} / TIMELINE table-valued function's plugin-driven arm delegates to the connector + * only when it declares this; a connector with no metadata table must NOT declare it so the TVF rejects the + * table with "not a hudi table". Hudi declares it connector-wide (every hudi table has a commit timeline).

    + */ + SUPPORTS_METADATA_TABLE } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java index 4ba5a7ca2d8033..ae7e1e2df373a0 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorMetadata.java @@ -228,6 +228,22 @@ default ConnectorTableHandle applyTopnLazyMaterialization(ConnectorSession sessi return handle; // default: connectors without column-pruned scan metadata ignore the signal } + /** + * Engine-neutral rows for a connector metadata table (e.g. the hudi commit timeline), one row per record in + * the fixed column order the table-valued function declares. The TVF owns the column schema; the connector + * returns only the {@code String} cell values in that order (a {@code null} cell renders as SQL NULL). + * {@code kind} selects the metadata table (currently only {@code "timeline"}). + * + *

    Default empty: a connector without a metadata table returns nothing. The plugin-driven TVF arm gates on + * {@link ConnectorCapability#SUPPORTS_METADATA_TABLE} before delegating here, so only a connector that + * declares the capability is ever asked. Connectors overriding this that read remote metadata off the + * planning thread must pin the TCCL to the plugin classloader themselves (fe-core does not).

    + */ + default List> getMetadataTableRows(ConnectorSession session, ConnectorTableHandle handle, + String kind) { + return Collections.emptyList(); + } + @Override default void close() throws IOException { } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index 1fc226e3b4a16f..4dc740c174c20f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hudi; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.DorisConnectorException; @@ -39,8 +40,10 @@ import java.io.IOException; import java.util.Collections; +import java.util.EnumSet; import java.util.Map; import java.util.Optional; +import java.util.Set; import java.util.concurrent.Callable; /** @@ -131,6 +134,17 @@ public boolean ownsHandle(ConnectorTableHandle handle) { return handle instanceof HudiTableHandle; } + /** + * Every hudi table exposes its commit timeline via the {@code hudi_meta()} / TIMELINE TVF, so declare the + * metadata-table capability connector-wide. fe-core's plugin-driven {@code hudiMetadataResult} arm reads this + * (via {@code hasScanCapability}) to admit a flipped hudi table, and the hive gateway reflects it onto a + * hudi-on-HMS table's delegated schema so hudi_meta keeps working through the sibling delegation. + */ + @Override + public Set getCapabilities() { + return EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE); + } + @Override public ConnectorScanPlanProvider getScanPlanProvider() { return new HudiScanPlanProvider(properties, context); diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 0d826faa268518..66dc9e861101f3 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -47,6 +47,7 @@ import org.apache.hadoop.conf.Configuration; import org.apache.hudi.common.table.HoodieTableMetaClient; import org.apache.hudi.common.table.TableSchemaResolver; +import org.apache.hudi.common.table.timeline.HoodieInstant; import org.apache.hudi.common.util.Option; import org.apache.hudi.internal.schema.InternalSchema; import org.apache.hudi.internal.schema.Types; @@ -56,6 +57,7 @@ import java.util.AbstractMap; import java.util.ArrayList; +import java.util.Arrays; import java.util.Collections; import java.util.HashMap; import java.util.LinkedHashMap; @@ -691,6 +693,33 @@ private long latestInstant(HudiTableHandle handle) { HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), handle.getBasePath()))); } + /** + * Engine-neutral rows for the {@code hudi_meta()} / TIMELINE metadata table: one row per instant of the FULL + * active timeline (all states, NOT the completed-only helper — the TVF shows a {@code state} column), each + * mapped to the 4 String cells the TVF renders (requestedTime / action / state / completionTime). The metaClient + * touch runs under the plugin auth + TCCL pin (like {@link #latestInstant} / {@link #collectPartitions}), so + * fe-core adds no pin of its own. {@code completionTime} is {@code null} for a non-completed instant (rendered + * SQL NULL) — byte-parity with the legacy fe-core inline loop. Unknown {@code kind} returns nothing. + */ + @Override + public List> getMetadataTableRows(ConnectorSession session, ConnectorTableHandle handle, + String kind) { + if (!"timeline".equals(kind)) { + return Collections.emptyList(); + } + HudiTableHandle hudiHandle = (HudiTableHandle) handle; + return metaClientExecutor.execute(() -> { + HoodieTableMetaClient metaClient = + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), hudiHandle.getBasePath()); + List> rows = new ArrayList<>(); + for (HoodieInstant instant : metaClient.getActiveTimeline().getInstants()) { + rows.add(Arrays.asList(instant.requestedTime(), instant.getAction(), + instant.getState().name(), instant.getCompletionTime())); + } + return rows; + }); + } + private boolean useHiveSyncPartition() { return Boolean.parseBoolean(properties.getOrDefault(USE_HIVE_SYNC_PARTITION, "false")); } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java index f86b5bc0b28c74..6cf314933361ee 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiConnectorOwnsHandleTest.java @@ -17,6 +17,7 @@ package org.apache.doris.connector.hudi; +import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.spi.ConnectorContext; @@ -24,6 +25,7 @@ import org.junit.jupiter.api.Test; import java.util.Collections; +import java.util.Set; /** * Pins {@link HudiConnector#ownsHandle} for the hms 3-way sibling routing: a flipped hms gateway embeds this @@ -59,4 +61,16 @@ public void ownsHandleOnlyForHudiTableHandle() { Assertions.assertFalse(connector.ownsHandle(new ConnectorTableHandle() { }), "a foreign (non-hudi) handle is NOT owned by the hudi connector"); } + + @Test + public void declaresMetadataTableCapabilityForTimeline() { + // WHY: the hudi_meta() / TIMELINE TVF's plugin-driven arm delegates only when the connector declares + // SUPPORTS_METADATA_TABLE (read via PluginDrivenExternalTable.hasScanCapability), and the hive gateway + // reflects it onto a hudi-on-HMS table's schema so hudi_meta keeps working through the sibling delegation. + // Every hudi table has a commit timeline, so it is connector-wide. MUTATION: dropping the capability -> a + // flipped hudi table's hudi_meta() returns "not a hudi table". + Set caps = connector().getCapabilities(); + Assertions.assertTrue(caps.contains(ConnectorCapability.SUPPORTS_METADATA_TABLE), + "hudi declares the metadata-table capability so hudi_meta() works post-flip"); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index afd2839b4bebae..6254af1f907e24 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -254,6 +254,17 @@ public boolean supportsNestedColumnPrune() { return hasScanCapability(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE); } + /** + * Returns whether THIS table exposes a connector metadata table (e.g. the hudi commit timeline) queryable via + * the {@code hudi_meta()} / TIMELINE TVF. Resolved per-table via {@link #hasScanCapability}: a uniform + * connector (hudi) declares it connector-wide, and the hive gateway reflects it onto a hudi-on-HMS table's + * schema so the TVF keeps serving it after the flip; a connector with no metadata table returns false and the + * TVF rejects the table. + */ + public boolean supportsMetadataTable() { + return hasScanCapability(ConnectorCapability.SUPPORTS_METADATA_TABLE); + } + /** * Whether this table supports a per-table-refinable scan-planning capability, resolved connector-wide OR * per-table. A uniform-format connector (iceberg — every table orc/parquet) declares the capability for all @@ -819,6 +830,33 @@ public Map> getNameToPartitionValues(Optional return nameToValues; } + /** + * Engine-neutral rows for a connector metadata table (the {@code hudi_meta()} / TIMELINE TVF), sourced from + * the connector's {@link ConnectorMetadata#getMetadataTableRows} in one round-trip. {@code kind} selects the + * metadata table (currently only {@code "timeline"}). Each returned row is the ordered String cell values the + * TVF renders (nulls -> SQL NULL). Empty for an unresolved handle. A NEW standalone method (like + * {@link #getNameToPartitionValues}) — byte/cost-invariant for paimon/iceberg, which never declare + * {@code SUPPORTS_METADATA_TABLE} and so never reach this via the capability-gated TVF arm. No TCCL pin here; + * the connector pins internally (bundled hudi timeline reflection). + */ + public List> getMetadataTableRows(String kind) { + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return Collections.emptyList(); + } + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + if (connector == null) { + return Collections.emptyList(); + } + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + if (!handleOpt.isPresent()) { + return Collections.emptyList(); + } + return metadata.getMetadataTableRows(session, handleOpt.get(), kind); + } + @Override public long getCachedRowCount() { // Do NOT call makeSureInitialized() here. diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiExternalMetaCache.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiExternalMetaCache.java index 258838711f0890..eeb3f7ba2d878a 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiExternalMetaCache.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiExternalMetaCache.java @@ -45,6 +45,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.List; @@ -103,6 +104,25 @@ public HoodieTableMetaClient getHoodieTableMetaClient(NameMapping nameMapping) { return metaClientEntry.get(nameMapping.getCtlId()).get(HudiMetaClientCacheKey.of(nameMapping)); } + /** + * Engine-neutral rows for the {@code hudi_meta()} / TIMELINE TVF for a LEGACY hms-backed hudi table: one row + * per instant of the full active timeline mapped to (requestedTime, action, state, completionTime), the same + * 4 String cells the TVF renders (completionTime null for a non-completed instant -> SQL NULL). Relocated here + * (this class already touches the hudi timeline and is part of the legacy delete-unit) so + * {@code MetadataGenerator} sheds its {@code org.apache.hudi} imports; byte-parity with the former inline loop. + * The flipped (plugin-driven) hudi table takes the connector path instead ({@code + * HudiConnectorMetadata.getMetadataTableRows}). Removed wholesale at the delete step with the legacy hudi package. + */ + public List> getTimelineRows(NameMapping nameMapping) { + HoodieTimeline timeline = getHoodieTableMetaClient(nameMapping).getActiveTimeline(); + List> rows = new ArrayList<>(); + for (HoodieInstant instant : timeline.getInstants()) { + rows.add(Arrays.asList(instant.requestedTime(), instant.getAction(), + instant.getState().name(), instant.getCompletionTime())); + } + return rows; + } + public HoodieTableFileSystemView getFsView(NameMapping nameMapping) { return fsViewEntry.get(nameMapping.getCtlId()).get(HudiFsViewCacheKey.of(nameMapping)); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java index 270d23e463cf76..2dd5d1a08ad259 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/tablefunction/MetadataGenerator.java @@ -126,8 +126,6 @@ import com.google.common.collect.ImmutableMap; import com.google.common.collect.Lists; import com.google.gson.Gson; -import org.apache.hudi.common.table.timeline.HoodieInstant; -import org.apache.hudi.common.table.timeline.HoodieTimeline; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; import org.apache.thrift.TException; @@ -457,32 +455,42 @@ private static TFetchSchemaTableDataResult hudiMetadataResult(TMetadataTableRequ return errorResult("The specified db or table does not exist"); } - if (!(dorisTable instanceof HMSExternalTable)) { - return errorResult("The specified table is not a hudi table: " + hudiMetadataParams.getTable()); + if (hudiQueryType != THudiQueryType.TIMELINE) { + return errorResult("Unsupported hudi inspect type: " + hudiQueryType); } - HMSExternalTable hudiTable = (HMSExternalTable) dorisTable; - List dataBatch = Lists.newArrayList(); - TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult(); - - switch (hudiQueryType) { - case TIMELINE: - HoodieTimeline timeline = Env.getCurrentEnv().getExtMetaCacheMgr() - .hudi(catalog.getId()) - .getHoodieTableMetaClient(hudiTable.getOrBuildNameMapping()) - .getActiveTimeline(); - for (HoodieInstant instant : timeline.getInstants()) { - TRow trow = new TRow(); - trow.addToColumnValue(new TCell().setStringVal(instant.requestedTime())); - trow.addToColumnValue(new TCell().setStringVal(instant.getAction())); - trow.addToColumnValue(new TCell().setStringVal(instant.getState().name())); - trow.addToColumnValue(new TCell().setStringVal(instant.getCompletionTime())); - dataBatch.add(trow); + // Dual-arm on table type (mirrors partitionValuesMetadataResult): a LEGACY hms-backed hudi table is an + // HMSExternalTable served from HudiExternalMetaCache; a flipped hudi table is a PluginDrivenExternalTable + // served by its connector via the SUPPORTS_METADATA_TABLE SPI. Timeline iteration/parsing lives OUTSIDE + // this class in both arms, so MetadataGenerator no longer imports org.apache.hudi. The plugin arm is + // dormant until the hudi catalog is flipped; the HMS arm keeps serving the 4 p2 hudi_meta suites today. + List> timelineRows; + switch (dorisTable.getType()) { + case HMS_EXTERNAL_TABLE: + timelineRows = Env.getCurrentEnv().getExtMetaCacheMgr().hudi(catalog.getId()) + .getTimelineRows(dorisTable.getOrBuildNameMapping()); + break; + case PLUGIN_EXTERNAL_TABLE: { + PluginDrivenExternalTable pluginTable = (PluginDrivenExternalTable) dorisTable; + if (!pluginTable.supportsMetadataTable()) { + return errorResult("The specified table is not a hudi table: " + hudiMetadataParams.getTable()); } + timelineRows = pluginTable.getMetadataTableRows("timeline"); break; + } default: - return errorResult("Unsupported hudi inspect type: " + hudiQueryType); + return errorResult("The specified table is not a hudi table: " + hudiMetadataParams.getTable()); } + + List dataBatch = Lists.newArrayList(); + for (List row : timelineRows) { + TRow trow = new TRow(); + for (String cell : row) { + trow.addToColumnValue(new TCell().setStringVal(cell)); + } + dataBatch.add(trow); + } + TFetchSchemaTableDataResult result = new TFetchSchemaTableDataResult(); result.setDataBatch(dataBatch); result.setStatus(new TStatus(TStatusCode.OK)); return result; diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index bfacdfb599e743..68a57341ef45f0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -394,6 +394,25 @@ public void supportsColumnAutoAnalyzeReflectsConnectorCapability() { EnumSet.noneOf(ConnectorCapability.class), autoAnalyzeMarker).supportsTopNLazyMaterialize()); } + @Test + public void supportsMetadataTableReflectsConnectorCapability() { + // The hudi_meta() / TIMELINE TVF's plugin arm gates on this. Hudi declares it connector-wide; the hive + // gateway reflects it onto a hudi-on-HMS table via the per-table marker (both resolved by hasScanCapability). + // MUTATION: hard-coding it / reading a different capability -> a flipped hudi table's hudi_meta() rejects + // with "not a hudi table". + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE)).supportsMetadataTable()); + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), + Collections.singletonMap(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, + ConnectorCapability.SUPPORTS_METADATA_TABLE.name())).supportsMetadataTable()); + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class)).supportsMetadataTable()); + // Independent of the other capabilities: declaring metadata-table must NOT enable auto-analyze. + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE)).supportsColumnAutoAnalyze()); + } + @Test public void supportsTopNLazyMaterializeReflectsConnectorCapability() { Assertions.assertTrue(pluginTableWithCapabilities( From 5af7e9535283cf76f58197400827e639f2d17669 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 16:03:04 +0800 Subject: [PATCH 180/334] [feat](catalog) Hive coupling-seams S3: ANALYZE ... WITH SAMPLE full port to plain-hive (dormant) Sampled ANALYZE works today for hive via HMSAnalysisTask.doSample. A flipped hive table is a PluginDrivenExternalTable, so three fe-core touchpoints go dead: AnalysisManager.canSample (instanceof HMSExternalTable + dlaType==HIVE) rejects at build time; PluginDrivenExternalTable .createAnalysisTask returns the generic ExternalAnalysisTask whose doSample throws NotImplementedException at runtime; AnalyzeTableCommand.isSamplingPartition degrades a many-partition table to analyzing every partition. Full port (user decision, full parity), all connector-agnostic: - ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE, emitted PER-TABLE by hive in getTableSchema for plain-hive tables only (supportsHiveSampleAnalyze = !view && detect==HIVE, ANY format -- legacy dlaType==HIVE had no format gate, unlike Top-N). iceberg/hudi-on-HMS and native iceberg/paimon never declare it, keeping their build-time reject. - PluginDrivenExternalTable.supportsSampleAnalyze() = hasScanCapability(...); additive capability arms in AnalysisManager.canSample and AnalyzeTableCommand.isSamplingPartition (legacy HMSExternalTable arm kept live for the un-flipped path, per supportAutoAnalyze). - New PluginDrivenSampleAnalysisTask extends ExternalAnalysisTask, carrying doSample/ getSampleInfo/needLimit ported VERBATIM from HMSAnalysisTask (they touch only base members -- table/tbl/col/tableSample + shared templates -- so scale factor, limit heuristic and linear-vs-DUJ1 estimator choice stay byte-identical). createAnalysisTask hands it back only when supportsSampleAnalyze() (iceberg/paimon stay on ExternalAnalysisTask, byte-identical). - Connector chunk-sizes SPI: ConnectorStatisticsOps.listFileSizes (raw per-file byte lengths, all partitions, not sampled/summed; default empty). HiveConnectorMetadata lists every partition's files' getLength() TCCL-pinned, best-effort empty-on-error, sibling-delegated. PluginDrivenExternalTable.getChunkSizes() overrides the throwing ExternalTable default to delegate to it; the Doris-type slot-width math stays fe-core-side. - Distribution-column parity (user chose full parity): hive emits bucket cols RAW under ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY; PluginDrivenExternalTable.getDistributionColumn Names() reads + lowercases them (mirroring legacy HMSExternalTable.getDistributionColumnNames) so a flipped bucketed hive table picks the same estimator. Stripped from SHOW CREATE. - StatisticsAutoCollector.processOneJob: force-FULL only for plugin tables that CANNOT sample (&& !supportsSampleAnalyze()), so a flipped plain-hive table keeps the background SAMPLE/FULL heuristic (its doSample works), matching legacy hive auto-analyze which could sample. Dormant until hms enters SPI_READY_TYPES. Tests: sample marker per plain-hive format / absent for view; distribution_columns emit; supportsSampleAnalyze + getDistributionColumnNames resolution. fe-core + fe-connector-hive BUILD SUCCESS, 0 checkstyle, import gate clean, suites green. The sampling SQL round-trip (doSample) + bucket-col estimator + per-partition listing are e2e-owed (heterogeneous HMS: FULL-vs-SAMPLE stat assertions on text/orc/partitioned hive). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/api/ConnectorCapability.java | 14 +- .../connector/api/ConnectorStatisticsOps.java | 15 ++ .../connector/api/ConnectorTableSchema.java | 13 ++ .../connector/hive/HiveConnectorMetadata.java | 63 +++++++ .../hive/HiveConnectorMetadataSchemaTest.java | 39 ++++ .../datasource/PluginDrivenExternalTable.java | 78 +++++++- .../plans/commands/AnalyzeTableCommand.java | 13 +- .../doris/statistics/AnalysisManager.java | 8 + .../PluginDrivenSampleAnalysisTask.java | 174 ++++++++++++++++++ .../statistics/StatisticsAutoCollector.java | 9 +- .../PluginDrivenExternalTableTest.java | 35 ++++ 11 files changed, 453 insertions(+), 8 deletions(-) create mode 100644 fe/fe-core/src/main/java/org/apache/doris/statistics/PluginDrivenSampleAnalysisTask.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java index d266a7e0821363..1e9f21416a41b6 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorCapability.java @@ -146,5 +146,17 @@ public enum ConnectorCapability { * only when it declares this; a connector with no metadata table must NOT declare it so the TVF rejects the * table with "not a hudi table". Hudi declares it connector-wide (every hudi table has a commit timeline).

    */ - SUPPORTS_METADATA_TABLE + SUPPORTS_METADATA_TABLE, + /** + * Indicates the connector's file-scan tables support {@code ANALYZE ... WITH SAMPLE} (scale-factor estimation + * from raw per-file byte sizes via {@link ConnectorStatisticsOps#listFileSizes}, with fe-core doing the + * Doris-type slot-width math). + * + *

    fe-core admits sampled analyze for a plugin-driven table only when it declares this. A heterogeneous + * connector (hive) emits it as a PER-TABLE marker in getTableSchema for its plain-hive tables only (legacy + * gated on {@code dlaType==HIVE}), so iceberg/hudi-on-HMS are excluded. Connectors whose {@code doSample} is + * unimplemented (native iceberg/paimon, JDBC, ES) must NOT declare it so sampled analyze stays rejected at + * build time.

    + */ + SUPPORTS_SAMPLE_ANALYZE } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java index 8d245ed90da547..2e89a09d7dd881 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java @@ -19,6 +19,8 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import java.util.Collections; +import java.util.List; import java.util.Optional; /** @@ -60,4 +62,17 @@ default Optional getColumnStatistics( default long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTableHandle handle) { return -1; } + + /** + * Returns the RAW byte length of every data file across ALL partitions of the table (not sampled, not summed), + * for {@code ANALYZE ... WITH SAMPLE}: fe-core seed-shuffles and cumulates these sizes to a sample scale + * factor, then does the Doris-type slot-width math itself. Unlike {@link #estimateDataSizeByListingFiles} it + * neither partition-samples nor sums, because the sampler needs the individual file sizes. A potentially + * expensive full remote listing, so connectors that cannot enumerate files cheaply must NOT override it + * (default empty -> the sampler falls back to scale factor 1). Best-effort: an override must return empty on + * any listing error rather than throw (statistics must not fail a query). + */ + default List listFileSizes(ConnectorSession session, ConnectorTableHandle handle) { + return Collections.emptyList(); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java index f637363fa128ea..fe535ab0538a70 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java @@ -69,6 +69,19 @@ public final class ConnectorTableSchema { */ public static final String PER_TABLE_CAPABILITIES_KEY = "connector.per-table-capabilities"; + /** + * Reserved property key carrying a CSV of the table's distribution (bucketing) column names, already + * lowercased. A heterogeneous connector (hive) whose bucketing varies per table cannot express it as a + * connector-wide trait, so it emits it here per-table. + * + *

    fe-core's {@code PluginDrivenExternalTable.getDistributionColumnNames()} reads it from the cached schema + * (no remote round-trip) so a flipped bucketed hive table picks the same NDV estimator as legacy + * {@code HMSExternalTable.getDistributionColumnNames} (a single bucket column selects the linear estimator in + * sampled analyze). A non-bucketed table omits it and connectors with no bucketing concept never emit it. + * Stripped from the user-facing SHOW CREATE TABLE PROPERTIES(...) block.

    + */ + public static final String DISTRIBUTION_COLUMNS_KEY = "connector.distribution-columns"; + private final String tableName; private final List columns; private final String tableFormatType; diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index dc9bf629324e4f..ff10cdfe1517c8 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -426,6 +426,9 @@ public ConnectorTableSchema getTableSchema( if (supportsHiveColumnAutoAnalyze(tableInfo)) { perTableCapabilities.add(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE.name()); } + if (supportsHiveSampleAnalyze(tableInfo)) { + perTableCapabilities.add(ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE.name()); + } if (supportsHiveTopNLazyMaterialize(tableInfo)) { perTableCapabilities.add(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name()); } @@ -434,6 +437,15 @@ public ConnectorTableSchema getTableSchema( String.join(",", perTableCapabilities)); } + // Distribution (bucketing) columns for the flipped table's getDistributionColumnNames() — legacy + // HMSExternalTable read getSd().getBucketCols(). Emitted RAW (fe-core lowercases, mirroring the legacy + // getDistributionColumnNames); only a bucketed table carries it. Consumed by sampled ANALYZE to pick the + // linear-vs-DUJ1 NDV estimator (a single bucket column that IS the analyzed column -> linear). + List bucketCols = tableInfo.getBucketCols(); + if (bucketCols != null && !bucketCols.isEmpty()) { + tableProperties.put(ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY, String.join(",", bucketCols)); + } + return new ConnectorTableSchema(tableName, allColumns, formatType, tableProperties); } @@ -794,6 +806,46 @@ public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTa } } + /** + * Returns the raw byte length of every data file across ALL partitions (not sampled, not summed), a port of + * legacy {@code HMSExternalTable.getChunkSizes} for {@code ANALYZE ... WITH SAMPLE}. Only plain-hive tables + * are listed (iceberg/hudi-on-HMS are served by their own connectors via the sibling divert; a view has no + * data files) — anything else returns empty. Lists EVERY partition (no {@link #STATS_PARTITION_SAMPLE_SIZE} + * sampling, unlike {@link #estimateDataSizeByListingFiles}) because the fe-core sampler needs the individual + * file sizes to seed-shuffle and cumulate. Best-effort: any listing error degrades to empty, never throwing + * (statistics must not fail a query). Pins the TCCL to the plugin classloader for the {@code FileSystem} + * reflection, exactly like estimateDataSizeByListingFiles (the statistics thread is not pinned by fe-core). + */ + @Override + public List listFileSizes(ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session, handle).listFileSizes(session, handle); + } + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + if (hiveHandle.getTableType() != HiveTableType.HIVE) { + return Collections.emptyList(); + } + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(getClass().getClassLoader()); + Configuration conf = buildHadoopConf(); + List sizes = new ArrayList<>(); + for (String location : resolvePartitionLocations(hiveHandle)) { + for (HiveFileStatus file : fileListingCache.listDataFiles( + hiveHandle.getDbName(), hiveHandle.getTableName(), location, conf)) { + sizes.add(file.getLength()); + } + } + return sizes; + } catch (RuntimeException e) { + LOG.warn("Failed to list hive file sizes for {}.{} for sample analyze", + hiveHandle.getDbName(), hiveHandle.getTableName(), e); + return Collections.emptyList(); + } finally { + Thread.currentThread().setContextClassLoader(previous); + } + } + /** * Sampling + summing + scale-up core of {@link #estimateDataSizeByListingFiles}, isolated from the * {@code FileSystem} I/O (injected as {@code sizeOf}) so the estimation math is unit-testable. Returns -1 @@ -1878,6 +1930,17 @@ private boolean supportsHiveColumnAutoAnalyze(HmsTableInfo tableInfo) { return !isView(tableInfo) && HiveTableFormatDetector.detect(tableInfo) == HiveTableType.HIVE; } + /** + * Whether {@code tableInfo} is a plain-hive data table (any file format) eligible for {@code ANALYZE ... WITH + * SAMPLE}, replicating legacy {@code AnalysisManager.canSample}'s {@code dlaType==HIVE} gate. Like + * {@link #supportsHiveColumnAutoAnalyze} there is NO orc/parquet restriction (legacy sampled any hive format); + * a view is excluded and an iceberg/hudi-on-HMS table is excluded ({@code detect() != HIVE}) so sampled + * analyze stays rejected for them (their {@code doSample} is unimplemented). + */ + private boolean supportsHiveSampleAnalyze(HmsTableInfo tableInfo) { + return !isView(tableInfo) && HiveTableFormatDetector.detect(tableInfo) == HiveTableType.HIVE; + } + /** Whether the HMS table is a view (tableType VIRTUAL_VIEW), mirroring legacy {@code HMSExternalTable.isView}. */ private static boolean isView(HmsTableInfo tableInfo) { return VIRTUAL_VIEW_TABLE_TYPE.equalsIgnoreCase(tableInfo.getTableType()); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index 4c609f2afe7b79..ace212ae256d4e 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -200,6 +200,45 @@ public void testColumnAutoAnalyzeMarkerAbsentForView() { Assertions.assertFalse(hasCapability(schema, ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE)); } + @Test + public void testSampleAnalyzeMarkerEmittedForEveryPlainHiveFormat() { + // Legacy AnalysisManager.canSample gated on dlaType==HIVE (any file format). Emit SUPPORTS_SAMPLE_ANALYZE + // per-table for every plain-hive table so fe-core admits ANALYZE ... WITH SAMPLE while excluding + // iceberg/hudi-on-HMS. Like auto-analyze there is NO orc/parquet restriction. MUTATION: gating on input + // format -> text/csv/json hive tables silently lose sample. + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE)); + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(ORC_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE)); + Assertions.assertTrue(hasCapability(schemaOf(unpartitionedTable(TEXT_INPUT_FORMAT).build()), + ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE)); + } + + @Test + public void testSampleAnalyzeMarkerAbsentForView() { + // A view is not sampled (legacy canSample excluded it via dlaType); excluded before the format check. + ConnectorTableSchema schema = schemaOf( + unpartitionedTable(PARQUET_INPUT_FORMAT).tableType("VIRTUAL_VIEW").build()); + Assertions.assertFalse(hasCapability(schema, ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE)); + } + + @Test + public void testDistributionColumnsEmittedRawForBucketedTable() { + // Bucketing columns are emitted RAW (fe-core lowercases, mirroring legacy getDistributionColumnNames); + // only a bucketed table carries the marker. MUTATION: not emitting -> a flipped bucketed hive table loses + // the linear NDV estimator in sampled analyze. + ConnectorTableSchema schema = schemaOf( + unpartitionedTable(PARQUET_INPUT_FORMAT).bucketCols(Arrays.asList("Id", "region")).build()); + Assertions.assertEquals("Id,region", + schema.getProperties().get(ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY)); + } + + @Test + public void testDistributionColumnsAbsentForNonBucketedTable() { + Assertions.assertNull(schemaOf(unpartitionedTable(PARQUET_INPUT_FORMAT).build()) + .getProperties().get(ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY)); + } + @Test public void testTopNLazyCapabilityMarkerAbsentForView() { // A view is excluded even when its SD carries a parquet input format, mirroring legacy diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 6254af1f907e24..7757411d837a66 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -50,6 +50,7 @@ import org.apache.doris.statistics.ColumnStatistic; import org.apache.doris.statistics.ColumnStatisticBuilder; import org.apache.doris.statistics.ExternalAnalysisTask; +import org.apache.doris.statistics.PluginDrivenSampleAnalysisTask; import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; @@ -61,6 +62,7 @@ import java.util.ArrayList; import java.util.Collections; import java.util.EnumSet; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -265,6 +267,19 @@ public boolean supportsMetadataTable() { return hasScanCapability(ConnectorCapability.SUPPORTS_METADATA_TABLE); } + /** + * Returns whether THIS table supports {@code ANALYZE ... WITH SAMPLE}. Consulted by + * {@code AnalysisManager.canSample}, {@code AnalyzeTableCommand.isSamplingPartition}, {@link + * #createAnalysisTask} (to return a sample-capable task) and the background auto-analyze method choice. + * Resolved per-table via {@link #hasScanCapability}: hive emits it for its plain-hive tables only (legacy + * {@code dlaType==HIVE}), so iceberg/hudi-on-HMS are excluded; native iceberg/paimon never declare it (their + * {@code doSample} is unimplemented), keeping their current build-time reject. Mirrors + * {@link #supportsTopNLazyMaterialize}. + */ + public boolean supportsSampleAnalyze() { + return hasScanCapability(ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE); + } + /** * Whether this table supports a per-table-refinable scan-planning capability, resolved connector-wide OR * per-table. A uniform-format connector (iceberg — every table orc/parquet) declares the capability for all @@ -691,7 +706,8 @@ public Map getTableProperties() { || ConnectorTableSchema.SHOW_LOCATION_KEY.equals(key) || ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY.equals(key) || ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY.equals(key) - || ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY.equals(key)) { + || ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY.equals(key) + || ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY.equals(key)) { continue; } result.put(entry.getKey(), entry.getValue()); @@ -944,6 +960,12 @@ public void gsonPostProcess() throws IOException { @Override public BaseAnalysisTask createAnalysisTask(AnalysisInfo info) { makeSureInitialized(); + if (supportsSampleAnalyze()) { + // A flipped plain-hive table keeps ANALYZE ... WITH SAMPLE working (ExternalAnalysisTask.doSample + // throws NotImplementedException). iceberg/paimon do NOT declare the capability, so they stay on the + // byte-identical ExternalAnalysisTask path — the extra check is one cached capability lookup. + return new PluginDrivenSampleAnalysisTask(info); + } return new ExternalAnalysisTask(info); } @@ -982,6 +1004,60 @@ public Optional getColumnStatistic(String colName) { return toColumnStatistic(statsOpt.get(), getColumn(colName)); } + /** + * The raw per-file byte sizes that {@code ANALYZE ... WITH SAMPLE} seed-shuffles and cumulates into a sample + * scale factor, from the connector's file listing (like legacy {@code HMSExternalTable.getChunkSizes}). The + * connector returns only the raw byte lengths; the Doris-type slot-width math stays fe-core-side in the sample + * task. Overrides {@link ExternalTable#getChunkSizes()} (which throws {@code NotImplementedException}); returns + * empty on any miss (non-plugin catalog / null connector / unresolved handle) so a connector that cannot list + * degrades the sampler to scale factor 1. Inert for iceberg/paimon — only reached from the sample task, which + * they never create. No TCCL pin here; the hive {@code listFileSizes} impl pins internally (parity with + * {@link #fetchRowCount()} / {@link #getColumnStatistic}). + */ + @Override + public List getChunkSizes() { + makeSureInitialized(); + if (!(catalog instanceof PluginDrivenExternalCatalog)) { + return Collections.emptyList(); + } + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + if (connector == null) { + return Collections.emptyList(); + } + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + if (!handleOpt.isPresent()) { + return Collections.emptyList(); + } + return metadata.listFileSizes(session, handleOpt.get()); + } + + /** + * The table's distribution (bucketing) column names, lowercased, from the connector's per-table + * {@code connector.distribution-columns} schema marker (read from the already-cached schema, no round-trip). + * Overrides the {@code TableIf} empty default so a flipped bucketed hive table matches legacy + * {@code HMSExternalTable.getDistributionColumnNames} (which lowercased on this side too). Empty for a + * non-bucketed table and for connectors that emit no marker (paimon/iceberg) — byte-invariant for them. Used by + * sampled ANALYZE to pick the linear-vs-DUJ1 NDV estimator. + */ + @Override + public Set getDistributionColumnNames() { + String csv = rawTableProperties().get(ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY); + if (csv == null || csv.isEmpty()) { + return Collections.emptySet(); + } + Set result = new HashSet<>(); + for (String name : csv.split(",")) { + String trimmed = name.trim(); + if (!trimmed.isEmpty()) { + result.add(trimmed.toLowerCase()); + } + } + return result; + } + /** * Turns the connector's raw {@link ConnectorColumnStatistics} into a Doris {@link ColumnStatistic}, * doing the Doris-type-dependent size math the connector cannot (legacy {@code setStatData} parity): a diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommand.java index 05ff1f976b40ef..92dfb635f990b5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/AnalyzeTableCommand.java @@ -32,6 +32,7 @@ import org.apache.doris.common.FeNameFormat; import org.apache.doris.common.UserException; import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.mysql.privilege.PrivPredicate; import org.apache.doris.nereids.trees.plans.PlanType; @@ -312,13 +313,19 @@ public boolean isPartitionOnly() { * isSamplingPartition */ public boolean isSamplingPartition() { - if (!(table instanceof HMSExternalTable) || partitionNames != null) { + // Additive: a flipped plain-hive table is a PluginDrivenExternalTable declaring SUPPORTS_SAMPLE_ANALYZE + // per-table; keep the legacy HMSExternalTable arm live for the un-flipped path. iceberg/hudi-on-HMS and + // native iceberg/paimon do not declare it, so they stay non-partition-sampled as before. + boolean sampleable = table instanceof HMSExternalTable + || (table instanceof PluginDrivenExternalTable + && ((PluginDrivenExternalTable) table).supportsSampleAnalyze()); + if (!sampleable || partitionNames != null) { return false; } int partNum = ConnectContext.get().getSessionVariable().getExternalTableAnalyzePartNum(); - if (partNum == -1 || partitionNames != null) { + if (partNum == -1) { return false; } - return table instanceof HMSExternalTable && table.getPartitionNames().size() > partNum; + return table.getPartitionNames().size() > partNum; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java index b2b4c0d57f63a1..72da54e82fdb57 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/AnalysisManager.java @@ -43,6 +43,7 @@ import org.apache.doris.common.util.Util; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.InternalCatalog; +import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.info.TableNameInfoUtils; import org.apache.doris.metric.MetricRepo; @@ -1481,6 +1482,13 @@ public boolean canSample(TableIf table) { if (table instanceof OlapTable) { return true; } + // Additive: a flipped plain-hive table is a PluginDrivenExternalTable declaring SUPPORTS_SAMPLE_ANALYZE + // per-table (iceberg/hudi-on-HMS and native iceberg/paimon do NOT declare it). Keeps the legacy + // HMSExternalTable arm below live for the un-flipped path, like StatisticsUtil.supportAutoAnalyze. + if (table instanceof PluginDrivenExternalTable + && ((PluginDrivenExternalTable) table).supportsSampleAnalyze()) { + return true; + } return table instanceof HMSExternalTable && ((HMSExternalTable) table).getDlaType().equals(HMSExternalTable.DLAType.HIVE); } diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/PluginDrivenSampleAnalysisTask.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/PluginDrivenSampleAnalysisTask.java new file mode 100644 index 00000000000000..220c985ee1eda2 --- /dev/null +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/PluginDrivenSampleAnalysisTask.java @@ -0,0 +1,174 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.statistics; + +import org.apache.doris.catalog.Column; +import org.apache.doris.common.Pair; +import org.apache.doris.qe.SessionVariable; +import org.apache.doris.statistics.util.StatisticsUtil; + +import org.apache.commons.text.StringSubstitutor; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Random; +import java.util.Set; + +/** + * Sample-capable analyze task for a flipped plugin-driven file-scan table (plain-hive after the HMS cutover). + * + *

    The generic {@link ExternalAnalysisTask} throws {@code NotImplementedException} from {@link #doSample()}, + * which is correct for connectors that cannot serve raw per-file sizes (iceberg/paimon/JDBC/ES). A plain-hive + * table CAN (legacy {@code HMSExternalTask.doSample}), so {@code PluginDrivenExternalTable.createAnalysisTask} + * hands back THIS task when the connector declares {@code SUPPORTS_SAMPLE_ANALYZE} per-table. The + * {@code doSample}/{@code getSampleInfo}/{@code needLimit} bodies are a verbatim port of the legacy + * {@code HMSAnalysisTask} equivalents — they touch only base-class members ({@link #table}, {@code tbl}, + * {@code col}, {@code tableSample} and the shared templates/helpers), never an HMS-specific field, so the scale + * factor, limit heuristic and linear-vs-DUJ1 estimator choice stay byte-identical to legacy. The raw file sizes + * come from {@code PluginDrivenExternalTable.getChunkSizes()} (connector's {@code listFileSizes}); the + * Doris-type slot-width math stays here. Full (non-sample) analyze falls through to the base {@link #doFull()}. + */ +public class PluginDrivenSampleAnalysisTask extends ExternalAnalysisTask { + + public PluginDrivenSampleAnalysisTask() { + super(); + } + + public PluginDrivenSampleAnalysisTask(AnalysisInfo info) { + super(info); + } + + @Override + protected void doSample() { + StringBuilder sb = new StringBuilder(); + Map params = buildSqlParams(); + params.put("min", getMinFunction()); + params.put("max", getMaxFunction()); + params.put("dataSizeFunction", getDataSizeFunction(col, false)); + Pair sampleInfo = getSampleInfo(); + params.put("scaleFactor", String.valueOf(sampleInfo.first)); + params.put("hotValueCollectCount", String.valueOf(SessionVariable.getHotValueCollectCount())); + if (LOG.isDebugEnabled()) { + LOG.debug("Will do sample collection for column {}", col.getName()); + } + boolean limitFlag = false; + boolean bucketFlag = false; + // If sample size is too large, use limit to control the sample size. + if (needLimit(sampleInfo.second, sampleInfo.first)) { + limitFlag = true; + long columnSize = 0; + for (Column column : table.getFullSchema()) { + columnSize += column.getDataType().getSlotSize(); + } + double targetRows = (double) sampleInfo.second / columnSize; + // Estimate the new scaleFactor based on the schema. + if (targetRows > StatisticsUtil.getHugeTableSampleRows()) { + params.put("limit", "limit " + StatisticsUtil.getHugeTableSampleRows()); + params.put("scaleFactor", + String.valueOf(sampleInfo.first * targetRows / StatisticsUtil.getHugeTableSampleRows())); + } + } + // Single distribution column is not fit for DUJ1 estimator, use linear estimator. + Set distributionColumns = tbl.getDistributionColumnNames(); + if (distributionColumns.size() == 1 && distributionColumns.contains(col.getName().toLowerCase())) { + bucketFlag = true; + sb.append(LINEAR_ANALYZE_TEMPLATE); + params.put("ndvFunction", "ROUND(NDV(`${colName}`) * ${scaleFactor})"); + params.put("rowCount", "ROUND(COUNT(1) * ${scaleFactor})"); + params.put("rowCount2", "(SELECT COUNT(1) FROM cte1 WHERE `${colName}` IS NOT NULL)"); + } else { + sb.append(DUJ1_ANALYZE_TEMPLATE); + params.put("subStringColName", getStringTypeColName(col)); + params.put("dataSizeFunction", getDataSizeFunction(col, true)); + params.put("ndvFunction", getNdvFunction("ROUND(SUM(t1.count) * ${scaleFactor})")); + params.put("rowCount", "ROUND(SUM(t1.count) * ${scaleFactor})"); + params.put("rowCount2", "(SELECT SUM(`count`) FROM cte1 WHERE `col_value` IS NOT NULL)"); + } + LOG.info("Sample for column [{}]. Scale factor [{}], " + + "limited [{}], is distribute column [{}]", + col.getName(), params.get("scaleFactor"), limitFlag, bucketFlag); + StringSubstitutor stringSubstitutor = new StringSubstitutor(params); + String sql = stringSubstitutor.replace(sb.toString()); + runQuery(sql); + } + + /** + * Get the pair of sample scale factor and the file size going to sample. While analyzing, the result of + * count, null count and data size need to multiply this scale factor to get a more accurate result. + * @return Pair of sample scale factor and the file size going to sample. + */ + protected Pair getSampleInfo() { + if (tableSample == null) { + return Pair.of(1.0, 0L); + } + long target; + // Get list of all files' size in this table (from the connector, via getChunkSizes()). + List chunkSizes = table.getChunkSizes(); + Collections.shuffle(chunkSizes, new Random(tableSample.getSeek())); + long total = 0; + // Calculate the total size of this table. + for (long size : chunkSizes) { + total += size; + } + if (total == 0) { + return Pair.of(1.0, 0L); + } + // Calculate the sample target size for percent and rows sample. + if (tableSample.isPercent()) { + target = total * tableSample.getSampleValue() / 100; + } else { + int columnSize = 0; + for (Column column : table.getFullSchema()) { + columnSize += column.getDataType().getSlotSize(); + } + target = columnSize * tableSample.getSampleValue(); + } + // Calculate the actual sample size (cumulate). + long cumulate = 0; + for (long size : chunkSizes) { + cumulate += size; + if (cumulate >= target) { + break; + } + } + return Pair.of(Math.max(((double) total) / cumulate, 1), cumulate); + } + + /** + * If the size to sample is larger than LIMIT_SIZE (1GB) and is much larger (1.2*) than the size the user + * wants to sample, use limit to control the total sample size. + * @param sizeToRead The file size to sample. + * @param factor sizeToRead * factor = Table total size. + * @return True if need to limit. + */ + protected boolean needLimit(long sizeToRead, double factor) { + long total = (long) (sizeToRead * factor); + long target; + if (tableSample.isPercent()) { + target = total * tableSample.getSampleValue() / 100; + } else { + int columnSize = 0; + for (Column column : table.getFullSchema()) { + columnSize += column.getDataType().getSlotSize(); + } + target = columnSize * tableSample.getSampleValue(); + } + return sizeToRead > LIMIT_SIZE && sizeToRead > target * LIMIT_FACTOR; + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java index a1d114f6cba099..7abec42ff1cc2f 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java +++ b/fe/fe-core/src/main/java/org/apache/doris/statistics/StatisticsAutoCollector.java @@ -149,9 +149,12 @@ protected void processOneJob(TableIf table, Set> columns, J analysisMethod = AnalysisMethod.FULL; } if (table instanceof PluginDrivenExternalTable - && ((PluginDrivenExternalTable) table).supportsColumnAutoAnalyze()) { - // Post-flip plugin tables (iceberg/paimon) only support full analyze: ExternalAnalysisTask.doSample - // throws, so the SAMPLE default would fail. Mirrors the legacy IcebergExternalTable arm above. + && ((PluginDrivenExternalTable) table).supportsColumnAutoAnalyze() + && !((PluginDrivenExternalTable) table).supportsSampleAnalyze()) { + // Force FULL only for plugin tables that CANNOT sample (iceberg/paimon): ExternalAnalysisTask.doSample + // throws, so the SAMPLE default would fail. A flipped plain-hive table declares SUPPORTS_SAMPLE_ANALYZE + // and keeps the SAMPLE/FULL heuristic above (its PluginDrivenSampleAnalysisTask.doSample works), matching + // legacy hive background auto-analyze which could sample. analysisMethod = AnalysisMethod.FULL; } boolean isSampleAnalyze = analysisMethod.equals(AnalysisMethod.SAMPLE); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index 68a57341ef45f0..2032fe872a7316 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -44,9 +44,11 @@ import org.junit.jupiter.api.Test; import org.mockito.Mockito; +import java.util.Arrays; import java.util.Collections; import java.util.EnumSet; import java.util.HashMap; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -413,6 +415,39 @@ public void supportsMetadataTableReflectsConnectorCapability() { EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE)).supportsColumnAutoAnalyze()); } + @Test + public void supportsSampleAnalyzeReflectsConnectorCapability() { + // AnalysisManager.canSample / AnalyzeTableCommand.isSamplingPartition / createAnalysisTask gate on this. + // Hive emits it per-table for plain-hive only (legacy dlaType==HIVE); iceberg/hudi-on-HMS and native + // iceberg/paimon do NOT declare it, keeping their build-time reject. MUTATION: hard-coding it / reading a + // different capability -> sampled ANALYZE wrongly admitted or wrongly rejected. + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.of(ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE)).supportsSampleAnalyze()); + Assertions.assertTrue(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class), + Collections.singletonMap(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY, + ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE.name())).supportsSampleAnalyze()); + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.noneOf(ConnectorCapability.class)).supportsSampleAnalyze()); + // Independent: sample must NOT enable auto-analyze (iceberg-on-HMS gets auto-analyze but not sample). + Assertions.assertFalse(pluginTableWithCapabilities( + EnumSet.of(ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE)).supportsColumnAutoAnalyze()); + } + + @Test + public void getDistributionColumnNamesReadsLowercasedMarker() { + // Bucketing columns come from the connector's per-table marker (emitted RAW), lowercased HERE to mirror + // legacy HMSExternalTable.getDistributionColumnNames. Consumed by sampled analyze's linear-vs-DUJ1 NDV + // estimator choice. MUTATION: not lowercasing / not reading the marker -> the estimator choice regresses + // for a flipped bucketed hive table. + PluginDrivenExternalTable table = pluginTableWithCapabilities(EnumSet.noneOf(ConnectorCapability.class), + Collections.singletonMap(ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY, "Id,Region")); + Assertions.assertEquals(new HashSet<>(Arrays.asList("id", "region")), table.getDistributionColumnNames()); + // No marker -> empty (paimon/iceberg unchanged, TableIf default). + Assertions.assertTrue(pluginTableWithCapabilities(EnumSet.noneOf(ConnectorCapability.class)) + .getDistributionColumnNames().isEmpty()); + } + @Test public void supportsTopNLazyMaterializeReflectsConnectorCapability() { Assertions.assertTrue(pluginTableWithCapabilities( From f3ea98402fbd199c65d9a4115bd16d6d1bc7fbb5 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 16:03:58 +0800 Subject: [PATCH 181/334] [doc](catalog) Hive coupling-seams W6: soften over-cautious write-path TCCL comment The getWritePlanProvider(handle) javadoc claimed the write-path TCCL pin was "a separate flip-time concern", implying an unaddressed gap. Recon verified it is a false gap: a foreign (iceberg/hudi-on-HMS) write is delegated to the sibling's per-handle write provider, and the sibling already carries the pin in its own TcclPinningConnectorContext (IcebergConnector wraps executeAuthenticated, classloader-thread-independent), so no additional pin is needed at this seam. Update the comment to say so; the iceberg-on-HMS write path stays e2e-owed. Comment-only. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../java/org/apache/doris/connector/hive/HiveConnector.java | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 9966f390c77024..c747cd45e70dd2 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -205,7 +205,10 @@ public ConnectorWritePlanProvider getWritePlanProvider() { * handle is passed through UNMODIFIED and NEVER cast (its concrete sibling type is invisible across the loader * split — a cast would CCE). A HUDI-stamped HiveTableHandle stays on the hive write path. Mirrors {@link * #getScanPlanProvider(ConnectorTableHandle)}; dormant until hms enters SPI_READY_TYPES. The returned sibling - * provider runs its planWrite on fe-core threads — the write-path TCCL pin is a separate flip-time concern. + * provider's planWrite runs on fe-core threads, but the write-path TCCL pin is already carried by the sibling's + * own {@code TcclPinningConnectorContext} (e.g. {@code IcebergConnector} wraps {@code executeAuthenticated}, + * classloader-thread-independent), so no additional pin is needed here — verified, not an open flip-time gap. + * The iceberg-on-HMS write path is e2e-owed on a heterogeneous HMS catalog. */ @Override public ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { From 3257b6af1b4ef6bcbfc47fe4d501d170a22ca9d6 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 16:08:11 +0800 Subject: [PATCH 182/334] [doc](catalog) Hive coupling-seams: S2/S3/S4 + W6 all landed; HANDOFF -> clean-room review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Record the completed step across the three authoritative docs: - hive-coupling-seams-step-design: check S2/S3/S4/W6 boxes with commit hashes + the S4 hidden-depth resolution (iceberg-on-HMS resolves capability from the HIVE connector, not the sibling -> user chose Option C: reflect the sibling's capability set onto the delegated schema, Trino table-redirection) and the S3 distribution-column full-parity choice; add a "Discovered follow-ups" section (the two out-of-seam org.apache.hudi VisibleForTesting imports that break the delete step; the partition-level FULL-analyze delta inherent to the plugin migration); expand the e2e-owed list. - hms-cutover-execution-plan: §2.3 all four seams marked LANDED (S1-S4 commit hashes); §2.5 W6 marked a verified false gap (no code); §5 e2e matrix grown with the coupling-seam rows. - HANDOFF: rolled forward to "all seams landed, next = clean-room adversarial review" (deliberately not started per the stop-before-review instruction); records this session's two full-parity decisions. Code is unchanged (docs only). Clean-room review of S1-S4 + W6 is the next session's first task. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 23 ++++---- ...e-coupling-seams-step-design-2026-07-10.md | 59 +++++++++++++------ .../hms-cutover-execution-plan-2026-07-10.md | 16 ++--- 3 files changed, 63 insertions(+), 35 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index ea743b5c7bf7ff..dbe6afd69afa7e 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -10,21 +10,22 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： - **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 -- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → **3 个 loud-break 耦合缝 + W6（本步，进行中）** → 剩余。 +- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → **4 个耦合缝(S1–S4) + W6 ✅（代码落地，欠净室复审）** → 剩余(复审 → 原子翻闸前置项)。 - **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 -**⭐ 本轮（2026-07-10 晚）= Phase 1 第三项「翻闸耦合缝」开工：HEAD 对齐侦察 + 3 处用户决策(全选逐位一致) + 第一缝落地。** -权威设计 = **`tasks/hive-coupling-seams-step-design-2026-07-10.md`**（**起步必读**；含 4 缝 HEAD 锚点 + 修法 + 用户决策 + TODO + e2e 欠账）。侦察 = `wf_dfe1cb86-df4`（4 缝读者 + 完备性审查；完备性审查确认 4 缝之外无未护 loud break，另抓出 auto-analyze 静默扩大缝）。 +**⭐ 本轮（2026-07-10 晚）= Phase 1 第三项「翻闸耦合缝」全部落地（4 缝 + W6），停在复审前。** +权威设计 = **`tasks/hive-coupling-seams-step-design-2026-07-10.md`**（**起步必读**；4 缝 TODO 已全勾 + 2 处发现的后续项 + e2e 欠账）。本轮侦察 = `wf_ecf9f292-892`（4 缝读者 + 完备性审查，HEAD 逐条证）。**用户本轮 2 处决策（全选 FULL PARITY）**：① iceberg-on-HMS 统计资格机制 = **做法C**（Hive 转交时反射兄弟连接器能力集，Trino 表重定向语义，顺带关 Top-N/嵌套列裁剪隐患）；② 分桶 Hive 采样 NDV 估算 = **做法2**（透传分桶列完全对齐）。 -- **W6 写路径 TCCL = 已核实为假警报，无需改代码**：iceberg-on-HMS 写委派穿过 iceberg 兄弟已绑定的 `TcclPinningConnectorContext`（`IcebergConnector.java:174`，线程无关）。只欠 e2e。可选：软化 `HiveConnector.java:206-208` 过度谨慎注释（纯文档）。 -- **用户 3 处决策（2026-07-10，全选 FULL PARITY）**：① 时间线函数 `hudi_meta` = **保留(改造连接器驱动)**（侦察推翻"删"倾向——4 个 p2 hudi 套件在用）；② Hive `ANALYZE … WITH SAMPLE` = **完整移植**（今天可用，不移植翻闸后会 `DdlException`）；③ 后台列自动统计 = **按表细分排除 hudi-on-HMS**（复刻旧 `HIVE||ICEBERG` 门）。 -- **✅ 已落地：S1「partition_values() 通用插件臂」** commit `166515cdc88`（3 个 fe-core 文件；镜像 `$partitions`：闸门放行 `PluginDrivenExternalCatalog`+`PLUGIN_EXTERNAL_TABLE`、加 `PluginDrivenExternalTable` 臂无 HMS 强转、`getTableColumns` 上提基类；新增 SPI `PluginDrivenExternalTable.getNameToPartitionValues`（**独立方法**、不动 live `getNameToPartitionItems` 保 paimon/iceberg 字节+成本不变）；`MetadataGenerator` 加 `PLUGIN_EXTERNAL_TABLE` 臂 + HMS/插件共用 `partitionValuesRows` 构建器保 `HIVE_DEFAULT_PARTITION`→NULL）。**对 paimon/iceberg 是加法(此前抛错的 TVF 现可用)**、对 hive 休眠至翻闸。fe-core BUILD SUCCESS + 0 checkstyle + import-gate 干净。功能校验 e2e-owed。 +- **✅ S1 partition_values** `166515cdc88`（上一轮）。 +- **✅ S4 auto-analyze 按表门 + 兄弟能力反射** `89c6f9454bb`：侦察坐实 iceberg-on-HMS 的能力从 **hive 连接器**解析(非兄弟)——单去 hive 连接器级 flag 会静默回归 iceberg-on-HMS。修法=`supportsColumnAutoAnalyze()`→`hasScanCapability`、去 hive 连接器级 flag(排除 hudi-on-HMS)、plain-hive 按表发标记、**委派分支 `reflectSiblingScanCapabilities` 把兄弟连接器能力集反射成按表标记**(做法C)。休眠。4 套件绿。 +- **✅ S2 hudi_meta 连接器驱动** `d8f2d01978a`：中立 `getMetadataTableRows` SPI(`List>`)+`SUPPORTS_METADATA_TABLE`+`HudiConnector` 全活时间线(TCCL pin)+双臂 `hudiMetadataResult`(HMS 臂下沉到 `HudiExternalMetaCache.getTimelineRows`)。**MetadataGenerator 摆脱 `org.apache.hudi`**。休眠。 +- **✅ S3 sample-analyze 完整移植** `8469a033abd`：`SUPPORTS_SAMPLE_ANALYZE`(plain-hive 按表)+`canSample`/`isSamplingPartition` 加臂+`PluginDrivenSampleAnalysisTask`(逐字移植 doSample/getSampleInfo/needLimit)+`ConnectorStatisticsOps.listFileSizes` SPI+`getChunkSizes` override+**分桶列透传**(`DISTRIBUTION_COLUMNS_KEY`)+`StatisticsAutoCollector` force-FULL 收窄 `&& !supportsSampleAnalyze()`。休眠。 +- **✅ W6 写路径 TCCL = 核实假警报**：兄弟自带 `TcclPinningConnectorContext`，无未固定路径。软化注释 `f53a71f5260`(纯文档)。 -**⭐ 下一步 = 本步剩余 3 缝 + 收尾（权威=`hive-coupling-seams-step-design-2026-07-10.md` TODO）：** -1. **S4 auto-analyze 按表门**（最小，但**先侦察**）：`PluginDrivenExternalTable.supportsColumnAutoAnalyze()` 由连接器级 `getCapabilities().contains(SUPPORTS_COLUMN_AUTO_ANALYZE)` 改为 `hasScanCapability(...)`（对 native iceberg/paimon 无变——它们仍连接器级声明）；hive 连接器从 `getCapabilities` 的 EnumSet 去掉该 flag，改在 `HiveConnectorMetadata.getTableSchema` 按表发标记（`PER_TABLE_CAPABILITIES_KEY`，precedent `:402-414` 的 TOPN 标记）。**隐藏深度须先查清**：翻闸后 iceberg-on-HMS / hudi-on-HMS 表的 `SUPPORTS_COLUMN_AUTO_ANALYZE` 是从 **hive 连接器**解析还是从**委派的兄弟连接器**解析？旧门 admit `dlaType==ICEBERG`。若 iceberg-on-HMS 已经从 iceberg 兄弟(连接器级声明)拿到，则 hive 只需对 plain-hive 发标记、对 hudi 不发，"排除 hudi"就落到"hudi 兄弟是否声明该 flag"。**动码前先在真实代码里把这个解析路径查清**（`HiveConnector` 的 3-way 路由 / 兄弟委派）。 -2. **S2 hudi_meta 连接器驱动**（保留）：加中立"元数据行"SPI（镜像 `ConnectorProcedureResult` 行返回）+ `HudiConnector` 实现（时间线数据已在连接器侧：`HudiMetaClientExecutor`/`getActiveTimeline().getInstants()`）；重写 `MetadataGenerator.hudiMetadataResult`（gate 改通用插件/能力型、去 `HMSExternalTable` 强转、fe-core 摆脱 `org.apache.hudi` timeline import `:128-129`）。休眠。委派须 pin TCCL。parity = 4 个 p2 套件行(e2e)。旧 body 无论如何删除步移除。 -3. **S3 sample-analyze 完整移植**（最大）：新 `ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE`（hive **按表**发给 plain-hive 表——旧 `dlaType==HIVE`，排除 iceberg/hudi-on-HMS；native iceberg/paimon 不发保持现拒绝）+ `AnalysisManager.canSample`/`AnalyzeTableCommand.isSamplingPartition` 加 `PluginDrivenExternalTable.supportsSampleAnalyze()` 臂（走 `hasScanCapability`）+ `PluginDrivenExternalTable.createAnalysisTask` 返回可抽样任务(移植 `HMSAnalysisTask.doSample`+`getSampleInfo`+`needLimit`) + `getChunkSizes` override(经新 `Connector` chunk-sizes SPI 拿原始字节长度，type-math 留 fe-core)。非休眠单测端到端(发真抽样 SQL)→ e2e-owed。铁律：能力**按表**(连接器级会误 admit iceberg/hudi-on-HMS)。 -4. **收尾**：（可选）软化 W6 注释；**净室对抗复审**跨全部 seam commit（memory `clean-room-adversarial-review-pref`）；把 e2e 欠账登记进 execution-plan §4；勾选设计文档 TODO；更新本 HANDOFF。 +**⭐ 下一步 = 净室对抗复审（本轮按用户要求"停在复审前"，未开工）：** +1. **净室对抗复审跨全部 seam commit（S1–S4 + W6）**：memory `clean-room-adversarial-review-pref`——多 agent 先独立判断、后交叉核对；重点核 ① S4 做法C 反射是否漏/误透能力(只应影响 `hasScanCapability` 消费者)、② S3 移植 doSample 字节等价 + `listFileSizes`/`getChunkSizes` TCCL 方向 + 分桶列大小写、③ S2 双臂无破 4 p2 套件(HMS 臂仍活)、④ 全部对 paimon/iceberg 字节+成本不变。修确认发现。 +2. **勾设计文档 TODO / 登记 e2e 欠账**：本轮已做（execution-plan §2.3/§2.5/§5 + 设计文档 TODO + 本 HANDOFF）。 +3. **2 处本轮发现的后续项（非 3 缝、勿丢，见设计文档「Discovered follow-ups」）**：① 删除步 build-break——`StatisticsAutoCollector.java:36`/`StatisticsCache.java:44` 误 import `org.apache.hudi.common.util.VisibleForTesting`(注解)，hudi 出 fe-core classpath 时会挂，删除步换 guava/doris `@VisibleForTesting`(现无害未改)；② 分区级 FULL analyze——`ExternalAnalysisTask.doFull` 不做旧 `HMSAnalysisTask.doPartitionTable` 分区级(所有插件表通性、非 S3 引入)，若要分区级外表 analyze 另起一步。 **⚠ 与本步解耦、仍待用户拍板的 3 处 D2 缓存复审项**（上一步遗留，见旧 HANDOFF/review2 报告；本轮未动）：① 自发 DROP/CREATE TABLE 不清连接器缓存(HIGH)；② 分区缓存容量单位悄变(MEDIUM)；③ paimon/iceberg `latestSnapshotCache` 现网同形删建洞(线上活性 bug)。下个 session 可择机连同上面 seam 一起或单独找用户定。 diff --git a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md index cffe20df6c2d80..68a9c32154490f 100644 --- a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md +++ b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md @@ -134,23 +134,48 @@ keep the connector-wide flag → unchanged. e2e-owed: hudi-on-HMS NOT auto-analy - [x] **S1** `partition_values` plugin arm (edits A–E + `getNameToPartitionValues` SPI). ✅ commit `166515cdc88`. fe-core BUILD SUCCESS + 0 checkstyle + import-gate clean. Functional test (paimon/ iceberg live rows; hive post-flip == legacy) = e2e-owed. **Additive for paimon/iceberg**, dormant hive. -- [ ] **S4** auto-analyze per-table gate (`supportsColumnAutoAnalyze` → `hasScanCapability`; hive emits - per-table marker, drops connector-wide flag). ⚠ **INVESTIGATE FIRST** (hidden depth): post-flip do - iceberg-on-HMS / hudi-on-HMS tables resolve `SUPPORTS_COLUMN_AUTO_ANALYZE` from the HIVE connector or - from their delegated sibling connector? Legacy admitted `dlaType==ICEBERG` too. If iceberg-on-HMS - already gets it via the iceberg sibling's connector-wide flag, then hive only needs to emit the marker - for plain-hive (and NOT hudi), and the "exclude hudi" fix reduces to: does the hudi sibling declare - it? Nail this before editing. Per-table marker precedent = `HiveConnectorMetadata.java:402-414` - (`SUPPORTS_TOPN_LAZY_MATERIALIZE`). -- [ ] **S2** `hudi_meta` connector-driven (neutral metadata-rows SPI + `HudiConnector` impl + rewrite - `hudiMetadataResult`, shed `org.apache.hudi` from fe-core). Dormant unit test for the plugin arm. -- [ ] **S3** sample-analyze full port (capability + per-table `supportsSampleAnalyze`/`isSamplingPartition` - arms + sample-capable plugin task + chunk-sizes SPI + hive marker/impl). -- [ ] (optional) soften `HiveConnector.java:206-208` comment (W6 doc-only). -- [ ] clean-room adversarial review over all seam commits; fix confirmed findings. -- [ ] update HANDOFF + this doc's checkboxes; record e2e-owed rows into execution-plan §4. +- [x] **S4** auto-analyze per-table gate. ✅ commit `89c6f9454bb`. **Recon RESOLVED the hidden depth:** + iceberg-on-HMS resolves capability from the **HIVE** connector, NEVER the iceberg sibling (proven at + HEAD by the completeness critic) — so dropping the hive connector-wide flag alone would silently + regress iceberg-on-HMS (legacy admitted `dlaType==ICEBERG`). Fix: `supportsColumnAutoAnalyze()` → + `hasScanCapability`; drop the hive connector-wide flag (de-admits hudi-on-HMS); emit the per-table + marker for plain-hive; **and (user chose Option C, full parity) reflect the OWNING sibling's + connector-wide capability set onto the delegated iceberg/hudi-on-HMS schema as a per-table marker** + (`HiveConnectorMetadata.reflectSiblingScanCapabilities`, Trino table-redirection semantics). This + restores iceberg-on-HMS auto-analyze AND closes the same-root-cause Top-N / nested-column-prune loss + for iceberg-on-HMS in one place; hudi-on-HMS (sibling declares neither) is correctly withheld. 0 + checkstyle, import-gate clean, 4 suites green. Parity e2e-owed. +- [x] **S2** `hudi_meta` connector-driven. ✅ commit `d8f2d01978a`. Neutral `getMetadataTableRows` SPI + (`List>` — TVF owns the schema, lighter than `ConnectorProcedureResult`) + + `SUPPORTS_METADATA_TABLE` + `HudiConnector` impl (full active timeline, TCCL-pinned) + dual-arm + `hudiMetadataResult` (HMS arm sources from the relocated `HudiExternalMetaCache.getTimelineRows`). + **MetadataGenerator sheds its two `org.apache.hudi` imports.** Dormant unit tests for the plugin arm. + Timeline-row parity e2e-owed (4 p2 suites, enableHudiTest). +- [x] **S3** sample-analyze full port. ✅ commit `8469a033abd`. `SUPPORTS_SAMPLE_ANALYZE` (per-table, + plain-hive only) + additive `canSample`/`isSamplingPartition` arms + `PluginDrivenSampleAnalysisTask` + (verbatim `doSample`/`getSampleInfo`/`needLimit`) + `ConnectorStatisticsOps.listFileSizes` SPI + + `getChunkSizes` override + **(user chose full parity) distribution-column port** (`DISTRIBUTION_COLUMNS_KEY`, + fe-core lowercases) + `StatisticsAutoCollector` force-FULL refined to `&& !supportsSampleAnalyze()`. + Sampling SQL round-trip + estimator + per-partition listing are e2e-owed. +- [x] (optional) soften `HiveConnector` W6 write-path TCCL comment (doc-only). ✅ commit `f53a71f5260`. +- [ ] **clean-room adversarial review over all seam commits (S1–S4 + W6); fix confirmed findings.** ← NEXT + (deliberately NOT started this session — stop-before-review). +- [x] update HANDOFF + this doc's checkboxes; record e2e-owed rows into execution-plan §4. + +## Discovered follow-ups (surfaced by recon/impl, NOT in the 3 seams — do-not-drop) +- **Delete-step build-break (out-of-seam):** `StatisticsAutoCollector.java:36` and `StatisticsCache.java:44` + import `org.apache.hudi.common.util.VisibleForTesting` (annotation-only, wrong package). Harmless now + (hudi still on fe-core's classpath via the delete-unit), but a compile break the moment the hudi jar + leaves fe-core at the delete step. Swap to the guava/doris `@VisibleForTesting` then. NOT fixed here + (unrelated to hudi_meta; surgical scope). +- **Partition-level FULL analyze (inherent to the plugin migration, not S3):** `ExternalAnalysisTask.doFull` + (which every flipped table incl. the new sample task inherits) does not do the legacy + `HMSAnalysisTask.doPartitionTable` partition-level analyze under `enablePartitionAnalyze`. This is a + property of ALL plugin tables (iceberg/paimon too), not introduced by S3; track for a full-analyze parity + pass if partition-level external analyze is required. ## e2e-owed (Phase 4, do-not-drop) partition_values over heterogeneous HMS == legacy hive rows; hudi_meta timeline rows == the 4 p2 suites; -hive ANALYZE WITH SAMPLE FULL-vs-SAMPLE stat assertions; auto-analyze admits hive/iceberg-on-HMS but not -hudi-on-HMS; W6 iceberg-on-HMS write no-CCE on bundled-AWS S3. +hive ANALYZE WITH SAMPLE FULL-vs-SAMPLE stat assertions (text + orc + partitioned + bucketed); auto-analyze +admits hive/iceberg-on-HMS but not hudi-on-HMS; iceberg-on-HMS regains Top-N lazy / nested-column prune via +the sibling-capability reflection; W6 iceberg-on-HMS write no-CCE on bundled-AWS S3. diff --git a/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md b/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md index ab0b8f73cf9629..0b6a3cc92e642a 100644 --- a/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md +++ b/plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md @@ -61,10 +61,12 @@ Design: `hms-event-pipeline-findings-2026-07-07.md` (Model B). **0% built** (gre ### 2.2 ⛔ D2 — connector-owned scan-side cache (retire the fe-core caches) Decision D2 (LOCKED 07-07) = connector-owned. **0% built.** All three fe-core caches present; `ExternalMetaCacheRouteResolver.java:66` still routes hms→HIVE+HUDI+ICEBERG caches. The hive connector must own scan-side caching (paimon/iceberg-native precedent) so that at the flip, routing collapsing to `ENGINE_DEFAULT` is *harmless*. Retire `HiveExternalMetaCache` / `HudiExternalMetaCache` / `IcebergExternalMetaCache` and the four routing `instanceof` gates (`ExternalMetaCacheRouteResolver:66`, `HiveExternalMetaCache:203/274`, `HudiExternalMetaCache:221`, `IcebergExternalMetaCache:234`) **with the flip set**. Also unblocks §2.6 below. -### 2.3 Coupling seams that break LOUD at the flip (must be dormant-staged) -- **`partition_values()` TVF** — no PluginDriven arm today → post-flip throws `"Catalog of type 'hms' is not allowed"` and `MetadataGenerator:2088-2092` casts to `HMSExternalTable`. Add a PluginDriven catalog+table arm to the gate + a `partitionValuesMetadataResultForPluginTable` path (mirror the already-done `$partitions`). `PartitionValuesTableValuedFunction.java:113,130`. -- **`hudi_meta()` / TIMELINE TVF** (NEW — not in the 07-07 doc; the hudi line added it) — post-flip `!(instanceof HMSExternalTable)` → returns *"not a hudi table"*, and its body depends on the deletion-unit hudi cache. **Decide: connector-driven timeline path, or drop the TVF.** `MetadataGenerator.java:459-483`. -- **`canSample` / `SUPPORTS_SAMPLE_ANALYZE`** (optional) — no capability today → post-flip hive `ANALYZE … WITH SAMPLE` silently falls back to FULL (correct, slower). Add `SUPPORTS_SAMPLE_ANALYZE` + a PluginDriven `canSample` arm, **or** accept the degrade. `AnalysisManager.java:1484-1485`. +### 2.3 Coupling seams that break LOUD at the flip (must be dormant-staged) — ✅ **ALL LANDED (dormant)** +> Detailed design + landing = `hive-coupling-seams-step-design-2026-07-10.md` (S1–S4 + W6). All chose FULL parity (user 2026-07-10). Each an independent dormant commit; clean-room review pending (deliberately not started). +- **`partition_values()` TVF** — ✅ **S1** `166515cdc88` (PluginDriven gate+arm + `getNameToPartitionValues` SPI, mirror `$partitions`). +- **`hudi_meta()` / TIMELINE TVF** — ✅ **S2** `d8f2d01978a` (KEEP, connector-driven: neutral `getMetadataTableRows` SPI + `SUPPORTS_METADATA_TABLE` + `HudiConnector` impl; MetadataGenerator sheds `org.apache.hudi`). +- **`canSample` / `SUPPORTS_SAMPLE_ANALYZE`** — ✅ **S3** `8469a033abd` (FULL port, not the degrade: capability + sample task + `listFileSizes` SPI + distribution-column parity). +- **auto-analyze per-table gate** — ✅ **S4** `89c6f9454bb` (drop hive connector-wide `SUPPORTS_COLUMN_AUTO_ANALYZE`; per-table marker; **sibling-capability reflection** at the delegation branch so iceberg-on-HMS keeps auto-analyze + regains Top-N/nested-prune, hudi-on-HMS excluded — Trino table-redirection semantics). ### 2.4 Deletion-unblocking extractions (→ RE-SEQUENCED to Phase 3 per the 2026-07-10 decision) > **Deferred to the deletion phase** (deletion is last). Post-flip the legacy classes still exist (dead-for-hms), so their live consumers keep compiling; these extractions are only needed at the moment of deletion. Listed here for completeness; do them in Phase 3, immediately before deleting the dirs. @@ -75,8 +77,8 @@ Three members of the deletion unit are consumed by **live surviving code** and m - `HiveSplit` branch in **live** `FileQueryScanNode.java:465` (remove the branch). - **`IcebergUtils` 6-member extraction** to a new fe-core SDK-free util (`ICEBERG_ROW_ID_COL`, `ICEBERG_LAST_UPDATED_SEQUENCE_NUMBER_COL`, `ICEBERG_ROW_LINEAGE_MIN_VERSION`, `isIcebergRowLineageColumn(String)`, `isIcebergRowLineageColumn(Column)`, `getEffectiveIcebergFormatVersion` with its 3 iceberg string constants inlined). **The target util does not exist at HEAD.** Repoint `BindExpression`, `CreateTableInfo:1145-1171`, `IcebergMergeCommand`, `IcebergUpdateCommand`. -### 2.5 W6 — write-path TCCL pin (verify-first, land dormant if needed) -`fe-connector-hive` has no `TcclPinningConnectorContext`; `HiveConnector.java:195` flags the write/DDL-thread pin as a separate flip-time concern. Iceberg-on-HMS *writes* run name-based reflection (`planWrite` TDataSink build, `getWritePartitioning`) on fe-core app-loader threads → classloader split-brain CCE risk. **Verify** whether the path is unpinned; if so add an fe-core write-path pin keyed off `provider.getClass().getClassLoader()`. Not dormant-testable → e2e-owed (Phase 4). +### 2.5 W6 — write-path TCCL pin (verify-first) — ✅ **VERIFIED FALSE GAP, NO CODE** +Recon confirmed the pin is already carried: an iceberg/hudi-on-HMS write is delegated to the sibling's per-handle write provider, and the sibling already wraps its write/commit in its own `TcclPinningConnectorContext` (`IcebergConnector` around `executeAuthenticated`, classloader-thread-independent). So there is **no unpinned fe-core write-path** at this seam. Over-cautious comment softened (`f53a71f5260`, doc-only). Iceberg-on-HMS write no-CCE stays **e2e-owed** (Phase 4). ### 2.6 `getNewestUpdateVersionOrTime` real max-partition-time (coupled to D2) Flipped plain-hive returns constant `0` (names-only `listPartitions` → every `lastModifiedMillis` UNKNOWN `-1` → filtered → `orElse(0L)`), so a **hive-backed SQL dictionary / MV auto-refresh never sees a newer source version → silently stale**. Fix needs the connector to surface a real max-partition modify time cheaply — **which needs D2 first**. Alternatively, sign off an explicit temporary regression + e2e assertion. `PluginDrivenMvccExternalTable.java:713-714`. @@ -120,7 +122,7 @@ The 07-07 §7 gate is **materially stale** (predates the whole hudi line, names - **Harness (i) — two-URLClassLoader routing fixture:** loads iceberg/hudi handles in a child-first loader and proves the gateway diverts each foreign handle to the correct sibling **with no CCE** across the loader split. Unit tests can't: test classpaths load everything on the app loader, so an illegal foreign-handle cast passes silently (the shipped `HiveConnectorThreeWayRoutingTest` is same-loader — proves only `ownsHandle` logic). - **Harness (ii) — heterogeneous-HMS docker e2e:** ONE `type=hms` catalog holding plain-hive + iceberg-on-HMS + hudi-on-HMS, each served by the right provider, mixed queries in one session correct. -**Consolidated e2e matrix** (every landed step's "⚠残留 e2e" note): transactional-hive read (full-ACID+insert-only in scope, full-ACID write hard-rejected); HMS event replay in sync post-flip; iceberg-on-HMS **INSERT/DELETE/MERGE/OVERWRITE/ALTER(14)/EXECUTE** == independent iceberg catalog (NET-NEW, signed 2026-07-08); MTMV freshness + time-travel over iceberg-on-HMS & hive base; hudi COW/MOR + hive-sync/non-hive-sync partition-value fidelity; hudi FOR TIME AS OF; hudi **@incr straddling-base-file** fixture; hudi schema-evolved reads (rename/reorder + **mixed-case nested struct must not SIGABRT**); hudi dropped-partition-set-at-instant; hudi MTMV refresh; hudi write/DDL/EXECUTE **fail-loud reject**; Kerberos-HMS smoke; `HmsGsonCompatReplayTest`. +**Consolidated e2e matrix** (every landed step's "⚠残留 e2e" note): transactional-hive read (full-ACID+insert-only in scope, full-ACID write hard-rejected); HMS event replay in sync post-flip; iceberg-on-HMS **INSERT/DELETE/MERGE/OVERWRITE/ALTER(14)/EXECUTE** == independent iceberg catalog (NET-NEW, signed 2026-07-08); MTMV freshness + time-travel over iceberg-on-HMS & hive base; hudi COW/MOR + hive-sync/non-hive-sync partition-value fidelity; hudi FOR TIME AS OF; hudi **@incr straddling-base-file** fixture; hudi schema-evolved reads (rename/reorder + **mixed-case nested struct must not SIGABRT**); hudi dropped-partition-set-at-instant; hudi MTMV refresh; hudi write/DDL/EXECUTE **fail-loud reject**; Kerberos-HMS smoke; `HmsGsonCompatReplayTest`; **coupling-seam rows (S1–S4, all dormant → live at flip):** `partition_values()` over heterogeneous HMS == legacy hive rows; `hudi_meta()` timeline rows == the 4 p2 suites (`timestamp/action/state/state_transition_time`, enableHudiTest); hive `ANALYZE … WITH SAMPLE` FULL-vs-SAMPLE stat assertions on **text + orc + partitioned + bucketed** plain-hive; background auto-analyze admits plain-hive + iceberg-on-HMS but **NOT** hudi-on-HMS; iceberg-on-HMS regains Top-N lazy + nested-column prune (via the S4 sibling-capability reflection) == an independent iceberg catalog. **4 rows are genuine CORRECTNESS risks the flip cannot ship without** (both critics ranked these above the docker run): 1. **hudi COW-@incr row-level `_hoodie_commit_time` filter** — INC-4 delivered the iron-rule-compliant fix (neutral `ConnectorExpression` reverse-converter + `CheckPolicy.collectConnectorSyntheticPredicates`, no fe-core source branch); the "possibly unfixable" §5.1 worry is **superseded**, but the straddling-base-file correctness is **e2e-owed**. From edae8f20bc29605bb6d9a278cb80823002213b36 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 17:05:10 +0800 Subject: [PATCH 183/334] [fix](catalog) Hive coupling-seams clean-room: hudi-on-HMS metadata-table (hudi_meta/TIMELINE) sibling divert + lock Clean-room adversarial review over the seam commits surfaced one HIGH cross-cutting defect between the hudi_meta connector-drive and the auto-analyze sibling-capability reflection: - reflectSiblingScanCapabilities copies the OWNING sibling's connector-wide capability set onto the delegated (hudi-on-HMS) table's per-table marker. HudiConnector declares {SUPPORTS_METADATA_TABLE}, so post-flip PluginDrivenExternalTable.supportsMetadataTable() returns true and MetadataGenerator's PLUGIN arm passes the "not a hudi table" gate and calls getMetadataTableRows("timeline"). - But HiveConnectorMetadata never OVERRODE getMetadataTableRows, while every other per-handle read guard-and-forwards to the sibling (getColumnHandles, getTableStatistics, estimateDataSizeByListingFiles, listFileSizes, listPartitions, ...). A foreign HudiTableHandle therefore fell through to the ConnectorMetadata SPI default (Collections.emptyList) instead of reaching HudiConnectorMetadata's real active-timeline producer. Result post-flip: hudi_meta()/TIMELINE on hudi-on-HMS returns OK-but-EMPTY vs the still-live HMS arm's real timeline -- a silent wrong result. The delegation test's EXPECTED_METHODS completeness lock omitted the method, so the missing divert was uncaught. Fix (mirrors listFileSizes/estimateDataSizeByListingFiles): - HiveConnectorMetadata.getMetadataTableRows override: forward a foreign handle to siblingMetadata(session, handle).getMetadataTableRows(...) (HudiConnectorMetadata already pins TCCL via its meta-client executor; fe-core/gateway add no pin), plain-hive returns empty (it has no metadata table). Iceberg-on-HMS is unaffected: IcebergConnector does not declare SUPPORTS_METADATA_TABLE, so its gate stays false and the arm still rejects. - HiveConnectorMetadataSiblingDelegationTest: drive getMetadataTableRows through the foreign-handle forwarding test, add it to EXPECTED_METHODS, record it in RecordingSiblingMetadata (SIBLING_TIMELINE_ROWS sentinel), and assert the returned rows are the sibling's -- so a future dropped divert fails loud. Also closes a Rule-9 test gap the same review flagged (LOW): the hudi-on-HMS auto-analyze WITHHOLDING (this seam's headline parity claim) had no representative test -- the existing fixtures used an iceberg-shape sibling (already carries auto-analyze) or an empty sibling (hits the isEmpty early-return), so a reflect-side over-add of SUPPORTS_COLUMN_AUTO_ANALYZE passed silently. Add foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling modeling the REAL hudi sibling (non-empty {SUPPORTS_METADATA_TABLE}, no auto-analyze): assert the marker carries SUPPORTS_METADATA_TABLE but NOT SUPPORTS_COLUMN_AUTO_ANALYZE / Top-N / nested-prune. Correct the misleading mutation comment on the empty-sibling test (it guards the isEmpty early-return, not hudi admission). Dormant until hms enters SPI_READY_TYPES; row-value parity stays e2e-owed (the 4 p2 hudi suites). fe-connector-hive BUILD SUCCESS, HiveConnectorMetadataSiblingDelegationTest 11/11, 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 18 +++++++ ...onnectorMetadataSiblingDelegationTest.java | 53 +++++++++++++++++-- 2 files changed, 67 insertions(+), 4 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index ff10cdfe1517c8..53ff791d885c8f 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -846,6 +846,24 @@ public List listFileSizes(ConnectorSession session, ConnectorTableHandle h } } + /** + * Engine-neutral rows for a connector metadata table (currently only the hudi commit timeline). A plain-hive + * table has no metadata table, so the hive branch returns nothing; a foreign (hudi-on-HMS) handle is diverted + * to the owning sibling connector, whose {@code getMetadataTableRows} produces the real timeline (and pins the + * TCCL itself). Without this divert a flipped hudi-on-HMS table would fall through to the SPI-default empty list + * even though {@link #reflectSiblingScanCapabilities} reflects the hudi sibling's {@code SUPPORTS_METADATA_TABLE} + * and so makes the {@code hudi_meta()} / TIMELINE gate pass — i.e. OK-but-empty instead of the real timeline. + * Mirrors the {@link #listFileSizes} / {@link #estimateDataSizeByListingFiles} per-handle divert. + */ + @Override + public List> getMetadataTableRows(ConnectorSession session, ConnectorTableHandle handle, + String kind) { + if (!(handle instanceof HiveTableHandle)) { + return siblingMetadata(session, handle).getMetadataTableRows(session, handle, kind); + } + return Collections.emptyList(); + } + /** * Sampling + summing + scale-up core of {@link #estimateDataSizeByListingFiles}, isolated from the * {@code FileSystem} I/O (injected as {@code sizeOf}) so the estimation math is unit-testable. Returns -1 diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 233f47d10aef58..4798f513f3123e 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -120,6 +120,7 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { md.getTableStatistics(null, foreignHandle); md.getColumnStatistics(null, foreignHandle, "c"); long size = md.estimateDataSizeByListingFiles(null, foreignHandle); + List> timelineRows = md.getMetadataTableRows(null, foreignHandle, "timeline"); Optional> filter = md.applyFilter(null, foreignHandle, null); List partNames = md.listPartitionNames(null, foreignHandle); md.listPartitions(null, foreignHandle, Optional.empty()); @@ -153,6 +154,9 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { "listPartitionNames must return the sibling's names"); Assertions.assertEquals(Collections.singletonList("snapshots"), sysTables, "iceberg-on-HMS system tables must resolve through the sibling (hive exposes none)"); + Assertions.assertSame(RecordingSiblingMetadata.SIBLING_TIMELINE_ROWS, timelineRows, + "getMetadataTableRows must return the sibling's timeline rows, not hive's empty default — a " + + "hudi-on-HMS hudi_meta()/TIMELINE read gets its rows from the hudi sibling post-flip"); // Handle-out methods must return the sibling's handle/result UNMODIFIED (a rewrap poisons a scan cast). Assertions.assertSame(siblingMetadata.filterResult, filter, "applyFilter must return the sibling result"); @@ -361,15 +365,46 @@ public void foreignHandleSchemaReflectsSiblingScanCapabilitiesAsPerTableMarker() @Test public void foreignHandleSchemaUnchangedWhenSiblingDeclaresNoCapabilities() { - // A hudi sibling declares no scan capabilities -> no marker is stamped -> the embedded table stays out of - // auto-analyze etc. (hudi-on-HMS parity: legacy StatisticsUtil.supportAutoAnalyze excluded it). MUTATION: - // unconditionally stamping a marker -> hudi-on-HMS would be wrongly admitted -> red here. + // A sibling declaring an EMPTY capability set hits the ownerCaps.isEmpty() early-return in + // reflectSiblingScanCapabilities -> the sibling schema is returned untouched -> no marker is stamped. This + // guards the empty-owner branch specifically; the real hudi-on-HMS withholding (a NON-empty sibling that + // lacks auto-analyze) is pinned by foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling below. + // MUTATION: dropping the isEmpty() early-return and stamping an (empty) marker unconditionally -> red here. HiveConnectorMetadata md = withSibling(); // RecordingSiblingConnector declares no capabilities ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); Assertions.assertNull(schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY), "no marker when the sibling declares no capabilities"); } + @Test + public void foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling() { + // The REAL hudi sibling declares a NON-EMPTY connector-wide set that does NOT include auto-analyze + // (HudiConnector.getCapabilities() = {SUPPORTS_METADATA_TABLE}), so it never takes the isEmpty() + // early-return — it goes through the copy loop. reflectSiblingScanCapabilities copies EXACTLY that set, so a + // flipped hudi-on-HMS table gains the metadata-table capability (hudi_meta()/TIMELINE works) but stays OUT of + // background column auto-analyze — legacy StatisticsUtil.supportAutoAnalyze excluded dlaType HUDI. This pins + // the copy-fidelity path that actually governs hudi withholding (the empty-sibling test only exercises the + // early-return). MUTATION: reflecting auto-analyze for any non-empty owner (or HudiConnector gaining the + // flag) -> the marker would contain SUPPORTS_COLUMN_AUTO_ANALYZE -> red here. + HiveConnectorMetadata md = new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext(), + SUPPLIER_MUST_NOT_BE_USED, SUPPLIER_MUST_NOT_BE_USED, + handle -> new CapabilityDeclaringSiblingConnector( + EnumSet.of(ConnectorCapability.SUPPORTS_METADATA_TABLE))); + + ConnectorTableSchema schema = md.getTableSchema(null, foreignHandle); + String csv = schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); + Assertions.assertNotNull(csv, "a non-empty hudi sibling must still stamp its declared capabilities"); + List names = Arrays.asList(csv.split(",")); + Assertions.assertTrue(names.contains(ConnectorCapability.SUPPORTS_METADATA_TABLE.name()), + "the hudi sibling's metadata-table capability must survive delegation (hudi_meta works post-flip)"); + Assertions.assertFalse(names.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE.name()), + "hudi-on-HMS must stay OUT of background auto-analyze (legacy excluded dlaType HUDI)"); + Assertions.assertFalse(names.contains(ConnectorCapability.SUPPORTS_TOPN_LAZY_MATERIALIZE.name()), + "hudi-on-HMS gains no Top-N lazy from a sibling that does not declare it"); + Assertions.assertFalse(names.contains(ConnectorCapability.SUPPORTS_NESTED_COLUMN_PRUNE.name()), + "hudi-on-HMS gains no nested-column prune from a sibling that does not declare it"); + } + private static void assertThrowsMessage(Executable exec, String expectedMessage) { DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, exec); Assertions.assertEquals(expectedMessage, e.getMessage(), @@ -419,12 +454,15 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata static final long SENTINEL_SNAPSHOT_ID = 99L; static final List SIBLING_PREDICATES = Collections.singletonList( new ConnectorColumnRef("sibling-pred", ConnectorType.of("STRING"))); + static final List> SIBLING_TIMELINE_ROWS = Collections.singletonList( + Arrays.asList("20260101000000000", "commit", "COMPLETED", "20260101000001000")); // The exact set + order of forwarded methods the foreign-handle test drives (a Rule-9 completeness lock: // dropping a guard, or adding one that should not forward, changes this list and fails the test). static final List EXPECTED_METHODS = Collections.unmodifiableList(Arrays.asList( "getTableSchema", "getColumnHandles", "getTableStatistics", "getColumnStatistics", - "estimateDataSizeByListingFiles", "applyFilter", "listPartitionNames", "listPartitions", + "estimateDataSizeByListingFiles", "getMetadataTableRows", + "applyFilter", "listPartitionNames", "listPartitions", "beginQuerySnapshot", "getTableFreshness", "getPartitionFreshnessMillis", "dropTable", "truncateTable", "getTableSchemaAtSnapshot", "getMvccPartitionView", "resolveTimeTravel", "applySnapshot", "getSyntheticScanPredicates", "applyRewriteFileScope", @@ -482,6 +520,13 @@ public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTa return SENTINEL_SIZE; } + @Override + public List> getMetadataTableRows(ConnectorSession session, ConnectorTableHandle handle, + String kind) { + calls.add("getMetadataTableRows"); + return SIBLING_TIMELINE_ROWS; + } + @Override public Optional> applyFilter(ConnectorSession session, ConnectorTableHandle handle, ConnectorFilterConstraint constraint) { From bc505f369d91e372a0d377eac53470e4b68ec45b Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 17:05:32 +0800 Subject: [PATCH 184/334] [test](catalog) Hive coupling-seams clean-room: pin sample/metadata capabilities out of hive connector-wide set Clean-room review flagged a Rule-9 guard hole (LOW; production set is correct today): PluginDrivenExternalTable.hasScanCapability checks the CATALOG connector-wide set FIRST, so any capability mistakenly added to HiveConnector.getCapabilities() applies to EVERY hms table (plain-hive + iceberg-on-HMS + hudi-on-HMS). The designated guard perTableScanCapabilitiesAreNotConnectorWide pinned TOPN_LAZY / NESTED_COLUMN_PRUNE / COLUMN_AUTO_ANALYZE out of the set but OMITTED the two capabilities the later seams added: - SUPPORTS_SAMPLE_ANALYZE is per-table (emitted in getTableSchema for plain-hive tables only); a connector-wide flag would over-admit iceberg/hudi-on-HMS to sampled ANALYZE (legacy gated dlaType==HIVE). - SUPPORTS_METADATA_TABLE reaches a hudi-on-HMS table only via reflectSiblingScanCapabilities (the hudi sibling declares it); hive declaring it connector-wide would make every hms table wrongly pass the hudi_meta()/TIMELINE gate. Add both assertFalse pins so declaring either connector-wide turns the guard red -- the exact mistake it exists to catch. Test-only; no production change. HiveConnectorCapabilitiesTest 3/3, 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorCapabilitiesTest.java | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java index f5546610ddd0bc..72482e3a0030c3 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorCapabilitiesTest.java @@ -86,5 +86,15 @@ public void perTableScanCapabilitiesAreNotConnectorWide() { // re-admits hudi-on-HMS -> red here. Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_COLUMN_AUTO_ANALYZE), "auto-analyze is a per-table marker (excludes hudi-on-HMS), not connector-wide"); + // SUPPORTS_SAMPLE_ANALYZE is likewise per-table (getTableSchema emits it for plain-hive tables only, + // any format). A connector-wide flag would over-admit iceberg/hudi-on-HMS to sampled ANALYZE, which + // legacy gated on dlaType==HIVE. MUTATION: declaring it connector-wide -> red here. + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_SAMPLE_ANALYZE), + "sample-analyze is a per-table marker (plain-hive only), not connector-wide"); + // SUPPORTS_METADATA_TABLE reaches a hudi-on-HMS table ONLY via reflectSiblingScanCapabilities (the hudi + // sibling declares it connector-wide); hive must NEVER declare it, or every hms table (incl. plain-hive and + // iceberg-on-HMS) would wrongly pass the hudi_meta()/TIMELINE gate. MUTATION: declaring it -> red here. + Assertions.assertFalse(caps.contains(ConnectorCapability.SUPPORTS_METADATA_TABLE), + "metadata-table reaches hudi-on-HMS via sibling reflection, never hive connector-wide"); } } From 6a13709a0ce60a5222004580644a9b1e839ccf1b Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 17:24:12 +0800 Subject: [PATCH 185/334] [fix](catalog) Hive coupling-seams clean-room: sampled-analyze listFileSizes fails loud on a listing error (parity) Clean-room review flagged a real error-semantics divergence (user chose fail-loud parity): HiveConnectorMetadata.listFileSizes swallowed a RuntimeException (DorisConnectorException / HiveDirectoryListingException) to Collections.emptyList(). That list feeds PluginDrivenSampleAnalysisTask.getSampleInfo for ANALYZE ... WITH SAMPLE, where an empty chunk list -> total == 0 -> scaleFactor 1.0, WHILE ExternalAnalysisTask still injects TABLESAMPLE(N) (getSampleHint fires whenever tableSample != null, independent of the scale factor). So a transient FS/HMS listing error during a sampled ANALYZE on a non-empty flipped plain-hive table made the sample scan read a fraction, multiply by 1.0, and persist silently-undercounted row_count/ndv/data_size with the task reporting SUCCESS. Legacy HMSExternalTable.getChunkSizes had no catch: the listing error propagated and failed the command loud, writing nothing. Fix (parity, minimal): drop the catch(RuntimeException)->emptyList in listFileSizes (keep the finally that restores the TCCL) so the listing error propagates through getChunkSizes -> getSampleInfo -> doSample -> the analyze task, failing the sampled ANALYZE exactly like legacy. A genuinely empty table still yields an empty list naturally (no throw), preserving the correct empty-table scaleFactor-1.0 path; the non-HIVE tableType guard / foreign-handle divert still return early. "statistics must not fail a query" governs user SELECTs / query planning (estimateDataSizeByListingFiles keeps its best-effort -1), NOT an explicit ANALYZE command. Test: listFileSizesPropagatesListingErrorAndRestoresClassLoader injects a throwing HiveFileListingCache (deterministic, no reliance on real FS behavior) and asserts listFileSizes propagates the error AND still restores the TCCL on the throw path (try/finally). Dormant until hms enters SPI_READY_TYPES; the sampling scale-factor round-trip stays e2e-owed. fe-connector-hive BUILD SUCCESS, HiveConnectorMetadataFileListStatsTest 11/11, 0 checkstyle violations. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../connector/hive/HiveConnectorMetadata.java | 14 +++---- ...iveConnectorMetadataFileListStatsTest.java | 40 +++++++++++++++++++ 2 files changed, 47 insertions(+), 7 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 53ff791d885c8f..28b07eb9fbb724 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -812,9 +812,13 @@ public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTa * are listed (iceberg/hudi-on-HMS are served by their own connectors via the sibling divert; a view has no * data files) — anything else returns empty. Lists EVERY partition (no {@link #STATS_PARTITION_SAMPLE_SIZE} * sampling, unlike {@link #estimateDataSizeByListingFiles}) because the fe-core sampler needs the individual - * file sizes to seed-shuffle and cumulate. Best-effort: any listing error degrades to empty, never throwing - * (statistics must not fail a query). Pins the TCCL to the plugin classloader for the {@code FileSystem} - * reflection, exactly like estimateDataSizeByListingFiles (the statistics thread is not pinned by fe-core). + * file sizes to seed-shuffle and cumulate. A listing error PROPAGATES here (unlike + * {@link #estimateDataSizeByListingFiles}'s best-effort {@code -1}): this backs an explicit + * {@code ANALYZE ... WITH SAMPLE}, and legacy {@code HMSExternalTable.getChunkSizes} failed the command loud + * rather than let the sampler collapse the scale factor to {@code 1.0} while {@code TABLESAMPLE} still fires + * (a silent stat undercount); a genuinely empty table still yields an empty list naturally. Pins the TCCL to + * the plugin classloader for the {@code FileSystem} reflection (the statistics thread is not pinned by + * fe-core), restored on the throw path by the finally. */ @Override public List listFileSizes(ConnectorSession session, ConnectorTableHandle handle) { @@ -837,10 +841,6 @@ public List listFileSizes(ConnectorSession session, ConnectorTableHandle h } } return sizes; - } catch (RuntimeException e) { - LOG.warn("Failed to list hive file sizes for {}.{} for sample analyze", - hiveHandle.getDbName(), hiveHandle.getTableName(), e); - return Collections.emptyList(); } finally { Thread.currentThread().setContextClassLoader(previous); } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java index cafc04264a12a8..3d7ea3dc8bf8c7 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.hadoop.conf.Configuration; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -152,6 +153,32 @@ public void publicEntryDegradesToMinusOneAndRestoresClassLoaderOnError() { } } + @Test + public void listFileSizesPropagatesListingErrorAndRestoresClassLoader() { + // ANALYZE ... WITH SAMPLE reads raw per-file sizes here. UNLIKE estimateDataSizeByListingFiles (best-effort + // -1 for query planning), a listing failure must PROPAGATE: legacy HMSExternalTable.getChunkSizes failed the + // sampled ANALYZE loud rather than let the sampler collapse the scale factor to 1.0 while TABLESAMPLE still + // fires (a silent stat undercount). The TCCL pin must still be restored on the throw path (try/finally). + // MUTATION: re-adding a catch -> Collections.emptyList swallows the error -> assertThrows red. + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new PartitionFakeHmsClient(Collections.emptyList()), Collections.emptyMap(), new FakeConnectorContext(), + () -> null, () -> null, handle -> null, new ThrowingFileListingCache()); + HiveTableHandle handle = unpartitioned("s3://wh/t"); + + ClassLoader marker = new URLClassLoader(new URL[0], + HiveConnectorMetadataFileListStatsTest.class.getClassLoader()); + ClassLoader original = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(marker); + try { + Assertions.assertThrows(RuntimeException.class, () -> metadata.listFileSizes(null, handle), + "a listing failure during sample analyze must fail loud, not degrade to empty"); + Assertions.assertSame(marker, Thread.currentThread().getContextClassLoader(), + "the TCCL pin must be restored on the throw path (try/finally)"); + } finally { + Thread.currentThread().setContextClassLoader(original); + } + } + @Test public void nonHiveTableTypeIsNotEstimated() { // A hudi/iceberg-on-HMS table is served by its own connector; the hive gateway must return -1 BEFORE @@ -164,6 +191,19 @@ public void nonHiveTableTypeIsNotEstimated() { .estimateDataSizeByListingFiles(null, hudiHandle)); } + /** A {@link HiveFileListingCache} whose listing always fails, to prove listFileSizes propagates (not swallows). */ + private static final class ThrowingFileListingCache extends HiveFileListingCache { + ThrowingFileListingCache() { + super(Collections.emptyMap()); + } + + @Override + public List listDataFiles(String dbName, String tableName, String location, + Configuration conf) { + throw new RuntimeException("simulated listing failure"); + } + } + /** * {@link HmsClient} double serving a fixed set of partition names, each with a synthetic location * {@code s3://wh/t/}. {@code listPartitionNames} echoes the names; {@code getPartitions} builds an From 06cdf84b33b1e7bd04ab7ffb36cf9f09934a7e03 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 17:28:27 +0800 Subject: [PATCH 186/334] [doc](catalog) Hive coupling-seams clean-room review DONE: 4 findings fixed; HANDOFF -> D2 cache decisions + atomic-flip prereqs Clean-room adversarial review over the seam commits (S1-S4 + W6) complete: 27-agent multi-agent clean-room (independent finders judging code cold vs each commit's pre-image + legacy, 3-lens adversarial verify, completeness critic), 4 findings survived (2 refuted), all fixed + verified. Record the outcome in the seam design doc (final TODO checked) and roll the HANDOFF forward: the "stop-before-review" debt is cleared; the next Phase-1 remaining work is the 3 pending D2 cache-review decisions (last external input before the flip) then the atomic-flip prerequisites. Fixes committed: a5015800abd (HIGH: hudi_meta/TIMELINE sibling divert + lock, + S4 withholding test gap), a63ab391171 (LOW: capabilities-guard sample/metadata pins), d85c16f2cda (LOW: listFileSizes fail-loud parity, user decision). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 25 ++++++++--------- ...e-coupling-seams-step-design-2026-07-10.md | 27 +++++++++++++++++-- 2 files changed, 36 insertions(+), 16 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index dbe6afd69afa7e..e6fc2521593b9f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -10,24 +10,21 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： - **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 -- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → **4 个耦合缝(S1–S4) + W6 ✅（代码落地，欠净室复审）** → 剩余(复审 → 原子翻闸前置项)。 +- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → **净室对抗复审 ✅（本轮，4 发现全修）** → 剩余(3 处 D2 缓存复审项待拍板 → 原子翻闸前置项)。 - **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 -**⭐ 本轮（2026-07-10 晚）= Phase 1 第三项「翻闸耦合缝」全部落地（4 缝 + W6），停在复审前。** -权威设计 = **`tasks/hive-coupling-seams-step-design-2026-07-10.md`**（**起步必读**；4 缝 TODO 已全勾 + 2 处发现的后续项 + e2e 欠账）。本轮侦察 = `wf_ecf9f292-892`（4 缝读者 + 完备性审查，HEAD 逐条证）。**用户本轮 2 处决策（全选 FULL PARITY）**：① iceberg-on-HMS 统计资格机制 = **做法C**（Hive 转交时反射兄弟连接器能力集，Trino 表重定向语义，顺带关 Top-N/嵌套列裁剪隐患）；② 分桶 Hive 采样 NDV 估算 = **做法2**（透传分桶列完全对齐）。 +**⭐ 本轮（2026-07-10）= 翻闸耦合缝的净室对抗复审全部完成，4 个确认发现已修复+验证（"停在复审前"的欠账已还清）。** +复审 = `wf_498114c4-3e1`（7 独立读者冷判"每 commit 前像 + 老实现"、每发现 3 lens 对抗核实、完备性审查；27 agent，2 refuted）。权威设计 = **`tasks/hive-coupling-seams-step-design-2026-07-10.md`**（**起步必读**；TODO 全勾，含 4 发现明细 + Discovered follow-ups + e2e 欠账）。**修复 = 3 个独立 commit，fe-connector-hive 全绿 0 checkstyle：** -- **✅ S1 partition_values** `166515cdc88`（上一轮）。 -- **✅ S4 auto-analyze 按表门 + 兄弟能力反射** `89c6f9454bb`：侦察坐实 iceberg-on-HMS 的能力从 **hive 连接器**解析(非兄弟)——单去 hive 连接器级 flag 会静默回归 iceberg-on-HMS。修法=`supportsColumnAutoAnalyze()`→`hasScanCapability`、去 hive 连接器级 flag(排除 hudi-on-HMS)、plain-hive 按表发标记、**委派分支 `reflectSiblingScanCapabilities` 把兄弟连接器能力集反射成按表标记**(做法C)。休眠。4 套件绿。 -- **✅ S2 hudi_meta 连接器驱动** `d8f2d01978a`：中立 `getMetadataTableRows` SPI(`List>`)+`SUPPORTS_METADATA_TABLE`+`HudiConnector` 全活时间线(TCCL pin)+双臂 `hudiMetadataResult`(HMS 臂下沉到 `HudiExternalMetaCache.getTimelineRows`)。**MetadataGenerator 摆脱 `org.apache.hudi`**。休眠。 -- **✅ S3 sample-analyze 完整移植** `8469a033abd`：`SUPPORTS_SAMPLE_ANALYZE`(plain-hive 按表)+`canSample`/`isSamplingPartition` 加臂+`PluginDrivenSampleAnalysisTask`(逐字移植 doSample/getSampleInfo/needLimit)+`ConnectorStatisticsOps.listFileSizes` SPI+`getChunkSizes` override+**分桶列透传**(`DISTRIBUTION_COLUMNS_KEY`)+`StatisticsAutoCollector` force-FULL 收窄 `&& !supportsSampleAnalyze()`。休眠。 -- **✅ W6 写路径 TCCL = 核实假警报**：兄弟自带 `TcclPinningConnectorContext`，无未固定路径。软化注释 `f53a71f5260`(纯文档)。 +- **✅ HIGH（翻闸前必修）** `a5015800abd`：hudi_meta/TIMELINE 兄弟转发缺失——`reflectSiblingScanCapabilities` 把 hudi 兄弟的 `SUPPORTS_METADATA_TABLE` 反射进按表标记→翻闸后 gate 通过，但 `HiveConnectorMetadata` **无 `getMetadataTableRows` override**(其它逐柄读都 guard-and-forward)→ foreign `HudiTableHandle` 掉进 SPI 默认空表 = 静默空时间线(非真实)。补 guard-and-forward override(仿 `listFileSizes`)+完整性锁(`EXPECTED_METHODS`/`RecordingSiblingMetadata`/行断言)+真 hudi 形态 withholding 测试(顺带修 S4 测试盲区)。iceberg-on-HMS 不受影响。 +- **✅ LOW(测试盲区)** `a63ab391171`：能力守卫补 `SUPPORTS_SAMPLE_ANALYZE`/`SUPPORTS_METADATA_TABLE` 两 pin(误设连接器级会静默过度接纳 iceberg/hudi-on-HMS)。 +- **✅ LOW(真错，用户拍板"响亮失败"对齐老实现)** `d85c16f2cda`：`listFileSizes` 吞列文件错误到空→采样 `scaleFactor` 塌成 1.0 但 `TABLESAMPLE` 仍触发→静默低估统计、任务报成功；老实现响亮失败。去 catch(留 finally 复原 TCCL)让错误照传 + 确定性传播单测(注入抛错 cache)。 +- **S1 partition_values + W6 = CLEAN**（无存活发现；paimon/iceberg 字节+成本不变已核）。 -**⭐ 下一步 = 净室对抗复审（本轮按用户要求"停在复审前"，未开工）：** -1. **净室对抗复审跨全部 seam commit（S1–S4 + W6）**：memory `clean-room-adversarial-review-pref`——多 agent 先独立判断、后交叉核对；重点核 ① S4 做法C 反射是否漏/误透能力(只应影响 `hasScanCapability` 消费者)、② S3 移植 doSample 字节等价 + `listFileSizes`/`getChunkSizes` TCCL 方向 + 分桶列大小写、③ S2 双臂无破 4 p2 套件(HMS 臂仍活)、④ 全部对 paimon/iceberg 字节+成本不变。修确认发现。 -2. **勾设计文档 TODO / 登记 e2e 欠账**：本轮已做（execution-plan §2.3/§2.5/§5 + 设计文档 TODO + 本 HANDOFF）。 -3. **2 处本轮发现的后续项（非 3 缝、勿丢，见设计文档「Discovered follow-ups」）**：① 删除步 build-break——`StatisticsAutoCollector.java:36`/`StatisticsCache.java:44` 误 import `org.apache.hudi.common.util.VisibleForTesting`(注解)，hudi 出 fe-core classpath 时会挂，删除步换 guava/doris `@VisibleForTesting`(现无害未改)；② 分区级 FULL analyze——`ExternalAnalysisTask.doFull` 不做旧 `HMSAnalysisTask.doPartitionTable` 分区级(所有插件表通性、非 S3 引入)，若要分区级外表 analyze 另起一步。 - -**⚠ 与本步解耦、仍待用户拍板的 3 处 D2 缓存复审项**（上一步遗留，见旧 HANDOFF/review2 报告；本轮未动）：① 自发 DROP/CREATE TABLE 不清连接器缓存(HIGH)；② 分区缓存容量单位悄变(MEDIUM)；③ paimon/iceberg `latestSnapshotCache` 现网同形删建洞(线上活性 bug)。下个 session 可择机连同上面 seam 一起或单独找用户定。 +**⭐ 下一步（本轮未开工，Phase 1 剩余 = 最后一块外部输入 → 翻闸前置项）：** +1. **3 处 D2 缓存复审项待用户拍板**（更早遗留，见 review2 报告；本轮未动）：① 自发 DROP/CREATE TABLE 不清连接器缓存(HIGH)；② 分区缓存容量单位悄变(MEDIUM)；③ paimon/iceberg `latestSnapshotCache` 现网同形删建洞(线上活性 bug)。**这是翻闸前 fe-core 建设唯一未决的外部输入**——下个 session 起步先找用户定这 3 项(中文讲清背景+选项，见 memory `ask-user-explain-in-chinese-first`)。 +2. **原子翻闸前置项**（Phase 2 集：`SPI_READY += hms`、GSON `HMSExternalTable`→`PluginDrivenMvccExternalTable` 重映射、4 守卫改接新子系统、死臂/删除排序）——见 execution-plan §2/§3；D2 三项定完即可开工。 +3. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失(所有插件表通性、非本轮引入)。 **⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步(读/写-拒/schema-evolution/缓存/跨加载器委派/本步 4 缝)只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + `hive-coupling-seams-step-design` e2e-owed 段 + memory `hms-iceberg-delegation-needs-e2e`）。删除排序最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 diff --git a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md index 68a9c32154490f..3e48d398f87475 100644 --- a/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md +++ b/plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md @@ -158,8 +158,31 @@ keep the connector-wide flag → unchanged. e2e-owed: hudi-on-HMS NOT auto-analy fe-core lowercases) + `StatisticsAutoCollector` force-FULL refined to `&& !supportsSampleAnalyze()`. Sampling SQL round-trip + estimator + per-partition listing are e2e-owed. - [x] (optional) soften `HiveConnector` W6 write-path TCCL comment (doc-only). ✅ commit `f53a71f5260`. -- [ ] **clean-room adversarial review over all seam commits (S1–S4 + W6); fix confirmed findings.** ← NEXT - (deliberately NOT started this session — stop-before-review). +- [x] **clean-room adversarial review over all seam commits (S1–S4 + W6); fix confirmed findings.** ✅ + Multi-agent clean-room (`wf_498114c4-3e1`: 7 independent finders judging code cold vs each commit's pre-image + + legacy, 3-lens adversarial verify per finding, completeness critic; 27 agents). 4 findings survived (2 refuted). + All fixed + verified (fe-connector-hive BUILD SUCCESS, 0 checkstyle): + - **HIGH (S2+S4 cross-cutting)** `a5015800abd`: `reflectSiblingScanCapabilities` reflects the hudi sibling's + `SUPPORTS_METADATA_TABLE` onto the delegated per-table marker → post-flip `supportsMetadataTable()`==true and + the `hudi_meta()`/TIMELINE gate PASSES, but `HiveConnectorMetadata` had NO `getMetadataTableRows` override + (every other per-handle read guard-and-forwards) → a foreign `HudiTableHandle` fell through to the SPI-default + empty list = OK-but-EMPTY timeline vs the still-live HMS arm's real one. Fix = add the guard-and-forward + override (mirrors listFileSizes) + drive it in the delegation completeness-lock (`EXPECTED_METHODS` + + `RecordingSiblingMetadata` + row assertion). iceberg-on-HMS unaffected (no `SUPPORTS_METADATA_TABLE`). + - **LOW (S4 test gap)** `a5015800abd`: hudi-on-HMS auto-analyze WITHHOLDING (the headline parity claim) had no + representative test — fixtures used an iceberg-shape sibling (already carries auto-analyze) or an empty sibling + (isEmpty early-return), so a reflect-side over-add passed silently. Added + `foreignHandleSchemaWithholdsAutoAnalyzeFromRealHudiSibling` (real hudi shape {METADATA_TABLE}, no auto-analyze) + + corrected the misleading empty-sibling mutation comment. + - **LOW (S3 test gap)** `a63ab391171`: capabilities guard omitted `SUPPORTS_SAMPLE_ANALYZE` / + `SUPPORTS_METADATA_TABLE` connector-wide pins → declaring either connector-wide would silently over-admit + iceberg/hudi-on-HMS. Added both assertFalse pins. + - **LOW (S3, real error-path)** `d85c16f2cda` (**user chose fail-loud parity**): `listFileSizes` swallowed a + listing `RuntimeException` to empty → sample `scaleFactor` collapses to 1.0 while `TABLESAMPLE` still fires → + silent stat undercount with the task SUCCESS; legacy `HMSExternalTable.getChunkSizes` failed loud. Fix = drop + the catch (keep the finally TCCL-restore) so the error propagates like legacy; a genuinely empty table stays + natural-empty. Deterministic propagation unit test added (injects a throwing `HiveFileListingCache`). + - **S1 (`partition_values`) + W6 = CLEAN** (no surviving findings; paimon/iceberg byte+cost invariance confirmed). - [x] update HANDOFF + this doc's checkboxes; record e2e-owed rows into execution-plan §4. ## Discovered follow-ups (surfaced by recon/impl, NOT in the 3 seams — do-not-drop) From 98f563fd3652eb7f3147a7a2f94c7c0eedd03bcd Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 18:05:53 +0800 Subject: [PATCH 187/334] [doc](catalog) Connector-cache invalidation follow-ups: step design (D1 fe-core hook + D2 per-partition keying) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second cache clean-room review raised 4 invalidation findings; user signed off two fixes (2026-07-10). Design doc grounds them at HEAD 1d9a5e61473 (recon wf_daed84c1-4ea): - §2.1 (REFRESH -> iceberg/hudi sibling) already FIXED at HEAD -> no action. - D1 (path A): fix "Doris-issued DROP/CREATE never invalidates the connector cache" once at the fe-core generic PluginDrivenExternalCatalog DDL seam -> covers dormant hive (§2.2) AND the LIVE paimon/iceberg drop+recreate stale-pin bug (§3) in one connector-agnostic change (Trino CachingHiveMetastore self-invalidation parity). Plus paimon PaimonSchemaAtMemo gains invalidate() (its narrow time-travel hole). - D2 (option per-partition): restructure CachingHmsClient partition-object cache from request-LIST keying to per-partition VALUE keying so cap=100000 again bounds 100000 partition objects (Trino + legacy parity; no shared-framework/weigher change). Correctness proven parse-fidelity-independent (misparse -> reload, never wrong). Decomposition F1/F1b (live) -> F2 (dormant); unified clean-room re-review after. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- ...nvalidation-followups-design-2026-07-10.md | 216 ++++++++++++++++++ 1 file changed, 216 insertions(+) create mode 100644 plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md diff --git a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md new file mode 100644 index 00000000000000..b37226ed9c00ab --- /dev/null +++ b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md @@ -0,0 +1,216 @@ +# Connector-cache invalidation follow-ups — step design (2026-07-10) + +> Follow-up to the connector-owned-cache clean-room re-reviews +> (`plan-doc/reviews/hive-connector-cache-cleanroom-review-2026-07-10.md` + +> `...-review2-2026-07-10.md`). Three defects the second review surfaced needed user sign-off; this doc +> is the implementer's starting point for the two the user chose to fix now. HEAD `1d9a5e61473`, branch +> `catalog-spi-11-hive`. Every line number below was re-verified against HEAD by a 5-agent recon +> (`wf_daed84c1-4ea`) — trust HEAD, re-confirm before editing. + +--- + +## 0. What this step does (and what it deliberately does NOT) + +The second clean-room review raised four cache-invalidation findings. Their HEAD status + disposition: + +| review finding | HEAD status | disposition | +|---|---|---| +| §2.1 REFRESH not forwarded to the built iceberg/hudi siblings | **already FIXED** (`forEachBuiltSibling` wired into `HiveConnector.invalidateTable/invalidateDb/invalidateAll/invalidatePartition`, `HiveConnector.java:296/322/342/365/409`) | no action | +| §2.2 Doris-issued DROP/CREATE never invalidate the connector caches (hive, dormant-until-flip) | CONFIRMED present | **FIX 1** (this step) | +| §3 same-shape LIVE bug in paimon/iceberg (drop+recreate serves a stale snapshot/schema pin) | CONFIRMED present, **LIVE today** | **FIX 1 + FIX 1b** (this step) | +| §2.3 partition-object cache bounds by request-LIST count, not partition count (hive, dormant) | CONFIRMED present | **FIX 2** (this step) | + +**User sign-off (2026-07-10):** +- **D1 → fix the self-DDL/drop+recreate invalidation once at the fe-core generic layer** (covers dormant + hive AND live paimon/iceberg in one connector-agnostic change). *Rejected*: per-plugin self-invalidation + (more surface, and would defer the live bug or fan it across three connectors). +- **D2 → restructure the partition-object cache to per-partition keying** (aligns with BOTH Trino and legacy + Doris; no shared-framework change). *Rejected*: a weigher (touches the paimon/iceberg-bundled + `fe-connector-cache`), a smaller default, or comment-only. + +Non-goals (recorded so they are not silently dropped): the review's §2.4 test-adequacy inserts +(`createClient` wrap seam, public-hook-with-built-client) — optional cheap insurance, not required by D1/D2; +event-driven incremental invalidation (Model B); deleting legacy fe-core caches (final deletion phase). + +--- + +## 1. FIX 1 — fe-core generic DDL invalidation hook (D1) + +### 1.1 The bug (HEAD-verified) +`PluginDrivenExternalCatalog`'s schema-level DDL overrides never reach `connector.invalidate*`: +- `dropTable` (`:536-596`) ends at `metadata.dropTable` + `logDropTable` + `unregisterTable` (`:593-594`); + the view branch (`:565-575`) ends at `dropView` + `unregisterTable` (`:572`). +- `createTable` (`:380-443`) ends at `metadata.createTable` + `logCreateTable` + `resetMetaCacheNames` + (`:439`). +- `dropDb` (`:500-525`) ends at `metadata.dropDatabase` + `logDropDb` + `unregisterDatabase` (`:523`). + +`unregisterTable`/`unregisterDatabase`/`resetMetaCacheNames` touch only the fe-core ENGINE caches +(`ExtMetaCacheMgr`), never the connector-owned caches. The ONLY paths that reach `connector.invalidate*` +are the three REFRESH verbs (`RefreshManager.refreshTableInternal:245-257` REFRESH TABLE — also reached by +TRUNCATE and ALTER via `afterExternalDdl`; `refreshDbInternal:119-129` REFRESH DATABASE; +`onRefreshCache:252-261` REFRESH CATALOG). So a Doris-issued `DROP TABLE t; CREATE TABLE t (...)` (or the +DB equivalent) keeps serving the dropped object's cached metadata/snapshot until the 24h TTL. + +Two victims, different severity: +- **hive** — dormant (`"hms"` not in `SPI_READY_TYPES`); latent until the flip. +- **paimon/iceberg** — **LIVE**: `IcebergLatestSnapshotCache` / `PaimonLatestSnapshotCache` (key = plain + `(db,table)`, value = pinned `(snapshotId[,schemaId])`, 86400s **access**-based TTL) pin the DROPPED + table's snapshot onto the recreated same-name table until the TTL (kept alive by continuous querying). + +### 1.2 Trino parity (per the user's request to consult Trino — recon-confirmed) +Trino's `CachingHiveMetastore` self-invalidates synchronously after every `createTable`/`dropTable`/ +`dropDatabase`/`replaceTable`/`renameTable`/`update*Statistics` (a `finally`-block +`invalidateTable`/`invalidateDatabase`). D1 puts the same eager invalidation at the natural Doris seam — +the generic `PluginDrivenExternalCatalog` DDL entry — so it lands once for every SPI connector instead of +per-plugin. + +### 1.3 The change +Add, immediately after each successful mutation (using the already-non-null `connector` field these methods +already dereference, and the REMOTE names — the exact form `RefreshManager` passes and the connectors key +on): +- `dropTable` table branch (after `metadata.dropTable`, `:590`) → + `connector.invalidateTable(dorisTable.getRemoteDbName(), dorisTable.getRemoteName())`. +- `dropTable` view branch (after `dropView`, `:570`) → same call (harmless no-op if the connector has no + entry for a view; keeps the two branches uniform). +- `createTable` (after `metadata.createTable`, `:427`) → + `connector.invalidateTable(db.getRemoteName(), createTableInfo.getTableName())`. The table name is NOT + remote-resolved (parity with the existing code: a new table has no local→remote mapping). +- `dropDb` (after `metadata.dropDatabase`, `:519`) → `connector.invalidateDb(db.getRemoteName())`. + +**`dropTable` is the load-bearing call** for the drop+recreate bug (it clears the stale pin so the later +`CREATE`'s exists-probe and the next `SELECT` both go live). `createTable`-invalidation is uniform +belt-and-suspenders (one line, harmless — invalidating a just-created table only forces the next read +live). **`createDb` is intentionally NOT hooked**: a brand-new database has no table-keyed connector +entries to clear that `dropDb` did not already clear, and Doris routes the db-name-list refresh through +`resetMetaCacheNames` already. Documented, not asked. + +### 1.4 Behavior / cost analysis (this is an INTENTIONAL live change — NOT byte-neutral) +D1 deliberately changes live paimon/iceberg behavior (that is the §3 fix). It is *strictly beneficial and +bounded*: +- **Correctness**: invalidating a dropped/created table's connector entry is always correct — the entry is + either stale (drop) or absent (create). No path reads a "must-stay-pinned" value across a DROP/CREATE. +- **Cost**: one `Cache.invalidate`/`asMap().keySet().removeIf` per DDL — O(1)/O(cache-size), off the query + hot path (DDLs are rare). No new RPC, no force-build (`connector` is already initialized here). +- **No-cache connectors** (jdbc/es and any without caches): `Connector.invalidateTable/invalidateDb` are + no-op SPI defaults → inert. +- **TCCL**: mirrors `RefreshManager`'s existing `getConnector().invalidateTable(...)` call, which pins + nothing — `invalidate*` runs the plugin's own method (no reflection-by-name), so no TCCL pin is needed + (memory `catalog-spi-plugin-tccl-classloader-gotcha` applies to name-reflection, not direct dispatch). +- **hive**: dormant (no `HiveConnector` built pre-flip) — inert until the flip, then armed. + +### 1.5 Tests +- fe-core (`PluginDrivenExternalCatalog` DDL): a fake `Connector` recording `invalidateTable/invalidateDb` + calls; assert `dropTable` (table + view branches) / `createTable` / `dropDb` each invoke it with the + REMOTE db/table names, and `createDb` does NOT. Assert a no-cache connector path is a no-op (default SPI). +- iceberg / paimon (live): after `invalidateTable(db,t)` the next `beginQuerySnapshot` re-pins (drop the + cached `(db,t)` entry). See FIX 1b for the paimon schema-memo half. + +--- + +## 2. FIX 1b — paimon `PaimonSchemaAtMemo` invalidation (completes §3 for paimon) + +### 2.1 The gap +`PaimonConnector.invalidateTable`/`invalidateAll` (`PaimonConnector.java:233/240`) clear only +`latestSnapshotCache`. The second connector-owned cache, `PaimonSchemaAtMemo` (`PaimonSchemaAtMemo.java`, a +plain `ConcurrentHashMap`, key = `(db,table,sysTable,branch,schemaId)`), +has **no invalidate method at all** — cleared only by its `maxSize` valve or a connector rebuild (REFRESH +CATALOG). A drop+recreate that reuses a `schemaId` (e.g. schema 0) with different content yields a stale +time-travel hit even after FIX 1 routes `invalidateTable` to the connector. + +### 2.2 The change +- Add `PaimonSchemaAtMemo.invalidate(String db, String table)` → `cache.keySet().removeIf(k -> + k matches (db,table))` (add a package-private `MemoKey.matches(db,table)` mirroring its equals fields), and + `invalidateAll()` → `cache.clear()`. Pure `ConcurrentHashMap` ops; no framework change. +- Wire `PaimonConnector.invalidateTable` → also `schemaAtMemo.invalidate(dbName, tableName)`; + `invalidateAll` → also `schemaAtMemo.invalidateAll()`. + +### 2.3 Cost / correctness +The memo value is immutable and only *skips* a re-read on a hit; dropping entries only forces a re-read +(the pre-memo behavior) — never a stale/wrong value. The keyspace is tiny; `removeIf` scan is negligible. +Iceberg needs no analog: `IcebergManifestCache` is path-keyed (unique paths, drop+recreate cannot collide) +and its `invalidateAll` (REFRESH CATALOG) already drops it; the `latestSnapshotCache` half is covered by +FIX 1 through `IcebergConnector.invalidateTable` (already clears it, `IcebergConnector.java:412`). + +### 2.4 Tests +paimon unit: populate the memo via `getOrLoad`, call `invalidateTable(db,t)`, assert the `(db,t)` entries +are gone and other-table entries survive; `invalidateAll` clears all. + +--- + +## 3. FIX 2 — per-partition keying of the hive partition-object cache (D2) + +### 3.1 The bug (HEAD-verified) +`CachingHmsClient.partitionsCache` (`CachingHmsClient.java:111`) is +`MetaCacheEntry>`: `PartitionsKey` (`:381-422`) = `(db, table, +requested-name-LIST)` and the value is the WHOLE list as ONE entry. `DEFAULT_PARTITION_CAPACITY=100000` +(`:105`) with a comment claiming "partition objects" parity — but the shared `CacheFactory` has only +`maximumSize`, no weigher (`CacheFactory.java:109`), so 100000 now bounds **distinct request-lists**, each +holding arbitrarily many partition objects (and duplicating them across overlapping lists). Legacy +`HiveExternalMetaCache` keyed ONE partition object per entry (`:121`, cap +`max_hive_partition_cache_num`); **Trino likewise keys per-partition** (`Cache`, flat +`maximumSize`). The list-keyed shape is unique to this cache and under-bounds FE heap. + +### 3.2 The design — value-keyed per-partition entries (matches Trino + legacy; no framework change) +The scan caller (`HiveScanPlanProvider.convertPartitions:332-349`) and every other `getPartitions` caller +(`HiveConnectorMetadata:933/1018/1176/1205`, `HiveWritePlanProvider:257`, `HiveConnectorTransaction:503`) +consume the returned `HmsPartitionInfo` **as a set** — they iterate and read `getValues()`/`getParameters()` +/`getLocation()` and never rely on result order or on a 1:1 name↔result correspondence (the delegate +`get_partitions_by_names` never guaranteed order/cardinality anyway). So: + +- Change `partitionsCache` to `MetaCacheEntry`, `PartitionKey = (db, table, + List values)` — keyed by the partition's OWN values. +- `getPartitions(db, table, partNames)`: + 1. for each requested name, parse it to values (`toPartitionValues`) and `getIfPresent((db,table,values))`; + hits go straight into the result, unparseable/missing names go into a `missNames` list; + 2. if `missNames` non-empty, `delegate.getPartitions(db,table,missNames)` → for each returned info, + `put((db,table, info.getValues()), info)` (keyed by the ACTUAL values — always correct) and add to the + result. +- **Correctness is independent of parse fidelity**: a store always uses the partition's real values; a + lookup misparse simply misses → the name falls to `missNames` → the delegate reloads it → correct result, + only no cache benefit for that (rare, escaped-value) name. Distinct partitions have distinct values, so no + false hit; a requested name with no partition is omitted by the delegate (parity, no negative caching). +- This is the SAME values-keyed correlation `HiveConnectorTransaction:518` already uses + (`partitionsByValues.get(HiveWriteUtils.toPartitionValues(name))`). + +### 3.3 The name→values parser +`HiveWriteUtils.toPartitionValues` (fe-connector-hive) is the byte-faithful port but lives in a module +`fe-connector-hms` cannot import (hms is below hive). Source options at implementation: a small private +helper in `CachingHmsClient` mirroring `HiveWriteUtils.toPartitionValues` (split on `/`, `=`, unescape via +inlined `unescapePathName`), OR `org.apache.hadoop.hive.common.FileUtils` (already a hms-module dependency — +`HmsEventParser` uses `FileUtils.makePartName`). Prefer the FileUtils route to avoid duplicating the +unescape table, unless it complicates the decorator; either is correct (misparse = perf-only). + +### 3.4 What stays the same +- `flush(db,table)` / `flushDb(db)` / `flushAll` (`:164-185`) keep using `invalidateIf(key -> + key.matches(db,table))` / `matchesDb(db)` / `invalidateAll` — `PartitionKey` still carries `(db,table)`, + so the invalidation interface is unchanged (and stays correct under FIX 1's new callers). +- `DEFAULT_PARTITION_CAPACITY=100000` now correctly bounds **100000 partition objects** (legacy semantics); + fix the comment to say so. +- The other three caches (`table`/`partition_names`/`column_stats`) are untouched. +- Dormant: no `HiveConnector`/`CachingHmsClient` is built for a live catalog pre-flip. + +### 3.5 Tests +`CachingHmsClientTest` additions: (a) two overlapping name-lists share per-partition entries (second call +hits, delegate called only for the new names); (b) a requested name with no partition is omitted and not +negative-cached; (c) `flush(db,table)` drops that table's partition entries only; (d) capacity bounds +partition COUNT (put N>cap distinct partitions → size ≤ cap); (e) an escaped-value name still returns the +correct partition (via reload) — no wrong/dropped result. + +--- + +## 4. Decomposition — independent commits +| step | what | module | live? | +|---|---|---|---| +| **F1** | fe-core `PluginDrivenExternalCatalog` DDL → `connector.invalidateTable/invalidateDb` + tests | fe-core | **live** (paimon/iceberg §3) + arms hive | +| **F1b** | `PaimonSchemaAtMemo.invalidate/invalidateAll` + wire into `PaimonConnector` + tests | fe-connector-paimon | **live** | +| **F2** | `CachingHmsClient` partition cache → per-partition value keying + tests + comment | fe-connector-hms | dormant | + +Order: F1 → F1b (both live, review together) → F2 (dormant). Each is its own commit with tests, checkstyle +0, import gate net. **After all land: unified clean-room adversarial re-review** (F1/F1b touch live +paimon/iceberg — per practice + memory `plugindriven-mvcc-table-is-live-not-dormant` / `clean-room-adversarial-review-pref`). + +## 5. e2e debt (heterogeneous-HMS docker; on top of the review's list) +- Drop+recreate freshness: `DROP TABLE t; CREATE TABLE t (new schema/loc); SELECT` sees the NEW table with + no REFRESH — for paimon + iceberg NOW (live), and for hive post-flip. +- `DROP DATABASE d; CREATE DATABASE d; ...` likewise sees no stale table entries. +- paimon time-travel after drop+recreate reusing a schemaId returns the new schema (schema-memo cleared). From d60833483a9f5d01726c5031447b3b3752b95ec0 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 18:15:36 +0800 Subject: [PATCH 188/334] [fix](catalog) Connector cache: Doris-issued DROP/CREATE invalidates the connector's own cache (D1, fe-core generic hook) The generic PluginDrivenExternalCatalog DDL overrides never reached connector.invalidate* (only the three REFRESH verbs did), so a Doris-issued `DROP TABLE t; CREATE TABLE t (...)` (or the DB equivalent) kept serving the dropped object's connector-owned cache until the 24h TTL. Dormant for hive (latent until the flip) but a LIVE bug for paimon/iceberg: their latest-snapshot cache (access-based 24h TTL) pins the DROPPED table's snapshot/schema onto the recreated same-name table. Fix once at the fe-core generic seam (connector-agnostic; Trino CachingHiveMetastore self-invalidation parity), keyed by the REMOTE names RefreshManager already uses: - dropTable (table + view branches) -> connector.invalidateTable [load-bearing] - createTable -> connector.invalidateTable [belt-and-suspenders] - dropDb -> connector.invalidateDb - createDb is intentionally NOT hooked (a fresh db has no table-keyed entries dropDb did not already clear). No-cache connectors (jdbc/es) get a no-op SPI default. Intentional live change (NOT byte-neutral): strictly beneficial (invalidating a dropped/ created entry is always correct), O(1) off the DDL path, no new RPC/TCCL. Completes the iceberg half of the live drop+recreate bug (IcebergConnector.invalidateTable already clears its latest-snapshot cache); paimon's schema-memo half lands in the next commit. PluginDrivenExternalCatalogDdlRoutingTest +1 (createDb-no-invalidate) + assertions on drop/create/dropDb (remote names) and the abort path; 56 pass, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../PluginDrivenExternalCatalog.java | 21 ++++++++++++ ...inDrivenExternalCatalogDdlRoutingTest.java | 33 +++++++++++++++++++ 2 files changed, 54 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index cf41447ab0cc8b..0026dd34d8b7b8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -425,6 +425,12 @@ public boolean createTable(CreateTableInfo createTableInfo) throws UserException } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } + // Drop any stale connector-owned cache entry for this name before the new table goes live + // (belt-and-suspenders with the DROP path, which is the load-bearing invalidation for drop+recreate). + // Connector-agnostic: invalidateTable is a no-op SPI default; hive/iceberg/paimon drop their own + // per-table caches (metastore/file-listing, latest-snapshot pin). The table name is intentionally NOT + // remote-resolved (a new table has no local->remote mapping — parity with the create request + editlog). + connector.invalidateTable(db.getRemoteName(), createTableInfo.getTableName()); org.apache.doris.persist.CreateTableInfo persistInfo = new org.apache.doris.persist.CreateTableInfo( getName(), @@ -517,6 +523,12 @@ public void dropDb(String dbName, boolean ifExists, boolean force) throws DdlExc } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } + // Drop the connector's own caches for every table in this db so a subsequent same-name CREATE + // DATABASE and the next reads go live rather than serving dropped tables up to the connector TTL. + // Connector-agnostic (no-op SPI default); keyed by the REMOTE db name, mirroring + // RefreshManager.refreshDbInternal. (createDb is intentionally NOT hooked: a brand-new db has no + // table-keyed connector entries that this dropDb did not already clear.) + connector.invalidateDb(db.getRemoteName()); // Edit log + cache invalidation intentionally use the LOCAL name: followers replay the // persisted DropDbInfo and the on-FE cache is keyed by local name (follower-replay parity). Env.getCurrentEnv().getEditLog().logDropDb(new DropDbInfo(getName(), dbName)); @@ -568,6 +580,9 @@ public void dropTable(String dbName, String tableName, boolean isView, boolean i } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } + // Uniform with the table branch: drop the connector's own caches for this name (harmless no-op + // for a view, which carries no snapshot pin). Keyed by the REMOTE names. + connector.invalidateTable(dorisTable.getRemoteDbName(), dorisTable.getRemoteName()); Env.getCurrentEnv().getEditLog().logDropTable(new DropInfo(getName(), dbName, tableName)); getDbForReplay(dbName).ifPresent(d -> d.unregisterTable(tableName)); LOG.info("finished to drop view {}.{}.{}", getName(), dbName, tableName); @@ -588,6 +603,12 @@ public void dropTable(String dbName, String tableName, boolean isView, boolean i } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } + // Drop the connector's own caches for this table (paimon/iceberg latest-snapshot pin, hive + // metastore + file-listing) so a subsequent same-name CREATE and the next read go live rather than + // serving the dropped table up to the connector TTL — the load-bearing fix for drop+recreate. + // Connector-agnostic (no-op SPI default); keyed by the REMOTE db/table names the connector caches + // under, mirroring RefreshManager.refreshTableInternal. + connector.invalidateTable(dorisTable.getRemoteDbName(), dorisTable.getRemoteName()); // Edit log and cache invalidation deliberately use the LOCAL db/table names for // follower-replay consistency; only the connector-bound names are remote-resolved. Env.getCurrentEnv().getEditLog().logDropTable(new DropInfo(getName(), dbName, tableName)); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java index c24ca47681034f..ee2172b51af60b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -136,6 +136,18 @@ public void testCreateDbRoutesToConnectorAndInvalidatesCache() throws Exception "createDb must invalidate the catalog db-name cache (legacy afterCreateDb parity)"); } + @Test + public void testCreateDbDoesNotInvalidateConnectorCache() throws Exception { + catalog.createDb("db1", false, new HashMap<>()); + + // WHY (Rule 9): createDb is DELIBERATELY not hooked to connector.invalidate* — a brand-new database + // has no table-keyed connector cache entries to clear that a prior dropDb did not already clear (the + // db-name-list is refreshed via resetMetaCacheNames, asserted above). This pins that scoping choice: + // a mutation that adds connector.invalidateDb/invalidateTable to createDb turns this red. + Mockito.verify(connector, Mockito.never()).invalidateDb(Mockito.any()); + Mockito.verify(connector, Mockito.never()).invalidateTable(Mockito.any(), Mockito.any()); + } + @Test public void testCreateDbIfNotExistsShortCircuitsWhenDbExists() throws Exception { catalog.dbNullableResult = Mockito.mock(ExternalDatabase.class); @@ -311,6 +323,10 @@ public void testDropDbResolvesRemoteNameRoutesAndUnregisters() throws Exception "edit-log DropDbInfo must carry the LOCAL db name for follower replay"); Assertions.assertEquals("db1", catalog.unregisteredDb, "cache invalidation must use the LOCAL db name"); + // WHY (Rule 9): the connector's own caches for every table in this db must be dropped on DROP + // DATABASE with the REMOTE db name (mirrors RefreshManager.refreshDbInternal). A mutation dropping + // the call — or passing the LOCAL "db1" — makes this red. + Mockito.verify(connector).invalidateDb("REMOTE_DB1"); } // ==================== DROP TABLE ==================== @@ -357,6 +373,12 @@ public void testDropTableResolvesRemoteNamesRoutesAndUnregisters() throws Except "cache invalidation must look up the LOCAL db name"); Mockito.verify(replayDb).unregisterTable("t1"); Mockito.verify(db, Mockito.never()).unregisterTable(Mockito.anyString()); + // WHY (Rule 9): the connector's OWN caches (paimon/iceberg latest-snapshot pin, hive metastore + + // file-listing) must be dropped on DROP TABLE with the REMOTE names, so a subsequent same-name + // CREATE + read go live instead of serving the dropped table up to the connector TTL (the LIVE + // paimon/iceberg drop+recreate stale-pin fix). A mutation dropping the call — or passing the LOCAL + // "db1"/"t1" — makes this verify red. + Mockito.verify(connector).invalidateTable("DB1", "TBL1"); } @Test @@ -446,6 +468,10 @@ public void testDropTableWrapsConnectorException() { DdlException ex = Assertions.assertThrows(DdlException.class, () -> catalog.dropTable("db1", "t1", false, false, false, false, false, false)); Assertions.assertTrue(ex.getMessage().contains("boom")); + // WHY (Rule 9 / Rule 12): a remote drop failure must abort BEFORE the cache is invalidated — the + // invalidate sits AFTER the successful metadata.dropTable, so the connector cache stays consistent + // with the (unchanged) remote. A mutation moving invalidateTable ahead of the mutation goes red. + Mockito.verify(connector, Mockito.never()).invalidateTable(Mockito.any(), Mockito.any()); } // B2: DROP on a flipped iceberg VIEW must route to metadata.dropView (mirroring legacy @@ -487,6 +513,8 @@ public void testDropTableRoutesViewToDropViewAndUnregisters() throws Exception { Assertions.assertEquals("db1", catalog.lastGetDbForReplayArg, "cache invalidation must look up the LOCAL db name"); Mockito.verify(replayDb).unregisterTable("v1"); + // The view branch drops the connector caches too (uniform with the table branch), keyed by REMOTE names. + Mockito.verify(connector).invalidateTable("DB1", "V1"); } @Test @@ -672,6 +700,11 @@ public void testCreateTableInvalidatesDbCacheUsingLocalNames() throws UserExcept Assertions.assertEquals("db1", catalog.lastGetDbForReplayArg, "cache invalidation must look up the LOCAL db name"); Mockito.verify(replayDb).resetMetaCacheNames(); + // WHY (Rule 9): CREATE TABLE also drops any stale connector cache entry for the new name + // (belt-and-suspenders with the DROP path). Keyed by the REMOTE db name "DB1" but the + // (non-remote-resolved) table name "t1" — a mutation flipping the table name to remote, or + // dropping the call, makes this red. + Mockito.verify(connector).invalidateTable("DB1", "t1"); } } From 1a3f4b34764c3417c6a0ba534752e20306954aad Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 18:24:21 +0800 Subject: [PATCH 189/334] [fix](catalog) Connector cache: hive partition-object cache keys per-partition, not per request-list (D2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CachingHmsClient.partitionsCache keyed the FULL requested-name-LIST and stored the whole List as ONE entry, so DEFAULT_PARTITION_CAPACITY=100000 bounded distinct request-LISTS (each holding arbitrarily many partitions, duplicated across overlapping lists) rather than the 100000 partition OBJECTS the comment claimed — FE heap could grow far past legacy's bound under sliding-predicate workloads. Legacy HiveExternalMetaCache and Trino CachingHiveMetastore both key per-partition. Restructure to one entry PER PARTITION, keyed by (db, table, partition-values): - getPartitions looks up each requested name (parsed to values via FileUtils, matching the write path's HiveWriteUtils.toPartitionValues) and fetches only the misses in ONE delegate round-trip, storing each returned partition under its OWN values. - Correctness is parse-fidelity-independent: a STORE is always keyed by the partition's real values, so a lookup-name misparse (rare escaped value) merely re-fetches — never a wrong or dropped partition. Callers consume the result as a SET (no order / 1:1 name-correspondence reliance; the delegate get_partitions_by_names never guaranteed either — verified all 7 call-sites). flush/flushDb/invalidateAll unchanged (PartitionKey still carries db/table); capacity now truly bounds partition objects. Dormant (no HiveConnector built pre-flip). CachingHmsClientTest: list-keying test replaced by per-partition sharing + missing-partition-not-negative-cached + parse-divergence-safety; the fake now returns per-name partition values. 15 pass, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../doris/connector/hms/CachingHmsClient.java | 101 +++++++++++++--- .../connector/hms/CachingHmsClientTest.java | 109 +++++++++++++++--- 2 files changed, 179 insertions(+), 31 deletions(-) diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index b89e9a60454cf5..963eedd3237ced 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -20,6 +20,8 @@ import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.cache.MetaCacheEntry; +import org.apache.hadoop.hive.common.FileUtils; + import java.io.IOException; import java.util.ArrayList; import java.util.Collections; @@ -50,10 +52,13 @@ *

  • {@code listPartitionNames} — keyed by {@code (db, table, maxParts)} → partition-name list. Real * callers pass the unbounded {@code maxParts}, so this is effectively one entry per table; keeping * {@code maxParts} in the key keeps a bounded request from ever being served a fuller list.
    • - *
  • {@code getPartitions} — keyed by {@code (db, table, requested-name-list)} → partition list. This is - * RPC-argument granularity: a different requested set is a separate entry (no cross-request - * assembly). {@link HmsPartitionInfo} carries {@code transient_lastDdlTime} in its parameters, which - * a later step reads through this cache for the table max-modify-time.
    • + *
  • {@code getPartitions} — one entry PER PARTITION, keyed by {@code (db, table, partition-values)} → + * {@link HmsPartitionInfo}. A bulk request looks up each requested name (parsed to its values) and + * fetches only the misses in a single delegate call, storing each returned partition under its OWN + * values — so overlapping requests SHARE partition entries and the capacity bounds partition OBJECTS + * (legacy {@code HiveExternalMetaCache} / Trino {@code CachingHiveMetastore} shape), not request-lists. + * {@link HmsPartitionInfo} carries {@code transient_lastDdlTime} in its parameters, which a later step + * reads through this cache for the table max-modify-time.
    • *
  • {@code getTableColumnStatistics} — keyed by {@code (db, table, requested-column-list)} → the * (possibly sparse or empty) stats list. Same RPC-argument granularity; the empty-list "no stats" * result is a legitimate cached value (only {@code null} loads are skipped). This is the planner @@ -108,7 +113,7 @@ public class CachingHmsClient implements HmsClient { private final HmsClient delegate; private final MetaCacheEntry tableCache; private final MetaCacheEntry> partitionNamesCache; - private final MetaCacheEntry> partitionsCache; + private final MetaCacheEntry partitionsCache; private final MetaCacheEntry> columnStatsCache; public CachingHmsClient(HmsClient delegate, Map properties) { @@ -147,8 +152,67 @@ public List listPartitionNames(String dbName, String tableName, int maxP @Override public List getPartitions(String dbName, String tableName, List partNames) { - return partitionsCache.get(new PartitionsKey(dbName, tableName, partNames), - key -> delegate.getPartitions(key.dbName, key.tableName, key.partNames)); + if (partNames == null || partNames.isEmpty()) { + return Collections.emptyList(); + } + // Per-partition assembly (Trino CachingHiveMetastore / legacy HiveExternalMetaCache shape): serve each + // requested partition from its own entry and fetch only the misses in ONE delegate round-trip, so + // overlapping requests share partition objects and the capacity bounds partition OBJECTS, not + // request-lists. Correctness is independent of name-parse fidelity: a LOOKUP is keyed by the requested + // name parsed to values, but a STORE is always keyed by the partition's OWN values, so a name whose + // parse diverges (a rare escaped value) simply misses and is re-fetched — never a wrong or dropped + // partition. Callers consume the result as a SET (they never rely on order or 1:1 name↔result + // correspondence — the delegate get_partitions_by_names never guaranteed either). + List result = new ArrayList<>(partNames.size()); + List missNames = null; + for (String name : partNames) { + HmsPartitionInfo hit = + partitionsCache.getIfPresent(new PartitionKey(dbName, tableName, toPartitionValues(name))); + if (hit != null) { + result.add(hit); + } else { + if (missNames == null) { + missNames = new ArrayList<>(); + } + missNames.add(name); + } + } + if (missNames != null) { + for (HmsPartitionInfo info : delegate.getPartitions(dbName, tableName, missNames)) { + partitionsCache.put(new PartitionKey(dbName, tableName, info.getValues()), info); + result.add(info); + } + } + return result; + } + + /** + * Splits a Hive partition name ("c1=a/c2=b") into its ordered values ("a", "b"), unescaping each via + * Hive's {@code FileUtils} (already a hms-module dependency — {@code HmsEventParser} uses it). Semantics + * match the write path's {@code HiveWriteUtils.toPartitionValues}, so scan and write correlate partitions + * identically. Only used to build the per-partition LOOKUP key: a parse that diverges from the stored + * partition's own values just misses and re-fetches (never a wrong/dropped partition), so this is a + * hit-rate optimization, not a correctness dependency. + */ + private static List toPartitionValues(String partitionName) { + List values = new ArrayList<>(); + int start = 0; + while (true) { + while (start < partitionName.length() && partitionName.charAt(start) != '=') { + start++; + } + start++; + int end = start; + while (end < partitionName.length() && partitionName.charAt(end) != '/') { + end++; + } + if (start > partitionName.length()) { + break; + } + values.add(FileUtils.unescapePathName(partitionName.substring(start, end))); + start = end + 1; + } + return values; } @Override @@ -378,19 +442,20 @@ public int hashCode() { } } - static final class PartitionsKey { + static final class PartitionKey { private final String dbName; private final String tableName; - // Order-sensitive, defensively copied: same requested list (same order) → hit; a different set or - // order → separate entry. Simple and correct (no partition-name reconstruction from values). - private final List partNames; + // ONE partition's ordered values (defensively copied). The cache stores one entry per partition keyed + // by these values (legacy / Trino per-partition shape), so the capacity bounds partition OBJECTS and + // overlapping requests share entries rather than duplicating partitions across request-list keys. + private final List values; - PartitionsKey(String dbName, String tableName, List partNames) { + PartitionKey(String dbName, String tableName, List values) { this.dbName = dbName; this.tableName = tableName; - this.partNames = partNames == null + this.values = values == null ? Collections.emptyList() - : Collections.unmodifiableList(new ArrayList<>(partNames)); + : Collections.unmodifiableList(new ArrayList<>(values)); } boolean matches(String db, String table) { @@ -406,18 +471,18 @@ public boolean equals(Object o) { if (this == o) { return true; } - if (!(o instanceof PartitionsKey)) { + if (!(o instanceof PartitionKey)) { return false; } - PartitionsKey that = (PartitionsKey) o; + PartitionKey that = (PartitionKey) o; return Objects.equals(dbName, that.dbName) && Objects.equals(tableName, that.tableName) - && Objects.equals(partNames, that.partNames); + && Objects.equals(values, that.values); } @Override public int hashCode() { - return Objects.hash(dbName, tableName, partNames); + return Objects.hash(dbName, tableName, values); } } diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java index d2346392824d18..04894c5261017a 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -25,8 +25,10 @@ import java.util.Arrays; import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; import java.util.List; import java.util.Map; +import java.util.Set; /** * Tests {@link CachingHmsClient}: the caching decorator over an {@link HmsClient}. @@ -114,24 +116,79 @@ public void listPartitionNamesCachesByDbTableAndMaxParts() { // ---- getPartitions ---- @Test - public void getPartitionsCachesByRequestedNameList() { + public void getPartitionsSharesPerPartitionEntriesAcrossRequests() { RecordingHmsClient delegate = new RecordingHmsClient(); CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); - List names = Arrays.asList("p=1", "p=2"); - List a = cache.getPartitions("db", "t", names); - List b = cache.getPartitions("db", "t", new ArrayList<>(names)); - // WHY: same requested set+order hits even if the caller passes a fresh list instance. - Assertions.assertSame(a, b); + // First request loads BOTH partitions in one delegate round-trip and caches each PER PARTITION. + List a = cache.getPartitions("db", "t", Arrays.asList("p=1", "p=2")); + Assertions.assertEquals(2, a.size()); Assertions.assertEquals(1, delegate.getPartitionsCalls); - // WHY: a different requested set is a distinct entry (RPC-argument granularity). + // WHY (Rule 9 / the D2 fix): an OVERLAPPING subset request must be served entirely from the shared + // per-partition entries — no new delegate call. The OLD list-keyed cache re-fetched any distinct + // request list (this was `getPartitionsCalls == 2` here); a mutation reverting to list keying — + // storing the whole list under a request-name-list key — makes this re-fetch and go red. cache.getPartitions("db", "t", Arrays.asList("p=1")); - Assertions.assertEquals(2, delegate.getPartitionsCalls); + Assertions.assertEquals(1, delegate.getPartitionsCalls, + "p=1 is served from the shared per-partition entry (no re-fetch)"); + + // WHY: order-independent too (the old list key was order-sensitive and re-loaded on a reversed list); + // both partitions are already cached, so a reversed request still hits. + List rev = cache.getPartitions("db", "t", Arrays.asList("p=2", "p=1")); + Assertions.assertEquals(2, rev.size()); + Assertions.assertEquals(1, delegate.getPartitionsCalls, "reversed order still hits the shared entries"); + + // WHY: only a genuinely new partition triggers a delegate fetch — and ONLY for the miss (p=1 stays + // cached), proving misses are fetched in one round-trip while hits are served locally. + cache.getPartitions("db", "t", Arrays.asList("p=1", "p=3")); + Assertions.assertEquals(2, delegate.getPartitionsCalls, "only the new p=3 is fetched; p=1 stays cached"); + Assertions.assertEquals(Arrays.asList("p=3"), delegate.lastGetPartitionsArg, + "the delegate is asked for the MISS only, not the whole requested list"); + } - // WHY: the key is order-sensitive by design (no reorder assembly) — reversed order re-loads. - cache.getPartitions("db", "t", Arrays.asList("p=2", "p=1")); - Assertions.assertEquals(3, delegate.getPartitionsCalls); + @Test + public void getPartitionsOmitsMissingPartitionWithoutNegativeCaching() { + RecordingHmsClient delegate = new RecordingHmsClient(); + delegate.absentPartitionNames.add("p=9"); // HMS has no such partition -> omitted from the response + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + List r = cache.getPartitions("db", "t", Arrays.asList("p=1", "p=9")); + // WHY: a non-existent partition is OMITTED (get_partitions_by_names parity), never fabricated. + Assertions.assertEquals(1, r.size(), "the absent partition is omitted, not fabricated"); + Assertions.assertEquals(1, delegate.getPartitionsCalls); + + // WHY (Rule 9): the missing p=9 must NOT be negative-cached — a later request re-attempts it (so once + // the partition is created + REFRESH'd it is picked up). Only p=9 re-fetches; p=1 stays cached. + cache.getPartitions("db", "t", Arrays.asList("p=1", "p=9")); + Assertions.assertEquals(2, delegate.getPartitionsCalls, + "the absent partition is re-attempted (no negative cache); p=1 still hits"); + Assertions.assertEquals(Arrays.asList("p=9"), delegate.lastGetPartitionsArg); + } + + @Test + public void getPartitionsStaysCorrectWhenParsedNameDivergesFromStoredValues() { + // Pathological: the delegate returns a partition whose values do NOT match the requested name's parse + // (models a value the name-parse cannot round-trip). The decorator keys the STORE by the partition's + // OWN values but the LOOKUP by the parsed name, so they never match -> the partition is re-fetched + // every time. WHY (Rule 9 / Rule 12): this pins the safety contract — a parse divergence degrades to a + // reload (perf), NEVER a wrong or dropped partition. A mutation that keyed the STORE by the parsed name + // instead would make the lookup "hit" a mis-keyed entry (or drop the partition) -> the size/values + // asserts go red. + RecordingHmsClient delegate = new RecordingHmsClient(); + delegate.forcedValues = Arrays.asList("EXOTIC"); // stored values != parse("p=1") == ["1"] + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + List r1 = cache.getPartitions("db", "t", Arrays.asList("p=1")); + Assertions.assertEquals(1, r1.size(), "the partition is returned even though its values diverge from the name"); + Assertions.assertEquals(Arrays.asList("EXOTIC"), r1.get(0).getValues()); + Assertions.assertEquals(1, delegate.getPartitionsCalls); + + List r2 = cache.getPartitions("db", "t", Arrays.asList("p=1")); + Assertions.assertEquals(1, r2.size()); + Assertions.assertEquals("EXOTIC", r2.get(0).getValues().get(0)); + Assertions.assertEquals(2, delegate.getPartitionsCalls, + "divergence degrades to a reload, never a wrong/dropped result"); } // ---- column statistics ---- @@ -366,6 +423,14 @@ private static final class RecordingHmsClient implements HmsClient { int dropTableCalls; int closeCalls; RuntimeException getTableError; + // Partition names the fake has NO partition for (mirrors HMS omitting non-existent partitions). + final Set absentPartitionNames = new HashSet<>(); + // When set, every returned partition carries these exact values regardless of the requested name + // (used to model a value the name-parse cannot round-trip, exercising the store-by-real-values path). + List forcedValues; + // The partition-name list the decorator actually asked the delegate for on the LAST getPartitions call + // (so a test can assert the decorator fetches only the MISSES, not the whole requested list). + List lastGetPartitionsArg; @Override public HmsTableInfo getTable(String dbName, String tableName) { @@ -385,13 +450,31 @@ public List listPartitionNames(String dbName, String tableName, int maxP @Override public List getPartitions(String dbName, String tableName, List partNames) { getPartitionsCalls++; + lastGetPartitionsArg = new ArrayList<>(partNames); List out = new ArrayList<>(); - for (int i = 0; i < partNames.size(); i++) { - out.add(new HmsPartitionInfo(Arrays.asList("v"), "loc", null, null, null, null)); + for (String name : partNames) { + if (absentPartitionNames.contains(name)) { + continue; // no such partition -> omitted (get_partitions_by_names parity) + } + // The partition's OWN values must correspond to its name ("k=v/..." -> ["v", ...]) so the + // decorator can key it per-partition the same way it parses the lookup name; forcedValues + // overrides this to model a value the name-parse cannot round-trip. + List values = forcedValues != null ? forcedValues : valuesOf(name); + out.add(new HmsPartitionInfo(values, "loc/" + name, null, null, null, null)); } return out; } + // "p=1" -> ["1"]; "k1=a/k2=b" -> ["a", "b"] (simple split; test names carry no escaped characters). + private static List valuesOf(String partitionName) { + List values = new ArrayList<>(); + for (String seg : partitionName.split("/")) { + int eq = seg.indexOf('='); + values.add(eq >= 0 ? seg.substring(eq + 1) : seg); + } + return values; + } + @Override public List getTableColumnStatistics(String dbName, String tableName, List columns) { From 346ba1d34af953e0e3d7cbde5cdd4e0bf65446c3 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 18:26:40 +0800 Subject: [PATCH 190/334] =?UTF-8?q?[fix](catalog)=20Connector=20cache:=20p?= =?UTF-8?q?aimon=20schema-at=20memo=20invalidates=20on=20REFRESH/DROP-CREA?= =?UTF-8?q?TE=20(completes=20D1=20=C2=A73=20for=20paimon)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit PaimonConnector.invalidateTable/invalidateAll cleared only latestSnapshotCache; the second connector-owned cache, PaimonSchemaAtMemo (keyed (db,table,sysTable,branch,schemaId)), had NO invalidate method, so even after the fe-core generic DDL hook routed invalidateTable to the connector, a drop+recreate reusing a schemaId (e.g. schema 0) with different content would still serve a stale time-travel schema. Add PaimonSchemaAtMemo.invalidate(db,table) (removeIf across all schemaIds/sysTables/ branches) + invalidateAll(), wire both into PaimonConnector.invalidateTable/invalidateAll. The memo value is immutable, so dropping an entry only forces a re-read (the pre-memo behavior), never a stale/wrong value. Iceberg needs no analog: its manifest cache is path-keyed (drop+recreate cannot collide) and its latest-snapshot half is already cleared by IcebergConnector.invalidateTable (reached by the same fe-core hook). PaimonSchemaAtMemoTest +2 (per-table invalidate drops all schemaIds but spares other tables/ dbs; invalidateAll clears all); 5 pass, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../connector/paimon/PaimonConnector.java | 11 ++++- .../connector/paimon/PaimonSchemaAtMemo.java | 21 +++++++++ .../paimon/PaimonSchemaAtMemoTest.java | 46 +++++++++++++++++++ 3 files changed, 76 insertions(+), 2 deletions(-) diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java index 71e3f5a6a2e306..fa43d8c8d28d13 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java @@ -231,14 +231,21 @@ public ConnectorMetadata getMetadata(ConnectorSession session) { @Override public void invalidateTable(String dbName, String tableName) { - // REFRESH TABLE: drop the cached latest snapshot id so the next read goes live. Keyed by the REMOTE - // db/table names, matching the key beginQuerySnapshot stores (PaimonTableHandle carries remote names). + // REFRESH TABLE (and, via the generic PluginDrivenExternalCatalog DDL hook, a Doris-issued + // DROP/CREATE of this name): drop the cached latest snapshot id so the next read goes live. Keyed by + // the REMOTE db/table names, matching the key beginQuerySnapshot stores (PaimonTableHandle carries + // remote names). latestSnapshotCache.invalidate(Identifier.create(dbName, tableName)); + // Also drop the time-travel schema memo for this table: unlike the snapshot cache it is keyed by + // (db,table,sysTable,branch,schemaId) and would otherwise serve a stale schema-at-snapshot after a + // drop+recreate that reuses a schemaId (the memo's narrow write-once-per-schemaId assumption breaks). + schemaAtMemo.invalidate(dbName, tableName); } @Override public void invalidateAll() { latestSnapshotCache.invalidateAll(); + schemaAtMemo.invalidateAll(); } @Override diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java index 4b7e6ca743029b..1a0f8bad071ae0 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java @@ -90,6 +90,22 @@ int size() { return cache.size(); } + /** + * Drop every memoized schema for {@code (db, table)} across all schemaIds / sys-tables / branches. Wired + * onto {@code REFRESH TABLE} and — via the generic {@code PluginDrivenExternalCatalog} DDL hook — onto a + * Doris-issued DROP/CREATE of the same name, so a drop+recreate that reuses a schemaId (e.g. schema 0) + * with different content does not serve a stale time-travel schema. The memo value is immutable, so + * dropping an entry only forces a re-read (the pre-memo behavior), never a stale/wrong value. + */ + void invalidate(String databaseName, String tableName) { + cache.keySet().removeIf(key -> key.matches(databaseName, tableName)); + } + + /** Drop the whole memo. Wired onto {@code REFRESH CATALOG} (alongside the connector rebuild). */ + void invalidateAll() { + cache.clear(); + } + /** * Cache key = the handle's identity (db, table, sysTableName, branchName) plus the pinned schemaId. * @@ -114,6 +130,11 @@ static final class MemoKey { this.schemaId = schemaId; } + /** True if this key belongs to {@code (db, table)} (any schemaId / sys-table / branch). */ + boolean matches(String db, String table) { + return databaseName.equals(db) && tableName.equals(table); + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java index a47dd7cc75e633..61d43adba22883 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java @@ -114,4 +114,50 @@ public void overflowEvictsAndReReadsNeverStale() { Assertions.assertEquals(4, loads.get(), "an evicted key must re-read (never serve a stale value)"); Assertions.assertTrue(memo.size() <= 2, "the memo must stay bounded"); } + + @Test + public void invalidateDropsAllSchemaIdsOfOneTableOnly() { + PaimonSchemaAtMemo memo = new PaimonSchemaAtMemo(100); + memo.getOrLoad(handle("db", "t"), 0L, PaimonSchemaAtMemoTest::snap); + memo.getOrLoad(handle("db", "t"), 1L, PaimonSchemaAtMemoTest::snap); // same (db,t), other schemaId + memo.getOrLoad(handle("db", "other"), 0L, PaimonSchemaAtMemoTest::snap); + memo.getOrLoad(handle("db2", "t"), 0L, PaimonSchemaAtMemoTest::snap); // same table name, other db + Assertions.assertEquals(4, memo.size()); + + memo.invalidate("db", "t"); + + // WHY (Rule 9 / the §3 drop+recreate fix): invalidate must drop EVERY schemaId of (db,t) so a recreate + // reusing schema 0 with different content cannot serve a stale schema-at memo, while leaving other + // tables/dbs intact. A mutation matching on schemaId, or matching db only, changes this count. + Assertions.assertEquals(2, memo.size(), "both (db,t) schemaIds gone; (db,other) and (db2,t) survive"); + + AtomicInteger loads = new AtomicInteger(); + memo.getOrLoad(handle("db", "other"), 0L, () -> { + loads.incrementAndGet(); + return snap(); + }); + memo.getOrLoad(handle("db2", "t"), 0L, () -> { + loads.incrementAndGet(); + return snap(); + }); + Assertions.assertEquals(0, loads.get(), "unrelated (db,other) and (db2,t) must stay cached hits"); + memo.getOrLoad(handle("db", "t"), 0L, () -> { + loads.incrementAndGet(); + return snap(); + }); + Assertions.assertEquals(1, loads.get(), "(db,t) must re-read after invalidate"); + } + + @Test + public void invalidateAllClearsEverything() { + PaimonSchemaAtMemo memo = new PaimonSchemaAtMemo(100); + memo.getOrLoad(handle("db", "t"), 0L, PaimonSchemaAtMemoTest::snap); + memo.getOrLoad(handle("db2", "t2"), 0L, PaimonSchemaAtMemoTest::snap); + Assertions.assertEquals(2, memo.size()); + + memo.invalidateAll(); + + // REFRESH CATALOG parity: the whole memo is dropped (the connector is rebuilt around it too). + Assertions.assertEquals(0, memo.size(), "invalidateAll must clear the whole memo"); + } } From 7be53efae65a9efc198fac933df669aaa8c7a958 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 18:30:34 +0800 Subject: [PATCH 191/334] [doc](catalog) Connector-cache invalidation follow-ups DONE: D1+D1b+D2 landed+verified; HANDOFF -> review triage + Phase 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the three cache-invalidation fixes (fe-core generic DDL hook, paimon schema-memo invalidation, hive per-partition keying) as landed+verified with per-fix build results, and the recon correction that review2 §2.1 was already fixed at HEAD. Next: triage the running clean-room re-review (wf_fe6ddef4-777) of the live paimon/iceberg changes, then atomic-flip prereqs. Notes the paimon `install`-not-`test` build gotcha (hive-shade at package phase). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- plan-doc/HANDOFF.md | 48 ++++++++++--------- ...nvalidation-followups-design-2026-07-10.md | 10 ++++ 2 files changed, 35 insertions(+), 23 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index e6fc2521593b9f..9e155d3e019dc0 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -10,33 +10,34 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： - **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 -- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → **净室对抗复审 ✅（本轮，4 发现全修）** → 剩余(3 处 D2 缓存复审项待拍板 → 原子翻闸前置项)。 +- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → 耦合缝净室复审 ✅ → **缓存失效收尾 3 项 ✅（本轮：用户拍板 + 已修 + 已验）** → 剩余（本轮 3 修的净室对抗复核，正在跑 → 原子翻闸前置项）。 - **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 -**⭐ 本轮（2026-07-10）= 翻闸耦合缝的净室对抗复审全部完成，4 个确认发现已修复+验证（"停在复审前"的欠账已还清）。** -复审 = `wf_498114c4-3e1`（7 独立读者冷判"每 commit 前像 + 老实现"、每发现 3 lens 对抗核实、完备性审查；27 agent，2 refuted）。权威设计 = **`tasks/hive-coupling-seams-step-design-2026-07-10.md`**（**起步必读**；TODO 全勾，含 4 发现明细 + Discovered follow-ups + e2e 欠账）。**修复 = 3 个独立 commit，fe-connector-hive 全绿 0 checkstyle：** +**⭐ 本轮（2026-07-10）= 连接器缓存第二轮净室复审留下的 4 个"缓存失效"发现，逐一对照 HEAD 核实后收尾。** +先做的独立核实（`wf_daed84c1-4ea`）有一处**重要更正**：其中一个 HIGH（REFRESH 刷不到 iceberg 兄弟快照缓存）**在 HEAD 已经修好了**（`forEachBuiltSibling` 已接进 invalidate\*），撤销无需处理。剩下 3 项按用户拍板落地，**每项独立 commit + 独立构建验证（BUILD SUCCESS + 0 checkstyle）**： -- **✅ HIGH（翻闸前必修）** `a5015800abd`：hudi_meta/TIMELINE 兄弟转发缺失——`reflectSiblingScanCapabilities` 把 hudi 兄弟的 `SUPPORTS_METADATA_TABLE` 反射进按表标记→翻闸后 gate 通过，但 `HiveConnectorMetadata` **无 `getMetadataTableRows` override**(其它逐柄读都 guard-and-forward)→ foreign `HudiTableHandle` 掉进 SPI 默认空表 = 静默空时间线(非真实)。补 guard-and-forward override(仿 `listFileSizes`)+完整性锁(`EXPECTED_METHODS`/`RecordingSiblingMetadata`/行断言)+真 hudi 形态 withholding 测试(顺带修 S4 测试盲区)。iceberg-on-HMS 不受影响。 -- **✅ LOW(测试盲区)** `a63ab391171`：能力守卫补 `SUPPORTS_SAMPLE_ANALYZE`/`SUPPORTS_METADATA_TABLE` 两 pin(误设连接器级会静默过度接纳 iceberg/hudi-on-HMS)。 -- **✅ LOW(真错，用户拍板"响亮失败"对齐老实现)** `d85c16f2cda`：`listFileSizes` 吞列文件错误到空→采样 `scaleFactor` 塌成 1.0 但 `TABLESAMPLE` 仍触发→静默低估统计、任务报成功；老实现响亮失败。去 catch(留 finally 复原 TCCL)让错误照传 + 确定性传播单测(注入抛错 cache)。 -- **S1 partition_values + W6 = CLEAN**（无存活发现；paimon/iceberg 字节+成本不变已核）。 +- **✅ D1（fe-core 通用钩子，线上改动）** `3b66982fedf`：Doris 自执行 `DROP/CREATE TABLE`、`DROP DATABASE` 从不清连接器自持缓存（只有 REFRESH 清）→ **同名 drop+recreate 命中老表快照/元数据（最长 24h TTL）**。休眠的 hive（翻闸后触发）+ **线上的 paimon/iceberg（现在就触发）** 一次修好：在 `PluginDrivenExternalCatalog` 的 drop/create-table、dropDb 里统一调 `connector.invalidate*`（REMOTE 名，照搬 Trino"谁改动谁清"）；`createDb` 有意不接。`PluginDrivenExternalCatalogDdlRoutingTest` 56 绿。 +- **✅ D1 补丁（paimon 第二层缓存）** `7b8fed012be`：`PaimonSchemaAtMemo`（时间旅行 schema memo）原本无 invalidate 入口 → 补 `invalidate/invalidateAll` 并接进 `PaimonConnector.invalidateTable/invalidateAll`，补齐 paimon 侧的 drop+recreate 洞（iceberg 无需：manifest 按路径 key + 快照缓存已被同一钩子清）。`PaimonSchemaAtMemoTest` 5 绿。 +- **✅ D2（分区缓存容量单位）** `982db925659`：`CachingHmsClient` 分区对象缓存原按"整条请求名列表"做 key（100000 变成"10 万个列表"而非"10 万分区对象"，可撑爆 FE 堆）→ 改成**每分区一个 value key**（对齐 Trino + 老实现），批量取分区做部分命中 + 只补缺失；正确性与名字解析保真度无关（store 永远按分区真实 values，误解析只退化成重取，绝不错/漏）。休眠。`CachingHmsClientTest` 15 绿。 -**⭐ 下一步（本轮未开工，Phase 1 剩余 = 最后一块外部输入 → 翻闸前置项）：** -1. **3 处 D2 缓存复审项待用户拍板**（更早遗留，见 review2 报告；本轮未动）：① 自发 DROP/CREATE TABLE 不清连接器缓存(HIGH)；② 分区缓存容量单位悄变(MEDIUM)；③ paimon/iceberg `latestSnapshotCache` 现网同形删建洞(线上活性 bug)。**这是翻闸前 fe-core 建设唯一未决的外部输入**——下个 session 起步先找用户定这 3 项(中文讲清背景+选项，见 memory `ask-user-explain-in-chinese-first`)。 -2. **原子翻闸前置项**（Phase 2 集：`SPI_READY += hms`、GSON `HMSExternalTable`→`PluginDrivenMvccExternalTable` 重映射、4 守卫改接新子系统、死臂/删除排序）——见 execution-plan §2/§3；D2 三项定完即可开工。 -3. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失(所有插件表通性、非本轮引入)。 +设计文档（**起步必读**，含正确性论证 + e2e 欠账）= **`tasks/hive-cache-invalidation-followups-design-2026-07-10.md`**（§6 状态：3 修全勾 + 复审在跑）。 -**⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步(读/写-拒/schema-evolution/缓存/跨加载器委派/本步 4 缝)只在翻闸后 live，须异构 HMS docker e2e 断言（清单见 execution plan §4/§5 + `hive-coupling-seams-step-design` e2e-owed 段 + memory `hms-iceberg-delegation-needs-e2e`）。删除排序最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 +**⭐ 下一步：** +1. **本轮 3 修的净室对抗复核（正在后台跑，`wf_fe6ddef4-777`）** — 4 维独立冷读（D1/D1补丁/D2/完备性）+ 每发现 3-lens 对抗核实。**下个 session 起步先取它的结论**：无存活发现即收尾；有则按 severity 修（D1/D1补丁是线上路径，务必核对 paimon/iceberg 行为/成本；见 memory `plugindriven-mvcc-table-is-live-not-dormant`）。 +2. **原子翻闸前置项**（Phase 2 集：`SPI_READY += hms`、GSON `HMSExternalTable`→`PluginDrivenMvccExternalTable` 重映射、4 守卫改接新子系统、死臂/删除排序）——见 execution-plan §2/§3。**翻闸前 fe-core 建设的外部输入已全部消化**（缓存决策已拍板落地），复核通过即可开 Phase 2。 +3. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失（所有插件表通性、非本轮引入）。 -**⚠ 并行 session 风险**：曾有两个 session 同工作树互相 amend；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手。 +**⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步 + 本轮 3 修只在翻闸后 live（hive）；**paimon/iceberg 的 drop+recreate 修复现在就 live，可即刻 e2e 回归**（`DROP TABLE t; CREATE TABLE t(新schema/位置); SELECT` 无需 REFRESH 见新表；paimon 时间旅行复用 schemaId 见新 schema）。异构 HMS docker e2e 清单见 `hive-cache-invalidation-followups-design` §5 + execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`。删除排序最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 + +**⚠ 并行 session 风险**：曾有两个 session 同工作树互相 amend；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手（本轮见到的 `gq` 用户 maven 是**别的工作树**，非本树）。 --- # 🧠 起步必读（读文档，别炒 git log 历史） 1. **权威翻闸计划** = `tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。 -2. **本步（翻闸耦合缝）设计** = `tasks/hive-coupling-seams-step-design-2026-07-10.md`（4 缝 + 3 决策 + TODO；S1 已勾）。**起步先读，再对照 HEAD 核行号**。 -3. 已完成的前两项 Phase-1 步：`tasks/hive-connector-cache-step-design-2026-07-10.md`（D2 缓存）、`tasks/hive-event-pipeline-step-design-2026-07-10.md`（事件管道）。 +2. **本轮（缓存失效收尾）设计** = `tasks/hive-cache-invalidation-followups-design-2026-07-10.md`（D1 fe-core 钩子 + D2 每分区 keying + 正确性论证 + §6 状态 + e2e 欠账）。**起步先读，再对照 HEAD 核行号**。 +3. 已完成的前几项 Phase-1 步：`tasks/hive-connector-cache-step-design-2026-07-10.md`（D2 缓存本体）、`tasks/hive-event-pipeline-step-design-2026-07-10.md`（事件管道）、`tasks/hive-coupling-seams-step-design-2026-07-10.md`（4 耦合缝）。第二轮缓存复审报告 = `plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md`（本轮 3 修的来源；注意 §2.1 已在 HEAD 修好、§2.3 说"Trino 用 weigher"经复核为**不准确**——Trino 也是每分区一条目按数量封顶）。 4. **补充权威**：`tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + §6 D1–D6 决策）；`tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 + CCE/TCCL 硬约束）；hudi = `tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `tasks/hudi-schema-evolution-step-design-2026-07-09.md`。 5. **样板**：`tasks/P5-paimon-migration.md`、`tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 6. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 @@ -52,20 +53,21 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms # ⚙️ 操作须知（构建/测试，复用） -- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**）。连接器：`-pl fe-connector/ -am`。单测加 `-Dtest=`。 -- **验证信 LOG 不信 exit**：`-q` + `| tail` 会吞掉 `BUILD SUCCESS` 且管道 exit=tail 的(不可信)；改**重定向到文件**跑（不加 `-q`），再 grep `BUILD SUCCESS`/`[ERROR].*\.java:`/checkstyle `You have N violations`（memory `doris-build-verify-gotchas`）。test-compile 的 `validate` 阶段已跑 checkstyle（本轮见 fe-core `0 violations`）。 -- **build 中 ANTLR `mismatched input '->'`/`super::` 噪音 = 非致命**（某 codegen/工具 Java 语法解析器不识 lambda/方法引用；本轮 BUILD SUCCESS 照出）。 +- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**）。连接器：`-pl :fe-connector- -am`。单测加 `-Dtest=`。 +- **⚠️ paimon 模块必须用 `install`（不是 `test`）验证**：`fe-connector-paimon` 的 `HiveConf` 来自 `fe-connector-paimon-hive-shade`，其 maven-shade 绑在 `package` 阶段；`mvn test` 停在 `test` 阶段→shade jar 没产出→paimon 编译报 `package org.apache.hadoop.hive.conf does not exist`（**误导：非你的改动**）。用 `-pl :fe-connector-paimon -am install`。hms/hive 无此坑（hive-common 是普通依赖）。 +- **验证信 LOG 不信 exit**：`-q` + `| tail` 会吞掉 `BUILD SUCCESS`；改**重定向到文件**跑（不加 `-q`），再 grep `BUILD SUCCESS`/`[ERROR].*\.java:`/`Tests run:`/checkstyle `You have N violations`（memory `doris-build-verify-gotchas`）。 +- **build 中 ANTLR `mismatched input '->'`/`super::` 噪音 = 非致命**。 - **import gate**：`bash tools/check-connector-imports.sh`（repo 根跑）。HMS `HiveVersionUtil` 命中 = 误报（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 -- **⚠️ bash 默认 timeout 120s**：fe-core 全量编译 ~3min+ → 后台跑 + 读 LOG。**⚠️ `/mnt/disk1` 盘紧(~82%)；勿用 worktree 隔离编译 agent**。cwd 会被重置 → 绝对路径。 -- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`mock(接口)` 不跑 default 方法返 null；`mockStatic(Env)` 惯用）。checkstyle 禁 static import、扫 test 源（memory `catalog-spi-fe-core-test-infra`）。 +- **⚠️ bash 默认 timeout 120s**：全量编译 ~3min+ → 后台跑 + 读 LOG。**⚠️ `/mnt/disk1` 盘紧；勿用 worktree 隔离编译 agent**。cwd 会被重置 → 绝对路径。 +- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`PluginDrivenExternalCatalogDdlRoutingTest` 用 `mock(Connector.class)` 直接 verify invalidate\*）。checkstyle 禁 static import、扫 test 源。 # 🔒 铁律（fe-core 约束） -- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。能力按表判要用 `hasScanCapability`/`PER_TABLE_CAPABILITIES_KEY`（连接器发标记，fe-core 不看格式/dlaType）。 +- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。本轮 D1 钩子调 `connector.invalidateTable/invalidateDb` 是**通用 SPI 调用**（no-op 默认），不违反。 - fe-core **不解析属性**（storage→fe-filesystem、meta→fe-connector，均插件侧；memory `catalog-spi-no-property-parsing-in-fecore`）。 -- 跨插件/跨边界**须 pin TCCL** 到连接器 classloader（扫描线程/写-DDL 引擎线程/iceberg worker 池/事件轮询后台线程；memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 +- 跨插件/跨边界**须 pin TCCL** 到连接器 classloader（扫描线程/写-DDL 引擎线程/iceberg worker 池/事件轮询后台线程；memory `catalog-spi-plugin-tccl-classloader-gotcha`）。invalidate\* 是直接方法派发（非按名反射），照搬 RefreshManager 现有调用不需 pin。 - `history_schema_info` 嵌套字段名逐层 lowercase（否则 BE SIGABRT；memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 -- `PluginDrivenMvccExternalTable`/`PluginDrivenExternalTable` 是 paimon/iceberg **实时**基类：改其共享方法须对二者字节+成本双不变（本轮 S1 即因此把 `getNameToPartitionValues` 做成独立方法、不动 `getNameToPartitionItems`；memory `plugindriven-mvcc-table-is-live-not-dormant`）。 +- `PluginDrivenMvccExternalTable`/`PluginDrivenExternalTable` 是 paimon/iceberg **实时**基类：改其共享方法须对二者字节+成本双不变（memory `plugindriven-mvcc-table-is-live-not-dormant`）。**本轮 D1 是有意的线上行为改动（修 drop+recreate 洞），非字节中性——已配净室复审 + e2e 欠账把关。** # 🗂 memory 相关项 diff --git a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md index b37226ed9c00ab..ce18335b43885f 100644 --- a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md +++ b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md @@ -214,3 +214,13 @@ paimon/iceberg — per practice + memory `plugindriven-mvcc-table-is-live-not-do no REFRESH — for paimon + iceberg NOW (live), and for hive post-flip. - `DROP DATABASE d; CREATE DATABASE d; ...` likewise sees no stale table entries. - paimon time-travel after drop+recreate reusing a schemaId returns the new schema (schema-memo cleared). + +## 6. Status (landed 2026-07-10) +- [x] **F1** fe-core generic DDL invalidation hook — `3b66982fedf` (`PluginDrivenExternalCatalogDdlRoutingTest` + 56 pass, checkstyle 0). Live for paimon/iceberg; arms hive post-flip. +- [x] **F1b** paimon `PaimonSchemaAtMemo` invalidate + wiring — `7b8fed012be` (`PaimonSchemaAtMemoTest` 5 pass). +- [x] **F2** `CachingHmsClient` per-partition value keying — `982db925659` (`CachingHmsClientTest` 15 pass). + Dormant (hive not flipped). +- [ ] **Unified clean-room adversarial re-review** of F1/F1b (live paths) — running (`wf_fe6ddef4-777`). +- [ ] **e2e** (§5) — owed at the flip / heterogeneous-HMS docker (paimon/iceberg drop+recreate is testable + NOW as a live regression). From a6afec61c080f6ffe3a1621c06156ad868033a08 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 18:56:42 +0800 Subject: [PATCH 192/334] [doc](catalog) Cache-invalidation clean-room review: 4 confirmed gaps (3 live + 1 dormant); HANDOFF -> fix R1-R4 then Phase 2 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adversarial review wf_fe6ddef4-777 (28 agents; 4 dimension readers -> 3-lens verify each, confirm >=2/3) over the three landed cache-invalidation fixes. Verdict: net improvements but INCOMPLETE (none a regression vs pre-fix; each an incomplete realization): - R1 (live): DROP/CREATE/DROPDB invalidation is coordinator-only; followers/observers never propagate on editlog replay (mirror the existing replayTruncateTable override). - R2 (live): iceberg/paimon lack an invalidateDb override -> the dropDb hook AND the pre-existing REFRESH DATABASE are no-ops for their snapshot/schema caches. - R3 (dormant): getPartitions raw put bypasses the invalidateGeneration guard the old get(key,loader) path had (REFRESH-vs-in-flight race); needs an additive guarded put on the connector MetaCacheEntry. - R4 (live, weak 1/3 but solid): RENAME (maybe replace/CTAS-overwrite) never invalidates. Full report + per-finding fix approach + order (R2->R1->R4->R3) in plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md. Design doc §6 + HANDOFF next-step updated. No code change this commit. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- plan-doc/HANDOFF.md | 16 ++-- ...nvalidation-cleanroom-review-2026-07-10.md | 83 +++++++++++++++++++ ...nvalidation-followups-design-2026-07-10.md | 13 ++- 3 files changed, 105 insertions(+), 7 deletions(-) create mode 100644 plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 9e155d3e019dc0..29369c0756503c 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -10,7 +10,7 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： - **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 -- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → 耦合缝净室复审 ✅ → **缓存失效收尾 3 项 ✅（本轮：用户拍板 + 已修 + 已验）** → 剩余（本轮 3 修的净室对抗复核，正在跑 → 原子翻闸前置项）。 +- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → 耦合缝净室复审 ✅ → 缓存失效收尾 3 修 ✅（用户拍板+已修+已验）→ 对抗复核 ✅ **查出 4 个待修（3 线上+1 休眠，见下）** → 剩余（修完 4 项 → 原子翻闸前置项）。 - **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 **⭐ 本轮（2026-07-10）= 连接器缓存第二轮净室复审留下的 4 个"缓存失效"发现，逐一对照 HEAD 核实后收尾。** @@ -22,10 +22,16 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms 设计文档（**起步必读**，含正确性论证 + e2e 欠账）= **`tasks/hive-cache-invalidation-followups-design-2026-07-10.md`**（§6 状态：3 修全勾 + 复审在跑）。 -**⭐ 下一步：** -1. **本轮 3 修的净室对抗复核（正在后台跑，`wf_fe6ddef4-777`）** — 4 维独立冷读（D1/D1补丁/D2/完备性）+ 每发现 3-lens 对抗核实。**下个 session 起步先取它的结论**：无存活发现即收尾；有则按 severity 修（D1/D1补丁是线上路径，务必核对 paimon/iceberg 行为/成本；见 memory `plugindriven-mvcc-table-is-live-not-dormant`）。 -2. **原子翻闸前置项**（Phase 2 集：`SPI_READY += hms`、GSON `HMSExternalTable`→`PluginDrivenMvccExternalTable` 重映射、4 守卫改接新子系统、死臂/删除排序）——见 execution-plan §2/§3。**翻闸前 fe-core 建设的外部输入已全部消化**（缓存决策已拍板落地），复核通过即可开 Phase 2。 -3. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失（所有插件表通性、非本轮引入）。 +**⭐ 净室对抗复核结论（`wf_fe6ddef4-777`，已完成，28 agent）：3 修是净改进但 INCOMPLETE，查出 4 个待修（3 线上 + 1 休眠），全部 ≥2/3 对抗票（R4 仅 1/3 但论证扎实）。报告 = `plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md`（含每项修法）。均非"比修前更糟"（修前哪都不清），只是 D1 修得不完整。** + +**⭐ 下一步（下个 session 起步先修这 4 项，再开 Phase 2）：** +1. **R2（线上，先修，一箭三雕）** — iceberg/paimon 没 override `invalidateDb` → 我的 `DROP DATABASE` 钩子 + **老早就存在的 `REFRESH DATABASE`** 对它俩的快照/schema 缓存都是空操作。给 `IcebergConnector`/`PaimonConnector` 补 `invalidateDb`（按 db 前缀 removeIf 清 `latestSnapshotCache` + paimon `PaimonSchemaAtMemo`）。顺带修好 `DROP DATABASE FORCE` 级联 + hive 翻闸后 `forEachBuiltSibling(sibling.invalidateDb)`。 +2. **R1（线上）** — DROP/CREATE/DROPDB 失效**只在执行 DDL 的那台 FE 生效**；follower/observer 走 editlog replay（`replayDropTable/replayCreateTable/replayDropDb` 只清 FE 名字缓存）不传到 `connector.invalidate*`。仿现成的 `replayTruncateTable` override + `replayRefreshTable→refreshTableInternal` 补 replay 侧传播。 +3. **R4（线上，弱票但论证实）** — `RENAME TABLE`（可能还有 replace/CTAS-overwrite）也没清连接器缓存（`afterExternalRename` 零 `invalidate*`、不走 `refreshTableInternal`）→ 原子换名 t↔t2 读到老快照。同 D1 类，设计文档还引了 Trino rename 自失效做样板却漏了它。 +4. **R3（休眠，最后）** — `getPartitions` 直接 `put` 绕过了老代码有的 invalidateGeneration 守卫（REFRESH 撞正在飞的取数 → 重缓存陈旧值到 TTL）。给连接器版 `MetaCacheEntry` 加个 additive 的守卫式 put（`invalidationGeneration()` + `putIfNotInvalidatedSince`），RPC 前抓 generation、逐分区守卫式 put，结果仍直接从 delegate 收（保 misparse-never-drop）。 + - **修法/顺序/测试细节全在复核报告里**；R1/R2/R4 是线上路径，改完再跑一轮针对性对抗复核（memory `plugindriven-mvcc-table-is-live-not-dormant`）。 +5. **之后**：原子翻闸前置项（Phase 2 集：`SPI_READY += hms`、GSON 重映射、4 守卫改接、死臂/删除排序）——见 execution-plan §2/§3。 +6. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失（所有插件表通性、非本轮引入）。 **⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步 + 本轮 3 修只在翻闸后 live（hive）；**paimon/iceberg 的 drop+recreate 修复现在就 live，可即刻 e2e 回归**（`DROP TABLE t; CREATE TABLE t(新schema/位置); SELECT` 无需 REFRESH 见新表；paimon 时间旅行复用 schemaId 见新 schema）。异构 HMS docker e2e 清单见 `hive-cache-invalidation-followups-design` §5 + execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`。删除排序最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 diff --git a/plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md b/plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md new file mode 100644 index 00000000000000..f5bb21c9487909 --- /dev/null +++ b/plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md @@ -0,0 +1,83 @@ +# Cache-invalidation fixes — clean-room adversarial review (2026-07-10) + +> Adversarial review of the three connector-cache invalidation fixes landed this round +> (`3b66982fedf` D1 fe-core hook, `7b8fed012be` D1b paimon memo, `982db925659` D2 per-partition keying). +> Run `wf_fe6ddef4-777`: 4 blind dimension readers (D1 / D1b / D2 / completeness) → each finding +> adversarially verified by 3 independent lenses (correctness / concurrency-or-parity / does-it-reproduce), +> confirmed at ≥2/3. Totals: **8 raised, 4 CONFIRMED (all 3/3), 2 weak (1/3), 2 dropped.** +> +> **Verdict: the fixes are net improvements but INCOMPLETE. 3 live gaps + 1 dormant race remain; must be +> fixed before this phase is called done.** None is a regression that makes things worse than pre-fix (before, +> nothing was invalidated anywhere); each is an incomplete realization of the stated fix. + +## Confirmed findings + fix approach + +### R1 — DROP/CREATE invalidation is coordinator-only; followers/observers stay stale (MEDIUM, LIVE) +`PluginDrivenExternalCatalog` invalidates the connector cache only on the FE that RUNS the DDL. Followers/ +observers replay via `ExternalCatalog.replayDropTable`/`replayCreateTable`/`replayDropDb` (the PluginDriven +branch, metadataOps==null) which touch only the FE name cache — never `connector.invalidate*`. So a +follower that had queried paimon/iceberg `d.t` keeps its `latestSnapshotCache[(d,t)]` pinned to the dropped +table's snapshot until the 24h access-TTL. +- **Precedent (this is normally done):** `RefreshManager.replayRefreshTable → refreshTableInternal:254-257` + DOES call `connector.invalidateTable` on replay; and `PluginDrivenExternalCatalog` already added a + `replayTruncateTable` override that routes through `refreshTableInternal` for exactly this reason. +- **Fix:** override `replayDropTable`/`replayCreateTable`/`replayDropDb` in `PluginDrivenExternalCatalog` + (or the shared replay seam) to also call `connector.invalidateTable`/`invalidateDb` after the base + bookkeeping, mirroring `replayTruncateTable`. Needs the remote names on the replay path (resolve like the + coordinator does, or persist them in the log). + +### R2 — iceberg/paimon do not override `invalidateDb` → DROP DATABASE + REFRESH DATABASE are no-ops (MEDIUM, LIVE) [= confirmed #2 + #4, same root] +Only `HiveConnector` overrides `invalidateDb`; `IcebergConnector`/`PaimonConnector` inherit the SPI no-op +default (`Connector.java:324`). So: +- The new `dropDb → connector.invalidateDb(d)` hook (D1) is INERT for iceberg/paimon (the dropDb comment's + "drops every table in this db" is false for them). `DROP DATABASE d FORCE` cascades table drops INSIDE the + connector, bypassing per-table `invalidateTable`, so the cascaded tables' `latestSnapshotCache`/ + `PaimonSchemaAtMemo` entries survive. +- **Pre-existing (independent of this round):** `RefreshManager.refreshDbInternal:126 → invalidateDb` — so + `REFRESH DATABASE d` on iceberg/paimon has ALWAYS been a silent no-op for their snapshot/schema caches. +- Also: hive's `forEachBuiltSibling(sibling.invalidateDb)` is a no-op against the iceberg/hudi siblings + post-flip for the same reason. +- **Fix:** add `invalidateDb(db)` overrides to `IcebergConnector` (db-scoped removeIf over + `latestSnapshotCache` keys whose namespace==db) and `PaimonConnector` (`latestSnapshotCache` + + `PaimonSchemaAtMemo`, db-scoped). Requires a db-scoped invalidate on `IcebergLatestSnapshotCache` / + `PaimonLatestSnapshotCache` / `PaimonSchemaAtMemo` (mirror the existing per-table `matches`). Fixes the + D1 dropDb hook AND the pre-existing REFRESH DATABASE gap in one go. + +### R3 — `getPartitions` raw `put` bypasses the invalidateGeneration guard (MEDIUM, DORMANT) +The rewritten `getPartitions` uses `partitionsCache.put(...)` directly (`CachingHmsClient.java:182`), which +has no generation check. The pre-commit code used `partitionsCache.get(key, loader)` → `getWithManualLoad`, +which captures `invalidateGeneration` before the load and drops the put if a `flush` raced it +(`MetaCacheEntry.java:240-256`). So a `REFRESH TABLE` (`flush`) racing an in-flight cold-cache fetch now +re-caches the pre-refresh partitions, which survive until TTL/next REFRESH. `getTable`/`listPartitionNames`/ +`getTableColumnStatistics` still use the guarded path — only `getPartitions` lost it. +- Currently dormant (hive not flipped; `flush` not yet wired to REFRESH for a live hms catalog). +- **Fix (keep bulk RPC + per-partition keying + divergence-safety):** add a generation-guarded put to the + connector copy of `MetaCacheEntry` — `long invalidationGeneration()` + `putIfNotInvalidatedSince(gen, key, + value)` (put under the key's stripe lock, then `removeLoadedValue` if the generation moved). In + `getPartitions`, capture the generation BEFORE the delegate RPC, then guard each per-partition put with it; + keep adding the delegate results to the result list directly (preserves the misparse→never-drop safety). + This is an ADDITIVE framework method (no behavior/byte change for paimon/iceberg; same Caffeine version). + +### R4 — RENAME TABLE never invalidates the connector cache (weak 1/3, but strong reasoning; LIVE) +`renameTable → afterExternalRename` (`:~649/1007`) runs unregister+resetMetaCacheNames+constraint+editlog and +NEVER calls `connector.invalidate*`, and does NOT route through `refreshTableInternal` (unlike truncate / +column-ALTER). So an iceberg/paimon atomic table-swap (`RENAME t→t_arch; RENAME t_new→t`) leaves the stale +`latestSnapshotCache[(db,t)]` pinned → the recreated `t` reads the OLD snapshot. Same drop+recreate class +D1 targets; the design doc even cites Trino's `renameTable` self-invalidation as the parity model, then +FIX 1 hooks only dropTable/createTable/dropDb. **Fix:** invalidate the source (and target) name's connector +cache in the rename path (source on the coordinator + replay; and `replayRenameTable` for followers). Verify +whether `replaceTable`/CTAS-overwrite has the same gap. + +## Weak / not-fixing +- **W1 (paimon memo TOCTOU, LOW, 1/3):** a lock-free invalidate-vs-in-flight-load window can memoize an + OLD schema for a reused schemaId if a concurrent drop+recreate lands between the loader read and the + `putIfAbsent`. NOT introduced/worsened by this round (inherent to every cache here incl. the Caffeine + snapshot cache); the commit message's unconditional "immutability ⇒ never stale" is the only overclaim. + No fix required; the 3 primary axes (CHM keySet removeIf safety, `matches` over/under-match + null-safety, + remote-name identity parity) are all correct. If ever hardened: computeIfAbsent-style load-under-key or an + invalidate-generation counter. + +## Next-session task (fix before Phase 2) +Order: **R2** (fixes 2 confirmed + a pre-existing live bug; connector-local) → **R1** (fe-core replay) → +**R4** (rename; same class as R1) → **R3** (dormant; framework method). Each its own commit + tests + build +(paimon uses `install`, see HANDOFF). Re-run a targeted adversarial pass on R1/R2/R4 (live paths) after. diff --git a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md index ce18335b43885f..ab5c72b82bdf2a 100644 --- a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md +++ b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md @@ -221,6 +221,15 @@ paimon/iceberg — per practice + memory `plugindriven-mvcc-table-is-live-not-do - [x] **F1b** paimon `PaimonSchemaAtMemo` invalidate + wiring — `7b8fed012be` (`PaimonSchemaAtMemoTest` 5 pass). - [x] **F2** `CachingHmsClient` per-partition value keying — `982db925659` (`CachingHmsClientTest` 15 pass). Dormant (hive not flipped). -- [ ] **Unified clean-room adversarial re-review** of F1/F1b (live paths) — running (`wf_fe6ddef4-777`). +- [x] **Unified clean-room adversarial re-review** of F1/F1b (live paths) — `wf_fe6ddef4-777`, DONE. **Verdict: + fixes are NET IMPROVEMENTS but INCOMPLETE — 4 follow-ups required (3 live + 1 dormant). See + `plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md`.** +- [ ] **R1** DROP/CREATE/DROPDB invalidation is coordinator-only — followers/observers don't propagate on + replay (mirror `replayTruncateTable`). LIVE. +- [ ] **R2** iceberg/paimon lack an `invalidateDb` override → DROP DATABASE hook + pre-existing REFRESH + DATABASE are no-ops for their snapshot/schema caches. LIVE (fixes a pre-existing bug too). +- [ ] **R3** `getPartitions` raw `put` bypasses the invalidateGeneration guard (REFRESH-vs-in-flight race). + DORMANT; needs an additive guarded-put on the connector `MetaCacheEntry`. +- [ ] **R4** RENAME (and maybe replace/CTAS-overwrite) never invalidates the connector cache. LIVE. - [ ] **e2e** (§5) — owed at the flip / heterogeneous-HMS docker (paimon/iceberg drop+recreate is testable - NOW as a live regression). + NOW as a live regression, once R1/R2/R4 land). From e5fb97662fcb369315794759e3a8504ead346202 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 19:26:56 +0800 Subject: [PATCH 193/334] [fix](catalog) Connector cache: iceberg/paimon gain a db-scoped invalidateDb (REFRESH/DROP DATABASE) (R2) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit IcebergConnector/PaimonConnector inherited the SPI no-op invalidateDb default, so BOTH the generic PluginDrivenExternalCatalog dropDb hook (Doris-issued DROP DATABASE) AND the pre-existing RefreshManager.refreshDbInternal (REFRESH DATABASE) were silent no-ops for their snapshot/schema caches: every table in the db stayed pinned to its stale snapshot (and paimon's time-travel schema memo stayed populated) up to the 24h access-TTL. DROP DATABASE FORCE was worse — the connector cascades table drops internally, bypassing per-table invalidateTable, so the cascaded tables' pins survived even the D1 fix. This also armed the hive gateway's forEachBuiltSibling(sibling.invalidateDb) for the iceberg sibling post-flip (it was a no-op against the inherited default). Fix (db-scoped analogue of invalidateTable; Trino CachingHiveMetastore.invalidateDatabase parity): add invalidateDb overrides that remove every entry keyed under the db. - IcebergLatestSnapshotCache.invalidateDb: entries are TableIdentifier.of(db,table) (single-level namespace [db], per beginQuerySnapshot) -> invalidateIf(namespace==db). - PaimonLatestSnapshotCache.invalidateDb: entries are Identifier.create(db,table) -> invalidateIf(getDatabaseName==db). - PaimonSchemaAtMemo.invalidateDb + MemoKey.matchesDb: removeIf(matchesDb) drops every (table,schemaId) of the db so a same-name recreate cannot serve a stale time-travel schema. PaimonConnector.invalidateDb wires BOTH caches. Uses MetaCacheEntry.invalidateIf (existing framework predicate); no framework change. Correctness/cost: dropping a db's entries is always safe (each is stale or absent after a DROP/REFRESH); one removeIf per DDL, off the query hot path; no-cache connectors (jdbc/es) inherit the inert SPI default. Live for paimon/iceberg today; arms hive post-flip. Tests: IcebergLatestSnapshotCacheTest.invalidateDbClearsOnlyThatDbsTables (16 pass); PaimonLatestSnapshotCacheTest.invalidateDbClearsOnlyThatDbsTables + PaimonSchemaAtMemoTest.invalidateDbDropsEveryTableOfOneDbOnly + connector no-throw smokes (17 pass). iceberg BUILD SUCCESS / paimon install BUILD SUCCESS, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../connector/iceberg/IcebergConnector.java | 16 +++++++++ .../iceberg/IcebergLatestSnapshotCache.java | 12 +++++++ .../iceberg/IcebergConnectorCacheTest.java | 1 + .../IcebergLatestSnapshotCacheTest.java | 34 +++++++++++++++++++ .../connector/paimon/PaimonConnector.java | 16 +++++++++ .../paimon/PaimonLatestSnapshotCache.java | 10 ++++++ .../connector/paimon/PaimonSchemaAtMemo.java | 16 +++++++++ .../paimon/PaimonConnectorCacheTest.java | 13 +++++++ .../paimon/PaimonLatestSnapshotCacheTest.java | 30 ++++++++++++++++ .../paimon/PaimonSchemaAtMemoTest.java | 29 ++++++++++++++++ 10 files changed, 177 insertions(+) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 5f8c2157633ff8..937c7a3c57d25c 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -448,6 +448,22 @@ public void invalidateTable(String dbName, String tableName) { latestSnapshotCache.invalidate(TableIdentifier.of(dbName, tableName)); } + /** + * REFRESH DATABASE hook (also reached by a Doris-issued {@code DROP DATABASE} via the generic + * {@code PluginDrivenExternalCatalog} dropDb hook, and by the hive gateway's + * {@code forEachBuiltSibling} for an iceberg-on-HMS sibling): drop the cached latest snapshot for + * EVERY table in one database so the next query re-pins live. Db-scoped analogue of + * {@link #invalidateTable}; the name is the REMOTE db name (RefreshManager / the dropDb hook pass + * remote names). Without this override iceberg inherited the SPI no-op default, so REFRESH DATABASE + * and DROP DATABASE (incl. its FORCE table cascade, which bypasses per-table invalidateTable) left + * the snapshot pins stale up to the TTL. The path-keyed manifest cache is intentionally NOT cleared + * (legacy parity — only REFRESH CATALOG drops manifests). + */ + @Override + public void invalidateDb(String dbName) { + latestSnapshotCache.invalidateDb(dbName); + } + /** * REFRESH CATALOG hook: drop ALL of this catalog's connector-owned caches. Clears both the latest-snapshot * cache and the (path-keyed) manifest cache — mirroring legacy {@code IcebergExternalMetaCache}'s diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java index e939de92c33e02..b953e178fbe377 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCache.java @@ -20,6 +20,7 @@ import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.cache.MetaCacheEntry; +import org.apache.iceberg.catalog.Namespace; import org.apache.iceberg.catalog.TableIdentifier; import java.util.concurrent.ForkJoinPool; @@ -98,6 +99,17 @@ void invalidate(TableIdentifier identifier) { entry.invalidateKey(identifier); } + /** + * Drops every cached entry for one database so the next read of any of its tables goes live + * (REFRESH DATABASE / a Doris-issued DROP DATABASE). Entries are keyed by + * {@code TableIdentifier.of(db, table)} (single-level namespace = {@code [db]}, see + * {@code IcebergConnectorMetadata.beginQuerySnapshot}), so a db match is namespace equality. + */ + void invalidateDb(String dbName) { + Namespace ns = Namespace.of(dbName); + entry.invalidateIf(id -> id.namespace().equals(ns)); + } + /** Drops all cached entries. */ void invalidateAll() { entry.invalidateAll(); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java index 555b37f26e33df..bc816df8c48e22 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorCacheTest.java @@ -124,6 +124,7 @@ public void invalidateHooksAreNoThrowOnFreshConnector() { IcebergConnector connector = new IcebergConnector(Collections.emptyMap(), new RecordingConnectorContext()); Assertions.assertDoesNotThrow(() -> connector.invalidateTable("db1", "t1")); + Assertions.assertDoesNotThrow(() -> connector.invalidateDb("db1")); Assertions.assertDoesNotThrow(connector::invalidateAll); } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCacheTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCacheTest.java index 08dc8e19ae2ddd..c592472ffffb08 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCacheTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergLatestSnapshotCacheTest.java @@ -135,4 +135,38 @@ public void invalidateAllClearsEverything() { c.invalidateAll(); Assertions.assertEquals(0, c.size()); } + + @Test + public void invalidateDbClearsOnlyThatDbsTables() { + AtomicInteger loads = new AtomicInteger(); + IcebergLatestSnapshotCache c = new IcebergLatestSnapshotCache(100, 1000); + c.getOrLoad(TableIdentifier.of("db1", "t1"), + () -> new IcebergLatestSnapshotCache.CachedSnapshot(1L, 11L)); + c.getOrLoad(TableIdentifier.of("db1", "t2"), + () -> new IcebergLatestSnapshotCache.CachedSnapshot(2L, 22L)); + c.getOrLoad(TableIdentifier.of("db2", "t1"), + () -> new IcebergLatestSnapshotCache.CachedSnapshot(3L, 33L)); + Assertions.assertEquals(3, c.size()); + + // REFRESH DATABASE db1 (or a Doris DROP DATABASE db1) must drop BOTH db1 tables and leave db2 intact. + // MUTATION: invalidateDb a no-op (the inherited SPI default this fix replaces) -> db1.t1 still cached + // -> loads stays 0 / after.snapshotId == 1 -> red. + c.invalidateDb("db1"); + Assertions.assertEquals(1, c.size(), "only db2's single entry must survive"); + + IcebergLatestSnapshotCache.CachedSnapshot afterDb1 = c.getOrLoad(TableIdentifier.of("db1", "t1"), () -> { + loads.incrementAndGet(); + return new IcebergLatestSnapshotCache.CachedSnapshot(9L, 99L); + }); + Assertions.assertEquals(9L, afterDb1.snapshotId, "db1.t1 must reload live after invalidateDb"); + Assertions.assertEquals(1, loads.get()); + + // db2 was untouched: still the original pin, no reload. + IcebergLatestSnapshotCache.CachedSnapshot db2 = c.getOrLoad(TableIdentifier.of("db2", "t1"), () -> { + loads.incrementAndGet(); + return new IcebergLatestSnapshotCache.CachedSnapshot(7L, 77L); + }); + Assertions.assertEquals(3L, db2.snapshotId, "db2 must keep its cached pin (not dropped by invalidateDb(db1))"); + Assertions.assertEquals(1, loads.get(), "db2 read must be a hit (no extra load)"); + } } diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java index fa43d8c8d28d13..5e44561056307c 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnector.java @@ -242,6 +242,22 @@ public void invalidateTable(String dbName, String tableName) { schemaAtMemo.invalidate(dbName, tableName); } + /** + * REFRESH DATABASE hook (also reached by a Doris-issued {@code DROP DATABASE} via the generic + * {@code PluginDrivenExternalCatalog} dropDb hook, and by the hive gateway's + * {@code forEachBuiltSibling} for a paimon sibling): drop BOTH connector-owned caches for EVERY table + * in one database — the latest-snapshot pin and the time-travel schema memo — so the next query + * re-reads live. Db-scoped analogue of {@link #invalidateTable}; the name is the REMOTE db name. + * Without this override paimon inherited the SPI no-op default, so REFRESH DATABASE and DROP DATABASE + * (incl. its FORCE table cascade, which bypasses per-table invalidateTable) left both caches stale up + * to the TTL. + */ + @Override + public void invalidateDb(String dbName) { + latestSnapshotCache.invalidateDb(dbName); + schemaAtMemo.invalidateDb(dbName); + } + @Override public void invalidateAll() { latestSnapshotCache.invalidateAll(); diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCache.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCache.java index c20590b0bcbed6..33b62686d07386 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCache.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCache.java @@ -82,6 +82,16 @@ void invalidate(Identifier identifier) { entry.invalidateKey(identifier); } + /** + * Drops every cached entry for one database so the next read of any of its tables goes live + * (REFRESH DATABASE / a Doris-issued DROP DATABASE). Entries are keyed by + * {@code Identifier.create(db, table)} (see {@code PaimonConnectorMetadata.beginQuerySnapshot}), so a + * db match is {@code getDatabaseName()} equality. + */ + void invalidateDb(String dbName) { + entry.invalidateIf(id -> id.getDatabaseName().equals(dbName)); + } + /** Drops all cached entries. */ void invalidateAll() { entry.invalidateAll(); diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java index 1a0f8bad071ae0..1a48e70929ec06 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemo.java @@ -101,6 +101,17 @@ void invalidate(String databaseName, String tableName) { cache.keySet().removeIf(key -> key.matches(databaseName, tableName)); } + /** + * Drop every memoized schema for database {@code databaseName} across all its tables / schemaIds / + * sys-tables / branches. Wired onto {@code REFRESH DATABASE} and — via the generic + * {@code PluginDrivenExternalCatalog} dropDb hook — onto a Doris-issued {@code DROP DATABASE} of the + * same name (incl. its FORCE table cascade), so a drop+recreate of a table in that db that reuses a + * schemaId does not serve a stale time-travel schema. Db-scoped analogue of {@link #invalidate}. + */ + void invalidateDb(String databaseName) { + cache.keySet().removeIf(key -> key.matchesDb(databaseName)); + } + /** Drop the whole memo. Wired onto {@code REFRESH CATALOG} (alongside the connector rebuild). */ void invalidateAll() { cache.clear(); @@ -135,6 +146,11 @@ boolean matches(String db, String table) { return databaseName.equals(db) && tableName.equals(table); } + /** True if this key belongs to database {@code db} (any table / schemaId / sys-table / branch). */ + boolean matchesDb(String db) { + return databaseName.equals(db); + } + @Override public boolean equals(Object o) { if (this == o) { diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorCacheTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorCacheTest.java index 81d6fb3fd2844f..7e7aa033229238 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorCacheTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorCacheTest.java @@ -70,4 +70,17 @@ public void schemaTtlOverrideIgnoresUnparseableValue() { // A malformed value must not break catalog schema caching; fall back to no override (engine default). Assertions.assertEquals(OptionalLong.empty(), connector(props("not-a-number")).schemaCacheTtlSecondOverride()); } + + @Test + public void invalidateHooksAreNoThrowOnFreshConnector() { + // Smoke: the REFRESH TABLE / REFRESH DATABASE / REFRESH CATALOG hooks must be safe on a fresh connector + // (they only touch the connector-internal latest-snapshot cache + schema memo; the actual db-scoped + // invalidate semantics are in PaimonLatestSnapshotCacheTest / PaimonSchemaAtMemoTest). invalidateDb + // wires BOTH caches — a mutation dropping the schemaAtMemo half still passes this smoke but fails those. + // MUTATION: an NPE on an empty cache -> red. + PaimonConnector connector = connector(Collections.emptyMap()); + Assertions.assertDoesNotThrow(() -> connector.invalidateTable("db1", "t1")); + Assertions.assertDoesNotThrow(() -> connector.invalidateDb("db1")); + Assertions.assertDoesNotThrow(connector::invalidateAll); + } } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCacheTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCacheTest.java index 4f95e641592c7d..de85e2153b627a 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCacheTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonLatestSnapshotCacheTest.java @@ -128,4 +128,34 @@ public void invalidateAllClearsEverything() { c.invalidateAll(); Assertions.assertEquals(0, c.size()); } + + @Test + public void invalidateDbClearsOnlyThatDbsTables() { + AtomicInteger loads = new AtomicInteger(); + PaimonLatestSnapshotCache c = new PaimonLatestSnapshotCache(100, 1000); + c.getOrLoad(Identifier.create("db1", "t1"), () -> 1L); + c.getOrLoad(Identifier.create("db1", "t2"), () -> 2L); + c.getOrLoad(Identifier.create("db2", "t1"), () -> 3L); + Assertions.assertEquals(3, c.size()); + + // REFRESH DATABASE db1 (or a Doris DROP DATABASE db1) must drop BOTH db1 tables and leave db2 intact. + // MUTATION: invalidateDb a no-op (the inherited SPI default this fix replaces) -> db1.t1 still cached + // -> loads stays 0 / after == 1 -> red. + c.invalidateDb("db1"); + Assertions.assertEquals(1, c.size(), "only db2's single entry must survive"); + + long afterDb1 = c.getOrLoad(Identifier.create("db1", "t1"), () -> { + loads.incrementAndGet(); + return 9L; + }); + Assertions.assertEquals(9L, afterDb1, "db1.t1 must reload live after invalidateDb"); + Assertions.assertEquals(1, loads.get()); + + long db2 = c.getOrLoad(Identifier.create("db2", "t1"), () -> { + loads.incrementAndGet(); + return 7L; + }); + Assertions.assertEquals(3L, db2, "db2 must keep its cached id (not dropped by invalidateDb(db1))"); + Assertions.assertEquals(1, loads.get(), "db2 read must be a hit (no extra load)"); + } } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java index 61d43adba22883..5b3aecd623c71a 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonSchemaAtMemoTest.java @@ -148,6 +148,35 @@ public void invalidateDropsAllSchemaIdsOfOneTableOnly() { Assertions.assertEquals(1, loads.get(), "(db,t) must re-read after invalidate"); } + @Test + public void invalidateDbDropsEveryTableOfOneDbOnly() { + PaimonSchemaAtMemo memo = new PaimonSchemaAtMemo(100); + memo.getOrLoad(handle("db", "t"), 0L, PaimonSchemaAtMemoTest::snap); + memo.getOrLoad(handle("db", "t"), 1L, PaimonSchemaAtMemoTest::snap); // same (db,t), other schemaId + memo.getOrLoad(handle("db", "other"), 0L, PaimonSchemaAtMemoTest::snap); // same db, other table + memo.getOrLoad(handle("db2", "t"), 0L, PaimonSchemaAtMemoTest::snap); // other db + Assertions.assertEquals(4, memo.size()); + + memo.invalidateDb("db"); + + // WHY (R2 / the DROP DATABASE + REFRESH DATABASE fix): invalidateDb must drop EVERY table (and every + // schemaId) of db so a same-name recreate under that db cannot serve a stale time-travel schema, while + // leaving other dbs intact. A mutation matching (db,table) instead of db-only, or a no-op, changes this. + Assertions.assertEquals(1, memo.size(), "all of db's entries gone; only (db2,t) survives"); + + AtomicInteger loads = new AtomicInteger(); + memo.getOrLoad(handle("db2", "t"), 0L, () -> { + loads.incrementAndGet(); + return snap(); + }); + Assertions.assertEquals(0, loads.get(), "the other db's entry must stay a cached hit"); + memo.getOrLoad(handle("db", "other"), 0L, () -> { + loads.incrementAndGet(); + return snap(); + }); + Assertions.assertEquals(1, loads.get(), "a table in the invalidated db must re-read"); + } + @Test public void invalidateAllClearsEverything() { PaimonSchemaAtMemo memo = new PaimonSchemaAtMemo(100); From 949aba750bf3ae96f3f5620360c43f89fe6ae040 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 19:33:53 +0800 Subject: [PATCH 194/334] [fix](catalog) Connector cache: DROP/CREATE/DROP-DB invalidation propagates on editlog replay (R1) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The coordinator PluginDrivenExternalCatalog dropTable/createTable/dropDb hooks drop the connector's OWN cache, but only on the FE that RUNS the DDL. Followers/observers replay via the base ExternalCatalog.replayDropTable/replayCreateTable/replayDropDb plugin branch, which touches only the FE name cache — never connector.invalidate*. So a follower that had queried a paimon/iceberg table kept its latestSnapshotCache[(db,t)] (and paimon's schema memo) pinned to the dropped table's snapshot until the 24h access-TTL (kept alive by continuous querying). Fix (mirrors the existing replayTruncateTable override + RefreshManager.replayRefreshTable → refreshTableInternal, which already forward connector.invalidateTable on replay): override replayDropTable/replayCreateTable/replayDropDb in PluginDrivenExternalCatalog to also call connector.invalidateTable/invalidateDb, keyed exactly like the coordinator (drop/dropDb resolve the REMOTE names from the still-cached object BEFORE the base unregisters it; createTable's table name is NOT remote-resolved, parity with the coordinator). Never force-initializes during replay: getConnector() runs only inside the getDbForReplay/ getTableForReplay match, which is present only when the catalog is already initialized on this FE (both return empty otherwise). A never-initialized catalog has no connector cache to drop, so skipping it is correct — mirrors HiveConnector.forEachBuiltSibling and preserves the base's no-force-init replay behavior (verified by testReplayDropTableUninitializedCatalogSkipsInvalidate). Tests: PluginDrivenExternalCatalogDdlRoutingTest +4 (replayDropTable/replayDropDb/replayCreateTable propagate to the connector by remote names; uninitialized-catalog replay skips it) — 60 pass, fe-core BUILD SUCCESS, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../PluginDrivenExternalCatalog.java | 48 ++++++++++++ ...inDrivenExternalCatalogDdlRoutingTest.java | 74 +++++++++++++++++++ 2 files changed, 122 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index 0026dd34d8b7b8..03d20b42e638e5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -705,6 +705,54 @@ public void replayTruncateTable(TruncateTableInfo info) { Env.getCurrentEnv().getRefreshManager().refreshTableInternal(db, tbl, info.getUpdateTime()))); } + /** + * Propagates the coordinator {@link #dropTable} hook's connector-cache invalidation to followers/observers + * on edit-log replay. The base {@link ExternalCatalog#replayDropTable} plugin branch only touches the FE + * name cache ({@code unregisterTable}); without this, a follower that had queried a paimon/iceberg table + * keeps its latest-snapshot pin (and paimon's schema memo) for the dropped name until the 24h access-TTL — + * the coordinator-only half of the drop+recreate fix. Resolves the REMOTE names from the still-cached + * table BEFORE the base unregisters it, keyed exactly like the coordinator (mirrors + * {@code RefreshManager.replayRefreshTable → refreshTableInternal}'s connector hook). + * + *

    Never force-initializes during replay: {@code getConnector()} runs only inside the + * {@code getDbForReplay}/{@code getTableForReplay} match, which is present only when this catalog is + * already initialized on this FE (both return empty otherwise). A never-initialized catalog has no + * connector cache to drop, so skipping it is correct — mirroring {@code HiveConnector.forEachBuiltSibling} + * ("a never-built sibling has no cache") and preserving the base's no-force-init replay behavior. + */ + @Override + public void replayDropTable(String dbName, String tblName) { + getDbForReplay(dbName).ifPresent(db -> + db.getTableForReplay(tblName).ifPresent(tbl -> + getConnector().invalidateTable(db.getRemoteName(), tbl.getRemoteName()))); + super.replayDropTable(dbName, tblName); + } + + /** + * Replay analogue of the coordinator {@link #dropDb} hook's connector-cache invalidation — clears every + * table's connector cache for the dropped database on followers/observers (the base + * {@link ExternalCatalog#replayDropDb} plugin branch only unregisters the FE db). Resolves the REMOTE db + * name BEFORE the base unregisters the database. See {@link #replayDropTable} for the no-force-init + * rationale. + */ + @Override + public void replayDropDb(String dbName) { + getDbForReplay(dbName).ifPresent(db -> getConnector().invalidateDb(db.getRemoteName())); + super.replayDropDb(dbName); + } + + /** + * Replay analogue of the coordinator {@link #createTable} hook's belt-and-suspenders connector-cache + * invalidation (uniform with the drop path). The table name is NOT remote-resolved — parity with the + * coordinator, where a brand-new table has no local→remote mapping. See {@link #replayDropTable} for the + * no-force-init rationale. + */ + @Override + public void replayCreateTable(String dbName, String tblName) { + super.replayCreateTable(dbName, tblName); + getDbForReplay(dbName).ifPresent(db -> getConnector().invalidateTable(db.getRemoteName(), tblName)); + } + /** * Routes {@code ALTER TABLE ... ADD/DROP/RENAME/MODIFY/REORDER COLUMN} through the SPI's * {@code ConnectorTableOps} column-evolution methods instead of the legacy {@code metadataOps} path used diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java index ee2172b51af60b..b1649340541fb5 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -830,6 +830,80 @@ public void testCreateTableLocalConflictWithoutIfNotExistsRejects() throws Excep } } + // ==================== EDIT-LOG REPLAY (follower / observer propagation) ==================== + // R1: the coordinator dropTable/createTable/dropDb hooks drop the connector's OWN cache, but editlog + // replay on followers/observers went through the base ExternalCatalog.replay* plugin branch, which only + // touched the FE name cache — so a follower kept the dropped/renamed object's snapshot pin to the TTL. + // These overrides propagate connector.invalidate* on replay too, keyed like the coordinator, WITHOUT + // force-initializing a catalog (the getDbForReplay/getTableForReplay match is present only when already + // initialized on this FE). + + @Test + public void testReplayDropTableInvalidatesConnectorOnFollower() { + // local db1.t1 maps to remote DB1.TBL1; the table is still in the replay cache when the drop replays. + ExternalDatabase replayDb = mockExternalDatabase(); + Mockito.when(replayDb.getRemoteName()).thenReturn("DB1"); + ExternalTable cached = Mockito.mock(ExternalTable.class); + Mockito.when(cached.getRemoteName()).thenReturn("TBL1"); + Mockito.doReturn(Optional.of(cached)).when(replayDb).getTableForReplay("t1"); + catalog.dbForReplayResult = Optional.of(replayDb); + + catalog.replayDropTable("db1", "t1"); + + // WHY (Rule 9 / R1): without the override a follower kept the dropped table's latest-snapshot pin (and + // paimon schema memo) to the 24h TTL — the coordinator-only half of the drop+recreate fix. Replay must + // drop it, keyed by the REMOTE names resolved from the still-cached table BEFORE unregister. MUTATION: + // removing the override, or passing the LOCAL names, turns this red. + Mockito.verify(connector).invalidateTable("DB1", "TBL1"); + // Base bookkeeping is preserved: the table is still unregistered from the FE name cache. + Mockito.verify(replayDb).unregisterTable("t1"); + } + + @Test + public void testReplayDropTableUninitializedCatalogSkipsInvalidate() { + // A follower that never initialized this catalog: getDbForReplay returns empty -> no connector cache + // exists to drop, and the override must NOT force-initialize the catalog (no invalidate, no throw). + catalog.dbForReplayResult = Optional.empty(); + + catalog.replayDropTable("db1", "t1"); + + // WHY (R1 no-force-init): the connector is untouched when the catalog is not initialized on this FE. + // MUTATION: using getConnector() unconditionally (force-init) would invoke the connector here -> red. + Mockito.verify(connector, Mockito.never()).invalidateTable(Mockito.any(), Mockito.any()); + } + + @Test + public void testReplayDropDbInvalidatesConnectorOnFollower() { + ExternalDatabase replayDb = mockExternalDatabase(); + Mockito.when(replayDb.getRemoteName()).thenReturn("DB1"); + catalog.dbForReplayResult = Optional.of(replayDb); + + catalog.replayDropDb("db1"); + + // WHY (R1): DROP DATABASE replay must drop every table's connector cache for the db (keyed by the + // REMOTE db name), mirroring the coordinator dropDb hook. MUTATION: removing the override or passing + // the LOCAL "db1" turns this red. + Mockito.verify(connector).invalidateDb("DB1"); + // Base bookkeeping is preserved (the db is unregistered by the LOCAL name). + Assertions.assertEquals("db1", catalog.unregisteredDb); + } + + @Test + public void testReplayCreateTableInvalidatesConnectorOnFollower() { + ExternalDatabase replayDb = mockExternalDatabase(); + Mockito.when(replayDb.getRemoteName()).thenReturn("DB1"); + catalog.dbForReplayResult = Optional.of(replayDb); + + catalog.replayCreateTable("db1", "t1"); + + // WHY (R1): belt-and-suspenders parity with the coordinator createTable hook — keyed by the REMOTE db + // name "DB1" but the (non-remote-resolved) table name "t1". MUTATION: flipping the table name to remote + // or dropping the call turns this red. + Mockito.verify(connector).invalidateTable("DB1", "t1"); + // Base bookkeeping is preserved (the db's table-name cache is refreshed). + Mockito.verify(replayDb).resetMetaCacheNames(); + } + // ==================== COLUMN EVOLUTION (B2) ==================== // The 6 column-op overrides resolve the connector handle by REMOTE names (like dropTable), convert the // Doris Column/ColumnPosition to the neutral SPI types, dispatch, wrap DorisConnectorException as From 0dabe25fa0cd65bae604d051fc2bafdb61ce81e0 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 19:44:44 +0800 Subject: [PATCH 195/334] [fix](catalog) Connector cache: RENAME TABLE invalidates the connector cache (coordinator + replay) (R4) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit renameTable → afterExternalRename ran unregister + resetMetaCacheNames + constraint rename + editlog but NEVER called connector.invalidate*, and did NOT route through refreshTableInternal (unlike truncate / column-ALTER). So an iceberg/paimon atomic table-swap (RENAME t→t_arch; RENAME t_new→t) left the stale latestSnapshotCache[(db,t)] (and paimon's schema memo) pinned under BOTH names → the recreated t read the OLD snapshot up to the TTL. Same drop+recreate class the dropTable hook covers; the design doc even cited Trino's renameTable self-invalidation as the parity model, then only hooked dropTable/createTable/dropDb. Fix (both the coordinator and the follower/observer replay, since rename is logged as a createForRenameTable refresh-external-table entry): - PluginDrivenExternalCatalog.renameTable: after a successful metadata.renameTable, drop the connector caches for BOTH the source (resolved REMOTE names) and the target (new name in the same remote db, not remote-resolved — parity with createTable). Placed after the mutation so a remote failure aborts before touching the cache. - RefreshManager.replayRefreshTable rename branch: propagate the same source+target invalidation, resolving the source's REMOTE names from the still-cached table BEFORE unregister. getConnector() does not force-init here — the branch is reached only for an already-initialized catalog (db + table resolved from the replay cache), mirroring the else branch's refreshTableInternal hook. No replaceTable/CTAS-overwrite path exists for plugin catalogs (verified), so RENAME is the only remaining gap of this class. Tests: PluginDrivenExternalCatalogDdlRoutingTest — rename now asserts connector.invalidateTable for source (DB1.TBL1) + target (DB1.t2), and the failure test asserts no invalidate on abort (60 pass); new RefreshManagerRenameReplayTest — replay propagates source+target by remote names, and an uninitialized-catalog replay skips it (2 pass). fe-core BUILD SUCCESS, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../apache/doris/catalog/RefreshManager.java | 14 ++ .../PluginDrivenExternalCatalog.java | 8 ++ .../RefreshManagerRenameReplayTest.java | 124 ++++++++++++++++++ ...inDrivenExternalCatalogDdlRoutingTest.java | 10 ++ 4 files changed, 156 insertions(+) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerRenameReplayTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java index bec60a78c918fb..3eb669810388ef 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/catalog/RefreshManager.java @@ -21,6 +21,7 @@ import org.apache.doris.common.DdlException; import org.apache.doris.common.ThreadPoolManager; import org.apache.doris.common.UserException; +import org.apache.doris.connector.api.Connector; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.CatalogLog; import org.apache.doris.datasource.ExternalCatalog; @@ -191,6 +192,19 @@ public void replayRefreshTable(ExternalObjectLog log) { } if (!Strings.isNullOrEmpty(log.getNewTableName())) { // this is a rename table op + // R4: propagate the coordinator renameTable's connector-cache invalidation (source + target) to + // followers/observers — the base replay below only fixes the FE name cache, so a follower kept the + // source name's snapshot pin (and paimon's schema memo) to the TTL after an atomic table swap. + // Resolve the source's REMOTE names from the still-cached table BEFORE unregister; the target keeps + // the new name (parity with the coordinator). getConnector() does not force-init here: this branch + // is reached only for an already-initialized catalog (db + table were resolved from the replay + // cache above), mirroring the else branch's refreshTableInternal -> getConnector() hook. + if (catalog instanceof PluginDrivenExternalCatalog) { + Connector connector = ((PluginDrivenExternalCatalog) catalog).getConnector(); + String remoteDb = db.get().getRemoteName(); + connector.invalidateTable(remoteDb, table.get().getRemoteName()); + connector.invalidateTable(remoteDb, log.getNewTableName()); + } db.get().unregisterTable(log.getTableName()); db.get().resetMetaCacheNames(); Env.getCurrentEnv().getConstraintManager().renameTable( diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index 03d20b42e638e5..a9c5c5195c78b7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -646,6 +646,14 @@ public void renameTable(String dbName, String oldTableName, String newTableName) } catch (DorisConnectorException e) { throw new DdlException(e.getMessage(), e); } + // R4: drop the connector's OWN caches for BOTH the source and target names so an atomic swap + // (RENAME t->t_arch; RENAME t_new->t) doesn't serve the pre-rename pinned snapshot under either name — + // afterExternalRename only fixes the FE name cache. Same drop+recreate class the dropTable hook covers. + // Connector-agnostic (no-op SPI default); the source is keyed by its resolved REMOTE names, the target + // by the new name in the same remote db (parity with createTable: a rename target has no prior + // local->remote mapping). Followers propagate this via RefreshManager.replayRefreshTable's rename branch. + connector.invalidateTable(dorisTable.getRemoteDbName(), dorisTable.getRemoteName()); + connector.invalidateTable(dorisTable.getRemoteDbName(), newTableName); afterExternalRename(dbName, oldTableName, newTableName); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerRenameReplayTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerRenameReplayTest.java new file mode 100644 index 00000000000000..e0647a69ad45ac --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/RefreshManagerRenameReplayTest.java @@ -0,0 +1,124 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.catalog; + +import org.apache.doris.catalog.constraint.ConstraintManager; +import org.apache.doris.connector.api.Connector; +import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.ExternalDatabase; +import org.apache.doris.datasource.ExternalObjectLog; +import org.apache.doris.datasource.ExternalTable; +import org.apache.doris.datasource.PluginDrivenExternalCatalog; + +import org.junit.jupiter.api.AfterEach; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.MockedStatic; +import org.mockito.Mockito; + +import java.util.Optional; + +/** + * Tests {@link RefreshManager#replayRefreshTable}'s RENAME branch (R4). A connector-driven RENAME TABLE is + * logged as a {@code createForRenameTable} external-object log and replayed here on followers/observers. + * + *

    Before R4 the replay only fixed the FE name cache ({@code unregisterTable} + {@code resetMetaCacheNames} + * + constraint rename) — it never dropped the connector's OWN caches, so a follower that had queried the + * source table kept its latest-snapshot pin (and paimon's schema memo) to the 24h TTL after an atomic table + * swap. This pins that the replay now propagates {@code connector.invalidateTable} for BOTH the source and + * target names, keyed exactly like the coordinator {@code PluginDrivenExternalCatalog.renameTable} hook. + */ +public class RefreshManagerRenameReplayTest { + + private MockedStatic mockedEnv; + private CatalogMgr catalogMgr; + private RefreshManager refreshManager; + + @BeforeEach + public void setUp() { + Env mockEnv = Mockito.mock(Env.class); + catalogMgr = Mockito.mock(CatalogMgr.class); + ConstraintManager constraintManager = Mockito.mock(ConstraintManager.class); + mockedEnv = Mockito.mockStatic(Env.class); + mockedEnv.when(Env::getCurrentEnv).thenReturn(mockEnv); + Mockito.when(mockEnv.getCatalogMgr()).thenReturn(catalogMgr); + Mockito.when(mockEnv.getConstraintManager()).thenReturn(constraintManager); + refreshManager = new RefreshManager(); + } + + @AfterEach + public void tearDown() { + if (mockedEnv != null) { + mockedEnv.close(); + } + } + + @SuppressWarnings("unchecked") + private static ExternalDatabase mockDb() { + return (ExternalDatabase) Mockito.mock(ExternalDatabase.class); + } + + @Test + public void testRenameReplayInvalidatesConnectorSourceAndTargetOnFollower() { + // local db1.t1 -> t2, mapping to remote DB1.TBL1 (source). The source table is still in the replay + // cache when the rename replays (getDbForReplay/getTableForReplay resolve it), so the catalog is + // already initialized on this FE and getConnector() does not force-init. + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Connector connector = Mockito.mock(Connector.class); + ExternalDatabase db = mockDb(); + ExternalTable table = Mockito.mock(ExternalTable.class); + // doReturn (not when().thenReturn) because getCatalog returns a wildcard CatalogIf + // whose capture rejects a concrete PluginDrivenExternalCatalog in the typed stubbing form. + Mockito.doReturn(catalog).when(catalogMgr).getCatalog(7L); + Mockito.when(catalog.getName()).thenReturn("c"); + Mockito.when(catalog.getConnector()).thenReturn(connector); + Mockito.doReturn(Optional.of(db)).when(catalog).getDbForReplay("db1"); + Mockito.when(db.getRemoteName()).thenReturn("DB1"); + Mockito.doReturn(Optional.of(table)).when(db).getTableForReplay("t1"); + Mockito.when(table.getRemoteName()).thenReturn("TBL1"); + + refreshManager.replayRefreshTable(ExternalObjectLog.createForRenameTable(7L, "db1", "t1", "t2")); + + // WHY (Rule 9 / R4): both the source (REMOTE DB1.TBL1) and the target (DB1.t2, new name NOT + // remote-resolved — parity with the coordinator) connector caches must be dropped so an atomic swap + // doesn't serve the pre-rename pin. MUTATION: removing the rename-branch invalidation, or passing the + // LOCAL names, turns this red. + Mockito.verify(connector).invalidateTable("DB1", "TBL1"); + Mockito.verify(connector).invalidateTable("DB1", "t2"); + // Base bookkeeping is preserved: the source name is unregistered and the db's name cache reset. + Mockito.verify(db).unregisterTable("t1"); + Mockito.verify(db).resetMetaCacheNames(); + } + + @Test + public void testRenameReplayUninitializedCatalogSkipsInvalidate() { + // A follower that never initialized this catalog: getDbForReplay returns empty, so the replay resolves + // no db/table and returns early — the connector is never consulted (no force-init, no cache to drop). + PluginDrivenExternalCatalog catalog = Mockito.mock(PluginDrivenExternalCatalog.class); + Connector connector = Mockito.mock(Connector.class); + Mockito.doReturn(catalog).when(catalogMgr).getCatalog(7L); + Mockito.doReturn(Optional.empty()).when(catalog).getDbForReplay("db1"); + + refreshManager.replayRefreshTable(ExternalObjectLog.createForRenameTable(7L, "db1", "t1", "t2")); + + // WHY (R4 no-force-init): an uninitialized catalog has no connector cache to drop; the replay must not + // touch (or force-build) the connector. MUTATION: force-initializing / invalidating here -> red. + Mockito.verify(catalog, Mockito.never()).getConnector(); + Mockito.verifyNoInteractions(connector); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java index b1649340541fb5..b57b29d3ca7ec5 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalCatalogDdlRoutingTest.java @@ -580,6 +580,13 @@ public void testRenameTableResolvesRemoteSourceRoutesAndFixesCache() throws Exce Assertions.assertEquals("db1", logCap.getValue().getDbName()); Assertions.assertEquals("t1", logCap.getValue().getTableName()); Assertions.assertEquals("t2", logCap.getValue().getNewTableName()); + // WHY (Rule 9 / R4): the connector's own caches for BOTH the source (REMOTE DB1.TBL1) and the target + // (DB1.t2, new name NOT remote-resolved) must be dropped so an atomic swap (RENAME t->t_arch; + // RENAME t_new->t) doesn't serve the pre-rename pinned snapshot under either name. Before R4 + // afterExternalRename fixed only the FE name cache. MUTATION: dropping either call — or passing the + // LOCAL names — turns this red. + Mockito.verify(connector).invalidateTable("DB1", "TBL1"); + Mockito.verify(connector).invalidateTable("DB1", "t2"); } @Test @@ -619,6 +626,9 @@ public void testRenameTableWrapsConnectorExceptionAndSkipsBookkeeping() { // so the FE cache + constraints stay consistent with the unchanged remote. Mockito.verify(mockEditLog, Mockito.never()).logRefreshExternalTable(Mockito.any()); Mockito.verifyNoInteractions(mockConstraintManager); + // WHY (R4): the connector-cache invalidation sits AFTER the successful renameTable, so a remote + // failure must NOT drop the connector cache — it stays consistent with the unchanged remote. + Mockito.verify(connector, Mockito.never()).invalidateTable(Mockito.any(), Mockito.any()); } // ==================== CREATE TABLE ==================== From ab0a9923cfab22a123e0fc5c5f165c861dbbd60b Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 19:45:41 +0800 Subject: [PATCH 196/334] [fix](catalog) Connector cache: hive getPartitions put is generation-guarded vs a racing REFRESH (R3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The D2-rewritten getPartitions used partitionsCache.put(...) directly, which has no generation check. The pre-D2 code went through partitionsCache.get(key, loader) -> getWithManualLoad, which captures invalidateGeneration before the load and drops the put if a flush raced it. So a REFRESH TABLE (flush) racing an in-flight cold-cache fetch now re-caches the pre-refresh partitions, which survive until TTL/next REFRESH. getTable / listPartitionNames / getTableColumnStatistics still use the guarded get path — only getPartitions' per-partition put had lost it. (Dormant: hive not flipped, flush not yet wired to REFRESH for a live hms catalog.) Fix — additive generation-guarded put on the connector MetaCacheEntry framework copy (no behavior/ byte change for paimon/iceberg, which don't call it; same Caffeine version): - MetaCacheEntry.invalidationGeneration() + putIfNotInvalidatedSince(gen, key, value): put under the key's stripe lock, skip if the generation moved before the put and removeLoadedValue if it moved between the put and the recheck — exactly the guard getWithManualLoad already applies to its single-key load. - CachingHmsClient.getPartitions: capture the generation BEFORE the delegate RPC, then guard each per-partition put with it. The delegate results still populate the RESULT list directly, so the misparse->never-drop safety is preserved (only the CACHE put is guarded). Not synced to the fe-core MetaCacheEntry copy: fe-core has no getPartitions-style raw put and its legacy caches are retired at the flip, so the addition is connector-only (Rule 3, surgical). Tests: MetaCacheEntryTest.putIfNotInvalidatedSinceHonorsGenerationGuard (7 pass); CachingHmsClientTest.getPartitionsFlushRacingInFlightFetchDoesNotRecacheStale — a flush fired mid-RPC (fake hook) must leave nothing cached so a follow-up read re-fetches (16 pass). fe-connector-cache + fe-connector-hms BUILD SUCCESS, checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- .../doris/connector/cache/MetaCacheEntry.java | 39 +++++++++++++++++++ .../connector/cache/MetaCacheEntryTest.java | 31 +++++++++++++++ .../doris/connector/hms/CachingHmsClient.java | 10 ++++- .../connector/hms/CachingHmsClientTest.java | 31 +++++++++++++++ 4 files changed, 110 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheEntry.java b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheEntry.java index 0e38e11a91899a..425697ef9b1098 100644 --- a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheEntry.java +++ b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/MetaCacheEntry.java @@ -157,6 +157,45 @@ public void put(K key, V value) { data.put(key, value); } + /** + * The current invalidation generation. Capture this BEFORE a slow external load, then hand it to + * {@link #putIfNotInvalidatedSince} so a {@code flush}/{@code invalidate*} that raced the load does not + * get its clear silently undone by a stale write-back. Mirrors the guard the manual-miss-load path + * ({@link #getWithManualLoad}) applies around its own put; exposed so a caller that does its OWN bulk + * external read (e.g. a decorator batching a multi-key RPC and putting each result under its own key) + * can reuse the same generation guard instead of an unguarded {@link #put}. + */ + public long invalidationGeneration() { + return invalidateGeneration.get(); + } + + /** + * Generation-guarded put: caches {@code (key, value)} only if no invalidation has happened since + * {@code generation} was captured (before the caller's external load). If a {@code flush}/{@code + * invalidate*} raced the load — bumping the generation either before the put (skip) or between the put + * and the recheck (drop only the value we wrote, via {@link #removeLoadedValue}) — the stale value is + * NOT left cached, exactly as {@link #getWithManualLoad} does for its single-key load. Additive: the + * existing {@link #put} is unchanged; a disabled entry is a no-op. + */ + public void putIfNotInvalidatedSince(long generation, K key, V value) { + if (!effectiveEnabled) { + return; + } + synchronized (loadLock(key)) { + // A racing flush already bumped the generation before we could put: skip so a stale pre-flush + // value is not re-cached (mirrors getWithManualLoad's pre-put guard). + if (generation != invalidateGeneration.get()) { + return; + } + data.put(key, value); + // A flush landing between the check and the put: drop only the value we just wrote, keeping any + // newer replacement intact (mirrors getWithManualLoad's post-put guard). + if (generation != invalidateGeneration.get()) { + removeLoadedValue(key, value); + } + } + } + public void invalidateKey(K key) { invalidateGeneration.incrementAndGet(); if (data.asMap().remove(key) != null) { diff --git a/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheEntryTest.java b/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheEntryTest.java index 8782bf9f1439d9..8284f4ae1ed672 100644 --- a/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheEntryTest.java +++ b/fe/fe-connector/fe-connector-cache/src/test/java/org/apache/doris/connector/cache/MetaCacheEntryTest.java @@ -212,6 +212,37 @@ public void manualMissLoadDeduplicatesConcurrentSameKey() throws Exception { } } + @Test + public void putIfNotInvalidatedSinceHonorsGenerationGuard() { + ExecutorService refreshExecutor = Executors.newSingleThreadExecutor(); + try { + // Contextual + manual-miss entry, mirroring the hive partition-object cache that uses the guarded put. + CacheSpec cacheSpec = CacheSpec.of(true, CacheSpec.CACHE_NO_TTL, 10L); + MetaCacheEntry entry = new MetaCacheEntry<>( + "test", null, cacheSpec, refreshExecutor, false, true, 0L, true); + + // No invalidation since the captured generation -> the guarded put caches normally. + long g1 = entry.invalidationGeneration(); + entry.putIfNotInvalidatedSince(g1, "a", 1); + Assertions.assertEquals(Integer.valueOf(1), entry.getIfPresent("a")); + + // An invalidation between the capture and the put (the flush-races-an-in-flight-load case) must make + // the put a no-op, so a stale pre-invalidation value is NOT re-cached to the TTL. + long g2 = entry.invalidationGeneration(); + entry.invalidateAll(); // bumps the generation, as a racing flush / REFRESH would + entry.putIfNotInvalidatedSince(g2, "b", 2); + // MUTATION: an unguarded put (the pre-R3 raw put) leaves "b" cached here -> getIfPresent == 2 -> red. + Assertions.assertNull(entry.getIfPresent("b"), "a stale-generation put must not re-cache the value"); + + // A generation captured AFTER the invalidation puts normally again (the guard is not a one-shot latch). + long g3 = entry.invalidationGeneration(); + entry.putIfNotInvalidatedSince(g3, "c", 3); + Assertions.assertEquals(Integer.valueOf(3), entry.getIfPresent("c")); + } finally { + refreshExecutor.shutdownNow(); + } + } + @SuppressWarnings("unchecked") private static LoadingCache extractLoadingCache(MetaCacheEntry entry) throws Exception { Field field = MetaCacheEntry.class.getDeclaredField("loadingData"); diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index 963eedd3237ced..ce70d268da1ee3 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -178,8 +178,16 @@ public List getPartitions(String dbName, String tableName, Lis } } if (missNames != null) { + // Capture the invalidation generation BEFORE the delegate RPC so a REFRESH (flush) that races this + // in-flight cold-cache fetch does not get silently undone by re-caching the pre-refresh partitions. + // The pre-D2 code went through partitionsCache.get(key, loader) -> getWithManualLoad, which had this + // guard; the per-partition put must restore it (getTable/listPartitionNames/getTableColumnStatistics + // still use the guarded get path). The delegate results still populate the RESULT list directly, + // preserving the misparse->never-drop safety (only the CACHE put is generation-guarded). + long generation = partitionsCache.invalidationGeneration(); for (HmsPartitionInfo info : delegate.getPartitions(dbName, tableName, missNames)) { - partitionsCache.put(new PartitionKey(dbName, tableName, info.getValues()), info); + partitionsCache.putIfNotInvalidatedSince( + generation, new PartitionKey(dbName, tableName, info.getValues()), info); result.add(info); } } diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java index 04894c5261017a..8c90fcc6bf2e77 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -191,6 +191,31 @@ public void getPartitionsStaysCorrectWhenParsedNameDivergesFromStoredValues() { "divergence degrades to a reload, never a wrong/dropped result"); } + @Test + public void getPartitionsFlushRacingInFlightFetchDoesNotRecacheStale() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Model a REFRESH TABLE (flush) landing DURING the cold-cache delegate RPC: getPartitions captures the + // invalidation generation BEFORE the RPC, the flush bumps it mid-RPC, so the per-partition guarded put + // must be dropped rather than re-cache the pre-refresh partition. The in-flight query still returns the + // freshly-fetched partition (only the CACHE put is guarded). + delegate.onGetPartitions = () -> cache.flush("db", "t"); + List r = cache.getPartitions("db", "t", Arrays.asList("p=1")); + Assertions.assertEquals(1, r.size(), "the in-flight query still returns the delegate's partition"); + Assertions.assertEquals(1, delegate.getPartitionsCalls); + + // WHY (Rule 9 / R3): the racing flush must have prevented the stale put, so a follow-up read is a MISS + // and re-fetches — it is NOT served the pre-refresh partition up to the TTL. MUTATION: the pre-R3 raw + // partitionsCache.put re-caches the stale partition here, so the next read hits and getPartitionsCalls + // stays 1 -> red. (getTable/listPartitionNames/getTableColumnStatistics kept the guard via get(); only + // getPartitions' per-partition put had lost it.) + delegate.onGetPartitions = null; + cache.getPartitions("db", "t", Arrays.asList("p=1")); + Assertions.assertEquals(2, delegate.getPartitionsCalls, + "a flush racing the in-flight fetch must not leave the stale partition cached (guarded put)"); + } + // ---- column statistics ---- @Test @@ -431,6 +456,9 @@ private static final class RecordingHmsClient implements HmsClient { // The partition-name list the decorator actually asked the delegate for on the LAST getPartitions call // (so a test can assert the decorator fetches only the MISSES, not the whole requested list). List lastGetPartitionsArg; + // Optional hook fired INSIDE getPartitions (after counting, before returning) to model a concurrent + // mutation (e.g. a REFRESH flush) racing the in-flight cold-cache RPC. + Runnable onGetPartitions; @Override public HmsTableInfo getTable(String dbName, String tableName) { @@ -451,6 +479,9 @@ public List listPartitionNames(String dbName, String tableName, int maxP public List getPartitions(String dbName, String tableName, List partNames) { getPartitionsCalls++; lastGetPartitionsArg = new ArrayList<>(partNames); + if (onGetPartitions != null) { + onGetPartitions.run(); + } List out = new ArrayList<>(); for (String name : partNames) { if (absentPartitionNames.contains(name)) { From f1724abebb0c804cd7ac32121044a041fe7855c5 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 20:09:04 +0800 Subject: [PATCH 197/334] [doc](catalog) Cache-invalidation R1-R4 DONE + re-review CLEAN; HANDOFF -> Phase 2 cutover MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The connector-cache invalidation follow-up line (D1/D2 -> R1-R4) is complete and validated: - R2 7b7b3c25953 (iceberg/paimon db-scoped invalidateDb), R1 a562c91e55b (drop/create/dropdb editlog-replay propagation), R4 43db3e8214f (RENAME coordinator+replay), R3 d26bfa52eea (getPartitions generation-guarded put) — each its own commit, BUILD SUCCESS + 0 checkstyle. - Targeted adversarial re-review wf_b730a7d4-6a3 (6 blind finders + 3-lens verify) returned CLEAN: 0 findings, the four fixes are correct AND complete. Confirms hudi correctly needs no invalidateDb (raw ThriftHmsClient, reads fresh), no other same-class path unfixed, replay never force-inits. New report: plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md. Design doc §6 all-checked; HANDOFF rewritten: Phase 1 cache line DONE, next = Phase 2 atomic cutover prerequisites (SPI_READY += hms, GSON remap, 4 guards, dead-arm/deletion ordering) + the now-live paimon/iceberg drop+recreate / rename-swap e2e regression. Fixed stale tasks/ -> plan-doc/tasks/ path prefixes in the startup-reading section. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: catalog-spi-11-hive cache-invalidation-followups --- plan-doc/HANDOFF.md | 43 +++++++------- ...-invalidation-fixes-rereview-2026-07-10.md | 57 +++++++++++++++++++ ...nvalidation-followups-design-2026-07-10.md | 33 ++++++++--- 3 files changed, 100 insertions(+), 33 deletions(-) create mode 100644 plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 29369c0756503c..e84e267ca0ba10 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -10,30 +10,23 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： - **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 -- **Phase 1 翻闸前 fe-core 建设（进行中，真正剩余主活）** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → 耦合缝净室复审 ✅ → 缓存失效收尾 3 修 ✅（用户拍板+已修+已验）→ 对抗复核 ✅ **查出 4 个待修（3 线上+1 休眠，见下）** → 剩余（修完 4 项 → 原子翻闸前置项）。 +- **Phase 1 翻闸前 fe-core 建设** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → 耦合缝净室复审 ✅ → 缓存失效收尾 3 修 ✅ → 对抗复核查出 4 个待修 ✅ → **缓存失效 4 修(R1–R4)全部 ✅ 落地+构建验证+净室对抗复核 CLEAN（见下）**。**Phase 1 缓存线收尾完毕，真正剩余主活 = Phase 2 原子翻闸前置项。** - **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 -**⭐ 本轮（2026-07-10）= 连接器缓存第二轮净室复审留下的 4 个"缓存失效"发现，逐一对照 HEAD 核实后收尾。** -先做的独立核实（`wf_daed84c1-4ea`）有一处**重要更正**：其中一个 HIGH（REFRESH 刷不到 iceberg 兄弟快照缓存）**在 HEAD 已经修好了**（`forEachBuiltSibling` 已接进 invalidate\*），撤销无需处理。剩下 3 项按用户拍板落地，**每项独立 commit + 独立构建验证（BUILD SUCCESS + 0 checkstyle）**： +**⭐ 本轮（2026-07-10）= 缓存失效对抗复核留下的 4 个待修（R1–R4）全部收尾 + 复核 CLEAN。连接器缓存失效这条线（D1/D2 → R1–R4）到此 DONE。** +每项独立 commit + 独立构建验证（BUILD SUCCESS + 0 checkstyle）： -- **✅ D1（fe-core 通用钩子，线上改动）** `3b66982fedf`：Doris 自执行 `DROP/CREATE TABLE`、`DROP DATABASE` 从不清连接器自持缓存（只有 REFRESH 清）→ **同名 drop+recreate 命中老表快照/元数据（最长 24h TTL）**。休眠的 hive（翻闸后触发）+ **线上的 paimon/iceberg（现在就触发）** 一次修好：在 `PluginDrivenExternalCatalog` 的 drop/create-table、dropDb 里统一调 `connector.invalidate*`（REMOTE 名，照搬 Trino"谁改动谁清"）；`createDb` 有意不接。`PluginDrivenExternalCatalogDdlRoutingTest` 56 绿。 -- **✅ D1 补丁（paimon 第二层缓存）** `7b8fed012be`：`PaimonSchemaAtMemo`（时间旅行 schema memo）原本无 invalidate 入口 → 补 `invalidate/invalidateAll` 并接进 `PaimonConnector.invalidateTable/invalidateAll`，补齐 paimon 侧的 drop+recreate 洞（iceberg 无需：manifest 按路径 key + 快照缓存已被同一钩子清）。`PaimonSchemaAtMemoTest` 5 绿。 -- **✅ D2（分区缓存容量单位）** `982db925659`：`CachingHmsClient` 分区对象缓存原按"整条请求名列表"做 key（100000 变成"10 万个列表"而非"10 万分区对象"，可撑爆 FE 堆）→ 改成**每分区一个 value key**（对齐 Trino + 老实现），批量取分区做部分命中 + 只补缺失；正确性与名字解析保真度无关（store 永远按分区真实 values，误解析只退化成重取，绝不错/漏）。休眠。`CachingHmsClientTest` 15 绿。 +- **✅ R2（线上）** `7b7b3c25953`：iceberg/paimon 补 `invalidateDb`（按 db 前缀 removeIf 清快照缓存 + paimon `PaimonSchemaAtMemo`）→ `DROP DATABASE` 钩子 + 老早存在的 `REFRESH DATABASE` 对它俩生效（含 `DROP DATABASE FORCE` 级联），并 arm 了 hive `forEachBuiltSibling(sibling.invalidateDb)`。iceberg 16 + paimon 17 绿。 +- **✅ R1（线上）** `a562c91e55b`：`PluginDrivenExternalCatalog` override `replayDropTable/replayCreateTable/replayDropDb`（仿 `replayTruncateTable`），follower/observer 回放也传播 `connector.invalidate*`，按 REMOTE 名 keying；靠 getDbForReplay 门控绝不 force-init。DDL routing 60 绿。 +- **✅ R4（线上）** `43db3e8214f`：`renameTable` 改名成功后清源+目标；`RefreshManager.replayRefreshTable` rename 分支同步传播到 follower。已确认插件目录无 replace/CTAS-overwrite 路径 → 改名是同类唯一漏洞。DDL routing 60 + 新 `RefreshManagerRenameReplayTest` 2 绿。 +- **✅ R3（休眠）** `d26bfa52eea`：连接器版 `MetaCacheEntry` 加 additive `invalidationGeneration()`/`putIfNotInvalidatedSince()`，`CachingHmsClient.getPartitions` RPC 前抓 generation、逐分区守卫式 put（结果仍直接从 delegate 收，保 misparse-never-drop）。`MetaCacheEntryTest` 7 + `CachingHmsClientTest` 16 绿。 -设计文档（**起步必读**，含正确性论证 + e2e 欠账）= **`tasks/hive-cache-invalidation-followups-design-2026-07-10.md`**（§6 状态：3 修全勾 + 复审在跑）。 +**✅ 针对性对抗复核（`wf_b730a7d4-6a3`，6 盲审 finder + 每发现 3 视角验证）：CLEAN — 0 发现，四修既正确又完整，无后续。** 关键确认：**hudi 兄弟不需要 `invalidateDb`**（它建裸 `ThriftHmsClient` 直读、非 `CachingHmsClient`）；无其它同类未修路径；replay 绝不 force-init；原子换名不留陈旧钉。报告 = `plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md`。设计文档 = `plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md`（§6 全勾）。 -**⭐ 净室对抗复核结论（`wf_fe6ddef4-777`，已完成，28 agent）：3 修是净改进但 INCOMPLETE，查出 4 个待修（3 线上 + 1 休眠），全部 ≥2/3 对抗票（R4 仅 1/3 但论证扎实）。报告 = `plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md`（含每项修法）。均非"比修前更糟"（修前哪都不清），只是 D1 修得不完整。** - -**⭐ 下一步（下个 session 起步先修这 4 项，再开 Phase 2）：** -1. **R2（线上，先修，一箭三雕）** — iceberg/paimon 没 override `invalidateDb` → 我的 `DROP DATABASE` 钩子 + **老早就存在的 `REFRESH DATABASE`** 对它俩的快照/schema 缓存都是空操作。给 `IcebergConnector`/`PaimonConnector` 补 `invalidateDb`（按 db 前缀 removeIf 清 `latestSnapshotCache` + paimon `PaimonSchemaAtMemo`）。顺带修好 `DROP DATABASE FORCE` 级联 + hive 翻闸后 `forEachBuiltSibling(sibling.invalidateDb)`。 -2. **R1（线上）** — DROP/CREATE/DROPDB 失效**只在执行 DDL 的那台 FE 生效**；follower/observer 走 editlog replay（`replayDropTable/replayCreateTable/replayDropDb` 只清 FE 名字缓存）不传到 `connector.invalidate*`。仿现成的 `replayTruncateTable` override + `replayRefreshTable→refreshTableInternal` 补 replay 侧传播。 -3. **R4（线上，弱票但论证实）** — `RENAME TABLE`（可能还有 replace/CTAS-overwrite）也没清连接器缓存（`afterExternalRename` 零 `invalidate*`、不走 `refreshTableInternal`）→ 原子换名 t↔t2 读到老快照。同 D1 类，设计文档还引了 Trino rename 自失效做样板却漏了它。 -4. **R3（休眠，最后）** — `getPartitions` 直接 `put` 绕过了老代码有的 invalidateGeneration 守卫（REFRESH 撞正在飞的取数 → 重缓存陈旧值到 TTL）。给连接器版 `MetaCacheEntry` 加个 additive 的守卫式 put（`invalidationGeneration()` + `putIfNotInvalidatedSince`），RPC 前抓 generation、逐分区守卫式 put，结果仍直接从 delegate 收（保 misparse-never-drop）。 - - **修法/顺序/测试细节全在复核报告里**；R1/R2/R4 是线上路径，改完再跑一轮针对性对抗复核（memory `plugindriven-mvcc-table-is-live-not-dormant`）。 -5. **之后**：原子翻闸前置项（Phase 2 集：`SPI_READY += hms`、GSON 重映射、4 守卫改接、死臂/删除排序）——见 execution-plan §2/§3。 -6. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失（所有插件表通性、非本轮引入）。 - -**⚠ 翻闸/e2e 欠账（非静默，勿丢）**：所有连接器休眠步 + 本轮 3 修只在翻闸后 live（hive）；**paimon/iceberg 的 drop+recreate 修复现在就 live，可即刻 e2e 回归**（`DROP TABLE t; CREATE TABLE t(新schema/位置); SELECT` 无需 REFRESH 见新表；paimon 时间旅行复用 schemaId 见新 schema）。异构 HMS docker e2e 清单见 `hive-cache-invalidation-followups-design` §5 + execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`。删除排序最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 +**⭐ 下一步（下个 session）= Phase 2 原子翻闸前置项 + 现在就能做的 e2e 回归：** +1. **Phase 2 原子翻闸集**（权威 = `hms-cutover-execution-plan-2026-07-10.md` §2/§3）：`SPI_READY_TYPES += "hms"`、GSON 重映射、4 守卫改接新子系统、死臂/删除排序。**Phase 3 删除旧代码最后做**，最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 +2. **e2e 欠账（非静默，勿丢）**：**paimon/iceberg 的 drop+recreate + rename-swap 修复现在就 live，可即刻 e2e 回归**（`DROP TABLE t; CREATE TABLE t(新schema/位置); SELECT` 无需 REFRESH 见新表；`DROP DATABASE d; CREATE DATABASE d`；`RENAME` 原子换名；paimon 时间旅行复用 schemaId 见新 schema）。hive 侧休眠步翻闸后才 live。异构 HMS docker e2e 清单见 design §5 + execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`。 +3. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失（所有插件表通性、非本轮引入）。 **⚠ 并行 session 风险**：曾有两个 session 同工作树互相 amend；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手（本轮见到的 `gq` 用户 maven 是**别的工作树**，非本树）。 @@ -41,11 +34,13 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms # 🧠 起步必读（读文档，别炒 git log 历史） -1. **权威翻闸计划** = `tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。 -2. **本轮（缓存失效收尾）设计** = `tasks/hive-cache-invalidation-followups-design-2026-07-10.md`（D1 fe-core 钩子 + D2 每分区 keying + 正确性论证 + §6 状态 + e2e 欠账）。**起步先读，再对照 HEAD 核行号**。 -3. 已完成的前几项 Phase-1 步：`tasks/hive-connector-cache-step-design-2026-07-10.md`（D2 缓存本体）、`tasks/hive-event-pipeline-step-design-2026-07-10.md`（事件管道）、`tasks/hive-coupling-seams-step-design-2026-07-10.md`（4 耦合缝）。第二轮缓存复审报告 = `plan-doc/reviews/hive-connector-cache-cleanroom-review2-2026-07-10.md`（本轮 3 修的来源；注意 §2.1 已在 HEAD 修好、§2.3 说"Trino 用 weigher"经复核为**不准确**——Trino 也是每分区一条目按数量封顶）。 -4. **补充权威**：`tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + §6 D1–D6 决策）；`tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 + CCE/TCCL 硬约束）；hudi = `tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `tasks/hudi-schema-evolution-step-design-2026-07-09.md`。 -5. **样板**：`tasks/P5-paimon-migration.md`、`tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 +> **路径注意**：设计/计划文档都在 `plan-doc/tasks/` 下，复审报告在 `plan-doc/reviews/` 下。 + +1. **权威翻闸计划（下个 session 起步必读）** = `plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。**行号信 HEAD 不信文档**。 +2. **缓存失效收尾（本轮，已 DONE）设计** = `plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md`（§6 全勾）；复核报告 = `plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md`（CLEAN）+ 上一轮 `plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md`（R1–R4 来源）。 +3. 已完成的前几项 Phase-1 步：`plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md`（D2 缓存本体）、`plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md`（事件管道）、`plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md`（4 耦合缝）。 +4. **补充权威**：`plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + §6 D1–D6 决策）；`plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 + CCE/TCCL 硬约束）；hudi = `plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md`。 +5. **样板**：`plan-doc/tasks/P5-paimon-migration.md`、`plan-doc/tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 6. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 --- diff --git a/plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md b/plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md new file mode 100644 index 00000000000000..8f169e286b26ea --- /dev/null +++ b/plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md @@ -0,0 +1,57 @@ +# Cache-invalidation fixes (R1–R4) — targeted adversarial re-review (2026-07-10) + +> Re-review of the four fixes that closed the prior clean-room review's findings +> (`cache-invalidation-cleanroom-review-2026-07-10.md`). Commits `6df649d722d..HEAD`: +> `7b7b3c25953` R2, `a562c91e55b` R1, `43db3e8214f` R4, `d26bfa52eea` R3. +> Run `wf_b730a7d4-6a3`: **6 blind finders** (one per fix R1/R2/R3/R4 + cross-cutting completeness + +> regression/invariance), each finding to be adversarially verified by 3 independent lenses +> (correctness / does-it-reproduce / refute-hardest) and kept only at ≥2/3. +> +> **Verdict: CLEAN. 0 findings raised, 0 survived. The four fixes are correct AND complete — no +> follow-ups required.** (531k subagent tokens, 191 tool calls, 0 agent errors; each finder did +> 11–64 code-grounded tool calls before concluding.) Contrast the prior review, which found the +> *pre-fix* code incomplete; the fixes now fully close all four findings with nothing new introduced. + +## What each lens verified (all concluded "no genuine defect") + +- **R1 — replay propagation** (drop/create/dropDb). EditLog routing reaches the overrides + (`OP_DROP_TABLE → ctl.replayDropTable`, `OP_DROP_DB → Env.replayDropDb → externalCatalog.replayDropDb`, + `OP_CREATE_TABLE → externalCatalog.replayCreateTable`); coordinator and replay key identically; + `DropInfo` maps correctly; the drop resolves the remote name BEFORE `super` unregisters; the view + and table drop branches are both covered by the single replay override; `createDb`/`replayCreateDb` + intentionally not hooked (coordinator parity). +- **R2 — iceberg/paimon `invalidateDb`.** The db-scoped predicate matches exactly the keys + `beginQuerySnapshot` stores (iceberg `namespace().equals(Namespace.of(db))`, paimon + `getDatabaseName().equals(db)`); all three callers (`dropDb`, replay, `RefreshManager`) pass the + REMOTE db name — the same correlation the accepted `invalidateTable` uses; `PaimonConnector.invalidateDb` + clears BOTH `latestSnapshotCache` AND `PaimonSchemaAtMemo`; `DROP DATABASE FORCE`'s connector-internal + cascade is covered because the clear is db-scoped; iceberg's path-keyed manifest cache is intentionally + kept. +- **R3 — guarded put** (dormant). The guard mirrors `getWithManualLoad`'s pre-put + post-put + (`removeLoadedValue`) checks exactly; a newer concurrent write for the same key is never dropped; the + captured generation is shared safely across the multi-key loop (any flush bumps the shared counter); + line 189 is the ONLY raw `put` in `CachingHmsClient` (the other three reads use the guarded `get` + path); the framework additions are additive (no behavior change for iceberg/paimon, which never call + them). +- **R4 — RENAME** (coordinator + replay). An atomic swap (`RENAME t→t2; RENAME t3→t`) leaves no stale + pin — each rename invalidates the source's old remote name (the live read-pin key) plus the target's + new name, and the vacated local name is unregistered so nothing re-pins it; invalidate-after-mutate is + safe (a failed rename aborts before the cache is touched); coordinator↔replay keys match; replay never + force-inits. +- **Completeness (cross-cutting).** No unfixed path of the same class remains on the live surface: + maxcompute/jdbc/es/trino are SPI-ready but hold NO connector-owned metadata cache; **hudi** (reached by + the hive gateway's `forEachBuiltSibling`) builds a raw `ThriftHmsClient` and reads fresh, so + `HudiConnector` correctly needs no `invalidateDb`; every replay hook resolves through + `getDbForReplay`/`getTableForReplay` (empty when uninitialized), so no follower force-init. +- **Regression / invariance (live paimon/iceberg).** No recursion via `forEachBuiltSibling` (siblings + clear their own caches, don't re-fan-out; standalone paimon/iceberg don't fan out at all); no NPE / + deadlock; no cost added to the query hot path (invalidate\* is DDL-only); coordinator↔replay key parity + holds; the deliberate "target/create name not remote-resolved" is consistent on both sides (not a new + divergence); `invalidateDb` stays a no-op SPI default (fe-core connector-agnostic, non-caching + connectors safe). + +## Disposition +The connector-cache invalidation follow-up (D1/D2 → R1–R4) is **DONE and validated**. Remaining owed +work is the heterogeneous-HMS docker **e2e** (§5 of the design doc — paimon/iceberg drop+recreate and +rename-swap are testable NOW as live regressions), and the next phase is the atomic HMS cutover (Phase 2 +of `hms-cutover-execution-plan-2026-07-10.md`). diff --git a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md index ab5c72b82bdf2a..45ccc07b50bbe5 100644 --- a/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md +++ b/plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md @@ -224,12 +224,27 @@ paimon/iceberg — per practice + memory `plugindriven-mvcc-table-is-live-not-do - [x] **Unified clean-room adversarial re-review** of F1/F1b (live paths) — `wf_fe6ddef4-777`, DONE. **Verdict: fixes are NET IMPROVEMENTS but INCOMPLETE — 4 follow-ups required (3 live + 1 dormant). See `plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md`.** -- [ ] **R1** DROP/CREATE/DROPDB invalidation is coordinator-only — followers/observers don't propagate on - replay (mirror `replayTruncateTable`). LIVE. -- [ ] **R2** iceberg/paimon lack an `invalidateDb` override → DROP DATABASE hook + pre-existing REFRESH - DATABASE are no-ops for their snapshot/schema caches. LIVE (fixes a pre-existing bug too). -- [ ] **R3** `getPartitions` raw `put` bypasses the invalidateGeneration guard (REFRESH-vs-in-flight race). - DORMANT; needs an additive guarded-put on the connector `MetaCacheEntry`. -- [ ] **R4** RENAME (and maybe replace/CTAS-overwrite) never invalidates the connector cache. LIVE. -- [ ] **e2e** (§5) — owed at the flip / heterogeneous-HMS docker (paimon/iceberg drop+recreate is testable - NOW as a live regression, once R1/R2/R4 land). +- [x] **R2** iceberg/paimon db-scoped `invalidateDb` — `7b7b3c25953`. Adds `IcebergConnector`/`PaimonConnector` + `invalidateDb` (+ db-scoped removeIf on the snapshot caches + `PaimonSchemaAtMemo`), so the DROP DATABASE + hook AND the pre-existing REFRESH DATABASE both take effect (incl. the FORCE table cascade), and arms + hive's `forEachBuiltSibling(sibling.invalidateDb)`. LIVE. iceberg 16 + paimon 17 pass, checkstyle 0. +- [x] **R1** DROP/CREATE/DROPDB invalidation propagates on editlog replay — `a562c91e55b`. Overrides + `replayDropTable`/`replayCreateTable`/`replayDropDb` in `PluginDrivenExternalCatalog` (mirrors the existing + `replayTruncateTable`), keyed like the coordinator; never force-inits (gated by getDbForReplay). LIVE. + `PluginDrivenExternalCatalogDdlRoutingTest` 60 pass, checkstyle 0. +- [x] **R4** RENAME invalidates the connector cache (coordinator + replay) — `43db3e8214f`. + `PluginDrivenExternalCatalog.renameTable` drops source+target after the mutation; + `RefreshManager.replayRefreshTable` rename branch propagates to followers. No `replaceTable`/CTAS-overwrite + path exists for plugin catalogs (verified) → RENAME was the only remaining gap. LIVE. DDL routing 60 + + new `RefreshManagerRenameReplayTest` 2 pass, checkstyle 0. +- [x] **R3** `getPartitions` generation-guarded put — `d26bfa52eea`. Additive + `MetaCacheEntry.invalidationGeneration()`/`putIfNotInvalidatedSince()` (connector copy only) + wired into + `CachingHmsClient.getPartitions` (capture generation before the RPC). DORMANT. `MetaCacheEntryTest` 7 + + `CachingHmsClientTest` 16 pass, checkstyle 0. +- [x] **Targeted adversarial re-review** of R1/R2/R4 (live paths) + R3 — `wf_b730a7d4-6a3` (6 blind finders + + 3-lens verify). **Verdict: CLEAN — 0 findings; the four fixes are correct AND complete, no follow-ups.** + Report = `plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md` (confirms hudi correctly + needs no `invalidateDb` — raw ThriftHmsClient, reads fresh; no other same-class path unfixed; replay + never force-inits). The whole D1/D2 → R1–R4 follow-up is DONE. +- [ ] **e2e** (§5) — owed at the flip / heterogeneous-HMS docker (paimon/iceberg drop+recreate AND rename-swap + are testable NOW as live regressions, since R1/R2/R4 are landed). From b8c5140231f6c46371a2a82d29acaab61a5b40b2 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 21:20:44 +0800 Subject: [PATCH 198/334] [fix](catalog) Connector event sync: follower cursor feed routes to the Model-B driver for flipped catalogs MetaIdMappingsLog replay (the follower's master-committed event-id cursor feed) unconditionally fed the legacy MetastoreEventsProcessor. Post-flip a hive catalog is a PluginDrivenExternalCatalog driven by the new MetastoreEventSyncDriver, whose masterLastSyncedEventIdMap would then never populate -> masterUpperBound stays -1 -> HmsEventSource.pollForFollower returns nothing -> followers silently stop applying incremental metastore changes. Route by catalog kind: PluginDrivenExternalCatalog -> new driver, else the legacy processor (both key by catalogId only, no HMSExternalCatalog cast). Dormant pre-flip (no hms-typed PluginDriven catalog yet). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/datasource/ExternalMetaIdMgr.java | 21 ++++++++++++------- 1 file changed, 13 insertions(+), 8 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java index 5de639566fe701..4b81c05947f539 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalMetaIdMgr.java @@ -18,7 +18,6 @@ package org.apache.doris.datasource; import org.apache.doris.catalog.Env; -import org.apache.doris.datasource.hive.event.MetastoreEventsProcessor; import com.google.common.base.Preconditions; import com.google.common.collect.Maps; @@ -114,14 +113,20 @@ public void replayMetaIdMappingsLog(@NotNull MetaIdMappingsLog log) { handleMetaIdMapping(mapping, ctlMetaIdMgr); } if (log.isFromHmsEvent()) { - // Propagate the master's synced-event-id cursor to this FE, keyed by catalogId only - // (the log already carries it). Never cast the live catalog to HMSExternalCatalog: - // once an HMS catalog is served by a generic PluginDrivenExternalCatalog, that cast - // would throw ClassCastException and abort edit-log replay, wedging FE startup. + // Propagate the master's synced-event-id cursor to this FE, keyed by catalogId only (the log + // already carries it). Route to the driver that actually owns this catalog's event sync: a flipped + // hms catalog is a generic PluginDrivenExternalCatalog driven by MetastoreEventSyncDriver, whose + // follower cursor map must be fed here (otherwise its masterUpperBound stays -1 and followers stop + // receiving incremental updates); a not-yet-flipped legacy HMS catalog is still driven by the legacy + // MetastoreEventsProcessor. Both branches key by catalogId only — never cast to HMSExternalCatalog + // (that cast would ClassCastException for a PluginDrivenExternalCatalog and abort replay). CatalogIf catalogIf = Env.getCurrentEnv().getCatalogMgr().getCatalog(catalogId); - if (catalogIf != null) { - MetastoreEventsProcessor metastoreEventsProcessor = Env.getCurrentEnv().getMetastoreEventsProcessor(); - metastoreEventsProcessor.updateMasterLastSyncedEventId(catalogId, log.getLastSyncedEventId()); + if (catalogIf instanceof PluginDrivenExternalCatalog) { + Env.getCurrentEnv().getMetastoreEventSyncDriver() + .updateMasterLastSyncedEventId(catalogId, log.getLastSyncedEventId()); + } else if (catalogIf != null) { + Env.getCurrentEnv().getMetastoreEventsProcessor() + .updateMasterLastSyncedEventId(catalogId, log.getLastSyncedEventId()); } } } From 56991fbf6b05c983b6a8aeb859b760ded3fba250 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 21:20:44 +0800 Subject: [PATCH 199/334] [fix](catalog) Connector event sync: master/follower force-init hook for flipped event-source catalogs MetastoreEventSyncDriver.realRun skipped un-initialized catalogs (to keep idle paimon/iceberg/jdbc byte-inert), so a flipped hms catalog never queried on an FE never seeds its cursor -> never event-synced. Mirror the legacy poller's per-cycle force-init, but only for the event-source type ("hms"), gated on the pre-init getType() (reads catalogProperty, no makeSureInitialized) so idle non-event PluginDriven catalogs stay inert. Runs on every FE (no isMaster gate: each FE needs the catalog initialized to obtain its event source / seed its own cursor / forward REFRESH CATALOG), one-shot per catalog via the !isInitialized guard, swallow-on-throw like the legacy path. Dormant pre-flip (no hms-typed PluginDriven catalog matches the gate). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../datasource/MetastoreEventSyncDriver.java | 29 ++++++++++++++----- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java index 8729f144a2fc4b..feb738a5d2f563 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/MetastoreEventSyncDriver.java @@ -101,15 +101,28 @@ private void realRun() { continue; } PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; - // Probe only already-initialized catalogs: calling getConnector() force-initializes, and we must - // NOT force-init the idle paimon/iceberg/jdbc/hudi PluginDriven catalogs that already exist (they - // stay byte-inert pre-flip). LIMITATION vs the legacy poller (which force-initialized every HMS - // catalog on the master via getHmsProperties): a flipped HMS catalog that is NEVER accessed on the - // master but IS queried on a follower never seeds its cursor, so that follower stops receiving - // incremental updates. Closing this without force-initing non-event catalogs needs a flip-time - // hook that initializes flipped event-source catalogs on the master (owed at the flip). if (!pluginCatalog.isInitialized()) { - continue; + // Flip-time force-init parity: the legacy MetastoreEventsProcessor force-initialized EVERY hms + // catalog every cycle on every FE (via getHmsProperties() -> makeSureInitialized()), so a flipped + // hms catalog seeds its cursor even if it is never queried on this FE. This is required on + // followers too (each FE runs its own driver with its own cursor, and a follower must have the + // catalog initialized to obtain its event source, seed its cursor and forward REFRESH CATALOG) — + // hence no isMaster gate. Mirror that ONLY for the event-source type ("hms", the sole connector + // exposing a ConnectorEventSource), keyed on the pre-init type string so idle paimon/iceberg/ + // jdbc/hudi PluginDriven catalogs stay byte-inert: getType() reads catalogProperty and does NOT + // force-init. Guarded by !isInitialized(), so it is a one-shot per catalog (subsequent cycles + // take the initialized fast path). Pre-flip there is no hms-typed PluginDriven catalog, so this + // block matches nothing and the driver stays dormant. + if (!"hms".equalsIgnoreCase(pluginCatalog.getType())) { + continue; + } + try { + pluginCatalog.makeSureInitialized(); + } catch (Exception e) { + // Missing/invalid params this cycle -> skip (mirrors the legacy skip-on-throw around + // getHmsProperties()); retried next cycle, the error is already surfaced via SHOW CATALOGS. + continue; + } } ConnectorEventSource eventSource; try { From b25b15c357fdfcf3e55d072d64ef0c13022c10b2 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 21:58:09 +0800 Subject: [PATCH 200/334] [feat](catalog) HMS SPI cutover: flip type=hms to the plugin-driven path (SPI_READY + GSON remap + buildDbForInit) The atomic flip that routes catalog type "hms" (plain-hive + iceberg-on-HMS + hudi-on-HMS) through the plugin connector SPI, so all production hms flows use the new code. Mirrors the paimon/iceberg flips. No legacy code deleted (that is the last phase); post-flip the legacy HMSExternal* classes still exist (dead-for-hms) so their consumers keep compiling. - CatalogFactory: SPI_READY_TYPES += "hms"; delete the now-dead `case "hms" -> new HMSExternalCatalog` fallback and its import (a fresh hms catalog is always handled by the SPI branch). - GsonUtils: three registerSubtype -> registerCompatibleSubtype remaps so old images deserialize as the generic PluginDriven classes: catalog -> PluginDrivenExternalCatalog, db -> PluginDrivenExternalDatabase, table -> PluginDrivenMvccExternalTable (the hive connector declares SUPPORTS_MVCC_SNAPSHOT, matching paimon/iceberg; the deserialized object is rebuilt via buildTableInternal on replay). Locked with SPI_READY_TYPES (label uniqueness + write-registration), so both ship in one commit. - ExternalCatalog.buildDbForInit: HMS arm -> new PluginDrivenExternalDatabase, mirroring the ICEBERG arm (replay-defense for an old InitCatalogLog persisted with Type.HMS). - HmsGsonCompatReplayTest (new): pins that legacy HMSExternal{Catalog,Database,Table} tags replay as the PluginDriven classes (table -> the MVCC variant, exact-class assert). Mirrors Iceberg/PaimonGsonCompatReplayTest. - Disable HmsCatalogTest / HmsQueryCacheTest / HiveDDLAndDMLPlanTest: they build a type=hms catalog via the routed path (no hms provider on the fe-core test classpath -> throws) and assert legacy HMSExternal* behavior, which is dead-for-hms post-flip. @Disabled with a pointer; removed with the legacy subsystem in the deletion phase. The event-sync activation this flip depends on landed in the two preceding commits (follower cursor feed + master force-init). The 4 legacy instanceof gates are deliberately LEFT (they self-exclude a PluginDriven catalog and stay correct for any un-flipped legacy HMS catalog); the connector cache path auto-engages. Verified: mvn -pl fe-core -am test-compile BUILD SUCCESS + 0 Checkstyle; HmsGsonCompatReplayTest 3/3, Iceberg/PaimonGsonCompatReplayTest 3/3 each, ExternalMetaCacheRouteResolverTest 7/7, HmsCatalogTest skipped. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../doris/datasource/CatalogFactory.java | 18 ++- .../doris/datasource/ExternalCatalog.java | 10 +- .../apache/doris/persist/gson/GsonUtils.java | 23 ++-- .../datasource/HmsGsonCompatReplayTest.java | 110 ++++++++++++++++++ .../hive/HiveDDLAndDMLPlanTest.java | 6 + .../doris/external/hms/HmsCatalogTest.java | 6 + .../apache/doris/qe/HmsQueryCacheTest.java | 6 + 7 files changed, 158 insertions(+), 21 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/HmsGsonCompatReplayTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java index 4a2a3dcf2766d5..a02609945b1ce1 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/CatalogFactory.java @@ -25,7 +25,6 @@ import org.apache.doris.connector.DefaultConnectorContext; import org.apache.doris.connector.api.Connector; import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; -import org.apache.doris.datasource.hive.HMSExternalCatalog; import org.apache.doris.datasource.test.TestExternalCatalog; import org.apache.doris.nereids.trees.plans.commands.CreateCatalogCommand; @@ -44,16 +43,16 @@ public class CatalogFactory { private static final Logger LOG = LogManager.getLogger(CatalogFactory.class); // Only these catalog types are routed through the SPI connector path; every other type falls through to the - // built-in ExternalCatalog switch below. "hms" is deliberately absent: a hive catalog still uses the built-in - // HMSExternalCatalog until the single-line HMS cutover adds "hms" here (which flips hive + iceberg-on-HMS + - // hudi-on-HMS to the SPI path atomically). + // built-in ExternalCatalog switch below. "hms" is now SPI-ready: an hms catalog is a PluginDrivenExternalCatalog + // driven by the hive connector, which flips plain-hive + iceberg-on-HMS + hudi-on-HMS to the SPI path (the + // latter two served as embedded siblings of the hms gateway). // Do NOT add "hudi": there is no standalone hudi catalog type and no HudiExternalCatalog — a hudi table is // parasitic on an HMS catalog (HMSExternalTable, dlaType==HUDI) and is served post-cutover as an embedded // SIBLING of the hms gateway via ConnectorContext.createSiblingConnector("hudi", ...), which bypasses this // set. Adding "hudi" here would build a standalone PluginDrivenExternalCatalog around HudiConnector with no // fe-core catalog class backing it. private static final Set SPI_READY_TYPES = - ImmutableSet.of("jdbc", "es", "trino-connector", "max_compute", "paimon", "iceberg"); + ImmutableSet.of("jdbc", "es", "trino-connector", "max_compute", "paimon", "iceberg", "hms"); /** * create the catalog instance from catalog log. @@ -136,12 +135,9 @@ private static CatalogIf createCatalog(long catalogId, String name, String resou // Fallback to built-in catalog types if no SPI connector matched. if (catalog == null) { switch (catalogType) { - case "hms": - catalog = new HMSExternalCatalog(catalogId, name, resource, props, comment); - break; - // iceberg is routed through the SPI connector path (SPI_READY_TYPES) and never reaches this - // built-in fallback; its legacy IcebergExternalCatalogFactory case was removed at the GSON - // cutover (its GSON subtype is now registerCompatibleSubtype-only -> PluginDriven). + // hms and iceberg are routed through the SPI connector path (SPI_READY_TYPES) and never reach + // this built-in fallback; their legacy built-in cases were removed at the GSON cutover (their + // GSON subtypes are now registerCompatibleSubtype-only -> PluginDriven). case "lakesoul": throw new DdlException("Lakesoul catalog is no longer supported"); case "doris": diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java index bc68f947e2b644..81402a32b81091 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/ExternalCatalog.java @@ -41,7 +41,6 @@ import org.apache.doris.common.util.Util; import org.apache.doris.datasource.connectivity.CatalogConnectivityTestCoordinator; import org.apache.doris.datasource.doris.RemoteDorisExternalDatabase; -import org.apache.doris.datasource.hive.HMSExternalDatabase; import org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase; import org.apache.doris.datasource.infoschema.ExternalMysqlDatabase; import org.apache.doris.datasource.lakesoul.LakeSoulExternalDatabase; @@ -1066,7 +1065,14 @@ protected ExternalDatabase buildDbForInit(String remote } switch (logType) { case HMS: - return new HMSExternalDatabase(this, dbId, localDbName, remoteDbName); + // Hive (hms) is flipped to the plugin path (PluginDrivenExternalCatalog); the HMSExternalDatabase + // entity class is dead-for-hms (deleted with the legacy subsystem in the deletion phase). This + // case only fires when replaying an old InitCatalogLog persisted with Type.HMS (a base + // ExternalCatalog whose logType was serialized as HMS) — build the post-flip runtime type so the + // db (and the PluginDrivenMvccExternalTable it builds) matches the GSON remap. Keep the case + // label: deleting it would fall through to `return null` and break db init on replay. The + // Type.HMS enum is retained for old-image deserialization. + return new PluginDrivenExternalDatabase(this, dbId, localDbName, remoteDbName); case JDBC: return new PluginDrivenExternalDatabase(this, dbId, localDbName, remoteDbName); case ICEBERG: diff --git a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java index 93e8dd25ef1223..c5cbb67c146ce7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java +++ b/fe/fe-core/src/main/java/org/apache/doris/persist/gson/GsonUtils.java @@ -147,9 +147,6 @@ import org.apache.doris.datasource.PluginDrivenExternalTable; import org.apache.doris.datasource.PluginDrivenMvccExternalTable; import org.apache.doris.datasource.doris.RemoteDorisExternalCatalog; -import org.apache.doris.datasource.hive.HMSExternalCatalog; -import org.apache.doris.datasource.hive.HMSExternalDatabase; -import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.infoschema.ExternalInfoSchemaDatabase; import org.apache.doris.datasource.infoschema.ExternalInfoSchemaTable; import org.apache.doris.datasource.infoschema.ExternalMysqlDatabase; @@ -363,7 +360,6 @@ public class GsonUtils { static { dsTypeAdapterFactory = RuntimeTypeAdapterFactory.of(CatalogIf.class, "clazz") .registerSubtype(CloudInternalCatalog.class, CloudInternalCatalog.class.getSimpleName()) - .registerSubtype(HMSExternalCatalog.class, HMSExternalCatalog.class.getSimpleName()) .registerSubtype(LakeSoulExternalCatalog.class, LakeSoulExternalCatalog.class.getSimpleName()) .registerSubtype(TestExternalCatalog.class, TestExternalCatalog.class.getSimpleName()) .registerSubtype(RemoteDorisExternalCatalog.class, RemoteDorisExternalCatalog.class.getSimpleName()) @@ -408,7 +404,11 @@ public class GsonUtils { .registerCompatibleSubtype( PluginDrivenExternalCatalog.class, "IcebergJdbcExternalCatalog") .registerCompatibleSubtype( - PluginDrivenExternalCatalog.class, "IcebergS3TablesExternalCatalog"); + PluginDrivenExternalCatalog.class, "IcebergS3TablesExternalCatalog") + // Migrate old HMS (hive) catalogs to PluginDriven on deserialization; the hms gateway serves + // plain-hive + hudi-on-HMS + iceberg-on-HMS through the hive connector. + .registerCompatibleSubtype( + PluginDrivenExternalCatalog.class, "HMSExternalCatalog"); if (Config.isNotCloudMode()) { dsTypeAdapterFactory .registerSubtype(InternalCatalog.class, InternalCatalog.class.getSimpleName()); @@ -444,7 +444,6 @@ public class GsonUtils { private static RuntimeTypeAdapterFactory dbTypeAdapterFactory = RuntimeTypeAdapterFactory.of( DatabaseIf.class, "clazz") .registerSubtype(ExternalDatabase.class, ExternalDatabase.class.getSimpleName()) - .registerSubtype(HMSExternalDatabase.class, HMSExternalDatabase.class.getSimpleName()) .registerSubtype(LakeSoulExternalDatabase.class, LakeSoulExternalDatabase.class.getSimpleName()) .registerSubtype(ExternalInfoSchemaDatabase.class, ExternalInfoSchemaDatabase.class.getSimpleName()) .registerSubtype(ExternalMysqlDatabase.class, ExternalMysqlDatabase.class.getSimpleName()) @@ -463,12 +462,14 @@ public class GsonUtils { PluginDrivenExternalDatabase.class, "PaimonExternalDatabase") // Migrate old Iceberg databases to PluginDriven on deserialization .registerCompatibleSubtype( - PluginDrivenExternalDatabase.class, "IcebergExternalDatabase"); + PluginDrivenExternalDatabase.class, "IcebergExternalDatabase") + // Migrate old HMS (hive) databases to PluginDriven on deserialization + .registerCompatibleSubtype( + PluginDrivenExternalDatabase.class, "HMSExternalDatabase"); private static RuntimeTypeAdapterFactory tblTypeAdapterFactory = RuntimeTypeAdapterFactory.of( TableIf.class, "clazz").registerSubtype(ExternalTable.class, ExternalTable.class.getSimpleName()) .registerSubtype(OlapTable.class, OlapTable.class.getSimpleName()) - .registerSubtype(HMSExternalTable.class, HMSExternalTable.class.getSimpleName()) .registerSubtype(LakeSoulExternalTable.class, LakeSoulExternalTable.class.getSimpleName()) .registerSubtype(ExternalInfoSchemaTable.class, ExternalInfoSchemaTable.class.getSimpleName()) .registerSubtype(ExternalMysqlTable.class, ExternalMysqlTable.class.getSimpleName()) @@ -492,6 +493,12 @@ public class GsonUtils { // declaring SUPPORTS_MVCC_SNAPSHOT) so a flipped iceberg table is a PluginDrivenMvccExternalTable .registerCompatibleSubtype( PluginDrivenMvccExternalTable.class, "IcebergExternalTable") + // HMS (hive) tables migrate to the MVCC variant too: the hive connector declares + // SUPPORTS_MVCC_SNAPSHOT (snapshots/time-travel/MTMV freshness, and the gateway serves the + // MVCC-capable iceberg-on-HMS + hudi-on-HMS siblings), so a flipped hms table is a + // PluginDrivenMvccExternalTable — matching what buildTableInternal rebuilds it as on replay. + .registerCompatibleSubtype( + PluginDrivenMvccExternalTable.class, "HMSExternalTable") .registerSubtype(BrokerTable.class, BrokerTable.class.getSimpleName()) .registerSubtype(EsTable.class, EsTable.class.getSimpleName()) .registerSubtype(FunctionGenTable.class, FunctionGenTable.class.getSimpleName()) diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/HmsGsonCompatReplayTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/HmsGsonCompatReplayTest.java new file mode 100644 index 00000000000000..43a81834d6c558 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/HmsGsonCompatReplayTest.java @@ -0,0 +1,110 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.TableIf; +import org.apache.doris.persist.gson.GsonUtils; + +import com.google.common.collect.Maps; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Map; + +/** + * Guards the hms (hive) SPI cutover edit-log compatibility (mirrors {@link IcebergGsonCompatReplayTest} / + * {@link PaimonGsonCompatReplayTest}): an FE image / edit log written by a pre-cutover version persisted hms + * catalogs/databases/tables under their legacy class simple names (the GSON {@code "clazz"} discriminator). + * After the cutover those legacy classes are no longer {@code registerSubtype}'d, so on replay the + * {@code registerCompatibleSubtype} mappings in {@link GsonUtils} MUST redirect every legacy tag to the + * generic PluginDriven class — otherwise the FE crashes on startup with a {@code JsonParseException} (tag not + * registered) or a downstream {@code ClassCastException}. + * + *

    Why this matters / what would break it: the three GSON registries (catalog, db, table) must + * migrate atomically. Unlike iceberg/paimon (which have several catalog flavors), hms has a single catalog + * class {@code HMSExternalCatalog} — but the table tag is the load-bearing one: {@code HMSExternalTable} is + * the gateway class for plain-hive AND iceberg-on-HMS AND hudi-on-HMS, and the hive connector declares + * {@code SUPPORTS_MVCC_SNAPSHOT} (snapshots/time-travel/MTMV freshness), so a flipped hms table is a + * {@code PluginDrivenMvccExternalTable}. Therefore {@code HMSExternalTable} MUST replay as the MVCC variant, + * not the base {@code PluginDrivenExternalTable} — replaying as the base would silently downgrade a persisted + * hms table and lose the MVCC behavior on every FE restart.

    + * + *

    Each case round-trips a valid PluginDriven object through GSON, rewrites only the {@code "clazz"} + * discriminator to the legacy tag (faithfully reproducing old-image bytes without depending on the legacy + * HMSExternal* classes), then deserializes and asserts the resolved runtime class.

    + */ +public class HmsGsonCompatReplayTest { + + private static String swapClazz(String json, String currentTag, String legacyTag) { + String needle = "\"clazz\":\"" + currentTag + "\""; + // Sanity: the polymorphic serialization must emit the discriminator we are about to rewrite. + Assertions.assertTrue(json.contains(needle), + "expected discriminator " + needle + " in serialized json: " + json); + return json.replace(needle, "\"clazz\":\"" + legacyTag + "\""); + } + + @Test + public void testLegacyHmsCatalogTagReplaysAsPluginDriven() { + Map props = Maps.newHashMap(); + props.put("type", "hms"); + // 6-arg ctor sets logType=PLUGIN and a non-null catalogProperty, so gsonPostProcess replays cleanly. + PluginDrivenExternalCatalog catalog = + new PluginDrivenExternalCatalog(1L, "hms_ctl", "", props, "c", null); + String baseJson = GsonUtils.GSON.toJson(catalog, CatalogIf.class); + + String json = swapClazz(baseJson, "PluginDrivenExternalCatalog", "HMSExternalCatalog"); + // MUTATION: removing the registerCompatibleSubtype for HMSExternalCatalog throws + // "cannot deserialize ... subtype named HMSExternalCatalog" here; a wrong target class fails instanceof. + CatalogIf restored = GsonUtils.GSON.fromJson(json, CatalogIf.class); + Assertions.assertTrue(restored instanceof PluginDrivenExternalCatalog, + "legacy edit-log tag 'HMSExternalCatalog' must replay as PluginDrivenExternalCatalog " + + "(no crash/ClassCastException)"); + } + + @Test + public void testLegacyHmsDatabaseTagReplaysAsPluginDriven() { + PluginDrivenExternalDatabase db = new PluginDrivenExternalDatabase(); + db.id = 2L; + db.name = "hms_db"; + String baseJson = GsonUtils.GSON.toJson(db, DatabaseIf.class); + + String json = swapClazz(baseJson, "PluginDrivenExternalDatabase", "HMSExternalDatabase"); + // MUTATION: dropping the db registerCompatibleSubtype makes this throw on deserialize. + DatabaseIf restored = GsonUtils.GSON.fromJson(json, DatabaseIf.class); + Assertions.assertTrue(restored instanceof PluginDrivenExternalDatabase, + "legacy 'HMSExternalDatabase' tag must replay as PluginDrivenExternalDatabase"); + } + + @Test + public void testLegacyHmsTableTagReplaysAsMvccPluginDriven() { + PluginDrivenMvccExternalTable table = new PluginDrivenMvccExternalTable(); + table.id = 3L; + table.name = "hms_tbl"; + table.dbName = "hms_db"; + String baseJson = GsonUtils.GSON.toJson(table, TableIf.class); + + String json = swapClazz(baseJson, "PluginDrivenMvccExternalTable", "HMSExternalTable"); + TableIf restored = GsonUtils.GSON.fromJson(json, TableIf.class); + // hms tables must replay as the MVCC variant. instanceof would also pass for a subclass, so assert the + // EXACT class to catch a mistaken mapping to the base PluginDrivenExternalTable. + Assertions.assertSame(PluginDrivenMvccExternalTable.class, restored.getClass(), + "legacy 'HMSExternalTable' tag must replay as PluginDrivenMvccExternalTable (the hive connector" + + " is MVCC-capable), not the base PluginDrivenExternalTable"); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveDDLAndDMLPlanTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveDDLAndDMLPlanTest.java index 5ff126ac30e372..2797b0e2d2b966 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveDDLAndDMLPlanTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HiveDDLAndDMLPlanTest.java @@ -58,6 +58,7 @@ import org.apache.hadoop.hive.metastore.api.FieldSchema; import org.apache.hadoop.hive.metastore.api.Table; import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import org.mockito.MockedConstruction; import org.mockito.Mockito; @@ -70,6 +71,11 @@ import java.util.Optional; import java.util.Set; +// Disabled at the hms SPI cutover: `CREATE CATALOG ... type=hms` now routes through the plugin SPI, and the +// fe-core test classpath has no hms connector provider, so the catalog setup here throws. This test drives the +// legacy hive DDL/DML planning path (HMSExternalCatalog + HiveMetadataOps), which is dead-for-hms in production +// post-flip. The legacy subsystem and this test are removed together in the Phase-3 deletion. +@Disabled("Legacy hive path retired at the hms SPI cutover; removed with the legacy subsystem in Phase 3") public class HiveDDLAndDMLPlanTest extends TestWithFeService { private static final String mockedCtlName = "hive"; private static final String mockedDbName = "mockedDb"; diff --git a/fe/fe-core/src/test/java/org/apache/doris/external/hms/HmsCatalogTest.java b/fe/fe-core/src/test/java/org/apache/doris/external/hms/HmsCatalogTest.java index 744933d5ee498e..359d944513f0b0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/external/hms/HmsCatalogTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/external/hms/HmsCatalogTest.java @@ -43,12 +43,18 @@ import com.google.common.collect.Maps; import org.junit.Assert; import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import org.mockito.Mockito; import java.util.List; import java.util.Optional; +// Disabled at the hms SPI cutover: `CREATE CATALOG ... type=hms` now routes through the plugin SPI, and the +// fe-core test classpath has no hms connector provider, so the catalog setup here throws. This test asserts the +// legacy HMSExternalCatalog/HMSExternalTable behavior (hive views/query cache), which is dead-for-hms in +// production post-flip. The legacy subsystem and this test are removed together in the Phase-3 deletion. +@Disabled("Legacy hive path retired at the hms SPI cutover; removed with the legacy subsystem in Phase 3") public class HmsCatalogTest extends AnalyzeCheckTestBase { private static final String HMS_CATALOG = "hms_ctl"; private static final long NOW = System.currentTimeMillis(); diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/HmsQueryCacheTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/HmsQueryCacheTest.java index 9c3274b00e8d29..fc7d848ad00dfe 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/qe/HmsQueryCacheTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/qe/HmsQueryCacheTest.java @@ -50,6 +50,7 @@ import com.google.common.collect.Lists; import com.google.common.collect.Maps; import org.junit.Assert; +import org.junit.jupiter.api.Disabled; import org.junit.jupiter.api.Test; import org.mockito.Mockito; @@ -57,6 +58,11 @@ import java.util.List; import java.util.Optional; +// Disabled at the hms SPI cutover: `CREATE CATALOG ... type=hms` now routes through the plugin SPI, and the +// fe-core test classpath has no hms connector provider, so the catalog setup here throws. This test asserts the +// legacy HMSExternalTable query-cache behavior, which is dead-for-hms in production post-flip. The legacy +// subsystem and this test are removed together in the Phase-3 deletion. +@Disabled("Legacy hive path retired at the hms SPI cutover; removed with the legacy subsystem in Phase 3") public class HmsQueryCacheTest extends AnalyzeCheckTestBase { private static final String HMS_CATALOG = "hms_ctl"; private static final long NOW = System.currentTimeMillis(); From 8ab2a6a166ed40bdf501adbd395c3a05c6d7648b Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 21:58:31 +0800 Subject: [PATCH 201/334] [feat](catalog) HMS cutover: serve external views unconditionally + de-iceberg-word the view throw (D5) Post-flip a hive view is PLUGIN_EXTERNAL_TABLE and hits the shared plugin-view arm in BindRelation (previously iceberg-only). Make that arm serve any SUPPORTS_VIEW connector (hive + iceberg) without a config gate, and drop the "iceberg" wording so a hive view's messages are not iceberg-labelled: - BindRelation: remove the enable_query_iceberg_views gate (views served unconditionally); neutralize the time-travel throw message "iceberg view not supported ..." -> "view not supported with snapshot time/version travel" (a view + AS OF is meaningless for hive and iceberg alike). Update the arm comment. - Config: deprecate enable_query_hive_views / enable_query_iceberg_views to retained no-ops (@Deprecated, fields kept one release so operator fe.conf still parses). - test_iceberg_view_query_p0.groovy: update the 16 message assertions to the neutralized wording. This is a visible behavior change to already-flipped iceberg (a view is now served regardless of the config, and the throw message drops "iceberg"), so it ships with the hms flip rather than leaking early. The legacy HMS_EXTERNAL_TABLE view arm (still reading enable_query_hive_views) is dead-for-hms post-flip and is removed in the deletion phase. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- .../java/org/apache/doris/common/Config.java | 10 ++++-- .../nereids/rules/analysis/BindRelation.java | 22 ++++++------- .../iceberg/test_iceberg_view_query_p0.groovy | 32 +++++++++---------- 3 files changed, 33 insertions(+), 31 deletions(-) diff --git a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java index 00526509177152..689ed4ea56c2ff 100644 --- a/fe/fe-common/src/main/java/org/apache/doris/common/Config.java +++ b/fe/fe-common/src/main/java/org/apache/doris/common/Config.java @@ -2252,12 +2252,18 @@ public class Config extends ConfigBase { public static boolean enable_fqdn_mode = false; /** - * If set to true, doris will try to parse the ddl of a hive view and try to execute the query - * otherwise it will throw an AnalysisException. + * @deprecated No-op since the hms/iceberg SPI cutover: external views (hive, iceberg) are always served. + * Retained for one release so operator fe.conf that sets it still parses; will be removed later. */ + @Deprecated @ConfField(mutable = true) public static boolean enable_query_hive_views = true; + /** + * @deprecated No-op since the iceberg SPI cutover: external views (hive, iceberg) are always served. + * Retained for one release so operator fe.conf that sets it still parses; will be removed later. + */ + @Deprecated @ConfField(mutable = true) public static boolean enable_query_iceberg_views = true; diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java index c35b96265dd450..e31dc2ae72de22 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java @@ -785,25 +785,21 @@ private LogicalPlan getLogicalPlan(TableIf table, UnboundRelation unboundRelatio case PLUGIN_EXTERNAL_TABLE: if (table instanceof PluginDrivenExternalTable && ((PluginDrivenExternalTable) table).isView()) { - // Flipped iceberg view: reproduce the legacy ICEBERG_EXTERNAL_TABLE view arm on the - // neutral plugin path (only iceberg declares SUPPORTS_VIEW, so isView() is true only - // for it). Same config gate + time-travel rejection + view-body dispatch as legacy; - // the config-off message is thrown before the time-travel check, matching the legacy - // arm's ordering (a config-off view errors with the config message regardless of any - // time/version travel). The view body is converted by the session dialect inside - // parseAndAnalyzeExternalView (shared with HMS hive-view), which is fully neutral. + // Plugin view (hive after the hms cutover, or iceberg): any connector that declares + // SUPPORTS_VIEW serves its view here, unconditionally — the legacy + // enable_query_hive_views / enable_query_iceberg_views switches are deprecated no-ops, + // so a view is served regardless of them. The view body is converted by the session + // dialect inside parseAndAnalyzeExternalView (shared with the legacy HMS hive-view + // path), which is fully neutral. PluginDrivenExternalTable pluginViewTable = (PluginDrivenExternalTable) table; - if (!Config.enable_query_iceberg_views) { - throw new UnsupportedOperationException( - "please set enable_query_iceberg_views=true to enable query iceberg views"); - } if (unboundRelation.getTableSnapshot().isPresent()) { - // iceberg view not supported with snapshot time/version travel + // A view cannot be combined with snapshot time/version travel (meaningless for a + // view, for hive and iceberg alike). // note that enable_fallback_to_original_planner should be set with false // or else this exception will not be thrown // because legacy planner will retry and thrown other exception throw new UnsupportedOperationException( - "iceberg view not supported with snapshot time/version travel"); + "view not supported with snapshot time/version travel"); } isView = true; String pluginCatalog = pluginViewTable.getCatalog().getName(); diff --git a/regression-test/suites/external_table_p0/iceberg/test_iceberg_view_query_p0.groovy b/regression-test/suites/external_table_p0/iceberg/test_iceberg_view_query_p0.groovy index a5197835ab9c9d..f27dad2b6b6cf2 100644 --- a/regression-test/suites/external_table_p0/iceberg/test_iceberg_view_query_p0.groovy +++ b/regression-test/suites/external_table_p0/iceberg/test_iceberg_view_query_p0.groovy @@ -112,85 +112,85 @@ suite("test_iceberg_view_query_p0", "p0,external") { try { sql """select * from v_with_partitioned_table FOR TIME AS OF '2025-06-11 20:17:01' order by col1 limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_partitioned_table FOR VERSION AS OF 5497706844625725452 order by col1 limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_partitioned_table FOR TIME AS OF '2025-06-11 20:17:01'""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_partitioned_table FOR VERSION AS OF 5497706844625725452""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_unpartitioned_table FOR TIME AS OF '2025-06-11 20:17:01' order by col1 limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_unpartitioned_table FOR VERSION AS OF 5497706844625725452 order by col1 limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_unpartitioned_table FOR TIME AS OF '2025-06-11 20:17:01'""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_unpartitioned_table FOR VERSION AS OF 5497706844625725452""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_partitioned_column FOR TIME AS OF '2025-06-11 20:17:01' order by col5 limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_partitioned_column FOR VERSION AS OF 5497706844625725452 order by col5 limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_partitioned_column FOR TIME AS OF '2025-06-11 20:17:01'""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_partitioned_column FOR VERSION AS OF 5497706844625725452""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_joint_table FOR TIME AS OF '2025-06-11 20:17:01' order by sale_date limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select * from v_with_joint_table FOR VERSION AS OF 5497706844625725452 order by sale_date limit 10""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_joint_table FOR TIME AS OF '2025-06-11 20:17:01'""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } try { sql """select count(*) from v_with_joint_table FOR VERSION AS OF 5497706844625725452""" } catch (Exception e) { - assertTrue(e.getMessage().contains("iceberg view not supported with snapshot time/version travel"), e.getMessage()) + assertTrue(e.getMessage().contains("view not supported with snapshot time/version travel"), e.getMessage()) } sql """drop view v_with_partitioned_table""" From 3b6985c7c8899310201bdf43a6dc887971ecfb14 Mon Sep 17 00:00:00 2001 From: morningman Date: Fri, 10 Jul 2026 22:21:41 +0800 Subject: [PATCH 202/334] [doc](catalog) HMS SPI cutover flip DONE (A-D) + clean-room CLEAN; HANDOFF -> e2e then Phase 3 delete MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Design/record of the Phase-2 atomic flip: hms routed through the plugin SPI (all production flows on new code), event-sync activation feeds, D5 view cleanup, 3 legacy tests disabled+deferred. Corrects the execution-plan §3 checklist (leave the 4 instanceof gates; leave dead Nereids arms for Phase 3). Full instanceof-surface classification + verification (build + targeted tests + clean-room review CLEAN, 0 confirmed defects). HANDOFF rolled to: e2e (user runs), then Phase 3 legacy deletion. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx --- plan-doc/HANDOFF.md | 64 +++++------ .../tasks/hms-spi-cutover-flip-2026-07-10.md | 100 ++++++++++++++++++ 2 files changed, 128 insertions(+), 36 deletions(-) create mode 100644 plan-doc/tasks/hms-spi-cutover-flip-2026-07-10.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index e84e267ca0ba10..3bc67011333ab6 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,41 +7,35 @@ # 🎯 当前状态（2026-07-10） -catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms-cutover-execution-plan-2026-07-10.md`**（4 阶段 + DONE 账本 + 已签字决策 + 硬门；**起步必读 #1，行号信 HEAD 不信文档**）。四阶段： +**⭐ 本轮 = Phase 2 原子翻闸完成：catalog 类型 `hms` 已接入插件 SPI，所有生产流程改走新连接器代码（plain-hive + iceberg-on-HMS + hudi-on-HMS）。未删任何旧代码（Phase 3 才删）。** 4 个独立 commit（均构建+靶向单测验证；净室对抗复核见下）： -- **Phase 0 连接器休眠补齐** ✅ **DONE**（读 SPI + iceberg/hudi 兄弟委派 + 整条 hudi 线）。 -- **Phase 1 翻闸前 fe-core 建设** — 连接器自持缓存(D2) ✅ → 事件管道 Model B ✅ → 4 个耦合缝(S1–S4)+W6 ✅ → 耦合缝净室复审 ✅ → 缓存失效收尾 3 修 ✅ → 对抗复核查出 4 个待修 ✅ → **缓存失效 4 修(R1–R4)全部 ✅ 落地+构建验证+净室对抗复核 CLEAN（见下）**。**Phase 1 缓存线收尾完毕，真正剩余主活 = Phase 2 原子翻闸前置项。** -- **Phase 2 原子翻闸**（`SPI_READY += hms` + GSON 重映射 + 死臂/删除 + 4 守卫改接新子系统）→ **Phase 3 删除旧代码（最后做）** → **Phase 4 e2e/硬门**。 +- **✅ A** `1af7f063c2d`：事件同步·跟随者游标改接新驱动。`ExternalMetaIdMgr.replayMetaIdMappingsLog` 原无条件喂旧 `MetastoreEventsProcessor`；改双臂——`PluginDrivenExternalCatalog`→新 `MetastoreEventSyncDriver`，否则旧 processor（都按 catalogId keying，不 cast HMS）。翻闸前休眠。**不改则翻闸后从库增量元数据同步静默永久停摆。** +- **✅ B** `eef6fe7b8c2`：事件同步·主/从强制初始化钩子。`MetastoreEventSyncDriver.realRun` 对未初始化目录按 `getType()=="hms"`（读 catalogProperty，不触发 init）强制 `makeSureInitialized()`，每 FE（无 isMaster 门，从库也要）、`!isInitialized()` 一次性、异常吞掉。翻闸前休眠。**不改则从没被查过的翻闸 hms 目录永不同步。** +- **✅ C（原子翻闸本体）** `fb1624be757`：`CatalogFactory` SPI_READY_TYPES += `"hms"` + 删死 `case "hms"`+import；`GsonUtils` 三处 registerSubtype→registerCompatibleSubtype（catalog→PluginDrivenExternalCatalog、db→PluginDrivenExternalDatabase、**table→PluginDrivenMvccExternalTable**，hive 连接器声明 SUPPORTS_MVCC_SNAPSHOT，对齐 paimon/iceberg）+ 删 3 个 orphaned import；`ExternalCatalog.buildDbForInit` HMS 臂→PluginDrivenExternalDatabase（镜像 ICEBERG 臂）；新增 `HmsGsonCompatReplayTest`（3 绿）；**禁用 3 个遗留测试**（`HmsCatalogTest`/`HmsQueryCacheTest`/`HiveDDLAndDMLPlanTest`——它们经路由 create 建 type=hms 目录、fe-core 测试无 hms 插件→抛错，且断言遗留 HMSExternal* 行为；`@Disabled` 指向 Phase 3）。 +- **✅ D（D5 视图收尾）** `320702b8166`：翻闸后 hive 视图=PLUGIN_EXTERNAL_TABLE 走共享插件视图臂。删 `enable_query_iceberg_views` 门（视图无条件服务）+ 中立化 “iceberg view not supported…”→“view not supported with snapshot time/version travel” + 两个 @ConfField 标记 `@Deprecated` no-op + 改 iceberg 视图回归 16 处断言。**对已上线 iceberg 是可见行为变更，随翻闸一起发。** -**⭐ 本轮（2026-07-10）= 缓存失效对抗复核留下的 4 个待修（R1–R4）全部收尾 + 复核 CLEAN。连接器缓存失效这条线（D1/D2 → R1–R4）到此 DONE。** -每项独立 commit + 独立构建验证（BUILD SUCCESS + 0 checkstyle）： +**已签字决策（用户 2026-07-10）**：① 3 个遗留测试=禁用+延后 Phase 3 删除（非改造）；② D5 视图收尾=本轮一起做。 -- **✅ R2（线上）** `7b7b3c25953`：iceberg/paimon 补 `invalidateDb`（按 db 前缀 removeIf 清快照缓存 + paimon `PaimonSchemaAtMemo`）→ `DROP DATABASE` 钩子 + 老早存在的 `REFRESH DATABASE` 对它俩生效（含 `DROP DATABASE FORCE` 级联），并 arm 了 hive `forEachBuiltSibling(sibling.invalidateDb)`。iceberg 16 + paimon 17 绿。 -- **✅ R1（线上）** `a562c91e55b`：`PluginDrivenExternalCatalog` override `replayDropTable/replayCreateTable/replayDropDb`（仿 `replayTruncateTable`），follower/observer 回放也传播 `connector.invalidate*`，按 REMOTE 名 keying；靠 getDbForReplay 门控绝不 force-init。DDL routing 60 绿。 -- **✅ R4（线上）** `43db3e8214f`：`renameTable` 改名成功后清源+目标；`RefreshManager.replayRefreshTable` rename 分支同步传播到 follower。已确认插件目录无 replace/CTAS-overwrite 路径 → 改名是同类唯一漏洞。DDL routing 60 + 新 `RefreshManagerRenameReplayTest` 2 绿。 -- **✅ R3（休眠）** `d26bfa52eea`：连接器版 `MetaCacheEntry` 加 additive `invalidationGeneration()`/`putIfNotInvalidatedSince()`，`CachingHmsClient.getPartitions` RPC 前抓 generation、逐分区守卫式 put（结果仍直接从 delegate 收，保 misparse-never-drop）。`MetaCacheEntryTest` 7 + `CachingHmsClientTest` 16 绿。 +**验证**：`mvn -pl fe-core -am test-compile` BUILD SUCCESS + 0 checkstyle；靶向 `mvn test` 17 跑 0 败 0 错 1 skip（HmsGson 3/3、Iceberg/PaimonGson 各 3/3、ExternalMetaCacheRouteResolver 7/7、HmsCatalog skip）。净室对抗复核（`wf_728cad25-62a`，4 独立审查 + 逐条对抗验证）= **CLEAN**（1 发现 0 确认缺陷；唯一 minor=翻闸 hive 表失去 Nereids SQL 结果缓存资格 @ `BindRelation:729`，经验证 by-design：fail-safe、`enable_hive_sql_cache` 默认关、与 paimon/iceberg-native 一致、被禁用的 `HmsQueryCacheTest` 已明记该遗留缓存 dead-for-hms）。 -**✅ 针对性对抗复核（`wf_b730a7d4-6a3`，6 盲审 finder + 每发现 3 视角验证）：CLEAN — 0 发现，四修既正确又完整，无后续。** 关键确认：**hudi 兄弟不需要 `invalidateDb`**（它建裸 `ThriftHmsClient` 直读、非 `CachingHmsClient`）；无其它同类未修路径；replay 绝不 force-init；原子换名不留陈旧钉。报告 = `plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md`。设计文档 = `plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md`（§6 全勾）。 +**⭐ 下一步（下个 session）：** +1. **e2e 回归（用户自跑，勿丢，非静默）**：异构 `type=hms` 目录跑 hive/iceberg-on-HMS/hudi-on-HMS 读/写/DDL/procedure/MTMV/time-travel/@incr；**从库事件同步陈旧**（新跟随者游标喂路从未在任何目录跑过，hms 是第一个事件消费者）；Kerberos-HMS 冒烟；升级镜像 GSON replay；耦合缝行（partition_values/hudi_meta/sample-analyze/auto-analyze）；hive 视图（现无条件服务）。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 +2. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂（PhysicalPlanTranslator HMS/hudi 臂、INC-5 stale throws、CheckPolicy hudi 臂、遗留 BindRelation HMS_EXTERNAL_TABLE 臂）+ 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 -**⭐ 下一步（下个 session）= Phase 2 原子翻闸前置项 + 现在就能做的 e2e 回归：** -1. **Phase 2 原子翻闸集**（权威 = `hms-cutover-execution-plan-2026-07-10.md` §2/§3）：`SPI_READY_TYPES += "hms"`、GSON 重映射、4 守卫改接新子系统、死臂/删除排序。**Phase 3 删除旧代码最后做**，最硬约束：`datasource/hive|hudi|iceberg/` ~90 个 HMS 支撑类删不掉直到翻闸切消费者（execution plan §2.4/§3）。 -2. **e2e 欠账（非静默，勿丢）**：**paimon/iceberg 的 drop+recreate + rename-swap 修复现在就 live，可即刻 e2e 回归**（`DROP TABLE t; CREATE TABLE t(新schema/位置); SELECT` 无需 REFRESH 见新表；`DROP DATABASE d; CREATE DATABASE d`；`RENAME` 原子换名；paimon 时间旅行复用 schemaId 见新 schema）。hive 侧休眠步翻闸后才 live。异构 HMS docker e2e 清单见 design §5 + execution plan §4/§5 + memory `hms-iceberg-delegation-needs-e2e`。 -3. **设计文档「Discovered follow-ups」2 项欠账（勿丢）**：① 删除步把 `StatisticsAutoCollector.java`/`StatisticsCache.java` 的 `org.apache.hudi...VisibleForTesting` 换 guava/doris；② 分区级 FULL analyze 缺失（所有插件表通性、非本轮引入）。 +**⚠ 关键纠正（execution plan §3 已过时，本轮已核实纠正，见 `hms-spi-cutover-flip-2026-07-10.md` §2）**：§3.7「rewire 4 个 gate」**错**——两个 instanceof gate（MetastoreEventsProcessor:116、ExternalMetaCacheRouteResolver:66）须**保留**（自动排除翻闸目录、对未翻闸旧目录仍正确；删则破坏旧目录同步/失效）；缓存路自动接管（连接器 CachingHmsClient），事件路靠上面 A/B 两个 ADD-feed（非删 gate）。死 Nereids 臂**翻闸不删**（对齐 iceberg 翻闸留死臂的先例，Phase 3 统删）。 -**⚠ 并行 session 风险**：曾有两个 session 同工作树互相 amend；起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手（本轮见到的 `gq` 用户 maven 是**别的工作树**，非本树）。 +**⚠ 并行 session 风险**：起步先查 `git log`/`git status`/运行中 maven/近 90s mtime 再动手（memory `concurrent-sessions-shared-worktree-hazard`）。 --- # 🧠 起步必读（读文档，别炒 git log 历史） -> **路径注意**：设计/计划文档都在 `plan-doc/tasks/` 下，复审报告在 `plan-doc/reviews/` 下。 +> **路径**：设计/计划在 `plan-doc/tasks/`，复审报告在 `plan-doc/reviews/`。 -1. **权威翻闸计划（下个 session 起步必读）** = `plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 已签字决策 + 原子翻闸集 + 硬门）。**行号信 HEAD 不信文档**。 -2. **缓存失效收尾（本轮，已 DONE）设计** = `plan-doc/tasks/hive-cache-invalidation-followups-design-2026-07-10.md`（§6 全勾）；复核报告 = `plan-doc/reviews/cache-invalidation-fixes-rereview-2026-07-10.md`（CLEAN）+ 上一轮 `plan-doc/reviews/cache-invalidation-cleanroom-review-2026-07-10.md`（R1–R4 来源）。 -3. 已完成的前几项 Phase-1 步：`plan-doc/tasks/hive-connector-cache-step-design-2026-07-10.md`（D2 缓存本体）、`plan-doc/tasks/hive-event-pipeline-step-design-2026-07-10.md`（事件管道）、`plan-doc/tasks/hive-coupling-seams-step-design-2026-07-10.md`（4 耦合缝）。 -4. **补充权威**：`plan-doc/tasks/hms-cutover-retype-design-2026-07-07.md`（原子翻闸模型 + §6 D1–D6 决策）；`plan-doc/tasks/hms-cutover-sibling-connector-decomposition-2026-07-08.md`（兄弟委派 + CCE/TCCL 硬约束）；hudi = `plan-doc/tasks/hudi-on-hms-delegation-plan-2026-07-08.md` + `plan-doc/tasks/hudi-schema-evolution-step-design-2026-07-09.md`。 -5. **样板**：`plan-doc/tasks/P5-paimon-migration.md`、`plan-doc/tasks/P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式）。 -6. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 +1. **本轮翻闸设计（下个 session 起步必读）** = `plan-doc/tasks/hms-spi-cutover-flip-2026-07-10.md`（做了什么 + 对 execution plan 的纠正 + instanceof 全分类 + 验证 + e2e 欠账 + Phase 3 清单）。**行号信 HEAD 不信文档。** +2. **权威翻闸计划（历史，§3 清单本轮已纠正）** = `plan-doc/tasks/hms-cutover-execution-plan-2026-07-10.md`（4 阶段 + DONE 账本 + 硬门；其 §2「Phase 1 未建」已过时，Phase 1 早已 DONE）。 +3. **样板**：`plan-doc/tasks/P5-paimon-migration.md`、`P6-iceberg-migration.md`（净室复审 + 能力孪生 + GSON replay 范式；iceberg 翻闸=加 SPI_READY + GSON compat + 留死臂到 Phase 3）。 +4. 完成工作明细 = `git log`（commit message 详尽）；勿在 HANDOFF 里重述。 --- @@ -49,26 +43,24 @@ catalog-SPI 迁移的剩余工作 = **HMS 翻闸**。权威计划 = **`tasks/hms - **工作分支 = `catalog-spi-11-hive`**（off `branch-catalog-spi`）。PR base = `branch-catalog-spi`，**squash 合并**。 - **公开 tracking issue = apache/doris#65185**（进度按已合入 `branch-catalog-spi` PR 口径）。 -- **⚠️ path-whitelist `git add`，严禁 `git add -A`**：工作树有大量历史遗留 scratch（`*.bak` / `regression-conf.groovy` 明文 key / `.audit-scratch/` / `conf.cmy/` / `META-INF/` / `docker/...` / `plan-doc/reviews/P5-*` / `.claude/` / `failed-cases.out`——**非本线程产物，勿混入任何 commit**）。 +- **⚠️ path-whitelist `git add`，严禁 `git add -A`**：工作树大量历史遗留 scratch（`*.bak` / `regression-conf.groovy` 明文 key / `.audit-scratch/` / `conf.cmy/` / `META-INF/` / `docker/...` / `plan-doc/reviews/P5-*` / `.claude/` / `failed-cases.out`——**非本线程产物，勿混入任何 commit**）。 - commit message：`[feat|fix|doc](catalog) …` + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + `Claude-Session: …`。**每子步/每条 fix = 独立 commit**；HANDOFF + 设计文档单独 commit（与 code 分开）。上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 # ⚙️ 操作须知（构建/测试，复用） -- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**）。连接器：`-pl :fe-connector- -am`。单测加 `-Dtest=`。 -- **⚠️ paimon 模块必须用 `install`（不是 `test`）验证**：`fe-connector-paimon` 的 `HiveConf` 来自 `fe-connector-paimon-hive-shade`，其 maven-shade 绑在 `package` 阶段；`mvn test` 停在 `test` 阶段→shade jar 没产出→paimon 编译报 `package org.apache.hadoop.hive.conf does not exist`（**误导：非你的改动**）。用 `-pl :fe-connector-paimon -am install`。hms/hive 无此坑（hive-common 是普通依赖）。 -- **验证信 LOG 不信 exit**：`-q` + `| tail` 会吞掉 `BUILD SUCCESS`；改**重定向到文件**跑（不加 `-q`），再 grep `BUILD SUCCESS`/`[ERROR].*\.java:`/`Tests run:`/checkstyle `You have N violations`（memory `doris-build-verify-gotchas`）。 -- **build 中 ANTLR `mismatched input '->'`/`super::` 噪音 = 非致命**。 -- **import gate**：`bash tools/check-connector-imports.sh`（repo 根跑）。HMS `HiveVersionUtil` 命中 = 误报（memory `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 -- **⚠️ bash 默认 timeout 120s**：全量编译 ~3min+ → 后台跑 + 读 LOG。**⚠️ `/mnt/disk1` 盘紧；勿用 worktree 隔离编译 agent**。cwd 会被重置 → 绝对路径。 -- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**（`PluginDrivenExternalCatalogDdlRoutingTest` 用 `mock(Connector.class)` 直接 verify invalidate\*）。checkstyle 禁 static import、扫 test 源。 +- maven：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false`（**漏 `-am`→假错 `${revision}`**）。连接器：`-pl :fe-connector- -am`。**靶向单测**加 `-Dtest= -DfailIfNoTests=false`（`-am` + `-Dtest` 会因上游模块无匹配测试报 “No tests were executed!” 假失败）。 +- **⚠️ paimon 模块必须用 `install`（不是 `test`）验证**（shade jar 绑 `package` 阶段）；hms/hive 无此坑。 +- **验证信 LOG 不信 exit**：后台 task 通知的 exit code 是 wrapper 的（本轮见过 rc=1 但通知报 exit 0）；重定向到文件跑（不加 `-q`），grep `BUILD SUCCESS`/`BUILD FAILURE`/`[ERROR].*\.java:`/`Tests run:`/`You have N Checkstyle`（memory `doris-build-verify-gotchas`）。 +- **⚠️ bash 默认 timeout 120s**：全量编译 ~6min → 后台跑 + 读 LOG。**⚠️ `/mnt/disk1` 盘紧；勿用 worktree 隔离编译 agent**。cwd 会被重置 → 绝对路径。 +- **连接器测试无 Mockito**（真 recording fake）；**fe-core 测有 Mockito**。checkstyle 禁 static import、扫 test 源、`UnusedImports` 会 fail build。 # 🔒 铁律（fe-core 约束） -- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。本轮 D1 钩子调 `connector.invalidateTable/invalidateDb` 是**通用 SPI 调用**（no-op 默认），不违反。 -- fe-core **不解析属性**（storage→fe-filesystem、meta→fe-connector，均插件侧；memory `catalog-spi-no-property-parsing-in-fecore`）。 -- 跨插件/跨边界**须 pin TCCL** 到连接器 classloader（扫描线程/写-DDL 引擎线程/iceberg worker 池/事件轮询后台线程；memory `catalog-spi-plugin-tccl-classloader-gotcha`）。invalidate\* 是直接方法派发（非按名反射），照搬 RefreshManager 现有调用不需 pin。 -- `history_schema_info` 嵌套字段名逐层 lowercase（否则 BE SIGABRT；memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 -- `PluginDrivenMvccExternalTable`/`PluginDrivenExternalTable` 是 paimon/iceberg **实时**基类：改其共享方法须对二者字节+成本双不变（memory `plugindriven-mvcc-table-is-live-not-dormant`）。**本轮 D1 是有意的线上行为改动（修 drop+recreate 洞），非字节中性——已配净室复审 + e2e 欠账把关。** +- fe-core **不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 引擎名判别；通用 SPI 节点 connector-agnostic（memory `catalog-spi-plugindriven-no-source-specific-code`）。本轮 B 的 `getType()=="hms"` 门是**事件源类型探测（对齐旧 poller 的 instanceof HMSExternalCatalog）**，非源判别式违规。 +- fe-core **不解析属性**（storage→fe-filesystem、meta→fe-connector；memory `catalog-spi-no-property-parsing-in-fecore`）。 +- 跨插件/跨边界**须 pin TCCL**（memory `catalog-spi-plugin-tccl-classloader-gotcha`）。 +- `history_schema_info` 嵌套字段名逐层 lowercase（memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 +- `PluginDrivenMvccExternalTable`/`PluginDrivenExternalTable` 是 paimon/iceberg/**翻闸后 hms** 实时基类（memory `plugindriven-mvcc-table-is-live-not-dormant`）。 # 🗂 memory 相关项 diff --git a/plan-doc/tasks/hms-spi-cutover-flip-2026-07-10.md b/plan-doc/tasks/hms-spi-cutover-flip-2026-07-10.md new file mode 100644 index 00000000000000..e72437904ddae0 --- /dev/null +++ b/plan-doc/tasks/hms-spi-cutover-flip-2026-07-10.md @@ -0,0 +1,100 @@ +# HMS SPI cutover — the atomic flip (2026-07-10) + +> Phase 2 of the HMS cutover: route catalog type `hms` (plain-hive + iceberg-on-HMS + hudi-on-HMS) through the +> plugin SPI so all production flows use the new connector code. **No legacy code deleted** — that is Phase 3. +> Grounded on HEAD by two recon workflows (`wf_c0e650aa-b62` flip-site grounding + 2 adversarial critics; +> `wf_78b374d6-a29` full instanceof-surface classification + force-init design). The execution-plan doc +> (`hms-cutover-execution-plan-2026-07-10.md` §3) was the starting checklist but was **corrected** here — see §"Corrections". + +--- + +## 1. What shipped + +Two dormant prerequisites (event sync), then the atomic flip, then the view-gate cleanup. + +### Commit A — follower event-cursor feed → new driver (dormant) +`ExternalMetaIdMgr.replayMetaIdMappingsLog` fed the **legacy** `MetastoreEventsProcessor.updateMasterLastSyncedEventId`. +Post-flip a hive catalog is a `PluginDrivenExternalCatalog` driven by the new `MetastoreEventSyncDriver`; without +repointing, its `masterLastSyncedEventIdMap` never populates → `masterUpperBound` stays -1 → `HmsEventSource.pollForFollower` +returns nothing → **followers silently stop applying incremental metastore changes**. Now dual-armed: +`PluginDrivenExternalCatalog` → new driver, else legacy processor (both key by catalogId, no HMS cast). Dormant pre-flip. + +### Commit B — master/follower force-init hook (dormant) +`MetastoreEventSyncDriver.realRun` skipped un-initialized catalogs (to keep idle paimon/iceberg inert), so a flipped +hms catalog **never queried on an FE never seeds its cursor → never event-synced**. Added a per-cycle force-init gated +on the pre-init `getType()=="hms"` (reads `catalogProperty`, no `makeSureInitialized`), on every FE (no isMaster gate — +each FE needs the catalog initialized to obtain its event source / seed its own cursor / forward `REFRESH CATALOG`), +one-shot via `!isInitialized()`, swallow-on-throw like the legacy poller. Dormant pre-flip. + +### Commit C — THE ATOMIC FLIP +- `CatalogFactory`: `SPI_READY_TYPES += "hms"`; remove the dead `case "hms" → new HMSExternalCatalog` + its import. +- `GsonUtils`: three `registerSubtype` → `registerCompatibleSubtype` remaps (catalog→`PluginDrivenExternalCatalog`, + db→`PluginDrivenExternalDatabase`, **table→`PluginDrivenMvccExternalTable`** — the hive connector declares + `SUPPORTS_MVCC_SNAPSHOT`, matching paimon/iceberg); remove the three orphaned HMS imports. Locked with `SPI_READY` + (label-uniqueness static-init throw + write-registration) → one commit. +- `ExternalCatalog.buildDbForInit`: HMS arm → `new PluginDrivenExternalDatabase` (mirrors the ICEBERG arm; replay-defense). +- `HmsGsonCompatReplayTest` (new): pins that legacy `HMSExternal{Catalog,Database,Table}` tags replay as the PluginDriven + classes (table → the MVCC variant, exact-class assert). Mirrors `Iceberg`/`PaimonGsonCompatReplayTest`. +- Disable 3 legacy fe-core tests (`HmsCatalogTest`, `HmsQueryCacheTest`, `HiveDDLAndDMLPlanTest`) — they create a + `type=hms` catalog via the **routed** path (no hms provider on the fe-core test classpath → throws) and assert legacy + `HMSExternal*` behavior. `@Disabled` with a Phase-3 pointer (removed with the legacy subsystem). **User sign-off: + disable + defer (2026-07-10).** + +### Commit D — view-gate cleanup (D5) +Post-flip a hive view is `PLUGIN_EXTERNAL_TABLE` → hits the shared plugin-view arm in `BindRelation` (today +iceberg-worded). Removed the `enable_query_iceberg_views` config gate (served unconditionally), neutralized the +"iceberg view not supported with snapshot time/version travel" message → "view not supported ..."; deprecated both +`enable_query_hive_views`/`enable_query_iceberg_views` `@ConfField`s to retained no-ops; updated the 16 assertions in +`test_iceberg_view_query_p0.groovy`. **Visible iceberg behavior change — ships with the flip. User sign-off: do now (2026-07-10).** + +--- + +## 2. Corrections to the execution-plan §3 checklist (verified against HEAD) + +- **"Rewire the 4 gates" is WRONG.** Both `instanceof HMSExternalCatalog` gates (`MetastoreEventsProcessor:116`, + `ExternalMetaCacheRouteResolver:66`) must be **LEFT** — they self-exclude a flipped PluginDriven catalog and stay + correct for any still-legacy HMS catalog; deleting them breaks legacy sync/invalidation. The **cache** path + auto-engages (connector `CachingHmsClient` + base metaCache; invalidation via PluginDriven overrides) — zero edits, + co-proven by paimon/iceberg. The **event** path needed the two ADD-feeds above, not gate removal. +- **Dead Nereids arms are NOT removed in the flip.** The iceberg flip (precedent, in HEAD) left its dead + `PhysicalPlanTranslator`/`IcebergScanNode` arms; they + the hms/hudi arms + the legacy `BindRelation` + `HMS_EXTERNAL_TABLE` arm + `CheckPolicy` hudi arm are all dead-harmless post-flip and are removed together in Phase 3. + (Keeps this flip minimal per "其他的先不动".) +- **Phase-1 was already DONE at HEAD** (event Model B + connector cache D2 + R1–R4), contrary to the plan doc's stale §2. + +## 3. instanceof-HMS surface (full classification — `wf_78b374d6-a29`) +15 DUAL_ARMED_SAFE (write/DDL sinks, INSERT OVERWRITE, CREATE TABLE engine, TVFs, SHOW CREATE DB, MetadataGenerator), +9 LEAVE_DEAD_HARMLESS (legacy caches/scan internals — Phase 3), 5 ALREADY_COVERED_BY_SEAM (S1/S3/S4 + MaterializeProbe), +and 3 SINGLE_ARMED read-path items that are **plugin-model parity, not bugs** (accepted, no action): +- `BindRelation:729` — flipped hive loses SQL-result-cache eligibility → **same as paimon/iceberg-native** (cache off, + never stale, plan §6). +- `ShowPartitionsCommand:154/222` — WHERE/ORDER-BY column strictness → flipped hive now matches already-flipped + paimon/iceberg (which skip the guard). +`RefreshManager:216` was flagged by a critic as a partition-invalidation regression but is **NOT** — the live +event-driven path (`refreshPartitions:307-311`) is already flip-aware (partition-granular `connector.invalidatePartition`), +and the replay fallback (`refreshTableInternal:268-270`) is connector-aware full-table invalidation (safe superset). + +--- + +## 4. Verification +- `mvn -pl fe-core -am test-compile`: **BUILD SUCCESS, 0 Checkstyle** (whole reactor). +- Targeted `mvn test` (17 run, 0 failures, 0 errors, 1 skipped): `HmsGsonCompatReplayTest` 3/3, + `IcebergGsonCompatReplayTest` 3/3, `PaimonGsonCompatReplayTest` 3/3, `ExternalMetaCacheRouteResolverTest` 7/7 green + (still routes a directly-constructed legacy `HMSExternalCatalog` to hive+hudi+iceberg); `HmsCatalogTest` **skipped** (@Disabled). +- Clean-room adversarial review (`wf_728cad25-62a`, 4 independent reviewers + per-finding adversarial verify): + **CLEAN** — 1 finding, 0 confirmed real. The one minor finding (flipped hive loses Nereids SQL-result-cache + eligibility at `BindRelation:729`) was verified **by-design**: fail-safe, `enable_hive_sql_cache` defaults off, + parity with paimon/iceberg-native, and acknowledged by the disabled `HmsQueryCacheTest` (§3 above). +- e2e is owed to the user (they run it). See §5. + +## 5. e2e owed (user runs) +Heterogeneous `type=hms` catalog: plain-hive + iceberg-on-HMS + hudi-on-HMS in one session; hive/iceberg/hudi read, +write, DDL, procedure, MTMV freshness, time-travel, `@incr`; **follower event-sync staleness** (the new follower feed +has never run for any catalog — hms is the first event consumer); Kerberos-HMS smoke; upgrade-image GSON replay; +`partition_values()`/`hudi_meta()`/sample-analyze/auto-analyze coupling-seam rows; hive view query (now served +unconditionally). Full matrix: `hms-cutover-execution-plan-2026-07-10.md` §4/§5. + +## 6. NOT done (Phase 3, deferred) +Delete the ~90-class legacy cyclic unit (`datasource/hive|hudi|iceberg`), the dead Nereids arms + INC-5 stale throws + +`CheckPolicy` hudi arm + legacy `BindRelation` HMS arm, the deletion-unblocking extractions (`HiveUtil`/`HiveSplit`/ +`IcebergUtils`), and the 3 disabled tests. Ordering + set: execution plan §2.4/§3/§4. From 79c489ec3e429b3889b9015334cabd16775a4ab4 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 04:50:15 +0800 Subject: [PATCH 203/334] [doc](catalog) #65185 reverify: third-party review re-checked vs post-HMS-cutover branch The third-party #65185 review targeted branch-catalog-spi (P1-P6 only, no hive/hms). This branch already completed the HMS SPI cutover ("hms" in SPI_READY_TYPES), so every finding is re-verified against current code (located by symbol; report line numbers are stale) via a multi-agent workflow: 3 recon + 34 finding-cluster verifiers + adversarial refutation of each surviving high/medium, cross-checked against deviations/decisions-log. 79 findings classified. Headline: the report's "dormant P7 hudi landmines" are now ACTIVATED by the cutover into 4 live high-severity regressions on hudi-on-HMS (partition unescape / datetime-ISO / HMS-name-as-storage-path / mixed-case-Avro JNI crash); several others fixed by P7 work (HD-A4 COW/MOR) or refuted (P3b-1, P3b-3 kerberos). Adds both the third-party input report and the dated reverify (severity-sorted, per-finding current status + evidence + fix, plus a "not-applicable / already-fixed / parity" section). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- ...og-spi-review-65185-reverify-2026-07-11.md | 255 +++++++++++ plan-doc/reviews/catalog-spi-review-65185.md | 422 ++++++++++++++++++ 2 files changed, 677 insertions(+) create mode 100644 plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md create mode 100644 plan-doc/reviews/catalog-spi-review-65185.md diff --git a/plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md b/plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md new file mode 100644 index 00000000000000..a1d1ecca274f71 --- /dev/null +++ b/plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md @@ -0,0 +1,255 @@ +# #65185 第三方评审 — 对当前分支 `catalog-spi-11-hive` 的逐条核实 + +> **输入**：`plan-doc/reviews/catalog-spi-review-65185.md`(第三方评审，基线 = `upstream-apache/branch-catalog-spi` tip `3ba75b7cf8a`，**仅含 P1–P6，无 P7/hive**)。 +> **核实对象**：当前本地分支 `catalog-spi-11-hive`(HEAD `3b6985c7c88`),**HMS SPI 原子翻闸已完成**。 +> **方法**:多 agent 工作流(3 recon + 34 finding-cluster 核实 + 对每条存活的高/中发现做对抗证伪),按 **符号/行为** 定位现码(报告行号已过时),交叉核对 `deviations-log`/`decisions-log` 避免把已验收偏差当新 bug。79 条结论,62 条存活为真/活跃。 +> **纪律**:不动代码,仅核实分析。所有 file:line 均对 HEAD 核过。 + +--- + +## 0. 一句话结论 + +报告的核心前提**已过时**:它写于「hive 未迁移、hudi 休眠」的分支;当前分支 `"hms"` 已进 `SPI_READY_TYPES`(`CatalogFactory.java:55`),**plain-hive + iceberg-on-HMS + hudi-on-HMS 全部改走插件路径**(hudi 作为 hms 网关的兄弟连接器)。因此报告的每条结论必须重新落到「当前分支」这一坐标: + +- **报告的「休眠 P7 地雷」中,hudi 的 4 个高危已被翻闸激活为真实回归**(silent 丢行 / JNI 崩溃),这是本次核实**最重要的产出**,比报告更严重。 +- 翻闸配套的大量 P7 工作(hive 耦合缝、连接器缓存失效、事件同步、hive 写事务、HD-A4 表类型硬化)**已修掉若干报告条目**。 +- 与连接器无关的 P0/P2/P4/P5/P6(iceberg/paimon/maxcompute)发现**基本仍然成立**。 +- 一批被对抗证伪 / 属 parity / 已登记验收 的条目**在当前分支不构成需修问题**,单列一节说明(§6),以免误改。 + +**行动优先级**:先处理 §2 的 4 个 hudi 高危(翻闸即静默丢行/崩),再看 §3 的中危。 + +--- + +## 1. 严重程度总表(仅列「真实/活跃」的条目) + +> 严重度取「对抗验证后的最终值」;`新激活` = 报告标休眠、翻闸后变活;`仍然` = 与连接器无关一直存在。 + +| # | ID | 现状 | 严重度 | 范围 | 一句话 | +|---|---|---|---|---|---| +| H1 | P3-hudi-unescape | 新激活 | 🔴高 | hudi-on-HMS | 分区名不 unescape,含 `:/%/空格/=` 的值静默剪掉→丢行 | +| H2 | P3-hudi-datetime | 新激活 | 🔴高 | hudi-on-HMS | datetime 分区谓词渲染成 ISO(`T`、省秒)永不等于 Hive 文本→整表剪到 0 行 | +| H3 | P3-hudi-storagepath | 新激活 | 🔴高 | hudi-on-HMS | 剪枝把 HMS hive-style 名当 Hudi 存储路径喂 fsView→非 hive-style 表带 filter 0 split | +| H4 | P3-hudi-avro-jni | 新激活 | 🔴高 | hudi-on-HMS | JNI 列名发原始大小写,BE 精确匹配 lowercase slot→每个 MOR/JNI split 崩 | +| M1 | P1-2 | 新激活 | 🟠中 | hive | `TABLESAMPLE` 在插件路径静默全表扫(不转发+无采样管线) | +| M2 | P4-batch-hive | 新激活 | 🟠中 | hive | 翻闸 hive 完全丢批量/异步 split(连接器未 opt-in 任何 batch flavor) | +| M3 | P4-1 | 仍然 | 🟠中 | maxcompute | batch 闸门 `!=NOT_PRUNED`→`!isPruned`,无谓词大分区表丢异步 batch | +| M4 | P4-2 | 仍然 | 🟠中 | maxcompute | 分区值缓存删除,每次规划一次全量 ODPS `listPartitions` | +| M5 | P6-3 | 仍然(iceberg-on-HMS 新激活) | 🟠中 | iceberg | `computeRowCount` 丢 equality-delete gate→MOR/CDC 表统计膨胀误导 CBO | +| M6 | P6-4 | 仍然 | 🟠中 | iceberg | s3tables 无显式凭证即硬失败,丢 EC2 instance-profile 默认链 | +| M7 | P6-5 | 仍然 | 🟠中 | iceberg | REST vended-cred region 别名收窄,丢 `AWS_REGION`/`*.signing-region`→"Unable to load region" | +| M8 | P6-2 | 仍然(hms 放大) | 🟠中(运营) | 全 SPI | 升级只换 `lib/` 不部署 `plugins/connector`→所有存量 hms/iceberg 目录首访抛异常 | +| L1 | P0-5 | 仍然 | 🟡低 | 门禁 | import-gate 三洞(漏 `import static`/漏 6 个包/只扫 main) | +| L2 | P1-3-sqlcache | 新激活 | 🟡低 | hive | 翻闸 hive 丢 SQL 结果缓存资格 + `COUNTER_QUERY_HIVE_TABLE` 停增 | +| L3 | P2-1 | 仍然 | 🟡低 | trino | 元数据方法开 Trino 事务从不 commit/close(与 master parity,量级放大) | +| L4 | P2-2 | 仍然 | 🟡低 | trino | `getInstance(pluginDir)` 首胜单例 vs per-catalog `trino.plugin.dir`(fail-loud) | +| L5 | P2-4 | 仍然 | 🟡低 | trino | `listTableNames` 丢 LinkedHashSet 去重 | +| L6 | P2-5 | 仍然 | 🟡低 | trino | guard 字段先于其余字段发布→并发首访瞬时 NPE(volatile 自愈) | +| L7 | P3b-2 | 仍然 | 🟡低 | kerberos | Kerberos 认证器每次 login/refresh 无条件 `UGI.setConfiguration`+强设 authorization(丢 first-writer guard);metastore 路 parity | +| L8 | P3b-4 | 仍然 | 🟡低 | kerberos | `doAs` 把 `InterruptedException`→`IOException` 不 restore interrupt | +| L9 | P4-4 | 仍然 | 🟡低 | maxcompute | 谓词下推全有全无(一个不可转 conjunct 丢整个 filter);perf-only | +| L10 | P5-1 | 仍然(hive 新激活) | 🟡低 | 全 SPI | EXPLAIN 节点名 `V{PAIMON,ICEBERG,HIVE}_SCAN_NODE`→`VPluginDrivenScanNode`;未登记 | +| L11 | P5-2 | 仍然 | 🟡低 | paimon | JNI/COUNT range 用表级 `file.format`(默认 parquet)非首文件后缀;默认 JNI reader 不消费故影响窄 | +| L12 | P5-3 | 仍然 | 🟡低 | paimon/iceberg | `selectedPartitionNum` 语义:SDK-distinct→Nereids-剪枝数;影响 EXPLAIN + `sql_block_rule` | +| L13 | P5-4 | 仍然 | 🟡低 | paimon | to-Paimon 建表丢嵌套 nullability(comment 半已登记 DV-035c) | +| L14 | P5-5 | 仍然 | 🟡低 | paimon | `ignore_split_type` session 变量插件路径静默 no-op | +| L15 | P5b-1 | 仍然 | 🟡低 | paimon | `PAIMON_SCAN_METRICS` 常量悬空 0 writer;FE 侧 paimon scan 剖面回归(已登记) | +| L16 | P6-1 | 部分已修 | 🟡低 | iceberg | 快照缓存 vs schema 缓存偏斜的结构隐患仍在,但触发已被同 TTL+原子失效大幅收窄 | +| L17 | P6-6 | 仍然 | 🟡低 | iceberg | 同表多版本自连接:schema 绑定 version-blind vs scan version-aware→字段 id 偏斜(窄触发,fail-loud) | +| L18 | P6-9 | 仍然 | 🟡低 | iceberg | 未知/v3 类型(TIMESTAMP_NANO/GEOMETRY…)静默降 UNSUPPORTED,legacy schema-load 抛 | +| L19 | MAGICKEY-collision | 仍然 | 🟡低 | 全 SPI | `partition_columns` 魔法键与源 TBLPROPERTY 撞名→非分区表被误当分区(窄) | +| L20 | P4-3-EQ | 待实测 | 🟡低 | maxcompute | EQ 发 `==`(SDK/legacy 是 `=`);ODPS 是否容忍需 live A/B | +| — | (已登记/验收的低危) | — | 🟡低 | — | P4-5(DV-018)、P4-6(DV-034)、P4-SHOWPART:limit(DV-021)、P4-SHOWPART:where、P6-7、P6-8、P6-10(DV-049③) — 见 §5 | +| — | (设计债 15 条) | — | ⚪ | — | 见 §4(含 **DUPLICATION:partition-prune** — 承载 H1–H3 的复制块,修复须一处落地) | + +--- + +## 2. 🔴 高危(4 条,均由 HMS 翻闸新激活,均在 hudi-on-HMS 上) + +> 共同背景:hudi 表寄生于 HMS 目录,翻闸后经 hive 网关 `createSiblingConnector("hudi")` → `HudiConnectorMetadata.applyFilter`(由 `PluginDrivenScanNode.convertPredicate:648` 实时调用)。报告标 CONFIRMED-but-dormant,现全部 **可达**。四者集中在同一段被复制的 EQ/IN 剪枝代码里(见 §4 DUPLICATION:partition-prune),修复应在共享 helper 一处落地。 + +### H1 — hudi 分区值不 unescape → 静默丢行 + +- **现证据**:`HudiConnectorMetadata.parsePartitionName:1054-1064` 用 `part.substring(eq+1)` 存值,**无 unescape**;`matchesPredicates:1067-1078` 直接 `allowedValues.contains(actualValue)` 字符串比。候选名来自 `hmsClient.listPartitionNames:247`(`ThriftHmsClient.listPartitionNames:214-218` 原样转发 HMS,返回 Hive 转义名如 `ts=2024-01-01 10%3A00%3A00`);谓词侧 `extractLiteralValue:1030-1036` 返回未转义值 `2024-01-01 10:00:00`。二者永不相等→`matched=0`,而 `matched(0) != all(N)` 使 `applyFilter:255` 的 bail 不触发→`prunedPartitionPaths` 空→`resolvePartitions:587` 返回空→扫 0 分区→**对真实存在的值返回 0 行**。 +- **对比 legacy**:`HudiExternalMetaCache.loadPartitionNames:231` 对 HMS 名做 `FileUtils::unescapePathName`,再按 typed Nereids key 剪枝。故这是**回归**,非 parity。 +- **P7 是否已修**:否。P7 的 unescape 修复落在**另一个**方法 `HudiScanPlanProvider.parsePartitionValues:723-737`(scan 侧值保真),`HudiPartitionValuesTest` 覆盖的是它;**剪枝决策用的 `parsePartitionName` 未改、未测**(`HudiPartitionPruningTest` 只用干净的 `year=2024`)。 +- **触发面**:仅当分区值含 Hive 转义字符(时间戳、含空格/特殊字符的字符串);纯日期/整数不受影响。但触发时是**静默数据丢失**,无报错。 +- **解决方案**:在 `parsePartitionName` 存值前 unescape(复用连接器已有的 `HudiScanPlanProvider.unescapePathName` 或 Hive `FileUtils.unescapePathName`),对齐 legacy。补 `HudiPartitionPruningTest`:HMS 名 `ts=2024-01-01 10%3A00%3A00` + 谓词 `ts='2024-01-01 10:00:00'` 断言命中而非剪掉。**首选在 §4 的共享 helper 里修一处**。 + +### H2 — hudi datetime 分区谓词 ISO 化 → 整表剪到 0 行 + +- **现证据**:`ExprToConnectorExpressionConverter.convertDateLiteral:309-322` 把非 DATE 的 datetime 字面量包成 `LocalDateTime`;`extractLiteralValue:1030-1036` 用 `String.valueOf(val)` = `LocalDateTime.toString()`,对 `2024-01-01 10:00:00` 产出 `2024-01-01T10:00`(ISO 用 `T` 分隔、**省略零秒**)。`matchesPredicates` 纯字符串比 HMS 值文本 `2024-01-01 10:00:00`(空格分隔、全秒)→**永不相等→每个分区被剪→0 行**。DATE 列安全(`LocalDate.toString()`=`2024-01-01` 匹配)。 +- **对比 legacy**:hudi 分区走 Nereids typed 剪枝(`HudiExternalMetaCache:205` 喂 typed 列类型),不做字符串比,故不受影响。**回归**。 +- **P7 是否已修**:否。 +- **解决方案**:对时间类型不做字符串剪枝——`extractLiteralValue` 把 `LocalDateTime` 按 Hive 规范分区文本(空格分隔、全 `HH:mm:ss`)渲染;更对齐 legacy 的是让 `matchesPredicates` type-aware(两侧过列类型),或对时间列干脆退回 fe-core 的 typed Nereids 剪枝。补 DATETIME 分区列 + `= '2024-01-01 10:00:00'` 的剪枝测试。 +- 注:此条的对抗证伪 agent 未完成(schema 重试超限),但机制确定、与 H1/H3 同源且二者已 CONFIRMED,判定保留 🔴高。 + +### H3 — hudi 把 HMS 名当存储路径喂 fsView → 非 hive-style 表带 filter 0 split + +- **现证据**:`applyFilter:244-267` **无条件**把 `prunedPartitionPaths` 设为 `hmsClient.listPartitionNames` 的子集(hive-style 名 `year=2024/month=01`);`resolvePartitions:587-590` 原样返回,`collectCowSplits:394`/`collectMorSplits:429` 把它当**第一参**传给 `fsView.getLatestBaseFilesBeforeOrOn(partitionPath,...)`,而 FileSystemView 按 **Hudi 相对存储路径** 索引。对 `hive_style_partitioning=false`(Hudi 默认)的表,物理布局是 `2024/01` 而 HMS 名是 `year=2024/month=01`→fsView 找不到文件→0 split;**不带 filter 时** `resolvePartitions` 回退 `listAllPartitionPaths`(Hudi 元数据,正确)→有行。即报告的"带 filter 0 split、不带 filter 有行"。 +- **对比 legacy**:默认 `use_hive_sync_partition=false` 从 Hudi 元数据 `getAllPartitionNames` 取相对路径;hive-sync 分支也经 `FSUtils.getRelativePartitionPath` 相对化(`HudiScanNode:410-411`)。**回归**。连接器**自己的**列举路 `collectPartitions:636-665` 是 `use_hive_sync` 感知的,但 `applyFilter` 绕过了它。`HudiScanPlanProvider:716-721` 的 javadoc 自称此坑"翻闸前已闭合"是**过时的**。 +- **P7 是否已修**:否(HD-A3 只修了分区**值**保真 + 列举路)。 +- **解决方案**:让 `applyFilter` 的候选分区源 `use_hive_sync_partition` 感知,复用 `collectPartitions`:`!useHiveSyncPartition` 时对 `listAllPartitionPaths`(相对路径 `2024/01`)剪枝;hive-sync 时用 `FSUtils.getRelativePartitionPath` 相对化 HMS LOCATION。补非 hive-style 表断言"带 filter 分区集 == 不带 filter"。 + +### H4 — hudi 混大小写 Avro 列名崩 JNI/MOR reader + +- **现证据**:两处 lowercase 修的是**别的**路径——Doris 列 schema(`HudiConnectorMetadata.avroSchemaToColumns:905` `.toLowerCase(ROOT)`)和 native `history_schema_info` 字典(`HudiSchemaUtils.buildField:137`)。但 JNI reader 的列名列表**仍是原始大小写**:`HudiScanPlanProvider.planScan:180-181` `.map(Schema.Field::name)` 无 lowercase,经 `HudiScanRange:220` 塞进 `THudiFileDesc.column_names`。BE `HadoopHudiJniScanner.initRequiredColumnsAndTypes:212-229` 用**原始大小写** key 建 `hudiColNameToType`,再对每个 **lowercase** 的 `requiredField` 做精确 `containsKey`,不命中即 `throw IllegalArgumentException`——`Id/Name/Addr` 表下 lowercase `id` 不在 `{Id,Name,Addr}`→**每个 JNI split 抛**。 +- **对比 legacy**:`HMSExternalTable:749` 把 Avro 名 lowercase 进 Column,`HudiScanNode:223` `map(Column::getName)` 发 lowercase 名,匹配。**回归**。该 PR 自己的 `HudiSchemaParityTest`(用 `Id/Name/Addr`)只断言 `avroSchemaToColumns` 和 native 字典,**从不断言 JNI 列名列表**,抓不到。 +- **触发面**:混大小写列 **且** MOR-带-log 或 `force_jni`;纯 COW / 全小写不受影响。但触发时**每个 split 硬崩→整查询失败**。 +- **解决方案**:`HudiScanPlanProvider.planScan:181` 把 `.map(Schema.Field::name)` 改为 `.map(f -> f.name().toLowerCase(Locale.ROOT))`(与 `:905`/`HudiSchemaUtils:137` 一致);列类型顺序不变仍对齐;Hive ObjectInspector 大小写不敏感,文件读仍解析。给 `HudiSchemaParityTest` 补 JNI 列名断言。 + +--- + +## 3. 🟠 中危(8 条) + +### M1 — P1-2 `TABLESAMPLE` 在翻闸 hive 上静默全表扫(新激活) + +- **现证据**:`PhysicalPlanTranslator.visitPhysicalFileScan:812` **先**匹配 `PluginDrivenExternalTable`,只 `setSelectedPartitions:820`,**不转发 `getTableSample()`**(唯一转发在 legacy hive 臂 `:837-840`,翻闸后死)。`PluginDrivenScanNode.getSplits:998-1019` 从 connector ranges 1:1 建 split,**零** tableSample 引用;`ConnectorScanPlanProvider.planScan:119` 也无采样参数。样本已由 `BindRelation:812-816` 带进 `LogicalFileScan`,故 `SELECT ... TABLESAMPLE(...)` **静默丢弃**(非报错)→返回全表基数。 +- **对比**:翻闸前 hive 走 `HiveScanNode`(`:332,447-463`)会应用采样。无登记偏差(recon 确认 TABLESAMPLE 不在 deviations/decisions)。现有 `test_hive_tablesample_p0.groovy` 只断言 EXPLAIN 子串,抓不到。 +- **解决方案**:(1)translator 插件臂在 `setSelectedPartitions` 后镜像 legacy:`if (fileScan.getTableSample().isPresent()) pluginScanNode.setTableSample(...)`;(2)在 `PluginDrivenScanNode.getSplits` 实现**通用、connector-agnostic** 的按 split 大小采样(仿 `HiveScanNode:448-458`,对通用 `PluginDrivenSplit` 大小操作,不按源分支)。强化 groovy 断言采样后基数。 + +### M2 — P4-batch-hive:翻闸 hive 完全丢批量/异步 split(新激活) + +- **现证据**:翻闸后 hive 表全走 `PluginDrivenScanNode`。`computeBatchMode:1085-1113` 需连接器 opt-in:`streamingSplitEstimate`(仅 iceberg override)或 `supportsBatchScan`(仅 MaxCompute override)。`HiveScanPlanProvider` 二者都不 override(javadoc `:62` 自称"No batch/lazy split mode"),继承 `ConnectorScanPlanProvider.supportsBatchScan=false`(`:231`)→`shouldUseBatchMode` 恒 false→**同步 `getSplits`**,把所有选中分区的所有文件 split 一次性物化进 FE 堆。 +- **对比 legacy**:`HiveScanNode.isBatchMode:283-294` 纯按 `prunedPartitions.size() >= num_partitions_in_batch_mode`(默认 1024)启用异步 `startSplit` streaming,**无 opt-in gate**。hive 是实际最大分区源→FE 堆/延迟影响潜在最大。fail-safe(结果正确)。D2 连接器缓存只恢复列举**性能**,不恢复异步/streaming split。 +- **解决方案**:给 hive 连接器加 batch 通路——override `HiveScanPlanProvider.supportsBatchScan`(分区表 true)+ `planScanForPartitionBatch`,仿 `MaxComputeScanPlanProvider`,无需改 fe-core。或明确登记为验收偏差 + 大分区 e2e 真值门(**当前既未修也未登记**)。 + +### M3 — P4-1:MaxCompute batch 闸门 `!=NOT_PRUNED`→`!isPruned`(仍然) + +- **现证据**:`PluginDrivenScanNode.shouldUseBatchMode:1136` 是 `if (selectedPartitions == null || !selectedPartitions.isPruned) return false;`;legacy `MaxComputeScanNode:227` 是 `!= NOT_PRUNED`。二者**不等价**:无谓词分区表 `ExternalTable.initSelectedPartitions:447` 返回 `SelectedPartitions(size, fullMap, isPruned=false)`——非 `NOT_PRUNED` 哨兵。legacy→batch;SPI→`!false`→返回 false→**无 batch**。`MaxComputeScanPlanProvider.supportsBatchScan:254` opt-in(`getFileNum()>0`),默认 `num_partitions_in_batch_mode=1024`。故 `SELECT col FROM odps_t`(≥1024 分区、无谓词)走同步全枚举,FE 规划延迟+堆压。行内 `:1129-1132` 注释自称 parity **是错的**(只覆盖非分区 `NOT_PRUNED` 情形);同作者在 `displayPartitionCounts:297` 却正确用 `!= NOT_PRUNED`。 +- **范围澄清**:**仅 MaxCompute**(唯一 opt-in `supportsBatchScan`);iceberg 走 streaming flavor 不看 `isPruned`;hive 是**另一根因**(M2)。DV-019/D-035 只登记了 wiring/test-gap 并**误称 parity**,该语义分歧未登记。 +- **解决方案**:`:1136` 改回 `selectedPartitions == SelectedPartitions.NOT_PRUNED`(非分区表仍 →false;无谓词分区表落到 `:1145` 的 size≥阈值 检查),对齐 legacy;补纯 helper 单测覆盖 `(isPruned=false, 非 NOT_PRUNED, size≥阈值)`;订正 `:1129-1132` 注释。 + +### M4 — P4-2:MaxCompute 分区值缓存删除(仍然) + +- **现证据**:`PluginDrivenExternalTable.getNameToPartitionItems:780-781` 每次无条件 `metadata.listPartitions(...)`(注释 `:776-779` 自认"no FE-side partition-value cache");连接器 `MaxComputeConnectorMetadata.listPartitions:256-273`→`McStructureHelper.getPartitions:112-118` = 裸 ODPS SDK 调用(javadoc `:253-254` 自认"no connector-side cache")。FE、连接器两层**都无缓存**。`initSelectedPartitions:439-447` 每次规划调一次→**每查询一次全量 ODPS `getPartitions()`**。 +- **对比 legacy**:`MaxComputeExternalMetaCache` 的 TTL Caffeine `partitionValuesEntry`,重复查询零 ODPS 往返。数万分区表每次规划多秒 + 限流风险。correctness 安全(BE 重过滤)。**hive/hms 不受此累**——P7 给 hive 加了连接器自有缓存(`CachingHmsClient`,commit `f742651990d`);唯 maxcompute 显式弃缓存。 +- **解决方案**:在 maxcompute 连接器内加 TTL/容量 Caffeine(仿已落地的 `CachingHmsClient`/`HiveFileListingCache`),keyed `(project,db,table)`,`REFRESH` 失效——**不**在 fe-core 加二级缓存(违背 connector-owned 方向)。若延后,须在 deviations-log 正式登记(CACHE-P1 从未登记)。 + +### M5 — P6-3:iceberg `computeRowCount` 丢 equality-delete gate(仍然;iceberg-on-HMS 新激活) + +- **现证据**:`IcebergConnectorMetadata.computeRowCount:631-646` 只读 `total-records`、`total-position-deletes`,无条件返回 `totalRecords - positionDeletes`,无 equality-delete gate(类只声明 `TOTAL_RECORDS`/`TOTAL_POSITION_DELETES`)。其 javadoc(`:624-629`)与单测 `equalityDeletesDoNotGateTableStatistics:195-210` 断言"legacy 不 gate"——**此前提为假**:legacy `IcebergUtils.getIcebergRowCount:1202`→`getCountFromSummary`,`:231-233` `if (!equalityDeletes.equals("0")) return UNKNOWN_ROW_COUNT`。COUNT(*) 下推副本 `IcebergScanPlanProvider.getCountFromSummary:1572-1574` **保留**了 gate→**不对称**:有 equality-delete 时 COUNT(*) 正确退让,但 `getTableStatistics` 报膨胀行数→误导 CBO。 +- **翻闸影响**:type=iceberg 自 P6 已在 SPI(故"仍然");但翻闸让 **iceberg-on-HMS** 也走 `computeRowCount`(legacy HMS 路 `HMSExternalTable:547` 走带 gate 的 `getIcebergRowCount`)→对 iceberg-on-HMS 是**新激活**。非登记偏差。 +- **解决方案**:加 `TOTAL_EQUALITY_DELETES` 常量,`computeRowCount` 在减法前 `null || !="0"→返回 -1(UNKNOWN)`,复现 legacy;订正 javadoc + 重写把回归锁死的单测(应断言 UNKNOWN)。 + +### M6 — P6-4:iceberg s3tables 无显式凭证即硬失败(仍然) + +- **现证据**:`IcebergConnector.createS3TablesCatalog:735-739` `if (!chosenS3.isPresent()) throw ...`;`chosenS3` 来自 `S3FileSystemProvider.supports:64-74`,要求 `hasCredential(AK/SK||roleARN||credentials_provider_type) && hasLocation`(除非 explicit-S3)。EC2 instance-profile s3tables 目录(仅 `s3.region`+warehouse ARN,无静态凭证/无 marker)→`supports()=false`→无 S3 storage→`chosenS3` 空→**建表首访抛**。 +- **对比 legacy**:`IcebergS3TablesMetaStoreProperties` 直接 `S3Properties.of(origProps)`(无凭证 gate)+ `createAwsCredentialsProvider(...,false)`→无静态凭证时返回 `DefaultCredentialsProvider`(instance-profile 默认链)→region+warehouse-only 可用。现 `supports()` 比报告**更松**(接受 `credentials_provider_type`),故可用 `s3.credentials_provider_type` 绕过,但纯默认链场景仍硬失败。**回归**,fail-loud,非登记。 +- **解决方案**:`createS3TablesCatalog` 在 `chosenS3` 空但能从原始 props 解析出 region 时(复用 M7 拓宽的别名),用 `DefaultCredentialsProvider`+`Region.of(...)` 建 client 而非抛,镜像 legacy;仅当既无 storage 又无 region 才 fail-loud。 + +### M7 — P6-5:iceberg REST vended-cred region 别名收窄(仍然) + +- **现证据**:`IcebergCatalogFactory.java:83` `S3_REGION_ALIASES = {s3.region, aws.region, region, client.region}`;`appendS3FileIO:187-194` 在 `chosenS3` 空的臂上以 `firstNonBlank(props, S3_REGION_ALIASES)` 为**唯一** region 源。REST vended-cred 目录(无静态 AK/SK→`chosenS3` 空)若 region 仅经 `AWS_REGION`/`iceberg.rest.signing-region`/`rest.signing-region` 提供→`firstNonBlank` 返回 null→`CLIENT_REGION` 不发→iceberg S3FileIO 落 `DefaultAwsRegionProviderChain`→**"Unable to load region"**。javadoc `:78-83` 自称"mirror legacy `getRegionFromProperties`"**不实**:legacy 扫所有 `@ConnectorProperty(isRegionField=true)` 别名(S3 就 10 个,含上述被丢的)。 +- **解决方案**:拓宽 `S3_REGION_ALIASES` 对齐 legacy `getRegionFromProperties`(至少加 `AWS_REGION`/`iceberg.rest.signing-region`/`rest.signing-region`,以及 `REGION`/`glue.region`/`aws.glue.region`);fe-connector 不能 import fe-core,连接器侧复制别名列表(与 ENDPOINT 列表同模式)。订正 `:78-83` 注释。 + +### M8 — P6-2:升级只换 `lib/` 不部署 `plugins/connector`(仍然;hms 放大) + +- **现证据**:`PluginDrivenExternalCatalog.java:135` `throw new RuntimeException("No ConnectorProvider found...")`——连接器插件未部署时首访即抛。`CatalogFactory` 的 degraded 分支(`:119-127`)**只护 replay/启动**(注册 null-connector 目录,启动不抛),外部目录懒初始化→抛在**首查**。built-in fallback switch(`:136-157`)已删 `case "hms"`/`case "iceberg"`。 +- **翻闸放大**:报告 tip 时只 iceberg 暴露(hms 仍走 legacy `case "hms"`)。翻闸把 `"hms"` 入 `SPI_READY_TYPES` 并删 legacy fallback→**所有** type=hms 目录(plain-hive+iceberg-on-HMS+hudi-on-HMS)首查即抛,hms/hive 是实际主力外部目录类型→lib-only 升级几乎击穿全部外部目录访问。 +- **性质**:对抗验证把它从"高"降为"设计/运营"——因为它是**有意的 fail-loud**(消息明确、启动有 per-catalog WARN、无静默/无数据损坏),正确升级(部署 `plugins/connector`)零问题。但 blast radius 因翻闸显著扩大,仍按 🟠中(运营)排。 +- **解决方案**:主线是**发布/升级工具**必须把连接器 jar 部署到 `connector_plugin_root`(build.sh/部署步骤)+ 响亮 release note"post-cutover 不支持 lib-only 换包"。代码侧防御(可选):replay 结束后聚合输出一条 ERROR 枚举所有 degraded(connector==null)目录,让运维在启动而非首查时发现。**保留** first-access throw,**不**加 legacy fallback(那会倒退翻闸)。 + +--- + +## 4. ⚪ 设计债 / 架构趋势(真实但按现行规则非"须修 bug";逐条已核实存在) + +> 铁律提醒:fe-core 有意 connector-agnostic + 不解析属性(用户拍板),故"SPI 表面长出连接器词汇"是**已知设计张力**,记为设计债。以下均已对 HEAD 核实为真。 + +| ID | 现状 | 要点(file:line) | 建议 | +|---|---|---|---| +| **DUPLICATION:partition-prune** | **新激活·承载 H1–H3** | `HiveConnectorMetadata:1995-2093` 与 `HudiConnectorMetadata:980-1078` 的 7 方法 EQ/IN 剪枝块**逐字节相同**(仅末尾 `}` 不同),现两份都在生产路径 | **最高优先 de-dup**:抽 `HmsStylePartitionPruner` 到共享模块,H1–H3 的 unescape/相对化修复一处落地,防"修一漏一"连接器专属分歧 | +| P4-D / 5.3 引擎名 switch | 仍然·翻闸增至 4 case | 三处 fe-core 字符串 switch 手动同步:`PluginDrivenExternalTable.getEngine:1182`、`getEngineTableTypeName:1220`、`CreateTableInfo.pluginCatalogTypeToEngine:927`;javadoc 自认"两个 switch 须同步" | 引 `Connector.getLegacyEngineName()`/`getLegacyTableTypeName()` SPI,连接器声明一次,删三处 switch;或共享静态表 | +| P6-S(a) SHOW CREATE 无脱敏 | 仍然(安全) | `Env.java:4897-4907` 插件 PROPERTIES 逐字 append,无 `DatasourcePrintableMap`/SENSITIVE_KEY masking,仅靠 capability gate;`ConnectorCapability:80-82` 自认 flag 是唯一防线 | ~5 行:该处改走 `new DatasourcePrintableMap<>(props," = ",true,true,hidePassword)`,把 javadoc 握手变纵深防御。**注**:当前无可达泄漏(hive 未声明该 cap,SHOW CREATE 只出注释;paimon/iceberg 只渲染表级 option) | +| P6-S(b) 敏感键硬编码 fe-core | 仍然(安全) | `DatasourcePrintableMap.SENSITIVE_KEY` 硬列 MC/iceberg-REST/glue/dlf 键(`:57-75`,"Keep in sync"注释) | 折进已有的 `registerSensitiveKeys(...)` SPI 聚合(filesystem 已用);至少把 iceberg REST 键移到连接器注册。**masking 当前完好**,翻闸后 hms 秘密不泄 | +| P6-D1 RowLevelDmlRegistry iceberg 形 | 仍然(潜伏) | `RowLevelDmlRegistry:37` 仅 `IcebergRowLevelDmlTransform`;gate 泛化(capability)但 body 硬绑 `__DORIS_ICEBERG_ROWID_COL__`。**hive 未触发**(仅 iceberg 声明 DELETE/MERGE;hive `WritePlanProvider:124` 只 INSERT/OVERWRITE) | 加 fail-loud 契约检查(非 iceberg 连接器声明 DELETE/MERGE 时拒绝),刷新过时 javadoc;第二个 row-level-DML 连接器落地前登记验收 | +| 4.2 stringly-typed 契约 | 仍然 | `ConnectorPartitionField.transform:37`(String,未知静默 CUSTOM)、`transformArgs:38`(仅 `List`)、`ConnectorBucketSpec.algorithm:39`(free string) | 硬化时换 enum+CUSTOM(String)、拓宽 args;近期至少共享常量类 | +| 4.3 SPI 连接器化 | 仍然·翻闸轻微放大 | `supportsWriteBlockAllocation/allocateWriteBlockRange`(仅 MC)、`deriveStorageProperties`(仅 iceberg)、7 个 branch/tag/partition-field 方法(iceberg 形)、`cleanupEmptyManagedLocation`(iceberg 单消费者)。**订正报告**:后者的 `["data","metadata"]` 在**连接器侧**(`IcebergSchemaBuilder:71`),fe-core 只收参数,未硬编码 iceberg 目录 | 连接器专属能力收进 capability-discovered facet(如 `getProcedureOps()` 范式);登记 4.3 为验收架构张力 | +| P6-D2 millis/micros 命名 | 仍然 | `ConnectorMvccPartitionView.getNewestUpdateTimeMillis:113` 名说毫秒实微秒(iceberg `last_updated_at`);javadoc 已警示 | 廉价:重命名 `getNewestUpdateMarker()`/`...Raw()`,单实现者+单消费者,零行为变更 | +| P6-D3 raw-path 字符串契约 | 仍然 | `ConnectorMetadata.applyRewriteFileScope:204`/`applyTopnLazyMaterialization:226` 的相等/全 schema 契约只活在 javadoc,违反=**静默数据损坏**非报错;单消费者 iceberg | 引 `ConnectorFilePath` 不透明 token 令相等 by-construction;debug 断言全 schema 覆盖;或登记验收 | +| MAGICKEY:channel | 仍然(未被翻闸激活) | SHOW CREATE 子句经保留魔法键传**连接器预渲染的 Doris SQL 文本**(`IcebergConnectorMetadata:458-502` 渲 `` BUCKET(N,`c`) `` 等),fe-core 逐字 append 再按名 strip。**部分改进**:`show.*` 现是声明常量,但 `partition_columns`/`primary_keys` 仍裸字面量 | 换结构化 `ConnectorTableDdl`(transform+sort 作数据),fe-core 单一 altitude 渲染;至少把两裸键提为常量 | +| P6-D4:tccl 复制 | 仍然 | `TcclPinningConnectorContext` 仍 iceberg/paimon 两份(~73% 同);hive **未加第 3 份**但在 `HiveConnectorMetadata:798-805,832-845` **内联** open-code 同一 TCCL pin(第 3 种形态) | 抽 `Tccl.pin(loader,Runnable)` 到 spi/support,三处共用 | +| P5-D:hiveconf 复制 | 仍然 | `assembleHiveConf`/`buildHadoopConfiguration` 仍 iceberg/paimon 两份;hive 未加 3rd,但**连接器内** `buildHadoopConf` 三重复(`HiveConnector:621`/`HiveScanPlanProvider:444`/`HiveConnectorMetadata:967`) | 合并 hive 三处为一私有 helper;长期共享 `HadoopConfBuilder` | +| DUPLICATION:fakestub | 仍然·翻闸恶化 | `Fake*HmsClient` 测试桩从 ~3 增至 ~11 份(hive/hudi 各测各造) | 加共享 `AbstractFakeHmsClient` 测试夹具 | +| P2-3 preCreateValidation 急切 | 仍然 | `TrinoDorisConnector:78` CREATE CATALOG 时急切加载插件+构造连接器(legacy 惰性);replay 跳校验 | 视为有意 fail-fast,**登记 deviations-log**(当前未登记);或改 best-effort | +| P4-SHOWPART:admission | 仍然 | `ShowPartitionsCommand:202-206` admit 所有 `PluginDrivenExternalCatalog`;jdbc/es/trino 错误类别变化 | 无需改;已登记决策 D-028 | +| P5-6 convertAnd sound | 仍然 | `PaimonPredicateConverter:121-130` OR 下保留部分合取;**sound**(超集,BE 重过滤),甚至比 legacy 剪得多 | 无需修;可注释登记免复审再标 | +| P6-D4:flavor-provider | **部分已修** | flavor wrapper 复制经共享 `Abstract*MetaStoreProperties` 基类大幅收敛,余薄子类矩阵;hms 复用共享基类 | 无需动;此行 5.4 是**改善**非恶化 | +| P8 前置(5 项) | 仍然·翻闸增 | `SPI_READY_TYPES` 字符串门(`CatalogFactory:54`)、`getEngine` switch、data-cache allowlist(`FileQueryScanNode:112` 已含 "hms",**hive 数据缓存正常**)、`TableType.ICEBERG_EXTERNAL_TABLE`(8 处/4 文件,活+死混合)、`PlanNode.printNestedColumns instanceof IcebergScanNode`(见 §6 死码) | 均为 **P8 删除前置**,非 bug;P8 换连接器声明属性/按注册路由。**当前对 hive 均无错误行为** | + +--- + +## 5. 🟡 已登记 / 验收偏差 / benign(真实但已决策或无害,列此避免误改) + +| ID | 现状 | 要点 | 处置 | +|---|---|---|---| +| P4-5 | 仍然(hive 新激活) | `PluginDrivenInsertExecutor.doAfterCommit:167-175` 吞 post-commit refresh 失败,INSERT 报成功 | **DV-018 已签字**(数据已提交,报失败会诱发重复写);翻闸后 hive INSERT 继承同行为 | +| P4-6 | 仍然(hive 新激活) | `createDb/dropDb` 丢 errno 1007/1008(createTable 恢复了 1050) | **DV-034/DV-021 GAP3 已登记**;修法 = 仿 createTable 加 `ErrorReport.reportDdlException` | +| P4-SHOWPART:limit | 新激活 | 翻闸 hive SHOW PARTITIONS 丢远端 LIMIT 下推(全取-FE 排序-分页) | **DV-021 GAP9 已登记**;升序结果等价,仅多元数据 payload | +| P4-SHOWPART:where | 新激活 | 翻闸 hive SHOW PARTITIONS 静默接受非 PartitionName WHERE/ORDER(legacy 拒绝) | 已登记为 plugin-model parity(对齐 paimon/iceberg);比 legacy hive 更松 | +| P6-7 | 仍然 | `FOR TIME AS OF '<数字串>'` 读作 epoch millis(legacy 拒) | benign 超集,javadoc 已声明有意;可补登记 | +| P6-8 | 仍然 | 无 ctx `resolveSessionZone` 回退 UTC vs legacy FE 默认 tz | **无可达无 ctx 时间旅行路径**(时旅恒有 ConnectContext);latent | +| P6-10 | 仍然 | `$position_deletes` 丢定向错误文案,变通用 not-found | **DV-049③ 已签字**,reg-test 已同步;语义等价 | + +--- + +## 6. ✅ 报告条目在当前分支**不成立 / 已修 / parity**(核实要点) + +> 用户特别关注"哪些结论不适用于当前分支"。以下逐条为报告发现,但**当前分支不构成需修问题**——或被 P7 修掉,或被对抗证伪,或与 master parity/属死码。**不要据此改代码**。 + +| ID | 判定 | 为什么不成立(要点) | +|---|---|---| +| P0-1 | parity | 翻闸 hive 全支持 CREATE/DROP;它拒绝的 ALTER 在 legacy 与新路都无 catalog 名(仅措辞变);errno 另登记 DV-034 | +| P0-2 | parity(不可达) | `readBucketNum` 的 `translateToCatalogStyle().getBuckets()` 证明不抛→catch 是死防御码,silent-0 不可达 | +| P0-3 | parity(不可达) | `convertFields` else-drop 对 hive(identity)/iceberg(transform)分区不可达;grammar 只产 slot/transform | +| P0-4 | 不适用 | hive/iceberg/paimon 均 override 完整 `createTable(request)`,降级默认不可达 | +| P0-6(stats) | 不适用 | `invalidateStatistics` no-op **零生产调用**;事件走 `MetastoreEventSyncDriver`(中立 CatalogMgr/RefreshManager),不经 `ConnectorMetaInvalidator`;legacy 事件也从不失效统计 | +| P0-6(part) | parity(有意) | 活的分区事件路是 `HiveConnector.invalidatePartition`(整表 flush,pull-based 正确),报告引的是死方法;已登记 | +| P1-1 | 不适用 | 翻闸 hudi 绑 `LogicalFileScan`→`visitPhysicalFileScan:812` 建 `PluginDrivenScanNode` 并 `setSelectedPartitions:820`+`setDistributeExprLists:866`;`visitPhysicalHudiScan` 插件臂是**死码**,分区剪枝/bucket-shuffle 已 parity | +| P1-3(iceberg) | 不适用 | `PlanNode.printNestedColumns instanceof IcebergScanNode` 对翻闸 iceberg 是死码(已 `PluginDrivenScanNode:377` 注释 + FU-h10-deadcode 跟踪),纯 cosmetic,与 hive 无关 | +| **P3-hudi COW/MOR UNKNOWN** | **已修** | **HD-A4(`cf8710691cf`)**:`planScan:140` `isCow = metaClient.getTableType()==COPY_ON_WRITE` 从**权威** Hudi 配置取读路径;`detectHudiTableType` 的 UNKNOWN 启发式仅用于 per-split 覆盖的 node 默认 + 信息属性,不再选错读路径→陈旧行风险消除 | +| **P3b-1** | **证伪(parity)** | 已发布/长期基线本就是 `"hadoop"`+doAs(legacy fe-core `DFSFileSystem` 全经 `getHadoopAuthenticator().doAs`,默认 user "hadoop");master 的 process-user 只活在**未发布**的 fs-SPI 脚手架(#62023)且与 master 自身 metastore 侧不一致。#64655 consolidation 是**有意**统一。无升级回归 | +| **P3b-3** | **证伪(机制错)** | `UGI.setConfiguration`→`initialize` 本身是 `private static synchronized` on 单一 parent-loaded `UGI.class`(`org.apache.hadoop.` 在 FS 是 parent-first);外层 `synchronized(HadoopKerberosAuthenticator.class)` 冗余,拆成两 monitor 也不丢保护。连接器侧 `org.apache.hadoop.` 非 parent-first→各自隔离 UGI,无共享全局可损。双载 jar 真实但**无害** | +| P4-3-IN | 已修 | `convertIn:169-184` 方向正确(`col IN (...)`),legacy 反向 bug 已修;缺一条定向回归测试(建议补) | +| P4-3-EQ | 待实测 | 算子仍 `==`(`:145-146`);ODPS 是否容忍需 live A/B。建议直接对齐 `=` 消除不确定性(见 §L20) | +| P5-7 | parity | `getInitialValues()` 仅测试填充,`VALUES IN` 静默吞 = master 同款;死 API 面。新增 `hasExplicitPartitionValues` 仅 hive 消费,paimon 未 opt-in | +| P6-S:cap-enum | 不适用 | capability enum 破坏性重写是一次性迁移成本,已落地,无运行期缺陷 | +| ENGINE-SWITCH:i3-hive | 已修 | 三处 switch 翻闸都加了正确 `case "hms"`→SHOW TABLE STATUS 显 "hms" 非通用 "Plugin";I3 回归被规避。建议补一条断言锁 parity | + +--- + +## 7. 建议的行动清单 + +**翻闸后、更大范围放量前(高)** +1. **hudi 4 高危(H1–H4)**:翻闸即在 hudi-on-HMS 上静默丢行/崩。修复应在 §4 `DUPLICATION:partition-prune` 抽出的共享剪枝 helper 一处落地(H1/H3)+ H2 时间类型不做字符串比 + H4 `planScan:181` lowercase;每条补能抓到的回归测试(现有 parity 测试写法抓不到)。 +2. **hive/MC 批量丢失(M1/M2/M3)**:TABLESAMPLE 转发+通用采样;hive `supportsBatchScan` opt-in;MC `!=NOT_PRUNED` 复原。 +3. **iceberg 统计/凭证(M5/M6/M7)**:equality-delete gate、s3tables 默认链、region 别名。 +4. **升级坑(M8)**:发布工具部署 `plugins/connector` + release note + 启动聚合告警。 + +**贯穿性(建议开跟踪项,对齐 #65185)** +5. `Connector.getLegacyEngineName()` SPI 收口三处引擎名 switch(P4-D);EXPLAIN 节点名(P5-1)登记或按连接器声明 legacy 名。 +6. `HmsStylePartitionPruner` / `Tccl.pin` / `HadoopConfBuilder` / `AbstractFakeHmsClient` 共享化(§4 复制项)。 +7. SHOW CREATE 脱敏纵深(P6-S(a))+ 敏感键 SPI 聚合(P6-S(b))。 +8. import-gate 三洞补齐(P0-5);MC 分区缓存(M4)。 + +**待实测** +9. P4-3-EQ(`==` vs `=`)live ODPS A/B;或直接改 `=`。 +10. P5-2 option-unset ORC paimon-cpp e2e(默认 JNI reader 不受影响,窄)。 + +--- + +## 附录 A:方法与统计 + +- **工作流**:`wf_588a73bd-294`。3 recon(类定位图 / P7-翻闸修复台账 / 已登记偏差摘要)→ 34 finding-cluster 核实(按符号定位现码,交叉核对 deviations/decisions-log)→ 对每条存活的高/中发现派对抗证伪 agent(尽力 REFUTE:找 guard / master-parity / P7-fix / 不可达 / 登记偏差)。 +- **规模**:60 agent,58 完成,2 个对抗 agent schema 重试超限(丢失 H2 与 P6-6 的对抗判定;二者由同源已 CONFIRMED 的姐妹发现佐证,判定保留)。 +- **口径**:79 条结论。状态分布 — STILL_REAL 52 / NEWLY_ACTIVE 10 / NO_LONGER_APPLICABLE 6 / PARITY 5 / FIXED 3 / PARTIALLY_FIXED 2 / CANNOT_VERIFY 1。对抗判定 — CONFIRMED_REAL 10 / DOWNGRADED 9 / REFUTED 2。 +- **原始逐条证据**(含每条 file:line、失败场景、对抗推理)落 scratchpad `findings_detail.md`;工作流返回值落 `journal.jsonl`。 + +## 附录 B:与报告 §10 优先级的对照 + +报告 §10 的 4 个"合 master 前高危"在当前分支的落点: +- P6-1(缓存偏斜崩 BE):**部分已修**(T07/T08 原子 pin + 同 TTL + 原子失效),对抗降为低危结构隐患(触发极窄); +- P6-2(升级坑):**仍然**,blast radius 因翻闸放大(M8); +- P4-1(batch-mode):**仍然**(M3),另 hive 有独立 M2; +- P6-S(SHOW CREATE 脱敏):**仍然**(设计/安全,§4),当前无可达泄漏。 + +报告 §10"P7 前必查(dormant)"的 hudi 4 高危 → **已翻闸激活为 §2 的 H1–H4**(报告的预警准确兑现);TABLESAMPLE→M1;CacheAnalyzer instanceof→L2;kerberos 锁分裂→**证伪**(§6 P3b-3)。 diff --git a/plan-doc/reviews/catalog-spi-review-65185.md b/plan-doc/reviews/catalog-spi-review-65185.md new file mode 100644 index 00000000000000..42a1fb04ecb2ff --- /dev/null +++ b/plan-doc/reviews/catalog-spi-review-65185.md @@ -0,0 +1,422 @@ +# Apache Doris Catalog SPI 迁移 — 完整评审 + +> **范围**:Issue [#65185](https://github.com/apache/doris/issues/65185) 伞形跟踪的全部 9 个 PR(P0–P6),分支 `branch-catalog-spi` +> **基线**:所有结论对齐分支最新 tip `3ba75b7cf8a`,并与当前 `apache/master`(legacy 代码)逐一对照 +> **方法**:20 个独立 finder(按角度+子系统分片)→ 对 tip 复核 → 6 个对抗验证器(CONFIRMED/PLAUSIBLE/REFUTED)→ live 集群实测(paimon filesystem + iceberg hadoop catalog) +> **注**:9 个 PR 均已 MERGED。本文既是对已合入代码的记录性评审,也是给 P7(hive)/P8(清理)的风险清单。 + +--- + +## 目录 + +1. [执行摘要](#1-执行摘要) +2. [迁移做了什么(架构概览)](#2-迁移做了什么架构概览) +3. [架构评估](#3-架构评估) +4. [抽象评估](#4-抽象评估) +5. [扩展性评估](#5-扩展性评估) +6. [贯穿性正确性主题](#6-贯穿性正确性主题) +7. [安全观察](#7-安全观察) +8. [逐 PR 详细发现](#8-逐-pr-详细发现) +9. [验证方法与统计](#9-验证方法与统计) +10. [优先级建议](#10-优先级建议) +- [附录 A:被证伪的候选](#附录-a被证伪的候选) +- [附录 B:live 集群实测证据](#附录-b-live-集群实测证据) + +--- + +## 1. 执行摘要 + +这是一次**规矩、低单点风险、但把大量"期票"推给 P7/P8 兑现**的重构。 + +**整体判断**:SPI 形状直接对标 Trino,分层清晰,对已迁移连接器(jdbc/es)真正零影响,GSON 升级兼容(invariant #2)在 iceberg/paimon/maxcompute 上都验证干净,写路径/事务提交协议/10 个 iceberg procedure 逐字节忠实移植。**架构本身没有硬伤。** + +**真正的风险全是行为细节和结构债**,分三类: + +- **已确认的正确性回归**(对 master):P4 batch-mode 闸门塌陷(高)、P4 分区值缓存删除(每查询往返 ODPS)、P6 iceberg 缓存偏斜→BE DCHECK 崩溃(高,需特定 TTL 时序)、P6 equality-delete 行数不 gate、P6 s3tables 默认凭证链回归、P6 region 别名收窄。 +- **休眠的 P7 地雷**:P3 hudi plugin 路径全套 4 个高危 bug(分区值不 unescape、datetime ISO 匹配不上、HMS 名当存储路径、混大小写 Avro 崩 JNI)——今天不触发(hudi 不在 `SPI_READY_TYPES`),但 P3 建立的正是 P7 要信赖的"测试基线",而 parity 测试写法抓不到这些。 +- **运营/升级坑**:P6 升级只换 `lib/` 不部署 `plugins/connector` → 所有存量 iceberg catalog 首次访问即抛异常(invariant #2 的"透明迁移"只在完整部署时成立)。 + +**贯穿性设计问题**(比单个 bug 更值得 PMC 关注): +- EXPLAIN 节点名 `VPluginDrivenScanNode` 系统性违反 invariant #3("EXPLAIN output preserved"),且 `deviations-log` 无记录。 +- 三处 per-connector 字符串 switch(引擎名/表类型名)必须手改同步,无编译期保护。 +- SPI 表面正在"每迁移一个连接器就长出该连接器私有词汇"(MVCC 微秒/毫秒、iceberg 的 branch/tag 方法、odps 的 block 分配),偏离"中立 SPI"承诺。 +- `SUPPORTS_SHOW_CREATE_DDL` 无引擎侧脱敏——靠连接器作者自觉不泄密码。 + +--- + +## 2. 迁移做了什么(架构概览) + +**目标**:把 FE 里硬编码的外部数据源(hive/iceberg/paimon/hudi/trino-connector/maxcompute/jdbc/es)从 `fe-core` 解耦成 `fe/fe-connector/*` 下可独立加载的插件,`fe-core` 只保留通用设施和稳定的 catalog SPI 桥。终态:每个连接器一个自包含 zip;`fe-core` 无任何连接器的编译期知识。 + +**三条不变量(验收标准)**: +- **I1 — 单向依赖** `fe-connector → fe-core`;连接器禁止 `import org.apache.doris.{catalog,common,datasource,qe,analysis,nereids,planner}`,CI grep 守门。 +- **I2 — 元数据向后兼容**:旧镜像里持久化的 `IcebergExternalCatalog` 等 GSON 类型必须透明反序列化+迁移到 `PluginDrivenExternalCatalog`。 +- **I3 — 用户可见行为零回归**:`SHOW CREATE CATALOG`、`information_schema`、EXPLAIN 输出、错误信息、catalog type/engine 名全保留。 + +**阶段**:P0(SPI 基线)→ P1(scan-node 路由收口)→ P2(trino,首个端到端样板)→ P3(hudi 加固,cutover 推迟到 P7)→ P3b(kerberos 收拢进 fe-kerberos)→ P4(maxcompute,首个删 legacy)→ P5+P5b(paimon 迁移+删 legacy)→ P6(iceberg,最大,7 flavor+MVCC+写路径+procedures)。P7(hive)/P8(删 allowlist)未做。 + +**核心桥接类**(全在 fe-core): +| 类 | 作用 | +|---|---| +| `PluginDrivenExternalCatalog extends ExternalCatalog` | catalog 层桥,路由 DDL/元数据到 `connector.getMetadata(session)` | +| `PluginDrivenExternalTable` / `PluginDrivenMvccExternalTable` | 表层桥,后者实现 `MvccTable`/`MTMVRelatedTableIf` | +| `PluginDrivenScanNode extends FileQueryScanNode` | 扫描节点,统一承载所有插件连接器的 scan | +| `ConnectorMvccSnapshotAdapter implements MvccSnapshot` | 把 SPI 快照套进引擎既有 MVCC pin 管线 | +| `PluginDrivenTransactionManager` | 双入口事务记账(legacy auto-commit + SPI ConnectorTransaction) | +| `CatalogFactory.SPI_READY_TYPES` | allowlist,决定哪些 catalog type 走插件路径(当前:jdbc/es/trino-connector/max_compute/paimon/iceberg) | + +--- + +## 3. 架构评估 + +### 3.1 SPI 分层与依赖方向 —— ✅ 做对了 + +- **血统清晰**:`ConnectorSession` / `ConnectorMetadata`(聚合 6 个细粒度 Ops 接口:Schema/Table/Pushdown/Statistics/Write/Identifier)/ `ConnectorTableHandle` / `applyFilter`/`applyProjection` handle-update 模式,直接对标 Trino SPI。行业验证过的形状,后来者好上手,增量迁移成本最低。 +- **全 default 方法**:连接器只 override 自己支持的能力,对已迁移的 jdbc/es 真正零影响(FakeConnectorPlugin 测试锁死了每个 default 的行为)。 +- **反向通道做对了**:`ConnectorMetaInvalidator` 经 `ConnectorContext` 下发,连接器回调引擎丢缓存而不 import 引擎;`ConnectorMvccSnapshotAdapter` 用适配器把 SPI 类型包进引擎既有 `MvccSnapshot`,fe-core 类型不泄漏进 SPI。 +- **守门存在但不完备**(见 [6.5](#65-升级兼容-invariant-2) 与 P0 发现):grep 黑名单,漏 static import、漏 `persist/transaction/fs/statistics` 等包、只扫 `src/main/java`。是"聊胜于无"而非"机器保证 I1"。 + +### 3.2 桥接层 —— ✅ 结构合理,但 `PluginDrivenScanNode` 在膨胀 + +`PluginDrivenScanNode` 从 P4(~200 行)到 P6(~1000+ 行)承载了所有连接器的两套 batch-scan 模型、MVCC pin、field-id dict、分区展示……它是"统一收口"的收益点,也是复杂度积聚点。P6 里 `hasSnapshotPin` 分支的 field-id dict 构建 + 独立 schema 缓存,正是 [6.4](#64-缓存一致性) 那个 BE 崩溃隐患的所在。**建议**:随着 P7 加入,考虑把 MVCC/dict 逻辑拆成可组合的 helper,别让这个类继续成为所有连接器 scan 复杂度的垃圾场。 + +### 3.3 类加载隔离模型 —— ⚠️ 有一处真实的锁失效 + +每个插件 zip 走 `ChildFirstClassLoader`,`FS_PARENT_FIRST_PREFIXES` 声明哪些包父加载优先,`TcclPinningConnectorContext` 在调用连接器 SDK 前 pin 线程上下文 classloader。 + +**隐患**(P3b-C5,CONFIRMED medium):`fe-filesystem-hdfs` 现在编译依赖 `fe-kerberos`,而 plugin-zip 的 dependencySet 没排除它(违反自己"fe-core classpath 已有的 jar 要排除"的策略,fe-core 也 ship fe-kerberos)。`FS_PARENT_FIRST_PREFIXES` 列了 `org.apache.hadoop.` 但**没列 `org.apache.doris.kerberos.`** → 目录加载的 HDFS 插件拿到 child-first 的第二份 `HadoopKerberosAuthenticator`,其 `synchronized (HadoopKerberosAuthenticator.class)` 类锁与 fe-core 那份**不互斥**,而两份都在改同一个父加载的 `UserGroupInformation` 全局状态。修复一行:把 fe-kerberos 从 plugin lib/ 排除,或把前缀加进父加载优先列表。 + +--- + +## 4. 抽象评估 + +### 4.1 抽象得好的地方 + +- **数据面演进友好**:请求/结果全用不可变 POJO + builder(`ConnectorCreateTableRequest` 等),加字段=加 setter 不破签名;`ConnectorPartitionInfo` 加统计字段用"哨兵 UNKNOWN + 保留旧构造器"。 +- **向后兼容内建在默认实现里**:新旧 `createTable` 的降级链、`beginTransaction` 默认 throw = 引擎按 auto-commit 处理。 + +### 4.2 Stringly-typed 契约 —— ⚠️ 零编译期保护 + +- partition transform 是字符串(`"year"`/`"bucket"`,不认识的**静默当 CUSTOM**); +- bucket 算法是字符串(`"doris_default"`/`"hive_hash"` 占位); +- `ConnectorPartitionField.transformArgs` 只能 `List`(paimon 的非整数 transform 参数放不进); +- 契约活在 javadoc 和 RFC 附录里,typo 直接变语义。 + +### 4.3 SPI 表面正在"连接器化" —— ⚠️ 这是最该管的抽象趋势 + +**核心观察:每迁移一个连接器,就往"中立" SPI 上长出该连接器的私有词汇。** + +| SPI 成员 | 归属连接器 | 问题 | +|---|---|---| +| `ConnectorTransaction.supportsWriteBlockAllocation()` / `allocateWriteBlockRange()` | maxcompute(ODPS tunnel block-id) | 其他连接器全无用,javadoc 自认"e.g. maxcompute" | +| `Connector.deriveStorageProperties()` | iceberg(hadoop-catalog warehouse→fs.defaultFS) | 唯一 override 是 iceberg | +| `ConnectorContext.cleanupEmptyManagedLocation(location, tableChildDirs)` | iceberg | 硬编码 iceberg `["data","metadata"]` 目录布局 | +| `ConnectorTableOps` 的 7 个 branch/tag/partition-field 方法 | iceberg | `PartitionFieldChange` 建模 iceberg 的 `transformName + Integer transformArg`,hive/jdbc/es 永远实现不了 | +| `ConnectorMvccPartitionView.getNewestUpdateTimeMillis()` | iceberg | 名字说毫秒,javadoc 说是**微秒**(源定义,只依赖单调性)——下一个 MVCC 连接器按名字返回真毫秒就混单位 | + +**历史教训已经发生**:P0 定义的 MVCC 三方法 `beginQuerySnapshot`/`getSnapshotAt`/`getSnapshotById`,到 P6 iceberg 真接入时被推翻重塑成 `resolveTimeTravel(spec)`/`applySnapshot(handle, snapshot)`——三分之二没活过第一个真实用户。这印证了 foundation-first 的固有风险:**没有 consumer 的 API 设计活不过第一个真实用户**,所以 P0 定义的每个 SPI 面都得到对应 consumer PR 里回头验证。 + +**建议**:把连接器专属能力收进可选 facet 接口(像 `getProcedureOps()` 那样能力发现),而不是堆在根接口当 default 方法。否则到 P8,"中立" SPI 会是每个 format 私有词汇的并集,实现者分不清 ~40 个 default 里哪些对自己是 load-bearing。 + +### 4.4 Magic-key 通道 —— ⚠️ 把 Doris SQL 方言塞进字符串 map + +SHOW CREATE TABLE 的子句通过 table-properties map 的保留魔法键传递:`show.location` / `show.partition-clause` / `show.sort-clause`——**连接器预渲染 Doris SQL 文本**(`PARTITION BY ... BUCKET(8, \`c\`)`),fe-core 逐字 append 后再按名字 strip。外加 `partition_columns` / `primary_keys` 两个 undeclared 魔法键。 + +问题:①每个连接器都得内嵌 Doris 的 SQL 方言/引号规则(altitude 倒置);②真名为 `show.location` 的用户属性会被误 strip;③连接器 jar 与 fe-core 现在是独立版本化的构件,Doris 改 SHOW CREATE 语法时,每个连接器预渲染的字符串就与引擎版本漂移。**更正确**:一个结构化的 `ConnectorTableDdl` 载体(partition transform + sort key 作为数据),由 fe-core 渲染,方言知识留在一个 altitude。 + +--- + +## 5. 扩展性评估 + +### 5.1 P7 hive 能零改动接入吗? —— ❌ 不能 + +发现若干"plugin-first 路由默认不转发,只 legacy hive 分支转发"的能力,P7 切换即静默失效: + +- **TABLESAMPLE**:`PluginDrivenScanNode` 完全无 table-sample 管线,SPI 无表达。`SELECT ... TABLESAMPLE(10 PERCENT)` 切换后静默全表扫(P1 发现)。 +- **SQL cache**:`CacheAnalyzer` 用 `instanceof HiveScanNode` 统计可缓存性(:305-325),hive 变 `PluginDrivenScanNode` 后 SQL cache 静默不匹配,`COUNTER_QUERY_HIVE_TABLE` 也停止递增。 +- **kerberos**:hive 是最重的 kerberos 用户(HMS + HDFS + metastore event poller),会把 [3.3](#33-类加载隔离模型--️-有一处真实的锁失效) 的锁分裂 + N 份独立登录/续期计时器放大。 +- **hudi-on-HMS 的 DLA dispatch**(P3 设计):一个 catalog 服务两种表格式,dispatch 是否泛化到 iceberg-on-HMS、P7 full hive,还是 hudi-shaped,值得在 P7 前定型。 + +### 5.2 P8(删 allowlist)不是纯删除 —— ⚠️ + +P6 把 flavor 知识从 `instanceof` 搬进了通用类里的**字符串 switch**,而非让它消失: + +- `PluginDrivenExternalTable.getEngine()` 硬编码 `case "iceberg"`; +- `FileQueryScanNode` 硬编码数据缓存 allowlist `Arrays.asList("hms","iceberg","paimon")`; +- `PlanNode.printNestedColumns` 仍 `instanceof IcebergScanNode`(cutover 后死代码); +- `CatalogFactory.SPI_READY_TYPES` 仍按字符串 gate 插件路径; +- `TableType.ICEBERG_EXTERNAL_TABLE` 还有 5 处编译期引用。 + +**后果**:一个全新连接器插件在 tip 上仍会拿到错误引擎名、被数据缓存排除、根本加载不了——直到有人改 fe-core。**连接器在 tip 上还不是真正可插拔的**,flavor 知识只是从 instanceof 挪进了字符串 switch。P8 要真删 allowlist,得先把这些 switch 换成 Connector SPI 声明的属性。 + +### 5.3 N-连接器 × N-编辑 的 switch —— ⚠️ 无编译期保护 + +引擎名/表类型名现在是**三处**必须手改同步的 fe-core switch(`CreateTableInfo.pluginCatalogTypeToEngine` + `PluginDrivenExternalTable.getEngine()` + `getEngineTableTypeName()`),javadoc 自认"the two switches must stay in sync"。P6 已经给三处都加了 iceberg case,证明了 N×N 轨迹。漏改一处 → `SHOW TABLE STATUS`/`information_schema` 里静默显示通用 `Plugin` 引擎名——正是这些 switch 存在要防的 I3 回归,却没有编译期或测试把它们绑在一起。**建议**:`Connector.getLegacyEngineName()`,每个插件声明一次。 + +### 5.4 跨连接器复制 —— ⚠️ + +| 复制物 | 副本 | 相似度 | +|---|---|---| +| `TcclPinningConnectorContext` | iceberg / paimon | ~97%(paimon javadoc:"the paimon analogue of the iceberg connector's") | +| `buildHadoopConfiguration` / `assembleHiveConf` | PaimonCatalogFactory / IcebergCatalogFactory | iceberg javadoc:"mirror PaimonCatalogFactory" | +| flavor-provider wrapper | dlf/hms/jdbc/rest × iceberg/paimon | 4 对,~80-92% | +| EQ/IN 分区裁剪块(~110 行)+ FakeHmsClient stub | hive / hudi | 3 份 stub | + +TCCL decorator 守的是 classloader 隔离正确性——一份修了另一份漏,产生连接器专属的 classloader bug。一个共享 `fe-connector-support` 模块能止住这种线性增长。 + +--- + +## 6. 贯穿性正确性主题 + +### 6.1 EXPLAIN / 行为 parity(invariant #3)—— 系统性违约 + +- **EXPLAIN 节点名**:legacy 打印 `VPAIMON_SCAN_NODE`/`VICEBERG_SCAN_NODE`(master `PaimonScanNode.java:159` 传 `"PAIMON_SCAN_NODE"`),插件路径打印 `VPluginDrivenScanNode` + `CONNECTOR: x`。live 实测坐实。回归测试是**改期望值接受**而非保留行为。`deviations-log` 无记录。修复廉价:`planNodeName = connectorType.toUpperCase() + "_SCAN_NODE"`。 +- **错误信息 drift**:P0 的 "CREATE TABLE not supported"(丢 catalog 名)、P4 的 DB DDL errno(丢 1007/1008)、P6 的 `position_deletes`/FOR TIME 越界错误文本。 +- **SHOW PARTITIONS**:P4 现在 admit 所有 `PluginDrivenExternalCatalog`(jdbc/es/trino 原本被定向拒绝),错误类别对非分区连接器改变;MC 的 LIMIT/OFFSET 语义从远端分页变成全取-排序-分页。 +- **selectedPartitionNum 语义**(P5,CONFIRMED medium):从"SDK 规划 split 里的 distinct 分区"变成"Nereids FE 剪枝后分区数",影响 EXPLAIN `partition=N/M` 和 `sql_block_rule` 的 `partition_num` 检查——legacy 通过的规则现在可能 block 查询。 + +### 6.2 休眠的 P7 地雷(P3 hudi,全部 CONFIRMED,当前 dormant) + +hudi plugin 路径今天不触发(不在 `SPI_READY_TYPES`),但 P3 是 P7 的"测试基线",且 parity 测试写法抓不到这些: + +- **分区值不 unescape**(high):HMS 存 `ts=2024-01-01 10%3A00%3A00`,`parsePartitionName` 拆原始文本直接比,含 `空格/:/%/=` 的分区值永远匹配不上→静默剪掉→丢行。legacy 用 `FileUtils.unescapePathName`。 +- **datetime ISO 匹配不上**(high):fe-core 把 datetime 字面量包成 `LocalDateTime`,`String.valueOf` 出 `2024-01-01T10:00`(T 分隔、秒省略),永远不等于 Hive 路径文本→整表分区剪光→0 行。 +- **HMS 名当 Hudi 存储路径**(high):`prunedPartitionPaths` 存 HMS 名 `year=2024/month=01`,喂给 `fsView.getLatestBaseFilesBeforeOrOn` 却期望 Hudi 相对存储路径;非 hive-style 分区(hudi 默认)存储布局是 `2024/01`→带 filter 0 split、不带 filter 有行。legacy 对分区 LOCATION 做相对化。 +- **混大小写 Avro 崩 JNI**(high):`getTableSchema` 小写化,但 planScan 送原始 `Schema.Field::name`;BE `HadoopHudiJniScanner` 精确大小写查找→`IllegalArgumentException` 崩每个 MOR/JNI split。该 PR 自己的 `HudiSchemaParityTest` 用的就是 `Id`/`Name`/`Addr`。 +- **COW/MOR 3-态 `"UNKNOWN"` 静默当 MOR**(medium):违背该 PR 自己的 fail-loud 设计;丢了 legacy 的 `skip_merge`/`flink.table.type` COW 信号;`spark.sql.sources.provider→COW` 是发明的启发式(无 legacy 对应),把 spark 注册的 MOR 表误读成 COW→跳 delta log→陈旧行。 + +### 6.3 谓词下推分歧 + +- **P4 maxcompute 变全有全无**(CONFIRMED low/perf):一个不可转换的 conjunct 丢掉整个 ODPS filter(legacy 逐 conjunct 保留可转的)。correctness 安全(BE 重过滤),但选择性查询扫描量暴涨。 +- **P4 `==` vs `=`**(PLAUSIBLE medium):EQ 发 `==`,ODPS SDK 描述是 `=`,ODPS 解析器是否容忍 `==` 静态无法确定,需 live A/B。**注:同一处的 IN 方向改动其实修了 legacy 的一个反向 bug**(legacy 发的是反的 IN 谓词),这是 bugfix,建议加回归测试。 +- **P5 paimon 三个"更激进"候选被 cast-guard 证伪**(REFUTED):`value.toString()` VARCHAR、Number 截断、LIKE 不查类型——都因 `supportsCastPredicatePushdown()=false` 在转换前 strip 掉含 cast 的 conjunct 而不可达。**这个 guard 是 load-bearing 的**,建议加测试/注释 pin 住,免得未来"启用 cast 下推"静默放出这些 arm。 +- **P5 `convertAnd` 部分合取**(CONFIRMED low/sound):OR 下的部分 AND 保留可转子集(legacy 要求两侧都可转否则 null)。验证为 sound(部分 AND 严格更弱 + BE 重过滤),只是 plan/profile 可见差异。 + +### 6.4 缓存一致性 + +- **P6 快照缓存 vs schema 缓存偏斜 → BE DCHECK 崩溃**(PLAUSIBLE high):`beginQuerySnapshot` 从 `IcebergLatestSnapshotCache` pin `snapshotId+schemaId`,但 slot schema 从独立的 name-keyed fe-core schema 缓存绑定,两缓存独立 TTL 过期。外部 ALTER 后两者错时重载→pinned schemaId 与 bound slots 偏斜→BE field-id dict 缺 scan slot→`IcebergScanPlanProvider` 自己注释说会触发 BE StructNode DCHECK 崩溃。legacy 结构免疫(schema 缓存 keyed BY 快照的 schemaId,单一原子源)。无单查询复现(需独立 TTL 时序),但崩溃模式严重。**建议**:schema 缓存 key off pinned schemaId,照 legacy。 +- **P6 同表双引用 version-blind**(PLAUSIBLE medium):`t JOIN t FOR VERSION AS OF old`,schema 绑定用 version-blind 取快照(偏好 latest)而 scan 用 version-aware pin old→同类偏斜。legacy 是"differently-unsound"(table-only key,两引用共享一快照),javadoc 承认此限。 +- **P4 分区值缓存删除**(CONFIRMED medium):`getNameToPartitionItems` 每查询做一次完整 ODPS `listPartitions` 往返(legacy 从 `MaxComputeExternalMetaCache` 供)。数万分区的表每次规划多秒 + API 限流风险。 + +### 6.5 升级兼容(invariant #2) + +- **✅ GSON shim 干净**:iceberg 8 flavor + database + table、paimon 5 flavor、maxcompute 全有 `registerCompatibleSubtype` shim + 回放测试;logType 保留;`gsonPostProcess` 从 logType 回填 `type` 且不改 catalog 属性。 +- **⚠️ 运营坑**(P6,high operational):只换 `lib/doris-fe.jar` 不部署 `plugins/connector/iceberg`(经典升级手法)→镜像加载 OK,但首次访问任何存量 iceberg catalog 抛 `RuntimeException("No ConnectorProvider found...")`(`PluginDrivenExternalCatalog.java:132`),因为 legacy fe-core `iceberg` fallback 已删。"透明迁移"只在完整部署新 plugins 目录时成立。**需响亮的 release note + 升级文档步骤**(build.sh 部署到 `plugins/connector` 而非 `lib/`)。 +- **⚠️ 守门有洞**(P0):grep 黑名单漏 `import static`、漏 `persist/transaction/fs/statistics/mysql/service` 包、只扫 `src/main/java`。今天无 live 违规,但这是 I1 的机器执行——建议反转为 allowlist(只许 `connector/thrift/filesystem/extension`)+ 匹配 `^import (static )?` + 扫测试源。 + +--- + +## 7. 安全观察 + +- **`SUPPORTS_SHOW_CREATE_DDL` 无引擎侧脱敏**:引擎把 `getTableProperties()` 逐字渲染进 SHOW CREATE TABLE(`Env.java:~4891`),仅靠 capability gate。capability 的 javadoc 自认:唯一防止密码泄露的是连接器作者记得在属性含凭证时不声明该 flag。一个未来的 JDBC 系 lakehouse 连接器若声明了 flag 而属性含密码,任何有 SHOW 权限的用户都能看到明文。**建议**:DDL 渲染路径加敏感键过滤(defense in depth)。 +- **敏感键 masking 硬编码 per-connector 进 fe-core**:`DatasourcePrintableMap` 的 `SENSITIVE_KEY` 加 `MCProperties.SECRET_KEY`、iceberg REST 键块(注释"Keep in sync with the connector's sensitive REST keys")——每个新连接器要么改 fe-core,要么在 SHOW CREATE CATALOG 里静默泄密。 + +--- + +## 8. 逐 PR 详细发现 + +> 严重度:🔴 高 / 🟠 中 / 🟡 低。验证:**C**=CONFIRMED / **P**=PLAUSIBLE。dormant=当前不触发(P7 切换后触发)。 + +### P0 #63582 — SPI 基线 + DDL/Partition + import gate + +*29 文件。无独立验证器;finder 对 tip 自校验,6 条全部 still_at_tip。已在 tip 修掉的 2 条(CTAS 已存在语义、建表后缓存失效)未报。* + +| # | 严重度 | 文件:行 | 发现 | 失败场景 | +|---|---|---|---|---| +| P0-1 | 🟠 I3 | `PluginDrivenExternalCatalog.java:407` + `ConnectorTableOps.java:152` | 不支持的 CREATE TABLE 报 "CREATE TABLE not supported"(丢 catalog 名),legacy 报 "Create table is not supported for catalog: ``";DROP/RENAME/ADD COLUMN 同样 drift | 客户端错误断言/日志解析失配,用户丢失 catalog 名 | +| P0-2 | 🟠 | `CreateTableInfoToConnectorRequestConverter.java:212` | `readBucketNum` catch-all 吞异常返回 0 桶 | `translateToCatalogStyle()` 抛异常时,连接器拿 `numBuckets=0` 建出错误分布的表;违反 AGENTS.md "report errors or crash" | +| P0-3 | 🟠 | `...Converter.java:159` | `convertFields` 静默丢不认识的分区表达式形状 | 不支持的 partition transform → 建出无分区/错分区表且报成功;丢失发生在 fe-core,连接器无法察觉 | +| P0-4 | 🟠 | `ConnectorTableOps.java:167` | SPI 默认 `createTable(request)` 降级到旧签名,静默丢 partition/bucket/external/ifNotExists | 只实现旧签名的连接器接受 `PARTITION BY` 建出无分区表报成功 | +| P0-5 | 🟡 I1 | `tools/check-connector-imports.sh:48` | 守门三洞:漏 `import static`、漏 persist/transaction/fs/... 包、只扫 main | 后续 phase 加 `import static org.apache.doris.catalog...` 或 `import org.apache.doris.persist.EditLog` 时守门放行,静默重耦合 | +| P0-6 | 🟡 | `ExternalMetaCacheInvalidator.java:72` | `invalidateStatistics` 静默 no-op;`invalidatePartition` 退化整表 | 未来连接器(HMS 事件)调 invalidateStatistics 期望丢陈旧行数,什么都没发生,优化器继续用陈旧统计;告诫只在 fe-core 实现注释里,SPI 侧不可见 | + +### P1 #63641 — plugin-first 路由收口 + +*5 文件。3 条候选在 tip 已被后续 phase 修掉(FileScan 分支 setSelectedPartitions、hudi 增量/时旅 fail-loud、嵌套列裁剪 capability 化)——未报,验证了"对 tip 复核"的价值。* + +| # | 严重度 | 文件:行 | 发现 | +|---|---|---|---| +| P1-1 | 🟠 dormant | `PhysicalPlanTranslator.java:836/911` | `visitPhysicalHudiScan` plugin 分支缺 `setSelectedPartitions` + `setDistributeExprLists`(FileScan 兄弟分支已修,这条漏了)→ P7 hudi 切换后分区不裁剪 + bucket-shuffle 计划退化。根因:dispatch 块两个 visitor 复制粘贴已漂移 | +| P1-2 | 🟠 dormant | `PhysicalPlanTranslator.java:740` | plugin 分支不转发 `getTableSample()`,`PluginDrivenScanNode` 无 table-sample 管线 → P7 hive 切换后 TABLESAMPLE 静默全表扫 | +| P1-3 | 🟡 | `CacheAnalyzer.java:305` / `PlanNode.java:949` | `instanceof HiveScanNode`/`IcebergScanNode` 残留:iceberg 已由 P6 切换使 PlanNode 那处 arm 死代码;hive 待 P7 切换后 SQL cache 静默失效 | + +### P2 #64096 — trino(首个端到端样板) + +*33 文件。验证器:C1/C7/C8 REFUTED(见附录 A)。首个迁移样板,设计模式被 P4/P5/P6 复制。* + +| # | 严重度 | 验证 | 文件:行 | 发现 | 失败场景 | +|---|---|---|---|---|---| +| P2-1 | 🟠 | C | `TrinoConnectorDorisMetadata.java:243` | 每个元数据方法开 Trino 事务从不 commit/rollback/close;legacy 每次 schema load 缓存一个复用 | 有状态 Trino 连接器(hive TransactionManager)每查询泄漏 3+ 未关事务→FE 内存随查询数无界增长 | +| P2-2 | 🟠 | C | `TrinoBootstrap.java:136` | `getInstance(pluginDir)` 首胜单例,但 `trino.plugin.dir` 宣传为 per-catalog | 第二个 catalog 用不同 dir 静默用第一个的→"Cannot find Trino ConnectorFactory" | +| P2-3 | 🟡 I3 | C | `TrinoDorisConnector.java:78` | `preCreateValidation` 在 CREATE CATALOG 时急切加载插件+构造连接器;legacy 惰性 | 先建 catalog 后铺插件目录的脚本现在 CREATE 就失败;replay 跳过校验→create-vs-replay 分歧 | +| P2-4 | 🟡 | P | `TrinoConnectorDorisMetadata.java:107` | `listTableNames` 丢了 legacy 的 LinkedHashSet 去重 + prefix 过滤 | listTables 返回重复 SchemaTableName 的连接器→SHOW TABLES 重复行 | +| P2-5 | 🟡 | P | `TrinoDorisConnector.java:176` | DCL 发布顺序:guard 字段 `trinoConnector` 先于 `trinoSession` 赋值,fast-path 只查前者 | 并发首次访问落入两次 store 之间→`trinoSession.toConnectorSession` 瞬时 NPE(volatile 自愈) | + +### P3 #64143 — hudi 加固(cutover 推迟 P7) + +*28 文件。验证器:**8/8 CONFIRMED**。全部 dormant(hudi 未进 allowlist),但 P3 是 P7 基线。详见 [6.2](#62-休眠的-p7-地雷p3-hudi全部-confirmed当前-dormant)。* + +4 高危(unescape / datetime ISO / HMS 名 vs 存储路径 / 混大小写 Avro JNI)+ 2 中(UNKNOWN 静默 MOR / 检测器丢 legacy COW 信号)+ 2 低(ENUM→STRING 误标 parity / 三份复制)。 + +### P3b #64655 — kerberos 收拢进 fe-kerberos + +*82 文件。验证器分类 master-parity-break / intra-branch / new-code。C3/C6/C7/C8 REFUTED(master 同病=parity,见附录 A)。* + +| # | 严重度 | 验证 | 分类 | 发现 | +|---|---|---|---|---| +| P3b-1 | 🟠 | C | parity-break | simple-auth 无 `hadoop.username` 时身份从"FE 进程用户"翻转为 remote user `"hadoop"`+doAs;按 FE 进程用户授权 ACL 的 HDFS 访问升级后 `AccessControlException` | +| P3b-2 | 🟠 | C | parity-break | `HadoopKerberosAuthenticator` 每次 login/refresh 无条件 `UGI.setConfiguration` + 强设 `authorization=true`;删了 legacy 的 first-writer-wins guard;混 simple+kerberos 多 catalog 更频繁抢写 JVM 全局 UGI | +| P3b-3 | 🟠 | C | new-code | plugin zip 双载 fe-kerberos,类锁分裂(见 [3.3](#33-类加载隔离模型--️-有一处真实的锁失效)) | +| P3b-4 | 🟡 | C | parity-break | `doAs` 把 `InterruptedException` 转 `IOException` 不 `Thread.currentThread().interrupt()`;查询取消对 filesystem 路径的重试/drain 循环不可见 | + +### P4 #64300 — maxcompute(首个删 legacy) + +*203 文件。验证器:C4 REFUTED(page cache parity,见附录 A)。首个删 legacy,模式被 P5/P6 复制。* + +| # | 严重度 | 验证 | 文件:行 | 发现 | 失败场景 | +|---|---|---|---|---|---| +| P4-1 | 🔴 | C | `PluginDrivenScanNode.java:173` | batch-mode 闸门从 `!= NOT_PRUNED` 塌成 `!isPruned`;无 filter 的分区表 `initSelectedPartitions` 返回非哨兵 `isPruned=false` | ≥1024 分区表的无谓词全扫失去异步 batch split→FE 规划延迟/内存暴涨,EXPLAIN 变化 | +| P4-2 | 🟠 | C | `PluginDrivenExternalTable.java:266` | 分区值缓存删除,每查询完整 ODPS `listPartitions` 往返 | 数万分区表每次规划多秒 + API 限流 | +| P4-3 | 🟠 | P | `MaxComputePredicateConverter.java:146` | EQ 发 `==`(ODPS SDK 描述是 `=`),容忍性需 live A/B。**IN 方向改动是 bugfix**(legacy 发反向 IN),建议加回归测试 | +| P4-4 | 🟡 | C | `MaxComputePredicateConverter.java:117` | 谓词下推全有全无(一个 leaf 抛异常丢整个 filter);perf-only(BE 重过滤) | +| P4-5 | 🟡 | C | `PluginDrivenInsertExecutor.java:159` | `doAfterCommit` 吞 post-commit refresh 失败(DV-018);INSERT 报成功而 legacy 传播 DdlException。数据已提交,arguably 更安全,但 success-vs-error 用户可见 | +| P4-6 | 🟡 | C | `PluginDrivenExternalCatalog.java:493` | DROP DATABASE 丢 ERR_DB_DROP_EXISTS(1008),CREATE DATABASE 丢 ERR_DB_CREATE_EXISTS(1007)prechecK;createTable 的 ERR_TABLE_EXISTS 已恢复,说明这些码属契约 | +| P4-7 | 🟡 | C | `ShowPartitionsCommand.java:203` | SHOW PARTITIONS admit 所有 plugin catalog(jdbc/es/trino 原被定向拒绝);MC LIMIT/OFFSET 从远端分页变全取分页 | +| P4-D | 设计 | — | `ConnectorTransaction`/`Transaction`/`CreateTableInfo`/`MCConnectorClientFactory` | ODPS block 分配爬上通用事务接口(见 [4.3](#43-spi-表面正在连接器化--️-这是最该管的抽象趋势));第三处引擎名 switch(见 [5.3](#53-n-连接器--n-编辑-的-switch--️-无编译期保护));MC 凭证/客户端构建 FE 侧(MCConnectorClientFactory)vs BE-JNI 侧(MCUtils)复制,auth 修一处漏一处→split-brain | + +### P5 #64446 — paimon 迁移 + cutover + +*283 文件。验证器:C1/C2/C3 REFUTED(cast-guard,见附录 A);L2(SHOW CREATE 无分区)DISPROVEN(master 同样不渲染)。* + +| # | 严重度 | 验证 | 文件:行 | 发现 | +|---|---|---|---|---| +| P5-1 | 🟠 I3 | C | `CatalogFactory.java:51`(加 paimon) | EXPLAIN 节点名 `VPAIMON_SCAN_NODE`→`VPluginDrivenScanNode`;deviations-log 无记录(live 实测坐实) | +| P5-2 | 🟠 | P | `PaimonScanPlanProvider.java:787` | JNI/COUNT range 的 `file_format` 取表级 `file.format`(默认 parquet),legacy 从首个数据文件后缀推;native 路径仍推后缀。ORC 数据+无 option→paimon-cpp 误读 | +| P5-3 | 🟠 | C | `PluginDrivenScanNode`(selectedPartitionNum) | 语义从"SDK split distinct 分区"变"FE 剪枝分区数",影响 EXPLAIN + sql_block_rule partition_num | +| P5-4 | 🟡 | C | `PaimonTypeMapping.java:254` | to-Paimon(CREATE TABLE)丢嵌套 nullability + struct 字段注释;非 SPI 限制(ConnectorColumnConverter 有 childrenNullable/Comments,iceberg 用了,paimon 没读) | +| P5-5 | 🟡 | C | `PaimonScanPlanProvider.java:457` | `ignore_split_type` session 变量插件路径无人消费(legacy 4 处);变量仍存在→`SET` 静默 no-op | +| P5-6 | 🟡 | C | `PaimonPredicateConverter.java:121` | `convertAnd` OR 下部分合取(sound,plan/profile 差异) | +| P5-7 | 设计 | — | `PaimonSchemaBuilder.java:127` | `getInitialValues()` 从不读,`VALUES IN` 静默吞(live 实测;master 同样吞=parity,但死 API 面 + 接受做不到的 DDL) | +| P5-D | 设计 | — | `PaimonCatalogFactory` | `buildHadoopConfiguration`/`assembleHiveConf` 共享设施放连接器里(iceberg 已复制,见 [5.4](#54-跨连接器复制--️)) | + +### P5b #64653 — paimon 删 legacy + +*75 文件,-8422。删除对账干净(GSON shim、sys 表、SHOW PARTITIONS 列、属性 passthrough 全有插件对应物)。* + +| # | 严重度 | 文件:行 | 发现 | +|---|---|---|---| +| P5b-1 | 🟡 | `SummaryProfile.java:158` | FE 侧 paimon scan-plan 剖面指标(`Paimon Scan Metrics` 段:manifest scan 时间、skip split 数)随 `PaimonScanMetricsReporter` 删除无替代;`PAIMON_SCAN_METRICS` 常量零 writer 悬空。慢 split 规划诊断能力回归;plan-doc 登记为已知回归。建议:连接器无关的 scan-metrics 钩子(也服务 iceberg/hudi),或删悬空常量 | + +### P6 #64688 — iceberg(最大,7 flavor + MVCC + 写路径 + procedures) + +*685 文件。5 个子系统 finder + 独立验证器。写路径/事务/10 procedure/CTAS 回滚验证忠实(未报)。* + +| # | 严重度 | 验证 | 文件:行 | 发现 | +|---|---|---|---|---| +| P6-1 | 🔴 | P | `IcebergConnectorMetadata` + `PluginDrivenMvccExternalTable:482` + `IcebergScanPlanProvider:1072` | 快照缓存 vs schema 缓存独立 TTL 偏斜→BE field-id dict 缺 scan slot→StructNode DCHECK 崩溃(见 [6.4](#64-缓存一致性)) | +| P6-2 | 🔴 | C(operational) | `PluginDrivenExternalCatalog.java:132` | 只换 lib/ 不部署 plugins/connector→所有存量 iceberg catalog 首次访问抛"No ConnectorProvider found"(见 [6.5](#65-升级兼容-invariant-2)) | +| P6-3 | 🟠 | C | `IcebergConnectorMetadata.java:645` | `computeRowCount` 丢 equality-delete gate:`totalRecords - positionDeletes` 无条件,legacy 有 equality-delete!=0→UNKNOWN;COUNT(*) 下推路径保留了 gate(不对称)。MOR/CDC 表统计膨胀,误导 CBO | +| P6-4 | 🟠 | C | `IcebergConnector.java:611` | s3tables 无显式凭证即硬失败;`S3FileSystemProvider.supports()` 要求 AK/SK/roleARN。只有 region+warehouse ARN 的 EC2 instance-profile catalog 无法创建;legacy 走 PROVIDER_CHAIN 默认链 | +| P6-5 | 🟠 | C | `IcebergCatalogFactory.java:83` | region 别名收窄成 4 个,丢 `AWS_REGION`/`iceberg.rest.signing-region`/`rest.signing-region`;REST vended-cred catalog 用被丢别名→S3FileIO "Unable to load region";注释声称"mirror" legacy 不实 | +| P6-6 | 🟠 | P | `PluginDrivenMvccExternalTable.java:475` | 同表双引用 version-blind 取快照(见 [6.4](#64-缓存一致性)) | +| P6-7 | 🟡 | C | `IcebergConnectorMetadata.java:626` | `FOR TIME AS OF '<数字串>'` 现读作 epoch millis(legacy 拒绝);benign superset | +| P6-8 | 🟡 | P | `IcebergTimeUtils.java:76` | 无上下文线程 `resolveSessionZone` 回退 UTC,legacy 回退 FE 默认 session tz;无具体可达的无上下文时间解析路径 | +| P6-9 | 🟡 | C | `IcebergTypeMapping.java:143` | 未知/v3 类型(TIMESTAMP_NANO/GEOMETRY)静默降级 UNSUPPORTED,legacy schema load 抛异常;TIME 是 parity(master 也 UNSUPPORTED) | +| P6-10 | 🟡 | C | `IcebergConnectorMetadata.java:1292` | `$position_deletes` 丢定向错误信息("not supported yet"),变通用 unknown-table | +| P6-S | 安全 | — | `ConnectorCapability` + `Env.java:4891` | `SUPPORTS_SHOW_CREATE_DDL` 无脱敏(见 [7](#7-安全观察));P6 还破坏性重写 capability enum(~14 值删除,强迫同 PR 重写 jdbc/mc/paimon) | +| P6-D1 | 设计 | — | `RowLevelDmlRegistry.java:38` | 能力门控通用,身体 iceberg-shaped(`IcebergDeleteCommand` 绑 `__DORIS_ICEBERG_ROWID_COL__`);下一个声明 DELETE 的连接器被路由进 iceberg transform→unresolvable-slot | +| P6-D2 | 设计 | — | `ConnectorMvccPartitionView.java:113` | `getNewestUpdateTimeMillis` 名说毫秒实微秒(见 [4.3](#43-spi-表面正在连接器化--️-这是最该管的抽象趋势)) | +| P6-D3 | 设计 | — | `ConnectorMetadata.java:142` | `applyRewriteFileScope`/`applyTopnLazyMaterialization` 契约靠散文 javadoc:raw path 字符串相等匹配"两侧都不归一化",违反→静默数据损坏(rewrite 重复行/OCC)非报错 | +| P6-D4 | 设计 | — | `TcclPinningConnectorContext` ×2 + flavor provider ×4 | 跨连接器复制(见 [5.4](#54-跨连接器复制--️)) | +| P6-D5 | 设计 | — | 多处 | P8 blocker(见 [5.2](#52-p8删-allowlist不是纯删除--️)) | + +--- + +## 9. 验证方法与统计 + +**流水线**:每个 PR 按角度(逐行正确性 / 删除行为审计 / 跨文件 / 设计·抽象·扩展 / 复用·简化·效率 / conventions)+ 子系统(P6 切 5 刀:catalog flavor / MVCC / 写路径 / 删除+GSON / SPI 设计)派 finder → 每个候选**强制对最新 tip 复核**(后续 phase 修掉的不算)→ 对抗验证器逐条 CONFIRMED/PLAUSIBLE/REFUTED,主动找 guard、不可达状态、master parity。 + +**"对 tip 复核"救回的误报**:P0 有 2 条、P1 有 3 条候选在最新 tip 已被后续 phase 修掉——只读 PR diff 会误报。 + +**对抗验证证伪的**(保护信誉,不发掺水账): +- P2:3 条(trino-spi 自己小写列名 / create_time 建库必写 / executor 是 legacy 自带) +- P3b:4 条(全 master 同病=parity 非回归) +- P4:1 条(page cache preprocessor master 同样不可达) +- P5:3 条(cast-guard 拦截)+ 1 条(SHOW CREATE 分区 master 同样不渲染) + +**finder 措辞纠错**:P4 finder 说 IN 方向反转是 bug,验证器查出**恰恰相反——tip 修了 legacy 的反向 bug**。据此把评论改成给作者 credit + 建议加测试。这就是验证不能省的原因。 + +**已发**:9 篇 GitHub review,26 条 inline 评论 + 每篇 body 汇总。 + +| PR | inline | 净发现(证伪后) | +|---|---|---| +| P0 #63582 | 6 | 6 | +| P1 #63641 | 2 | 2 + 1 body | +| P2 #64096 | 2 | 5(3 证伪) | +| P3 #64143 | 4 | 8(8/8 CONFIRMED)| +| P3b #64655 | 0 (body) | 4(4 证伪) | +| P4 #64300 | 2 | 7 + 设计(1 证伪) | +| P5 #64446 | 6 | 7 + 设计(4 证伪) | +| P5b #64653 | 0 (body) | 1 | +| P6 #64688 | 4 | 10 + 设计/安全 | + +--- + +## 10. 优先级建议 + +**合 master 前应处理(高)**: +1. **P6 缓存偏斜**(P6-1):schema 缓存 key off pinned schemaId,消除 BE DCHECK 崩溃窗口。 +2. **P6 升级坑**(P6-2):升级文档明确"必须部署 plugins/connector",或加 fe-core 侧 degraded 兜底。 +3. **P4 batch-mode**(P4-1):恢复 `!= NOT_PRUNED` 判据,避免大分区表 FE 规划暴涨。 +4. **安全**(P6-S):SHOW CREATE DDL 渲染路径加敏感键过滤。 + +**P7 前必查(中,dormant)**: +5. **P3 hudi 4 高危**(6.2):unescape / datetime / HMS-name-vs-storage-path / 混大小写 Avro——P7 切换即静默丢行或崩。P3 的 parity 测试要能抓到这些。 +6. **P7 hive 适配缺口**:TABLESAMPLE SPI 表达、CacheAnalyzer instanceof、kerberos 锁分裂。 + +**贯穿性(应有跟踪 issue)**: +7. **EXPLAIN 节点名**(6.1):要么保留 per-connector 名,要么在 #65185 登记为明示偏差。 +8. **三处引擎名 switch → Connector SPI 声明**(5.3)。 +9. **SPI 表面连接器化**(4.3):连接器专属能力收进 facet,别堆根接口。 +10. **跨连接器复制 → fe-connector-support**(5.4)。 +11. **守门反转为 allowlist**(6.5)。 + +**P4/P5 待验证(中)**: +12. **P4 `==` EQ**(P4-3):live ODPS A/B。 +13. **P5 file_format**(P5-2):option-unset ORC 表 paimon-cpp e2e。 +14. **P5 cast-guard**(6.3):加测试 pin 住 `supportsCastPredicatePushdown()=false`,免得未来放出激进 arm。 + +--- + +## 附录 A:被证伪的候选(体现覆盖面) + +| PR | 候选 | 判决 | 理由 | +|---|---|---|---| +| P2 | 列名大小写失配(columnHandleMap 小写 vs columnMetadataMap 原样) | REFUTED | trino-spi `ColumnMetadata` 构造器自身小写化名字;master 同样不对称=parity | +| P2 | create_time 每 planScan 新铸→BE 缓存不命中 | REFUTED | create_time 自 #18778(2023)每 catalog 建时必写 | +| P2 | 每 planScan 新建线程池不 shutdown | REFUTED | legacy `TrinoConnectorScanNode` 逐字自带;daemon 线程 60s 后 GC | +| P3b | 惰性 vs 急切 kerberos 登录 | REFUTED | master 同样惰性;bad-keytab 都在首次 IO/DDL 校验时报 | +| P3b | buildAuthenticator 判据分歧 | REFUTED | `HdfsConfigBuilder` 逐字 master 相同 | +| P3b | 空 username → createRemoteUser("") IAE | REFUTED | master fe-common 逐字相同,pre-existing | +| P3b | fe-kerberos 混中立类型+hadoop 机器 | REFUTED | 中立类型(AuthType/KerberosAuthSpec)machinery-free,JVM 惰性加载不可达 NoClassDefFoundError | +| P4 | INSERT INTO SELECT 不再关 page cache | REFUTED | master 的 `visitLogicalMaxComputeTableSink` 对 INSERT-SELECT 同样不可达(preprocess 早于 BindSink)=parity | +| P5 | VARCHAR `value.toString()` datetime/bool | REFUTED | `supportsCastPredicatePushdown()=false` 在转换前 strip 含 cast conjunct;VARCHAR arm 只见 String | +| P5 | 整数 Number 截断把谓词推强 | REFUTED | 同 cast-guard;cast-free 只交型匹配整型字面量 | +| P5 | LIKE 不查类型→ClassCastException | REFUTED | 同 cast-guard;CHAR 残留无 ClassCastException(CHAR stats 是 BinaryString) | +| P5 | SHOW CREATE TABLE 无 PARTITION BY(L2) | DISPROVEN | master paimon arm 同样只渲染 comment+LOCATION+PROPERTIES,无分区子句=parity | +| P6 | txn-id 双命名空间碰撞(P0 遗留) | 已缓解 | 连接器统一从 `session.allocateTransactionId()→Env.getNextId()` 取号 | + +--- + +## 附录 B:live 集群实测证据 + +worktree `/Users/lanhuajian/github/doris-catalog-spi`(tip `3ba75b7cf8a`),单 FE(JDWP :35005)+ 单 BE,端口偏移 +30000。 + +- **paimon filesystem catalog**(`paimon_debug.spi_db`):CREATE CATALOG/DB/TABLE(踩 P0 DDL 转换器)/ SELECT / EXPLAIN 全通;谓词成功下推(`predicatesFromPaimon: GreaterThan(id, 1)`)。 +- **实测坐实**:①EXPLAIN 打 `VPluginDrivenScanNode / CONNECTOR: paimon`(master 是 `VPAIMON_SCAN_NODE`)→ P5-1;②`PARTITION BY LIST (region) (PARTITION p VALUES IN ('x'))` 静默吞值定义 → P5-7;③paimon INSERT 被拒(supportsInsert=false)——但 master 也不支持 paimon 写=parity,非回归。 +- **数据注入**:paimon 无 Doris 写路径,写了 `PaimonSeeder.java` 直接用插件目录里的 paimon 1.3.1 SDK Batch Write API 灌数据(绕过 Doris);t_orders(4 行含 NULL)、t_part(3 分区)、t_list(4 分区,含 `__HIVE_DEFAULT_PARTITION__`)。外部写入即时可见(无需 REFRESH)→ 快照查询时现 pin;SHOW PARTITIONS 统计列有值 → P0 的 `ConnectorPartitionInfo` 统计字段生效;分区裁剪穿透 SPI(`partition=1/3`)。 +- **iceberg hadoop catalog**(`iceberg_legacy.ice_db`,legacy 对照):INSERT/SELECT 可用;EXPLAIN 打 `VICEBERG_SCAN_NODE`。坑:file:// warehouse 首次 INSERT 前需手工 `mkdir -p ///data`(BE 本地写不建目录)。 + +--- + +*本评审综合了对 9 个 PR 的系统性多智能体审查(20 finder + 6 对抗验证器,全部对 tip 复核)与 live 集群实测。每条发现均带 master 对照的 file:line 和具体失败场景,可直接定位。* From 368a6f5b7eb4f156e78794c399f868face70bf0d Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 04:50:15 +0800 Subject: [PATCH 204/334] [doc](catalog) #65185 reverify: fix task-list series (H1-H4/M1-M8/L1-L20/D-series) Tracking doc for fixing the reverify findings one-by-one in later sessions. Follows the existing task-list-*-fixes.md convention: master progress table (32 tracked tasks) + per-task blocks (current file:line / fix / files / test intent), suggested batches by blast radius (4 hudi highs first), AGENT-PLAYBOOK single-task loop, module build/verify reminders. Uses the same H/M/L ids as the reverify report so the two cross-reference. Flags 4 decision-gated tasks (ask-user-first) and an explicit "do NOT touch" list (registered deviations + refuted / already-fixed findings) so later sessions don't mis-edit. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 230 +++++++++++++++++++++ 1 file changed, 230 insertions(+) create mode 100644 plan-doc/task-list-65185-reverify-fixes.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md new file mode 100644 index 00000000000000..64fb54656b30b0 --- /dev/null +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -0,0 +1,230 @@ +# Task List — #65185 复核修复系列(2026-07-11 起) + +> **来源(证据/推理详见)**：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md`。本文件是**跟踪表**,只记「做什么 / 改哪 / 怎么验 / 进度」;每条的完整证据、失败场景、对抗结论去 reverify 文档对应小节(§2 高 / §3 中 / §4 设计 / §5 已登记 / §6 不适用)。 +> **本系列范围**：reverify 中判定为「真实/活跃」且需**改代码**的条目。已登记/验收偏差(reverify §5)与不适用/已修/parity(reverify §6)**不在本系列**,见文末「明确不做」。 +> +> **处理纪律(AGENT-PLAYBOOK 单任务循环,每条一遍)**： +> 起步先读 `HANDOFF.md` + 本表 → 选一条 → **对 HEAD 复核现码**(reverify 行号可能又漂了) → 设计(`plan-doc/tasks/designs/FIX--design.md`) → 设计红队 → 实现 → 实现自验 → build + 靶向 UT → **独立 commit** → summary → 勾掉本表 → **更新 HANDOFF + commit**(memory `handoff-discipline-per-phase`)。 +> **每条一个独立 commit**;**path-whitelist `git add`,禁 `git add -A`**(工作树大量遗留 scratch,见 HANDOFF §分支须知)。上下文超 30% 找干净节点交接(memory `session-handoff-at-30pct-context`)。 +> +> **构建/验证备忘(memory `doris-build-verify-gotchas` + HANDOFF §操作须知)**： +> - fe-core：`mvn -o -f fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false`(漏 `-am`→假 `${revision}` 错)。 +> - 连接器:`-pl :fe-connector- -am`;靶向 UT 加 `-Dtest= -DfailIfNoTests=false`。 +> - **⚠ paimon 模块必须 `install`/`package`(shade jar 绑 package 阶段);hms/hive/hudi/iceberg/maxcompute 无此坑**。 +> - 连接器**无 Mockito**(真 recording fake);**fe-core 有 Mockito**;checkstyle 禁 static import、扫 test 源、`UnusedImports` fail build。 +> - `bash tools/check-connector-imports.sh` 须 exit 0(连接器不得 import fe-core)。 +> - **信 LOG 不信 exit**:后台 task 通知的 exit code 是 wrapper 的;重定向到文件 grep `BUILD SUCCESS`/`BUILD FAILURE`/`[ERROR].*\.java:`/`Tests run:`/`You have N Checkstyle`。全量编译 ~6min→后台跑。 +> - **e2e 多为 live-gated**(hudi/iceberg/s3/odps 需真集群);无法本地跑的须显式登记为 gated,别静默略过(Rule 12)。异构 `type=hms` 目录 e2e 见 memory `hms-iceberg-delegation-needs-e2e`。 +> - **铁律**:fe-core 不得新增 `if(hive/iceberg/hudi)`/`instanceof HMSExternal*`/源名判别;不解析属性(memory `catalog-spi-plugindriven-no-source-specific-code`、`catalog-spi-no-property-parsing-in-fecore`)。通用节点 connector-agnostic。 + +--- + +## 进度总表 + +Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) + +| # | id | 严重度 | 模块 | 一句话 | 设计 | 实现 | build+UT | 状态 | +|---|----|-------|------|--------|------|------|----------|------| +| 1 | **H1** | 🔴高 | hudi | 分区名不 unescape→丢行 | ⬜ | ⬜ | ⬜ | ⬜ | +| 2 | **H2** | 🔴高 | hudi | datetime 分区谓词 ISO→0 行 | ⬜ | ⬜ | ⬜ | ⬜ | +| 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ⬜ | ⬜ | ⬜ | ⬜ | +| 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ⬜ | ⬜ | ⬜ | ⬜ | +| 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | +| 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ⬜ | ⬜ | ⬜ | ⬜ | +| 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | +| 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ⬜ | ⬜ | ⬜ | ⬜ | +| 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ⬜ | ⬜ | ⬜ | ⬜ | +| 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ⬜ | ⬜ | ⬜ | ⬜ | +| 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ⬜ | ⬜ | ⬜ | ⬜ | +| 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⬜ | +| 13 | **L1** | 🟡低 | tools | import-gate 三洞 | ⬜ | ⬜ | ⬜ | ⬜ | +| 14 | **L2** | 🟡低 | fe-core | 翻闸 hive 丢 SQL 缓存资格 + COUNTER 停增 | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | +| 15 | **L3** | 🟡低 | trino | 元数据事务从不 commit/close | ⬜ | ⬜ | ⬜ | ⬜ | +| 16 | **L4** | 🟡低 | trino | plugin.dir 首胜单例(fail-loud) | ⬜ | ⬜ | ⬜ | ⬜ | +| 17 | **L5** | 🟡低 | trino | listTableNames 丢去重 | ⬜ | ⬜ | ⬜ | ⬜ | +| 18 | **L6** | 🟡低 | trino | guard 字段发布顺序→瞬时 NPE | ⬜ | ⬜ | ⬜ | ⬜ | +| 19 | **L7** | 🟡低 | kerberos | UGI.setConfiguration 无 guard(丢 first-writer) | ⬜ | ⬜ | ⬜ | ⬜ | +| 20 | **L8** | 🟡低 | kerberos | doAs 吞 interrupt 不 restore | ⬜ | ⬜ | ⬜ | ⬜ | +| 21 | **L9** | 🟡低 | maxcompute | 谓词下推全有全无 | ⬜ | ⬜ | ⬜ | ⬜ | +| 22 | **L10** | 🟡低 | fe-core | EXPLAIN 节点名 VPluginDrivenScanNode | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | +| 23 | **L11** | 🟡低 | paimon | JNI/COUNT file_format 用表级默认 | ⬜ | ⬜ | ⬜ | ⬜ | +| 24 | **L12** | 🟡低 | fe-core/paimon | selectedPartitionNum 语义(登记或对齐) | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | +| 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ⬜ | ⬜ | ⬜ | ⬜ | +| 26 | **L14** | 🟡低 | paimon | ignore_split_type 静默 no-op | ⬜ | ⬜ | ⬜ | ⬜ | +| 27 | **L15** | 🟡低 | fe-core | PAIMON_SCAN_METRICS 悬空常量 | ⬜ | ⬜ | ⬜ | ⬜ | +| 28 | **L16** | 🟡低 | iceberg | 快照/schema 缓存偏斜(防御性 union) | ⬜ | ⬜ | ⬜ | ⬜ | +| 29 | **L17** | 🟡低 | fe-core/iceberg | 同表多版本 version-blind schema 绑定 | ⬜ | ⬜ | ⬜ | ⬜ | +| 30 | **L18** | 🟡低 | iceberg | 未知/v3 类型静默 UNSUPPORTED | ⬜ | ⬜ | ⬜ | ⬜ | +| 31 | **L19** | 🟡低 | fe-core/iceberg | partition_columns 魔法键撞名→误判分区 | ⬜ | ⬜ | ⬜ | ⬜ | +| 32 | **L20** | 🟡低 | maxcompute | EQ 发 `==`(对齐 `=` 或 live A/B) | ⬜ | ⬜ | ⬜ | ⏸ 或 live | +| — | **D-系列** | ⚪设计债 | — | 见文末「设计债跟踪」(多为 P8 前置/需一次性重构) | — | — | — | ⏸ | + +--- + +## 建议批次(独立;按 blast radius 从小到大 + 高危优先) + +- **批次 0(高危,先做)**：`H4`(单行 lowercase,最小)→ `H1`+`H3`(建议先做 `D-PRUNE` 抽共享 helper,再一处修 H1/H3)→ `H2`。全部需 hudi 剪枝/schema 回归测试(现有 parity 测试抓不到)。 +- **批次 1(中危·连接器局部)**：`M5`→`M7`→`M6`(iceberg)、`M4`(mc)、`M2`(hive)。 +- **批次 2(中危·fe-core 通用节点,blast radius 较大,单测充分后再动)**：`M3`、`M1`。 +- **批次 3(运营/门禁)**：`M8`(发布工具+文档,无 fe 编译)、`L1`(gate)。 +- **批次 4(低危·连接器)**：`L3–L6`(trino)、`L7/L8`(kerberos)、`L9/L20`(mc)、`L11/L13/L14`(paimon)、`L18`(iceberg)。 +- **批次 5(需决策,先问用户再动)**：`L2`(SQL 缓存:恢复 vs 登记)、`L10`(EXPLAIN 名:登记 vs 按连接器声明)、`L12`(selectedPartitionNum:登记 vs 对齐)。 +- **批次 6(设计债/P8)**：`D-系列`,择机或随 P8。`D-PRUNE` 因承载 H1/H3,提前到批次 0。 + +> 决策类(⏸)条目**先在 session 里用中文讲清背景+选项问用户**(memory `ask-user-explain-in-chinese-first`),别擅自选一路实现。 + +--- + +## 🔴 高危(H1–H4) + +### H1 — hudi 分区名不 unescape → 丢行 · reverify §2 H1 +- [ ] **H1** +- **现码**:`fe-connector-hudi/.../HudiConnectorMetadata.java:1054-1064`(`parsePartitionName` 无 unescape)+ `:1067-1078`(`matchesPredicates` 裸字符串比);候选名 `:247` ← `ThriftHmsClient.listPartitionNames:214-218`(原样 Hive 转义)。 +- **Fix**:存值前 unescape(复用 `HudiScanPlanProvider.unescapePathName`,提升可见性;或 Hive `FileUtils.unescapePathName`),对齐 legacy `HudiExternalMetaCache.loadPartitionNames:231`。**首选落在 `D-PRUNE` 抽出的共享 helper**。 +- **Files**:`HudiConnectorMetadata.java`(或共享 `HmsStylePartitionPruner`)。 +- **Test intent**:`HudiPartitionPruningTest` 加 HMS 名 `ts=2024-01-01 10%3A00%3A00` + 谓词 `ts='2024-01-01 10:00:00'`,断言命中(RED:被剪)。live e2e gated。 + +### H2 — hudi datetime 分区谓词 ISO 化 → 0 行 · reverify §2 H2 +- [ ] **H2** +- **现码**:`ExprToConnectorExpressionConverter.convertDateLiteral:309-322`(非 DATE→`LocalDateTime`)+ `HudiConnectorMetadata.extractLiteralValue:1030-1036`(`String.valueOf`→`2024-01-01T10:00`)+ `matchesPredicates:1067` 字符串比 HMS `2024-01-01 10:00:00`。 +- **Fix**:时间类型不做字符串剪枝——`extractLiteralValue` 按 Hive 规范文本(空格分隔、全 `HH:mm:ss[.ffffff]`)渲染 `LocalDateTime`;更优:`matchesPredicates` type-aware 或对时间列退回 fe-core typed Nereids 剪枝。 +- **Files**:`HudiConnectorMetadata.java`(时间渲染逻辑;必要时连带 `extractLiteralValue`)。 +- **Test intent**:DATETIME 分区列 + `= '2024-01-01 10:00:00'` 断言命中(RED:整表剪到 0)。 +- 注:此条对抗验证 agent 未完成,但机制确定 + 与 H1/H3 同源(均 CONFIRMED),保留高危。 + +### H3 — hudi HMS 名当存储路径 → 非 hive-style 带 filter 0 split · reverify §2 H3 +- [ ] **H3** +- **现码**:`HudiConnectorMetadata.applyFilter:244-267`(无条件用 hive-style HMS 名作 `prunedPartitionPaths`)→ `HudiScanPlanProvider.resolvePartitions:587-590` → `collectCowSplits:394`/`collectMorSplits:429` 喂 `fsView.getLatestBaseFilesBeforeOrOn`(期望 Hudi 相对存储路径)。`use_hive_sync` 感知只在列举路 `collectPartitions:641`,`applyFilter` 绕过。 +- **Fix**:`applyFilter` 候选分区源 `use_hive_sync_partition` 感知:`!useHiveSyncPartition`→对 `listAllPartitionPaths`(`2024/01`)剪枝;`useHiveSyncPartition`→`FSUtils.getRelativePartitionPath(basePath, location)` 相对化。 +- **Files**:`HudiConnectorMetadata.java`(applyFilter 候选源)。 +- **Test intent**:非 hive-style 表断言「带 filter 分区集 == 不带 filter」(RED:带 filter 0 split)。live e2e gated。 + +### H4 — hudi 混大小写 Avro → JNI/MOR reader 崩 · reverify §2 H4 +- [ ] **H4**(最小,建议先做) +- **现码**:`HudiScanPlanProvider.planScan:180-181` `.map(Schema.Field::name)`(原始大小写)→ `HudiScanRange:220` → BE `HadoopHudiJniScanner.initRequiredColumnsAndTypes:227-229` 对 lowercase `requiredField` 精确 `containsKey`→throw。 +- **Fix**:`planScan:181` 改 `.map(f -> f.name().toLowerCase(java.util.Locale.ROOT))`(与 `HudiConnectorMetadata.avroSchemaToColumns:905`、`HudiSchemaUtils:137` 一致);列类型顺序不变。 +- **Files**:`HudiScanPlanProvider.java`。 +- **Test intent**:`HudiSchemaParityTest` 加 JNI 列名断言(混大小写源→lowercase 列表)。MOR/JNI live e2e gated。 + +--- + +## 🟠 中危(M1–M8) + +### M1 — TABLESAMPLE 插件路径静默全表扫 · reverify §3 M1 +- [ ] **M1** +- **现码**:`PhysicalPlanTranslator.visitPhysicalFileScan:812-821`(只 `setSelectedPartitions`,不转发 `getTableSample()`;legacy 转发臂 `:837-840` 死)+ `PluginDrivenScanNode.getSplits:998-1019`(零 tableSample)+ `ConnectorScanPlanProvider.planScan:119`(无采样参数)。 +- **Fix**:(1)translator 插件臂在 `setSelectedPartitions` 后镜像 legacy 转发 `setTableSample`;(2)`PluginDrivenScanNode.getSplits` 实现**通用 connector-agnostic** 按 split 大小采样(仿 `HiveScanNode:448-458`,操作通用 `PluginDrivenSplit` 大小,**不按源分支**)。 +- **Files**:`fe-core/.../PhysicalPlanTranslator.java`、`fe-core/.../PluginDrivenScanNode.java`。 +- **Test intent**:强化 `test_hive_tablesample_p0.groovy` 断言采样后基数(非仅 EXPLAIN 子串)。 + +### M2 — 翻闸 hive 丢批量/异步 split · reverify §3 M2 +- [ ] **M2** +- **现码**:`HiveScanPlanProvider.java:62`(自称 No batch;不 override `supportsBatchScan`/`streamingSplitEstimate`)→ `PluginDrivenScanNode.computeBatchMode:1085-1113` 恒非 batch。legacy `HiveScanNode.isBatchMode:283-294` 按 `prunedPartitions.size()>=1024` 异步。 +- **Fix**:override `HiveScanPlanProvider.supportsBatchScan`(分区表 true)+ `planScanForPartitionBatch`,仿 `MaxComputeScanPlanProvider`;无需改 fe-core。若决定接受回归,须登记 + 大分区 e2e 真值门。 +- **Files**:`fe-connector-hive/.../HiveScanPlanProvider.java`。 +- **Test intent**:单测断言分区表 `supportsBatchScan`;大分区异步 split live e2e gated。 + +### M3 — MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` · reverify §3 M3 +- [ ] **M3** +- **现码**:`PluginDrivenScanNode.shouldUseBatchMode:1136` `!selectedPartitions.isPruned`;应 `== SelectedPartitions.NOT_PRUNED`。`ExternalTable.initSelectedPartitions:447` 无谓词返 `isPruned=false` 非哨兵。行内 `:1129-1132` 注释误称 parity。仅 MC opt-in(`MaxComputeScanPlanProvider.supportsBatchScan:254`)。 +- **Fix**:`:1136` 改回 `== NOT_PRUNED` 早退语义;订正 `:1129-1132` 注释。 +- **Files**:`fe-core/.../PluginDrivenScanNode.java`。 +- **Test intent**:纯 helper 单测覆盖 `(isPruned=false, 非 NOT_PRUNED, size≥阈值)` 应 →batch(RED:返回 false)。 + +### M4 — MC 分区值缓存删除 · reverify §3 M4 +- [ ] **M4** +- **现码**:`PluginDrivenExternalTable.getNameToPartitionItems:780-781`(每次 `listPartitions`)→ `MaxComputeConnectorMetadata.listPartitions:256-273` → `McStructureHelper.getPartitions:112-118`(裸 ODPS SDK)。fe-core+连接器**两层无缓存**。 +- **Fix**:maxcompute 连接器内加 TTL/容量 Caffeine(仿 `CachingHmsClient`/`HiveFileListingCache`),keyed `(project,db,table)`,`REFRESH` 失效。**不**在 fe-core 加二级缓存。若延后须登记 CACHE-P1。 +- **Files**:`fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java`(+ 新缓存类)。 +- **Test intent**:连接器单测:同表两次 `listPartitions` 只一次 SDK 往返(recording fake 计数)。 + +### M5 — iceberg computeRowCount 丢 equality-delete gate · reverify §3 M5 +- [ ] **M5** +- **现码**:`IcebergConnectorMetadata.computeRowCount:631-646`(无 gate 减法)vs COUNT(*) 下推 `IcebergScanPlanProvider.getCountFromSummary:1572-1574`(保留 gate)。legacy `IcebergUtils.getCountFromSummary:231-233` gate 到 UNKNOWN。javadoc+单测前提为假。 +- **Fix**:加 `TOTAL_EQUALITY_DELETES` 常量,`computeRowCount` 减法前 `null||!="0"→返回 -1(UNKNOWN)`;订正 javadoc `:624-629`;重写 `equalityDeletesDoNotGateTableStatistics:195-210` 断言 UNKNOWN。 +- **Files**:`fe-connector-iceberg/.../IcebergConnectorMetadata.java` + 同名测试。 +- **Test intent**:100 records + 5 equality-delete → rowCount UNKNOWN(RED:当前锁死 100)。 + +### M6 — iceberg s3tables 无显式凭证硬失败 · reverify §3 M6 +- [ ] **M6** +- **现码**:`IcebergConnector.createS3TablesCatalog:735-739`(`chosenS3` 空即 throw)← `S3FileSystemProvider.supports:64-74`(要 `hasCredential`)。legacy 用 `DefaultCredentialsProvider` 默认链。 +- **Fix**:`chosenS3` 空但能从原始 props 解析 region 时(复用 M7 拓宽别名),用 `DefaultCredentialsProvider`+`Region.of(...)` 建 client 而非抛;仅无 storage 且无 region 才 fail-loud。 +- **Files**:`fe-connector-iceberg/.../IcebergConnector.java`。 +- **Test intent**:region+warehouse-only s3tables props 建 catalog 不抛(RED:抛)。live S3 e2e gated。 + +### M7 — iceberg REST vended-cred region 别名收窄 · reverify §3 M7 +- [ ] **M7** +- **现码**:`IcebergCatalogFactory.java:83` `S3_REGION_ALIASES` 仅 4 个 → `appendS3FileIO:187-194` 唯一 region 源。丢 `AWS_REGION`/`iceberg.rest.signing-region`/`rest.signing-region` 等。 +- **Fix**:拓宽别名对齐 legacy `getRegionFromProperties`(至少加上述 3 个 + `REGION`/`glue.region`/`aws.glue.region`);连接器侧复制列表(不 import fe-core);订正 `:78-83` 注释。 +- **Files**:`fe-connector-iceberg/.../IcebergCatalogFactory.java`。 +- **Test intent**:仅 `AWS_REGION` 的 props → `CLIENT_REGION` 被发(RED:null)。 + +### M8 — 升级只换 lib 不部署 plugins → 首访抛 · reverify §3 M8 +- [ ] **M8**(无 fe 编译;发布工具 + 文档) +- **现码**:`PluginDrivenExternalCatalog.java:135` throw;`CatalogFactory` degraded 只护启动 `:119-127`。翻闸把 blast radius 扩到全部 type=hms 目录。 +- **Fix**:主线 = 发布/升级工具把连接器 jar 部署到 `connector_plugin_root`(build.sh/部署步骤)+ 响亮 release note。代码侧可选防御:replay 后聚合 ERROR 枚举所有 degraded 目录。**保留** first-access throw,**不**加 legacy fallback。 +- **Files**:`build.sh`/部署脚本、release-note/升级文档;(可选)`fe-core/.../CatalogFactory.java` 或 replay 收尾处聚合日志。 +- **Test intent**:升级文档步骤评审;(可选)degraded 聚合日志单测。 + +--- + +## 🟡 低危(L1–L20) + +> 每条一行「现码 → fix」,详见 reverify §1 表 + 对应正文。⏸ 三条(L2/L10/L12)先问用户。 + +- [ ] **L1** import-gate 三洞 · `tools/check-connector-imports.sh:48,50`:grep 改 `^import (static )?${FORBIDDEN}[.]`;`FORBIDDEN` 补 `persist|transaction|fs|statistics|mysql|service`;glob 覆盖 `src/test/java`。保留 `is_vendored()`(HiveVersionUtil FP)。 +- [ ] **L2** ⏸ 翻闸 hive 丢 SQL 缓存资格 · `CacheAnalyzer.java:308/316/319`(instanceof HiveScanNode)+ `BindRelation.java:887`(instanceof HMSExternalTable)。**需决策**:加 `ConnectorCapability` 让 `PluginDrivenScanNode`/`PluginDrivenExternalTable` 被识别并恢复缓存 **vs** 登记为 fail-safe 验收偏差(`enable_hive_sql_cache` 默认关)。 +- [ ] **L3** trino 事务从不 commit/close · `TrinoConnectorDorisMetadata.java`(6 处)+ `TrinoScanPlanProvider.java:112`(scan 站有意保持):元数据 6 站 try/finally `commit(txn)`(read-only 廉价)。scan 站不动。 +- [ ] **L4** trino plugin.dir 首胜单例 · `TrinoBootstrap.java:136,316`:单例已存在且 pluginDir 不同时 fail-loud 抛(而非静默用旧);或删 per-catalog 分支只认全局 config。 +- [ ] **L5** trino listTableNames 丢去重 · `TrinoConnectorDorisMetadata.java:98`:收集加 `.distinct()`/LinkedHashSet 保序;不复原 prefix 过滤(冗余)。 +- [ ] **L6** trino guard 字段发布顺序 NPE · `TrinoDorisConnector.java:176`:`doInitialize` 把 guard 字段 `trinoConnector` 赋值移到**最后**(volatile happens-before);或原子发布不可变 holder。 +- [ ] **L7** kerberos UGI.setConfiguration 无 guard · `HadoopKerberosAuthenticator.java:53-59,83,115`:port master first-writer-wins(仅首次 setConfiguration,进程锁序列化,已匹配则 skip+WARN);refresh 分支不重跑 initializeAuthConfig。注:metastore 路 parity,仅 fe-filesystem HDFS 数据路是真变。 +- [ ] **L8** kerberos doAs 吞 interrupt · `HadoopAuthenticator.java:34`:`throw new IOException(e)` 前加 `Thread.currentThread().interrupt();`。一行。 +- [ ] **L9** MC 谓词下推全有全无 · `MaxComputePredicateConverter.java:87-97,117-123`:`convertFilter` 若根是 `ConnectorAnd` 则逐 top-level conjunct 独立 try/catch(丢+log 失败)再 AND 幸存者;**不**对 OR/嵌套 AND 逐子容错。 +- [ ] **L10** ⏸ EXPLAIN 节点名 · `PluginDrivenScanNode.java:170,320`。**需决策**:登记 display-only 验收偏差(cheapest,`CONNECTOR:` 行已披露)**vs** 加连接器声明的 legacy `*_SCAN_NODE` 名(连 `Connector.getLegacyEngineName` 一起,见 D-ENGINE)。注:报告的 `connectorType.toUpperCase()+"_SCAN_NODE"` 一行修法**不够**(hms 会出 `VHMS_SCAN_NODE` 非 `VHIVE_/VICEBERG_`)。 +- [ ] **L11** paimon file_format · `PaimonScanPlanProvider.java:812-813,848-849`:JNI DataSplit + COUNT 路按首数据文件后缀 `getFileFormatBySuffix("/"+dataFiles().get(0).fileName()).orElse(defaultFileFormat)`(仿 native 臂 `:540`/legacy)。注:默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。 +- [ ] **L12** ⏸ selectedPartitionNum 语义 · `PluginDrivenScanNode.java:297-303`。**需决策**:登记为「paimon/iceberg 对齐 MC/hive 的 Nereids-剪枝数」验收偏差(推荐)**vs** 连接器回报 SDK-distinct(重,不推荐)。同步 paimon/iceberg EXPLAIN `partition=N/M` 回归期望。**勿**在通用节点按源分支。 +- [ ] **L13** paimon 嵌套 nullability · `PaimonTypeMapping.java:254,257-259,269-281`:发 4-arg `DataField(id,name,type.copy(isChildNullable),comment)`;ARRAY 元素/MAP value `.copy(...)`。仿 `IcebergSchemaBuilder`/legacy。comment 半已登记 DV-035c。 +- [ ] **L14** paimon ignore_split_type no-op · `PaimonScanPlanProvider.java:407-451`:读 `ignore_split_type`(经 `ConnectorSession.getSessionProperties()`)按被忽略 reader 类型跳 split;或退休变量+登记。 +- [ ] **L15** PAIMON_SCAN_METRICS 悬空 · `SummaryProfile.java:158,218,277`:删三处死引用(P5 已验收弃 paimon FE scan metrics);或加 connector-neutral scan-metrics SPI(feature,非必需)。 +- [ ] **L16** iceberg 缓存偏斜(部分已修) · `IcebergScanPlanProvider.java:1077-1108`:hasSnapshotPin 臂把 field-id dict 建成「pinned schema ∪ requested latest columns」超集(传 requestedLowerNames 而非 `emptyList`),两向都超集;或 fe-core 侧 query-begin pin 解析 pinnedSchema 到 pinned schemaId(对齐 time-travel 臂 `:377-387`)。 +- [ ] **L17** iceberg 同表多版本 version-blind 绑定 · `PluginDrivenMvccExternalTable.java:475-485`:per-reference schema 绑定 version-aware(用 `StatementContext.getSnapshot(table, tableSnapshot, scanParams)` 的 pinnedSchema,fallback latest);与 L16 同根,建议一并。窄触发 + fail-loud。 +- [ ] **L18** iceberg 未知/v3 类型静默 UNSUPPORTED · `IcebergTypeMapping.java:91,143`:两 default 臂改 `throw DorisConnectorException("Cannot transform unknown type: "+...)`,保留显式 TIME/VARIANT UNSUPPORTED;或接受更松并登记。 +- [ ] **L19** partition_columns 魔法键撞名 · `PluginDrivenExternalTable.java:512,703-714` + `IcebergConnectorMetadata.java:409-445`:putAll 前从源 `table.properties()` 剔除保留键;iceberg buildTableSchema 在分区/非分区分支前移除已存在的 `partition_columns`,使非分区表不继承用户 `partition_columns`。 +- [ ] **L20** ⏸ MC EQ `==` · `MaxComputePredicateConverter.java:145-146`:直接 `case EQ: opDesc = "=";` 对齐 SDK/legacy 消除不确定性(推荐);或 live ODPS A/B 确认容忍。顺带补 IN 方向回归测试(P4-3-IN 已修但缺测)。 + +--- + +## ⚪ 设计债跟踪(D-系列;多为 P8 前置或一次性重构,择机) + +> 详见 reverify §4。均为真实设计张力,按现行铁律非「须修 bug」;择机或随 P8。**D-PRUNE 因承载 H1/H3,提前到高危批次。** + +- [ ] **D-PRUNE**(承载 H1/H3,优先)· 抽 `HiveConnectorMetadata:1995-2093` 与 `HudiConnectorMetadata:980-1078` 逐字节相同的 7 方法 EQ/IN 剪枝块到共享 `HmsStylePartitionPruner`(fe-connector-api pushdown 或 fe-connector-metastore-hms util),H1/H3 的 unescape/相对化修复一处落地,防连接器专属分歧。 +- [ ] **D-ENGINE** · `Connector.getLegacyEngineName()`/`getLegacyTableTypeName()` SPI 收口三处引擎名 switch(`PluginDrivenExternalTable.getEngine:1182`/`getEngineTableTypeName:1220`/`CreateTableInfo.pluginCatalogTypeToEngine:927`)。连 L10 EXPLAIN 名可一起。 +- [ ] **D-SHOWCREATE-MASK**(安全) · `Env.java:4897-4907` 插件 PROPERTIES 改走 `new DatasourcePrintableMap<>(props," = ",true,true,hidePassword)`(~5 行,纵深防御;当前无可达泄漏)。 +- [ ] **D-SENSITIVE-KEY**(安全) · `DatasourcePrintableMap.SENSITIVE_KEY:57-75` 的连接器键折进 `registerSensitiveKeys(...)` SPI 聚合;至少 iceberg REST 键移连接器注册。 +- [ ] **D-DML-REGISTRY** · `RowLevelDmlRegistry:37` 加 fail-loud 契约检查(非 iceberg 连接器声明 DELETE/MERGE 时拒绝)+ 刷新过时 javadoc;或连接器提供 transform SPI。 +- [ ] **D-TCCL** · 抽 `Tccl.pin(loader,Runnable)` 到 fe-connector-spi/support,iceberg/paimon/hive(内联 `HiveConnectorMetadata:798-805,832-845`)共用。 +- [ ] **D-HIVECONF** · 合并 hive 三处 `buildHadoopConf`(`HiveConnector:621`/`HiveScanPlanProvider:444`/`HiveConnectorMetadata:967`)为一私有 helper;长期共享 `HadoopConfBuilder`。 +- [ ] **D-FAKESTUB**(测试) · 加共享 `AbstractFakeHmsClient` 测试夹具,hive/hudi ~11 份桩收敛。 +- [ ] **D-MAGICKEY** · SHOW CREATE 子句换结构化 `ConnectorTableDdl`(transform+sort 作数据,fe-core 单一 altitude 渲染);至少把 `partition_columns`/`primary_keys` 提为声明常量。 +- [ ] **D-SPI-FACET** · 4.2 stringly-typed(transform/bucket enum 化)+ 4.3 连接器专属能力收进 capability-discovered facet;登记为验收架构张力。 +- [ ] **D-D2-MICROS** · `ConnectorMvccPartitionView.getNewestUpdateTimeMillis:113` 重命名 `getNewestUpdateMarker()`(去伪单位,零行为变更)。 +- [ ] **D-D3-PATHCONTRACT** · `applyRewriteFileScope:204`/`applyTopnLazyMaterialization:226` 引 `ConnectorFilePath` token / debug 全 schema 断言;或登记验收。 +- [ ] **D-P2-PRECREATE** · `TrinoDorisConnector:78` 急切 preCreateValidation 登记 deviations-log(或改 best-effort)。 +- [ ] **D-P8** · P8 前置(SPI_READY_TYPES 字符串门、data-cache allowlist、getEngine switch、TableType.ICEBERG_EXTERNAL_TABLE 残引、PlanNode instanceof 死臂)统一随 P8/Phase-3 处理。**当前对 hive 均无错误行为**,勿提前动。 + +--- + +## 明确不做(避免误改,详见 reverify §5/§6) + +- **已登记/验收偏差(§5)**:P4-5(DV-018 INSERT 吞 refresh)、P4-6(DV-034 DB errno)、P4-SHOWPART:limit(DV-021)、P4-SHOWPART:where、P6-7(benign 超集)、P6-8(latent 不可达)、P6-10(DV-049③)。→ 保持,除非用户要重开决策。 +- **不适用/已修/parity(§6)**:P0-1/2/3/4、P0-6(×2)、P1-1、P1-3-iceberg 死臂、**P3-hudi COW/MOR(HD-A4 已修)**、**P3b-1(证伪)**、**P3b-3(证伪)**、P4-3-IN(已修,仅缺测→并入 L20)、P5-7(parity)、P6-S:cap-enum、ENGINE-SWITCH:i3(已修)、P8:printNested 死臂。→ **不要据此改代码**。 + +--- + +## 每轮结束更新(滚动) + +> 每完成一条:此处记 ` DONE — 一句话` + 勾总表 + 更新 `HANDOFF.md`。 + +- (尚未开始) From 17a9aaf6272a17e1fa737e932e4b8d07067ccba7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 05:02:14 +0800 Subject: [PATCH 205/334] [fix](catalog) fe-connector-jdbc: disambiguate assertDoesNotThrow overload getWritePlanProvider() returns a non-void value, so the method-reference form Assertions.assertDoesNotThrow(connector::getWritePlanProvider, msg) matched both assertDoesNotThrow(Executable, String) and assertDoesNotThrow(ThrowingSupplier, String), failing testCompile on stricter javac (CI build 991899, pull/65474). Use a block-bodied lambda with no value-returning statement so it is void-compatible only and binds Executable unambiguously; the test only asserts no-throw, so the return value is intentionally discarded. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../apache/doris/connector/jdbc/JdbcDorisConnectorTest.java | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java index 3f5af295ad186c..cb7c49076f52cd 100644 --- a/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java +++ b/fe/fe-connector/fe-connector-jdbc/src/test/java/org/apache/doris/connector/jdbc/JdbcDorisConnectorTest.java @@ -201,8 +201,9 @@ void testGetWritePlanProviderWithoutDriverClassDoesNotThrow() { Map props = new HashMap<>(); props.put(JdbcConnectorProperties.JDBC_URL, "jdbc:postgresql://localhost:5432/test"); JdbcDorisConnector connector = new JdbcDorisConnector(props, testContext()); - Assertions.assertDoesNotThrow(connector::getWritePlanProvider, - "missing driver_class must not NPE inside HikariCP during client initialization"); + Assertions.assertDoesNotThrow(() -> { + connector.getWritePlanProvider(); + }, "missing driver_class must not NPE inside HikariCP during client initialization"); } @Test From 03f4c12dffa012234e126d4844a8bc24f43f2120 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 05:37:34 +0800 Subject: [PATCH 206/334] [fix](catalog) hudi H4: lowercase JNI reader column names Batch-0 (#65185 reverify) H4. After the HMS SPI cutover, a hudi-on-HMS table with mixed-case Avro columns (e.g. Id/Name/Addr) crashed every MOR/JNI split. Root cause: HudiScanPlanProvider.planScan built the JNI reader column list with `.map(Schema.Field::name)` (original case). BE HadoopHudiJniScanner.initRequiredColumnsAndTypes keys hudiColNameToType by these names, then resolves each requiredField with an EXACT lower-case containsKey; mixed-case "Id" misses lower-case "id" -> throw. The two sibling name paths were already lower-cased (avroSchemaToColumns:905, HudiSchemaUtils.buildField:137); legacy HudiScanNode:223 also emits lower-case names. This one path regressed. Fix: extract the inline transform into a package-private static `jniColumnNames(Schema)` that lower-cases each field name (toLowerCase(Locale.ROOT)); column ORDER preserved so the parallel columnTypes list stays positionally aligned. Extraction makes the transform offline-unit-testable (planScan needs a live metaClient). Test: HudiSchemaParityTest.jniColumnNamesAreLowerCased asserts the mixed-case Id/Name/Addr fixture lower-cases. MOR/JNI end-to-end read with mixed-case columns is live-gated (real hudi cluster). Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../connector/hudi/HudiScanPlanProvider.java | 25 +++++++++++++++++-- .../connector/hudi/HudiSchemaParityTest.java | 12 +++++++++ 2 files changed, 35 insertions(+), 2 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 720427a9d31c5c..80d7b04a965e9b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -57,6 +57,7 @@ import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Optional; import java.util.function.Function; @@ -177,8 +178,12 @@ public List planScan( // InternalSchema at the instant, D3) keeps the latest schema — byte-identical. Schema jniSchema = HudiSchemaUtils.resolveJniColumnSchema( schemaResolver, metaClient, avroSchema, hudiHandle.getQueryInstant()); - columnNames = jniSchema.getFields().stream() - .map(Schema.Field::name).collect(Collectors.toList()); + // JNI reader column names MUST be lower-cased: BE HadoopHudiJniScanner.initRequiredColumnsAndTypes + // keys hudiColNameToType by these names, then looks each requiredField up as an EXACT lower-case key + // (mixed-case "Id" would miss lower-case "id" -> throw, crashing every MOR/JNI split). Matches the + // Doris-column path (avroSchemaToColumns:905) and the native history_schema_info dict + // (HudiSchemaUtils.buildField:137); legacy HudiScanNode:223 also emits lower-case names. + columnNames = jniColumnNames(jniSchema); columnTypes = jniSchema.getFields().stream() .map(f -> HudiTypeMapping.toHiveTypeString(f.schema())) .collect(Collectors.toList()); @@ -827,6 +832,22 @@ private static String unescapePathName(String path) { return sb.toString(); } + /** + * The JNI (MOR-realtime) reader's required-column names, LOWER-CASED. BE + * {@code HadoopHudiJniScanner.initRequiredColumnsAndTypes} builds {@code hudiColNameToType} keyed by these + * names and resolves each requiredField with an EXACT lower-case {@code containsKey}, so a mixed-case Avro + * name ({@code "Id"}) would miss the lower-case slot ({@code "id"}) and throw, crashing every MOR/JNI split. + * Byte-consistent with the Doris-column casing ({@code HudiConnectorMetadata.avroSchemaToColumns} / + * {@code HudiSchemaUtils.buildField}) and legacy {@code HudiScanNode} (which emits lower-case Column names). + * Column ORDER is preserved so the parallel {@code columnTypes} list stays positionally aligned. Package- + * private static for offline unit testing (planScan itself needs a live metaClient). + */ + static List jniColumnNames(Schema jniSchema) { + return jniSchema.getFields().stream() + .map(f -> f.name().toLowerCase(Locale.ROOT)) + .collect(Collectors.toList()); + } + /** * Detect file format from file path suffix. Package-private static so the ported incremental relations * ({@code COWIncrementalRelation}) stamp the required explicit {@code fileFormat} on their native diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java index 18df581efdba11..9ee408c97f2ede 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiSchemaParityTest.java @@ -148,6 +148,18 @@ public void avroSchemaToColumnsPreservesMetaCommitTimeAsBindableStringColumn() { "_hoodie_commit_time must be STRING so the window compare is lexicographic over Hudi instants"); } + @Test + public void jniColumnNamesAreLowerCased() { + // H4: HudiScanPlanProvider.planScan feeds these names into THudiFileDesc.column_names, which BE + // (HadoopHudiJniScanner.initRequiredColumnsAndTypes) keys a map by and then resolves each requiredField + // as an EXACT lower-case containsKey. A mixed-case Avro name ("Id") missing the lower-case slot ("id") + // throws and crashes every MOR/JNI split, so the JNI column-name list MUST be lower-cased — consistent + // with avroSchemaToColumns and legacy HudiScanNode. RED before the fix: raw-case "Id"/"Name"/"Addr". + Assertions.assertEquals( + Arrays.asList("id", "name", "price", "event_date", "created_at", "tags", "props", "addr"), + HudiScanPlanProvider.jniColumnNames(schema())); + } + @Test public void testTopLevelNameLoweredButNestedStructNamePreserved() { List columns = HudiConnectorMetadata.avroSchemaToColumns(schema()); From e398278018dcb8487f3716f935f2644d5df3d947 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 05:38:44 +0800 Subject: [PATCH 207/334] [doc](catalog) #65185 batch-0: H4 design + track-off; record H1/H2 hive scope MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit H4 DONE (03f4c12dffa). Records the batch-0 scope decision: adversarial verification confirmed H1/H2 are NOT hudi-only — HiveConnectorMetadata's byte-identical prune block silently drops rows too. User signed off on "fix both copies in place" (no shared-helper extraction); D-PRUNE stays deferred as tracked design debt. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 8 +-- plan-doc/tasks/designs/FIX-H4-design.md | 58 ++++++++++++++++++++++ 2 files changed, 63 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-H4-design.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 64fb54656b30b0..a4888cec48897b 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -28,7 +28,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 1 | **H1** | 🔴高 | hudi | 分区名不 unescape→丢行 | ⬜ | ⬜ | ⬜ | ⬜ | | 2 | **H2** | 🔴高 | hudi | datetime 分区谓词 ISO→0 行 | ⬜ | ⬜ | ⬜ | ⬜ | | 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ⬜ | ⬜ | ⬜ | ⬜ | -| 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ⬜ | ⬜ | ⬜ | ⬜ | +| 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | | 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | | 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ⬜ | ⬜ | ⬜ | ⬜ | | 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | @@ -100,7 +100,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:非 hive-style 表断言「带 filter 分区集 == 不带 filter」(RED:带 filter 0 split)。live e2e gated。 ### H4 — hudi 混大小写 Avro → JNI/MOR reader 崩 · reverify §2 H4 -- [ ] **H4**(最小,建议先做) +- [x] **H4**(最小,建议先做) · DONE `03f4c12dffa`(设计 `designs/FIX-H4-design.md`) - **现码**:`HudiScanPlanProvider.planScan:180-181` `.map(Schema.Field::name)`(原始大小写)→ `HudiScanRange:220` → BE `HadoopHudiJniScanner.initRequiredColumnsAndTypes:227-229` 对 lowercase `requiredField` 精确 `containsKey`→throw。 - **Fix**:`planScan:181` 改 `.map(f -> f.name().toLowerCase(java.util.Locale.ROOT))`(与 `HudiConnectorMetadata.avroSchemaToColumns:905`、`HudiSchemaUtils:137` 一致);列类型顺序不变。 - **Files**:`HudiScanPlanProvider.java`。 @@ -227,4 +227,6 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) > 每完成一条:此处记 ` DONE — 一句话` + 勾总表 + 更新 `HANDOFF.md`。 -- (尚未开始) +**⭐ 批次 0 范围决策(用户 2026-07-11 签字)**：对抗 agent 已证实 **H1/H2 不是 hudi 独有——`HiveConnectorMetadata` 逐字节相同的剪枝块(parsePartitionName/matchesPredicates/extractLiteralValue)同样静默丢行**(fe-core 算出正确 typed 分区集但 hive `planScan` 丢弃、只认 applyFilter 那份 bug 结果)。用户选 **「两份就地各修」**(选项 2)：H1/H2 在 hive 和 hudi **两份副本各自就地修**(不抽共享 helper),H3/H4 仅 hudi。**D-PRUNE 抽取继续延后**为 ⚪ 设计债(reverify §4 DUPLICATION:partition-prune)。 + +- **H4** DONE `03f4c12dffa` — lowercase JNI reader 列名(hudi-only;`jniColumnNames` helper + UT)。 diff --git a/plan-doc/tasks/designs/FIX-H4-design.md b/plan-doc/tasks/designs/FIX-H4-design.md new file mode 100644 index 00000000000000..dc83ea929ed26d --- /dev/null +++ b/plan-doc/tasks/designs/FIX-H4-design.md @@ -0,0 +1,58 @@ +# FIX-H4 — hudi 混大小写 Avro 列名 → JNI/MOR reader 崩 + +> 来源：`plan-doc/task-list-65185-reverify-fixes.md` §H4；证据 `plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §2 H4。 +> 批次 0 第 1 条（最小、hudi-only）。 + +## Problem + +翻闸后 hudi-on-HMS 表若列名混大小写（如 `Id/Name/Addr`）**且**走 MOR-带-log 或 `force_jni`，每个 JNI split 硬崩、整查询失败。 + +## Root Cause + +`HudiScanPlanProvider.planScan:180-181` 用 `.map(Schema.Field::name)` 取 JNI reader 列名列表（**原始大小写**），经 `HudiScanRange` → `THudiFileDesc.column_names` 传给 BE。BE `HadoopHudiJniScanner.initRequiredColumnsAndTypes` 用**原始大小写** key 建 `hudiColNameToType`，再对每个 **lowercase** 的 `requiredField` 做精确 `containsKey`——`Id/Name/Addr` 表下 lowercase `id` 不在 `{Id,Name,Addr}` → `throw IllegalArgumentException`。 + +对比：同类的两条列名路径**已** lowercase——Doris 列 schema（`HudiConnectorMetadata.avroSchemaToColumns:905` `toLowerCase(ROOT)`）和 native `history_schema_info` 字典（`HudiSchemaUtils.buildField:137`）。唯 JNI 列名列表漏改。legacy `HudiScanNode:223` 发的是 lowercase（`map(Column::getName)`，Column 名已在 `HMSExternalTable:749` lowercase）。故这是**回归**。 + +## Design + +`planScan` 的 JNI 列名 `.map(Schema.Field::name)` 改为 `.map(f -> f.name().toLowerCase(Locale.ROOT))`，与 `avroSchemaToColumns:905` / `HudiSchemaUtils:137` 一致。列**类型**顺序不变（仍按 `jniSchema.getFields()` 顺序），名↔类型位置对应不变；Hive ObjectInspector 大小写不敏感，文件读仍解析。 + +为可离线单测（`planScan` 需 live metaClient，无法离线跑），把该 inline transform 抽成包私有静态 helper `jniColumnNames(Schema)`，最小 seam（仿 `parsePartitionValues`/`chooseJniSchema` 已有范式）。 + +## Implementation Plan + +`HudiScanPlanProvider.java`： +1. 新增 `import java.util.Locale;`（List 之后、Map 之前，checkstyle 顺序）。 +2. 新增包私有静态 helper： + ```java + static List jniColumnNames(Schema jniSchema) { + return jniSchema.getFields().stream() + .map(f -> f.name().toLowerCase(Locale.ROOT)) + .collect(Collectors.toList()); + } + ``` +3. `planScan:180-181` 的 `columnNames = jniSchema.getFields().stream().map(Schema.Field::name)...` 改为 `columnNames = jniColumnNames(jniSchema);`（`columnTypes` 那段不动）。 + +## Risk Analysis + +- 仅改列名大小写、不改顺序/类型 → 名↔类型对应不变；Hive OI 大小写不敏感，文件读不受影响。 +- 混大小写但**纯 COW/全小写**表不受此路径影响（`avroSchemaToColumns` 早已 lowercase Doris 列；仅 JNI column_names 这一路曾漏）。 +- 无 fe-core 改动、connector-agnostic 铁律无关。 + +## Test Plan + +### Unit Tests +`HudiSchemaParityTest`（复用其 `Id/Name/Addr` 混大小写 `SCHEMA_JSON` fixture）加一条： +```java +Assertions.assertEquals( + Arrays.asList("id","name","price","event_date","created_at","tags","props","addr"), + HudiScanPlanProvider.jniColumnNames(schema())); +``` +RED（改前）：`jniColumnNames` 不存在 / 若直接断言旧 inline 会得 `Id/Name/...` → 失败。GREEN：lowercase 列表。 + +### E2E Tests +MOR-带-log / `force_jni` + 混大小写列的读回归 = **live-gated**（需真 hudi 集群），显式登记 gated，不静默略过（Rule 12）。 + +## 守门结果（DONE，commit `03f4c12dffa`） + +`mvn -o -pl :fe-connector-hudi -am test -Dtest=HudiSchemaParityTest` → **BUILD SUCCESS**；`HudiSchemaParityTest` 5 run / 0 fail / 0 err / 0 skip（新 `jniColumnNamesAreLowerCased` + 4 既有）；`You have 0 Checkstyle violations`（全模块）。测试可 RED：fixture 混大小写 `Id/Name/Addr` vs 期望 lowercase，去掉 `.toLowerCase(Locale.ROOT)` 即变红。MOR/JNI 混大小写读端到端 = live-gated（真 hudi 集群）。 From 39a279e7c26e67bf26b12711ce63b56b63f34bca Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 05:48:31 +0800 Subject: [PATCH 208/334] [fix](catalog) hive+hudi H1: unescape partition value in prune matching Batch-0 (#65185 reverify) H1. After the HMS SPI cutover, a partitioned hive-on-HMS or hudi-on-HMS table silently DROPPED rows when a partition value contained a Hive-escaped character (e.g. ':' stored as '%3A') and the query carried an EQ/IN predicate on that column. Root cause: the connector partition-pruning decision uses a byte-identical block in both HiveConnectorMetadata and HudiConnectorMetadata. parsePartitionName stored the raw substring after '=' WITHOUT unescaping, while the predicate literal (extractLiteralValue) is unescaped, so matchesPredicates' string compare (escaped 'US%3ACA' vs unescaped 'US:CA') never matched -> the real partition was pruned out -> silent row loss. fe-core computes a correct typed requiredPartitions but the connector planScan ignores it and honors only applyFilter's prunedPartitions, so this pruning is authoritative. The sibling parse paths already unescape (HudiScanPlanProvider.parsePartitionValues, HiveWriteUtils.toPartitionValues); this pruning path regressed. Adversarial verification confirmed BOTH connectors are affected (not hudi-only). Fix (both copies in place, per scope decision): unescape the VALUE in each connector's parsePartitionName via its own same-package unescapePathName (widened private->package-private). The key (column name) is left as-is. parsePartitionName widened to package-private static for direct offline unit testing (mirrors the already-tested parsePartitionValues). Tests: direct parsePartitionName unescape assertions in HudiPartitionValuesTest and HiveConnectorMetadataPartitionPruningTest (durable across the later H3 restructure); plus a hive end-to-end applyFilter test proving an escaped partition value prunes-in instead of dropping. Escaped-value read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../connector/hive/HiveConnectorMetadata.java | 10 +++++-- .../doris/connector/hive/HiveWriteUtils.java | 2 +- ...ConnectorMetadataPartitionPruningTest.java | 27 +++++++++++++++++++ .../connector/hudi/HudiConnectorMetadata.java | 10 +++++-- .../connector/hudi/HudiScanPlanProvider.java | 2 +- .../hudi/HudiPartitionValuesTest.java | 14 ++++++++++ 6 files changed, 59 insertions(+), 6 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 28b07eb9fbb724..6f54212d465ee4 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -2066,14 +2066,20 @@ private List prunePartitionNames(List allPartNames, return matched; } - private Map parsePartitionName(String partName, + static Map parsePartitionName(String partName, List partKeyNames) { Map values = new HashMap<>(); String[] parts = partName.split("/"); for (String part : parts) { int eq = part.indexOf('='); if (eq > 0) { - values.put(part.substring(0, eq), part.substring(eq + 1)); + // Unescape the VALUE: HMS get_partition_names returns Hive-escaped names (e.g. "%3A" for ':'). + // The predicate literal side (extractLiteralValue) is unescaped, so matchesPredicates' string + // compare needs the value unescaped too — otherwise an escaped partition value silently drops + // rows. Mirrors the sibling partition-value parse (HiveWriteUtils.toPartitionValues) and legacy + // FileUtils.unescapePathName. The key is a column name (never escaped), left as-is. + values.put(part.substring(0, eq), + HiveWriteUtils.unescapePathName(part.substring(eq + 1))); } } return values; diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java index b3f7ea70be0bba..8e4c46dee4c92e 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveWriteUtils.java @@ -204,7 +204,7 @@ static List toPartitionValues(String partitionName) { * {@code org.apache.hadoop.hive.common.FileUtils.unescapePathName}, inlined to avoid a hive-common * dependency. */ - private static String unescapePathName(String path) { + static String unescapePathName(String path) { StringBuilder sb = new StringBuilder(); for (int i = 0; i < path.length(); i++) { char c = path.charAt(i); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java index ccde4c41f635ef..7204128ae0f2b0 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java @@ -136,6 +136,33 @@ public void testUnpartitionedTableIsNotTouched() { Assertions.assertFalse(result.isPresent()); } + @Test + public void parsePartitionNameUnescapesValues() { + // H1 (unit, durable): the pruning-decision parse must unescape values the same way the sibling + // HiveWriteUtils.toPartitionValues does, or an escaped partition value never string-equals the + // unescaped predicate literal. RED before the fix: "US%3ACA". + Map values = HiveConnectorMetadata.parsePartitionName( + "code=US%3ACA", Collections.singletonList("code")); + Assertions.assertEquals("US:CA", values.get("code"), "colon-escaped value must be decoded"); + } + + @Test + public void testEscapedPartitionValuePrunesInsteadOfDropping() { + // H1 (end-to-end via applyFilter): a partition value with a Hive-escaped char (":" stored as "%3A") + // must still match its unescaped predicate literal. RED before the fix: both escaped names fail the + // string compare -> the pruned set is EMPTY, dropping the real partition (silent row loss). + List escaped = Arrays.asList("code=US%3ACA", "code=EU%3ADE"); + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(escaped), Collections.emptyMap(), new FakeConnectorContext()); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.singletonList("code")) + .build(); + Optional> result = + metadata.applyFilter(null, handle, new ConnectorFilterConstraint(eq("code", "US:CA"))); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals(Collections.singletonList("code=US%3ACA"), prunedLocations(result)); + } + // ===== helpers ===== private Optional> applyFilter( diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 66dc9e861101f3..80f3ea81517484 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -1051,14 +1051,20 @@ private List prunePartitionNames(List allPartNames, return matched; } - private Map parsePartitionName(String partName, + static Map parsePartitionName(String partName, List partKeyNames) { Map values = new HashMap<>(); String[] parts = partName.split("/"); for (String part : parts) { int eq = part.indexOf('='); if (eq > 0) { - values.put(part.substring(0, eq), part.substring(eq + 1)); + // Unescape the VALUE: HMS get_partition_names returns Hive-escaped names (e.g. "%3A" for ':'). + // The predicate literal side (extractLiteralValue) is unescaped, so matchesPredicates' string + // compare needs the value unescaped too — otherwise an escaped partition value silently drops + // rows. Mirrors the sibling scan-side parse (HudiScanPlanProvider.parsePartitionValues) and + // legacy FileUtils.unescapePathName. The key is a column name (never escaped), left as-is. + values.put(part.substring(0, eq), + HudiScanPlanProvider.unescapePathName(part.substring(eq + 1))); } } return values; diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 80d7b04a965e9b..ecdb1fcd1af8a0 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -810,7 +810,7 @@ private static String escapePathName(String value) { * {@code org.apache.hadoop.hive.common.FileUtils.unescapePathName} (identical to the fe-connector-hive * inlined copy in {@code HiveWriteUtils}), so the connector needs no hive-common dependency. */ - private static String unescapePathName(String path) { + static String unescapePathName(String path) { StringBuilder sb = new StringBuilder(); for (int i = 0; i < path.length(); i++) { char c = path.charAt(i); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java index 26ab443398731d..3ae89e77b33922 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java @@ -106,6 +106,20 @@ public void multiColumnFragmentCountMismatchFailsLoud() { "the failure must name the offending partition path"); } + @Test + public void parsePartitionNameUnescapesValuesForPruning() { + // H1: the PRUNING-decision parse (HudiConnectorMetadata.parsePartitionName, fed the ESCAPED HMS + // get_partition_names output) must unescape values the same way the scan-side parsePartitionValues does. + // Otherwise an escaped partition value ("code=US%3ACA") never string-equals the unescaped predicate + // literal ("US:CA") in matchesPredicates, so the real partition is pruned OUT -> silent row loss. RED + // before the fix: values stay "US%3ACA" / "a%2Fb". + Map values = HudiConnectorMetadata.parsePartitionName( + "code=US%3ACA/kind=a%2Fb", Arrays.asList("code", "kind")); + + Assertions.assertEquals("US:CA", values.get("code"), "colon-escaped value must be decoded"); + Assertions.assertEquals("a/b", values.get("kind"), "slash-escaped value must be decoded"); + } + @Test public void emptyPartitionKeysReturnsEmptyForUnpartitionedTable() { // Unpartitioned tables reach here with an empty key list and an empty path; the result must be empty From e9a187541fdf69a3569222a7cb12a27f86dd02fc Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 05:49:12 +0800 Subject: [PATCH 209/334] [doc](catalog) #65185 batch-0: H1 design + track-off H1 DONE (39a279e7c26). Unescape partition value in the prune-matching parse in both hive and hudi copies. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 5 +- plan-doc/tasks/designs/FIX-H1-design.md | 79 ++++++++++++++++++++++ 2 files changed, 82 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-H1-design.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index a4888cec48897b..66c13d9a272d9f 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -25,7 +25,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | # | id | 严重度 | 模块 | 一句话 | 设计 | 实现 | build+UT | 状态 | |---|----|-------|------|--------|------|------|----------|------| -| 1 | **H1** | 🔴高 | hudi | 分区名不 unescape→丢行 | ⬜ | ⬜ | ⬜ | ⬜ | +| 1 | **H1** | 🔴高 | hudi**+hive** | 分区名不 unescape→丢行 | ✅ | ✅ | ✅ | ✅ | | 2 | **H2** | 🔴高 | hudi | datetime 分区谓词 ISO→0 行 | ⬜ | ⬜ | ⬜ | ⬜ | | 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ⬜ | ⬜ | ⬜ | ⬜ | | 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | @@ -78,7 +78,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) ## 🔴 高危(H1–H4) ### H1 — hudi 分区名不 unescape → 丢行 · reverify §2 H1 -- [ ] **H1** +- [x] **H1**(hive+hudi 两份) · DONE `39a279e7c26`(设计 `designs/FIX-H1-design.md`) - **现码**:`fe-connector-hudi/.../HudiConnectorMetadata.java:1054-1064`(`parsePartitionName` 无 unescape)+ `:1067-1078`(`matchesPredicates` 裸字符串比);候选名 `:247` ← `ThriftHmsClient.listPartitionNames:214-218`(原样 Hive 转义)。 - **Fix**:存值前 unescape(复用 `HudiScanPlanProvider.unescapePathName`,提升可见性;或 Hive `FileUtils.unescapePathName`),对齐 legacy `HudiExternalMetaCache.loadPartitionNames:231`。**首选落在 `D-PRUNE` 抽出的共享 helper**。 - **Files**:`HudiConnectorMetadata.java`(或共享 `HmsStylePartitionPruner`)。 @@ -230,3 +230,4 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) **⭐ 批次 0 范围决策(用户 2026-07-11 签字)**：对抗 agent 已证实 **H1/H2 不是 hudi 独有——`HiveConnectorMetadata` 逐字节相同的剪枝块(parsePartitionName/matchesPredicates/extractLiteralValue)同样静默丢行**(fe-core 算出正确 typed 分区集但 hive `planScan` 丢弃、只认 applyFilter 那份 bug 结果)。用户选 **「两份就地各修」**(选项 2)：H1/H2 在 hive 和 hudi **两份副本各自就地修**(不抽共享 helper),H3/H4 仅 hudi。**D-PRUNE 抽取继续延后**为 ⚪ 设计债(reverify §4 DUPLICATION:partition-prune)。 - **H4** DONE `03f4c12dffa` — lowercase JNI reader 列名(hudi-only;`jniColumnNames` helper + UT)。 +- **H1** DONE `39a279e7c26` — hive+hudi 剪枝 `parsePartitionName` unescape 值(两份就地各修;widen `unescapePathName`;直接单测跨 H3 稳定 + hive e2e applyFilter 测)。 diff --git a/plan-doc/tasks/designs/FIX-H1-design.md b/plan-doc/tasks/designs/FIX-H1-design.md new file mode 100644 index 00000000000000..292f252be47e19 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-H1-design.md @@ -0,0 +1,79 @@ +# FIX-H1 — 分区名不 unescape → 静默丢行（hive + hudi 两份就地各修） + +> 来源：`task-list-65185-reverify-fixes.md` §H1；证据 reverify §2 H1 + §4 DUPLICATION:partition-prune。 +> 批次 0 第 2 条。**范围决策**：对抗 agent 证实 H1 **不是 hudi 独有**——`HiveConnectorMetadata` 逐字节相同的剪枝块同样丢行。用户签字 **「两份就地各修」**（不抽共享 helper）。 + +## Problem + +翻闸后，对 hive-on-HMS 或 hudi-on-HMS 分区表，若**分区值含 Hive 转义字符**（`:`→`%3A`、`/`→`%2F`、`%`→`%25`、以及 `"#'*/=?\{[]^` 等；注：空格 **不**被 Hive 转义）且带该列的 `=`/`IN` 谓词，**静默丢行**（对真实存在的值返回 0 行、无报错）。 + +例：分区列 `code`（STRING），HMS 名 `code=US%3ACA`，谓词 `code='US:CA'` → 剪掉该分区 → 0 行。 + +## Root Cause（对 HEAD 核实） + +两连接器各有一份**逐字节相同**的剪枝块（reverify §4 DUPLICATION:partition-prune）： +- 候选名来自 `hmsClient.listPartitionNames`（`ThriftHmsClient` 原样转发 HMS `get_partition_names`，返回 **Hive 转义**名）。 +- `parsePartitionName`（hudi `HudiConnectorMetadata:1054-1065` / hive `HiveConnectorMetadata:2069-2080`）存 `part.substring(eq+1)` —— **原样转义值，无 unescape**。 +- `matchesPredicates`（hudi `:1067` / hive `:2082`）裸 `allowedValues.contains(actualValue)` 字符串比；谓词侧 `extractLiteralValue` 返回 **未转义** 字面量。 +- 转义值 `US%3ACA` ≠ 未转义 `US:CA` → 该分区落选 → `matched < all`（非全命中，`applyFilter` 的 bail 不触发）→ prunedPartitions 为错误子集 → 丢行。 + +对比：两连接器的**同名兄弟路径已 unescape** —— hudi 扫描侧 `HudiScanPlanProvider.parsePartitionValues:723`（`unescapePathName`）、hive fe-core 分区项路 `HiveWriteUtils.toPartitionValues`（`unescapePathName`）/ `CachingHmsClient.toPartitionValues`（`FileUtils.unescapePathName`）。唯**剪枝决策**这条 `parsePartitionName` 漏。legacy 经 Nereids typed 剪枝 + 只发 Hive-canonical 原文，从不转义比对。故 **回归**。 + +fe-core 其实算出正确 typed `requiredPartitions`，但连接器 `planScan` 只认 applyFilter 的 `prunedPartitions`（丢弃 requiredPartitions）→ 连接器剪枝是权威，其 bug 直接丢行（对抗 agent 已 walk 全链确认 hive+hudi 皆真）。 + +## Design + +**两份就地各修**：在**每个**连接器的 `parsePartitionName` 存值前 unescape，复用**本连接器同包**已有的 `unescapePathName`（与其兄弟 parse 路径一致；连接器不跨 import fe-core、亦不跨连接器互 import）。 + +- 键（分区列名）不 unescape（列名非转义；与 `parsePartitionValues` 一致——只 unescape 值）。 +- 为可离线单测（对齐已有 `HudiPartitionValuesTest` 直接测静态 parse helper 的范式，Rule 11）：把 `parsePartitionName` 从 `private` 提为**包私有 `static`**（纯函数、不用实例态），`unescapePathName` 从 `private static` 提为**包私有 static**。**每份只动 2 个方法**（parsePartitionName + unescapePathName）。 + +> H1/H3 纠缠说明：H3 稍后会重构 **hudi** 的 `applyFilter`（改分区源 + 相对化），届时非-hive-sync 臂改走 `parsePartitionValues`（本就 unescape），hive-sync 臂仍用 `parsePartitionName`（H1 修的）。故 H1 的 hudi 修在 hive-sync 臂长期有效；**用直接单测 `parsePartitionName`**（不测 applyFilter 全链）使 H1 测试**跨 H3 稳定**（applyFilter 级测试会被 H3 改语义）。hive 的 `applyFilter` H3 不动，稳定。 + +## Implementation Plan + +### hudi（`HudiConnectorMetadata.java` + `HudiScanPlanProvider.java`） +1. `HudiScanPlanProvider.unescapePathName`：`private static` → `static`。 +2. `HudiConnectorMetadata.parsePartitionName`：`private` → `static`；值改 + `HudiScanPlanProvider.unescapePathName(part.substring(eq + 1))`。 + +### hive（`HiveConnectorMetadata.java` + `HiveWriteUtils.java`） +3. `HiveWriteUtils.unescapePathName`：`private static` → `static`。 +4. `HiveConnectorMetadata.parsePartitionName`：`private` → `static`；值改 + `HiveWriteUtils.unescapePathName(part.substring(eq + 1))`。 + +（`prunePartitionNames` 内 `parsePartitionName(...)` 调用点：静态方法从实例上下文调用，编译不变，无需改。） + +## Risk Analysis + +| Risk | 处置 | +|---|---| +| unescape 误伤非转义值 | `unescapePathName` 只解码合法 `%XX`；`year=2024` 等无 `%` 值不变；含真实 `%`+2 hex 的数据值会被解码——但**与 legacy/兄弟路径 parity**（都 unescape），非本 fix 引入。 | +| 列名被 unescape | 只 unescape 值（`substring(eq+1)`），键不动。 | +| static 化破坏调用点 | 同类静态调用编译不变；checkstyle 无 static-import（是方法调用非 import）。 | +| 与 H2 交互 | 正交：H1 修 HMS-名侧转义，H2 修谓词侧 datetime 渲染；datetime 分区两者**都需**且**复合**（H1 unescape 后 `10:00:00`，H2 渲染 `2024-01-01 10:00:00` → 命中）。 | +| fe-core 铁律 | 仅连接器内改；无 fe-core 变更、无源名判别、无属性解析。import-gate 须 exit 0。 | + +## Test Plan + +### Unit Tests（直接测 `parsePartitionName`，跨 H3 稳定） +- **hudi** `HudiPartitionValuesTest` 加 `parsePartitionNameUnescapesValues`： + `HudiConnectorMetadata.parsePartitionName("code=US%3ACA/kind=a%2Fb", Arrays.asList("code","kind"))` + == `{code:"US:CA", kind:"a/b"}`。RED（改前）：`{code:"US%3ACA", kind:"a%2Fb"}`。 +- **hive** `HiveConnectorMetadataPartitionPruningTest` 加 `parsePartitionNameUnescapesValues`： + `HiveConnectorMetadata.parsePartitionName("code=US%3ACA", Collections.singletonList("code"))` + == `{code:"US:CA"}`。RED：`{code:"US%3ACA"}`。 +- **hive** 额外 applyFilter 级集成测试（hive applyFilter 稳定）：2 分区 `code=US%3ACA`/`code=EU%3ADE` + 谓词 `code='US:CA'` → 命中 `code=US%3ACA` 一条（pruned 非空且正确）。RED：命中集为空（两分区都因转义不等被剪）。 + +### E2E Tests +含转义字符分区值的 hive/hudi 分区表带谓词读回归 = **live-gated**（真集群）；显式登记 gated（Rule 12）。 + +## 决策类型 +明确修复（用户签字「两份就地各修」）。连接器局部、无 SPI 变更、与 legacy + 兄弟 unescape 路径 parity。D-PRUNE 抽取延后（reverify §4）。 + +## 守门结果（DONE，commit `39a279e7c26`） + +- hudi：`mvn -o -pl :fe-connector-hudi -am test -Dtest=HudiPartitionValuesTest,HudiPartitionPruningTest` → **BUILD SUCCESS**；17 run / 0 fail（`HudiPartitionValuesTest` 9 含新 `parsePartitionNameUnescapesValuesForPruning`、`HudiPartitionPruningTest` 8）；checkstyle 全 0。 +- hive：`mvn -o -pl :fe-connector-hive -am test -Dtest=HiveConnectorMetadataPartitionPruningTest` → **BUILD SUCCESS**；10 run / 0 fail（+2 新：`parsePartitionNameUnescapesValues`、`testEscapedPartitionValuePrunesInsteadOfDropping`）；checkstyle 全 0。 +- import-gate：`GATE_RC=0`（仅已知 `HiveVersionUtil` vendored 误报被 `is_vendored()` 跳过）。 +- 测试可 RED：断言 decoded `US:CA`/`a/b`，改前 parsePartitionName 出 `US%3ACA`/`a%2Fb` → 变红。含转义值分区表读端到端 = live-gated。 From cf540eebc3cebb134077f37921606db44e7ac6f1 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 06:03:56 +0800 Subject: [PATCH 210/334] [fix](catalog) hive+hudi H2: render datetime partition literal as Hive text Batch-0 (#65185 reverify) H2. After the HMS SPI cutover, an EQ/IN predicate on a DATETIME/TIMESTAMP partition column pruned a hive-on-HMS or hudi-on-HMS table to 0 rows (silent total row loss). DATE and STRING partition columns were unaffected. Root cause: the byte-identical prune block in both HiveConnectorMetadata and HudiConnectorMetadata renders the predicate literal via String.valueOf. ExprToConnectorExpressionConverter.convertDateLiteral wraps a non-DATE datetime literal as a LocalDateTime, and String.valueOf(LocalDateTime) == toString() == ISO "2024-01-01T10:00" (T separator, dropped zero seconds), which never string-equals the Hive-canonical stored partition value "2024-01-01 10:00:00" in matchesPredicates -> every partition is pruned out. Legacy pruned via typed Nereids comparison, never strings. Adversarial verification confirmed BOTH connectors are affected, and a real Hive fixture (run17.hql: `time_par timestamp` stored as time_par=2023-01-01 01%3A30%3A00) confirms the stored scale-0 form is exactly "yyyy-MM-dd HH:mm:ss". Fix (both copies in place, per scope decision): in extractLiteralValue, render a LocalDateTime value via a new package-private static hiveDateTimeString helper -> "yyyy-MM-dd HH:mm:ss" (space separator, full seconds), appending trailing-zero-trimmed microseconds only when present. All other value types keep String.valueOf; null stays null; LocalDate (DATE columns) is unaffected. No timezone conversion is involved (unlike the MaxCompute datetime-pushdown fix), so there is no ZoneId.of crash risk. This composes with H1 (H1 unescapes the stored value, H2 renders the predicate) so a datetime partition survives pruning. Tests: direct hiveDateTimeString render assertions (whole second, midnight, seconds, microseconds, trimmed) in both connectors; plus a hive end-to-end applyFilter test proving an escaped datetime partition value (H1) prunes-in against a datetime literal (H2). DATETIME partition read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../connector/hive/HiveConnectorMetadata.java | 49 +++++++++++++++++-- ...ConnectorMetadataPartitionPruningTest.java | 37 ++++++++++++++ .../connector/hudi/HudiConnectorMetadata.java | 49 +++++++++++++++++-- .../hudi/HudiPartitionValuesTest.java | 22 +++++++++ 4 files changed, 149 insertions(+), 8 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 6f54212d465ee4..495eddfbfa0f1b 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -74,6 +74,8 @@ import org.apache.logging.log4j.Logger; import java.nio.charset.StandardCharsets; +import java.time.LocalDateTime; +import java.time.format.DateTimeFormatter; import java.util.ArrayList; import java.util.Base64; import java.util.Collections; @@ -119,6 +121,12 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // identical 65533. private static final int MAX_VARCHAR_LENGTH = 65533; + // Hive-canonical partition text for a DATETIME/TIMESTAMP literal: space separator, full seconds. See + // hiveDateTimeString / extractLiteralValue (H2: String.valueOf(LocalDateTime) would yield ISO "…T…" and drop + // zero seconds, never matching the stored Hive partition value). + private static final DateTimeFormatter HIVE_DATETIME_SECONDS_FORMAT = + DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"); + // Hive input formats eligible for Top-N lazy materialization, replicating legacy // HMSExternalTable.SUPPORTED_HIVE_TOPN_LAZY_FILE_FORMATS (parquet/orc only). The match is on the EXACT // input-format class (not a substring), so a HoodieParquetInputFormatBase hive table — which contains @@ -2043,11 +2051,44 @@ private String extractColumnName(ConnectorExpression expr) { } private String extractLiteralValue(ConnectorExpression expr) { - if (expr instanceof ConnectorLiteral) { - Object val = ((ConnectorLiteral) expr).getValue(); - return val != null ? String.valueOf(val) : null; + if (!(expr instanceof ConnectorLiteral)) { + return null; } - return null; + Object val = ((ConnectorLiteral) expr).getValue(); + if (val == null) { + return null; + } + if (val instanceof LocalDateTime) { + // H2: a DATETIME/TIMESTAMP partition literal arrives as a LocalDateTime (DATE arrives as LocalDate). + // String.valueOf would call toString() -> ISO "2024-01-01T10:00" (T separator, dropped zero seconds), + // which never string-equals the Hive-canonical stored partition value "2024-01-01 10:00:00" in + // matchesPredicates -> the whole table prunes to 0 rows. Render Hive-canonical text instead. + return hiveDateTimeString((LocalDateTime) val); + } + return String.valueOf(val); + } + + /** + * Renders a DATETIME/TIMESTAMP literal as Hive-canonical partition text: {@code yyyy-MM-dd HH:mm:ss} (space + * separator, full seconds), appending trailing-zero-trimmed microseconds only when a sub-second part is + * present. Matches the stored Hive partition value for a scale-0 DATETIME partition (the only realistic case) + * so the pruning string-compare hits. Package-private static for offline unit testing. + * + *

    Contract: {@code convertDateLiteral} produces microsecond precision (nano = micros*1000), so the nano is + * always a multiple of 1000; a sub-microsecond nano (unreachable on the pruning path) would be truncated. + */ + static String hiveDateTimeString(LocalDateTime ldt) { + String base = ldt.format(HIVE_DATETIME_SECONDS_FORMAT); + int nano = ldt.getNano(); + if (nano == 0) { + return base; + } + String micros = String.format("%06d", nano / 1000); + int end = micros.length(); + while (end > 1 && micros.charAt(end - 1) == '0') { + end--; + } + return base + "." + micros.substring(0, end); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java index 7204128ae0f2b0..0dc1484beb5b7d 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java @@ -35,6 +35,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.time.LocalDateTime; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -163,6 +164,42 @@ public void testEscapedPartitionValuePrunesInsteadOfDropping() { Assertions.assertEquals(Collections.singletonList("code=US%3ACA"), prunedLocations(result)); } + @Test + public void hiveDateTimeStringRendersHiveCanonicalText() { + // H2 (unit): a DATETIME/TIMESTAMP predicate literal (LocalDateTime) must render Hive-canonical text. + // String.valueOf would yield ISO "2024-01-01T10:00", never matching "2024-01-01 10:00:00". RED before. + Assertions.assertEquals("2024-01-01 10:00:00", + HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0))); + Assertions.assertEquals("2024-01-01 00:00:00", + HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 0, 0, 0))); + Assertions.assertEquals("2024-01-01 10:00:30", + HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 30))); + Assertions.assertEquals("2024-01-01 10:00:00.123456", + HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0, 123456 * 1000))); + } + + @Test + public void testDatetimePartitionPredicatePrunesWithHiveCanonicalText() { + // H1+H2 composed end-to-end: a DATETIME partition value stored escaped in HMS ("dt=2024-01-01 10%3A00%3A00") + // must prune-in against a DATETIME predicate literal. RED before H2: the literal renders ISO + // "2024-01-01T10:00" and matches nothing; RED before H1: the stored ":" stays "%3A". Both are required for + // the real partition to survive pruning (else silent row loss). + List parts = Arrays.asList("dt=2024-01-01 10%3A00%3A00", "dt=2024-01-02 00%3A00%3A00"); + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(parts), Collections.emptyMap(), new FakeConnectorContext()); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.singletonList("dt")) + .build(); + ConnectorComparison dtEq = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("dt", ConnectorType.of("DATETIMEV2", 6, 0)), + new ConnectorLiteral(ConnectorType.of("DATETIMEV2", 6, 0), LocalDateTime.of(2024, 1, 1, 10, 0, 0))); + Optional> result = + metadata.applyFilter(null, handle, new ConnectorFilterConstraint(dtEq)); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Collections.singletonList("dt=2024-01-01 10%3A00%3A00"), prunedLocations(result)); + } + // ===== helpers ===== private Optional> applyFilter( diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 80f3ea81517484..e96068277ca85e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -55,6 +55,8 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.time.LocalDateTime; +import java.time.format.DateTimeFormatter; import java.util.AbstractMap; import java.util.ArrayList; import java.util.Arrays; @@ -89,6 +91,12 @@ public class HudiConnectorMetadata implements ConnectorMetadata { // Catalog property gating the partition-name source (mirrors legacy HMSExternalTable.USE_HIVE_SYNC_PARTITION). private static final String USE_HIVE_SYNC_PARTITION = "use_hive_sync_partition"; + // Hive-canonical partition text for a DATETIME/TIMESTAMP literal: space separator, full seconds. See + // hiveDateTimeString / extractLiteralValue (H2: String.valueOf(LocalDateTime) would yield ISO "…T…" and drop + // zero seconds, never matching the stored Hive partition value). + private static final DateTimeFormatter HIVE_DATETIME_SECONDS_FORMAT = + DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"); + // Internal MVCC-snapshot property carrying the FOR TIME AS OF instant from resolveTimeTravel to applySnapshot // (FE-internal transport, paimon scan-options model). It is NEVER serialized to BE: fe-core only feeds the // snapshot to applySnapshot, which consumes this property and stamps HudiTableHandle.queryInstant. Not a @@ -1028,11 +1036,44 @@ private String extractColumnName(ConnectorExpression expr) { } private String extractLiteralValue(ConnectorExpression expr) { - if (expr instanceof ConnectorLiteral) { - Object val = ((ConnectorLiteral) expr).getValue(); - return val != null ? String.valueOf(val) : null; + if (!(expr instanceof ConnectorLiteral)) { + return null; } - return null; + Object val = ((ConnectorLiteral) expr).getValue(); + if (val == null) { + return null; + } + if (val instanceof LocalDateTime) { + // H2: a DATETIME/TIMESTAMP partition literal arrives as a LocalDateTime (DATE arrives as LocalDate). + // String.valueOf would call toString() -> ISO "2024-01-01T10:00" (T separator, dropped zero seconds), + // which never string-equals the Hive-canonical stored partition value "2024-01-01 10:00:00" in + // matchesPredicates -> the whole table prunes to 0 rows. Render Hive-canonical text instead. + return hiveDateTimeString((LocalDateTime) val); + } + return String.valueOf(val); + } + + /** + * Renders a DATETIME/TIMESTAMP literal as Hive-canonical partition text: {@code yyyy-MM-dd HH:mm:ss} (space + * separator, full seconds), appending trailing-zero-trimmed microseconds only when a sub-second part is + * present. Matches the stored Hive partition value for a scale-0 DATETIME partition (the only realistic case) + * so the pruning string-compare hits. Package-private static for offline unit testing. + * + *

    Contract: {@code convertDateLiteral} produces microsecond precision (nano = micros*1000), so the nano is + * always a multiple of 1000; a sub-microsecond nano (unreachable on the pruning path) would be truncated. + */ + static String hiveDateTimeString(LocalDateTime ldt) { + String base = ldt.format(HIVE_DATETIME_SECONDS_FORMAT); + int nano = ldt.getNano(); + if (nano == 0) { + return base; + } + String micros = String.format("%06d", nano / 1000); + int end = micros.length(); + while (end > 1 && micros.charAt(end - 1) == '0') { + end--; + } + return base + "." + micros.substring(0, end); } /** diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java index 3ae89e77b33922..f892c7a6f29d59 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionValuesTest.java @@ -22,6 +22,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.time.LocalDateTime; import java.util.Arrays; import java.util.Collections; import java.util.Map; @@ -120,6 +121,27 @@ public void parsePartitionNameUnescapesValuesForPruning() { Assertions.assertEquals("a/b", values.get("kind"), "slash-escaped value must be decoded"); } + @Test + public void hiveDateTimeStringRendersHiveCanonicalText() { + // H2: a DATETIME/TIMESTAMP predicate literal arrives as a LocalDateTime. It must render as Hive-canonical + // partition text (space separator, full seconds) so it string-matches the stored partition value in + // matchesPredicates. String.valueOf(LocalDateTime) yields ISO "2024-01-01T10:00" (T separator, dropped + // zero seconds) which never matches "2024-01-01 10:00:00" -> the whole table prunes to 0 rows. RED before. + Assertions.assertEquals("2024-01-01 10:00:00", + HudiConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0))); + // midnight: ISO would collapse to "2024-01-01T00:00" + Assertions.assertEquals("2024-01-01 00:00:00", + HudiConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 0, 0, 0))); + // non-zero seconds: ISO keeps the 'T' separator ("2024-01-01T10:00:30") + Assertions.assertEquals("2024-01-01 10:00:30", + HudiConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 30))); + // sub-second (nano = micros*1000): trailing-zero-trimmed microseconds + Assertions.assertEquals("2024-01-01 10:00:00.123456", + HudiConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0, 123456 * 1000))); + Assertions.assertEquals("2024-01-01 10:00:00.1", + HudiConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0, 100000 * 1000))); + } + @Test public void emptyPartitionKeysReturnsEmptyForUnpartitionedTable() { // Unpartitioned tables reach here with an empty key list and an empty path; the result must be empty From b911019d8f6db6114e5535e72ab86d8677902c73 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 06:04:47 +0800 Subject: [PATCH 211/334] [doc](catalog) #65185 batch-0: H2 design (red-team SOUND) + track-off H2 DONE (cf540eebc3c). Render DATETIME/TIMESTAMP partition literals as Hive-canonical text in both hive and hudi copies. Adversarial design review verdict SOUND with a real Hive fixture (run17.hql) confirming the stored scale-0 form. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 5 +- plan-doc/tasks/designs/FIX-H2-design.md | 115 +++++++++++++++++++++ 2 files changed, 118 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-H2-design.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 66c13d9a272d9f..80f9a366935746 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -26,7 +26,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | # | id | 严重度 | 模块 | 一句话 | 设计 | 实现 | build+UT | 状态 | |---|----|-------|------|--------|------|------|----------|------| | 1 | **H1** | 🔴高 | hudi**+hive** | 分区名不 unescape→丢行 | ✅ | ✅ | ✅ | ✅ | -| 2 | **H2** | 🔴高 | hudi | datetime 分区谓词 ISO→0 行 | ⬜ | ⬜ | ⬜ | ⬜ | +| 2 | **H2** | 🔴高 | hudi**+hive** | datetime 分区谓词 ISO→0 行 | ✅ | ✅ | ✅ | ✅ | | 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ⬜ | ⬜ | ⬜ | ⬜ | | 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | | 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | @@ -85,7 +85,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:`HudiPartitionPruningTest` 加 HMS 名 `ts=2024-01-01 10%3A00%3A00` + 谓词 `ts='2024-01-01 10:00:00'`,断言命中(RED:被剪)。live e2e gated。 ### H2 — hudi datetime 分区谓词 ISO 化 → 0 行 · reverify §2 H2 -- [ ] **H2** +- [x] **H2**(hive+hudi 两份) · DONE `cf540eebc3c`(设计 `designs/FIX-H2-design.md`，含设计红队 SOUND) - **现码**:`ExprToConnectorExpressionConverter.convertDateLiteral:309-322`(非 DATE→`LocalDateTime`)+ `HudiConnectorMetadata.extractLiteralValue:1030-1036`(`String.valueOf`→`2024-01-01T10:00`)+ `matchesPredicates:1067` 字符串比 HMS `2024-01-01 10:00:00`。 - **Fix**:时间类型不做字符串剪枝——`extractLiteralValue` 按 Hive 规范文本(空格分隔、全 `HH:mm:ss[.ffffff]`)渲染 `LocalDateTime`;更优:`matchesPredicates` type-aware 或对时间列退回 fe-core typed Nereids 剪枝。 - **Files**:`HudiConnectorMetadata.java`(时间渲染逻辑;必要时连带 `extractLiteralValue`)。 @@ -231,3 +231,4 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **H4** DONE `03f4c12dffa` — lowercase JNI reader 列名(hudi-only;`jniColumnNames` helper + UT)。 - **H1** DONE `39a279e7c26` — hive+hudi 剪枝 `parsePartitionName` unescape 值(两份就地各修;widen `unescapePathName`;直接单测跨 H3 稳定 + hive e2e applyFilter 测)。 +- **H2** DONE `cf540eebc3c` — hive+hudi `extractLiteralValue` 对 `LocalDateTime` 渲 Hive-canonical 空格文本(`hiveDateTimeString` helper;设计红队 SOUND+实证 fixture `run17.hql`;H1+H2 复合 e2e 测)。 diff --git a/plan-doc/tasks/designs/FIX-H2-design.md b/plan-doc/tasks/designs/FIX-H2-design.md new file mode 100644 index 00000000000000..9be8de0674f97a --- /dev/null +++ b/plan-doc/tasks/designs/FIX-H2-design.md @@ -0,0 +1,115 @@ +# FIX-H2 — datetime 分区谓词 ISO 化 → 整表剪到 0 行（hive + hudi 两份就地各修） + +> 来源：`task-list-65185-reverify-fixes.md` §H2；证据 reverify §2 H2。**范围决策**：H2 与 H1 同源、同属两连接器逐字节相同的剪枝块——对抗 agent 已证实 **hive 亦丢行**。用户签字「两份就地各修」。 +> 关联勘验（复用）：`P4-T06e-FIX-DATETIME-PUSHDOWN-FORMAT-design.md`（MaxCompute 同根因：`ExprToConnectorExpressionConverter.convertDateLiteral` 产 `LocalDateTime`；修法=直接 `format` 空格分隔文本而非 `String.valueOf`/`toString`）。 + +## Problem + +翻闸后，对 **DATETIME/DATETIMEV2/TIMESTAMP 分区列** 的 `=`/`IN` 谓词，剪枝把整表剪到 0 行（静默丢全部行）。DATE 列安全、STRING 列安全。 + +例：`dt` 为 `DATETIMEV2(0)` 分区列，谓词 `dt = '2024-01-01 10:00:00'` → 0 行。 + +## Root Cause（对 HEAD 核实 + 勘验 agent 实测） + +剪枝谓词侧 `extractLiteralValue`（hudi `HudiConnectorMetadata:1030-1036` / hive `HiveConnectorMetadata:2045-2051`）对字面量做 `String.valueOf(getValue())`。`convertDateLiteral` 对非 DATE 的时间字面量存入 **`LocalDateTime`**（DATE/DATEV2 存 `LocalDate`）。`String.valueOf(LocalDateTime)` = `LocalDateTime.toString()` = **ISO-8601**：`'T'` 分隔、且**省略末尾零秒/零分数**（`2024-01-01 10:00:00` → `"2024-01-01T10:00"`）。而 HMS 分区值文本（经 H1 unescape 后）是 Hive-canonical **空格分隔、全 `HH:mm:ss`**（`"2024-01-01 10:00:00"`）。`matchesPredicates` 裸字符串比 → **永不相等** → 每分区被剪 → 0 行。 + +- `LocalDate.toString()` = `"2024-01-01"` 恰与 Hive 文本一致 → **DATE 安全**（不改）。 +- STRING/VARCHAR 值本就 verbatim → 安全。 +- legacy 经 Nereids typed 剪枝（`PruneFileScanPartition`）比 typed 值、只发 Hive-canonical 原文，从不 ISO 化 → **回归**。 +- **与 MaxCompute 的关键差异**：MC 谓词下推涉及 TZ 转 UTC（故有 `ZoneId.of("CST")` 抛坑）；**分区剪枝是拿字面量与「存储的分区值文本」比，无任何 TZ 转换** → H2 无 TZ、无 `ZoneId.of` 崩坑，比 MC 简单。 + +## Design + +**两份就地各修**：在**每个**连接器的 `extractLiteralValue`，对 `LocalDateTime` 值渲染 **Hive-canonical 分区文本**（空格分隔、全 `yyyy-MM-dd HH:mm:ss`、有微秒才带 `.ffffff`），而非 `String.valueOf`。DATE(`LocalDate`)/STRING/其它分支不变（`String.valueOf`）。 + +渲染放**包私有 static** helper `hiveDateTimeString(LocalDateTime)`（可离线单测，对齐 H1 的 static 化范式）： + +```java +private static final DateTimeFormatter HIVE_DATETIME_SECONDS_FORMAT = + DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"); + +static String hiveDateTimeString(LocalDateTime ldt) { + String base = ldt.format(HIVE_DATETIME_SECONDS_FORMAT); // "yyyy-MM-dd HH:mm:ss"（全秒、空格） + int nano = ldt.getNano(); + if (nano == 0) { + return base; // 现实唯一场景（scale-0 datetime 分区） + } + // DATETIMEV2 是微秒精度（convertDateLiteral: nano = micro*1000）；追加 6 位微秒、去尾零（Hive 分数写法）。 + String micros = String.format("%06d", nano / 1000); + int end = micros.length(); + while (end > 1 && micros.charAt(end - 1) == '0') { + end--; + } + return base + "." + micros.substring(0, end); +} + +// extractLiteralValue: +Object val = ((ConnectorLiteral) expr).getValue(); +if (val == null) { + return null; +} +if (val instanceof LocalDateTime) { + return hiveDateTimeString((LocalDateTime) val); +} +return String.valueOf(val); +``` + +### 为何现实场景 bulletproof（关键论证） +唯一现实的 datetime 分区是 **scale-0**（按整秒；按微秒分区会分区爆炸，无人这么建表）：谓词字面量 scale-0 → `LocalDateTime` nano=0 → 渲 `2024-01-01 10:00:00`（无分数）；存储的整秒值 canonical 亦 `2024-01-01 10:00:00`（经 H1 unescape）→ **精确相等**。H1(unescape)+H2(渲染) **复合**：HMS 名 `dt=2024-01-01 10%3A00%3A00` → H1 unescape → `2024-01-01 10:00:00` == H2 渲染 → 命中。 +残余风险仅 scale>0 datetime 分区（实际不存在）且写入方分数写法与本渲染不一致时——**登记为 e2e 验证点**（见 Test Plan），非现实缺陷。 + +### 为何不做 type-aware / 退回 fe-core（对齐用户「最小」选择） +- type-aware（matchesPredicates 按列类型 typed 比）需把列类型穿进剪枝块、改两份数据结构 → 超出「每份就地小改」； +- 「退回 fe-core typed 剪枝」实际未接线（`planScan` 丢弃 fe-core `requiredPartitions`，勘验已证），退回=对 datetime 列不剪=全表扫（perf 回归）——比 (a) 渲染差。 +- 故取 task-list **primary** = (a) 渲染 canonical 文本，保留剪枝且现实场景正确。 + +## Implementation Plan + +对 `HudiConnectorMetadata.java` 与 `HiveConnectorMetadata.java` **各**： +1. 加 import `java.time.LocalDateTime`、`java.time.format.DateTimeFormatter`（java 组内按序）。 +2. 加常量 `HIVE_DATETIME_SECONDS_FORMAT` + 包私有 static `hiveDateTimeString(LocalDateTime)`。 +3. `extractLiteralValue`：`LocalDateTime` 分支渲染，其它 `String.valueOf`（null→null 不变）。 + +## Risk Analysis + +| Risk | 处置 | +|---|---| +| `instanceof LocalDateTime` 误分类 | `convertDateLiteral` 仅非 DATE 时间类型产 `LocalDateTime`，DATE 产 `LocalDate`；精确区分。 | +| scale>0 datetime 分区分数写法不匹配 | 现实不存在（微秒分区爆炸）；登记 e2e 点；非本 fix 引入 silent-loss（本 fix 只修 scale-0 现实场景，改前该场景 100% 丢）。 | +| 与 H1 交互 | 正交且复合（H1 unescape 名侧、H2 渲染谓词侧）；datetime 分区两者都需。UT 覆盖复合。 | +| TZ | 无——分区剪枝不转 TZ（与 MC 不同），无 `ZoneId.of` 崩坑。 | +| fe-core 铁律 | 仅连接器内；无 fe-core 变更、无源名判别、无属性解析。import-gate exit 0。 | +| checkstyle | 无 static import；新 import 按序；`%06d` 无需 Locale（整数无分组）。 | + +## Test Plan + +### Unit Tests（直接测 `hiveDateTimeString`，两连接器各一套；Rule 9 钉 WHY） +钉 WHY：datetime 分区谓词必须渲成 Hive-canonical 空格文本，否则整表剪到 0 行（静默丢全部行）。 +1. **整秒**：`hiveDateTimeString(LocalDateTime.of(2024,1,1,10,0,0))` == `"2024-01-01 10:00:00"`（**核心**，现实场景；改前 `String.valueOf`→`"2024-01-01T10:00"`）。 +2. **午夜/零分零秒**：`...of(2024,1,1,0,0,0)` == `"2024-01-01 00:00:00"`（ISO 会缩成 `"2024-01-01T00:00"`）。 +3. **带微秒**：`...of(2024,1,1,10,0,0, 123456*1000)` == `"2024-01-01 10:00:00.123456"`；去尾零 `...100000*1000` → `.1`。 +4. **秒非零**：`...of(2024,1,1,10,0,30)` == `"2024-01-01 10:00:30"`（ISO 出 `"2024-01-01T10:00:30"`，仍 `T`）。 +5. **mutation**（守门）：helper 改回 `ldt.toString()` → 断言 1/2/4 变红。 +6. **DATE 不回归**：`extractLiteralValue` 对 `LocalDate` 仍出 `"2024-01-01"`（经既有 pruning 测或补一条 `String.valueOf(LocalDate)` 断言；LocalDate 不入 LocalDateTime 分支）。 + +### E2E Tests（live-gated） +- DATETIME(scale-0) 分区列 + `= '2024-01-01 10:00:00'` 返回**正确行集**（改前 0 行）；含微秒 scale>0 分区（若真存在）作 e2e 验证点（真值闸）。真集群，显式登记 gated（Rule 12）。 + +## 设计红队（对抗 agent，实现前） + +**结论：DESIGN SOUND — 无 must-fix gap。** 逐条 CONFIRM： +1. `String.valueOf(LocalDateTime)`==`toString()`==ISO `T`+省零秒（`convertDateLiteral:309-322` DATE→`LocalDate`、非 DATE→`LocalDateTime`；TIMEV2 是 `TimeV2Literal` 非 `DateLiteral`→走 String，且 TIME 非合法 Hive 分区类型→无 gap；TIMESTAMP_NTZ→DATETIME→`LocalDateTime` 已覆盖）。 +2. 渲染逻辑逐例正确（含去尾零 `.1`/`.123456`）；`convertDateLiteral` 保证 nano=micro×1000（1000 倍数）→ 无越界；`%06d` 无需 Locale。 +3. **scale-0 存储形吻合有实证 fixture**：`docker/.../hive/scripts/create_preinstalled_scripts/run17.hql:7` `time_par timestamp` 分区，真实目录 `time_par=2023-01-01 01%3A30%3A00` → H1 unescape → `2023-01-01 01:30:00` == H2 渲染。Hive 写 + Doris 写（`be/.../vhive_utils.cpp`）均整秒无 `.0`。 +4. 完整性：`extractLiteralValue` 是唯一 literal→string 点，EQ + IN-list 均经它；其它连接器（MC 已空格化、Trino 转 epoch typed、ES 有意 ISO 自有 DSL）无需修 → 范围 hive+hudi 完整。 +5. 无回归：if-chain 对 null/LocalDate/Boolean/数值/String 行为不变；static 化无碍。 +- 非阻断 nit（已折入）：javadoc 注明 helper 亚微秒行为 out-of-contract（剪枝路不可达）。 + +## 守门结果（DONE，commit `cf540eebc3c`） + +- hudi：`mvn -o -pl :fe-connector-hudi -am test -Dtest=HudiPartitionValuesTest` → **BUILD SUCCESS**；10 run / 0 fail（+`hiveDateTimeStringRendersHiveCanonicalText`）；checkstyle 全 0。 +- hive：`mvn -o -pl :fe-connector-hive -am test -Dtest=HiveConnectorMetadataPartitionPruningTest` → **BUILD SUCCESS**；12 run / 0 fail（+`hiveDateTimeStringRendersHiveCanonicalText` 直接测 + `testDatetimePartitionPredicatePrunesWithHiveCanonicalText` H1+H2 复合 e2e applyFilter）；checkstyle 全 0。 +- import-gate `GATE_RC=0`。测试可 RED：断言空格文本 `2024-01-01 10:00:00`/`.123456`/`.1`，改前 `String.valueOf`→`2024-01-01T10:00` → 变红。DATETIME 分区读端到端 = live-gated。 + +## 决策类型 +明确修复（用户签字「两份就地各修」，task-list primary=(a) 渲染）。连接器局部、无 SPI 变更、与 legacy 语义 parity（现实场景）。D-PRUNE 抽取延后。 From 9c6fc584eb9becc99e01434ffd820678f67e038f Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 06:11:59 +0800 Subject: [PATCH 212/334] [fix](catalog) hudi H3: use_hive_sync_partition-aware prune partition source MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Batch-0 (#65185 reverify) H3, hudi-only. After the HMS SPI cutover, a filtered query on a non-hive-style hudi-on-HMS table (hive_style_partitioning =false, the Hudi default) returned 0 splits (silent total row loss); the same query without a filter returned rows. Root cause: HudiConnectorMetadata.applyFilter set prunedPartitionPaths UNCONDITIONALLY from hmsClient.listPartitionNames (HMS hive-style names, "year=2024/month=01"). HudiScanPlanProvider feeds those verbatim to fsView.getLatestBaseFilesBeforeOrOn(partitionPath,...), which is keyed by Hudi RELATIVE STORAGE paths. For a non-hive-style table the physical layout is positional ("2024/01"), so fsView matched nothing -> 0 splits. The unpruned scan path (resolvePartitions -> listAllPartitionPaths) is use_hive_sync_partition-aware and returns the relative paths, which is why an unfiltered query worked. applyFilter bypassed that awareness (collectPartitions already had it); the stale HudiScanPlanProvider javadoc claiming the pitfall was "closed before the catalog flip" was outdated. Fix: make applyFilter's candidate source use_hive_sync_partition-aware, mirroring collectPartitions. hive-sync -> keep hmsClient.listPartitionNames (the hive-style name IS the relative layout, fsView accepts it) pruned by the existing prunePartitionNames (parsePartitionName, H1-unescaped). non-hive-sync (default) -> list the relative storage paths via metaClientExecutor.execute(listAllPartitionPaths(buildMetaClient(...))) — the SAME source the unpruned scan uses (net-neutral: resolvePartitions short-circuits once prunedPaths is set) — pruned by a new static prunePartitionPaths built on parsePartitionValues (handles positional AND hive-style layouts + unescapes). matchesPredicates made static so both prune helpers share it. No FSUtils/getPartitions/getLocation (relativizing custom HMS locations exceeds legacy/collectPartitions parity). Tests: migrated HudiPartitionPruningTest to a stub metaClient executor (feeds the canned listing offline; existing assertions unchanged, now exercising the non-hive-sync branch); added a positional-path test pinning that non-hive-style tables prune to RELATIVE paths (RED under the old HMS-name code), a hive-sync-branch test, and a direct prunePartitionPaths unit. Non-hive-style filtered read on a real cluster is live-gated. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../connector/hudi/HudiConnectorMetadata.java | 70 ++++++++++++++----- .../hudi/HudiPartitionPruningTest.java | 69 +++++++++++++++++- 2 files changed, 122 insertions(+), 17 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index e96068277ca85e..b75fa00e365b8e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -249,29 +249,48 @@ public Optional> applyFilter( return Optional.empty(); } - // List candidate partition names from HMS (e.g. "year=2024/month=01"). These - // relative paths double as partition identifiers consumed by HudiScanPlanProvider. - // Keep maxParts=-1 (unlimited): no silent partition truncation. - List allPartNames = hmsClient.listPartitionNames( - hudiHandle.getDbName(), hudiHandle.getTableName(), -1); - if (allPartNames == null || allPartNames.isEmpty()) { - return Optional.empty(); + // H3: candidate partition paths MUST be the SAME shape the scan feeds fsView (Hudi RELATIVE STORAGE + // paths), and use_hive_sync_partition-aware (mirroring collectPartitions). The old code fed HMS hive-style + // names ("year=2024/month=01") unconditionally; for a non-hive-style table (Hudi default) the physical + // layout is positional ("2024/01"), so fsView (keyed by relative storage paths) matched nothing -> 0 + // splits for any filtered query. Keep maxParts=-1 (unlimited): no silent partition truncation. + boolean hiveSync = useHiveSyncPartition(); + List allPartPaths; + List matchedPartPaths; + if (hiveSync) { + // hive-sync: HMS registers the hive-style names, which ARE the relative storage layout, so fsView + // accepts them directly (no relativization, matching legacy / collectPartitions). Prune the HMS names. + allPartPaths = hmsClient.listPartitionNames( + hudiHandle.getDbName(), hudiHandle.getTableName(), -1); + if (allPartPaths == null || allPartPaths.isEmpty()) { + return Optional.empty(); + } + matchedPartPaths = prunePartitionNames(allPartPaths, partKeyNames, partitionPredicates); + } else { + // non-hive-sync (Hudi default): list the RELATIVE storage paths from Hudi metadata -- the SAME source + // the unpruned scan (resolvePartitions -> listAllPartitionPaths) uses -- under the plugin auth + TCCL + // pin. Net-neutral: resolvePartitions short-circuits once prunedPaths is set, so a filtered query lists + // exactly once (here) instead of there. parsePartitionValues handles the positional layout ("2024/01"). + allPartPaths = metaClientExecutor.execute(() -> + HudiScanPlanProvider.listAllPartitionPaths( + HudiScanPlanProvider.buildMetaClient(buildHadoopConf(), hudiHandle.getBasePath()))); + if (allPartPaths == null || allPartPaths.isEmpty()) { + return Optional.empty(); + } + matchedPartPaths = prunePartitionPaths(allPartPaths, partKeyNames, partitionPredicates); } - - List matchedPartNames = prunePartitionNames( - allPartNames, partKeyNames, partitionPredicates); - if (matchedPartNames.size() == allPartNames.size()) { + if (matchedPartPaths.size() == allPartPaths.size()) { // No pruning effect return Optional.empty(); } - LOG.info("Partition pruning: {}.{} all={} pruned={}", + LOG.info("Partition pruning: {}.{} hiveSync={} all={} pruned={}", hudiHandle.getDbName(), hudiHandle.getTableName(), - allPartNames.size(), matchedPartNames.size()); + hiveSync, allPartPaths.size(), matchedPartPaths.size()); - // Build updated handle carrying only the matched partition paths for scan planning. + // Build updated handle carrying only the matched (relative-shape) partition paths for scan planning. HudiTableHandle updatedHandle = hudiHandle.toBuilder() - .prunedPartitionPaths(matchedPartNames) + .prunedPartitionPaths(matchedPartPaths) .build(); return Optional.of(new FilterApplicationResult<>(updatedHandle, constraint.getExpression(), false)); @@ -1092,6 +1111,25 @@ private List prunePartitionNames(List allPartNames, return matched; } + /** + * Prunes Hudi RELATIVE partition paths (positional {@code "2024/01"} or hive-style {@code + * "year=2024/month=01"}) using {@link HudiScanPlanProvider#parsePartitionValues} (handles both layouts and + * unescapes) + {@link #matchesPredicates}. Used by the non-hive-sync {@link #applyFilter} branch, whose + * candidate source is the Hudi metadata listing — the same relative-path shape the scan feeds fsView. Static + + * package-private for offline unit testing. + */ + static List prunePartitionPaths(List allPartPaths, + List partKeyNames, Map> predicates) { + List matched = new ArrayList<>(); + for (String partPath : allPartPaths) { + Map partValues = HudiScanPlanProvider.parsePartitionValues(partPath, partKeyNames); + if (matchesPredicates(partValues, predicates)) { + matched.add(partPath); + } + } + return matched; + } + static Map parsePartitionName(String partName, List partKeyNames) { Map values = new HashMap<>(); @@ -1111,7 +1149,7 @@ static Map parsePartitionName(String partName, return values; } - private boolean matchesPredicates(Map partValues, + static boolean matchesPredicates(Map partValues, Map> predicates) { for (Map.Entry> entry : predicates.entrySet()) { String colName = entry.getKey(); diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java index d6000b6cd40b27..b65eb0cecbfe2e 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java @@ -41,6 +41,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.concurrent.Callable; /** * Tests {@link HudiConnectorMetadata#applyFilter} partition pruning (P3-T05). @@ -157,13 +158,60 @@ public void testUnpartitionedTableIsNotTouched() { Assertions.assertFalse(result.isPresent()); } + @Test + public void testNonHiveStylePositionalPathsPruneToRelativePaths() { + // H3 core: a non-hive-style Hudi table (hive_style_partitioning=false, the DEFAULT) has a POSITIONAL + // physical layout ("2024/01"), NOT the HMS hive-style name ("year=2024/month=01"). applyFilter must prune + // the RELATIVE storage paths (the Hudi metadata listing that the scan also feeds fsView), so the pruned + // set is the shape fsView is keyed by. RED before the fix: applyFilter fed HMS hive-style names to fsView, + // which finds nothing on a non-hive-style table -> 0 splits for any filtered query. + HudiConnectorMetadata metadata = new HudiConnectorMetadata( + new FakeHmsClient(PARTITIONS), // HMS hive-style names -- must NOT be used here + Collections.emptyMap(), // use_hive_sync_partition=false -> non-hive-sync + new StubMetaClientExecutor(Arrays.asList("2024/01", "2024/02", "2023/12"))); + Optional> result = + metadata.applyFilter(null, partitionedHandle(), new ConnectorFilterConstraint(eq("year", "2024"))); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals(Arrays.asList("2024/01", "2024/02"), prunedPaths(result)); + } + + @Test + public void testHiveSyncBranchPrunesHmsNames() { + // use_hive_sync_partition=true: partitions are registered in HMS and the hive-style name IS the relative + // storage layout, so applyFilter prunes the HMS names directly (no Hudi metadata listing / stub unused). + HudiConnectorMetadata metadata = new HudiConnectorMetadata( + new FakeHmsClient(PARTITIONS), + Collections.singletonMap("use_hive_sync_partition", "true"), + new StubMetaClientExecutor(Collections.emptyList())); + Optional> result = + metadata.applyFilter(null, partitionedHandle(), new ConnectorFilterConstraint(eq("year", "2024"))); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals( + Arrays.asList("year=2024/month=01", "year=2024/month=02"), prunedPaths(result)); + } + + @Test + public void prunePartitionPathsMatchesPositionalLayout() { + // Direct offline unit for the non-hive-sync prune helper: positional relative paths matched by the values + // parsePartitionValues extracts positionally. + Assertions.assertEquals( + Arrays.asList("2024/01", "2024/02"), + HudiConnectorMetadata.prunePartitionPaths( + Arrays.asList("2024/01", "2024/02", "2023/12"), + PART_KEYS, + Collections.singletonMap("year", Collections.singletonList("2024")))); + } + // ========== helpers ========== private Optional> applyFilter( HudiTableHandle handle, ConnectorExpression expr) { + // Default (use_hive_sync_partition=false) -> the non-hive-sync branch, whose candidate source is the Hudi + // metadata listing. Feed the canned partition list via the stub executor (no live metaClient). The + // hive-style names here parse the same via parsePartitionValues, so the pruning assertions are unchanged. HudiConnectorMetadata metadata = new HudiConnectorMetadata( new FakeHmsClient(PARTITIONS), Collections.emptyMap(), - new DirectHudiMetaClientExecutor()); + new StubMetaClientExecutor(PARTITIONS)); return metadata.applyFilter(null, handle, new ConnectorFilterConstraint(expr)); } @@ -263,4 +311,23 @@ public HmsPartitionInfo getPartition(String dbName, String tableName, List T execute(Callable action) { + return (T) canned; + } + } } From f0ee2ab06d25d4f81f9dbba3208f4fedc848f161 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 06:23:58 +0800 Subject: [PATCH 213/334] [test](catalog) batch-0 H2: DATE non-regression + hive micros-trim coverage Closes two low-risk coverage gaps flagged by the batch-0 adversarial review: - H2 design test-plan item 6 (DATE non-regression) was unimplemented: add a DATE partition + LocalDate predicate applyFilter test to both hive and hudi, proving a LocalDate literal is NOT diverted to the datetime render (stays String.valueOf = "2024-01-01") and prunes correctly. - The hive hiveDateTimeString test omitted the trailing-zero-trim ".1" case the hudi twin asserts; add it for parity. Test-only. hive 13/0-fail, hudi 12/0-fail, checkstyle clean. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- ...ConnectorMetadataPartitionPruningTest.java | 23 +++++++++++++++++++ .../hudi/HudiPartitionPruningTest.java | 20 ++++++++++++++++ 2 files changed, 43 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java index 0dc1484beb5b7d..e3eb2a049f57ad 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionPruningTest.java @@ -35,6 +35,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.time.LocalDate; import java.time.LocalDateTime; import java.util.ArrayList; import java.util.Arrays; @@ -176,6 +177,28 @@ public void hiveDateTimeStringRendersHiveCanonicalText() { HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 30))); Assertions.assertEquals("2024-01-01 10:00:00.123456", HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0, 123456 * 1000))); + Assertions.assertEquals("2024-01-01 10:00:00.1", + HiveConnectorMetadata.hiveDateTimeString(LocalDateTime.of(2024, 1, 1, 10, 0, 0, 100000 * 1000))); + } + + @Test + public void testDatePartitionPredicatePrunesUnchanged() { + // H2 non-regression: a DATE predicate literal arrives as a LocalDate (not LocalDateTime), so it is NOT + // diverted to hiveDateTimeString -- String.valueOf(LocalDate) = "2024-01-01" already matches the stored + // Hive DATE partition value. Guards against the datetime branch accidentally catching DATE columns. + List parts = Arrays.asList("dt=2024-01-01", "dt=2024-01-02"); + HiveConnectorMetadata metadata = new HiveConnectorMetadata( + new FakeHmsClient(parts), Collections.emptyMap(), new FakeConnectorContext()); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.singletonList("dt")) + .build(); + ConnectorComparison dateEq = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("dt", ConnectorType.of("DATEV2")), + new ConnectorLiteral(ConnectorType.of("DATEV2"), LocalDate.of(2024, 1, 1))); + Optional> result = + metadata.applyFilter(null, handle, new ConnectorFilterConstraint(dateEq)); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals(Collections.singletonList("dt=2024-01-01"), prunedLocations(result)); } @Test diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java index b65eb0cecbfe2e..4bb9b733759464 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiPartitionPruningTest.java @@ -35,6 +35,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.time.LocalDate; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -202,6 +203,25 @@ public void prunePartitionPathsMatchesPositionalLayout() { Collections.singletonMap("year", Collections.singletonList("2024")))); } + @Test + public void testDatePartitionPredicatePrunesUnchanged() { + // H2 non-regression: a DATE predicate literal is a LocalDate (not LocalDateTime), so it is NOT diverted to + // hiveDateTimeString -- String.valueOf(LocalDate) = "2024-01-01" already matches the stored DATE value. + HudiConnectorMetadata metadata = new HudiConnectorMetadata( + new FakeHmsClient(PARTITIONS), Collections.emptyMap(), + new StubMetaClientExecutor(Arrays.asList("2024-01-01", "2024-01-02"))); + HudiTableHandle handle = new HudiTableHandle.Builder("db", "t", "s3://b/t", "COPY_ON_WRITE") + .partitionKeyNames(Collections.singletonList("dt")) + .build(); + ConnectorComparison dateEq = new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("dt", ConnectorType.of("DATEV2")), + new ConnectorLiteral(ConnectorType.of("DATEV2"), LocalDate.of(2024, 1, 1))); + Optional> result = + metadata.applyFilter(null, handle, new ConnectorFilterConstraint(dateEq)); + Assertions.assertTrue(result.isPresent()); + Assertions.assertEquals(Collections.singletonList("2024-01-01"), prunedPaths(result)); + } + // ========== helpers ========== private Optional> applyFilter( From 6e2c881ef540911898ad05e0fd581a75b96ef4f4 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 06:25:08 +0800 Subject: [PATCH 214/334] [doc](catalog) #65185 batch-0 COMPLETE: H3 design + final review + track-off H3 DONE (9c6fc584eb9) + test-hardening (f0ee2ab06d2). Batch-0 (H1-H4) all done. Final batch-wide adversarial review (3 skeptics) = CLEAN: four fixes correct, compose correctly, no regressions, scope complete (other connectors independently verified immune), all 10 new tests RED-capable. Registers one pre-existing residual (use_hive_sync_partition=true + non-hive-style layout, at legacy parity, out of H3 scope) for the D-PRUNE/relativization follow-up. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 8 ++- plan-doc/tasks/designs/FIX-H3-design.md | 71 ++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-H3-design.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 80f9a366935746..1662f266ee30f6 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -27,7 +27,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) |---|----|-------|------|--------|------|------|----------|------| | 1 | **H1** | 🔴高 | hudi**+hive** | 分区名不 unescape→丢行 | ✅ | ✅ | ✅ | ✅ | | 2 | **H2** | 🔴高 | hudi**+hive** | datetime 分区谓词 ISO→0 行 | ✅ | ✅ | ✅ | ✅ | -| 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ⬜ | ⬜ | ⬜ | ⬜ | +| 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ✅ | ✅ | ✅ | ✅ | | 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | | 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | | 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ⬜ | ⬜ | ⬜ | ⬜ | @@ -93,7 +93,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - 注:此条对抗验证 agent 未完成,但机制确定 + 与 H1/H3 同源(均 CONFIRMED),保留高危。 ### H3 — hudi HMS 名当存储路径 → 非 hive-style 带 filter 0 split · reverify §2 H3 -- [ ] **H3** +- [x] **H3**(hudi-only) · DONE `9c6fc584eb9`(设计 `designs/FIX-H3-design.md`，含最终对抗复审 CORRECT&COMPLETE + 残余登记) - **现码**:`HudiConnectorMetadata.applyFilter:244-267`(无条件用 hive-style HMS 名作 `prunedPartitionPaths`)→ `HudiScanPlanProvider.resolvePartitions:587-590` → `collectCowSplits:394`/`collectMorSplits:429` 喂 `fsView.getLatestBaseFilesBeforeOrOn`(期望 Hudi 相对存储路径)。`use_hive_sync` 感知只在列举路 `collectPartitions:641`,`applyFilter` 绕过。 - **Fix**:`applyFilter` 候选分区源 `use_hive_sync_partition` 感知:`!useHiveSyncPartition`→对 `listAllPartitionPaths`(`2024/01`)剪枝;`useHiveSyncPartition`→`FSUtils.getRelativePartitionPath(basePath, location)` 相对化。 - **Files**:`HudiConnectorMetadata.java`(applyFilter 候选源)。 @@ -232,3 +232,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **H4** DONE `03f4c12dffa` — lowercase JNI reader 列名(hudi-only;`jniColumnNames` helper + UT)。 - **H1** DONE `39a279e7c26` — hive+hudi 剪枝 `parsePartitionName` unescape 值(两份就地各修;widen `unescapePathName`;直接单测跨 H3 稳定 + hive e2e applyFilter 测)。 - **H2** DONE `cf540eebc3c` — hive+hudi `extractLiteralValue` 对 `LocalDateTime` 渲 Hive-canonical 空格文本(`hiveDateTimeString` helper;设计红队 SOUND+实证 fixture `run17.hql`;H1+H2 复合 e2e 测)。 +- **H3** DONE `9c6fc584eb9` — hudi `applyFilter` 候选源 `use_hive_sync_partition` 感知(非 hive-sync 走 `listAllPartitionPaths` 相对路径+新 `prunePartitionPaths`,hive-sync 保留 HMS 名臂;`matchesPredicates` 提 static)。测迁移到 stub executor + 位置式/hive-sync/直接 helper 新测。 +- **批次 0 test-hardening** DONE `f0ee2ab06d2` — DATE 非回归测(hive+hudi)+ hive `.1` 微秒去尾零(补 H2 设计 item 6 + 复审软点)。 +- **批次 0 全量对抗复审(3 skeptic)= CLEAN**:四修正确/复合/无回归、范围完整、10 新测均可 RED。**登记残余(非本批修)**:`use_hive_sync_partition=true`+`hive_style_partitioning=false` 表 hive-sync 臂仍喂 HMS 名给 fsView→带 filter 0 split(同 H3 类、另一臂、pre-existing 与 legacy parity);D-PRUNE/相对化 location 时一并评估。见 `designs/FIX-H3-design.md` §最终对抗复审。 +- **⭐ 批次 0(H1–H4)全部 DONE。** 下一步见文首建议批次:批次 1(M5→M7→M6/M4/M2 连接器局部)。**所有 H1–H4 e2e 均 live-gated**(含转义值/DATETIME/非-hive-style/MOR-JNI 混大小写读),须真集群回归(memory `hms-iceberg-delegation-needs-e2e`)。 diff --git a/plan-doc/tasks/designs/FIX-H3-design.md b/plan-doc/tasks/designs/FIX-H3-design.md new file mode 100644 index 00000000000000..c7355811582312 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-H3-design.md @@ -0,0 +1,71 @@ +# FIX-H3 — hudi 把 HMS 名当存储路径喂 fsView → 非 hive-style 表带 filter 0 split（hudi-only） + +> 来源：`task-list-65185-reverify-fixes.md` §H3；证据 reverify §2 H3。批次 0 第 4 条（最复杂，hudi-only）。 +> 已由 recon agent 核实 hudi 1.0.2 API + 现码；本设计取 recon 推荐的「最小正确 + 可测」形。 + +## Problem + +翻闸后，对 `hive_style_partitioning=false`（Hudi 默认）的 hudi-on-HMS 分区表，**带 filter 的查询返回 0 split（不带 filter 有行）**——即静默丢全部匹配行。 + +## Root Cause（对 HEAD 核实） + +`HudiConnectorMetadata.applyFilter:247` **无条件**把候选分区源设为 `hmsClient.listPartitionNames`（hive-style HMS 名 `year=2024/month=01`），剪枝后原样存入 `prunedPartitionPaths`。scan 侧 `resolvePartitions:588` 直接返回它们，喂给 `fsView.getLatestBaseFilesBeforeOrOn(partitionPath, ...)`（`HudiScanPlanProvider:394/429`）——而 fsView 按 **Hudi 相对存储路径** 索引。非 hive-style 表物理布局是 `2024/01`，与 HMS 名 `year=2024/month=01` 不符 → fsView 找不到文件 → 0 split。不带 filter 时 `resolvePartitions` 回退 `listAllPartitionPaths`（Hudi 元数据，相对路径，正确）→ 有行。 + +`applyFilter` **绕过了** `use_hive_sync_partition` 感知：连接器自己的列举路 `collectPartitions:641` 是感知的（非 hive-sync 用 `listAllPartitionPaths` 相对路径），但 `applyFilter` 没有。`HudiScanPlanProvider:716-721` 的 javadoc 自称此坑「翻闸前已闭合」是**过时的**（翻闸已发生、修未做）。legacy 默认从 Hudi 元数据取相对路径 → 回归。 + +## Design（option A — 保留 H1 的 `parsePartitionName`，surgical 加分支） + +`applyFilter` 候选分区源改 **`use_hive_sync_partition` 感知**，镜像 `collectPartitions`；候选路径必须与 scan 喂 fsView 的**同形**（Hudi 相对存储路径）： + +- **hive-sync**（`useHiveSyncPartition()==true`）：`hmsClient.listPartitionNames`（hive-style 名 == 相对存储布局，fsView 直接认，无需相对化——recon 证实 legacy/collectPartitions 均如此，`FSUtils.getRelativePartitionPath`/`getLocation` 是**红鲱鱼**，超出 parity，不引入）。剪枝复用现有 `prunePartitionNames`（→ `parsePartitionName`，**H1 已修 unescape**）。 +- **非 hive-sync**（默认）：`metaClientExecutor.execute(() -> HudiScanPlanProvider.listAllPartitionPaths(buildMetaClient(buildHadoopConf(), basePath)))`——**与不带 filter 的 scan（`resolvePartitions`）同一相对路径源**，成本 net-neutral（`resolvePartitions` 见 `prunedPaths!=null` 即短路，不重复列举）。剪枝用**新** static `prunePartitionPaths`（→ `parsePartitionValues`，处理 hive-style **与** 位置式 `2024/01` 两种布局 + unescape）。 + +两臂产出的 matched 均为相对存储路径形；`prunedPartitionPaths(matched)`。保留「无剪枝效果 → `Optional.empty()`」与空集守卫。 + +> **为何 option A 而非统一 `parsePartitionValues`（B）**：B 会使 hudi `parsePartitionName`/`prunePartitionNames` 死掉、需删除、连带撤掉 H1 刚加的 hudi 方法+直接测（同批次内加了又删，churn）。A 让 hive-sync 臂继续用 `parsePartitionName`（HMS hive-style 名的天然解析器，H1 修的 unescape 在此长期有效），非 hive-sync 臂用 `parsePartitionValues`（位置式）——各源用各自天然解析器，surgical、保留已提交工作（Rule 3）。`matchesPredicates` 提为 static 供两个 prune helper 共用。 + +## Implementation Plan + +`HudiConnectorMetadata.java`： +1. `applyFilter`：候选源按 `useHiveSyncPartition()` 二分（如上）；`matchedPartPaths` 二分求得后走共同尾（size 比较 bail + LOG + build handle）。LOG 加 `hiveSync` 便于诊断。 +2. 新增 static `prunePartitionPaths(List allPartPaths, List partKeyNames, Map> predicates)`：逐路径 `matchesPredicates(HudiScanPlanProvider.parsePartitionValues(path, partKeyNames), predicates)`。 +3. `matchesPredicates`：`private` → `static`（纯函数；`prunePartitionNames` 与新 `prunePartitionPaths` 共用）。`parsePartitionName`/`prunePartitionNames` 保留（hive-sync 臂用）。 + +无 fe-core 改动；无 `FSUtils`/`getPartitions`/`getLocation`（recon 证实为 parity 红鲱鱼）。 + +## Risk Analysis + +| Risk | 处置 | +|---|---| +| 非 hive-sync 臂新建 metaClient 增成本 | net-neutral：`resolvePartitions` 见 pruned 即短路，过滤查询只在 `applyFilter` 列举一次（= 不过滤查询在 `resolvePartitions` 的同一次列举）。hive-sync 臂仍用廉价 HMS 列举。 | +| hive-sync 是否需相对化 LOCATION | 否（recon + legacy 证实：标准 hive-sync 布局 name==相对路径；相对化自定义 location 超 parity，不做）。 | +| 与 H1 交互 | hive-sync 臂用 `parsePartitionName`（H1 unescape 有效）；非 hive-sync 臂用 `parsePartitionValues`（本就 unescape）→ 两臂均 unescape 正确。 | +| 位置式路径解析 | `parsePartitionValues` 已证处理 `2024/01`（`HudiPartitionValuesTest.nonHiveStylePositionalPathMapsByPosition`）+ hive-style + 单列 whole-path fallback + fail-loud。 | +| fe-core 铁律 | 仅连接器内；`use_hive_sync_partition` 是**连接器**属性读取（非 fe-core 源名判别），与 `collectPartitions` 同款。 | + +## Test Plan + +现有 `HudiPartitionPruningTest` 用 `DirectHudiMetaClientExecutor`（**运行** action）+ emptyMap（非 hive-sync）→ 修后非 hive-sync 臂会真建 metaClient（假 basePath）→ 抛。迁移 + 补： + +### Unit Tests +1. **迁移现有 8 条**：`applyFilter` helper 的 executor 换成 `StubMetaClientExecutor(PARTITIONS)`（返回 canned、不跑 action → 不建真 metaClient）。现有断言不变（`parsePartitionValues` 处理 hive-style 名，matched 相对路径形 == 原 hive-style 名）→ 全绿；现覆盖**非 hive-sync 臂**（默认、原被破坏的臂）。 +2. **H3 核心 RED 测**：`testNonHiveStylePositionalPathsPruneToRelativePaths` — `FakeHmsClient(hive-style 名)` + `StubMetaClientExecutor(["2024/01","2024/02","2023/12"])` + emptyMap + `eq("year","2024")` → 断言 pruned == `["2024/01","2024/02"]`（**相对路径**，非 HMS 名）。RED（改前）：旧码用 HMS 名 → pruned == `["year=2024/month=01",...]` ≠ 断言。 +3. **hive-sync 臂测**：`testHiveSyncBranchPrunesHmsNames` — `use_hive_sync_partition=true` + `FakeHmsClient(PARTITIONS)` → 走 HMS 名臂（`parsePartitionName`）→ pruned == matched hive-style 名。 +4. **直接 helper 测**：`prunePartitionPaths(["2024/01","2024/02","2023/12"], ["year","month"], {year:[2024]})` == `["2024/01","2024/02"]`（位置式匹配，离线）。 + +### E2E Tests +非 hive-style hudi 表「带 filter 分区集 == 不带 filter 分区集（都命中）」= **live-gated**（真 hudi 集群），显式登记（Rule 12）。memory `hms-iceberg-delegation-needs-e2e` 口径。 + +## 守门结果（DONE，commit `9c6fc584eb9`） + +`mvn -o -pl :fe-connector-hudi -am test -Dtest=HudiPartitionPruningTest,HudiPartitionValuesTest,HudiConnectorPartitionListingTest` → **BUILD SUCCESS**；39 run / 0 fail（`HudiPartitionPruningTest` 11→迁移 8 + 新 3；`HudiConnectorPartitionListingTest` 18 回归通过，证 `matchesPredicates` static + applyFilter 重构未破坏列举路）；checkstyle 全 0；import-gate `GATE_RC=0`。后续 test-hardening `f0ee2ab06d2` 加 DATE 非回归测（12 run）。 + +## 最终对抗复审（批次 0 全量，3 并行 skeptic 审已提交码） + +**结论：H3 CORRECT & COMPLETE（其声明范围内）；批次 0 四修正确、复合正确、无回归、范围完整（其它连接器独立核实免疫）；10 条新测均可 RED + 编码 WHY。** 详见 session。登记两项（均非本 fix 引入、非阻断）： + +- **残余 gap（登记，非本批修）**：`use_hive_sync_partition=true` **且** `hive_style_partitioning=false` 的表——hive-sync 臂喂 HMS hive-style 名给 fsView，但物理布局是位置式 `2024/01` → 带 filter 0 split（同类 bug 的另一臂）。**非 H3 引入**（改前所有表都用 HMS 名，此臂与改前及 legacy `HMSExternalTable` 逐字节相同），且与本 fix「不相对化自定义 location（超 collectPartitions/legacy parity）」立场一致；不带 filter 该组合正常（`resolvePartitions` 恒用相对路径）。→ **登记为 CACHE-later / D-PRUNE 时一并评估**；当前对最常见组合（hive-sync+hive-style、非-hive-sync+任意布局）全部正确。 +- **net-neutral 微注**：仅「EQ/IN 恰好命中全部分区」的 no-effect 角落会二次列举 Hudi 元数据（applyFilter 列举→bail→resolvePartitions 再列举）；与改前 HMS-名码同模式、非新回归、非正确性问题。 + +## 决策类型 +明确修复（hudi-only；用户批次 0 范围内）。连接器局部、无 SPI 变更、与 `collectPartitions`/legacy parity。D-PRUNE 抽取延后。 From 54b764abd028f70a490098fdf6febe9a78b43bdf Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 06:26:22 +0800 Subject: [PATCH 215/334] [doc](catalog) HANDOFF: batch-0 (H1-H4 reverify fixes) DONE; next = batch 1 Records the #65185 reverify-fix series and batch-0 completion (H1-H4, all green, adversarial review CLEAN, one pre-existing residual registered). Next session: reverify batch 1 (connector-local M-series) + live-gated e2e. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 3bc67011333ab6..7959fd7d5aa434 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -18,9 +18,21 @@ **验证**：`mvn -pl fe-core -am test-compile` BUILD SUCCESS + 0 checkstyle；靶向 `mvn test` 17 跑 0 败 0 错 1 skip（HmsGson 3/3、Iceberg/PaimonGson 各 3/3、ExternalMetaCacheRouteResolver 7/7、HmsCatalog skip）。净室对抗复核（`wf_728cad25-62a`，4 独立审查 + 逐条对抗验证）= **CLEAN**（1 发现 0 确认缺陷；唯一 minor=翻闸 hive 表失去 Nereids SQL 结果缓存资格 @ `BindRelation:729`，经验证 by-design：fail-safe、`enable_hive_sql_cache` 默认关、与 paimon/iceberg-native 一致、被禁用的 `HmsQueryCacheTest` 已明记该遗留缓存 dead-for-hms）。 +--- + +# 🔧 复核修复系列（#65185 reverify，2026-07-11 起）——跟踪表 `plan-doc/task-list-65185-reverify-fixes.md` + +翻闸后第三方复核（`reviews/catalog-spi-review-65185-reverify-2026-07-11.md`）判定的真实/活跃需修条目，按批次做（H1–H4 高 / M1–M8 中 / L1–L20 低 / D-系列设计债）。**处理纪律**：每条 = 设计(`tasks/designs/FIX--design.md`) → 设计红队 → 实现 → build+靶向 UT → 独立 commit → 勾表 → 更新 HANDOFF。path-whitelist `git add`。 + +**⭐ 批次 0（H1–H4 hudi 高危）全部 DONE（2026-07-11）**： +- **关键发现+范围决策**：H1（分区名不 unescape 丢行）、H2（datetime 谓词 ISO 化剪到 0 行）**不是 hudi 独有**——`HiveConnectorMetadata` 逐字节相同剪枝块同样静默丢行（对抗 agent 证实）。**用户签字「两份就地各修」**（不抽共享 helper；D-PRUNE 延后）。H3（HMS 名当存储路径）、H4（JNI 列名原样大小写）hudi-only。 +- commits：`03f4c12dffa`(H4 lowercase JNI 列名) · `39a279e7c26`(H1 hive+hudi unescape) · `cf540eebc3c`(H2 hive+hudi datetime 渲染) · `9c6fc584eb9`(H3 hudi use_hive_sync 感知源) · `f0ee2ab06d2`(test-hardening) · 各配 doc commit。 +- **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 + **⭐ 下一步（下个 session）：** -1. **e2e 回归（用户自跑，勿丢，非静默）**：异构 `type=hms` 目录跑 hive/iceberg-on-HMS/hudi-on-HMS 读/写/DDL/procedure/MTMV/time-travel/@incr；**从库事件同步陈旧**（新跟随者游标喂路从未在任何目录跑过，hms 是第一个事件消费者）；Kerberos-HMS 冒烟；升级镜像 GSON replay；耦合缝行（partition_values/hudi_meta/sample-analyze/auto-analyze）；hive 视图（现无条件服务）。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 -2. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂（PhysicalPlanTranslator HMS/hudi 臂、INC-5 stale throws、CheckPolicy hudi 臂、遗留 BindRelation HMS_EXTERNAL_TABLE 臂）+ 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 +1. **复核修复续批**（跟踪表）：批次 1（M5→M7→M6 iceberg / M4 mc / M2 hive，连接器局部）→ 批次 2（M3/M1 fe-core 通用节点）→ ...。⏸ 决策类（L2/L10/L12/L20）先问用户。 +2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 +3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 **⚠ 关键纠正（execution plan §3 已过时，本轮已核实纠正，见 `hms-spi-cutover-flip-2026-07-10.md` §2）**：§3.7「rewire 4 个 gate」**错**——两个 instanceof gate（MetastoreEventsProcessor:116、ExternalMetaCacheRouteResolver:66）须**保留**（自动排除翻闸目录、对未翻闸旧目录仍正确；删则破坏旧目录同步/失效）；缓存路自动接管（连接器 CachingHmsClient），事件路靠上面 A/B 两个 ADD-feed（非删 gate）。死 Nereids 臂**翻闸不删**（对齐 iceberg 翻闸留死臂的先例，Phase 3 统删）。 From 84f580c9075d7f4bdfd7bde3d1db06b2ad84c259 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 07:56:31 +0800 Subject: [PATCH 216/334] [fix](catalog) iceberg table stats: restore equality-delete gate (align current legacy) IcebergConnectorMetadata.computeRowCount reported total-records minus total-position-deletes with no equality-delete gate, so an iceberg MOR/CDC table with equality deletes surfaced an inflated row count to the CBO, while the sibling COUNT(*) pushdown (IcebergScanPlanProvider.getCountFromSummary) correctly declined -- an asymmetry that misleads join reorder / cardinality. Root cause is a parity target that moved under us, NOT a misread: the table- stats override was recon'd 2026-06-29 (HEAD 6252ecc248b) when legacy IcebergUtils.getIcebergRowCount was total-records - total-position-deletes with no gate, so the connector faithfully mirrored "no gate". Two days later upstream 32a2651f66b (#64648) refactored the count into getCountFromSummary(summary, ignoreDanglingDelete) and ADDED the equality gate (total-equality-deletes null or != "0" -> UNKNOWN), rewiring getIcebergRowCount to getCountFromSummary( summary, true). That commit is on this branch, so legacy now gates and the connector diverged. Cutover also routes iceberg-on-HMS (legacy HMSExternalTable :547 used the gated getIcebergRowCount) through computeRowCount -> newly activated regression there. Fix reproduces current legacy getCountFromSummary(summary, true) connector- locally (no fe-core edit): add TOTAL_EQUALITY_DELETES constant (byte-identical to the scan provider copy + fe-core), read + null-guard all three counters, and gate `!equalityDeletes.equals("0") -> -1` before the subtraction (-1 is already mapped to UNKNOWN by getTableStatistics). Position deletes still net out (legacy ignoreDanglingDelete = true). Corrected the three stale "does NOT gate" comments and rewrote the mutation-locked test equalityDeletesDoNotGate... -> equalityDeletesGateTableStatisticsToUnknown (asserts UNKNOWN; RED before fix). Reverses the prior clean-room-signed-off no-gate decision (P6.6-FIX-H4); user signed off the reversal 2026-07-11 (align current legacy over register-as- deviation). fe-core untouched; connector imports no fe-core; e2e live-gated. Build SUCCESS, 0 checkstyle, IcebergConnectorMetadataStatisticsTest 7/7 green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../iceberg/IcebergConnectorMetadata.java | 40 ++++++++++++------- ...cebergConnectorMetadataStatisticsTest.java | 21 +++++----- 2 files changed, 38 insertions(+), 23 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index e67ca495db9fb3..3797416ec2746d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -120,10 +120,12 @@ public class IcebergConnectorMetadata implements ConnectorMetadata { // spec-stable iceberg strings — byte-identical to legacy IcebergUtils.TOTAL_* and to the COUNT(*) // pushdown copies in IcebergScanPlanProvider (themselves deliberately NOT org.apache.iceberg // .SnapshotSummary.* per that file's note). Duplicated rather than shared so this fix does not touch - // the unrelated scan provider. WHY two keys only: legacy getIcebergRowCount nets out position deletes - // but (unlike the COUNT pushdown) does NOT gate on equality deletes — see computeRowCount. + // the unrelated scan provider. All THREE keys are read: legacy getIcebergRowCount (via + // getCountFromSummary, upstream 32a2651f66b / #64648) nets out position deletes AND gates the count to + // UNKNOWN on any equality delete — see computeRowCount. private static final String TOTAL_RECORDS = "total-records"; private static final String TOTAL_POSITION_DELETES = "total-position-deletes"; + private static final String TOTAL_EQUALITY_DELETES = "total-equality-deletes"; // Doris-level table property carrying a user comment. Local literal copy of the fe-core constant // IcebergExternalTable.TABLE_COMMENT_PROP ("comment") — the connector cannot import fe-core. Read by @@ -588,8 +590,9 @@ public Map getColumnHandles( * (without this override the connector inherits {@code ConnectorStatisticsOps}'s {@code Optional.empty()}, * so every iceberg table reports rowCount -1 -> CBO collapses cardinality to 1 and disables join reorder). * Mirrors {@code PaimonConnectorMetadata.getTableStatistics} in STRUCTURE, but uses the legacy iceberg - * FORMULA ({@code IcebergUtils.getIcebergRowCount}: currentSnapshot summary {@code total-records - - * total-position-deletes}). Parity decisions: + * FORMULA ({@code IcebergUtils.getIcebergRowCount} -> {@code getCountFromSummary(summary, true)}: + * {@code total-records - total-position-deletes}, gated to UNKNOWN when equality deletes are present). + * Parity decisions: *

      *
    • System tables -> empty: legacy {@code IcebergSysExternalTable.fetchRowCount} is unconditionally * UNKNOWN; a sys handle would otherwise load the BASE table and misreport its data row count for a @@ -622,11 +625,14 @@ public Optional getTableStatistics( } /** - * Row count from the current snapshot summary: {@code total-records - total-position-deletes} (legacy - * {@code IcebergUtils.getIcebergRowCount}). NOT the COUNT(*)-pushdown formula in - * {@code IcebergScanPlanProvider.getCountFromSnapshot} — that one gates on equality deletes and honors the - * dangling-delete session var (scan cardinality), which would over-degrade table statistics. Empty table - * (no current snapshot) -> -1, which the caller maps to UNKNOWN. + * Row count from the current snapshot summary, a faithful port of legacy {@code IcebergUtils + * .getIcebergRowCount} (which calls {@code getCountFromSummary(summary, true)}, upstream 32a2651f66b / + * #64648): any equality delete ({@code total-equality-deletes} absent or {@code != "0"}) -> -1 (UNKNOWN), + * since equality deletes re-project at read time and the summary cannot net them out; otherwise + * {@code total-records - total-position-deletes}. Shares the equality-delete gate with the COUNT(*) + * pushdown {@code IcebergScanPlanProvider.getCountFromSummary}, differing only in dangling-delete handling + * (table statistics always net out position deletes; the pushdown honors the dangling-delete session var). + * Empty table (no current snapshot) -> -1, which the caller maps to UNKNOWN. */ private static long computeRowCount(Table table) { Snapshot snapshot = table.currentSnapshot(); @@ -634,13 +640,19 @@ private static long computeRowCount(Table table) { return -1; } Map summary = snapshot.summary(); - // Null-guard (upstream 32a2651f66b, #64648): compaction / replace / overwrite snapshots may omit a - // total-* counter, and the pre-fix Long.parseLong(null) NPE-d. -1 -> caller maps to UNKNOWN. NOTE: - // intentionally NOT the pushdown getCountFromSummary — this table-stats path deliberately omits the - // equality-delete gate (see the method javadoc), so only the NPE is fixed here, not the semantics. + // Equality-delete gate + null-guard, a faithful port of legacy IcebergUtils.getCountFromSummary( + // summary, true) (upstream 32a2651f66b, #64648): an absent total-* counter (compaction / replace / + // overwrite snapshots may omit one — the pre-fix Long.parseLong(null) NPE-d), or any equality delete + // (total-equality-deletes != "0"), makes the summary row count unsafe -> -1 (caller maps to UNKNOWN), + // because equality deletes re-project at read time and the summary cannot net them out. Same gate as + // the COUNT(*) pushdown IcebergScanPlanProvider.getCountFromSummary. + String equalityDeletes = summary.get(TOTAL_EQUALITY_DELETES); String totalRecords = summary.get(TOTAL_RECORDS); String positionDeletes = summary.get(TOTAL_POSITION_DELETES); - if (totalRecords == null || positionDeletes == null) { + if (equalityDeletes == null || totalRecords == null || positionDeletes == null) { + return -1; + } + if (!equalityDeletes.equals("0")) { return -1; } return Long.parseLong(totalRecords) - Long.parseLong(positionDeletes); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataStatisticsTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataStatisticsTest.java index ce10cde5818b03..df9f030c1d6057 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataStatisticsTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataStatisticsTest.java @@ -45,8 +45,9 @@ * {@code Optional.empty()}, so every iceberg base table reported row count -1 to the FE optimizer * (cardinality collapses to 1, join reorder disabled, SHOW TABLE STATUS = -1). The fix mirrors * {@code PaimonConnectorMetadata.getTableStatistics} in structure but uses the legacy iceberg FORMULA - * ({@code IcebergUtils.getIcebergRowCount}: currentSnapshot summary {@code total-records - - * total-position-deletes}). Tests run against a real {@link InMemoryCatalog} table (no Mockito), the + * ({@code IcebergUtils.getIcebergRowCount} -> {@code getCountFromSummary(summary, true)}: currentSnapshot + * summary {@code total-records - total-position-deletes}, gated to UNKNOWN when equality deletes are present, + * per upstream #64648). Tests run against a real {@link InMemoryCatalog} table (no Mockito), the * {@link RecordingIcebergCatalogOps} fake serving it through the seam. */ public class IcebergConnectorMetadataStatisticsTest { @@ -193,7 +194,7 @@ public void loadFailureDegradesToUnknown() { } @Test - public void equalityDeletesDoNotGateTableStatistics() { + public void equalityDeletesGateTableStatisticsToUnknown() { Table table = newTable(); table.newAppend().appendFile(dataFile("/data/f1.parquet", 100)).commit(); table.newRowDelta().addDeletes(equalityDeletes("/data/ed1.parquet", 5)).commit(); @@ -201,11 +202,13 @@ public void equalityDeletesDoNotGateTableStatistics() { Optional stats = metadataFor(table, new RecordingIcebergCatalogOps()).getTableStatistics(null, handle()); - // WHY: table statistics use total-records - total-position-deletes ONLY. The COUNT(*)-pushdown formula - // (IcebergScanPlanProvider.getCountFromSnapshot) returns -1 when equality deletes exist; misusing THAT - // here would report UNKNOWN for any table with equality deletes. MUTATION: routing through the equality - // gate -> empty; this asserts present(100) (equality deletes are NOT subtracted by the legacy stat). - Assertions.assertTrue(stats.isPresent(), "equality deletes must not blank out table statistics"); - Assertions.assertEquals(100L, stats.get().getRowCount()); + // WHY: with equality deletes present the snapshot summary cannot net out the deleted rows, so + // total-records (100) overstates the real count. Legacy getIcebergRowCount -> getCountFromSummary( + // summary, true) (upstream #64648) gates such tables to UNKNOWN, and this connector's own COUNT(*) + // pushdown (IcebergScanPlanProvider.getCountFromSummary) does the same; the table-stats path must match + // or the two disagree and the CBO is fed an inflated count. MUTATION: dropping the + // `!equalityDeletes.equals("0")` gate surfaces present(100) and this assertion fails. + Assertions.assertFalse(stats.isPresent(), + "equality deletes must gate table statistics to UNKNOWN (summary cannot net them out)"); } } From 0387bc95fc54423a7ac45846a677e0e96fcece3f Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 07:59:02 +0800 Subject: [PATCH 217/334] [doc](catalog) #65185 batch-1: M5 design + reconcile superseded H-4 docs + track-off M5 design (FIX-M5-design.md): frames the fix as a parity catch-up to upstream #64648 (legacy row count evolved to gate equality deletes after the 2026-06-29 recon), NOT a misread; records the user's 2026-07-11 sign-off to reverse the prior no-gate decision and align current legacy. Reconciles the three now-stale prior-decision docs with SUPERSEDED banners (preserving history, pointing to FIX-M5): P6.6-FIX-H4 design (key parity decision 1 + test #7 + mutation), its summary, and the iceberg flip-blockers tasklist H-4 entry. Ticks M5 in the reverify-fixes tracking table (+ batch-1 rolling notes: recon + adversarial red-team of all five items, all mechanisms confirmed at HEAD, no UNSOUND verdicts) and updates HANDOFF (M5 done; next M7->M6->M4->M2). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 10 +- .../P6.6-iceberg-flip-blockers-tasklist.md | 2 +- plan-doc/tasks/designs/FIX-M5-design.md | 93 +++++++++++++++++++ ...FIX-H4-rowcount-table-statistics-design.md | 7 ++ ...IX-H4-rowcount-table-statistics-summary.md | 4 + 6 files changed, 114 insertions(+), 4 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M5-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 7959fd7d5aa434..4edb1e7bd59738 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -30,7 +30,7 @@ - **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 **⭐ 下一步（下个 session）：** -1. **复核修复续批**（跟踪表）：批次 1（M5→M7→M6 iceberg / M4 mc / M2 hive，连接器局部）→ 批次 2（M3/M1 fe-core 通用节点）→ ...。⏸ 决策类（L2/L10/L12/L20）先问用户。 +1. **复核修复续批**（跟踪表）：**批次 1 进行中**——全 5 条已一轮 recon+对抗红队（`wf_40498e52-19f`，机制全 HEAD 确认、无 UNSOUND，设计存 `plan-doc/tasks/designs/FIX-M*-design.md` + scratch `batch1/`）。**M5 DONE**（`84f580c9075`：iceberg 表级行数恢复 equality-delete 护栏，对齐当前旧版；**推翻先前签字的「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 文档已批注 SUPERSEDED）。**续做 M7→M6→M4→M2**（红队要点见跟踪表滚动区：M7 仅注释措辞、M6 companion 升必做、M4 最干净、**M2 非照抄 MaxCompute 须额外 override `planScanForPartitionBatch`**）→ 批次 2（M3/M1 fe-core 通用节点）→ ...。⏸ 决策类（L2/L10/L12/L20）先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 1662f266ee30f6..5a184e06902a32 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -33,7 +33,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ⬜ | ⬜ | ⬜ | ⬜ | | 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | | 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ⬜ | ⬜ | ⬜ | ⬜ | -| 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ⬜ | ⬜ | ⬜ | ⬜ | +| 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | | 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ⬜ | ⬜ | ⬜ | ⬜ | | 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ⬜ | ⬜ | ⬜ | ⬜ | | 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⬜ | @@ -139,7 +139,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:连接器单测:同表两次 `listPartitions` 只一次 SDK 往返(recording fake 计数)。 ### M5 — iceberg computeRowCount 丢 equality-delete gate · reverify §3 M5 -- [ ] **M5** +- [x] **M5** · DONE `84f580c9075`（设计 `designs/FIX-M5-design.md`；recon+红队 SOUND_WITH_CHANGES）。根因=parity 目标被上游 #64648 移动（非误读）；恢复护栏对齐当前旧版（**用户 2026-07-11 签字推翻先前「不 gate」决定**）；连接器局部无 fe-core；3 处 P6.6-FIX-H4 stale 文档已批注 SUPERSEDED；`IcebergConnectorMetadataStatisticsTest` 7/7 绿（含倒置后的 gate 断言，RED-able）。e2e live-gated。 - **现码**:`IcebergConnectorMetadata.computeRowCount:631-646`(无 gate 减法)vs COUNT(*) 下推 `IcebergScanPlanProvider.getCountFromSummary:1572-1574`(保留 gate)。legacy `IcebergUtils.getCountFromSummary:231-233` gate 到 UNKNOWN。javadoc+单测前提为假。 - **Fix**:加 `TOTAL_EQUALITY_DELETES` 常量,`computeRowCount` 减法前 `null||!="0"→返回 -1(UNKNOWN)`;订正 javadoc `:624-629`;重写 `equalityDeletesDoNotGateTableStatistics:195-210` 断言 UNKNOWN。 - **Files**:`fe-connector-iceberg/.../IcebergConnectorMetadata.java` + 同名测试。 @@ -236,3 +236,9 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **批次 0 test-hardening** DONE `f0ee2ab06d2` — DATE 非回归测(hive+hudi)+ hive `.1` 微秒去尾零(补 H2 设计 item 6 + 复审软点)。 - **批次 0 全量对抗复审(3 skeptic)= CLEAN**:四修正确/复合/无回归、范围完整、10 新测均可 RED。**登记残余(非本批修)**:`use_hive_sync_partition=true`+`hive_style_partitioning=false` 表 hive-sync 臂仍喂 HMS 名给 fsView→带 filter 0 split(同 H3 类、另一臂、pre-existing 与 legacy parity);D-PRUNE/相对化 location 时一并评估。见 `designs/FIX-H3-design.md` §最终对抗复审。 - **⭐ 批次 0(H1–H4)全部 DONE。** 下一步见文首建议批次:批次 1(M5→M7→M6/M4/M2 连接器局部)。**所有 H1–H4 e2e 均 live-gated**(含转义值/DATETIME/非-hive-style/MOR-JNI 混大小写读),须真集群回归(memory `hms-iceberg-delegation-needs-e2e`)。 + +--- + +**⭐ 批次 1(M5→M7→M6/M4/M2 连接器局部)进行中**:recon+对抗红队一轮扫全 5 条(workflow `wf_40498e52-19f`,5 recon+5 红队),全部机制 HEAD 确认、verdict SOUND / SOUND_WITH_CHANGES(无 UNSOUND)。要点:M7 只需更正注释措辞(别名数组本身正确);M6 须把「data-plane client.region」companion 从可选升为必做 + 注意测试 UnusedImports;M4 最干净(SOUND,唯 Caffeine 2.9.3 版本一致性待 build-verify);M2 **非** trivial「照抄 MaxCompute」——hive `planScan` 非 partition-set-scoped,必须**额外** override `planScanForPartitionBatch` 否则 batch 重复 split(红队证实),另需登记 ACID→sync + 未过滤扫描→sync 两条偏差。 + +- **M5** DONE `84f580c9075`(code) — iceberg 表级行数恢复 equality-delete 护栏,对齐当前旧版(上游 #64648 移动了 parity 目标)。**推翻先前签字的「不 gate」决定,用户 2026-07-11 签字**;3 处 P6.6-FIX-H4 文档批注 SUPERSEDED;`IcebergConnectorMetadataStatisticsTest` 7/7 绿。e2e live-gated(equality-delete 表 SHOW TABLE STATS=UNKNOWN,独立 iceberg + iceberg-on-HMS 同表同结果)。 diff --git a/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md b/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md index 0947f1f06f41cd..80503b41c5db57 100644 --- a/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md +++ b/plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md @@ -61,7 +61,7 @@ ### 关键详情 - **H-2** ☑ DONE `b0c34ec8fe7`：`getMetadata()` 用 5-arg ctor 串 `external_catalog.name`，但 scan/write/procedure 用 1-arg → `toNamespace` 漏 external-catalog 级（三 provider 解析表全部且仅经 `catalogOps.loadTable`→`toNamespace`，`toNamespace` 只读 externalCatalogName）。**修**=抽私有 `newCatalogBackedOps()` 集中计算四门控值并返 5-arg ops，四处（getMetadata + 三 provider getter）共用（复刻 master 单 ops）；listing-only 三标志在 loadTable 路径 inert，仅平价线程；getMetadata ops 字节不变；修正三处过时注释。815 全绿/checkstyle0/mutation 3-3 KILLED+1 反向守护/clean-room 5 探针全 REFUTED。验收 e2e（3 级 REST SELECT/INSERT/EXECUTE）=flip-gated 未跑。 - **H-3** ☑ DONE `7850c6ad03f`：legacy `initCatalog` 翻闸后死码，认证须经 `initExecutionAuthenticator` 接入（`PluginDrivenExternalCatalog.initPreExecutionAuthenticator:153-154,174` 调它并注入连接器 context）。**⚠️RECONCILE（Rule 7）：review「各 flavor 均未覆写」收窄=仅 filesystem 是真翻闸回归**——jdbc：master `IcebergJdbcMetaStoreProperties` 根本无 ExecutionAuthenticator/Kerberos 逻辑→非翻闸回归（pre-existing，与 paimon jdbc 不同），出 H-3 范围；hms 在 `initNormalizeAndCheckProps` 接入（live）→ 不受影响；glue/dlf/s3tables/rest 云无 HDFS UGI。**修**=`IcebergFileSystemMetaStoreProperties` 覆写 `initExecutionAuthenticator` 委派既有 `buildExecutionAuthenticator`（Kerberos-only 守护保留=legacy parity，非 Kerberos 留基类 no-op）。无字段遮蔽 bug（`@Getter executionAuthenticator` 覆写基类 NOOP_AUTH getter，已核）。4 测绿/checkstyle0/mutation 2-2 KILLED（neuter override+drop isKerberos 守护）。验收 e2e（Kerberized HDFS hadoop SELECT/INSERT）=flip-gated 未跑。 -- **H-4** ☑ DONE `7b26fdbac88`：snapshot-summary 行数逻辑在 `IcebergScanPlanProvider.getCountFromSnapshot`（COUNT(*) 下推用）但未走表级统计。rowCount=-1 → StatsCalculator 基数塌 1 + `disableJoinReorderIfStatsInvalid` 失整链 CBO + SHOW TABLE STATUS=-1。**修**=`IcebergConnectorMetadata` 覆写 `getTableStatistics`，从 `currentSnapshot().summary()` 算 `total-records - total-position-deletes`（legacy `IcebergUtils.getIcebergRowCount` 公式，镜像 paimon **结构** 但用 iceberg **公式**）。3 关键 parity 决策：① 系统表→empty（legacy `IcebergSysExternalTable.fetchRowCount` 恒 UNKNOWN，刻意背离 paimon）；② `rowCount>0` 才报否则 empty（钉死 0→UNKNOWN，对齐新消费方 `>=0` 取值契约）；③ 刻意**不**复用 `getCountFromSnapshot`（带 equality-delete 门，轴不同）。0 新 SPI/fe-core/BE。新 `IcebergConnectorMetadataStatisticsTest` 7 测（真 InMemoryCatalog+真 delete 文件）。iceberg 连接器 822/0/1 绿、checkstyle 0、**mutation 4/4 KILLED**、clean-room 8 向量全 REFUTED（唯 1 LOW=瞬时失败缓存 -1，与 paimon sibling 一致不修）。e2e（SHOW TABLE STATUS / join CBO 重排序）= flip-gated 未跑。 +- **H-4** ☑ DONE `7b26fdbac88`：snapshot-summary 行数逻辑在 `IcebergScanPlanProvider.getCountFromSnapshot`（COUNT(*) 下推用）但未走表级统计。rowCount=-1 → StatsCalculator 基数塌 1 + `disableJoinReorderIfStatsInvalid` 失整链 CBO + SHOW TABLE STATUS=-1。**修**=`IcebergConnectorMetadata` 覆写 `getTableStatistics`，从 `currentSnapshot().summary()` 算 `total-records - total-position-deletes`（legacy `IcebergUtils.getIcebergRowCount` 公式，镜像 paimon **结构** 但用 iceberg **公式**）。3 关键 parity 决策：① 系统表→empty（legacy `IcebergSysExternalTable.fetchRowCount` 恒 UNKNOWN，刻意背离 paimon）；② `rowCount>0` 才报否则 empty（钉死 0→UNKNOWN，对齐新消费方 `>=0` 取值契约）；③ 刻意**不**复用 `getCountFromSnapshot`（带 equality-delete 门，轴不同）。**⚠️ 决策 ③ 已于 2026-07-11 推翻**：上游 #64648 给旧版行数新增了 equality 护栏，连接器已恢复护栏对齐当前旧版（commit `84f580c9075`，见 `designs/FIX-M5-design.md`）。0 新 SPI/fe-core/BE。新 `IcebergConnectorMetadataStatisticsTest` 7 测（真 InMemoryCatalog+真 delete 文件）。iceberg 连接器 822/0/1 绿、checkstyle 0、**mutation 4/4 KILLED**、clean-room 8 向量全 REFUTED（唯 1 LOW=瞬时失败缓存 -1，与 paimon sibling 一致不修）。e2e（SHOW TABLE STATUS / join CBO 重排序）= flip-gated 未跑。 - **H-5+H-6（同根，合并修）✅ DONE `d6758ff71f5`**：翻闸后 iceberg catalog=`PluginDrivenExternalCatalog`，route resolver 落 ENGINE_DEFAULT（只 schema 缓存），连接器自有 `IcebergLatestSnapshotCache`(TTL 24h)/`IcebergManifestCache` 仅 `connector.invalidateAll()` 清，而 REFRESH CATALOG / 快照过程从不调它；H-6 叠加 invalidateTable 传 **REMOTE** 名（缓存键按 LOCAL 匹配，name-mapped 目录失效成 no-op）。 - **实证纠正 review/HANDOFF 设想**：① REFRESH CATALOG 走 `onRefreshCache`（不经 `resetToUninitialized`，**不重建连接器**——侦察中一 agent 误判，控制流证伪）；② 「让 IcebergProcedureOps 传 LOCAL 名」不可行——连接器 `fromRemote*` 是恒等默认（名字映射真相在引擎侧 `ExternalTable` 的 NameMapping 上），连接器无法自行做正确的 LOCAL 失效；③ `IcebergManifestCache` 注释自称「只在 REFRESH CATALOG 重建时清」**是错的**（REFRESH CATALOG 不重建）；④ `context.getMetaInvalidator().invalidateTable` 的**唯一生产调用方**就是 `IcebergProcedureOps:164`。 - **修（用户裁=引擎统一接管）**：H-5=`PluginDrivenExternalCatalog.onRefreshCache` 覆写 invalidCache 时调 `connector.invalidateAll()`（通用 SPI，paimon 同受益；读 connector 字段+null 守护，reset 路径安全）+ `IcebergConnector.invalidateAll` 扩清 manifest（目录级平价，REFRESH TABLE 仍保留 manifest）+ `IcebergManifestCache.invalidateAll()` + 修错误注释；H-6=`ConnectorExecuteAction` 过程成功后调 `RefreshManager.refreshTableInternal`（follower/REFRESH TABLE 同一标准路径，从 `ExternalTable` 出发清两层、两套名字都对）+ 删 `IcebergProcedureOps` REMOTE-名旧通知 + 其单测重定义为「连接器 dispatch 不失效缓存」。 diff --git a/plan-doc/tasks/designs/FIX-M5-design.md b/plan-doc/tasks/designs/FIX-M5-design.md new file mode 100644 index 00000000000000..b7147b3eefd29c --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M5-design.md @@ -0,0 +1,93 @@ +# FIX-M5 — iceberg 表级行数统计恢复 equality-delete 护栏（对齐当前旧版） + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M5。 +> 跟踪表：`plan-doc/task-list-65185-reverify-fixes.md` 行 9。 +> recon+对抗复审：workflow `wf_40498e52-19f`（recon SOUND_WITH_CHANGES，机制 HEAD 确认）。 + +## Problem + +`IcebergConnectorMetadata.computeRowCount`（fe-connector-iceberg）把表级行数算作 +`total-records - total-position-deletes`，**没有 equality-delete 护栏**。对带 equality-delete 的 +Iceberg MOR/CDC 表，这个数**偏大**（equality-delete 删掉的行无法从快照 summary 里扣除），被喂给 +CBO 会误导 join 顺序 / 广播判断。而同一连接器里 COUNT(\*) 下推的 `IcebergScanPlanProvider +.getCountFromSummary` **已带**这道护栏 → 两条路自相矛盾。 + +## Root Cause（重点：不是读错代码，是旧版在脚下变了） + +- 表级统计覆写（`getTableStatistics` + `computeRowCount`）是先前 iceberg 翻闸阻断项的一环，**recon 于 + 2026-06-29、HEAD `6252ecc248b`**。当时旧版 `IcebergUtils.getIcebergRowCount` 就是 + `total-records - total-position-deletes`、**无** equality 护栏，故连接器忠实镜像为「不 gate」，并经 + 对抗评审 + 变异测试**锁死**（设计见 `P6.6-FIX-H4-rowcount-table-statistics-design.md` §关键 parity 决策 1）。 +- **2026-07-01**，上游 `32a2651f66b`（#64648，*Fix NPE in COUNT(\*) pushdown when snapshot summary omits + total-\* counters*）把行数函数重构成公共 helper `getCountFromSummary(summary, ignoreDanglingDelete)`， + 并**新增** equality 护栏（`total-equality-deletes` 为 null 或 `!= "0"` → `UNKNOWN_ROW_COUNT`），同时把 + `getIcebergRowCount` 改为 `getCountFromSummary(summary, true)`。这个上游 commit 也进了本分支。 +- 于是旧版路径现在遇到 equality-delete 表**报 UNKNOWN**，而连接器覆写仍报偏大的数。**根因 = 迁移镜像的 + parity 目标在 recon 之后被上游改动了**（非当初读错）。翻闸让 iceberg-on-HMS 也走此覆写 + （旧 `HMSExternalTable:547` 走带护栏的 `getIcebergRowCount`）→ 对 iceberg-on-HMS 是**新激活**回归。 + +## 决策 / 签字 + +- 恢复护栏 = **推翻**先前签字并变异锁死的「不 gate」决定。因其推翻的是用户签过字的决定，已在 session 中用 + 中文讲清背景+两个方向，**用户 2026-07-11 签字选「恢复护栏、对齐当前旧版」**（另一路 = 维持不 gate + 登记 + 验收偏差，被否）。 +- 既定迁移原则 = 与旧版行为对齐；当前旧版已 gate，故 gate 是 parity-correct 目标。 + +## Design + +连接器内忠实复刻当前旧版 `getCountFromSummary(summary, /*ignoreDanglingDelete=*/true)`，**不动 fe-core**： + +1. 加第三个 summary 键常量 `TOTAL_EQUALITY_DELETES = "total-equality-deletes"`——与连接器自身 + `IcebergScanPlanProvider`（:136）及 fe-core `IcebergUtils` 同串字节一致；沿用本文件「本地复制、不抽公共类、 + 不动无关 scan provider」的既有惯例。 +2. `computeRowCount` 读三个计数并三者一起 null-guard；减法前加护栏 `if (!equalityDeletes.equals("0")) + return -1;`。`-1` 已被 `getTableStatistics` 的 `rowCount > 0` 门映射为 UNKNOWN。位置删除仍照减（等价旧版 + `ignoreDanglingDelete = true` 分支：`deleteCount==0` 时 `totalRecords - 0 == totalRecords`，故简单减法即可）。 +3. 更正三处失效注释（常量块 WHY、方法 javadoc、方法内 NOTE），改述护栏**已**施加、且与 COUNT(\*) 下推仅在 + dangling-delete 处理上不同。 +4. 重写变异锁死的单测断言为 UNKNOWN，改名 `equalityDeletesGateTableStatisticsToUnknown`。 + +## Implementation Plan（均在 fe-connector-iceberg，无 fe-core/BE/thrift/pom 改） + +- `IcebergConnectorMetadata.java` 常量块：改 WHY 注释 + 加 `TOTAL_EQUALITY_DELETES` 常量。 +- `IcebergConnectorMetadata.java` `getTableStatistics` javadoc：FORMULA 子句补「gated on equality deletes」。 +- `IcebergConnectorMetadata.java` `computeRowCount` javadoc + body：护栏 + null-guard 三者 + 减法保留。 +- `IcebergConnectorMetadataStatisticsTest.java`：重写/改名 `equalityDeletesDoNotGateTableStatistics` → + 断言 empty；更新类 javadoc FORMULA 描述。 +- **无新 import。** + +## Risk Analysis + +- 连接器局部，无 fe-core 依赖；`check-connector-imports.sh` 不受影响（仅字符串字面量）。 +- 行为收窄 = 旧版 parity：summary 缺 `total-equality-deletes` → UNKNOWN，与当前旧版一致。 +- append / 仅位置删除表不受影响：iceberg `SnapshotSummary` 总发 `total-equality-deletes`（无则 ="0"）， + 新 null-guard 不误伤（由两个正数用例保持 GREEN 佐证——它们现也读新计数）。 +- `computeRowCount` 私有、唯一调用方 `getTableStatistics`，无其它 caller。 +- checkstyle：无 static / 无 unused import。 + +## 文档收口（红队 required change #2） + +先前的「不 gate」结论散落在三处签字文档，须更正为「上游 #64648 已改 gate，故对齐」，否则 stale 文档与 +已发代码 + 倒置后的锁死测试长期相反： +- `P6.6-FIX-H4-rowcount-table-statistics-design.md`（§关键 parity 决策 1 / Test #7 / Mutation 末条） +- `P6.6-FIX-H4-rowcount-table-statistics-summary.md` +- `P6.6-iceberg-flip-blockers-tasklist.md`（H-4 行） +以顶部「SUPERSEDED」批注 + 指向本文件的方式收口（不删历史）。 + +## Test Plan + +### Unit Tests（fe-connector-iceberg；真 InMemoryCatalog v2；无 Mockito） + +- **重写** `equalityDeletesGateTableStatisticsToUnknown`（原 `equalityDeletesDoNotGateTableStatistics`）： + 100 数据行 + 1 个 equality-delete 文件（5 行）→ `getTableStatistics` 返回 `Optional.empty()`（UNKNOWN）。 + **HEAD RED**：修前返回 100-0=100 → present(100)，`assertFalse(isPresent())` 挂；护栏返回 -1 后 GREEN。 + WHY 注释编码意图 + 变异（去 gate → present(100)）。 +- **非回归全类重跑**：`rowCountFromTotalRecords` present(100)、`rowCountNetsOutPositionDeletes` present(70)、 + `emptyTableReportsUnknown` empty、`zeroNetRowsReportUnknown` empty、`systemTableReportsUnknownWithoutLoading` + empty+无 seam load、`loadFailureDegradesToUnknown` empty+不抛。这些现也读新 `total-equality-deletes`， + 绿跑额外佐证 iceberg 对 append/位置删除快照发 "0"。 + +### E2E（live-gated，随 iceberg-on-HMS 批量 e2e 补，见 memory `hms-iceberg-delegation-needs-e2e`） + +- docker iceberg/HMS + equality-delete writer（Spark MERGE/DELETE）；断言 `SHOW TABLE STATS` / explain 估计 + 行数 = UNKNOWN(-1)，**独立 iceberg 目录 + iceberg-on-HMS 目录同表同结果**。 diff --git a/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-design.md b/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-design.md index ad47630e30343b..8ea9e5537d1125 100644 --- a/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-design.md +++ b/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-design.md @@ -1,5 +1,12 @@ # P6.6-FIX-H4 — iceberg 表级行数统计（getTableStatistics） +> **⚠️ SUPERSEDED（2026-07-11）— 下方「关键 parity 决策 1」（不 gate equality-delete）已被推翻。** +> 本文 recon 于 2026-06-29 忠实镜像当时旧版 `getIcebergRowCount`（无 equality 护栏）；两天后上游 +> `32a2651f66b`（#64648）把旧版重构为 `getCountFromSummary(summary, true)` 并**新增** equality-delete +> 护栏。为对齐当前旧版，连接器已恢复护栏（用户 2026-07-11 签字，commit `84f580c9075`）。 +> **决策 1、Test #7、Mutation 末条以此批注为准，权威见 `designs/FIX-M5-design.md`。** 本文其余决策 +> （系统表→empty、`rowCount>0` 门、best-effort 降级、常量本地化）仍有效。 + > 来源：`plan-doc/reviews/P6.6-iceberg-cleanroom-adversarial-review-2026-06-28.md` §H-4。 > 任务清单：`plan-doc/tasks/P6.6-iceberg-flip-blockers-tasklist.md` 行 54。 > recon 实证日期：2026-06-29，HEAD `6252ecc248b`，branch `catalog-spi-10-iceberg`。 diff --git a/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-summary.md b/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-summary.md index 508615a5e1e3bf..02487d265f4eaf 100644 --- a/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-summary.md +++ b/plan-doc/tasks/designs/P6.6-FIX-H4-rowcount-table-statistics-summary.md @@ -1,5 +1,9 @@ # P6.6-FIX-H4 — 总结：iceberg 表级行数统计 +> **⚠️ SUPERSEDED（2026-07-11）— 「不 gate equality-delete」结论已被推翻。** 上游 `32a2651f66b`（#64648） +> 在本文成文后给旧版行数新增 equality-delete 护栏；为对齐当前旧版，连接器已恢复护栏（用户签字，commit +> `84f580c9075`，权威见 `designs/FIX-M5-design.md`）。本文其余部分仍有效。 + > 设计：`P6.6-FIX-H4-rowcount-table-statistics-design.md`。commit ``（代码）+ ``（落档）。 ## Problem From f6de950e5bdaa5ff9364e3cae8e23123bfafddf6 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:01:41 +0800 Subject: [PATCH 218/334] [fix](catalog) iceberg REST vended-cred: widen client.region alias set to legacy parity On the no-bound-S3 path (REST vended credentials: no static AK/SK/role -> chosenS3 empty), IcebergCatalogFactory.appendS3FileIO derives iceberg's client.region SOLELY from firstNonBlank(props, S3_REGION_ALIASES). The array listed only {s3.region, aws.region, region, client.region}, so a catalog whose region arrives via AWS_REGION, iceberg.rest.signing-region, rest.signing-region, REGION, glue.region or aws.glue.region yielded null -> client.region omitted -> iceberg S3FileIO falls to the AWS DefaultAwsRegionProviderChain and the write commit fails "Unable to load region". The comment claimed it mirrored legacy getRegionFromProperties but did not. Widen S3_REGION_ALIASES to a verbatim connector-side copy of the fe-core S3Properties @ConnectorProperty(isRegionField=true) region aliases (all 10, same declared order so s3.region still wins on conflict). Connector-local literal copy (fe-connector must not import fe-core). OSS/COS/OBS/Minio subclass region aliases are deliberately excluded (irrelevant to an AWS-S3-backed vended REST catalog) and the comment is corrected to say so rather than overclaiming a full getRegionFromProperties mirror. New IcebergCatalogFactoryTest.buildCatalogPropertiesRestVendedResolvesRegionFromWidened Aliases (region via AWS_REGION and via iceberg.rest.signing-region -> client.region); RED before the widening. Existing s3.region test pins non-regression. Build SUCCESS, 0 checkstyle, IcebergCatalogFactoryTest 62/62 green. e2e live-gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../iceberg/IcebergCatalogFactory.java | 13 ++++++---- .../iceberg/IcebergCatalogFactoryTest.java | 24 +++++++++++++++++++ 2 files changed, 33 insertions(+), 4 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index 333e548d2584b1..284348d4940463 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -77,10 +77,15 @@ public final class IcebergCatalogFactory { // Region-field aliases scanned to propagate client.region when NO fe-filesystem S3 storage is bound // (e.g. REST vended credentials: no static AK/SK/role, so S3FileSystemProvider.supports is false and - // chosenS3 is empty). Mirrors the legacy AbstractIcebergProperties.toFileIOProperties chosen==null - // fallback (getRegionFromProperties). The raw s3.region copied by buildBaseCatalogProperties is inert - // because iceberg S3FileIO reads client.region, not s3.region. - private static final String[] S3_REGION_ALIASES = {"s3.region", "aws.region", "region", "client.region"}; + // chosenS3 is empty). Verbatim connector-side copy (fe-connector must not import fe-core) of the fe-core + // S3Properties @ConnectorProperty(isRegionField=true) region aliases — the S3 subset of the alias set the + // legacy getRegionFromProperties scanned; declared order preserved so s3.region still wins on conflict. + // The OSS/COS/OBS/Minio subclass region aliases are deliberately excluded (irrelevant to an AWS-S3-backed + // vended REST catalog). The raw s3.region copied by buildBaseCatalogProperties is inert because iceberg + // S3FileIO reads client.region, not s3.region. + private static final String[] S3_REGION_ALIASES = { + "s3.region", "AWS_REGION", "region", "REGION", "aws.region", "glue.region", + "aws.glue.region", "iceberg.rest.signing-region", "rest.signing-region", "client.region"}; private IcebergCatalogFactory() { } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index 9ec4efc0f9c057..60bc87845e913d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -720,6 +720,30 @@ public void buildCatalogPropertiesRestVendedPropagatesClientRegionWithoutBoundS3 "vended REST (no bound S3) must still translate s3.region -> client.region"); } + @Test + public void buildCatalogPropertiesRestVendedResolvesRegionFromWidenedAliases() { + // WHY: the empty-chosenS3 region fallback must scan the SAME region aliases legacy getRegionFromProperties + // did (the fe-core S3Properties isRegionField set), not just {s3.region, aws.region, region, client.region}. + // A vended REST catalog whose region arrives only via AWS_REGION or iceberg.rest.signing-region would + // otherwise yield no client.region -> AWS SDK DefaultAwsRegionProviderChain -> "Unable to load region". + // RED before widening: AWS_REGION (uppercase) does not match the narrow lowercase aws.region and + // iceberg.rest.signing-region is absent from the narrow 4-alias set -> client.region null. + Map viaAwsRegion = IcebergCatalogFactory.buildCatalogProperties( + props("iceberg.catalog.type", "rest", "uri", "https://rest", + "iceberg.rest.vended-credentials-enabled", "true", "AWS_REGION", "us-east-1"), + "rest", Optional.empty()); + Assertions.assertEquals("us-east-1", viaAwsRegion.get("client.region"), + "region supplied only via AWS_REGION must translate to client.region"); + + Map viaSigningRegion = IcebergCatalogFactory.buildCatalogProperties( + props("iceberg.catalog.type", "rest", "uri", "https://rest", + "iceberg.rest.vended-credentials-enabled", "true", + "iceberg.rest.signing-region", "eu-west-1"), + "rest", Optional.empty()); + Assertions.assertEquals("eu-west-1", viaSigningRegion.get("client.region"), + "region supplied only via iceberg.rest.signing-region must translate to client.region"); + } + // --------------------------------------------------------------------- // buildS3TablesCatalogProperties — bespoke s3tables options (NO catalog-impl, NO type removal) // --------------------------------------------------------------------- From 9471be085c7507b036971fc445646c1947556dd1 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:03:14 +0800 Subject: [PATCH 219/334] [doc](catalog) #65185 batch-1: M7 design + track-off FIX-M7-design.md (REST vended-cred client.region alias widening) + tick M7 in the reverify-fixes tracking table (progress row + detail + rolling log). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 5 ++- plan-doc/tasks/designs/FIX-M7-design.md | 51 ++++++++++++++++++++++ 2 files changed, 54 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M7-design.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 5a184e06902a32..3cb3696685d5ed 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -35,7 +35,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ⬜ | ⬜ | ⬜ | ⬜ | | 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | | 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ⬜ | ⬜ | ⬜ | ⬜ | -| 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ⬜ | ⬜ | ⬜ | ⬜ | +| 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ✅ | ✅ | ✅ | ✅ | | 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⬜ | | 13 | **L1** | 🟡低 | tools | import-gate 三洞 | ⬜ | ⬜ | ⬜ | ⬜ | | 14 | **L2** | 🟡低 | fe-core | 翻闸 hive 丢 SQL 缓存资格 + COUNTER 停增 | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | @@ -153,7 +153,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:region+warehouse-only s3tables props 建 catalog 不抛(RED:抛)。live S3 e2e gated。 ### M7 — iceberg REST vended-cred region 别名收窄 · reverify §3 M7 -- [ ] **M7** +- [x] **M7** · DONE `f6de950e5bd`（设计 `designs/FIX-M7-design.md`）。`S3_REGION_ALIASES` 4→10 逐字节对齐 fe-core `S3Properties` isRegionField 集（同声明序）；注释按红队更正为「S3 子集、OSS/COS/OBS/Minio 刻意排除」（不再过度声称 getRegionFromProperties 精确镜像）；新测覆盖 `AWS_REGION`/`iceberg.rest.signing-region`（RED-able）。`IcebergCatalogFactoryTest` 62/62 绿。e2e live-gated。 - **现码**:`IcebergCatalogFactory.java:83` `S3_REGION_ALIASES` 仅 4 个 → `appendS3FileIO:187-194` 唯一 region 源。丢 `AWS_REGION`/`iceberg.rest.signing-region`/`rest.signing-region` 等。 - **Fix**:拓宽别名对齐 legacy `getRegionFromProperties`(至少加上述 3 个 + `REGION`/`glue.region`/`aws.glue.region`);连接器侧复制列表(不 import fe-core);订正 `:78-83` 注释。 - **Files**:`fe-connector-iceberg/.../IcebergCatalogFactory.java`。 @@ -242,3 +242,4 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) **⭐ 批次 1(M5→M7→M6/M4/M2 连接器局部)进行中**:recon+对抗红队一轮扫全 5 条(workflow `wf_40498e52-19f`,5 recon+5 红队),全部机制 HEAD 确认、verdict SOUND / SOUND_WITH_CHANGES(无 UNSOUND)。要点:M7 只需更正注释措辞(别名数组本身正确);M6 须把「data-plane client.region」companion 从可选升为必做 + 注意测试 UnusedImports;M4 最干净(SOUND,唯 Caffeine 2.9.3 版本一致性待 build-verify);M2 **非** trivial「照抄 MaxCompute」——hive `planScan` 非 partition-set-scoped,必须**额外** override `planScanForPartitionBatch` 否则 batch 重复 split(红队证实),另需登记 ACID→sync + 未过滤扫描→sync 两条偏差。 - **M5** DONE `84f580c9075`(code) — iceberg 表级行数恢复 equality-delete 护栏,对齐当前旧版(上游 #64648 移动了 parity 目标)。**推翻先前签字的「不 gate」决定,用户 2026-07-11 签字**;3 处 P6.6-FIX-H4 文档批注 SUPERSEDED;`IcebergConnectorMetadataStatisticsTest` 7/7 绿。e2e live-gated(equality-delete 表 SHOW TABLE STATS=UNKNOWN,独立 iceberg + iceberg-on-HMS 同表同结果)。 +- **M7** DONE `f6de950e5bd`(code) — iceberg REST vended-cred `client.region` 别名 4→10 拓宽,逐字节对齐 fe-core `S3Properties` isRegionField 集;注释按红队更正为「S3 子集」。`IcebergCatalogFactoryTest` 62/62 绿。e2e live-gated(region 经 `AWS_REGION` 写提交不报 Unable to load region)。 diff --git a/plan-doc/tasks/designs/FIX-M7-design.md b/plan-doc/tasks/designs/FIX-M7-design.md new file mode 100644 index 00000000000000..2cbd92669718e4 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M7-design.md @@ -0,0 +1,51 @@ +# FIX-M7 — iceberg REST vended-cred client.region 别名收窄 → 拓宽对齐旧版 + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M7。 +> recon+对抗红队：`wf_40498e52-19f`（verdict SOUND_WITH_CHANGES：数组正确，唯一 required change = 注释措辞）。 + +## Problem + +REST + vended credentials 的 iceberg 目录不绑定任何 fe-filesystem S3 存储（无静态 AK/SK/role → +`S3FileSystemProvider.supports` 为 false → `chooseS3Compatible` 空）。此路 `IcebergCatalogFactory +.appendS3FileIO`（rest/jdbc/hadoop flavor）**唯一**的 region 来源是 `firstNonBlank(props, +S3_REGION_ALIASES)`。原数组只有 4 项 `{s3.region, aws.region, region, client.region}`。若 region 仅经 +`AWS_REGION` / `iceberg.rest.signing-region` / `rest.signing-region` 提供 → 返回 null → `client.region` +不发 → iceberg S3FileIO 落 AWS SDK `DefaultAwsRegionProviderChain` → 写提交报 **"Unable to load region"**。 + +## Root Cause + +`:78-83` 注释自称「mirror legacy getRegionFromProperties」但**不实**：旧版 `getRegionFromProperties` 扫所有 +`@ConnectorProperty(isRegionField=true)` 别名；fe-core `S3Properties`（:87-88）单 region field 就声明 **10** +个别名 `{s3.region, AWS_REGION, region, REGION, aws.region, glue.region, aws.glue.region, +iceberg.rest.signing-region, rest.signing-region, client.region}`。连接器手抄丢了 6 个 → 静默收窄。 + +## Design + +把连接器本地 `S3_REGION_ALIASES` 拓宽为 fe-core `S3Properties` `isRegionField` 别名集的**逐字节副本** +（10 个、同声明序，`s3.region` 仍首位胜出）。fe-connector 不 import fe-core（门禁），故本地复制（沿用 +`GLUE_ENDPOINT_PATTERN` 等既有 duplication 惯例）。 + +**注释措辞（红队 required change）**：不再声称是「getRegionFromProperties 精确镜像」——它是 `S3Properties` +`isRegionField` 别名的连接器副本，即旧版 `getRegionFromProperties` 所扫集合的 **S3 子集**；OSS/COS/OBS/Minio +子类 region 别名**刻意排除**（对 AWS-S3-backed vended REST 目录无关）。历史引用 `AbstractIcebergProperties +.toFileIOProperties` 已不存在于 fe-core，注释不再指其为「当前」。 + +## Implementation + +- `IcebergCatalogFactory.java` `:78-88`：更正注释 + 4 元数组→10 元多行数组（单消费点 `:192`；glue 路不动）。 +- `IcebergCatalogFactoryTest.java`：新增 `buildCatalogPropertiesRestVendedResolvesRegionFromWidenedAliases` + （region 仅经 `AWS_REGION`、仅经 `iceberg.rest.signing-region` → `client.region` 发出）；既有 `s3.region` + 用例保留为非回归钉。**无新 import。** + +## Risk + +- 单读点（`:192`）在 chosenS3 空臂；chosenS3-present（typed getter）与 glue（`resolveGlueRegion`）不受影响。 +- 序照 `S3Properties` 声明序 → 冲突解析 parity。无 fe-core import（字面量），门禁清；无 unused import。 +- 新增键均为合法 region 别名，空存储路无竞争值可误捕。 + +## Test + +- Unit：见上，RED（`AWS_REGION` 大写不匹配窄集小写 `aws.region`；`iceberg.rest.signing-region` 不在窄集）→ + 拓宽后 GREEN。`IcebergCatalogFactoryTest` 62/62 绿、0 checkstyle。 +- E2E live-gated：真 vended-cred REST 目录（region 经 `AWS_REGION`）写提交成功、无「Unable to load region」， + 随 P6 docker 门禁。 From 03bd4f581879a466952c8bac8e0a646cb7786d80 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:09:18 +0800 Subject: [PATCH 220/334] [fix](catalog) iceberg s3tables: fall back to SDK default credential chain (no-storage) createS3TablesCatalog hard-failed when no fe-filesystem S3 storage was bound ("requires S3-compatible storage properties"). But an EC2 instance-profile s3tables catalog carries only s3.region + a warehouse table-bucket ARN and no static AK/SK/role, so S3FileSystemProvider.supports is false -> chosenS3 empty -> throw on first access. Legacy IcebergS3TablesMetaStoreProperties supported this via the SDK DefaultCredentialsProvider chain -- a fail-loud regression. Invert the requirement: a REGION (from the bound storage or the raw props) is the sole hard requirement for s3tables; credentials fall back to the SDK default chain when no storage is bound. - IcebergCatalogFactory.resolveS3Region(props): single source of truth over the widened S3 region-alias set (M7); appendS3FileIO refactored to use it. - IcebergConnector.resolveS3TablesRegion(chosenS3, props): static/package-visible region gate (bound-storage region wins, else props), fail-loud only when neither supplies a region. Unit-testable offline without a live S3TablesClient. - createS3TablesCatalog drops the chosenS3-presence throw; buildS3TablesClient re-signatured to (Optional, region) and derives credentials as chosenS3.map(buildAwsCredentialsProvider).orElseGet(DefaultCredentialsProvider). - Companion (data-plane): buildS3TablesCatalogProperties now emits client.region from the props on the no-storage branch so S3FileIO honors an explicit s3.region rather than only IMDS. Tests (no Mockito): reframed s3TablesWithoutStorageFailsLoud -> s3TablesWithoutStorageOrRegionFailsLoud (RED at HEAD); +4 resolveS3TablesRegion gate tests (props fallback / widened alias / bound-storage-wins / fail-loud) and +1 factory companion test (RED at HEAD). fe-core untouched; connector imports no fe-core; no new production imports. Build SUCCESS, 0 checkstyle, IcebergCatalogFactoryTest 63/63 + IcebergConnectorTest 19/19 green. Live-S3 e2e gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../iceberg/IcebergCatalogFactory.java | 22 ++++++- .../connector/iceberg/IcebergConnector.java | 53 +++++++++++------ .../iceberg/IcebergCatalogFactoryTest.java | 22 ++++++- .../iceberg/IcebergConnectorTest.java | 57 ++++++++++++++++--- 4 files changed, 124 insertions(+), 30 deletions(-) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java index 284348d4940463..0a8594202c6690 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergCatalogFactory.java @@ -194,10 +194,20 @@ private static void appendS3FileIO(Map opts, Map if (chosenS3.isPresent()) { appendS3FileIOProperties(opts, chosenS3.get()); } else { - putIfNotBlank(opts, AwsClientProperties.CLIENT_REGION, firstNonBlank(props, S3_REGION_ALIASES)); + putIfNotBlank(opts, AwsClientProperties.CLIENT_REGION, resolveS3Region(props)); } } + /** + * Resolves the S3 region from the raw catalog props over {@link #S3_REGION_ALIASES} (the fe-core + * {@code S3Properties} {@code isRegionField} set). Single source of truth for the region-alias fallback, + * shared by {@link #appendS3FileIO} (the vended-cred S3FileIO branch) and the s3tables region gate + * ({@code IcebergConnector.resolveS3TablesRegion}). Returns null when no alias is set. + */ + public static String resolveS3Region(Map props) { + return firstNonBlank(props, S3_REGION_ALIASES); + } + /** * Emits ONLY the {@code s3.*} + {@link AwsClientProperties#CLIENT_REGION} S3FileIO dialect keys (no * credential-type block), shared by {@link #appendS3FileIOProperties} (rest/hadoop/jdbc/glue) and the @@ -374,7 +384,15 @@ public static Map buildCatalogProperties(Map pro public static Map buildS3TablesCatalogProperties(Map props, Optional chosenS3) { Map opts = buildBaseCatalogProperties(props); - chosenS3.ifPresent(s3 -> appendS3TablesFileIOProperties(opts, s3, props)); + if (chosenS3.isPresent()) { + appendS3TablesFileIOProperties(opts, chosenS3.get(), props); + } else { + // No bound S3 storage (e.g. an EC2 instance-profile s3tables catalog: region + warehouse ARN, no + // static creds): still propagate client.region from the raw props so the data-plane S3FileIO honors + // an explicit s3.region rather than only IMDS / DefaultAwsRegionProviderChain (mirrors the vended- + // cred branch in appendS3FileIO). Credentials are left to the SDK default chain (none are bound). + putIfNotBlank(opts, AwsClientProperties.CLIENT_REGION, resolveS3Region(props)); + } return opts; } diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java index 937c7a3c57d25c..1e0e493cfff160 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnector.java @@ -729,30 +729,44 @@ private Catalog buildRestSessionCatalogDefault(String catalogName, Map chosenS3) { - if (!chosenS3.isPresent()) { - throw new DorisConnectorException( - "Iceberg s3tables catalog requires S3-compatible storage properties (region + credentials)"); - } - S3CompatibleFileSystemProperties s3 = chosenS3.get(); - if (StringUtils.isBlank(s3.getRegion())) { - throw new DorisConnectorException( - "Iceberg s3tables catalog requires a region (set s3.region or a region-bearing endpoint)"); - } + String region = resolveS3TablesRegion(chosenS3, properties); Map catalogProps = IcebergCatalogFactory.buildS3TablesCatalogProperties(properties, chosenS3); - LOG.info("Creating Iceberg s3tables catalog '{}' region='{}'", catalogName, s3.getRegion()); + LOG.info("Creating Iceberg s3tables catalog '{}' region='{}' boundStorage={}", + catalogName, region, chosenS3.isPresent()); return buildCatalogAuthenticated(IcebergConnectorProperties.TYPE_S3_TABLES, () -> { - S3TablesClient client = buildS3TablesClient(s3); + S3TablesClient client = buildS3TablesClient(chosenS3, region); S3TablesCatalog catalog = new S3TablesCatalog(); catalog.initialize(catalogName, catalogProps, client); return catalog; }); } + /** + * Resolves the s3tables control-plane region: the bound fe-filesystem storage's region when present, else + * the raw catalog props (the widened S3 region-alias set, via {@link IcebergCatalogFactory#resolveS3Region}). + * A region is the SOLE hard requirement for s3tables; credentials fall back to the SDK default chain when no + * storage is bound. Fails loud only when NEITHER storage nor props supply a region. Static / package-visible + * so the gate is unit-testable offline without a live {@link S3TablesClient}. + */ + static String resolveS3TablesRegion( + Optional chosenS3, Map props) { + String region = chosenS3.map(S3CompatibleFileSystemProperties::getRegion) + .filter(StringUtils::isNotBlank) + .orElseGet(() -> IcebergCatalogFactory.resolveS3Region(props)); + if (StringUtils.isBlank(region)) { + throw new DorisConnectorException( + "Iceberg s3tables catalog requires a region (set s3.region or a region-bearing endpoint)"); + } + return region; + } + /** * Creates the bespoke {@code dlf} catalog, mirroring legacy {@code IcebergAliyunDLFMetaStoreProperties}: the * DLF metastore connection {@link Configuration} is built from the shared metastore-spi @@ -784,12 +798,17 @@ private Catalog createDlfCatalog(String catalogName, Optional chosenS3, String region) { + AwsCredentialsProvider credentialsProvider = chosenS3 + .map(s3 -> buildAwsCredentialsProvider(s3, properties)) + .orElseGet(DefaultCredentialsProvider::create); S3TablesClientBuilder builder = S3TablesClient.builder() - .region(Region.of(s3.getRegion())) - .credentialsProvider(buildAwsCredentialsProvider(s3, properties)); + .region(Region.of(region)) + .credentialsProvider(credentialsProvider); String endpoint = properties.get(S3TablesProperties.S3TABLES_ENDPOINT); if (StringUtils.isNotBlank(endpoint)) { builder.endpointOverride(URI.create(endpoint)); diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java index 60bc87845e913d..144c491139a2ec 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergCatalogFactoryTest.java @@ -826,9 +826,10 @@ public void buildS3TablesCatalogPropertiesExplicitStaticCredsSuppressAssumeRole( @Test public void buildS3TablesCatalogPropertiesWithoutStorageOmitsS3FileIo() { - // WHY: with no bound S3-compatible storage the props carry only the base (warehouse + manifest-cache); - // no S3FileIO keys are fabricated (the connector fails loud before building the client). MUTATION: - // emitting any s3.* / client.region without a storage -> red. + // WHY: with no bound S3-compatible storage AND no region alias in the props, only the base keys are present + // (warehouse + manifest-cache) — no s3.* credential keys are fabricated, and client.region stays absent + // because there is no region to propagate. (When a region IS present it is now emitted; see + // buildS3TablesCatalogPropertiesPropagatesClientRegionWithoutBoundS3.) MUTATION: fabricating any s3.* -> red. Map opts = IcebergCatalogFactory.buildS3TablesCatalogProperties( props("iceberg.catalog.type", "s3tables", "warehouse", "arn:aws:s3tables:us-east-1:1:bucket/b"), Optional.empty()); @@ -838,6 +839,21 @@ public void buildS3TablesCatalogPropertiesWithoutStorageOmitsS3FileIo() { Assertions.assertNull(opts.get("catalog-impl")); } + @Test + public void buildS3TablesCatalogPropertiesPropagatesClientRegionWithoutBoundS3() { + // WHY: an EC2 instance-profile s3tables catalog (no bound storage) must still propagate an explicit + // s3.region to the data-plane S3FileIO as client.region, or S3FileIO falls to IMDS / + // DefaultAwsRegionProviderChain. Parallels the REST vended-cred region test. RED at HEAD (the old + // ifPresent-only path emitted nothing without a storage). No s3.* credential keys (none are bound). + Map opts = IcebergCatalogFactory.buildS3TablesCatalogProperties( + props("iceberg.catalog.type", "s3tables", "warehouse", "arn:aws:s3tables:us-east-1:1:bucket/b", + "s3.region", "us-east-1"), + Optional.empty()); + Assertions.assertEquals("us-east-1", opts.get("client.region"), + "no-storage s3tables must propagate s3.region -> client.region for the data-plane S3FileIO"); + Assertions.assertNull(opts.get("s3.access-key-id"), "no credentials are bound"); + } + // --------------------------------------------------------------------- // resolveCatalogName — jdbc positional vs default // --------------------------------------------------------------------- diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java index d7f35004154309..5c5975cc4875bc 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorTest.java @@ -23,6 +23,7 @@ import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.handle.WriteOperation; +import org.apache.doris.filesystem.properties.S3CompatibleFileSystemProperties; import org.apache.doris.filesystem.properties.StorageProperties; import org.apache.iceberg.Schema; @@ -39,6 +40,7 @@ import java.util.EnumSet; import java.util.List; import java.util.Map; +import java.util.Optional; import java.util.Set; import java.util.function.Function; @@ -51,12 +53,13 @@ public class IcebergConnectorTest { @Test - public void s3TablesWithoutStorageFailsLoud() { - // WHY: legacy IcebergS3TablesMetaStoreProperties always derives from S3Properties.of(origProps); the - // connector needs a bound S3-compatible storage to derive the region + credentials for the control-plane - // S3TablesClient. Missing storage must fail loud (a clear DorisConnectorException), NOT silently route - // s3tables through the generic CatalogUtil path or fall back to an anonymous client. MUTATION: dropping - // the chosenS3 presence check -> a NullPointer / wrong-path error instead of this message -> red. + public void s3TablesWithoutStorageOrRegionFailsLoud() { + // WHY: a bound S3-compatible storage is NO LONGER required (M6): an EC2 instance-profile s3tables catalog + // carries only region + warehouse ARN and uses the SDK DefaultCredentialsProvider chain, mirroring legacy + // IcebergS3TablesMetaStoreProperties. The sole hard requirement is a REGION. With NEITHER a bound storage + // NOR a region alias in the props, the connector must still fail loud naming the missing region, before + // any AWS call. MUTATION: reinstating the chosenS3-presence throw -> the storage-worded message (which + // does not contain "requires a region") -> red. RecordingConnectorContext ctx = new RecordingConnectorContext(); IcebergConnector connector = new IcebergConnector( Map.of("iceberg.catalog.type", "s3tables", @@ -64,8 +67,46 @@ public void s3TablesWithoutStorageFailsLoud() { ctx); DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, () -> connector.getMetadata(null)); - Assertions.assertTrue(ex.getMessage().contains("S3-compatible storage"), - "expected a fail-loud message naming the missing S3-compatible storage, got: " + ex.getMessage()); + Assertions.assertTrue(ex.getMessage().contains("requires a region"), + "expected a fail-loud message naming the missing region, got: " + ex.getMessage()); + } + + @Test + public void s3TablesRegionResolvesFromPropsWhenNoStorageBound() { + // WHY: the M6 unblock — an EC2 instance-profile s3tables catalog (no bound storage) resolves its region + // from the raw props (s3.region here) instead of hard-failing, then uses the SDK default credential chain. + // MUTATION: reinstating the storage gate / removing the props fallback -> throws instead of resolving. + Assertions.assertEquals("us-east-1", + IcebergConnector.resolveS3TablesRegion(Optional.empty(), Map.of("s3.region", "us-east-1"))); + } + + @Test + public void s3TablesRegionResolvesFromWidenedAliasWhenNoStorageBound() { + // WHY: the props fallback scans the SAME widened S3 region-alias set as the vended-cred FileIO path (M7), + // so a region supplied only via AWS_REGION resolves. MUTATION: narrowing the alias set -> null -> throws. + Assertions.assertEquals("eu-west-1", + IcebergConnector.resolveS3TablesRegion(Optional.empty(), Map.of("AWS_REGION", "eu-west-1"))); + } + + @Test + public void s3TablesRegionPrefersBoundStorageRegion() { + // WHY: when a storage IS bound its typed region wins over a conflicting raw prop (parity with the bound- + // storage path). MUTATION: reading the props first -> us-east-1 instead of eu-west-1. + Optional bound = + Optional.of(new FakeS3CompatibleStorageProperties("S3").region("eu-west-1")); + Assertions.assertEquals("eu-west-1", + IcebergConnector.resolveS3TablesRegion(bound, Map.of("s3.region", "us-east-1"))); + } + + @Test + public void s3TablesRegionFailsLoudWhenNeitherStorageNorRegion() { + // WHY: a region is the sole hard requirement; neither a bound storage nor a props alias -> fail loud + // naming the region (Region.of("") would otherwise blow up deep in the SDK). MUTATION: returning "" / + // null instead of throwing -> no exception -> red. + DorisConnectorException ex = Assertions.assertThrows(DorisConnectorException.class, + () -> IcebergConnector.resolveS3TablesRegion(Optional.empty(), Map.of())); + Assertions.assertTrue(ex.getMessage().contains("requires a region"), + "expected a fail-loud message naming the missing region, got: " + ex.getMessage()); } @Test From 4d327fba0ad811d05b08996259ee65a288c6718e Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:11:12 +0800 Subject: [PATCH 221/334] [doc](catalog) #65185 batch-1: M6 design + track-off (iceberg subgroup M5/M7/M6 done) FIX-M6-design.md (s3tables default-credential-chain fallback; companion promoted to required) + tick M6 in the tracking table (progress row + detail + rolling log) + HANDOFF: iceberg subgroup (M5/M7/M6) done, next M4 -> M2. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 5 +- plan-doc/tasks/designs/FIX-M6-design.md | 67 ++++++++++++++++++++++ 3 files changed, 71 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M6-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4edb1e7bd59738..d4e4d099104d7a 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -30,7 +30,7 @@ - **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 **⭐ 下一步（下个 session）：** -1. **复核修复续批**（跟踪表）：**批次 1 进行中**——全 5 条已一轮 recon+对抗红队（`wf_40498e52-19f`，机制全 HEAD 确认、无 UNSOUND，设计存 `plan-doc/tasks/designs/FIX-M*-design.md` + scratch `batch1/`）。**M5 DONE**（`84f580c9075`：iceberg 表级行数恢复 equality-delete 护栏，对齐当前旧版；**推翻先前签字的「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 文档已批注 SUPERSEDED）。**续做 M7→M6→M4→M2**（红队要点见跟踪表滚动区：M7 仅注释措辞、M6 companion 升必做、M4 最干净、**M2 非照抄 MaxCompute 须额外 override `planScanForPartitionBatch`**）→ 批次 2（M3/M1 fe-core 通用节点）→ ...。⏸ 决策类（L2/L10/L12/L20）先问用户。 +1. **复核修复续批**（跟踪表）：**批次 1 进行中**——全 5 条已一轮 recon+对抗红队（`wf_40498e52-19f`，机制全 HEAD 确认、无 UNSOUND，设计存 `plan-doc/tasks/designs/FIX-M*-design.md` + scratch `batch1/`）。**iceberg 子组 M5/M7/M6 全 DONE**：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字的「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion)。**续做 M4（mc 分区缓存，最干净）→ M2（hive 批量：非照抄 MaxCompute，须额外 override `planScanForPartitionBatch`，另登记 ACID→sync + 未过滤扫描→sync 两条偏差）**→ 批次 2（M3/M1 fe-core 通用节点）→ ...。⏸ 决策类（L2/L10/L12/L20）先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 3cb3696685d5ed..3671c77280c5c9 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -34,7 +34,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | | 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ⬜ | ⬜ | ⬜ | ⬜ | | 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | -| 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ⬜ | ⬜ | ⬜ | ⬜ | +| 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ✅ | ✅ | ✅ | ✅ | | 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ✅ | ✅ | ✅ | ✅ | | 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⬜ | | 13 | **L1** | 🟡低 | tools | import-gate 三洞 | ⬜ | ⬜ | ⬜ | ⬜ | @@ -146,7 +146,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:100 records + 5 equality-delete → rowCount UNKNOWN(RED:当前锁死 100)。 ### M6 — iceberg s3tables 无显式凭证硬失败 · reverify §3 M6 -- [ ] **M6** +- [x] **M6** · DONE `03bd4f58187`（设计 `designs/FIX-M6-design.md`）。反转硬性要求=region 唯一必需（存储或 props）、无存储凭证回退 `DefaultCredentialsProvider`；新 `resolveS3Region`(factory)+`resolveS3TablesRegion`(connector 静态门)；`buildS3TablesClient(Optional,region)` 重签名；**companion 升为必做**=无存储臂发 data-plane `client.region`。测试 reframe 1 + gate 4 + companion 1（RED-able），`IcebergConnectorTest` 19/19 + `IcebergCatalogFactoryTest` 63/63 绿。依赖 M7 拓宽别名。e2e live-gated。 - **现码**:`IcebergConnector.createS3TablesCatalog:735-739`(`chosenS3` 空即 throw)← `S3FileSystemProvider.supports:64-74`(要 `hasCredential`)。legacy 用 `DefaultCredentialsProvider` 默认链。 - **Fix**:`chosenS3` 空但能从原始 props 解析 region 时(复用 M7 拓宽别名),用 `DefaultCredentialsProvider`+`Region.of(...)` 建 client 而非抛;仅无 storage 且无 region 才 fail-loud。 - **Files**:`fe-connector-iceberg/.../IcebergConnector.java`。 @@ -243,3 +243,4 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **M5** DONE `84f580c9075`(code) — iceberg 表级行数恢复 equality-delete 护栏,对齐当前旧版(上游 #64648 移动了 parity 目标)。**推翻先前签字的「不 gate」决定,用户 2026-07-11 签字**;3 处 P6.6-FIX-H4 文档批注 SUPERSEDED;`IcebergConnectorMetadataStatisticsTest` 7/7 绿。e2e live-gated(equality-delete 表 SHOW TABLE STATS=UNKNOWN,独立 iceberg + iceberg-on-HMS 同表同结果)。 - **M7** DONE `f6de950e5bd`(code) — iceberg REST vended-cred `client.region` 别名 4→10 拓宽,逐字节对齐 fe-core `S3Properties` isRegionField 集;注释按红队更正为「S3 子集」。`IcebergCatalogFactoryTest` 62/62 绿。e2e live-gated(region 经 `AWS_REGION` 写提交不报 Unable to load region)。 +- **M6** DONE `03bd4f58187`(code) — iceberg s3tables 无绑定存储不再硬失败:region 唯一必需(存储或 props),凭证回退 `DefaultCredentialsProvider` 默认链;companion 无存储臂发 data-plane `client.region`。`IcebergConnectorTest` 19/19 + `IcebergCatalogFactoryTest` 63/63 绿。**iceberg 子组(M5/M7/M6)全 DONE**。e2e live-gated(EC2 instance-profile s3tables 目录 CREATE+list 不抛)。 diff --git a/plan-doc/tasks/designs/FIX-M6-design.md b/plan-doc/tasks/designs/FIX-M6-design.md new file mode 100644 index 00000000000000..36b2e9ed4bb48c --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M6-design.md @@ -0,0 +1,67 @@ +# FIX-M6 — iceberg s3tables 无显式凭证即硬失败 → 回退 SDK 默认凭证链 + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M6。 +> recon+对抗红队：`wf_40498e52-19f`（SOUND_WITH_CHANGES：核心正确，红队要求 = data-plane client.region companion 从可选**升为必做** + 测试 UnusedImports 防护）。依赖 **M7**（拓宽后的 region 别名集）。 + +## Problem + +依赖 AWS 默认凭证链的 iceberg `s3tables` 目录（如 EC2 instance-profile：仅 `s3.region` + warehouse +table-bucket ARN，无静态 AK/SK、无 role、无 `credentials_provider_type`）首访即抛： +`Iceberg s3tables catalog requires S3-compatible storage properties (region + credentials)`。旧版 +`IcebergS3TablesMetaStoreProperties` 支持此场景（无静态凭证时用 `DefaultCredentialsProvider` 默认链）。fail-loud 回归。 + +## Root Cause + +`IcebergConnector.createS3TablesCatalog` 开头 `if (!chosenS3.isPresent()) throw ...`。`chosenS3` 来自 +`chooseS3Compatible(context.getStorageProperties())`；fe-filesystem 仅在 `S3FileSystemProvider.supports()` +为 true（要求 `hasCredential && hasLocation`）时绑定 S3 存储。region+warehouse-only 的 EC2 目录 `hasCredential=false` +→ `supports()=false` → 无绑定存储 → `chosenS3` 空 → 建表首访抛（早于任何 AWS 调用）。下游 `buildS3TablesClient` +本已能处理无静态凭证（`buildAwsCredentialsProvider` 对 DEFAULT/PROVIDER_CHAIN 返回 `DefaultCredentialsProvider`）， +故唯一阻断是 (a) 过早的 `!chosenS3.isPresent()` throw + (b) region 只从存储对象 `s3.getRegion()` 取、不看 raw props。 + +## Design + +反转硬性要求：**region**（可来自绑定存储或 raw props）是 s3tables 唯一硬性要求；无绑定存储时凭证回退 SDK 默认链。 + +1. `IcebergCatalogFactory.resolveS3Region(props)` 新 `public static`：`firstNonBlank(props, S3_REGION_ALIASES)`—— + region 别名回退的单一真源，`appendS3FileIO` 重构为调它（纯重构；`S3_REGION_ALIASES` 已由 M7 拓宽）。 +2. `IcebergConnector.resolveS3TablesRegion(chosenS3, props)` 新 `static` 包可见门：绑定存储 region 优先、否则 + `resolveS3Region(props)`；均空才 fail-loud（唯一硬失败=无 region）。静态包可见 → 离线单测无需活 `S3TablesClient`。 +3. `createS3TablesCatalog` 删 `!chosenS3.isPresent()` throw + 独立 blank-region throw，改用 `resolveS3TablesRegion`； + `buildS3TablesClient(Optional, String region)` 重签名，凭证= + `chosenS3.map(s3 -> buildAwsCredentialsProvider(s3, props)).orElseGet(DefaultCredentialsProvider::create)`、 + region=`Region.of(region)`。绑定存储路径字节不变（同 region、同凭证 provider）。 +4. **companion（红队升为必做）**：`buildS3TablesCatalogProperties` 无绑定存储臂也发 `client.region`（=`resolveS3Region(props)`）， + 使 **data-plane** `S3FileIO` 认显式 `s3.region` 而非只靠 IMDS/`DefaultAwsRegionProviderChain`（镜像 + `appendS3FileIO` vended-cred 臂）。核心修复单独已解 EC2（data-plane 经 IMDS 取 region），companion 覆盖显式 + `s3.region` ≠ host region / 非-EC2 默认链主机。 + +铁律：全连接器局部（fe-connector-iceberg；fe-filesystem-s3 不动）；无 fe-core 改/import；无源名判别；无 Mockito。 + +## Implementation + +- `IcebergCatalogFactory.java`：加 `resolveS3Region`；`appendS3FileIO` 改调它；`buildS3TablesCatalogProperties` + ifPresent→if/else（空臂发 client.region）。 +- `IcebergConnector.java`：`createS3TablesCatalog` 重写（删 throw）；加 `resolveS3TablesRegion`；`buildS3TablesClient` + 重签名 + 凭证 orElseGet 默认链；更新 3 处 javadoc。**无新 production import**（`DefaultCredentialsProvider`/ + `AwsCredentialsProvider`/`Region`/`Optional`/`StringUtils` 均已 import）。 +- 测试：见下。**测试新增 2 import（`java.util.Optional` + `S3CompatibleFileSystemProperties`）均被用**（`prefersBoundStorageRegion` + 用 typed `Optional` 局部，规避泛型不变性 + 满足 UnusedImports 门）。 + +## Risk + +- **M6→M7 序**：M6 无 M7 拓宽的别名集则 `AWS_REGION`/`*.signing-region` 供的 region 解析不到；M7 已先落（同文件、非重叠行）。 +- 行为变更（有意）：无存储不再致命，唯一 fail-loud=无 region；`s3TablesWithoutStorageFailsLoud` 断言措辞须 reframe。 +- 正常目录零回归：静态凭证/role/provider-chain/绑定存储 s3tables 路径字节不变（同 region、同凭证 provider）。 +- companion 令 `WithoutStorageOmitsS3FileIo` 的 WHY 过时（其 props 无 region 别名→client.region 仍 null，断言保持绿）， + 已更正注释 + 加正向 companion 测试。 + +## Test + +- Unit（`IcebergConnectorTest` + `IcebergCatalogFactoryTest`；无 Mockito；recording fake）： + - reframe `s3TablesWithoutStorageFailsLoud`→`s3TablesWithoutStorageOrRegionFailsLoud`（无存储无 region→抛「requires a region」，**RED at HEAD**：HEAD 抛存储措辞不含该串）。 + - `+ resolveS3TablesRegion` 4 测：props 回退 / 拓宽别名(AWS_REGION) / 绑定存储优先 / 均空 fail-loud。 + - `+ buildS3TablesCatalogPropertiesPropagatesClientRegionWithoutBoundS3`（无存储 + s3.region→client.region，**RED at HEAD**）。 + - 结果：`IcebergCatalogFactoryTest` 63/63、`IcebergConnectorTest` 19/19、0 checkstyle。 +- E2E live-gated（无本地 AWS）：真 S3/MinIO/instance-profile-emulated 目录，region+warehouse-only props CREATE CATALOG + + list namespaces 不抛 `DorisConnectorException`；随 P6.6 docker plugin-zip 门禁（memory `hms-iceberg-delegation-needs-e2e`）。 From c553c3c7696bddcbf9441aa148fc622b4724d559 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:23:41 +0800 Subject: [PATCH 222/334] [fix](catalog) maxcompute: add connector-side partition-value cache (restore legacy) Every query plan of a MaxCompute external table issued one full ODPS getPartitions() round trip: fe-core PluginDrivenExternalTable.getNameToPartitionItems (via initSelectedPartitions) calls ConnectorMetadata.listPartitions once per plan, and the connector served it with a bare uncached ODPS SDK call (both fe-core and connector layers uncached). On wide tables (tens of thousands of partitions) that is multi-second per plan plus ODPS throttling risk -- a pure perf regression versus the legacy MaxComputeExternalMetaCache.partitionValuesEntry (a TTL Caffeine cache) the SPI migration deleted. Correctness is unaffected (BE re-filters). Hive/HMS already re-homed its equivalent cache into the connector (HiveFileListingCache); MaxCompute was the only connector still explicitly cache-less. New MaxComputePartitionCache, a structural copy of HiveFileListingCache backed by the shared fe-connector-cache framework (CacheSpec + MetaCacheEntry, byte-identical contextual-only + manual-miss flags so the loader runs on the caller's TCCL-pinned thread). Keyed by (db, table); loader injected as a PartitionLister for offline testing. Held as a final field on the long-lived MaxComputeDorisConnector (metadata is rebuilt per query), injected into the metadata; all three partition-listing methods (listPartitions/listPartitionNames/listPartitionValues) route through it. The four Connector REFRESH hooks (invalidateTable/Db/All/Partition) flush the cache (invalidatePartition degrades to a whole-table flush). Config via meta.cache.max_compute.partition.{enable,ttl-second,capacity}, defaults mirroring legacy (ttl 86400s, capacity 10000). pom adds fe-connector-cache + Caffeine 2.9.3 (mirroring hive; the plugin zip bundles both automatically). No fe-core edit; connector imports no fe-core; no Mockito (recording fakes). Tests (MaxComputePartitionCacheTest, 9): 7 direct-cache + 2 metadata-integration encoding "N plans / cross-method = 1 ODPS round trip" (behaviorally RED under mutation). Build SUCCESS, 0 checkstyle, maxcompute module 113/113 (+1 live skip); plugin zip bundles exactly one caffeine-2.9.3.jar + fe-connector-cache. Live-ODPS e2e gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../fe-connector-maxcompute/pom.xml | 25 ++ .../MaxComputeConnectorMetadata.java | 21 +- .../maxcompute/MaxComputeDorisConnector.java | 46 ++- .../maxcompute/MaxComputePartitionCache.java | 169 ++++++++++ .../MaxComputeBuildTableDescriptorTest.java | 3 +- ...omputeConnectorMetadataCapabilityTest.java | 6 +- ...MaxComputeConnectorMetadataDropDbTest.java | 3 +- .../MaxComputePartitionCacheTest.java | 303 ++++++++++++++++++ .../MaxComputeValidateColumnsTest.java | 3 +- 9 files changed, 565 insertions(+), 14 deletions(-) create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java create mode 100644 fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCacheTest.java diff --git a/fe/fe-connector/fe-connector-maxcompute/pom.xml b/fe/fe-connector/fe-connector-maxcompute/pom.xml index 3ed6ac74ed0df9..37565fbc8a1140 100644 --- a/fe/fe-connector/fe-connector-maxcompute/pom.xml +++ b/fe/fe-connector/fe-connector-maxcompute/pom.xml @@ -45,6 +45,31 @@ under the License. ${project.version} + + + ${project.groupId} + fe-connector-cache + ${project.version} + + + + + com.github.ben-manes.caffeine + caffeine + 2.9.3 + + ${project.groupId} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java index bd27366b7749cd..43ccdf32c6b556 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java @@ -76,19 +76,22 @@ public class MaxComputeConnectorMetadata implements ConnectorMetadata { private final String endpoint; private final String quota; private final Map properties; + private final MaxComputePartitionCache partitionCache; public MaxComputeConnectorMetadata(Odps odps, McStructureHelper structureHelper, String defaultProject, String endpoint, String quota, - Map properties) { + Map properties, + MaxComputePartitionCache partitionCache) { this.odps = odps; this.structureHelper = structureHelper; this.defaultProject = defaultProject; this.endpoint = endpoint; this.quota = quota; this.properties = properties; + this.partitionCache = partitionCache; } @Override @@ -237,8 +240,8 @@ public org.apache.doris.thrift.TTableDescriptor buildTableDescriptor( public List listPartitionNames(ConnectorSession session, ConnectorTableHandle handle) { MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; - List partitions = structureHelper.getPartitions( - odps, mcHandle.getDbName(), mcHandle.getTableName()); + List partitions = partitionCache.getPartitions( + mcHandle.getDbName(), mcHandle.getTableName()); List names = new ArrayList<>(partitions.size()); for (Partition partition : partitions) { names.add(partition.getPartitionSpec().toString(false, true)); @@ -251,14 +254,16 @@ public List listPartitionNames(ConnectorSession session, * legacy SHOW PARTITIONS path ({@code MaxComputeExternalCatalog * #listPartitionNames}) returns the full partition set without pushing * predicates into ODPS, and this preserves that behavior. Partitions are - * read directly from ODPS with no connector-side cache (P4-T02 / OQ-4). + * served through the connector-owned {@link MaxComputePartitionCache} + * (keyed by db+table), so repeated / cross-method partition listings of the + * same table share one ODPS round trip. */ @Override public List listPartitions(ConnectorSession session, ConnectorTableHandle handle, Optional filter) { MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; - List partitions = structureHelper.getPartitions( - odps, mcHandle.getDbName(), mcHandle.getTableName()); + List partitions = partitionCache.getPartitions( + mcHandle.getDbName(), mcHandle.getTableName()); List result = new ArrayList<>(partitions.size()); for (Partition partition : partitions) { PartitionSpec spec = partition.getPartitionSpec(); @@ -276,8 +281,8 @@ public List listPartitions(ConnectorSession session, public List> listPartitionValues(ConnectorSession session, ConnectorTableHandle handle, List partitionColumns) { MaxComputeTableHandle mcHandle = (MaxComputeTableHandle) handle; - List partitions = structureHelper.getPartitions( - odps, mcHandle.getDbName(), mcHandle.getTableName()); + List partitions = partitionCache.getPartitions( + mcHandle.getDbName(), mcHandle.getTableName()); List> result = new ArrayList<>(partitions.size()); for (Partition partition : partitions) { PartitionSpec spec = partition.getPartitionSpec(); diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java index 10c7ce5d00a8a1..b435a88d0423d4 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeDorisConnector.java @@ -35,6 +35,7 @@ import org.apache.logging.log4j.Logger; import java.io.IOException; +import java.util.List; import java.util.Map; /** @@ -53,6 +54,12 @@ public class MaxComputeDorisConnector implements Connector { private final Map properties; private final ConnectorContext context; + // Connector-owned partition-listing cache, shared by the (per-call) metadata's three partition-listing + // methods. One per connector — the metadata is rebuilt per query, so the cache must live on the long-lived + // connector to survive across queries. Its loader captures this connector and reads structureHelper/odps + // lazily at query time (always post-init, since getMetadata calls ensureInitialized before use). + private final MaxComputePartitionCache partitionCache; + private Odps odps; private String endpoint; private String defaultProject; @@ -69,6 +76,8 @@ public MaxComputeDorisConnector(Map properties, ConnectorContext context) { this.properties = properties; this.context = context; + this.partitionCache = new MaxComputePartitionCache(properties, + (db, t) -> structureHelper.getPartitions(odps, db, t)); } private void ensureInitialized() { @@ -168,7 +177,42 @@ private EnvironmentSettings buildSettings() { public ConnectorMetadata getMetadata(ConnectorSession session) { ensureInitialized(); return new MaxComputeConnectorMetadata( - odps, structureHelper, defaultProject, endpoint, quota, properties); + odps, structureHelper, defaultProject, endpoint, quota, properties, partitionCache); + } + + /** + * REFRESH TABLE hook: drops this table's connector-owned partition listing. fe-core routes + * {@code REFRESH TABLE} to {@code connector.invalidateTable} for a plugin-driven catalog. Mirrors + * {@code HiveConnector.invalidateTable}. + */ + @Override + public void invalidateTable(String dbName, String tableName) { + partitionCache.invalidateTable(dbName, tableName); + } + + /** + * REFRESH DATABASE hook: drops the connector-owned partition listings for every table in one database. + * Mirrors {@code HiveConnector.invalidateDb}. + */ + @Override + public void invalidateDb(String dbName) { + partitionCache.invalidateDb(dbName); + } + + /** REFRESH CATALOG hook: drops the whole connector-owned partition cache. Mirrors {@code HiveConnector}. */ + @Override + public void invalidateAll() { + partitionCache.invalidateAll(); + } + + /** + * Invalidates a table's partition cache on a partition add/drop/alter. The cache is keyed by {@code (db, + * table)} and cannot target a single partition name, so this degrades to a whole-table flush (correctness + * -safe: the cache re-lists on the next miss). Mirrors {@code HiveConnector.invalidatePartition}. + */ + @Override + public void invalidatePartition(String dbName, String tableName, List partitionNames) { + partitionCache.invalidateTable(dbName, tableName); } @Override diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java new file mode 100644 index 00000000000000..71670e610fd6b4 --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java @@ -0,0 +1,169 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.cache.CacheSpec; +import org.apache.doris.connector.cache.MetaCacheEntry; + +import com.aliyun.odps.Partition; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.Objects; +import java.util.concurrent.ForkJoinPool; + +/** + * The MaxCompute connector's own partition-listing cache — a structural copy of the hive connector's + * {@code HiveFileListingCache}, backed by the shared + * {@code fe-connector-cache} framework ({@link CacheSpec} + {@link MetaCacheEntry}). It memoizes the (expensive) + * per-table ODPS partition listing ({@code structureHelper.getPartitions}), keyed by {@code (db, table)} — the + * ODPS project is constant per catalog, so it is NOT part of the key. + * + *

      Why this exists. Legacy fe-core kept partition listings in the engine-side external meta cache; once + * a MaxCompute catalog becomes plugin-driven that cache stops routing to it, so without this connector-owned + * cache every {@code SHOW PARTITIONS} / partition-pruning / partition-values call would re-list every partition + * from ODPS. The three metadata consumers ({@code listPartitions}, {@code listPartitionNames}, + * {@code listPartitionValues}) share ONE instance, held as a {@code final} field on the per-catalog + * {@link MaxComputeDorisConnector} (the only object that outlives a single query; the metadata is rebuilt per + * call), so a partition read warms the others. + * + *

      Read-only convention. The cached {@code List} is shared by reference. The consumers only + * read local partition accessors ({@code getPartitionSpec()}, {@code spec.keys()/get()}, {@code toString()}), + * never a per-partition lazy reload, so the shared list is safe as long as callers treat it as read-only (the + * codebase-wide metadata-cache convention). + * + *

      TCCL. The entry is contextual-only + manual-miss + no auto-refresh, so the loader runs synchronously + * on the CALLING thread — keeping TCCL/classloading byte-identical to today's uncached ODPS call (a background + * refresh thread would not inherit the caller's pin). Mirrors {@code CachingHmsClient} / {@code + * HiveFileListingCache}'s entry construction. + * + *

      Failures are not cached. The loader never caches a failed load (matching {@link MetaCacheEntry}'s + * null-is-a-miss / exception-propagates contract), so a transient ODPS failure does not poison the listing for + * the whole TTL. + */ +public class MaxComputePartitionCache { + + /** Engine token for the {@code meta.cache...*} property namespace. */ + static final String ENGINE = "max_compute"; + /** {@code meta.cache.max_compute.partition.*} — cached partition listings. */ + static final String ENTRY_PARTITION = "partition"; + + // Legacy fe-core Config values, mirrored locally (the connector never touches fe-core Config): + // TTL = Config.external_cache_expire_time_seconds_after_access (86400s = 24h) + // capacity = Config.max_external_file_cache_num (10000) + static final long DEFAULT_TTL_SECOND = 86400L; + static final long DEFAULT_PARTITION_CAPACITY = 10000L; + + /** + * The raw partition lister: {@code structureHelper.getPartitions(odps, db, table)} in production, a fake in + * unit tests. Injected so the cache's hit/miss/invalidation behaviour is testable without a live ODPS client + * (mirrors the hive connector's {@code HiveFileListingCache.DirectoryLister}). + */ + @FunctionalInterface + interface PartitionLister { + List list(String dbName, String tableName); + } + + private final MetaCacheEntry> cache; + private final PartitionLister lister; + + MaxComputePartitionCache(Map properties, PartitionLister lister) { + Map props = properties == null ? Collections.emptyMap() : properties; + CacheSpec spec = CacheSpec.fromProperties(props, ENGINE, ENTRY_PARTITION, + CacheSpec.of(true, DEFAULT_TTL_SECOND, DEFAULT_PARTITION_CAPACITY)); + // Contextual-only + manual-miss so the slow getPartitions runs on the caller (TCCL-pinned) thread outside + // Caffeine's sync compute lock, deduplicated by a striped lock — mirrors CachingHmsClient's entries. + this.cache = new MetaCacheEntry<>("max_compute.partition", null, spec, ForkJoinPool.commonPool(), + false, true, 0L, true); + this.lister = Objects.requireNonNull(lister, "lister can not be null"); + } + + /** + * Returns all partitions of {@code (dbName, tableName)}, served from the cache. Keyed by {@code (db, table)} + * so {@link #invalidateTable} can drop exactly one table's entry. The loader runs on the calling thread; a + * failure propagates and is NOT cached. The returned list is shared by reference — callers must treat it as + * read-only (the codebase-wide metadata-cache convention). + */ + public List getPartitions(String dbName, String tableName) { + return cache.get(new PartitionKey(dbName, tableName), + key -> lister.list(key.dbName, key.tableName)); + } + + /** Drops the cached partition listing for one table. Backs {@code REFRESH TABLE}. */ + public void invalidateTable(String dbName, String tableName) { + cache.invalidateIf(key -> key.matches(dbName, tableName)); + } + + /** Drops every cached partition listing for one database (all its tables). Backs {@code REFRESH DATABASE}. */ + public void invalidateDb(String dbName) { + cache.invalidateIf(key -> key.matchesDb(dbName)); + } + + /** Drops the whole partition cache. Backs {@code REFRESH CATALOG}. */ + public void invalidateAll() { + cache.invalidateAll(); + } + + /** Current number of cached partition listings — for unit tests only (mirrors HiveFileListingCache.size()). */ + long size() { + long[] count = {0L}; + cache.forEach((key, value) -> count[0]++); + return count[0]; + } + + /** + * Cache key: (db, table). db+table let {@link #invalidateTable} select one table's entry; db alone lets + * {@link #invalidateDb} select one database's entries. + */ + static final class PartitionKey { + private final String dbName; + private final String tableName; + + PartitionKey(String dbName, String tableName) { + this.dbName = dbName; + this.tableName = tableName; + } + + boolean matches(String db, String table) { + return Objects.equals(dbName, db) && Objects.equals(tableName, table); + } + + boolean matchesDb(String db) { + return Objects.equals(dbName, db); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof PartitionKey)) { + return false; + } + PartitionKey that = (PartitionKey) o; + return Objects.equals(dbName, that.dbName) + && Objects.equals(tableName, that.tableName); + } + + @Override + public int hashCode() { + return Objects.hash(dbName, tableName); + } + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java index e5ad71f91075df..64a7381b51b7db 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeBuildTableDescriptorTest.java @@ -58,7 +58,8 @@ public void buildsMaxComputeTableDescriptorWithAuthAndAddressing() { properties.put("mc.secret_key", "test-sk"); MaxComputeConnectorMetadata metadata = new MaxComputeConnectorMetadata( - null, null, "default_project", endpoint, quota, properties); + null, null, "default_project", endpoint, quota, properties, + null); // null: partition cache unused by this test // dbName / remoteName are already remote names at the real call site (OQ-7). long tableId = 42L; diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java index 6dcb647f1b8a87..b92c651635ea6c 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataCapabilityTest.java @@ -43,7 +43,8 @@ public class MaxComputeConnectorMetadataCapabilityTest { @Test public void maxComputeDeclaresSupportsCreateDatabase() { MaxComputeConnectorMetadata metadata = new MaxComputeConnectorMetadata( - null, null, "proj", "ep", "quota", Collections.emptyMap()); + null, null, "proj", "ep", "quota", Collections.emptyMap(), + null); // null: partition cache unused by this test Assertions.assertTrue(metadata.supportsCreateDatabase(), "MaxCompute must declare supportsCreateDatabase()=true so the FE " @@ -65,7 +66,8 @@ public void maxComputeDeclaresSupportsCreateDatabase() { @Test public void maxComputeDisablesCastPredicatePushdown() { MaxComputeConnectorMetadata metadata = new MaxComputeConnectorMetadata( - null, null, "proj", "ep", "quota", Collections.emptyMap()); + null, null, "proj", "ep", "quota", Collections.emptyMap(), + null); // null: partition cache unused by this test Assertions.assertFalse(metadata.supportsCastPredicatePushdown(null), "MaxCompute must disable CAST-predicate pushdown (F9): the converter unwraps CAST " diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java index 2c47bd3bd9d89f..f5fb465948e46e 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadataDropDbTest.java @@ -58,7 +58,8 @@ public class MaxComputeConnectorMetadataDropDbTest { private MaxComputeConnectorMetadata metadataWith(RecordingStructureHelper helper) { return new MaxComputeConnectorMetadata( - null /* odps */, helper, "proj", "ep", "quota", Collections.emptyMap()); + null /* odps */, helper, "proj", "ep", "quota", Collections.emptyMap(), + null); // null: partition cache unused by this test } @Test diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCacheTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCacheTest.java new file mode 100644 index 00000000000000..b957bb7a28579a --- /dev/null +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCacheTest.java @@ -0,0 +1,303 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.maxcompute; + +import org.apache.doris.connector.api.DorisConnectorException; + +import com.aliyun.odps.Odps; +import com.aliyun.odps.OdpsException; +import com.aliyun.odps.Partition; +import com.aliyun.odps.Table; +import com.aliyun.odps.TableSchema; +import com.aliyun.odps.Tables; +import com.aliyun.odps.table.TableIdentifier; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Collections; +import java.util.HashMap; +import java.util.Iterator; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests {@link MaxComputePartitionCache}: the connector-owned partition-listing cache (a structural copy of the + * hive connector's {@code HiveFileListingCache}), backed by the shared {@code fe-connector-cache} framework. + * + *

      WHY (Rule 9): after the max_compute cutover the fe-core engine-side external meta cache stops routing to a + * MaxCompute catalog, so without this connector-owned cache every {@code SHOW PARTITIONS} / partition-pruning / + * partition-values call would re-list every partition from ODPS. These tests pin the behaviours that make the + * re-homed cache correct: (1) a partition listing is cached keyed by {@code (db, table)} so it loads once; the + * db / table dimensions never collide; (2) {@code invalidateTable}/{@code invalidateDb}/{@code invalidateAll} + * drop exactly the intended scope (arming REFRESH TABLE / DATABASE / CATALOG); (3) the per-entry + * {@code meta.cache.max_compute.partition.*} knob turns it off; (4) a load failure propagates and is NOT cached. + * Two integration tests prove the metadata's partition-listing methods are served from the cache, so repeated / + * cross-method listings of the same table share ONE ODPS round trip.

      + */ +public class MaxComputePartitionCacheTest { + + // ==================== caching: hit / miss keyed by (db, table) ==================== + + @Test + public void partitionsAreCachedPerTable() { + CountingPartitionLister lister = new CountingPartitionLister(); + MaxComputePartitionCache cache = new MaxComputePartitionCache(Collections.emptyMap(), lister); + + List a = cache.getPartitions("db", "t"); + List b = cache.getPartitions("db", "t"); + // WHY: a hit must serve the cached listing without re-listing ODPS. + Assertions.assertSame(a, b); + Assertions.assertEquals(1, lister.totalCalls); + } + + @Test + public void keyScopedByDbAndTable() { + CountingPartitionLister lister = new CountingPartitionLister(); + MaxComputePartitionCache cache = new MaxComputePartitionCache(Collections.emptyMap(), lister); + + // WHY: (db, table) must both be part of the key so invalidateTable can scope one table — and so two + // tables that share a name across dbs never serve each other's listing. + cache.getPartitions("db1", "t"); + cache.getPartitions("db2", "t"); + cache.getPartitions("db1", "t2"); + Assertions.assertEquals(3, lister.totalCalls); + } + + // ==================== invalidation (arms REFRESH TABLE / DATABASE / CATALOG) ==================== + + @Test + public void invalidateTableDropsOnlyThatTable() { + CountingPartitionLister lister = new CountingPartitionLister(); + MaxComputePartitionCache cache = new MaxComputePartitionCache(Collections.emptyMap(), lister); + + cache.getPartitions("db", "t1"); + cache.getPartitions("db", "t2"); + Assertions.assertEquals(2, lister.totalCalls); + + cache.invalidateTable("db", "t1"); + + // WHY: t1 must re-list after its invalidation... + cache.getPartitions("db", "t1"); + Assertions.assertEquals(2, (int) lister.callsPerTable.get("db/t1")); + // ...while t2's entry (a different table) must survive — invalidateTable is scoped by (db, table). + cache.getPartitions("db", "t2"); + Assertions.assertEquals(1, (int) lister.callsPerTable.get("db/t2")); + } + + @Test + public void invalidateAllDropsEverything() { + CountingPartitionLister lister = new CountingPartitionLister(); + MaxComputePartitionCache cache = new MaxComputePartitionCache(Collections.emptyMap(), lister); + + cache.getPartitions("db", "t1"); + cache.getPartitions("db", "t2"); + Assertions.assertEquals(2, lister.totalCalls); + + cache.invalidateAll(); + + // WHY: every entry must reload after invalidateAll (REFRESH CATALOG). + cache.getPartitions("db", "t1"); + cache.getPartitions("db", "t2"); + Assertions.assertEquals(4, lister.totalCalls); + } + + @Test + public void invalidateDbDropsOneDb() { + CountingPartitionLister lister = new CountingPartitionLister(); + MaxComputePartitionCache cache = new MaxComputePartitionCache(Collections.emptyMap(), lister); + + cache.getPartitions("db1", "t"); + cache.getPartitions("db2", "t"); + Assertions.assertEquals(2, lister.totalCalls); + + cache.invalidateDb("db1"); + + // WHY: only db1's entries reload (REFRESH DATABASE db1)... + cache.getPartitions("db1", "t"); + Assertions.assertEquals(2, (int) lister.callsPerTable.get("db1/t")); + // ...db2's entry survives. + cache.getPartitions("db2", "t"); + Assertions.assertEquals(1, (int) lister.callsPerTable.get("db2/t")); + } + + // ==================== per-entry property knob ==================== + + @Test + public void disablingViaPropsBypassesTheCache() { + CountingPartitionLister lister = new CountingPartitionLister(); + MaxComputePartitionCache cache = new MaxComputePartitionCache( + Collections.singletonMap("meta.cache.max_compute.partition.enable", "false"), lister); + + cache.getPartitions("db", "t"); + cache.getPartitions("db", "t"); + // WHY: meta.cache.max_compute.partition.enable=false must bypass caching entirely — every call re-lists. + Assertions.assertEquals(2, lister.totalCalls); + } + + // ==================== failures are propagated, never cached ==================== + + @Test + public void loadFailureIsNotCached() { + CountingPartitionLister lister = new CountingPartitionLister(); + lister.error = new DorisConnectorException("boom"); + MaxComputePartitionCache cache = new MaxComputePartitionCache(Collections.emptyMap(), lister); + + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> cache.getPartitions("db", "t")); + Assertions.assertEquals("boom", e.getMessage()); + Assertions.assertEquals(1, lister.totalCalls); + // WHY (Rule 9): a failed load must leave NO cache entry, else a momentary ODPS blip would poison the + // listing for the whole TTL (a "catch -> return emptyList" loader mutation would cache an empty listing). + Assertions.assertEquals(0L, cache.size()); + + // WHY: after recovery the next clean call re-lists and succeeds. + lister.error = null; + cache.getPartitions("db", "t"); + Assertions.assertEquals(2, lister.totalCalls); + } + + // ==================== integration: the metadata's partition methods are cache-backed ==================== + + @Test + public void twoListPartitionsShareOneRoundTrip() { + CountingStructureHelper helper = new CountingStructureHelper(); + MaxComputePartitionCache cache = new MaxComputePartitionCache( + Collections.emptyMap(), (db, t) -> helper.getPartitions(null, db, t)); + MaxComputeConnectorMetadata md = new MaxComputeConnectorMetadata( + null, helper, "proj", "ep", "quota", Collections.emptyMap(), cache); + MaxComputeTableHandle handle = new MaxComputeTableHandle("db", "t", null, null); + + md.listPartitions(null, handle, Optional.empty()); + md.listPartitions(null, handle, Optional.empty()); + + // WHY: the second listPartitions is served from the cache — one ODPS round trip, not two. + Assertions.assertEquals(1, helper.getPartitionsCalls); + } + + @Test + public void crossMethodShareOneRoundTrip() { + CountingStructureHelper helper = new CountingStructureHelper(); + MaxComputePartitionCache cache = new MaxComputePartitionCache( + Collections.emptyMap(), (db, t) -> helper.getPartitions(null, db, t)); + MaxComputeConnectorMetadata md = new MaxComputeConnectorMetadata( + null, helper, "proj", "ep", "quota", Collections.emptyMap(), cache); + MaxComputeTableHandle handle = new MaxComputeTableHandle("db", "t", null, null); + + md.listPartitions(null, handle, Optional.empty()); + md.listPartitionNames(null, handle); + + // WHY: listPartitions and listPartitionNames of the SAME table share ONE cached listing (the §5th-cache + // coupling): a partition read warms the other method too. + Assertions.assertEquals(1, helper.getPartitionsCalls); + } + + /** + * A {@link MaxComputePartitionCache.PartitionLister} double: counts calls (total + per {@code db/table}) and + * returns a FRESH empty list per call, so reference identity distinguishes a cache hit from a reload (a real + * {@code Partition} needs a live ODPS client, which the connector test module has no Mockito to fake). Throws + * {@link #error} when set, to exercise the failure-not-cached path. + */ + private static final class CountingPartitionLister implements MaxComputePartitionCache.PartitionLister { + final Map callsPerTable = new HashMap<>(); + int totalCalls; + RuntimeException error; + + @Override + public List list(String dbName, String tableName) { + totalCalls++; + callsPerTable.merge(dbName + "/" + tableName, 1, Integer::sum); + if (error != null) { + throw error; + } + return new ArrayList<>(); + } + } + + /** + * Recording {@link McStructureHelper}: its {@code getPartitions} returns an empty list and increments a + * counter (= the SDK round trip), so an integration test can assert the metadata was served from the cache. + * Every other method returns a harmless default (none is invoked on the partition-listing path). + */ + private static final class CountingStructureHelper implements McStructureHelper { + int getPartitionsCalls; + + @Override + public List getPartitions(Odps mcClient, String dbName, String tableName) { + getPartitionsCalls++; + return Collections.emptyList(); + } + + // ---- unused on the partition-listing path: harmless defaults ---- + + @Override + public List listTableNames(Odps mcClient, String dbName) { + return Collections.emptyList(); + } + + @Override + public List listDatabaseNames(Odps mcClient, String defaultProject) { + return Collections.emptyList(); + } + + @Override + public boolean tableExist(Odps mcClient, String dbName, String tableName) { + return false; + } + + @Override + public boolean databaseExist(Odps mcClient, String dbName) { + return false; + } + + @Override + public TableIdentifier getTableIdentifier(String dbName, String tableName) { + return null; + } + + @Override + public Iterator getPartitionIterator(Odps mcClient, String dbName, String tableName) { + return Collections.emptyIterator(); + } + + @Override + public Table getOdpsTable(Odps mcClient, String dbName, String tableName) { + return null; + } + + @Override + public Tables.TableCreator createTableCreator(Odps mcClient, String dbName, + String tableName, TableSchema schema) { + return null; + } + + @Override + public void dropTable(Odps mcClient, String dbName, String tableName, boolean ifExists) + throws OdpsException { + } + + @Override + public void createDb(Odps mcClient, String dbName, boolean ifNotExists) { + } + + @Override + public void dropDb(Odps mcClient, String dbName, boolean ifExists) { + } + } +} diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java index 649085c76ab35b..1d9230adfdd0d6 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputeValidateColumnsTest.java @@ -48,7 +48,8 @@ public class MaxComputeValidateColumnsTest { private MaxComputeConnectorMetadata metadata() { return new MaxComputeConnectorMetadata( - null, null, "proj", "ep", "quota", Collections.emptyMap()); + null, null, "proj", "ep", "quota", Collections.emptyMap(), + null); // null: partition cache unused by this test } @Test From 5bbb1a81df70dbe2a45e263353e0e13b5184cfe6 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:25:06 +0800 Subject: [PATCH 223/334] [doc](catalog) #65185 batch-1: M4 design + track-off FIX-M4-design.md (maxcompute connector-side partition cache; records impl-subagent deviations + the verified caffeine-2.9.3 plugin-zip coherence) + tick M4 in the tracking table (progress row + detail + rolling log). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/task-list-65185-reverify-fixes.md | 5 +- plan-doc/tasks/designs/FIX-M4-design.md | 67 ++++++++++++++++++++++ 2 files changed, 70 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M4-design.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 3671c77280c5c9..a099235e9db7ac 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -32,7 +32,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | | 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ⬜ | ⬜ | ⬜ | ⬜ | | 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | -| 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ⬜ | ⬜ | ⬜ | ⬜ | +| 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ✅ | ✅ | ✅ | ✅ | | 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | | 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ✅ | ✅ | ✅ | ✅ | | 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ✅ | ✅ | ✅ | ✅ | @@ -132,7 +132,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:纯 helper 单测覆盖 `(isPruned=false, 非 NOT_PRUNED, size≥阈值)` 应 →batch(RED:返回 false)。 ### M4 — MC 分区值缓存删除 · reverify §3 M4 -- [ ] **M4** +- [x] **M4** · DONE `c553c3c7696`（设计 `designs/FIX-M4-design.md`；实现经 impl-subagent + 独立 build/test/zip 核验）。新 `MaxComputePartitionCache`=`HiveFileListingCache` 结构副本（共享 `fe-connector-cache`，contextual-only+manual-miss flag 字节一致），keyed(db,table)，持于 `MaxComputeDorisConnector`、注入 metadata、三方法走它、4 个 REFRESH 钩子刷；pom 加 `fe-connector-cache`+Caffeine 2.9.3。`MaxComputePartitionCacheTest` 9/9 + 模块 113/113、0 checkstyle、import 门 0、**插件 zip 恰含单个 caffeine-2.9.3.jar**。e2e live-gated。 - **现码**:`PluginDrivenExternalTable.getNameToPartitionItems:780-781`(每次 `listPartitions`)→ `MaxComputeConnectorMetadata.listPartitions:256-273` → `McStructureHelper.getPartitions:112-118`(裸 ODPS SDK)。fe-core+连接器**两层无缓存**。 - **Fix**:maxcompute 连接器内加 TTL/容量 Caffeine(仿 `CachingHmsClient`/`HiveFileListingCache`),keyed `(project,db,table)`,`REFRESH` 失效。**不**在 fe-core 加二级缓存。若延后须登记 CACHE-P1。 - **Files**:`fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java`(+ 新缓存类)。 @@ -244,3 +244,4 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **M5** DONE `84f580c9075`(code) — iceberg 表级行数恢复 equality-delete 护栏,对齐当前旧版(上游 #64648 移动了 parity 目标)。**推翻先前签字的「不 gate」决定,用户 2026-07-11 签字**;3 处 P6.6-FIX-H4 文档批注 SUPERSEDED;`IcebergConnectorMetadataStatisticsTest` 7/7 绿。e2e live-gated(equality-delete 表 SHOW TABLE STATS=UNKNOWN,独立 iceberg + iceberg-on-HMS 同表同结果)。 - **M7** DONE `f6de950e5bd`(code) — iceberg REST vended-cred `client.region` 别名 4→10 拓宽,逐字节对齐 fe-core `S3Properties` isRegionField 集;注释按红队更正为「S3 子集」。`IcebergCatalogFactoryTest` 62/62 绿。e2e live-gated(region 经 `AWS_REGION` 写提交不报 Unable to load region)。 - **M6** DONE `03bd4f58187`(code) — iceberg s3tables 无绑定存储不再硬失败:region 唯一必需(存储或 props),凭证回退 `DefaultCredentialsProvider` 默认链;companion 无存储臂发 data-plane `client.region`。`IcebergConnectorTest` 19/19 + `IcebergCatalogFactoryTest` 63/63 绿。**iceberg 子组(M5/M7/M6)全 DONE**。e2e live-gated(EC2 instance-profile s3tables 目录 CREATE+list 不抛)。 +- **M4** DONE `c553c3c7696`(code) — maxcompute 连接器内 `MaxComputePartitionCache`(HiveFileListingCache 结构副本)恢复被删的分区值缓存,消除每规划一次全量 ODPS listPartitions;4 REFRESH 钩子刷;pom+Caffeine 2.9.3。9/9+模块 113/113 绿,插件 zip 单 caffeine 版本。e2e live-gated。 diff --git a/plan-doc/tasks/designs/FIX-M4-design.md b/plan-doc/tasks/designs/FIX-M4-design.md new file mode 100644 index 00000000000000..84b7b086bfb562 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M4-design.md @@ -0,0 +1,67 @@ +# FIX-M4 — maxcompute 分区值缓存删除 → 连接器内重建（对齐旧版） + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M4。 +> recon+对抗红队：`wf_40498e52-19f`（verdict **SOUND**：连接器局部、正确 scope、忠实镜像 `HiveFileListingCache`，无铁律违背，测试编码真实 bug）。 + +## Problem + +MaxCompute 外表**每次查询规划**都发一次全量 ODPS `getPartitions()` 往返。fe-core +`PluginDrivenExternalTable.getNameToPartitionItems`（经 `initSelectedPartitions`）每规划调一次 +`ConnectorMetadata.listPartitions`，连接器用裸 ODPS SDK 调用服务它（fe-core + 连接器两层都无缓存）。宽表 +（数万分区）每规划多秒 + ODPS 限流风险。纯性能回归——旧版 `MaxComputeExternalMetaCache.partitionValuesEntry` +（TTL Caffeine 缓存）被 SPI 迁移删了；正确性不受影响（BE 重过滤）。Hive/HMS 已在 P7 把等价缓存 re-home 进连接器 +（`CachingHmsClient`/`HiveFileListingCache`），MaxCompute 是唯一仍显式无缓存的连接器。 + +## Root Cause + +`MaxComputeDorisConnector.getMetadata()` 每次建全新 `MaxComputeConnectorMetadata`，其 `listPartitions`/ +`listPartitionNames`/`listPartitionValues` 全调 `structureHelper.getPartitions(odps, db, table)`（网络 RPC） +无任何 memoization。metadata 对象 per-call，故缓存须落在长寿的 per-catalog `MaxComputeDorisConnector` 上并注入 +metadata——正是 `HiveConnector` 持 `HiveFileListingCache` 交给 metadata 的方式。 + +## Design（连接器局部；无 fe-core 改/不解析属性） + +1. 新 `MaxComputePartitionCache` = `HiveFileListingCache` 结构副本，靠共享 `fe-connector-cache`（`CacheSpec`+ + `MetaCacheEntry`）。keyed `(db, table)`（ODPS project per-catalog 恒定，不入 key）。value=`List` + 按引用缓存（只读约定；三消费方只读本地 accessor `getPartitionSpec()`/`spec.keys()/get()`/`toString()`，无 + per-partition 懒加载）。`MetaCacheEntry` **contextual-only + manual-miss**（flag 元组 `(false,true,0L,true)` + 与 `HiveFileListingCache` 字节一致）→ 慢 loader 在调用线程同步跑（striped-lock dedup），TCCL/classloading 与 + 今日裸调用字节一致。loader 经 `PartitionLister` 注入（无 Mockito 可测），生产 loader + `(db,t)->structureHelper.getPartitions(odps,db,t)`。config `meta.cache.max_compute.partition.{enable,ttl-second, + capacity}`，默认对齐旧版（ttl 86400s、capacity 10000）。 +2. `MaxComputeDorisConnector` 持 `final` 缓存（ctor 建，loader lambda 捕获 this、查询时惰读 structureHelper/odps， + 均 post-init），`getMetadata` 注入；override 4 个 `Connector` REFRESH 钩子（`invalidateTable/Db/All/Partition`） + 路由到缓存（`invalidatePartition` 降级为整表刷，正确安全——miss 重列）。镜像 `HiveConnector`。 +3. `MaxComputeConnectorMetadata` 加末位 ctor 参 + 三方法改走 `partitionCache.getPartitions`；订正 stale javadoc。 +4. `pom.xml` 加 `fe-connector-cache` + Caffeine `2.9.3`（compile；插件 zip 自动打入 `lib/`，镜像 hive）。 + +## 实现要点 / 与设计的偏差（实现 agent 记录，已核） + +- `MaxComputePartitionCache` 用**单个** package-private ctor `(Map, PartitionLister)`（非 hive 的双 ctor）： + hive 默认 lister 是自包含 static 方法故可给 `(Map)`-only 公共 ctor；MaxCompute loader 需活连接器状态 + （odps/structureHelper），按设计恒由连接器注入 lambda，故只留注入式 ctor。连接器与测试均同包，无碍。 +- 跨模块 javadoc `HiveFileListingCache`/`DirectoryLister` 引用软化为 `{@code}`（maxcompute 不依赖 hive， + `@link` 会 doclint 失败）；模块内 `@link`（CacheSpec/MetaCacheEntry/…）保留。 +- 5 处既有 metadata-ctor 测试站补 `null`（含注释）+ 1 处生产 `getMetadata` 注入真缓存。 + +## Risk + +- 陈旧（有意）：TTL 内重复规划见缓存分区集；带外新增分区 TTL/REFRESH 前不可见。恢复旧版行为，受 REFRESH 钩子 + + `ttl-second` 约束、正确安全（BE 重过滤）。release note 记。 +- 按引用缓存 `Partition`：仅因消费方读列表响应已填充的本地 accessor 而安全（无路径读懒加载字段）。 +- 打包：MaxCompute 原无 Caffeine；漏打 2.9.3 → 运行期 `NoClassDefFoundError`（memory + `catalog-spi-connector-cache-framework-caffeine-coherence`）。**已实测**插件 zip 恰含一个 `caffeine-2.9.3.jar` + + `fe-connector-cache`，odps-sdk 无冲突版本。 +- ctor churn：5 测试站传 `null`；未来触分区的测试在这些文件会 NPE（null-arg 注释已标）。 +- TCCL：须 contextual-only + manual-miss（loader 在 pin 的调用线程）；勿切 auto-refresh/后台条目。 + +## Test + +- Unit `MaxComputePartitionCacheTest`（JUnit5、无 Mockito、recording fake）9 测： + - 直接缓存 7（per-table 命中/键作用域/invalidate table·db·all/disable 绕过/失败不缓存）； + - metadata 集成 2：`twoListPartitionsShareOneRoundTrip`（两 listPartitions→helper 计数==1）、 + `crossMethodShareOneRoundTrip`（listPartitions+listPartitionNames→==1，杀「只缓存一个方法」的半修）。 + - RED：直接组 greenfield 非编译红；集成组变异红（只回退三方法体、留 ctor 参 → 计数读 2 → `assertEquals(1,..)` 挂）。 + - 结果：9/9 + 模块 113/113（1 live skip）、0 checkstyle、import 门 exit 0、插件 zip caffeine 单版本。 +- E2E live-gated：需真 MaxCompute 目录+凭证（同 `OdpsLiveConnectivityTest` 门），往返计数 SQL 不可观测→连接器单测为 + 权威验证（`HiveFileListingCache` 亦无 live e2e）。有 live env 则验修前后查询正确性不变 + REFRESH TABLE 拾带外新增分区。 From 702153885ab4e278428866a97e41d568235fa4ed Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:35:48 +0800 Subject: [PATCH 224/334] [fix](catalog) hive: restore batch/async partition-split path (post-cutover) After the HMS cutover every hive table scans through the generic PluginDrivenScanNode, which enters batch/async split generation only when the connector opts in. HiveScanPlanProvider opted into neither flavor (supportsBatchScan=false, streamingSplitEstimate=-1), so a large partitioned hive scan materialized every file split of every selected partition into the FE heap in one synchronous pass -- worst on hive precisely because it is the biggest partition-count source. Legacy HiveScanNode went async at prunedPartitions.size() >= num_partitions_in_batch_mode (default 1024). Results stayed correct; FE heap + planning latency regressed. This is NOT a clean MaxCompute mimic: MaxCompute overrides only supportsBatchScan and relies on the SPI default planScanForPartitionBatch, correct only because its planScan is partition-set-scoped. Hive's planScan resolves from handle.getPrunedPartitions() and ignores the passed set, so inheriting the default would re-scan the whole pruned set per batch -> every partition's files emitted once per batch -> DUPLICATE ROWS. So hive overrides BOTH: - supportsBatchScan: true iff partitioned AND not transactional. ACID excluded deliberately (same isTransactional() accessor planScan branches on): the scan opens one metastore read transaction; per-batch resolution on background threads would open and leak a read transaction per batch. ACID partitioned tables keep the synchronous path (correct, not streamed). - planScanForPartitionBatch: resolves ONLY the batch via hmsClient.getPartitions(db, table, partitionBatch) (the batch strings are the HMS-rendered key=value/... names getPartitions accepts), then the SAME helpers as planScan (format detect, split size, hadoop conf, convertPartitions, listAndSplitFiles). No new imports. Deviations registered (fail-safe, results unaffected): - BATCH-ACID-SYNC (permanent, by-design): ACID partitioned tables stay synchronous. - BATCH-UNPRUNED-SYNC (resolved by M3, batch 2): a truly-unfiltered scan (no WHERE) keeps isPruned=false so fe-core's shouldUseBatchMode stays false; M3's ==NOT_PRUNED fix re-enables batch for it. No fe-core edit; connector imports no fe-core; no Mockito (real HmsClient / DirectoryLister / ConnectorSession fakes). New HiveScanBatchModeTest (4): supportsBatchScan true/false/false + batch-scoping size==1 (RED under the default -> 3 ranges, encoding the duplicate-splits bug). Build SUCCESS, 0 checkstyle, hive module 284/284 green. Large-partition async e2e gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../connector/hive/HiveScanPlanProvider.java | 74 ++++- .../connector/hive/HiveScanBatchModeTest.java | 273 ++++++++++++++++++ 2 files changed, 346 insertions(+), 1 deletion(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 8c302e3a45539a..066214f4f27b08 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -59,7 +59,8 @@ *
        *
      • No file listing cache (lists files directly each time)
        • *
      • No ACID transaction support (non-transactional tables only)
        • - *
      • No batch/lazy split mode
        • + *
      • No file-count streaming split mode; partition-batch mode is limited to partitioned + * non-transactional tables (see {@link #supportsBatchScan})
        • *
      • No table sampling
        • *
      */ @@ -145,6 +146,77 @@ public List planScan( return ranges; } + /** + * Whether this hive table supports batched split generation (see + * {@link ConnectorScanPlanProvider#supportsBatchScan}): {@code true} iff the table is PARTITIONED (has + * partition keys) and NOT transactional. A large partitioned scan then streams its splits per partition + * batch via {@link #planScanForPartitionBatch} on a background pool instead of materializing every + * partition's files synchronously (legacy {@code HiveScanNode} went async at + * {@code num_partitions_in_batch_mode}). + * + *

      Transactional/ACID tables are excluded DELIBERATELY: their scan opens one metastore read transaction + * (see {@link #planAcidScan}); running the per-batch resolution on background threads would open — and leak + * — a read transaction per batch. ACID partitioned tables keep the synchronous {@link #planScan} path + * (correct, just not streamed). The transactional test uses the SAME {@code isTransactional()} accessor + * {@link #planScan} branches on.

      + */ + @Override + public boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle handle) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + List partKeyNames = hiveHandle.getPartitionKeyNames(); + return partKeyNames != null && !partKeyNames.isEmpty() && !hiveHandle.isTransactional(); + } + + /** + * Plans the scan for a SINGLE partition batch (see + * {@link ConnectorScanPlanProvider#planScanForPartitionBatch}). Unlike {@link #planScan} — which resolves + * partitions from {@code handle.getPrunedPartitions()} and IGNORES the passed partition set — this MUST scope + * resolution to {@code partitionBatch}: {@code PluginDrivenScanNode} slices the pruned partition NAMES into + * batches and calls this once per batch, so inheriting the SPI default (which re-runs the whole-pruned-set + * {@link #planScan} per batch) would emit every partition's files once per batch — duplicate rows. The batch + * strings are the metastore-rendered {@code key=value/...} partition names (the keys of the Nereids + * selected-partition map), exactly the form {@code hmsClient.getPartitions} accepts, so batch-scoped + * resolution is a clean {@code getPartitions(batch)} round-trip. Only the non-ACID path is reachable here + * ({@link #supportsBatchScan} excludes transactional tables). Reuses the SAME helpers as {@link #planScan}: + * format detect, split size, hadoop conf, {@link #convertPartitions}, {@link #listAndSplitFiles}. + */ + @Override + public List planScanForPartitionBatch( + ConnectorSession session, + ConnectorTableHandle handle, + List columns, + Optional filter, + long limit, + List partitionBatch) { + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + String dbName = hiveHandle.getDbName(); + String tableName = hiveHandle.getTableName(); + + // Resolve ONLY this batch's partitions (scoped to partitionBatch), NOT handle.getPrunedPartitions(). + List hmsPartitions = hmsClient.getPartitions(dbName, tableName, partitionBatch); + List partitions = convertPartitions( + hmsPartitions, hiveHandle.getPartitionKeyNames()); + if (partitions.isEmpty()) { + return Collections.emptyList(); + } + + HiveFileFormat fileFormat = HiveFileFormat.detect( + hiveHandle.getInputFormat(), hiveHandle.getSerializationLib(), + readHiveJsonInOneColumn(session), hiveHandle.isFirstColumnString()); + long targetSplitSize = getTargetSplitSize(session); + boolean splittable = fileFormat.isSplittable(); + Configuration hadoopConf = buildHadoopConf(); + + List ranges = new ArrayList<>(); + for (PartitionScanInfo partition : partitions) { + HiveFileFormat partFormat = partition.fileFormat != null + ? partition.fileFormat : fileFormat; + listAndSplitFiles(dbName, tableName, partition, partFormat, + splittable, targetSplitSize, hadoopConf, ranges); + } + return ranges; + } + /** * Plans a scan of a transactional (ACID) Hive table. * diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java new file mode 100644 index 00000000000000..b973a2c26be7d8 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java @@ -0,0 +1,273 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.scan.ConnectorScanRange; +import org.apache.doris.connector.hms.HmsClient; +import org.apache.doris.connector.hms.HmsDatabaseInfo; +import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.hms.HmsTableInfo; + +import org.apache.hadoop.conf.Configuration; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * Tests the two connector-local batch-mode overrides {@link HiveScanPlanProvider} adds so a large partitioned + * hive scan streams its splits per partition batch instead of materializing every file synchronously into the FE + * heap (post-HMS-cutover, hive scans through the generic {@code PluginDrivenScanNode}, which enters batch mode + * only when the connector opts in). + * + *

      WHY (Rule 9): the two overrides encode two correctness-critical decisions.

      + *
        + *
      • {@code supportsBatchScan}: batch iff PARTITIONED and NOT transactional. ACID tables are excluded so the + * per-batch resolution does not open (and leak) a metastore read transaction per batch — they keep the + * synchronous path.
        • + *
      • {@code planScanForPartitionBatch}: hive's {@code planScan} resolves partitions from the handle's pruned + * set and IGNORES the passed partition set, so hive MUST override the batch hook to scope resolution to the + * batch. If it inherited the SPI default (which re-runs the whole-pruned-set {@code planScan} per batch), + * every partition's files would be emitted once per batch — DUPLICATE ROWS. The {@code ranges.size()==1} + * (and single-location listing) assertion below fails precisely under that bug.
        • + *
      + * + *

      No Mockito: reuses the proven {@code FakeHmsClient} echo (getPartitions(names) -> one partition per name, + * location = name) and a counting {@link HiveFileListingCache.DirectoryLister} so the listed locations can be + * asserted, exactly as {@code HiveConnectorMetadataPartitionPruningTest} / {@code HiveFileListingCacheTest}.

      + */ +public class HiveScanBatchModeTest { + + private static final List PART_KEYS = Arrays.asList("year", "month"); + private static final String PARQUET_INPUT_FORMAT = + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat"; + private static final String PARQUET_SERDE = + "org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe"; + + // ==================== supportsBatchScan: partitioned AND non-transactional ==================== + + @Test + public void supportsBatchScanIsTrueForPartitionedNonTransactionalTable() { + HiveScanPlanProvider provider = provider(null, new CountingLister()); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(PART_KEYS) + .build(); + // WHY: a partitioned, non-transactional table is exactly the case that must stream its splits per batch. + Assertions.assertTrue(provider.supportsBatchScan(new FakeSession(), handle)); + } + + @Test + public void supportsBatchScanIsFalseForUnpartitionedTable() { + HiveScanPlanProvider provider = provider(null, new CountingLister()); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.emptyList()) + .build(); + // WHY: an unpartitioned table has a single partition (the table location); there is nothing to batch, so + // it stays on the synchronous planScan path. + Assertions.assertFalse(provider.supportsBatchScan(new FakeSession(), handle)); + } + + @Test + public void supportsBatchScanIsFalseForTransactionalPartitionedTable() { + HiveScanPlanProvider provider = provider(null, new CountingLister()); + Map txnParams = new HashMap<>(); + txnParams.put("transactional", "true"); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(PART_KEYS) + .tableParameters(txnParams) + .build(); + // WHY (Rule 9): ACID tables are excluded DELIBERATELY — running planScanForPartitionBatch per batch on + // background threads would open (and leak) a metastore read transaction per batch. They keep the + // synchronous path even though they are partitioned. + Assertions.assertFalse(provider.supportsBatchScan(new FakeSession(), handle)); + } + + // ==================== planScanForPartitionBatch: scoped to the batch, no duplication ==================== + + @Test + public void planScanForPartitionBatchResolvesOnlyTheBatch() { + CountingLister lister = new CountingLister(); + // getPartitions echoes each requested name back as a partition whose location IS the name, so the counting + // lister's keys are the partition names actually resolved for this batch. + HiveScanPlanProvider provider = provider(new FakeHmsClient(), lister); + + // The handle carries the FULL pruned set (all three partitions). If the batch hook were NOT scoped to the + // batch (SPI default -> whole-pruned-set planScan), all three would be listed -> 3 ranges. + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat(PARQUET_INPUT_FORMAT) + .serializationLib(PARQUET_SERDE) + .partitionKeyNames(PART_KEYS) + .prunedPartitions(Arrays.asList( + part("year=2023/month=12"), + part("year=2024/month=01"), + part("year=2024/month=02"))) + .build(); + + List ranges = provider.planScanForPartitionBatch( + new FakeSession(), handle, Collections.emptyList(), + Optional.empty(), -1L, Collections.singletonList("year=2024/month=01")); + + // WHY (Rule 9): exactly one range for the single-partition batch. This fails (== 3) precisely if the batch + // is not partition-scoped — the duplicate-splits bug the override exists to prevent. + Assertions.assertEquals(1, ranges.size()); + // ...and the file lister was asked ONLY for the batch's partition, never the other two. + Assertions.assertEquals(1, lister.totalCalls); + Assertions.assertEquals(1, (int) lister.callsPerLocation.get("year=2024/month=01")); + Assertions.assertNull(lister.callsPerLocation.get("year=2023/month=12")); + Assertions.assertNull(lister.callsPerLocation.get("year=2024/month=02")); + } + + // ===== helpers ===== + + private static HiveScanPlanProvider provider(HmsClient hmsClient, CountingLister lister) { + return new HiveScanPlanProvider(hmsClient, Collections.emptyMap(), + new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); + } + + private static HmsPartitionInfo part(String name) { + return new HmsPartitionInfo(Collections.emptyList(), name, null, null, null, Collections.emptyMap()); + } + + /** + * A {@link HiveFileListingCache.DirectoryLister} double: counts calls (total + per location) and returns a + * fresh single-file listing per call, so each partition location contributes exactly one range. + */ + private static final class CountingLister implements HiveFileListingCache.DirectoryLister { + final Map callsPerLocation = new HashMap<>(); + int totalCalls; + + @Override + public List list(String location, Configuration conf) { + totalCalls++; + callsPerLocation.merge(location, 1, Integer::sum); + return new ArrayList<>(Collections.singletonList(new HiveFileStatus(location + "/000000_0", 10L, 1L))); + } + } + + /** + * Minimal {@link HmsClient} double whose {@code getPartitions} echoes each requested name back as an + * {@link HmsPartitionInfo} whose location IS the name, so the batch-scoped resolution can be asserted through + * the listed locations. The rest fail loud. + */ + private static final class FakeHmsClient implements HmsClient { + @Override + public List getPartitions(String dbName, String tableName, List partNames) { + List result = new ArrayList<>(); + for (String name : partNames) { + result.add(new HmsPartitionInfo(Collections.emptyList(), name, + null, null, null, Collections.emptyMap())); + } + return result; + } + + @Override + public List listPartitionNames(String dbName, String tableName, int maxParts) { + throw new UnsupportedOperationException(); + } + + @Override + public List listDatabases() { + throw new UnsupportedOperationException(); + } + + @Override + public HmsDatabaseInfo getDatabase(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public List listTables(String dbName) { + throw new UnsupportedOperationException(); + } + + @Override + public boolean tableExists(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsTableInfo getTable(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public Map getDefaultColumnValues(String dbName, String tableName) { + throw new UnsupportedOperationException(); + } + + @Override + public HmsPartitionInfo getPartition(String dbName, String tableName, List values) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + } + + /** Minimal {@link ConnectorSession} (no split-size override, empty session properties). */ + private static final class FakeSession implements ConnectorSession { + @Override + public String getQueryId() { + return "q"; + } + + @Override + public String getUser() { + return "u"; + } + + @Override + public String getTimeZone() { + return "UTC"; + } + + @Override + public String getLocale() { + return "en_US"; + } + + @Override + public long getCatalogId() { + return 0L; + } + + @Override + public String getCatalogName() { + return "hive_catalog"; + } + + @Override + public T getProperty(String name, Class type) { + return null; + } + + @Override + public Map getCatalogProperties() { + return Collections.emptyMap(); + } + } +} From 9c9d159afcc58e6ca9593a26f4765f068eba0aa3 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:37:55 +0800 Subject: [PATCH 225/334] [doc](catalog) #65185 batch-1 COMPLETE: M2 design + track-off (all 5 done) FIX-M2-design.md (hive batch/async split path; registers BATCH-ACID-SYNC permanent + BATCH-UNPRUNED-SYNC resolved-by-M3) + tick M2 in the tracking table, mark batch 1 (M5/M7/M6/M4/M2) complete in the rolling log, and update HANDOFF (next = batch 2 M3->M1 fe-core generic node; M3 also resolves M2's unpruned-sync residual). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 6 +- plan-doc/tasks/designs/FIX-M2-design.md | 71 ++++++++++++++++++++++ 3 files changed, 76 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M2-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d4e4d099104d7a..26bcb1071c016b 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -30,7 +30,7 @@ - **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 **⭐ 下一步（下个 session）：** -1. **复核修复续批**（跟踪表）：**批次 1 进行中**——全 5 条已一轮 recon+对抗红队（`wf_40498e52-19f`，机制全 HEAD 确认、无 UNSOUND，设计存 `plan-doc/tasks/designs/FIX-M*-design.md` + scratch `batch1/`）。**iceberg 子组 M5/M7/M6 全 DONE**：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字的「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion)。**续做 M4（mc 分区缓存，最干净）→ M2（hive 批量：非照抄 MaxCompute，须额外 override `planScanForPartitionBatch`，另登记 ACID→sync + 未过滤扫描→sync 两条偏差）**→ 批次 2（M3/M1 fe-core 通用节点）→ ...。⏸ 决策类（L2/L10/L12/L20）先问用户。 +1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**下一步 = 批次 2（M3→M1，fe-core 通用节点，blast radius 较大，单测充分后再动）**；M3(`shouldUseBatchMode` 的 `!isPruned`→`==NOT_PRUNED`)顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。之后批次 3（M8/L1）… ⏸ 决策类（L2/L10/L12/L20）先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index a099235e9db7ac..12b7d378744439 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -30,7 +30,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ✅ | ✅ | ✅ | ✅ | | 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | | 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | -| 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ⬜ | ⬜ | ⬜ | ⬜ | +| 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ✅ | ✅ | ✅ | ✅ | | 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | | 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ✅ | ✅ | ✅ | ✅ | | 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | @@ -118,7 +118,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:强化 `test_hive_tablesample_p0.groovy` 断言采样后基数(非仅 EXPLAIN 子串)。 ### M2 — 翻闸 hive 丢批量/异步 split · reverify §3 M2 -- [ ] **M2** +- [x] **M2** · DONE `702153885ab`（设计 `designs/FIX-M2-design.md`；实现经 impl-subagent + 独立 diff 复核 + 全模块 test）。**两** override（非照抄 MaxCompute）：`supportsBatchScan`(分区∧非事务)+`planScanForPartitionBatch`(scope 到 batch 防重复 split)；ACID 刻意排除(同 `isTransactional()` accessor)。**登记两偏差**：BATCH-ACID-SYNC(永久)、BATCH-UNPRUNED-SYNC(由 M3 解)。`HiveScanBatchModeTest` 4/4 + hive 模块 284/284 绿。e2e live-gated。 - **现码**:`HiveScanPlanProvider.java:62`(自称 No batch;不 override `supportsBatchScan`/`streamingSplitEstimate`)→ `PluginDrivenScanNode.computeBatchMode:1085-1113` 恒非 batch。legacy `HiveScanNode.isBatchMode:283-294` 按 `prunedPartitions.size()>=1024` 异步。 - **Fix**:override `HiveScanPlanProvider.supportsBatchScan`(分区表 true)+ `planScanForPartitionBatch`,仿 `MaxComputeScanPlanProvider`;无需改 fe-core。若决定接受回归,须登记 + 大分区 e2e 真值门。 - **Files**:`fe-connector-hive/.../HiveScanPlanProvider.java`。 @@ -245,3 +245,5 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **M7** DONE `f6de950e5bd`(code) — iceberg REST vended-cred `client.region` 别名 4→10 拓宽,逐字节对齐 fe-core `S3Properties` isRegionField 集;注释按红队更正为「S3 子集」。`IcebergCatalogFactoryTest` 62/62 绿。e2e live-gated(region 经 `AWS_REGION` 写提交不报 Unable to load region)。 - **M6** DONE `03bd4f58187`(code) — iceberg s3tables 无绑定存储不再硬失败:region 唯一必需(存储或 props),凭证回退 `DefaultCredentialsProvider` 默认链;companion 无存储臂发 data-plane `client.region`。`IcebergConnectorTest` 19/19 + `IcebergCatalogFactoryTest` 63/63 绿。**iceberg 子组(M5/M7/M6)全 DONE**。e2e live-gated(EC2 instance-profile s3tables 目录 CREATE+list 不抛)。 - **M4** DONE `c553c3c7696`(code) — maxcompute 连接器内 `MaxComputePartitionCache`(HiveFileListingCache 结构副本)恢复被删的分区值缓存,消除每规划一次全量 ODPS listPartitions;4 REFRESH 钩子刷;pom+Caffeine 2.9.3。9/9+模块 113/113 绿,插件 zip 单 caffeine 版本。e2e live-gated。 +- **M2** DONE `702153885ab`(code) — hive 补 batch 通路:`supportsBatchScan`(分区∧非事务)+`planScanForPartitionBatch`(scope 到 batch,防重复 split——hive `planScan` 非 partition-scoped 故须额外 override)。**登记 BATCH-ACID-SYNC(永久)+BATCH-UNPRUNED-SYNC(M3 解)**。4/4+hive 模块 284/284 绿。e2e live-gated。 +- **⭐ 批次 1(M5/M7/M6/M4/M2)全部 DONE。** 5 连接器局部修全绿(iceberg 3 + mc 1 + hive 1),各配 RED-able 单测 + 独立 code/doc commit。**e2e 全 live-gated**(equality-delete 统计/vended-region/s3tables 默认链/mc 分区缓存往返/hive 大分区异步),须真集群回归(memory `hms-iceberg-delegation-needs-e2e`)。**下一步=批次 2(M3→M1,fe-core 通用节点,blast radius 较大)**;M3 顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。 diff --git a/plan-doc/tasks/designs/FIX-M2-design.md b/plan-doc/tasks/designs/FIX-M2-design.md new file mode 100644 index 00000000000000..045ff8078b4eb0 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M2-design.md @@ -0,0 +1,71 @@ +# FIX-M2 — 翻闸后 hive 丢批量/异步 split → 连接器补 batch 通路 + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M2。 +> recon+对抗红队：`wf_40498e52-19f`（SOUND_WITH_CHANGES：**非** trivial 照抄 MaxCompute——须**额外** override `planScanForPartitionBatch` 否则 batch 重复 split；红队证实机制 + 要求登记两条偏差）。 + +## Problem + +翻闸后（`"hms"` ∈ `SPI_READY_TYPES`）每张 hive 表走通用 `PluginDrivenScanNode`。该节点仅在连接器 opt-in 时进 +batch/async split。`HiveScanPlanProvider` 两 flavor 都不 opt-in → 大分区 hive 扫描把所有选中分区的所有文件 split +一次性同步物化进 FE 堆——hive 是最大分区数外部源，回归最重。结果正确（fail-safe），FE 堆 + 规划延迟回归。 + +## Root Cause + +`PluginDrivenScanNode.computeBatchMode` 仅在连接器返 `streamingSplitEstimate>=0`（iceberg-only）或 +`supportsBatchScan==true`（MaxCompute-only）时开 batch。`HiveScanPlanProvider` 两者都不 override → 继承 +`supportsBatchScan=false`/`streamingSplitEstimate=-1` → `shouldUseBatchMode` 恒 false → 同步 `getSplits`。legacy +`HiveScanNode.isBatchMode` 按 `prunedPartitions.size() >= num_partitions_in_batch_mode`(默认 1024) 异步、无 opt-in 门。 + +**关键 nuance（非照抄 MaxCompute）**：MaxCompute 只 override `supportsBatchScan`、靠 SPI **默认** +`planScanForPartitionBatch`——正确仅因其 `planScan` **partition-set-scoped**（消费 `requiredPartitions`）。hive 的 +`planScan` **非** partition-set-scoped：从 `handle.getPrunedPartitions()` 解析、**忽略** 传入分区集。 +`PluginDrivenScanNode` 把 Nereids 剪枝分区**名**切 batch、每 batch 调 +`planScanForPartitionBatch(...,batch)`。若 hive 继承默认 → 每 batch 重跑全剪枝集 `planScan` → 每分区文件被 emit +`num_batches` 次 → **重复行**（正确性 bug）。故 hive **必须也** override `planScanForPartitionBatch` 把解析 scope 到 batch。 + +batch 串 = Nereids selected-partition map 的键 = `ConnectorPartitionInfo.getPartitionName()` = HMS 渲染 +`"key=value/..."` 名 = `hmsClient.getPartitions(db,table,names)` 接受、连接器 `applyFilter` 已用的形式 → batch-scoped +解析 = 干净 `getPartitions(batch)` 往返。 + +## Design（`HiveScanPlanProvider` 两连接器局部 override；无 fe-core 改） + +1. `supportsBatchScan(session, handle)` → true iff **分区表**（`getPartitionKeyNames()` 非空）**且非事务** + （`!isTransactional()`）。**ACID 刻意排除**：其扫描在 `planScan`/`planAcidScan` 开一个 metastore 读事务；batch 路 + 在后台池线程 per-batch 跑 `planScanForPartitionBatch`，路由 ACID 会 per-batch 开→泄漏读事务。ACID 分区表保持同步 + 路径（正确、只是不流式）。排除用的 `isTransactional()` = `planScan` 分支 ACID 的同一 accessor。 +2. `planScanForPartitionBatch(session, handle, columns, filter, limit, partitionBatch)` → 仅解析 batch 分区： + `hmsClient.getPartitions(db, table, partitionBatch)` → `convertPartitions` → 格式/split-size/hadoopConf 检测 + （与 `planScan` 逐字节同）→ `listAndSplitFiles` per 分区 → 返回该 batch ranges。复用 `planScan` 同款 helper， + ~30 行、无新 import。ACID 路不可达（`supportsBatchScan` 已排除）。 +- 订正 class-javadoc `:62` stale bullet。 + +**为何不改 `planScan` 为 requiredPartitions-scoped**：会重做非-batch 路（读 getPrunedPartitions）、扩大 blast +radius；targeted `planScanForPartitionBatch` override 是外科选择。 + +## 登记偏差（fail-safe，结果不变；Rule 12 显式登记非静默） + +- **BATCH-ACID-SYNC（永久·by-design）**：ACID 分区表保持同步（legacy 曾对 ACID 也 batch）。per-batch-ACID-事务 + 方案会破坏单读事务不变式 / 泄漏共享读锁——已否。仅罕见 ACID+≥1024 分区丢异步。 +- **BATCH-UNPRUNED-SYNC（**由 M3 解**，批次 2）**：真正无过滤扫描（无 WHERE → `PruneFileScanPartition` 不触发 → + `initSelectedPartitions` 返 `isPruned=false`）→ fe-core `shouldUseBatchMode` 现用 `!isPruned` 恒 false → 保持同步； + legacy 无条件 batch。**非连接器可修**——是 fe-core `isPruned` 门；M3 的 `==NOT_PRUNED` 修复正是让无过滤扫描也 batch。 + +## Risk + +- 线程：`planScanForPartitionBatch` 在 `scheduleExecutor` 线程用线程安全共享 `HiveFileListingCache`、每调新 `Configuration`、 + 每 batch 一次 `getPartitions` RPC（legacy 付等价 per-分区列举成本）。TCCL 由 `PluginDrivenScanNode.onPluginClassLoader` pin。 +- 名保真：batch 名直传 `getPartitions`（无 unescape/比较）→ hudi-H1 转义陷阱不适用（往返恒等）。 +- Nereids vs 连接器剪枝：batch 仅在 `isPruned` 且 size≥阈值时触发；两者同谓词剪枝 → batch 集 == 同步集（不增删行）。 +- Scope：无 fe-core 改、无新 import、`planScan`/`planAcidScan` 不动。 + +## Test + +- Unit `HiveScanBatchModeTest`（无 Mockito；复用 `HiveFileListingCache.DirectoryLister` counting fake + `FakeHmsClient` + echo + `FakeSession`）4 测：`supportsBatchScan` 分区非事务→true / 非分区→false / 事务分区→false； + `planScanForPartitionBatch` 携全 3 剪枝分区、batch=`[year=2024/month=01]` → `ranges.size()==1` **且** lister 仅列该分区。 + - RED：`supportsBatchScan` 继承默认 false；batch hook 继承默认→委派 `planScan`→解析全 3→3 ranges/3 location→`size==1` 挂。 + `size==1` 编码 WHY（Rule 9）：非 partition-scoped（重复 split bug）即挂。 + - 结果：4/4 + hive 模块 284/284 绿、0 checkstyle、import 门 0。 +- E2E live-gated（docker HMS/HDFS + 真 FE+BE）：分区数 > `num_partitions_in_batch_mode` 且带谓词仍留≥阈值幸存者的表， + 断言 (a) 行数正确无重复、(b) 走异步（EXPLAIN 显 `isBatchMode()` 下的 `(approximate) inputSplitNum` 前缀）；可 A/B 把 + 阈值设高于分区数（同步）比对结果一致。 From fca288424fc38ba2a55934d6906b61a98805226f Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:58:06 +0800 Subject: [PATCH 226/334] [fix](catalog) maxcompute partition cache: correct TTL default to legacy parity (600s) Final adversarial review caught a parity miss in the M4 cache (c553c3c7696): the TTL default was copied from HiveFileListingCache's knob (external_cache_expire_time_seconds_after_access = 86400s / 24h), but the DELETED legacy MaxComputeExternalMetaCache.partitionValuesEntry used a DIFFERENT pair -- CacheSpec.of(Config.external_cache_refresh_time_minutes * 60, // default 10min = 600s Config.max_hive_partition_table_cache_num). // default 10000 Verified against the deletion commit (1da88365e85^) + Config.java defaults. The capacity (10000) coincidentally matched; the TTL was 144x too long -- a partition added directly in ODPS would have stayed invisible for up to 24h instead of the legacy ~10 min (both contextual-only expire-after-access, so this is a pure staleness regression vs the behavior this fix set out to restore). DEFAULT_TTL_SECOND 86400 -> 600; comment corrected to cite the real legacy knobs. No behavior change to capacity. MaxComputePartitionCacheTest 9/9 green (no test asserts the TTL value; the meta.cache.max_compute.partition.ttl-second override still works). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../maxcompute/MaxComputePartitionCache.java | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java index 71670e610fd6b4..759bbeffe4ee83 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePartitionCache.java @@ -64,10 +64,14 @@ public class MaxComputePartitionCache { /** {@code meta.cache.max_compute.partition.*} — cached partition listings. */ static final String ENTRY_PARTITION = "partition"; - // Legacy fe-core Config values, mirrored locally (the connector never touches fe-core Config): - // TTL = Config.external_cache_expire_time_seconds_after_access (86400s = 24h) - // capacity = Config.max_external_file_cache_num (10000) - static final long DEFAULT_TTL_SECOND = 86400L; + // Legacy MaxCompute partition-cache Config values, mirrored locally (the connector never touches fe-core + // Config). These are the knobs the DELETED MaxComputeExternalMetaCache.partitionValuesEntry actually used — + // TTL = Config.external_cache_refresh_time_minutes * 60 (default 10 min * 60 = 600s) + // capacity = Config.max_hive_partition_table_cache_num (default 10000) + // NOT the hive file-listing cache's knobs (external_cache_expire_time_seconds_after_access = 24h / + // max_external_file_cache_num): a MaxCompute table's partition set must re-list ~10 min after last access, + // matching legacy, so a partition added directly in ODPS becomes visible without an explicit REFRESH. + static final long DEFAULT_TTL_SECOND = 600L; static final long DEFAULT_PARTITION_CAPACITY = 10000L; /** From e3ae0ff7b8662e86061c6e6a35c7487dab0ffa97 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 08:59:07 +0800 Subject: [PATCH 227/334] [doc](catalog) #65185 batch-1: record final adversarial review (CLEAN) + M4 TTL fix Final review (wf_542c60b9-001, 5 per-fix skeptics + 1 cross-cut): M5/M6/M7/M2 and the whole-batch cross-cut CLEAN; M4 hit one medium defect (TTL default copied from the hive file cache's knob, 86400s, vs legacy MaxCompute partition cache's external_cache_refresh_time_minutes*60 = 600s) -> fixed in fca288424fc. Corrected FIX-M4-design.md, tracking table (rolling log + M4 detail), and HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 5 +++-- plan-doc/tasks/designs/FIX-M4-design.md | 4 +++- 3 files changed, 7 insertions(+), 4 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 26bcb1071c016b..36dc11b4adab46 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -30,7 +30,7 @@ - **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 **⭐ 下一步（下个 session）：** -1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**下一步 = 批次 2（M3→M1，fe-core 通用节点，blast radius 较大，单测充分后再动）**；M3(`shouldUseBatchMode` 的 `!isPruned`→`==NOT_PRUNED`)顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。之后批次 3（M8/L1）… ⏸ 决策类（L2/L10/L12/L20）先问用户。 +1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`+`fca288424fc`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3；**最终复核纠正 TTL 86400→600s 对齐旧版**) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**最终对抗复核（`wf_542c60b9-001`，5 per-fix + 1 cross-cut）= M5/M6/M7/M2/cross-cut CLEAN、M4 命中 1 medium(TTL)已修 → 全 CLEAN。****下一步 = 批次 2（M3→M1，fe-core 通用节点，blast radius 较大，单测充分后再动）**；M3(`shouldUseBatchMode` 的 `!isPruned`→`==NOT_PRUNED`)顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。之后批次 3（M8/L1）… ⏸ 决策类（L2/L10/L12/L20）先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 12b7d378744439..7bca81c989d127 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -132,7 +132,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:纯 helper 单测覆盖 `(isPruned=false, 非 NOT_PRUNED, size≥阈值)` 应 →batch(RED:返回 false)。 ### M4 — MC 分区值缓存删除 · reverify §3 M4 -- [x] **M4** · DONE `c553c3c7696`（设计 `designs/FIX-M4-design.md`；实现经 impl-subagent + 独立 build/test/zip 核验）。新 `MaxComputePartitionCache`=`HiveFileListingCache` 结构副本（共享 `fe-connector-cache`，contextual-only+manual-miss flag 字节一致），keyed(db,table)，持于 `MaxComputeDorisConnector`、注入 metadata、三方法走它、4 个 REFRESH 钩子刷；pom 加 `fe-connector-cache`+Caffeine 2.9.3。`MaxComputePartitionCacheTest` 9/9 + 模块 113/113、0 checkstyle、import 门 0、**插件 zip 恰含单个 caffeine-2.9.3.jar**。e2e live-gated。 +- [x] **M4** · DONE `c553c3c7696` + TTL parity 修 `fca288424fc`（设计 `designs/FIX-M4-design.md`；实现经 impl-subagent + 独立 build/test/zip 核验 + 最终对抗复核）。新 `MaxComputePartitionCache`=`HiveFileListingCache` 结构副本（共享 `fe-connector-cache`，contextual-only+manual-miss flag 字节一致），keyed(db,table)，持于 `MaxComputeDorisConnector`、注入 metadata、三方法走它、4 个 REFRESH 钩子刷；pom 加 `fe-connector-cache`+Caffeine 2.9.3。**复核纠正 TTL 默认 86400→600s（对齐旧版 `external_cache_refresh_time_minutes*60`）**。`MaxComputePartitionCacheTest` 9/9 + 模块 113/113、0 checkstyle、import 门 0、**插件 zip 恰含单个 caffeine-2.9.3.jar**。e2e live-gated。 - **现码**:`PluginDrivenExternalTable.getNameToPartitionItems:780-781`(每次 `listPartitions`)→ `MaxComputeConnectorMetadata.listPartitions:256-273` → `McStructureHelper.getPartitions:112-118`(裸 ODPS SDK)。fe-core+连接器**两层无缓存**。 - **Fix**:maxcompute 连接器内加 TTL/容量 Caffeine(仿 `CachingHmsClient`/`HiveFileListingCache`),keyed `(project,db,table)`,`REFRESH` 失效。**不**在 fe-core 加二级缓存。若延后须登记 CACHE-P1。 - **Files**:`fe-connector-maxcompute/.../MaxComputeConnectorMetadata.java`(+ 新缓存类)。 @@ -246,4 +246,5 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **M6** DONE `03bd4f58187`(code) — iceberg s3tables 无绑定存储不再硬失败:region 唯一必需(存储或 props),凭证回退 `DefaultCredentialsProvider` 默认链;companion 无存储臂发 data-plane `client.region`。`IcebergConnectorTest` 19/19 + `IcebergCatalogFactoryTest` 63/63 绿。**iceberg 子组(M5/M7/M6)全 DONE**。e2e live-gated(EC2 instance-profile s3tables 目录 CREATE+list 不抛)。 - **M4** DONE `c553c3c7696`(code) — maxcompute 连接器内 `MaxComputePartitionCache`(HiveFileListingCache 结构副本)恢复被删的分区值缓存,消除每规划一次全量 ODPS listPartitions;4 REFRESH 钩子刷;pom+Caffeine 2.9.3。9/9+模块 113/113 绿,插件 zip 单 caffeine 版本。e2e live-gated。 - **M2** DONE `702153885ab`(code) — hive 补 batch 通路:`supportsBatchScan`(分区∧非事务)+`planScanForPartitionBatch`(scope 到 batch,防重复 split——hive `planScan` 非 partition-scoped 故须额外 override)。**登记 BATCH-ACID-SYNC(永久)+BATCH-UNPRUNED-SYNC(M3 解)**。4/4+hive 模块 284/284 绿。e2e live-gated。 -- **⭐ 批次 1(M5/M7/M6/M4/M2)全部 DONE。** 5 连接器局部修全绿(iceberg 3 + mc 1 + hive 1),各配 RED-able 单测 + 独立 code/doc commit。**e2e 全 live-gated**(equality-delete 统计/vended-region/s3tables 默认链/mc 分区缓存往返/hive 大分区异步),须真集群回归(memory `hms-iceberg-delegation-needs-e2e`)。**下一步=批次 2(M3→M1,fe-core 通用节点,blast radius 较大)**;M3 顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。 +- **批次 1 最终对抗复核(5 per-fix skeptic + 1 cross-cut,`wf_542c60b9-001`)= M5/M6/M7/M2/cross-cut CLEAN；M4 命中 1 medium 缺陷已修**:M4 TTL 默认误抄 hive 文件缓存 knob(86400s)而非旧版 MaxCompute 分区缓存 knob(`external_cache_refresh_time_minutes*60`=600s),144x 过陈;经核旧版删除 commit `1da88365e85^`+Config 默认证实,已改 600s(**`fca288424fc`**,9/9 仍绿)。capacity 10000 巧合正确(旧版 `max_hive_partition_table_cache_num`)。 +- **⭐ 批次 1(M5/M7/M6/M4/M2)全部 DONE + 最终复核 CLEAN。** 5 连接器局部修全绿(iceberg 3 + mc 1 + hive 1),各配 RED-able 单测 + 独立 code/doc commit。**e2e 全 live-gated**(equality-delete 统计/vended-region/s3tables 默认链/mc 分区缓存往返/hive 大分区异步),须真集群回归(memory `hms-iceberg-delegation-needs-e2e`)。**下一步=批次 2(M3→M1,fe-core 通用节点,blast radius 较大)**;M3 顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。 diff --git a/plan-doc/tasks/designs/FIX-M4-design.md b/plan-doc/tasks/designs/FIX-M4-design.md index 84b7b086bfb562..0031ae9334f157 100644 --- a/plan-doc/tasks/designs/FIX-M4-design.md +++ b/plan-doc/tasks/designs/FIX-M4-design.md @@ -28,7 +28,9 @@ metadata——正是 `HiveConnector` 持 `HiveFileListingCache` 交给 metadata 与 `HiveFileListingCache` 字节一致）→ 慢 loader 在调用线程同步跑（striped-lock dedup），TCCL/classloading 与 今日裸调用字节一致。loader 经 `PartitionLister` 注入（无 Mockito 可测），生产 loader `(db,t)->structureHelper.getPartitions(odps,db,t)`。config `meta.cache.max_compute.partition.{enable,ttl-second, - capacity}`，默认对齐旧版（ttl 86400s、capacity 10000）。 + capacity}`，默认对齐**旧版 MaxCompute 分区缓存**（ttl=`external_cache_refresh_time_minutes*60`=**600s**、 + capacity=`max_hive_partition_table_cache_num`=10000）。**⚠ 最终复核纠正**：初版误抄 hive 文件缓存的 knob + （`external_cache_expire_time_seconds_after_access`=86400s）→ 144x 过陈；已改 600s（commit `fca288424fc`）。 2. `MaxComputeDorisConnector` 持 `final` 缓存（ctor 建，loader lambda 捕获 this、查询时惰读 structureHelper/odps， 均 post-init），`getMetadata` 注入；override 4 个 `Connector` REFRESH 钩子（`invalidateTable/Db/All/Partition`） 路由到缓存（`invalidatePartition` 降级为整表刷，正确安全——miss 重列）。镜像 `HiveConnector`。 From 92004ef1d0d3eafc43644c43d3ec39248e8d333c Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 10:10:22 +0800 Subject: [PATCH 228/334] [fix](catalog) hive: pin HMS metastore-client TCCL to plugin loader (fix SecurityUtil split-brain) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TeamCity #991951 (PR #65474, hive connector SPI migration) failed 49 external cases — hive / iceberg-on-HMS / hudi / mtmv / kerberos / tvf — all with the FE-side "Could not initialize class org.apache.hadoop.security.SecurityUtil". The cluster was healthy (366 passed, no BE crash/OOM); the "No backend available" lines are startup-only. Root cause: ThriftHmsClient.doAs pinned the TCCL to ClassLoader.getSystemClassLoader() before creating the metastore client. SecurityUtil. -> new Configuration() captures that TCCL to reflectively load DNSDomainNameResolver: the system loader holds fe-core's hadoop copy while SecurityUtil/DomainNameResolver resolve from the plugin's child-first copy, so DomainNameResolver.isAssignableFrom(DNSDomainNameResolver) is false ("class DNSDomainNameResolver not DomainNameResolver") -> ExceptionInInitializerError, which permanently poisons SecurityUtil JVM-wide. doAs is the single choke point for both metastore creation (createFreshClient) and every RPC (execute), and both the Kerberos (ugi.doAs) and non-Kerberos (context.executeAuthenticated) authAction paths run inside it. Fix: pin getClass().getClassLoader() — the plugin child-first loader that loaded ThriftHmsClient, a strict superset of the system loader — matching iceberg/paimon TcclPinningConnectorContext and HiveConnectorMetadata's stats pins. Test: ThriftHmsClientDoAsClassLoaderTest drives DoAsTcclProbe through an isolated child-first loader (mirrors OdpsClassloaderIsolationTest) so getClass().getClassLoader() differs from the system loader, making the exact bug observable: RED before the fix (PIN_WRONG_SYSTEM_LOADER), GREEN after. fe-connector-hms 40/40 green, 0 checkstyle. Real-cluster e2e rerun of the 49 cases still owed. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../doris/connector/hms/ThriftHmsClient.java | 13 +- .../doris/connector/hms/DoAsTcclProbe.java | 99 ++++++++++++++ .../ThriftHmsClientDoAsClassLoaderTest.java | 123 ++++++++++++++++++ 3 files changed, 233 insertions(+), 2 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/DoAsTcclProbe.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientDoAsClassLoaderTest.java diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index e0e03f9d1699c5..99e17f118e7408 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -631,8 +631,17 @@ private T execute(HmsAction action) { private T doAs(Callable callable) throws Exception { ClassLoader original = Thread.currentThread().getContextClassLoader(); try { - Thread.currentThread().setContextClassLoader( - ClassLoader.getSystemClassLoader()); + // Pin the TCCL to the plugin (child-first) classloader that loaded THIS client — NOT the system + // classloader. Metastore client creation runs Hadoop's SecurityUtil., whose internal + // `new Configuration()` captures the current TCCL and uses it to reflectively load + // DNSDomainNameResolver. The system classloader holds fe-core's own hadoop copy, while + // SecurityUtil/DomainNameResolver here resolve from the plugin's child-first copy — so a + // system-loader TCCL loads the two from different loaders ("class DNSDomainNameResolver not + // DomainNameResolver") and permanently poisons SecurityUtil JVM-wide. Pinning the plugin loader + // (a strict superset of the system loader) keeps every reflective load on the plugin side. Mirrors + // iceberg/paimon TcclPinningConnectorContext and HiveConnectorMetadata's stats pins; covers both the + // non-Kerberos (context.executeAuthenticated) and Kerberos (ugi.doAs) authAction paths. + Thread.currentThread().setContextClassLoader(getClass().getClassLoader()); return authAction.execute(callable); } finally { Thread.currentThread().setContextClassLoader(original); diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/DoAsTcclProbe.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/DoAsTcclProbe.java new file mode 100644 index 00000000000000..48dc0547b47ae3 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/DoAsTcclProbe.java @@ -0,0 +1,99 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.hadoop.hive.metastore.IMetaStoreClient; + +import java.lang.reflect.InvocationHandler; +import java.lang.reflect.Proxy; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; + +/** + * Test entrypoint, invoked reflectively THROUGH an isolated child-first classloader (see + * {@link ThriftHmsClientDoAsClassLoaderTest}). Because the child-first loader defines this class — and + * therefore {@link ThriftHmsClient} — itself, {@code ThriftHmsClient.class.getClassLoader()} inside here is + * that isolated loader, distinct from the system classloader. That is what reproduces the production two-copy + * topology (fe-core hadoop on the system loader, plugin hadoop child-first) so the difference between + * {@code doAs} pinning {@code getClass().getClassLoader()} (correct) and {@code getSystemClassLoader()} (the + * split-brain bug) becomes observable in a unit test. + */ +public final class DoAsTcclProbe { + + private DoAsTcclProbe() { + } + + /** + * Drives one metastore call and inspects the TCCL {@code doAs} pinned while creating the client. Returns + * {@code "OK"} iff {@code doAs} pinned the connector's own (this isolated) loader AND restored the caller's + * TCCL; otherwise a diagnostic string. Returning a plain String avoids any cross-loader type coupling with + * the invoking test. + */ + public static String check() throws Exception { + // A fake IMetaStoreClient (no Mockito): List-returning calls yield an empty list, else null. + InvocationHandler handler = (proxy, method, args) -> + List.class.isAssignableFrom(method.getReturnType()) ? new ArrayList<>() : null; + IMetaStoreClient fake = (IMetaStoreClient) Proxy.newProxyInstance( + DoAsTcclProbe.class.getClassLoader(), new Class[] {IMetaStoreClient.class}, handler); + + final ClassLoader[] observedDuringCreate = new ClassLoader[1]; + ThriftHmsClient.MetaStoreClientProvider provider = hiveConf -> { + if (observedDuringCreate[0] == null) { + observedDuringCreate[0] = Thread.currentThread().getContextClassLoader(); + } + return fake; + }; + + // poolSize 0 -> no pool: borrowClient() -> createFreshClient() -> doAs() -> provider.create(). + HmsClientConfig config = new HmsClientConfig(new HashMap<>(), 0); + ThriftHmsClient client = new ThriftHmsClient(config, null, provider, HmsTypeMapping.Options.DEFAULT); + + ClassLoader connectorLoader = ThriftHmsClient.class.getClassLoader(); + ClassLoader system = ClassLoader.getSystemClassLoader(); + // Drive under a DISTINCT caller TCCL so both "pinned to system" and "never pinned" are observable. + ClassLoader caller = new URLClassLoader(new URL[0], connectorLoader); + ClassLoader original = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(caller); + try { + client.listDatabases(); + } finally { + ClassLoader afterCall = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(original); + client.close(); + if (observedDuringCreate[0] == null) { + return "NO_CLIENT_CREATED"; + } + if (observedDuringCreate[0] == system && system != connectorLoader) { + return "PIN_WRONG_SYSTEM_LOADER"; + } + if (observedDuringCreate[0] == caller) { + return "PIN_MISSING_LEFT_CALLER_LOADER"; + } + if (observedDuringCreate[0] != connectorLoader) { + return "PIN_WRONG_OTHER_LOADER"; + } + if (afterCall != caller) { + return "TCCL_NOT_RESTORED"; + } + } + return "OK"; + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientDoAsClassLoaderTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientDoAsClassLoaderTest.java new file mode 100644 index 00000000000000..6ea2e24d4dbdd7 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientDoAsClassLoaderTest.java @@ -0,0 +1,123 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.io.File; +import java.lang.reflect.Method; +import java.net.URL; +import java.net.URLClassLoader; +import java.util.ArrayList; +import java.util.List; + +/** + * Tests {@link ThriftHmsClient}'s internal {@code doAs}: it MUST pin the thread-context classloader (TCCL) to + * the plugin (child-first) classloader that loaded this client while it creates/uses the metastore client, + * then restore the caller's TCCL. + * + *

      WHY: metastore client creation runs Hadoop's {@code SecurityUtil.}, whose internal + * {@code new Configuration()} captures the current TCCL to reflectively load {@code DNSDomainNameResolver}. In + * production the hive plugin bundles its own child-first hadoop copy while fe-core carries another on the + * system classloader; a system-loader TCCL loads {@code DNSDomainNameResolver} from fe-core's copy while + * {@code SecurityUtil}/{@code DomainNameResolver} resolve from the plugin's copy — a classloader split-brain + * ("class DNSDomainNameResolver not DomainNameResolver") that permanently poisons {@code SecurityUtil} + * JVM-wide. TeamCity build 991951 failed 49 hive/iceberg-on-HMS/hudi/mtmv/kerberos cases exactly this way. + * + *

      The bug is invisible under a plain surefire loader (there {@code getSystemClassLoader()} and + * {@code ThriftHmsClient.class.getClassLoader()} are the SAME object). To make it observable we reproduce the + * production two-copy topology: the {@link DoAsTcclProbe} is invoked THROUGH an isolated child-first loader, so + * inside it {@code ThriftHmsClient.class.getClassLoader()} is that isolated loader — distinct from the system + * loader. A regression to {@code getSystemClassLoader()}, to no pin, or to a dropped restore is then RED. This + * mirrors {@code OdpsClassloaderIsolationTest}'s isolation approach. + */ +public class ThriftHmsClientDoAsClassLoaderTest { + + @Test + public void doAsPinsPluginLoaderNotSystemAndRestores() throws Exception { + try (IsolatedChildFirstClassLoader loader = new IsolatedChildFirstClassLoader(classpathUrls())) { + Class probe = loader.loadClass("org.apache.doris.connector.hms.DoAsTcclProbe"); + Assertions.assertSame(loader, probe.getClassLoader(), + "sanity: the probe must be defined by the isolated child-first loader"); + Assertions.assertSame(loader, loader.loadClass(ThriftHmsClient.class.getName()).getClassLoader(), + "sanity: ThriftHmsClient must be defined by the isolated loader, so getClass().getClassLoader() " + + "differs from the system loader (that is what makes the split-brain observable)"); + + Method check = probe.getMethod("check"); + String result = (String) check.invoke(null); + Assertions.assertEquals("OK", result, + "doAs must pin the TCCL to the connector's own (plugin child-first) classloader — NOT the " + + "system classloader (the SecurityUtil split-brain root cause) — and restore the caller's " + + "TCCL afterwards; probe reported: " + result); + } + } + + private static URL[] classpathUrls() throws Exception { + String classpath = System.getProperty("java.class.path"); + String[] entries = classpath.split(File.pathSeparator); + List urls = new ArrayList<>(entries.length); + for (String entry : entries) { + if (!entry.isEmpty()) { + urls.add(new File(entry).toURI().toURL()); + } + } + return urls.toArray(new URL[0]); + } + + /** + * Child-first loader: defines every non-JDK class from its own URLs (delegating only JDK packages to the + * parent). This gives the probe — and the {@code ThriftHmsClient} + hadoop it touches — an isolated copy, + * a superset of the production plugin isolation. Mirrors {@code OdpsClassloaderIsolationTest}. + */ + private static final class IsolatedChildFirstClassLoader extends URLClassLoader { + + IsolatedChildFirstClassLoader(URL[] urls) { + super(urls, ClassLoader.getSystemClassLoader().getParent()); + } + + @Override + protected Class loadClass(String name, boolean resolve) throws ClassNotFoundException { + synchronized (getClassLoadingLock(name)) { + Class loaded = findLoadedClass(name); + if (loaded == null) { + if (isJdkClass(name)) { + loaded = super.loadClass(name, false); + } else { + try { + loaded = findClass(name); + } catch (ClassNotFoundException notLocal) { + loaded = super.loadClass(name, false); + } + } + } + if (resolve) { + resolveClass(loaded); + } + return loaded; + } + } + + private static boolean isJdkClass(String name) { + return name.startsWith("java.") || name.startsWith("javax.") + || name.startsWith("jdk.") || name.startsWith("sun.") + || name.startsWith("com.sun.") || name.startsWith("org.w3c.") + || name.startsWith("org.xml."); + } + } +} From 15d3df1dfd6fd2967656231cf7adf4c37fe1e721 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 10:10:36 +0800 Subject: [PATCH 229/334] [fix](catalog) hive: pin buildPluginAuthenticator TCCL to plugin loader (latent eager-UGI split-brain) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Latent-edge companion to the ThriftHmsClient.doAs fix (not hit by the 49 failing #991951 cases, hardened here per request). HiveConnector.buildPluginAuthenticator runs on the (unpinned) createClient thread; a catalog declaring hadoop.security.authentication=kerberos WITHOUT a principal/keytab falls back (AuthenticationConfig.getKerberosConfig) to a HadoopSimpleAuthenticator whose ctor EAGERLY calls UserGroupInformation.createRemoteUser -> SecurityUtil.. Its internal new Configuration() captures the TCCL, so an unpinned thread would load hadoop's DNSDomainNameResolver from fe-core's system-loader copy and split-brain-poison SecurityUtil against the plugin copy — the same failure doAs guards. buildHadoopConf sets only the OUTER conf's loader, not the TCCL that SecurityUtil's own Configuration reads, so a thread pin is required here too. Fix: pin HiveConnector.class.getClassLoader() around the whole resolution (try/finally), mirroring HudiConnector.metaClientExecutor and the doAs fix. Also refreshes the now-stale comment in createClient that described doAs as pinning the system classloader. Test: HiveConnectorPluginAuthenticatorTcclTest observes, via a TCCL-recording properties map, that the method body runs under the plugin loader and restores the caller's TCCL — RED without the pin (observed = caller marker), GREEN with it. fe-connector-hive 186/186 green (incl. the 5 existing buildPluginAuthenticator cases), 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../doris/connector/hive/HiveConnector.java | 46 ++++++---- ...eConnectorPluginAuthenticatorTcclTest.java | 84 +++++++++++++++++++ 2 files changed, 115 insertions(+), 15 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTcclTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index c747cd45e70dd2..2caa039a0dbca7 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -517,8 +517,9 @@ private HmsClient createClient() { // For a Kerberos catalog run the metastore RPC under the PLUGIN's UGI doAs (buildPluginAuthenticator), // NOT the FE-injected context: after the catalog flip that context resolves to NOOP (SIMPLE) auth, which // would silently downgrade a Kerberos HMS. AuthAction.execute is a generic method ( T execute(...)), - // so it cannot be a lambda — use an anonymous class. ThriftHmsClient.doAs already pins the RPC's TCCL to - // the system classloader; the plugin's HadoopAuthenticator only wraps it in a UGI doAs (no TCCL change). + // so it cannot be a lambda — use an anonymous class. ThriftHmsClient.doAs pins the RPC's TCCL to the + // plugin (child-first) classloader (so SecurityUtil. resolves hadoop from the plugin copy, not a + // split-brain against fe-core's copy); the plugin's HadoopAuthenticator only wraps it in a UGI doAs. HadoopAuthenticator auth = pluginAuthenticator(); ThriftHmsClient.AuthAction authAction; if (auth != null) { @@ -596,20 +597,35 @@ private HadoopAuthenticator pluginAuthenticator() { * Package-visible + static for KDC-free unit testing. */ static HadoopAuthenticator buildPluginAuthenticator(Map properties) { - if ("kerberos".equalsIgnoreCase(properties.get(HADOOP_SECURITY_AUTHENTICATION))) { - return HadoopAuthenticator.getHadoopAuthenticator(buildHadoopConf(properties)); - } - HmsMetaStoreProperties hms = (HmsMetaStoreProperties) MetaStoreProviders.bindForType( - HmsClientConfig.METASTORE_TYPE_HMS, properties, Collections.emptyMap()); - Optional spec = hms.kerberos(); - if (spec.isPresent() && spec.get().hasCredentials()) { - Configuration conf = buildHadoopConf(properties); - conf.set("hadoop.security.authentication", "kerberos"); - conf.set("hive.metastore.sasl.enabled", "true"); - return HadoopAuthenticator.getHadoopAuthenticator( - new KerberosAuthenticationConfig(spec.get().getPrincipal(), spec.get().getKeytab(), conf)); + // Pin the TCCL to the plugin (child-first) classloader for the whole resolution. A catalog that declares + // hadoop.security.authentication=kerberos WITHOUT a principal/keytab falls back to a + // HadoopSimpleAuthenticator whose ctor EAGERLY calls UserGroupInformation.createRemoteUser -> + // SecurityUtil., whose internal `new Configuration()` captures the current TCCL. This runs on the + // (unpinned) createClient thread, so without the pin it would resolve hadoop's DNSDomainNameResolver from + // fe-core's system-loader copy and split-brain-poison SecurityUtil against the plugin copy (the same + // failure ThriftHmsClient.doAs guards). buildHadoopConf pins only the OUTER conf's loader, not the TCCL + // that SecurityUtil's own Configuration reads — so the thread pin is required here too. + // (HadoopKerberosAuthenticator's keytab login is lazy in getUGI, already under doAs's pin.) + ClassLoader previous = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(HiveConnector.class.getClassLoader()); + if ("kerberos".equalsIgnoreCase(properties.get(HADOOP_SECURITY_AUTHENTICATION))) { + return HadoopAuthenticator.getHadoopAuthenticator(buildHadoopConf(properties)); + } + HmsMetaStoreProperties hms = (HmsMetaStoreProperties) MetaStoreProviders.bindForType( + HmsClientConfig.METASTORE_TYPE_HMS, properties, Collections.emptyMap()); + Optional spec = hms.kerberos(); + if (spec.isPresent() && spec.get().hasCredentials()) { + Configuration conf = buildHadoopConf(properties); + conf.set("hadoop.security.authentication", "kerberos"); + conf.set("hive.metastore.sasl.enabled", "true"); + return HadoopAuthenticator.getHadoopAuthenticator( + new KerberosAuthenticationConfig(spec.get().getPrincipal(), spec.get().getKeytab(), conf)); + } + return null; + } finally { + Thread.currentThread().setContextClassLoader(previous); } - return null; } /** diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTcclTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTcclTest.java new file mode 100644 index 00000000000000..5655e24cbbb4b7 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorPluginAuthenticatorTcclTest.java @@ -0,0 +1,84 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.net.URL; +import java.net.URLClassLoader; +import java.util.HashMap; + +/** + * Tests that {@link HiveConnector#buildPluginAuthenticator(java.util.Map)} runs under the plugin (child-first) + * classloader (TCCL pinned), then restores the caller's TCCL. + * + *

      WHY: a catalog that declares {@code hadoop.security.authentication=kerberos} WITHOUT a principal/keytab + * falls back to a {@code HadoopSimpleAuthenticator}, whose constructor EAGERLY calls + * {@code UserGroupInformation.createRemoteUser} → {@code SecurityUtil.} — whose internal + * {@code new Configuration()} captures the current TCCL. This resolution runs on the (unpinned) createClient + * thread, so an unpinned TCCL would load hadoop's {@code DNSDomainNameResolver} from fe-core's system-loader + * copy and split-brain-poison {@code SecurityUtil} against the plugin's copy (the same failure + * {@code ThriftHmsClient.doAs} guards). This is the latent-edge companion to that fix. + * + *

      The map records the TCCL the first time {@code buildPluginAuthenticator} reads a key (its very first + * statement), which is inside the pinned region. Simple-auth properties are used so the method returns quickly + * without an eager UGI side effect, yet still exercises the pin. Without the pin the observed loader would be + * the caller's marker — so the assertion is RED on a missing pin, and on a dropped restore. + */ +public class HiveConnectorPluginAuthenticatorTcclTest { + + /** A properties map that records the TCCL the first time a key is read. */ + private static final class TcclRecordingMap extends HashMap { + private static final long serialVersionUID = 1L; + private transient ClassLoader observed; + + @Override + public String get(Object key) { + if (observed == null) { + observed = Thread.currentThread().getContextClassLoader(); + } + return super.get(key); + } + } + + @Test + public void buildPluginAuthenticatorRunsUnderPluginClassLoaderAndRestores() { + TcclRecordingMap props = new TcclRecordingMap(); + props.put("hive.metastore.uris", "thrift://hms:9083"); + + ClassLoader marker = new URLClassLoader(new URL[0], getClass().getClassLoader()); + ClassLoader pluginLoader = HiveConnector.class.getClassLoader(); + ClassLoader original = Thread.currentThread().getContextClassLoader(); + Thread.currentThread().setContextClassLoader(marker); + try { + HiveConnector.buildPluginAuthenticator(props); + + Assertions.assertNotNull(props.observed, "buildPluginAuthenticator must have read the properties"); + Assertions.assertSame(pluginLoader, props.observed, + "buildPluginAuthenticator must run under the plugin classloader, so an eager " + + "HadoopSimpleAuthenticator UGI init cannot split-brain SecurityUtil"); + Assertions.assertNotSame(marker, props.observed, + "buildPluginAuthenticator must re-pin the TCCL away from the caller's loader"); + Assertions.assertSame(marker, Thread.currentThread().getContextClassLoader(), + "buildPluginAuthenticator must restore the caller's TCCL (try/finally)"); + } finally { + Thread.currentThread().setContextClassLoader(original); + } + } +} From cd0c12872a5eed57750c5e6d4bb03134d9d738e1 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 10:11:58 +0800 Subject: [PATCH 230/334] [doc](catalog) #991951 hive classloader split-brain: design + summary + HANDOFF RCA, fix design, and per-fix summary for the ThriftHmsClient.doAs / buildPluginAuthenticator TCCL classloader split-brain (CLR1 92004ef1d0d, CLR2 15d3df1dfd6). Records the e2e owed (real-cluster rerun of the 49 SecurityUtil cases) and the outlier classification (test_hdfs_parquet_group0 = BE ASAN mem flake, unrelated). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 11 ++++ .../FIX-CLR-classloader-splitbrain-design.md | 57 +++++++++++++++++++ .../FIX-CLR-classloader-splitbrain-summary.md | 23 ++++++++ plan-doc/tasks/task-list-CLR-991951.md | 14 +++++ 4 files changed, 105 insertions(+) create mode 100644 plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-design.md create mode 100644 plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-summary.md create mode 100644 plan-doc/tasks/task-list-CLR-991951.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 36dc11b4adab46..d1457a08087590 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -20,6 +20,17 @@ --- +# 🚑 插队任务 DONE（2026-07-11）——TeamCity #991951 hive connector CI classloader split-brain + +PR #65474（hive connector SPI 迁移）build **991951** 50 外表 case 失败。根因 = FE 侧 classloader split-brain：`ThriftHmsClient.doAs` 把 metastore-client 创建的 TCCL 钉成 `getSystemClassLoader()`，`SecurityUtil.` 的 `new Configuration()` 捕获它 → `DNSDomainNameResolver` 从 fe-core hadoop 副本加载、其父接口 `DomainNameResolver` 从 hive 插件 child-first 副本加载 → `isAssignableFrom` false → `SecurityUtil` 全 JVM 永久毒化 → 49 hive/iceberg-on-HMS/hudi/mtmv/kerberos/tvf 用例全挂（+outlier `test_hms_partitions_tvf` 同因，表象 comms-failure）。集群健康、非 case 问题。**已修**（设计+摘要 `plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-{design,summary}.md`，跟踪 `plan-doc/tasks/task-list-CLR-991951.md`，对抗验证 `wf_bf3a50e5-046` high-confidence）： +- `92004ef1d0d` **CLR1** `ThriftHmsClient.doAs`→`getClass().getClassLoader()`（plugin loader）+ 隔离 child-first loader 探针单测（RED/GREEN 双验）。fe-connector-hms 40/40。 +- `15d3df1dfd6` **CLR2** `HiveConnector.buildPluginAuthenticator` 方法体钉 plugin loader（挡 `HadoopSimpleAuthenticator` eager UGI latent 毒化）。fe-connector-hive 186/186。 +- **memory 新增第 4 个 TCCL-pin locus**：plain-hive HMS metastore-client 创建（见 `catalog-spi-plugin-tccl-classloader-gotcha`）。 +- **⚠ e2e 欠账（用户自跑）**：真集群重跑 991951 的 49 个 SecurityUtil 用例断言全绿（系统 vs 插件双 loader 只在真 child-first 环境复现，单测已用隔离 loader 复刻精确根因）。`test_hdfs_parquet_group0`（BE `MEM_LIMIT_EXCEEDED`/ASAN flake）与本 PR 无关。 +- **残余另开 ticket**：TVF analyze 阶段抛 `java.lang.Error` 未转 SQL 错误、拆连接（outlier1 表象来源；毒化去除后触发点消失）。 + +--- + # 🔧 复核修复系列（#65185 reverify，2026-07-11 起）——跟踪表 `plan-doc/task-list-65185-reverify-fixes.md` 翻闸后第三方复核（`reviews/catalog-spi-review-65185-reverify-2026-07-11.md`）判定的真实/活跃需修条目，按批次做（H1–H4 高 / M1–M8 中 / L1–L20 低 / D-系列设计债）。**处理纪律**：每条 = 设计(`tasks/designs/FIX--design.md`) → 设计红队 → 实现 → build+靶向 UT → 独立 commit → 勾表 → 更新 HANDOFF。path-whitelist `git add`。 diff --git a/plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-design.md b/plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-design.md new file mode 100644 index 00000000000000..aa41b79e766d99 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-design.md @@ -0,0 +1,57 @@ +# FIX-CLR — hive 连接器 HMS 路径 classloader split-brain（TeamCity #991951 / PR #65474） + +> 插队任务（2026-07-11）。TeamCity `Doris_External_Regression` build **991951**（PR #65474，hive connector 迁移）50 个外表 case 失败。经取回 fe.log + be.log + fe.conf/be.conf + 4 路对抗验证（`wf_bf3a50e5-046`，全部 high-confidence），定位为**一个** FE 侧 classloader split-brain 根因。 + +## Problem + +- 50 失败中 **49 直接** + **1（`test_hms_partitions_tvf`，表象为 `Communications link failure`）** = 同一根因；另 1（`test_hdfs_parquet_group0` BE `MEM_LIMIT_EXCEEDED`）与本 PR 无关（HDFS TVF，ASAN 内存压力 flake，重测/忽略）。 +- 集群健康：无 BE 宕机/OOM/配置错误；366 通过；"No backend available" 仅启动轮询期（05:52，测试前）。 +- 失败文案：FE 侧 `java.sql.SQLException`：首次 `ExceptionInInitializerError`，其后一律 `Could not initialize class org.apache.hadoop.security.SecurityUtil`（fe.log 185 次）——`SecurityUtil` 静态初始化失败被 JVM **永久毒化**。 + +## Root Cause + +首次失败栈（fe.log:10851/10922）： +``` +ThriftHmsClient.doAs(:636) → DefaultConnectorContext.executeAuthenticated(:178) + → RetryingMetaStoreClient.getUGI → UserGroupInformation.getCurrentUser → SecurityUtil.(:95) + → setConfigurationInternal(:123) → DomainNameResolverFactory.newInstance(:70) → Configuration.getClass(:2764) + ✗ RuntimeException: class org.apache.hadoop.net.DNSDomainNameResolver not org.apache.hadoop.net.DomainNameResolver +``` +`DNSDomainNameResolver extends DomainNameResolver`，`isAssignableFrom` 却为 false ⇒ 两类来自**不同 classloader**。运行时确有两份 Hadoop： +- fe-core `hadoop-client`（`fe-core/pom.xml:447`）→ **app/system** loader； +- hive 插件 `hadoop-common`（`fe-connector-hms/pom.xml:71` + `plugin-zip.xml`）→ **plugin child-first** loader（`ChildFirstClassLoader` allowlist 不含 `org.apache.hadoop.*` / `org.apache.doris.connector.hms.*`）。 + +**罪魁**：`ThriftHmsClient.doAs`（`fe-connector-hms/.../ThriftHmsClient.java:632-641`）在建 metastore client 前把 TCCL 钉成 `ClassLoader.getSystemClassLoader()`。`SecurityUtil.` 内 `new Configuration()` 捕获该 TCCL ⇒ `DNSDomainNameResolver` 从 system 副本加载，而其父接口 `DomainNameResolver`（由 plugin 加载的 `SecurityUtil` 引用）来自 plugin 副本 ⇒ 身份不一致 ⇒ 毒化。 + +**为何只有 hive 中招**：iceberg/paimon 在 `IcebergConnector:182`/`PaimonConnector:137` 用 `TcclPinningConnectorContext` 把 context 钉到 `getClass().getClassLoader()`（plugin loader）；`HiveConnector:113-117` 存裸 context，且 `doAs` 反钉 system loader（钉反）。`doAs` 是 hms+hive 两模块里**唯一** `getSystemClassLoader()` 钉点。同模块既有约定（`HiveConnectorMetadata:808/842`、`HudiConnector.metaClientExecutor`）均钉 plugin loader。 + +### 派生的 latent edge（未被 49 用例触发，用户要求本批一并加固） +`HiveConnector.buildPluginAuthenticator`（:598）在 `hadoop.security.authentication=kerberos` **但缺 principal/keytab** 的错配下 → `AuthenticationConfig.getKerberosConfig` 回落 `SimpleAuthenticationConfig`（`AuthenticationConfig.java:98-102`）→ `new HadoopSimpleAuthenticator`（`HadoopSimpleAuthenticator.java:37` **eager** `UGI.createRemoteUser`）→ 在**未钉的 createClient 线程**上初始化 `SecurityUtil` ⇒ 独立毒化。`buildHadoopConf` 只 `conf.setClassLoader(plugin)`（外层 conf），管不住 `SecurityUtil.` 内**另建**的 `new Configuration()` 捕获 TCCL，故须钉 TCCL。`HadoopKerberosAuthenticator` 的 login 是 lazy（在 `getUGI` 内，随 doAs 已被 FIX-CLR1 钉），不受影响。 + +## Design + +两个独立、最小、连接器局部的 TCCL 钉点（对齐 iceberg/paimon/hudi 与同模块 stats 方法先例；**不**在 fe-core 加任何 source-specific 代码）： + +- **FIX-CLR1（根因，解 50/50 里的 49+outlier1）**：`ThriftHmsClient.doAs` 把 `ClassLoader.getSystemClassLoader()` 改为 `getClass().getClassLoader()`（= 加载 ThriftHmsClient 的 plugin child-first loader，是 system loader 的严格超集）。`doAs` 是 client 创建（`createFreshClient:722`）与每次 RPC（`execute:618`）的**唯一咽喉**，且 Kerberos/非 Kerberos 两条 authAction 都在其内 ⇒ 一处修复覆盖两路（`TcclPinningConnectorContext` 式"包 context"改法修不了 Kerberos，因 Kerberos authAction 绕过 `context.executeAuthenticated`）。附带更新 `HiveConnector.java:520-521` 现已过时的注释（"system classloader" → "plugin classloader"）。 + +- **FIX-CLR2（latent 加固）**：在 `HiveConnector.buildPluginAuthenticator` 方法体外包 TCCL 钉到 `HiveConnector.class.getClassLoader()`，try/finally 还原。使 `HadoopSimpleAuthenticator` eager UGI 在 plugin loader 下初始化。 + +## Implementation Plan + +1. FIX-CLR1：改 `ThriftHmsClient.doAs` 一行 + 更新 `HiveConnector:520-521` 注释。 +2. FIX-CLR2：`buildPluginAuthenticator` 方法体包 `ClassLoader prev=TCCL; try{ set(HiveConnector.class.getClassLoader()); <原体> } finally { set(prev); }`。 + +## Risk Analysis + +- plugin child-first loader 委派父加载器解析未自带类 ⇒ 对 system loader 严格超集，无可见性回归；hadoop/hive/thrift 解析到 plugin 自带副本（正是所需）。 +- 铁律核对：无 fe-core 改动、无 source-specific 分支、不解析属性；仅连接器局部钉 TCCL（memory `catalog-spi-plugin-tccl-classloader-gotcha` 的 HMS-client-创建 locus，为其**第 4 个**登记 locus）。 +- 残余（本 PR 不 own）：TVF analyze 阶段抛 `java.lang.Error` 未转 SQL 错误、直接拆连接（outlier1 表象）——毒化去除后触发点即消失，另开 hardening ticket。 + +## Test Plan + +### Unit Tests（连接器模块无 Mockito；用 recording fake + child-first marker loader，须 RED-able） +- **FIX-CLR1** `ThriftHmsClientDoAsClassLoaderTest`（fe-connector-hms）：注入 recording `MetaStoreClientProvider` 捕获 `doAs` 内建 client 时的 TCCL；调 `listDatabases()`。断言：TCCL == `ThriftHmsClient.class.getClassLoader()`、!= 调用方 marker、事后还原 marker；当 `system != connectorLoader`（env 允许时）额外断言 != `getSystemClassLoader()`（精确根因回归门）。 +- **FIX-CLR2** `HiveConnectorPluginAuthenticatorTccl` 断言（并入/毗邻 `HiveConnectorPluginAuthenticatorTest`）：传一个在 `get()` 时记录 TCCL 的 Map，marker 下调 `buildPluginAuthenticator`；断言方法体内 TCCL == `HiveConnector.class.getClassLoader()`、!= marker（**未钉时停留在 marker ⇒ RED**）、事后还原 marker。 + +### E2E Tests（用户自跑，勿丢） +真集群重跑 build 991951 的 49 个 SecurityUtil 用例（hive/iceberg-on-HMS/hudi/mtmv/kerberos/tvf/export/cache），断言全绿；系统 vs 插件双 loader 拓扑只在真插件 child-first 环境复现，单测只钉 intent + 还原（对齐 `HiveConnectorMetadataFileListStatsTest` 先例）。`test_hdfs_parquet_group0` 与本 PR 无关，另议 BE mem_limit/测试数据。 diff --git a/plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-summary.md b/plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-summary.md new file mode 100644 index 00000000000000..178d44cd875175 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-summary.md @@ -0,0 +1,23 @@ +# FIX-CLR Summary — hive HMS classloader split-brain（TeamCity #991951 / PR #65474） + +## Problem +build 991951（hive connector SPI 迁移 PR）50 个外表 case 失败。集群健康（366 通过、无 BE 宕机/OOM、`priority_networks` 正确、"No backend available" 仅启动轮询期）。 + +## Root Cause +FE 侧 classloader split-brain：`ThriftHmsClient.doAs` 在建 metastore client 前把 TCCL 钉成 `ClassLoader.getSystemClassLoader()`。`SecurityUtil.` 内 `new Configuration()` 捕获该 TCCL 反射加载 `DNSDomainNameResolver`——system loader 是 fe-core 的 hadoop 副本，而 `SecurityUtil`/`DomainNameResolver` 来自 hive 插件 child-first 副本 → `isAssignableFrom` false（"class DNSDomainNameResolver not DomainNameResolver"）→ `ExceptionInInitializerError` → `SecurityUtil` 全 JVM 永久毒化 → 所有 hive/iceberg-on-HMS/hudi/mtmv/kerberos 操作报 "Could not initialize class SecurityUtil"。iceberg/paimon 用 `TcclPinningConnectorContext` 钉 plugin loader 故免疫；hive 独中招。4 路对抗验证（`wf_bf3a50e5-046`）high-confidence 确认。 + +## Fix +- **CLR1（根因）** `92004ef1d0d`：`ThriftHmsClient.doAs` → `getClass().getClassLoader()`（plugin child-first loader，system loader 严格超集）。单一咽喉覆盖 client 创建 + 每次 RPC + Kerberos/非 Kerberos 两 authAction。附带更新 `HiveConnector:517-521` stale 注释。 +- **CLR2（latent 加固）** `15d3df1dfd6`：`HiveConnector.buildPluginAuthenticator` 方法体钉 `HiveConnector.class.getClassLoader()`（try/finally），挡 kerberos-无凭证错配下 `HadoopSimpleAuthenticator` eager `UGI.createRemoteUser` 在未钉线程毒化。 + +## Tests +- `ThriftHmsClientDoAsClassLoaderTest`（+ `DoAsTcclProbe`）：经隔离 child-first loader 跑探针（镜像 `OdpsClassloaderIsolationTest`），使 `getClass().getClassLoader()` 与 system loader 不同 → 精确区分根因。**RED**（fix 还原）=`PIN_WRONG_SYSTEM_LOADER`、**GREEN**（fix）双向实测。 +- `HiveConnectorPluginAuthenticatorTcclTest`：TCCL-recording Map 观测方法体内 TCCL == plugin loader + 还原。**RED**（钉去除）=marker、**GREEN** 双向实测。 +- 回归：fe-connector-hms 40/40、fe-connector-hive 186/186（含 5 个既有 authenticator 用例）、0 checkstyle。 + +## Result +两连接器局部 TCCL 钉点，无 fe-core 改动、无 source-specific 分支、不解析属性（守铁律）。**系统 vs 插件双 loader 拓扑只在真 child-first 插件环境复现**（单测已用隔离 loader 复刻 CLR1 精确根因）：**e2e 欠账 = 真集群重跑 991951 的 49 个 SecurityUtil 用例断言全绿**（用户自跑）。 +`test_hdfs_parquet_group0`（BE `MEM_LIMIT_EXCEEDED`，HDFS TVF/ASAN 内存 flake）与本 PR 无关，重测/忽略（另议 BE mem_limit/测试数据）。 + +## 残余（非本批 own，另开 ticket） +- TVF analyze（Nereids parse 阶段）抛 `java.lang.Error` 未转 SQL 错误、直接拆 client 连接（outlier1 `test_hms_partitions_tvf` 的 comms-failure 表象来源）——毒化去除后触发点消失，可另开 FE 硬化 ticket。 diff --git a/plan-doc/tasks/task-list-CLR-991951.md b/plan-doc/tasks/task-list-CLR-991951.md new file mode 100644 index 00000000000000..594a886eb00f63 --- /dev/null +++ b/plan-doc/tasks/task-list-CLR-991951.md @@ -0,0 +1,14 @@ +# Task List — TeamCity #991951 / PR #65474 classloader split-brain (插队任务, 2026-07-11) + +设计 + RCA：`plan-doc/tasks/designs/FIX-CLR-classloader-splitbrain-design.md` +证据：fe.log/be.log/fe.conf/be.conf（build 991951 archive）+ 对抗验证 `wf_bf3a50e5-046`（4 agent，high-confidence）。 + +- [x] **FIX-CLR1**（根因，解 50 里的 49 + outlier `test_hms_partitions_tvf`）：`ThriftHmsClient.doAs` 钉 plugin loader（`getSystemClassLoader()`→`getClass().getClassLoader()`）+ 更新 `HiveConnector` stale 注释 + `ThriftHmsClientDoAsClassLoaderTest`（隔离 child-first loader 探针，RED=`PIN_WRONG_SYSTEM_LOADER`/GREEN 双验）。commit `92004ef1d0d`。fe-connector-hms 40/40 绿、0 checkstyle。 +- [x] **FIX-CLR2**（latent 加固，用户要求本批一并）：`HiveConnector.buildPluginAuthenticator` 方法体钉 plugin loader（挡 `HadoopSimpleAuthenticator` eager UGI 毒化）+ `HiveConnectorPluginAuthenticatorTcclTest`（RED=marker/GREEN 双验）。commit `15d3df1dfd6`。fe-connector-hive 186/186 绿（含 5 个既有 authenticator 用例）、0 checkstyle。 + +## 归类（50 失败；muted 忽略） +- 49 直接 `SecurityUtil` 毒化 + 1 outlier1（`test_hms_partitions_tvf`，comms 中断表象）= FIX-CLR1 解。 +- 1 outlier2（`test_hdfs_parquet_group0`，BE `MEM_LIMIT_EXCEEDED`，HDFS TVF/ASAN 内存 flake）= **与本 PR 无关**，重测/忽略（另议 BE mem_limit/测试数据）。 + +## e2e（用户自跑，勿丢） +真集群重跑 49 个 SecurityUtil 用例断言全绿；系统 vs 插件双 loader 只在真 child-first 环境复现（单测钉 intent+还原）。 From 6963de4124f78f84b0d0aa7c5f651934945750b8 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 11:07:16 +0800 Subject: [PATCH 231/334] [fix](catalog) batch-mode gate: restore != NOT_PRUNED for no-predicate large partition scans PluginDrivenScanNode.shouldUseBatchMode guarded on !selectedPartitions.isPruned, which diverges from legacy MaxComputeScanNode.isBatchMode's != NOT_PRUNED. A partitioned table with NO partition predicate is initialized by ExternalTable.initSelectedPartitions to a full, non-NOT_PRUNED map with isPruned=false (PruneFileScanPartition only runs under a LogicalFilter): legacy batches it, but !isPruned wrongly forced it synchronous, so >=num_partitions_in_batch_mode scans materialized every partition's splits into the FE heap at once (planning latency + heap pressure; rows correct). Fix: gate on == NOT_PRUNED (mirrors legacy and the sibling displayPartitionCounts). An enumeration of every SelectedPartitions producer confirms the only newly-batched state is exactly that no-predicate full-map case; non-partitioned tables still carry the NOT_PRUNED singleton and stay non-batch, so no gate is dropped. Restores async batch for MaxCompute and (post supportsBatchScan) hive no-predicate large scans, resolving the M2-registered BATCH-UNPRUNED-SYNC deviation. Split-generation strategy only; query rows unchanged. - Invert the deliberately-pinned test testUnprocessedPruningNeverBatches -> testNoPredicatePartitionedTableBatches (now asserts it batches). A prior review's "!isPruned is equivalent to != NOT_PRUNED and slightly stronger" note is superseded (proven false for this case); recorded in decisions-log D-035 / deviations-log DV-019. - Update docker-hive golden test_hive_partitions: plugin-SPI batch mode reports the approximate split count as the selected PARTITION count (=6, uniform across connectors / matches MaxCompute) rather than legacy hive's numSplitsPerPartition*partitions (=60); the "(approximate)" prefix returns, confirming async batch generation was chosen for the no-predicate full scan. fe-core UT PluginDrivenScanNodeBatchModeTest 12/12 green; docker-hive e2e live-gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../datasource/PluginDrivenScanNode.java | 21 ++++++++----- .../PluginDrivenScanNodeBatchModeTest.java | 30 ++++++++++++------- .../hive/test_hive_partitions.groovy | 7 ++++- 3 files changed, 40 insertions(+), 18 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 8f5a4470766c81..19bf958fbc9cc7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -1119,21 +1119,28 @@ private boolean computeBatchMode() { * {@link FileQueryScanNode} (the async/wiring half is covered by live e2e — see DV-019). * *

        - *
      • not partitioned / not pruned ({@code selectedPartitions} null or {@code !isPruned}) → false;
        • + *
      • null or the {@link SelectedPartitions#NOT_PRUNED} sentinel (non-partitioned, or Nereids + * pruning not applicable) → false;
        • *
      • no required slots → false;
        • *
      • connector does not support batch scan (incl. no scan provider) → false;
        • - *
      • otherwise batch iff {@code numPartitionsInBatchMode > 0} and the pruned partition count + *
      • otherwise batch iff {@code numPartitionsInBatchMode > 0} and the selected partition count * reaches that threshold.
        • *
      * - *

      The {@code !isPruned} check subsumes BOTH legacy gates ({@code getPartitionColumns().isEmpty()} - * and the reference check {@code != NOT_PRUNED}): a non-partitioned external table always carries - * {@code NOT_PRUNED} (which has {@code isPruned=false}), so collapsing them is not a dropped gate — - * it is in fact marginally stronger than legacy's reference identity check.

      + *

      The gate is {@code == NOT_PRUNED}, deliberately not {@code !isPruned} — mirroring legacy + * {@code MaxComputeScanNode.isBatchMode}'s {@code != NOT_PRUNED} and the sibling + * {@link #displayPartitionCounts}. The two are not equivalent: a partitioned table with no + * partition predicate is initialized by {@link ExternalTable#initSelectedPartitions} to a full, + * non-{@code NOT_PRUNED} map with {@code isPruned=false} ({@code PruneFileScanPartition} only runs + * under a {@code LogicalFilter}). Legacy batches that case — a large full scan is exactly what most + * needs async/streaming split generation — whereas an {@code !isPruned} gate wrongly forces it + * synchronous (the split-materialization regression this restores). The {@code == NOT_PRUNED} sentinel + * check still folds in the non-partitioned gate ({@code getPartitionColumns().isEmpty()}), which + * always carries the {@code NOT_PRUNED} singleton, so no gate is dropped.

      */ static boolean shouldUseBatchMode(SelectedPartitions selectedPartitions, boolean hasSlots, boolean supportsBatchScan, int numPartitionsInBatchMode) { - if (selectedPartitions == null || !selectedPartitions.isPruned) { + if (selectedPartitions == null || selectedPartitions == SelectedPartitions.NOT_PRUNED) { return false; } if (!hasSlots) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java index 9adc6e2315774f..fb7a7c7289d0a0 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeBatchModeTest.java @@ -42,7 +42,7 @@ * *

      Why this matters: batch mode mirrors legacy {@code MaxComputeScanNode.isBatchMode()}. * Getting any gate wrong has real consequences: enabling batch when it should not (e.g. dropping the - * "must be pruned" or "must have files" guard) spins up async read sessions for the wrong tables; + * "not the NOT_PRUNED sentinel" or "must have files" guard) spins up async read sessions for the wrong tables; * disabling it when it should fire (e.g. an off-by-one on the partition-count threshold) silently * regresses large-partition scans back to slow synchronous planning + large single sessions (the * exact OOM/latency risk this fix removes). The connector {@code fileNum > 0} check is folded into @@ -69,10 +69,11 @@ private static SelectedPartitions pruned(int count) { @Test public void testNotPrunedNeverBatches() { - // NOT_PRUNED = non-partitioned / pruning not applied -> never batch. NOTE: NOT_PRUNED carries - // an EMPTY map, so this case is non-discriminating for the !isPruned guard alone (0 >= THRESHOLD - // is false regardless); the guard mutant is killed by testUnprocessedPruningNeverBatches - // (populated map). This test documents the legacy NOT_PRUNED singleton path. + // NOT_PRUNED = non-partitioned / pruning not applicable -> never batch. NOTE: NOT_PRUNED carries + // an EMPTY map, so this case is non-discriminating for the == NOT_PRUNED guard alone (0 >= THRESHOLD + // is false regardless); the guard's discriminating counterpart is testNoPredicatePartitionedTableBatches + // (a full, non-sentinel isPruned=false map, which MUST batch). This test documents the NOT_PRUNED + // singleton path (non-partitioned / unpruned). Assertions.assertFalse( PluginDrivenScanNode.shouldUseBatchMode(SelectedPartitions.NOT_PRUNED, true, true, THRESHOLD)); } @@ -83,15 +84,24 @@ public void testNullSelectionNeverBatches() { } @Test - public void testUnprocessedPruningNeverBatches() { - // isPruned=false with a populated map is "pruning not processed" -> not batch. Pins the - // !isPruned guard: dropping it would batch on an unpruned (effectively full) selection. + public void testNoPredicatePartitionedTableBatches() { + // A partitioned table with NO partition predicate: ExternalTable.initSelectedPartitions returns a + // FULL, non-NOT_PRUNED map with isPruned=false (PruneFileScanPartition only runs under a + // LogicalFilter). Scanning every partition is EXACTLY the case that most needs async/streaming split + // generation, and legacy MaxComputeScanNode.isBatchMode (its != NOT_PRUNED gate) batched it. The gate + // is == NOT_PRUNED, NOT !isPruned: this object is not the sentinel, so it MUST batch once the + // partition count reaches the threshold. The old !isPruned gate wrongly returned false here (forcing + // slow synchronous planning on the largest scans) — the regression this now pins against. NOTE this + // inverts the former testUnprocessedPruningNeverBatches assertion: a prior review's "!isPruned is + // equivalent to != NOT_PRUNED and slightly stronger" note was mistaken (they diverge for exactly this + // case); that note and the test pinning it are superseded (decisions-log D-035 / deviations-log DV-019). Map items = new LinkedHashMap<>(); for (int i = 0; i < THRESHOLD; i++) { items.put("pt=" + i, Mockito.mock(PartitionItem.class)); } - SelectedPartitions notProcessed = new SelectedPartitions(THRESHOLD, items, false); - Assertions.assertFalse(PluginDrivenScanNode.shouldUseBatchMode(notProcessed, true, true, THRESHOLD)); + SelectedPartitions noPredicateFullScan = new SelectedPartitions(THRESHOLD, items, false); + Assertions.assertTrue( + PluginDrivenScanNode.shouldUseBatchMode(noPredicateFullScan, true, true, THRESHOLD)); } @Test diff --git a/regression-test/suites/external_table_p0/hive/test_hive_partitions.groovy b/regression-test/suites/external_table_p0/hive/test_hive_partitions.groovy index f5944e798a0634..35ecb5f99597ae 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_partitions.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_partitions.groovy @@ -197,7 +197,12 @@ suite("test_hive_partitions", "p0,external") { sql ("select * from partition_table") verbose (true) - contains "(approximate)inputSplitNum=60" + // Plugin-SPI hive batch mode reports the approximate split count as the SELECTED PARTITION + // count (numApproximateSplits = selectedPartitions.size() = 6), uniform across connectors + // (matches MaxCompute). Legacy HiveScanNode reported numSplitsPerPartition * partitions (=60); + // the "(approximate)" prefix still confirms async batch generation was chosen for this + // no-predicate full scan (num_partitions_in_batch_mode=1 forces it). + contains "(approximate)inputSplitNum=6" } sql """unset variable num_partitions_in_batch_mode""" } finally { From 97363fc9c3390541db7209bb13cc721a31eb2a29 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 11:09:13 +0800 Subject: [PATCH 232/334] [doc](catalog) #65185 batch-2: M3 batch-gate NOT_PRUNED fix design + supersession + tracking MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - FIX-M3-design.md: root cause (git-history legacy != NOT_PRUNED + full SelectedPartitions producer enumeration), design red-team wf_811e6242-d8b (3 lens; 1 blocker + 1 major resolved), final change list + e2e registration. - decisions-log D-035 / deviations-log DV-019: SUPERSEDED-IN-PART annotations — the prior impl-review LP-1 "!isPruned equivalent to != NOT_PRUNED" note is overturned (proven false for no-predicate partition tables); the pinning test was inverted. - task-list + HANDOFF: mark M3 DONE (6963de4124f), record the docker-hive golden 60->6 decision (user signoff 2026-07-11: uniform SPI partition-count semantics) and the concurrent-build contamination gotcha; next = M1. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 3 +- plan-doc/decisions-log.md | 2 +- plan-doc/deviations-log.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 11 +- plan-doc/tasks/designs/FIX-M3-design.md | 146 +++++++++++++++++++++ 5 files changed, 159 insertions(+), 5 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M3-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d1457a08087590..8b94b9daffe708 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -41,7 +41,8 @@ PR #65474（hive connector SPI 迁移）build **991951** 50 外表 case 失败 - **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 **⭐ 下一步（下个 session）：** -1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`+`fca288424fc`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3；**最终复核纠正 TTL 86400→600s 对齐旧版**) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**最终对抗复核（`wf_542c60b9-001`，5 per-fix + 1 cross-cut）= M5/M6/M7/M2/cross-cut CLEAN、M4 命中 1 medium(TTL)已修 → 全 CLEAN。****下一步 = 批次 2（M3→M1，fe-core 通用节点，blast radius 较大，单测充分后再动）**；M3(`shouldUseBatchMode` 的 `!isPruned`→`==NOT_PRUNED`)顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。之后批次 3（M8/L1）… ⏸ 决策类（L2/L10/L12/L20）先问用户。 +1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`+`fca288424fc`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3；**最终复核纠正 TTL 86400→600s 对齐旧版**) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**最终对抗复核（`wf_542c60b9-001`，5 per-fix + 1 cross-cut）= M5/M6/M7/M2/cross-cut CLEAN、M4 命中 1 medium(TTL)已修 → 全 CLEAN。** + **⭐ 批次 2（M3→M1，fe-core 通用节点）进行中：M3 DONE `6963de4124f`**——`shouldUseBatchMode` 的 `!isPruned`→`== NOT_PRUNED`，无谓词大分区表（MaxCompute + 翻闸 hive）恢复异步 batch split，**顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余**（legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证；git `1da88365e85^` 取证 + 全 producer 枚举证闭合）。设计红队 `wf_811e6242-d8b`（3 lens）命中 1 blocker + 1 major 均已解：**反转**被前次评审特意锁定的 pinning 测试（`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`）、**登记 supersession**（`decisions-log` D-035 / `deviations-log` DV-019 的 LP-1「`!isPruned` 等价」判定已批注 SUPERSEDED）、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`（**用户 2026-07-11 签字：采用 SPI 统一分区数口径**，不给 hive 补 split-count 估算；对齐 MaxCompute/Trino）。`PluginDrivenScanNodeBatchModeTest` 12/12 绿。设计 `designs/FIX-M3-design.md`。**⚠ 本轮踩并行 session 构建污染坑**（`be-java-extensions package -am -T 1C` 污染共享 target 报「cannot access 生成类」假失败，非本码；待其结束干净重跑 12/12）。**下一步 = M1**（TABLESAMPLE 插件路径静默全表扫；translator 转发 `setTableSample` + `PluginDrivenScanNode.getSplits` 通用按 split 大小采样，connector-agnostic）。之后批次 3（M8/L1）… ⏸ 决策类（L2/L10/L12/L20）先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 diff --git a/plan-doc/decisions-log.md b/plan-doc/decisions-log.md index 87fbcaf09f8a51..0bb4bb7fa35a77 100644 --- a/plan-doc/decisions-log.md +++ b/plan-doc/decisions-log.md @@ -45,7 +45,7 @@ | D-038 | P5-D2 | **P5 paimon MTMV + MVCC(时间旅行) scope = P5 内实现桥，翻闸 gated on 它（用户签字，design-only）**：SPI 当前 **MTMV 完全无面（E10 缺）**（`PluginDrivenExternalTable:62` 不 implements MTMVRelatedTableIf/MTMVBaseTableIf/MvccTable，框架靠 `instanceof MTMVRelatedTableIf` 分发——`MTMVPartitionUtil:265/497/588`、`StatementContext:987/1003`），E5(MVCC) `defined-no-consumer`（`ConnectorMvccSnapshotAdapter` 仅自身文件引用、`ConnectorScanRange` 无 snapshot 字段）。legacy `PaimonExternalTable:74` 实现全套。翻闸不机械阻断（plain SELECT 经 `getPaimonTable(empty)` 取 latest）但按 MC 样板直接翻闸=**静默回归** paimon-as-MTMV-base + 时间旅行。**用户定 = 方案 A**：P5 内落 fe-core `PaimonPluginDrivenExternalTable extends PluginDrivenExternalTable` 实现三接口 + 首个真 E5 消费者 override `beginQuerySnapshot` 三方法 + 新增 GAP-LISTPART-AT-SNAPSHOT 的 at-snapshot listPartitions；表级 staleness=`ConnectorMvccSnapshot.getSnapshotId()`(-1 空表)、分区级=`ConnectorPartitionInfo.getLastModifiedMillis()`(已存在)；MTMV 类型/PartitionItem 留 fe-core、连接器仅供 SPI-neutral 数据。**翻闸(B7) gated on MTMV 桥(B5)**；**禁**静默读 latest。否决 B（翻闸先行 + MTMV fail-loud 延后）。最高 correctness 风险=单-pin 不变式 + `lastFileCreationTime()` 跨 flavor 可靠性（SDK 行为，须 live 验）。设计 `tasks/P5-paimon-migration.md` §开放决策 D2 + recon §3.5/§4 | 2026-06-09 | ✅ | | D-037 | P5-D1 | **P5 paimon flavor(hms/filesystem/dlf/rest/jdbc) 装配 = 单 Catalog + `createCatalog` flavor switch（MC 一致，用户签字，design-only）**：连接器现走单 Catalog stub（`PaimonConnector.createCatalog:75-83` 把 `Options.fromMap` 直喂 paimon SDK CatalogFactory，无 Doris 侧 warehouse/HiveConf/StorageProperties/authenticator 装配）；5 个 `fe-connector-paimon-backend-*` 模块**是空壳**（仅 gitignore `.flattened-pom.xml`、零 src/未注册 Maven 模块）。legacy 装配在 fe-core `AbstractPaimonProperties`+5 子类+`PaimonPropertiesFactory`，全 import 禁用的 fe-core `StorageProperties`/`HMSBaseProperties`/`HadoopExecutionAuthenticator`。**用户定 = 方案 A**：`PaimonConnector.createCatalog` 内 switch on `paimon.catalog.type`，**拷** warehouse/conf/S3-normalize + 重建 Hadoop/HiveConf + **每-flavor ExecutionAuthenticator** 入模块（镜像 MC 拷 MCProperties→MCConnectorProperties；filesystem→hms→rest/jdbc/dlf 渐进）。**不**建 backend 模块 + ServiceLoader（否决 B：无 MC 先例、大 surface、空壳从零建）。约束：StorageProperties 从属性 map 重建（禁 import）；**每-flavor authenticator 必须保**（否则 Kerberized HMS/HDFS DDL 运行时炸、无离线测覆盖）。设计 `tasks/P5-paimon-migration.md` §开放决策 D1 + recon §3.4 | 2026-06-09 | ✅ | | D-036 | — | **P4-T06e FIX-CAST-PUSHDOWN MaxCompute 关 CAST 谓词下推 + 剥壳时抑制 source LIMIT（F9 静默丢行回归，review 原误判 known-degr 已推翻）**：共享 converter 无条件剥 CAST（`ExprToConnectorExpressionConverter:108`）、MaxCompute 不 override `supportsCastPredicatePushdown`（继承默认 true）→ `buildRemainingFilter` 不剔除含 CAST 的 conjunct → 剥壳谓词推入 ODPS read session（`CAST(str AS INT)=5`→源过滤 `str="5"` 按列 STRING quote）→ 源端 under-match 丢 `'05'/' 5'`、BE 复算只能过滤超集向下无法找回 → **静默丢行**。legacy `convertSlotRefToColumnName` 对 CAST 操作数抛异常→caught→丢弃该谓词（BE-only）→正确 ⇒ cutover 比 legacy 严格更紧 = **回归**（区别于 [DV-016] 仅 limit-opt 资格 CAST-unwrap、非丢行）。**对抗核验 `wzoa6dkvw` 0/3 refuted、verdict=real-unregistered-regression**。**用户定 Fix**。修 = ① 连接器 `MaxComputeConnectorMetadata.supportsCastPredicatePushdown→false`（激活既有 strip 路径、CAST conjunct 保留 BE-only、恢复 legacy parity；镜像 JDBC + `ConnectorPushdownOps` doc 处方；无 SPI 变更、无新路径）；② fe-core `getSplits` 在 CAST conjunct 被剥（`filteredToOriginalIndex!=null`）时抑制 source LIMIT 下推（抽纯静态 `effectiveSourceLimit`）——否则连接器收空 filter→limit-opt(ON 时) row-offset 读首 N 行无谓词→BE under-return（impl-review `wj2h0120n` F9-LIMITOPT-1 折入；`startSplit` 批路径已恒 -1[DEC-1] 故只改 getSplits）。守门：连接器 UT 2/2+mutation(false→true 红)、fe-core LimitStrip 2/2+BatchMode 9/9+mutation 2/2 向红、checkstyle 0、import-gate 净。真值闸=live ODPS CAST(str)=5 返回全集（DV-020，CI 跳）。out-of-scope surface：JDBC `applyLimit`+cast-off 理论同类（MC 不 override applyLimit、本修对 MC 完整）。commit `cc32521ed99` | 2026-06-08 | ✅ | -| D-035 | — | **P4-T06e FIX-BATCH-MODE-SPLIT 通用 batch SPI 路径恢复异步分批 split（Shape A，NG-7/F6=F13 minor）**：翻闸后 `PluginDrivenScanNode` 不 override `isBatchMode/numApproximateSplits/startSplit` → 继承 `SplitGenerator` 默认（false/-1/no-op）→ plugin-driven(含 MC) 读永走同步 `getSplits` 一次性枚举全(已裁剪)分区 split；legacy `MaxComputeScanNode:214-298` 分批异步建 read session 流式喂 split。P1-4 后降级收窄到「裁剪后仍 ≥`num_partitions_in_batch_mode` 分区」（规划慢 + 大 session 潜在 OOM、行正确）。**用户定「实现 batch SPI 路径」（非 DV）**。修 = **Shape A（薄 SPI + fe-core 编排、逐字镜像 legacy）**：① SPI `ConnectorScanPlanProvider` +2 additive default（`supportsBatchScan` 默认 false / `planScanForPartitionBatch` 默认委托 6 参 `planScan` over 子集）零破坏其余 6 连接器；② 连接器 `MaxComputeScanPlanProvider.supportsBatchScan`=`odpsTable.getFileNum()>0`（`planScanForPartitionBatch` 不 override，继承默认即批语义）；③ fe-core `PluginDrivenScanNode`（extends `FileQueryScanNode` 已继承 batch dispatch+stop；`PluginDrivenSplit extends FileSplit` 故 `:381` 转型安全）override `isBatchMode`（4 闸 isPruned+slots+supportsBatchScan+size≥阈值，含 SF-1 `getScanPlanProvider()` null-guard）/`numApproximateSplits`=size/`startSplit`（`getScheduleExecutor` outer/inner CompletableFuture 分批，`needMoreSplit/addToQueue/finishSchedule/setException/isStop` 契约，DEC-1 不下推 limit 传 -1 与 P3-9 limit-opt 互斥）+ 抽纯静态 `shouldUseBatchMode` 供单测。clean-room 设计验证 `wcpg9lblj` GO-WITH-EDITS（0 mustFix + 2 shouldFix：SF-1 null-guard NPE 修 + 预核文案，已折入）+ impl-review `wve7y1jst` GO-WITH-EDITS（0 mustFix + 1 shouldFix TQ-1 测试覆盖文案诚实降级 + 2 nit，已折入）。守门：编译 BUILD SUCCESS、fe-core UT 9/9、fe-connector-api UT 2/2、checkstyle 0、import-gate 净、mutation 5/5 向红。真值闸=大分区 live e2e（DV-019，CI 跳）。**Batch-D 红线**：legacy `MaxComputeScanNode` batch 逻辑须待本 fix 落才可删（读裁剪那半 P1-4 已清，本项为最后前置闸）。commit `ac8f0fc15eb` | 2026-06-08 | ✅ | +| D-035 | — | **P4-T06e FIX-BATCH-MODE-SPLIT 通用 batch SPI 路径恢复异步分批 split（Shape A，NG-7/F6=F13 minor）**：翻闸后 `PluginDrivenScanNode` 不 override `isBatchMode/numApproximateSplits/startSplit` → 继承 `SplitGenerator` 默认（false/-1/no-op）→ plugin-driven(含 MC) 读永走同步 `getSplits` 一次性枚举全(已裁剪)分区 split；legacy `MaxComputeScanNode:214-298` 分批异步建 read session 流式喂 split。P1-4 后降级收窄到「裁剪后仍 ≥`num_partitions_in_batch_mode` 分区」（规划慢 + 大 session 潜在 OOM、行正确）。**用户定「实现 batch SPI 路径」（非 DV）**。修 = **Shape A（薄 SPI + fe-core 编排、逐字镜像 legacy）**：① SPI `ConnectorScanPlanProvider` +2 additive default（`supportsBatchScan` 默认 false / `planScanForPartitionBatch` 默认委托 6 参 `planScan` over 子集）零破坏其余 6 连接器；② 连接器 `MaxComputeScanPlanProvider.supportsBatchScan`=`odpsTable.getFileNum()>0`（`planScanForPartitionBatch` 不 override，继承默认即批语义）；③ fe-core `PluginDrivenScanNode`（extends `FileQueryScanNode` 已继承 batch dispatch+stop；`PluginDrivenSplit extends FileSplit` 故 `:381` 转型安全）override `isBatchMode`（4 闸 isPruned+slots+supportsBatchScan+size≥阈值，含 SF-1 `getScanPlanProvider()` null-guard）/`numApproximateSplits`=size/`startSplit`（`getScheduleExecutor` outer/inner CompletableFuture 分批，`needMoreSplit/addToQueue/finishSchedule/setException/isStop` 契约，DEC-1 不下推 limit 传 -1 与 P3-9 limit-opt 互斥）+ 抽纯静态 `shouldUseBatchMode` 供单测。clean-room 设计验证 `wcpg9lblj` GO-WITH-EDITS（0 mustFix + 2 shouldFix：SF-1 null-guard NPE 修 + 预核文案，已折入）+ impl-review `wve7y1jst` GO-WITH-EDITS（0 mustFix + 1 shouldFix TQ-1 测试覆盖文案诚实降级 + 2 nit，已折入）。守门：编译 BUILD SUCCESS、fe-core UT 9/9、fe-connector-api UT 2/2、checkstyle 0、import-gate 净、mutation 5/5 向红。真值闸=大分区 live e2e（DV-019，CI 跳）。**Batch-D 红线**：legacy `MaxComputeScanNode` batch 逻辑须待本 fix 落才可删（读裁剪那半 P1-4 已清，本项为最后前置闸）。commit `ac8f0fc15eb`。**⚠ SUPERSEDED-IN-PART（2026-07-11，reverify M3 = [`FIX-M3-design.md`](./tasks/designs/FIX-M3-design.md)）**：本决策核心（实现 batch SPI 路径、逐字镜像 legacy）**成立不变**；但 ③ 记的「4 闸 **isPruned**」及 impl-review **LP-1**「`!isPruned` ≡ legacy `!= NOT_PRUNED`、等价且略强」判定**为误**——二者对**无谓词分区表**取值相反（`initSelectedPartitions:447` 满 map `isPruned=false` 非哨兵：legacy batch、`!isPruned` 反挡回同步）。已把闸门订正为 `== NOT_PRUNED`（恰恰恢复本决策自身的 legacy-parity 目标），并反转 **TQ-2** 特意留的 pinning 测试 `testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`。docker-hive golden `test_hive_partitions` `(approximate)inputSplitNum` `60→6`（用户签字 SPI 分区数口径）。 | 2026-06-08 | ✅ | | D-034 | — | **P4-T06e FIX-POSTCOMMIT-REFRESH 接受更安全的 post-commit 刷新 swallow、不回退 legacy 传播失败（无产线逻辑改动，NG-8/F15=F21 minor）**：翻闸后 `PluginDrivenInsertExecutor.doAfterCommit()` 用 try/catch 吞 `super.doAfterCommit()`（=`handleRefreshTable`）刷新失败、INSERT 仍报 OK；legacy `MCInsertExecutor` 不 override → 异常传播 → 报 FAILED。按生命周期序 `doBeforeCommit→commit（远端持久）→doAfterCommit`，`handleRefreshTable` 跑时数据已落 ODPS/远端、FE 无法回滚，且只刷 FE 缓存 + 写 external-table refresh editlog（follower 缓存失效提示、非数据真相源）、不碰已提交数据 → 报 FAILED 会诱发重试→**重复写**。**用户定（2026-06-08）：接受 swallow（更安全）+ Javadoc 泛化 + DV 登记，不回退**。改 = **无产线逻辑**：仅 Javadoc(`:164-176`) 从「只讲 JDBC_WRITE」泛化到覆盖 MC connector-transaction 路径（两路径数据均已持久；swallow 最坏只瞬时缓存 stale 自愈；显式注明有意分歧 legacy、引用 [DV-018]）。对抗性安全核查：master 先本地刷新(`RefreshManager:152`)后写 editlog(`:155`)，丢 editlog 仅 follower 缓存暂 stale 自愈、无正确性损失/无主从分裂。守门：checkstyle 0、import-gate 净（注释 only、字节码不变）。真值闸=CI-skip live e2e（MC INSERT 后人为令 refresh 失败→断言报 OK）。commit `1f2e00d3696` | 2026-06-08 | ✅ | | D-033 | — | **P4-T06e FIX-ISKEY-METADATA 连接器局部恢复 isKey=true（无 SPI 变更，NG-6/F3/F10 minor）**：翻闸后 `MaxComputeConnectorMetadata.getTableSchema` 用 5 参 `ConnectorColumn` ctor（isKey 默认 false）→ `DESCRIBE` 显示 Key=NO；legacy `MaxComputeExternalTable.initSchema` 全列 isKey=true。**用户定 Fix（isKey=true 恢复 parity）**。修 = 连接器局部、不动 SPI：抽 `buildColumn(...)` 静态助手用 6 参 ctor 置 isKey=true，data+partition 两 loop 经之；converter 已透传 isKey。**作用域更正**（设计验证 `wa9t0emta`）：`information_schema.columns.COLUMN_KEY` 受 `FrontendServiceImpl:962-965` OlapTable 门控、MC 前后皆空（已 parity，out-of-scope）→ 本修**仅影响 DESCRIBE**。**非纯展示**：isKey 亦喂 `UnequalPredicateInfer:278` + BE slot/column descriptor（非 OLAP 门控），但 legacy 即喂 true → 恢复 production 既有值、零新行为。clean-room 设计验证 `wa9t0emta` 0 mustFix + impl review `wrx0n11ol` 0 mustFix。UT 3/3(+37 collateral)、checkstyle 0、import-gate 净、mutation killed（isKey true→false→Failures 2）。commit `1b44cd4f065` | 2026-06-08 | ✅ | | D-032 | — | **P4-T06e FIX-LIMIT-SPLIT-DEFAULT 连接器局部恢复 limit-split 默认 OFF 三重闸（无 SPI 变更，NG-5/F11；并闭 minors F2/F12）**：翻闸后 `MaxComputeScanPlanProvider.planScan` 丢 legacy 三重闸——`checkOnlyPartitionEquality` 恒 false stub + 从不读 `enable_mc_limit_split_optimization`（默认 false）→ `useLimitOpt = limit>0 && !filter.isPresent()`：无过滤 LIMIT 默认即压成单 row-offset split（语义反转 + 静默无视 session var），分区等值 LIMIT 路径永不触发。**用户定 Fix（恢复三重闸）**。修 = 连接器局部、**不动 SPI**：① 加 hardcode 常量 `ENABLE_MC_LIMIT_SPLIT_OPTIMIZATION`（禁依赖 fe-core `SessionVariable`，同 JDBC 约定）经 `ConnectorSession.getSessionProperties()`（live 由 `from(ctx)`→`VariableMgr.toMap` 填）读 gate(1)；② 实 `checkOnlyPartitionEquality` 遍历 `ConnectorExpression` 树（`ConnectorAnd` 全 conjunct / `ConnectorComparison` EQ 且 col 左 lit 右 / `ConnectorIn` 非 NOT-IN 且 value 为分区列、全 literal），镜像 legacy `checkOnlyPartitionEqualityPredicate`；③ 纯静态 `shouldUseLimitOptimization` 合成 gate(1)&&gate(3)&&gate(2)。默认 OFF=保守回退 legacy。clean-room 设计验证 `w17wzd0el` 0 mustFix + impl review `walkff1vf` 1 mustFix（IN-value 守卫缺杀手测，已补 test+mutation G）收敛。UT 26/26、checkstyle 0、import-gate 净、mutation 8 向红。commit `952b08e0cc8` | 2026-06-08 | ✅ | diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md index a78ec7209323c0..188c48af131e37 100644 --- a/plan-doc/deviations-log.md +++ b/plan-doc/deviations-log.md @@ -47,7 +47,7 @@ | DV-022 | P4-T09 §8：fe-common 去 odps 暴露隐藏传递依赖（依赖卫生，非缺陷）——`odps-sdk-core` 此前**传递**为 fe-common 自身 `DorisHttpException`(io.netty) / `GsonUtilsBase`(com.google.protobuf) 提供 jar；删 odps-sdk-core 后编译暴露缺失，故 fe-common/pom 显式补 `netty-all`+`protobuf-java`（parent dependencyManagement 管版本）。设计 §8 原假设「odps 仅服务 MCUtils」不全 | [Batch-D 设计 §8](./tasks/designs/P4-batchD-maxcompute-removal-design.md) / [D-027] | 2026-06-09 | 🟢 已修正（显式声明，`409300a75b8`）| | DV-021 | P4-T3：Batch-D 删除后 4 条 Tier-3 接受项（minor，legacy 已删故现为既定行为，非丢数据，用户定接受不修）——**GAP3** CREATE DB 非-IFNE 远端已存→本地预抛 `ERR_DB_CREATE_EXISTS`(1007)；**GAP4** DROP TABLE 非-IF-EXISTS+远端缺→通用 `ERR_UNKNOWN_TABLE`(1109)；**GAP9** SHOW PARTITIONS `LIMIT`：sort-then-paginate（vs legacy paginate-then-sort，更合 ORDER-BY-LIMIT）；**GAP10** partitions() TVF schema-分区零实例表→返 0 行（vs legacy 抛，in-code 注释声明 intentional） | [Batch-D 红线](./task-list-batchD-redline-gaps.md) | 2026-06-09 | 🟢 已登记（Tier-3 接受）| | DV-020 | P4-T06e FIX-CAST-PUSHDOWN：getSplits 的 limit-suppress wiring + MC 端到端 CAST-strip 无 fe-core 单测（KNOWN-LIMITATION）+ JDBC applyLimit 同类 under-return（OUT-OF-SCOPE 备查）。**① harness gap**：纯静态 `effectiveSourceLimit(limit,stripped)` 已 UT 2 + mutation 2/2（drop-suppression/always-suppress）向红 pin；连接器 `supportsCastPredicatePushdown=false` 已 UT + mutation(false→true 红) pin；但「`getSplits` 据 `filteredToOriginalIndex!=null` 调 `effectiveSourceLimit`」+「`buildRemainingFilter` 对 MC 真剥 CAST conjunct 并保留 BE-only」的端到端 wiring **无 offline 直测**（构造 `PluginDrivenScanNode` 需 harness、本模块缺，同 [DV-015]）。覆盖经：strip-when-false 是 fe-core 共享逻辑（JDBC false 分支既覆盖）+ 纯 helper UT/mutation + **live e2e 真值闸**（STRING 列存 `"5"/"05"/" 5"`，`WHERE CAST(code AS INT)=5` 返回全部 3 行 / limit-opt ON+CAST+LIMIT 不 under-return；EXPLAIN 证 CAST 谓词不在下推 filter）。**② OUT-OF-SCOPE（Rule 12 surface）**：JDBC 若 session 关 cast-pushdown 且经 `applyLimit` 推 limit，理论同类 under-return；但 MaxCompute 不 override `applyLimit`（no-op）、F9 的 getSplits limit-param 抑制对 MC 完整，JDBC `applyLimit` 路径非本修范围（pre-existing、非 MC），登记备查、待评估。fail-safe：误关下推退化为多读行交 BE（非丢数据） | [FIX-CAST-PUSHDOWN 设计](./tasks/designs/P4-T06e-FIX-CAST-PUSHDOWN-design.md) / [D-036] | 2026-06-08 | 🟢 已登记（helper+capability UT/mutation；wiring 待 live e2e；JDBC applyLimit 备查）| -| DV-019 | P4-T06e FIX-BATCH-MODE-SPLIT 异步 batch wiring + `computeBatchMode` null-guard 无 fe-core 单测（KNOWN-LIMITATION，NG-7）：纯静态四闸 `shouldUseBatchMode` 已 UT 9 + mutation 5/5 向红 pin；但 ① `computeBatchMode` 的 SF-1 `scanProvider != null` null-guard（provider-less full-adopter 防 NPE，跑 dispatch+explain 两路径）与 ② `startSplit` 的 async 分批循环（`getScheduleExecutor` outer/inner CompletableFuture + `SplitAssignment` `needMoreSplit/addToQueue/finishSchedule/setException/isStop` 契约 + init 30s 首-split）+ ③ `numApproximateSplits` 取值——三处 wiring **无 offline 直测**：构造 `PluginDrivenScanNode`（`FileQueryScanNode` 子类）需绕 ctor + stub connector/session/handle/desc/sessionVariable/splitAssignment，本模块无现成轻量 spy/analyze harness（同 [DV-015]/[DV-014] 因）。覆盖经：逐字镜像 legacy `MaxComputeScanNode:214-298`（已验 parity）+ 纯 helper UT/mutation + **大分区 live e2e 真值闸**（EXPLAIN/profile 证 batched/streamed split、规划耗时/内存 ≪ 同步路；阈值边界 `num_partitions_in_batch_mode`=0/大于选中数→回退非-batch；全空选/单分区）。impl-review `wve7y1jst` TQ-1 已据此把测试 javadoc 的「null-provider 已覆盖」声明诚实降级。fail-safe：去 batch 退化为同步 `getSplits`（非丢数据） | [FIX-BATCH-MODE-SPLIT 设计](./tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md) / [D-035] | 2026-06-08 | 🟢 已登记（helper UT+mutation，wiring 待外表 scan harness / live e2e）| +| DV-019 | P4-T06e FIX-BATCH-MODE-SPLIT 异步 batch wiring + `computeBatchMode` null-guard 无 fe-core 单测（KNOWN-LIMITATION，NG-7）：纯静态四闸 `shouldUseBatchMode` 已 UT 9 + mutation 5/5 向红 pin；但 ① `computeBatchMode` 的 SF-1 `scanProvider != null` null-guard（provider-less full-adopter 防 NPE，跑 dispatch+explain 两路径）与 ② `startSplit` 的 async 分批循环（`getScheduleExecutor` outer/inner CompletableFuture + `SplitAssignment` `needMoreSplit/addToQueue/finishSchedule/setException/isStop` 契约 + init 30s 首-split）+ ③ `numApproximateSplits` 取值——三处 wiring **无 offline 直测**：构造 `PluginDrivenScanNode`（`FileQueryScanNode` 子类）需绕 ctor + stub connector/session/handle/desc/sessionVariable/splitAssignment，本模块无现成轻量 spy/analyze harness（同 [DV-015]/[DV-014] 因）。覆盖经：逐字镜像 legacy `MaxComputeScanNode:214-298`（已验 parity）+ 纯 helper UT/mutation + **大分区 live e2e 真值闸**（EXPLAIN/profile 证 batched/streamed split、规划耗时/内存 ≪ 同步路；阈值边界 `num_partitions_in_batch_mode`=0/大于选中数→回退非-batch；全空选/单分区）。impl-review `wve7y1jst` TQ-1 已据此把测试 javadoc 的「null-provider 已覆盖」声明诚实降级。fail-safe：去 batch 退化为同步 `getSplits`（非丢数据）。**⚠ SUPERSEDED-IN-PART（2026-07-11，reverify M3 = [`FIX-M3-design.md`](./tasks/designs/FIX-M3-design.md)）**：本偏差登记的 wiring/null-guard/async-harness test-gap **不变**；仅其关联的 impl-review LP-1「`!isPruned` 等价 `!= NOT_PRUNED`」判定被推翻（详见 [D-035] 批注）——闸门订正 `== NOT_PRUNED`、pinning 测试 `testUnprocessedPruningNeverBatches` 已反转为 `testNoPredicatePartitionedTableBatches`（assertTrue）。 | [FIX-BATCH-MODE-SPLIT 设计](./tasks/designs/P4-T06e-FIX-BATCH-MODE-SPLIT-design.md) / [D-035] | 2026-06-08 | 🟢 已登记（helper UT+mutation，wiring 待外表 scan harness / live e2e）| | DV-018 | P4-T06e FIX-POSTCOMMIT-REFRESH cutover post-commit 刷新 swallow 有意分歧于 legacy（无产线逻辑改动，NG-8/F15=F21 minor，regression=no）：`PluginDrivenInsertExecutor.doAfterCommit()` 用 try/catch 吞 `super.doAfterCommit()`（=`handleRefreshTable`）刷新失败、INSERT 报 OK；legacy `MCInsertExecutor` 不 override → 异常传播 → 报 FAILED。**cutover 更安全**：按生命周期序数据已落 ODPS/远端、FE 无法回滚，`handleRefreshTable` 只刷 FE 缓存 + 写 external-table refresh editlog（follower 失效提示、非数据真相源）、不碰已提交数据 → 报 FAILED 诱发重试→重复写。**用户定（2026-06-08）接受 + Javadoc 泛化（[D-034]）、不回退**。改 = 仅 Javadoc(`:164-176`) 从「只讲 JDBC_WRITE」泛化到覆盖 MC connector-transaction 路径（两路径数据均已持久；swallow 最坏只瞬时缓存 stale 自愈；显式注明分歧 legacy）。对抗性安全核查：master 先本地刷新(`RefreshManager:152`)后写 editlog(`:155`)，丢 editlog 仅 follower 缓存暂 stale 自愈、无正确性损失/无主从分裂。swallow 路径无新增 UT（注释 only、无可 pin 逻辑变化；异常吞行为 offline 直测受同类 harness 缺位限制，同 [DV-015]）；真值闸=CI-skip live e2e（MC INSERT 后人为令 refresh 失败→断言报 OK + warn）。守门 checkstyle 0、import-gate 净 | [FIX-POSTCOMMIT-REFRESH 设计](./tasks/designs/P4-T06e-FIX-POSTCOMMIT-REFRESH-design.md) / [D-034] | 2026-06-08 | 🟢 已登记（无逻辑改动，行为收敛接受；live 真值闸待跑）| | DV-017 | P4-T06e FIX-ISKEY-METADATA `getTableSchema→buildColumn` wiring 无连接器内单测（KNOWN-LIMITATION）：`buildColumn` 助手 isKey=true 不变式已 UT+mutation pin，但两 `getTableSchema` 调用点经 `buildColumn` 的 wiring 无 offline 测——`getTableSchema` deref live `com.aliyun.odps.Table`（唯一 ctor package-private）、模块无 Mockito（同 [DV-014]/[DV-015]/[DV-016] 类）；唯一 offline 变通=`com.aliyun.odps` 包内 fixture 子类 override `getSchema()`，repo 无先例（sibling `getColumnHandles` 同样未测）。绕过 `buildColumn`（回退 5 参 ctor）的回归仅由 CI-skip live e2e `DESCRIBE ` 显 Key=YES 捕获（load-bearing gate）。**作用域注**：`information_schema.columns.COLUMN_KEY` 受 `FrontendServiceImpl:962-965` OlapTable 门控、MC 前后皆空、已 parity、out-of-scope（不可断言其变非空）；isKey 非纯展示（亦喂 `UnequalPredicateInfer`/BE descriptor），但 legacy 即喂 true → 本修恢复既有值 | [FIX-ISKEY-METADATA 设计](./tasks/designs/P4-T06e-FIX-ISKEY-METADATA-design.md) / [D-033] | 2026-06-08 | 🟢 已登记（helper UT+mutation，wiring 待 live DESCRIBE）| | DV-016 | P4-T06e FIX-LIMIT-SPLIT-DEFAULT 三点（均 opt-in 默认 OFF、非丢行/非回归）：① **CAST-unwrap 致 limit-opt 资格略宽于 legacy**——converter `convert(CastExpr)→convert(child)` 在所有位置剥 CAST（左列/右 literal/IN 元素），故 `CAST(partcol AS T)=lit`、`partcol=CAST(lit AS T)`、`partcol IN (CAST(lit,…))` 经 `checkOnlyPartitionEquality` 判资格，legacy 见原始 `CastExpr` 子节点 instanceof 失败→false；② **嵌套-AND-作单 conjunct 略宽**——converter `flattenAnd` 把单 conjunct `(pt=1 AND region=cn)` 摊平成 flat `ConnectorAnd`→资格，legacy 见 `CompoundPredicate` conjunct→false（与①同安全类，且 conjunct 拆分通常上游已分）；③ **`LIMIT 0` 路径差**——本 fix `limit<=0` 拒 limit-opt 走标准多 split 路，legacy `hasLimit()`(`limit>-1`) 走 limit-opt 路；两者皆 0 行、且 `LIMIT 0` 被 Nereids 折成 EmptySet 不可达。①②均纯分区、correctness-safe（裁剪 Nereids `SelectedPartitions` 同算 + 转换后 `filterPredicate` 仍下推 read session 作 backstop，`:191/:208/:353`；LIMIT 无 ORDER BY 无序）。**另**：planScan 两行 wiring（`isLimitOptEnabled(session.getSessionProperties())` + `shouldUseLimitOptimization(...)` 收 live filter/partitionColumnNames）无连接器内单测——`planScan` 需 live odps `Table`、模块无 fe-core/Mockito（同 [DV-014]/[DV-015] 因）；纯 helper 全 UT(26)+mutation(8 向红) pin，wiring 半由 CI-skip live E2E 守。**附**：本 fix 实 `checkOnlyPartitionEquality` 同闭 F2/F12（旧恒 false stub minors）| [FIX-LIMIT-SPLIT-DEFAULT 设计](./tasks/designs/P4-T06e-FIX-LIMIT-SPLIT-DEFAULT-design.md) / [D-032] | 2026-06-08 | 🟢 已登记（opt-in 非回归 + 逻辑 UT/mutation，wiring 待 live E2E）| diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 7bca81c989d127..b537a66342edf3 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -31,7 +31,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | | 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | | 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ✅ | ✅ | ✅ | ✅ | -| 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ⬜ | ⬜ | ⬜ | ⬜ | +| 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ✅ | ✅ | ✅ | ✅ | | 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ✅ | ✅ | ✅ | ✅ | | 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | | 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ✅ | ✅ | ✅ | ✅ | @@ -125,7 +125,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:单测断言分区表 `supportsBatchScan`;大分区异步 split live e2e gated。 ### M3 — MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` · reverify §3 M3 -- [ ] **M3** +- [x] **M3** · DONE `6963de4124f`（设计 `designs/FIX-M3-design.md`；设计红队 `wf_811e6242-d8b` 3 lens：BLAST-RADIUS SOUND / LEGACY-PARITY SOUND_WITH_CHANGES / COMPLETENESS SOUND_WITH_CHANGES，1 blocker+1 major 已解）。`:1136` `!isPruned`→`== NOT_PRUNED`（对齐 legacy `MaxComputeScanNode:227` + sibling `displayPartitionCounts:298`；git `1da88365e85^` 取证 + 全 producer 枚举证闭合）。**反转** pinning 测试 `testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`（assertTrue）。**解 M2 的 BATCH-UNPRUNED-SYNC**。**登记 supersession**：D-035/DV-019 的 LP-1「`!isPruned` 等价」判定被推翻（已补 SUPERSEDED 批注）。**docker-hive golden**`test_hive_partitions:200``(approximate)inputSplitNum` `60→6`（**用户 2026-07-11 签字：SPI 统一分区数口径**；batch 模式 `selectedSplitNum=numApproximateSplits=分区数`；对齐 MaxCompute/Trino）。`PluginDrivenScanNodeBatchModeTest` 12/12 绿。e2e live-gated（docker-hive，本地不可跑；sweep 确认全 regression 仅此 1 处 `(approximate)` 断言受影响）。 - **现码**:`PluginDrivenScanNode.shouldUseBatchMode:1136` `!selectedPartitions.isPruned`;应 `== SelectedPartitions.NOT_PRUNED`。`ExternalTable.initSelectedPartitions:447` 无谓词返 `isPruned=false` 非哨兵。行内 `:1129-1132` 注释误称 parity。仅 MC opt-in(`MaxComputeScanPlanProvider.supportsBatchScan:254`)。 - **Fix**:`:1136` 改回 `== NOT_PRUNED` 早退语义;订正 `:1129-1132` 注释。 - **Files**:`fe-core/.../PluginDrivenScanNode.java`。 @@ -248,3 +248,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **M2** DONE `702153885ab`(code) — hive 补 batch 通路:`supportsBatchScan`(分区∧非事务)+`planScanForPartitionBatch`(scope 到 batch,防重复 split——hive `planScan` 非 partition-scoped 故须额外 override)。**登记 BATCH-ACID-SYNC(永久)+BATCH-UNPRUNED-SYNC(M3 解)**。4/4+hive 模块 284/284 绿。e2e live-gated。 - **批次 1 最终对抗复核(5 per-fix skeptic + 1 cross-cut,`wf_542c60b9-001`)= M5/M6/M7/M2/cross-cut CLEAN；M4 命中 1 medium 缺陷已修**:M4 TTL 默认误抄 hive 文件缓存 knob(86400s)而非旧版 MaxCompute 分区缓存 knob(`external_cache_refresh_time_minutes*60`=600s),144x 过陈;经核旧版删除 commit `1da88365e85^`+Config 默认证实,已改 600s(**`fca288424fc`**,9/9 仍绿)。capacity 10000 巧合正确(旧版 `max_hive_partition_table_cache_num`)。 - **⭐ 批次 1(M5/M7/M6/M4/M2)全部 DONE + 最终复核 CLEAN。** 5 连接器局部修全绿(iceberg 3 + mc 1 + hive 1),各配 RED-able 单测 + 独立 code/doc commit。**e2e 全 live-gated**(equality-delete 统计/vended-region/s3tables 默认链/mc 分区缓存往返/hive 大分区异步),须真集群回归(memory `hms-iceberg-delegation-needs-e2e`)。**下一步=批次 2(M3→M1,fe-core 通用节点,blast radius 较大)**;M3 顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余。 + +--- + +**⭐ 批次 2(M3→M1,fe-core 通用节点)进行中** + +- **M3** DONE `6963de4124f`(code) — batch 闸门 `!isPruned`→`== NOT_PRUNED`:无谓词大分区表(MaxCompute + 翻闸 hive)恢复异步 batch split(legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证;git `1da88365e85^` 取证 + 全 producer 枚举证闭合)。**顺带解 M2 的 BATCH-UNPRUNED-SYNC**。设计红队 `wf_811e6242-d8b`(3 lens:BLAST-RADIUS SOUND / LEGACY-PARITY SOUND_WITH_CHANGES / COMPLETENESS SOUND_WITH_CHANGES)命中 1 blocker(docker-hive golden 未 reconcile)+ 1 major(overturn 前次签字 LP-1/D-035「等价」未登记)均已解:**反转** pinning 测试(`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`,assertTrue)、**登记 supersession**(D-035/DV-019 补 SUPERSEDED 批注)、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`(**用户 2026-07-11 签字:SPI 统一分区数口径**,非 hive 专属 split-count 估算;对齐 MaxCompute/Trino「引擎层统一报分片」)。`PluginDrivenScanNodeBatchModeTest` **12/12** 绿、fe-core test-compile BUILD SUCCESS 0 checkstyle。**e2e live-gated**(docker-hive `(approximate)inputSplitNum=6` + MaxCompute 无谓词 ≥阈值分区表进 batch;sweep 确认全 regression 仅此 1 处 `(approximate)` 断言受影响,maxcompute p2 只断言 `partition=N/M`/结果不受影响)。**⚠ 构建坑**:本轮 UT 一度被并行 session 的 `be-java-extensions package -am -T 1C` 构建污染共享 target 报「cannot access 生成类」假失败(非本码);待其结束后干净重跑 12/12(memory `concurrent-sessions-shared-worktree-hazard`)。 +- **下一步 = M1**(TABLESAMPLE 插件路径静默全表扫;translator 转发 `setTableSample` + `PluginDrivenScanNode.getSplits` 通用按 split 大小采样,仿 legacy `HiveScanNode:448-458` 但 connector-agnostic)。 diff --git a/plan-doc/tasks/designs/FIX-M3-design.md b/plan-doc/tasks/designs/FIX-M3-design.md new file mode 100644 index 00000000000000..2668c25a179c8b --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M3-design.md @@ -0,0 +1,146 @@ +# FIX-M3 — batch 闸门 `!isPruned` → `== NOT_PRUNED`（无谓词大分区表恢复异步 batch split） + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M3。 +> 批次 2（fe-core 通用节点）。**同时解 M2 登记的 BATCH-UNPRUNED-SYNC 偏差**（`FIX-M2-design.md` §登记偏差）。 +> HEAD 复核（本轮，primary source）：legacy 从 git 历史取证、`SelectedPartitions` 全 producer 枚举、sibling gate 对照。 + +## Problem + +翻闸后（`"hms"` ∈ `SPI_READY_TYPES`）分区外表走通用 `PluginDrivenScanNode`。其 batch/async split 通路由纯静态门 +`shouldUseBatchMode`（`:1134`）决定。该门第一条守卫（`:1136`）用 `!selectedPartitions.isPruned`，与 legacy +`MaxComputeScanNode.isBatchMode` 的 `selectedPartitions != NOT_PRUNED` **语义不等价**： + +- **无谓词分区表**（`SELECT col FROM t`，无 WHERE → `PruneFileScanPartition` 不触发）： + `ExternalTable.initSelectedPartitions:447` 返 `new SelectedPartitions(size, fullMap, isPruned=false)`—— + 一个**非 `NOT_PRUNED` 哨兵**、`isPruned=false`、**满** map 的对象。 + - legacy：`!= NOT_PRUNED` → true → batch 合格（若 `size >= num_partitions_in_batch_mode`，默认 1024）。 + - 现 SPI：`!isPruned` = `!false` = true → 守卫**返回 false → 不 batch**。 +- 后果：≥1024 分区的无谓词全表扫**同步**一次性把所有分区所有文件 split 物化进 FE 堆 → 规划延迟 + 堆压 + （legacy 用异步 streaming batch）。结果正确（fail-safe），是**性能回归**。 +- 范围：现只影响 **opt-in `supportsBatchScan` 的连接器**——MaxCompute（`getFileNum()>0`）与**翻闸后的 hive** + （M2 刚加的 `supportsBatchScan`）。iceberg 走 streaming flavor（不看 `isPruned`），不受影响。 +- `:1129-1132` 行内 javadoc 自称「`!isPruned` subsumes BOTH legacy gates…marginally stronger」是**错的**—— + 它只对非分区（NOT_PRUNED）情形成立，恰恰漏了上面的无谓词分区情形。同文件同作者的 sibling + `displayPartitionCounts:298` 却**正确**用 `== NOT_PRUNED`（其 javadoc `:288-296` 明确解释此正是「无谓词分区表 + `isPruned=false` 但满 map、须当已处理」的情形）——证明该分歧是**误用**而非有意设计。 + +## Root Cause + +单点：`PluginDrivenScanNode.shouldUseBatchMode:1136` 用 `!selectedPartitions.isPruned` 作「非哨兵」判据， +但 `isPruned` 与「是否 NOT_PRUNED 哨兵」并不同义——无谓词分区表二者取值相反。 + +**Legacy 权威取证**（git，删除 commit `1da88365e85`，故取 `1da88365e85^`） +`.../maxcompute/source/MaxComputeScanNode.java:215-229`： +```java +public boolean isBatchMode() { + if (table.getPartitionColumns().isEmpty()) return false; // 非分区 → false + if (desc.getSlots().isEmpty() || odpsTable.getFileNum() <= 0) return false; // 无 slot / 无文件 → false + int numPartitions = sessionVariable.getNumPartitionsInBatchMode(); + return numPartitions > 0 + && selectedPartitions != SelectedPartitions.NOT_PRUNED // ← 权威门：!= NOT_PRUNED，非 isPruned + && selectedPartitions.selectedPartitions.size() >= numPartitions; +} +``` +- `getPartitionColumns().isEmpty()`（非分区）→ `initSelectedPartitions` 返 NOT_PRUNED → 被 `!= NOT_PRUNED` 门吸收 + （故 SPI 折叠这两门到一条 `== NOT_PRUNED` **确实无损**，此点原 javadoc 说对了）； +- `desc.getSlots().isEmpty()` → SPI `hasSlots`；`getFileNum()<=0` → 折进连接器 `supportsBatchScan`； +- `!= NOT_PRUNED` → **应落在 `:1136`，现被误写成 `!isPruned`**。 + +**Producer 枚举（证明修复闭合、无隐藏第三态）**——全 fe-core `new SelectedPartitions(...)`： +| 处 | isPruned | 是哨兵? | 何时 | +|----|----------|---------|------| +| `LogicalFileScan:275` NOT_PRUNED | false | **是** | 非分区 / 未剪枝 | +| `ExternalTable:447` | false | 否 | **无谓词分区表（满 map）← 到达本门的唯一分歧态** | +| `HMSExternalTable:485` | false | 否 | `initHudiSelectedPartitions`（**hudi-only**，经 translator:938 喂 `HudiScanNode`，**不经本门**）——与本修无关（红队更正：非「旧 HMS 死路」） | +| `PruneFileScanPartition:68` | true | 否 | 剪到空 | +| `PruneFileScanPartition:108` | true | 否 | 剪枝非空 | + +→ 到达 `PluginDrivenScanNode` 本门、`isPruned=false` 且非哨兵的**仅** `ExternalTable:447`「无谓词分区满 map」一态 +（`HMSExternalTable:485` 走 `HudiScanNode`，不经本门）。故把 `!isPruned` 换成 `== NOT_PRUNED` 后， +新增 batch 的**恰**是该态（+ 已一致的 `isPruned=true` 剪枝态），**无**别的对象被这门影响。 + +## Design（单点 fe-core；connector-agnostic，无源判别） + +1. `PluginDrivenScanNode.shouldUseBatchMode:1136`： + ```java + if (selectedPartitions == null || selectedPartitions == SelectedPartitions.NOT_PRUNED) { + return false; + } + ``` + 对齐 legacy `!= NOT_PRUNED` 与同文件 sibling `displayPartitionCounts:298`。守卫仍 connector-agnostic + （通用哨兵比较，非源名分支）；仅 opt-in `supportsBatchScan` 的连接器实际受益。 +2. 订正 javadoc： + - `:1122` summary bullet：`!isPruned` → `== NOT_PRUNED` 表述； + - `:1129-1132` 段落重写——`== NOT_PRUNED` 折叠「非分区 + 未 Nereids 剪枝的空哨兵」两态；无谓词分区满 map（非哨兵） + **落到 size≥阈值 检查**（对齐 legacy），指向 sibling `displayPartitionCounts` 的同款解释。 +3. **无**其它改动：`computeBatchMode` 流式 flavor、`numApproximateSplits`、async pump 均不动。 + +**解 M2 BATCH-UNPRUNED-SYNC**：M2 已 override hive `supportsBatchScan`（分区∧非事务），本修使无谓词分区 hive +表也进 batch（size≥阈值时），恰是 M2 设计 §登记偏差点名「由 M3 解」的一条。 + +## Risk + +- **结果安全**：batch 仅是 split 生成策略（异步 streaming vs 同步物化），产出 split 集/行数完全一致；不改查询结果。 +- **Blast radius**：MaxCompute + 翻闸 hive 的**无谓词 ≥阈值 分区表**从同步转异步 batch。iceberg（streaming flavor）、 + 非分区表、有谓词剪枝表、连接器未 opt-in `supportsBatchScan` 者——**行为不变**。 +- **`numApproximateSplits`**：新 batch 态走 `:1160` 返 `selectedPartitions.size()`（满分区数，非负），对齐 legacy + `MaxComputeScanNode.numApproximateSplits:233`（同 `selectedPartitions.size()`）。 +- **EXPLAIN 回归**：batch 表 EXPLAIN 可能出现 batch/approximate 标记差异；无谓词大分区 MaxCompute/hive 用例 + （`test_max_compute_partition_prune.groovy`、`test_hive_partitions.groovy` 设 `num_partitions_in_batch_mode`）须 e2e + 真集群回归（本地无法跑，live-gated）。 +- **铁律**：无新 `if(hive/iceberg)`/`instanceof`/属性解析；单点哨兵比较，通用节点 connector-agnostic。 + +## Test Plan + +### Unit（`PluginDrivenScanNodeBatchModeTest`，fe-core 有 Mockito） +- **反转** `testUnprocessedPruningNeverBatches`（`:86-95`）→ 更名 `testUnprocessedPruningStillBatches`： + `new SelectedPartitions(THRESHOLD, items(THRESHOLD), false)`（= `initSelectedPartitions:447` 对无谓词分区表的产物） + 在 `hasSlots=true, supportsBatchScan=true, threshold=THRESHOLD` 下断言 **`assertTrue`**。 + 注释改为编码 WHY：无谓词分区全扫是**最需**异步 batch 的态（对齐 legacy `!= NOT_PRUNED`）；此前 `assertFalse` 编码的 + 正是 M3 回归。**RED-able**：现码 `!isPruned` 返 false → 该 `assertTrue` 现会失败。 +- `testNotPrunedNeverBatches`（NOT_PRUNED 哨兵，空 map）、`testNullSelectionNeverBatches` 保持 `assertFalse` + （`== NOT_PRUNED` 守卫仍挡哨兵与 null）——守住「非分区 / 空哨兵不 batch」。 +- 其余门测（hasSlots / supportsBatchScan / threshold / 边界）不变。 + +### E2E（live-gated，须真集群；memory `hms-iceberg-delegation-needs-e2e`） +- MaxCompute ≥阈值 无谓词分区表 `SELECT col FROM odps_t`：断言进 batch（EXPLAIN/规划无 OOM）； +- 翻闸 hive ≥阈值 无谓词分区表：同断言（BATCH-UNPRUNED-SYNC 已解）； +- 二者结果与同步路径逐行一致（batch 不改行）。 +- 登记为 gated（本模块无 live harness，DV-019 同类）。 + +## 设计红队（`wf_811e6242-d8b`，3 lens 对抗）+ 最终确认 + +**verdict**：BLAST-RADIUS = **SOUND**（无遗漏 producer、行不变、仅 hive+MaxCompute 无谓词 ≥阈值 分区表受影响）； +LEGACY-PARITY = **SOUND_WITH_CHANGES**（MaxCompute 核心 parity **精确**、一行修无须改动，仅 doc 措辞更正）； +COMPLETENESS = **SOUND_WITH_CHANGES**（命中 1 blocker + 1 major，均已解，见下）。核心一行修**无争议**：restores legacy +`!= NOT_PRUNED`，producer 枚举证闭合。 + +**折入的更正：** +1. **（已改上表）** `HMSExternalTable:485` 是 hudi initializer 走 `HudiScanNode`，非「旧 HMS 死路」——不到达本门。 +2. **hive parity 仅「方向性」非「精确」**：legacy `HiveScanNode.isBatchMode`（git `785ae1d7dda:290-306`）**无** NOT_PRUNED/isPruned + 门、阈值用 `>= 0`（vs SPI `> 0`）、`numApproximateSplits = numSplitsPerPartition × partitions`（vs SPI = 分区数）。 + 这两处差异**均 pre-existing**（本一行修不碰阈值算子、不碰 numApproximateSplits），故不驳斥本修；但本 doc 的 hive parity 叙述 + 限定为「恢复无谓词场景的 batch **方向**」，非「与 legacy hive 全等」。**MaxCompute** 才是本修的精确 parity 目标 + （legacy `MaxComputeScanNode` 同样 `!= NOT_PRUNED` + `numApproximateSplits = 分区数`）。 + +**blocker（docker-hive golden，已解）**：`test_hive_partitions.groovy:200` 是 docker-gated（`enableHiveTest`）checked-in 断言 +`(approximate)inputSplitNum=60`，**非**「deferred live-gated e2e」。该表 6 分区、`num_partitions_in_batch_mode=1` 强制 batch、 +**期望** `(approximate)` 前缀——即它本身就是「无谓词全表扫应 batch」的旁证。翻闸+M2 后闸门 `!isPruned` 挡掉 batch → 前缀消失 → +**已 red**。本修恢复 batch → 前缀回；但 batch 模式 `selectedSplitNum = numApproximateSplits()`（`FileQueryScanNode:383`）= +分区数 `6`（非 legacy 的 `10×6=60`）。**用户 2026-07-11 签字：采用 SPI 统一口径（近似分片数=分区数）**，golden `60→6` +（对齐 MaxCompute、Trino「引擎层统一报分片」思路；不给 hive 补 split-count 估算）。**docker-gated，本地不可跑 → 源 golden 已改， +真值须真集群回归**（sweep 确认全 regression-test 仅此 1 处 `(approximate)` 断言受影响；maxcompute p2 只断言 `partition=N/M`/结果，不受影响）。 + +**major（supersession，已登记非静默）**：`!isPruned` 门 + 其 pinning 测试同随 commit `1da88365e85` 引入； +**LP-1**（P4-T06e impl-review，nit）曾判 `!isPruned` vs `!= NOT_PRUNED`「等价且略强」、**TQ-2** 特意留 +`testUnprocessedPruningNeverBatches` 钉住 `!isPruned`，签为 **D-035 / DV-019**。本修**证明该「等价」为假**（无谓词分区表二者相反）， +反转该测试为 `testNoPredicatePartitionedTableBatches`（assertTrue）。**已在 `decisions-log.md` D-035 / `deviations-log.md` DV-019 +补 SUPERSEDED 批注**（非静默；对齐 M5「推翻先前签字」的登记纪律）。 + +## 最终改动清单（本条落地） +- `PluginDrivenScanNode.java:1136` `!isPruned` → `== NOT_PRUNED` + javadoc（summary bullet + 段落）重写。 +- `PluginDrivenScanNodeBatchModeTest.java`：`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches` + （assertFalse→assertTrue，注释编码 WHY + supersession）；订正 `testNotPrunedNeverBatches` 注释（`!isPruned`→`== NOT_PRUNED`） + + class-javadoc「must be pruned」措辞。 +- `test_hive_partitions.groovy:200` golden `60→6`（+ 注释解释 SPI 口径）。 +- `decisions-log.md` D-035 / `deviations-log.md` DV-019：补 SUPERSEDED 批注。 From 4dbb8e02056186469da0fff393e92f9855257420 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 11:15:22 +0800 Subject: [PATCH 233/334] [doc](regression) Document Hive regression local environment ### What problem does this PR solve? Issue Number: None Related PR: None Problem Summary: Document the local Docker components and regression configuration needed to run external_table_p0/hive suites, including the implicit MySQL dependency from the default JuiceFS metadata URI and the observed local container state. ### Release note None ### Check List (For Author) - Test: No need to test (documentation-only change) - Behavior changed: No - Does this need documentation: No --- .../thirdparties/hive-regression-local-env.md | 162 ++++++++++++++++++ 1 file changed, 162 insertions(+) create mode 100644 docker/thirdparties/hive-regression-local-env.md diff --git a/docker/thirdparties/hive-regression-local-env.md b/docker/thirdparties/hive-regression-local-env.md new file mode 100644 index 00000000000000..62e82997299b60 --- /dev/null +++ b/docker/thirdparties/hive-regression-local-env.md @@ -0,0 +1,162 @@ +# Hive Regression Local Environment + +This document records the local Docker and regression configuration needed to run: + +```bash +regression-test/suites/external_table_p0/hive/ +``` + +## Required Thirdparty Components + +Run Hive2 and Hive3 through the thirdparty launcher: + +```bash +./docker/thirdparties/run-thirdparties-docker.sh -c hive2,hive3 --hive-mode refresh +``` + +The Hive startup script may also start `mysql` automatically. Both +`docker-compose/hive/hive-2x_settings.env` and +`docker-compose/hive/hive-3x_settings.env` default `JFS_CLUSTER_META` to: + +```bash +mysql://root:123456@(127.0.0.1:3316)/juicefs_meta +``` + +Because of that setting, `run-thirdparties-docker.sh` treats local MySQL 5.7 as +an implicit dependency for Hive. + +Each Hive version starts its own compose services: + +- Hive2: `hive2-server`, `hive2-metastore`, `hadoop2-namenode`, + `hadoop2-datanode`, `hive2-metastore-postgresql` +- Hive3: `hive3-server`, `hive3-metastore`, `hadoop3-namenode`, + `hadoop3-datanode`, `hive3-metastore-postgresql` +- MySQL 5.7: required by the default JuiceFS metadata URI when using the local + `127.0.0.1:3316` endpoint + +The Hive p0 directory does not require these components by default: + +- `pg` +- `iceberg` +- `minio` +- `hudi` +- `kerberos` +- `ranger` + +`test_external_catalog_hive.groovy` has a Ranger-specific branch guarded by +`enableRangerTest`; if this option is not set to `true`, that branch is skipped. + +## Current Running Containers Observed + +The following relevant containers were running when checked with +`sudo docker ps`: + +- Hive3 stack: `hive3-server`, `hive3-metastore`, `hadoop3-datanode`, + `hadoop3-namenode`, `hive3-metastore-postgresql`; all healthy +- Hive2 stack: `hive2-server`, `hive2-metastore`, `hadoop2-datanode`, + `hadoop2-namenode`, `hive2-metastore-postgresql`; all healthy +- MySQL 5.7: `mysql-doris--syt--mysql_57-1`, healthy, published as + `127.0.0.1:3316 -> 3306` + +Other Doris thirdparty containers were also running, but they are not required +for `external_table_p0/hive/`: + +- PostgreSQL 14 on `5442` +- Iceberg REST, Spark, Postgres, and MinIO +- Oracle on `1521` +- ClickHouse on `8123` +- SQLServer on `1433` +- OceanBase on `2881` + +The observed Hive2/Hive3 and MySQL compose projects came from another checkout +under `/mnt/disk2/suyiteng/doris`, not from this worktree. PostgreSQL and +Iceberg compose projects came from this worktree. + +## Regression Configuration + +The Hive p0 suite directly reads these keys from +`regression-test/conf/regression-conf.groovy`: + +```groovy +enableHiveTest +externalEnvIp +hive2HmsPort +hive2HdfsPort +hive3HmsPort +hive3HdfsPort +hdfsUser +``` + +It also uses framework helpers backed by: + +```groovy +brokerName +hdfsPasswd +hdfsFs +``` + +For local Hive2/Hive3 Docker with default ports, the relevant values are: + +```groovy +enableHiveTest = true + +hive2HmsPort = 9083 +hive2HdfsPort = 8020 +hive2ServerPort = 10000 +hive2PgPort = 5432 + +hive3HmsPort = 9383 +hive3HdfsPort = 8320 +hive3ServerPort = 13000 +hive3PgPort = 5732 + +hdfsUser = "doris-test" +brokerName = "broker_name" +hdfsPasswd = "" +``` + +`externalEnvIp` must be the address that Doris BE can use to reach the Hive +Docker services. For a fully local setup this is usually: + +```groovy +externalEnvIp = "127.0.0.1" +``` + +If BE runs in a context where `127.0.0.1` is not the Docker host, use the host +IP selected by `run-thirdparties-docker.sh` through `IP_HOST`. + +## Config Gaps Noted In regression-conf.groovy.bak + +`regression-test/conf/regression-conf.groovy.bak` already contains the core Hive +keys listed above. The notable follow-ups are: + +- `externalEnvIp` was set to `172.20.32.136`; verify this is reachable from BE, + or replace it with `127.0.0.1` for local host-mode Docker. +- FE/JDBC addresses in the file used `9033`, `9022`, and `8033`; align these + with the current Doris worktree cluster ports before running regression. +- `enableRangerTest` is not present. This is acceptable for the Hive p0 suite + because the Ranger-specific branch is guarded and skipped unless explicitly + enabled. + +## Thirdparty Script Settings + +Before starting Hive through `run-thirdparties-docker.sh`, set a unique +`CONTAINER_UID` in: + +```bash +docker/thirdparties/custom_settings.env +``` + +The default `CONTAINER_UID="doris--"` is rejected by the startup script. + +Hive baseline restore also depends on these settings: + +```bash +s3Endpoint +s3BucketName +HIVE_BASELINE_VERSION +HIVE_BASELINE_TARBALL_CACHE +``` + +If baseline download is unavailable, place the matching tarball manually under +`HIVE_BASELINE_TARBALL_CACHE`. From 99e0b4a6ade7c3d575c78f63e84c4fa04e734ce7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 12:01:12 +0800 Subject: [PATCH 234/334] [fix](catalog) hive: pin HiveConf classloader to plugin loader (fix MetaStoreFilterHook split-brain) HMS catalogs now route to the new HiveConnector (hms is in SPI_READY_TYPES), which builds its metastore client via HmsConfHelper.createHiveConf -> new HiveConf(). That constructor captures the calling thread's TCCL into Configuration.classLoader; on the FE query thread that is the fe-core app loader, which carries fe-core's own copy of the hive-metastore classes. HiveMetaStoreClient later copies this conf (new Configuration copies the classLoader field) and, in loadFilterHooks(), resolves metastore.filter.hook (default DefaultMetaStoreFilterHookImpl) via Configuration.getClass using that cached classLoader. The app-loader copy is not assignable to the plugin loader's MetaStoreFilterHook interface ("class DefaultMetaStoreFilterHookImpl not MetaStoreFilterHook"), so client creation fails before any metastore RPC and every hive-on-HMS read surfaces "Unknown database". The 92004ef1d0d doAs pin only fixes classes resolved via the LIVE TCCL (SecurityUtil.); the filter-hook path uses the conf's cached classLoader, so it must be pinned at HiveConf construction. Mirror paimon's already-fixed PaimonCatalogFactory.assembleHiveConf: hiveConf.setClassLoader(plugin loader). Co-Authored-By: Claude Opus 4.8 (1M context) --- .../apache/doris/connector/hms/HmsConfHelper.java | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java index 0a6f7df090a737..333fa7f56b55d5 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java @@ -46,6 +46,18 @@ private HmsConfHelper() { */ public static HiveConf createHiveConf(Map properties) { HiveConf hiveConf = new HiveConf(); + // Pin the conf classloader to the plugin loader, mirroring PaimonCatalogFactory.assembleHiveConf. + // HiveMetaStoreClient.loadFilterHooks resolves metastore.filter.hook via Configuration.getClass, which + // uses the conf's OWN classLoader field (= the thread-context CL captured at new HiveConf() above), NOT + // the live TCCL. createHiveConf runs in the ThriftHmsClient constructor on the FE query thread, BEFORE + // ThriftHmsClient.doAs pins the TCCL, so that captured CL is still the parent 'app' loader (fe-core's own + // hive-metastore copy). HiveMetaStoreClient later copies this conf (new Configuration(hiveConf) copies the + // classLoader field), so under child-first plugin loading it resolves DefaultMetaStoreFilterHookImpl from + // the parent while MetaStoreFilterHook is child-loaded, giving "class DefaultMetaStoreFilterHookImpl not + // MetaStoreFilterHook" and failing client creation before any metastore RPC. doAs pins the LIVE TCCL + // (fixes SecurityUtil.) but cannot fix this conf-cached CL. Pinning here keeps the whole + // hive-metastore class graph in one loader. + hiveConf.setClassLoader(HmsConfHelper.class.getClassLoader()); for (Map.Entry entry : properties.entrySet()) { hiveConf.set(entry.getKey(), entry.getValue()); } From 17b432dc1e10d7dee9f8f76229b07e1c61cec01c Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 12:23:06 +0800 Subject: [PATCH 235/334] [fix](catalog) TABLESAMPLE: forward to plugin scan + connector opt-in sampling (fixes silent full scan) Post-cutover, TABLESAMPLE was silently dropped for plugin-driven external tables: visitPhysicalFileScan's plugin arm forwarded selectedPartitions but not fileScan.getTableSample() (only the now-dead legacy hive arm did), so SELECT ... TABLESAMPLE(N) scanned the FULL table instead of a sample. Only hive ever sampled (via HiveScanNode.selectFiles), so this is a hive-only regression; the other connectors never sampled. The generic node cannot sample by Split.getLength() unconditionally: that length is -1 for MaxCompute's default byte-size strategy and Paimon's JNI ranges, and a row count (not bytes) for MaxCompute row_offset -- feeding those into size-weighted sampling returns garbage. So sampling is gated on a connector capability (mirrors supportsBatchScan / Trino's applySample), not a source-type branch. - fe-connector-api: add ConnectorScanPlanProvider.supportsTableSample() default false. - fe-connector-hive: HiveScanPlanProvider.supportsTableSample() -> true (its ranges carry positive byte lengths); drop the stale "No table sampling" javadoc. - PhysicalPlanTranslator: forward TABLESAMPLE in the plugin arm (mirrors legacy hive). - PluginDrivenScanNode: apply a new pure-static sampleSplits (port of HiveScanNode.selectFiles over the generic Split.getLength()) only when the connector opted in; a non-supporting connector no-ops with a WARN (not silently). Two gated completeness guards: suppress COUNT(*) pushdown while sampling (else a connector that collapses count-eligible splits into one precomputed-full-count range ignores the sample), and disable batch mode while sampling (the batch/startSplit path never samples; M3 just widened batch to no-predicate large partition tables). fe-core UT PluginDrivenScanNodeTableSampleTest 6/6, BatchMode 12/12; hive connector 285/285; 0 checkstyle; import-gate clean. docker-hive e2e strengthened + live-gated. User signoff 2026-07-11: scope = hive-only (capability opt-in), per the red-team finding that generic all-connector sampling is unsound. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../api/scan/ConnectorScanPlanProvider.java | 18 +++ .../connector/hive/HiveScanPlanProvider.java | 12 +- .../datasource/PluginDrivenScanNode.java | 72 +++++++++- .../translator/PhysicalPlanTranslator.java | 8 ++ .../PluginDrivenScanNodeTableSampleTest.java | 131 ++++++++++++++++++ .../hive/test_hive_tablesample_p0.groovy | 13 ++ 6 files changed, 252 insertions(+), 2 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeTableSampleTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index dbbf6c1d544828..418c76d9950e52 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -232,6 +232,24 @@ default boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle return false; } + /** + * Whether this connector's scan ranges carry meaningful byte lengths + * ({@link ConnectorScanRange#getLength()}) so the engine can apply {@code TABLESAMPLE} by + * size-weighted split selection ({@code PluginDrivenScanNode.sampleSplits}). Returning + * {@code false} (the default) makes {@code TABLESAMPLE} a no-op — the full table is scanned (with a + * warning) — matching these connectors' behavior before the SPI migration (only the legacy Hive scan + * node ever sampled). A connector must NOT return {@code true} unless EVERY range it plans exposes a + * positive, byte-proportional length: MaxCompute's default byte-size ranges and Paimon's JNI-read + * ranges report {@code -1}, and MaxCompute row_offset ranges report a ROW count (not bytes), so they + * must stay {@code false}. Mirrors {@link #supportsBatchScan}'s opt-in shape and Trino's + * {@code ConnectorMetadata.applySample}. + * + * @return whether split-size TABLESAMPLE is valid for this connector (default: false) + */ + default boolean supportsTableSample() { + return false; + } + /** * Plans the scan for a single batch of partitions (used by batch-mode scans). * diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 066214f4f27b08..dc62a2aef1139c 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -61,7 +61,6 @@ *

    • No ACID transaction support (non-transactional tables only)
      • *
    • No file-count streaming split mode; partition-batch mode is limited to partitioned * non-transactional tables (see {@link #supportsBatchScan})
      • - *
    • No table sampling
      • *
    */ public class HiveScanPlanProvider implements ConnectorScanPlanProvider { @@ -167,6 +166,17 @@ public boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle return partKeyNames != null && !partKeyNames.isEmpty() && !hiveHandle.isTransactional(); } + /** + * Hive scan ranges (base/insert files) carry positive byte lengths, so the engine can apply + * {@code TABLESAMPLE} by size-weighted split selection — restoring the legacy + * {@code HiveScanNode.selectFiles} behavior lost at the SPI cutover. Only Hive ever sampled; the + * other connectors keep the default {@code false} (their ranges do not carry byte-proportional lengths). + */ + @Override + public boolean supportsTableSample() { + return true; + } + /** * Plans the scan for a SINGLE partition batch (see * {@link ConnectorScanPlanProvider#planScanForPartitionBatch}). Unlike {@link #planScan} — which resolves diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 19bf958fbc9cc7..969d1c375ef3f6 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -21,6 +21,7 @@ import org.apache.doris.analysis.Expr; import org.apache.doris.analysis.ExprToSqlVisitor; import org.apache.doris.analysis.SlotDescriptor; +import org.apache.doris.analysis.TableSample; import org.apache.doris.analysis.ToSqlParams; import org.apache.doris.analysis.TupleDescriptor; import org.apache.doris.catalog.Column; @@ -74,6 +75,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.Random; import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.Executor; @@ -988,13 +990,28 @@ public List getSplits(int numBackends) throws UserException { // the rows the source returns, that would under-return. Legacy disabled limit-split whenever // a non-partition-equality (incl. CAST) predicate was present; this mirrors it. long sourceLimit = effectiveSourceLimit(limit, filteredToOriginalIndex != null); + // TABLESAMPLE (FIX-M1): applied (sampleSplits below) only when the connector declares its scan + // ranges carry byte lengths (supportsTableSample), so the size-weighted selection is valid. Only + // Hive sampled pre-SPI; connectors whose ranges lack byte-proportional lengths keep the default + // false and no-op the sample (full-table scan, as before) — surfaced with a warning rather than + // silently dropped. connector-agnostic: a generic capability, not a source-type branch. + boolean applySample = tableSample != null + && onPluginClassLoader(scanProvider, scanProvider::supportsTableSample); + if (tableSample != null && !applySample) { + LOG.warn("TABLESAMPLE is not supported by connector [{}]; scanning the full table", + desc.getTable().getDatabase().getCatalog().getType()); + } // Forward the no-grouping COUNT(*) signal to the connector (FIX-COUNT-PUSHDOWN). The op is set // on this node by the Nereids translator (PhysicalPlanTranslator) and shipped to BE via // FileScanNode.toThrift, but a connector that can serve a precomputed row count // (paimon DataSplit.mergedRowCount()) needs the signal here to emit it; otherwise BE // materializes the full post-merge row set just to count. Connectors that do not override the // count-pushdown overload ignore the flag (default delegates to the 6-arg planScan). - boolean countPushdown = getPushDownAggNoGroupingOp() == TPushAggOp.COUNT; + // Suppressed under TABLESAMPLE (applySample): a connector that collapses count-eligible splits + // into ONE range carrying the precomputed FULL-table count (paimon/iceberg) would ignore the + // sample and return full cardinality; with sampling active BE counts rows over the sampled splits + // instead (mirrors legacy HiveScanNode, whose tableSample branch precedes the count-only opt). + boolean countPushdown = getPushDownAggNoGroupingOp() == TPushAggOp.COUNT && !applySample; List ranges = onPluginClassLoader(scanProvider, () -> scanProvider.planScan( connectorSession, currentHandle, columns, remainingFilter, sourceLimit, requiredPartitions, countPushdown)); @@ -1016,9 +1033,54 @@ public List getSplits(int numBackends) throws UserException { // line renders the (-1) sentinel — the correctness-critical no-precomputed-count case. setPushDownCount(pushDownRowCount); } + // TABLESAMPLE (FIX-M1): keep a size-weighted random subset of the planned splits. Only reached + // when the connector opted in (applySample), i.e. its ranges carry byte lengths — so operating on + // the generic Split.getLength() is valid. estimatedRowSize (ROWS mode) = sum of column slot sizes, + // mirroring legacy HiveScanNode.selectFiles. + if (applySample) { + long estimatedRowSize = 0; + for (Column column : desc.getTable().getFullSchema()) { + estimatedRowSize += column.getDataType().getSlotSize(); + } + splits = sampleSplits(splits, tableSample, estimatedRowSize); + } return splits; } + /** + * Connector-agnostic TABLESAMPLE: keeps a size-weighted random subset of splits, mirroring legacy + * {@code HiveScanNode.selectFiles} but operating on the generic {@link Split#getLength()}. Only called + * when the connector declares {@code supportsTableSample()} (its ranges carry positive byte lengths), + * so a negative/row-count length can never corrupt the accumulation. PERCENT targets + * {@code totalSize * value / 100}; ROWS targets {@code estimatedRowSize * value} (estimatedRowSize = sum + * of column slot sizes). The shuffle is seeded by the sample's REPEATABLE seek so a repeated query + * returns the same subset. Pure static so the size-accumulation + seed determinism is unit-testable + * without driving a full {@code planScan}. + */ + static List sampleSplits(List splits, TableSample tableSample, long estimatedRowSize) { + long totalSize = 0; + for (Split split : splits) { + totalSize += split.getLength(); + } + long sampleSize; + if (tableSample.isPercent()) { + sampleSize = totalSize * tableSample.getSampleValue() / 100; + } else { + sampleSize = estimatedRowSize * tableSample.getSampleValue(); + } + Collections.shuffle(splits, new Random(tableSample.getSeek())); + long selectedSize = 0; + int index = 0; + for (Split split : splits) { + selectedSize += split.getLength(); + index += 1; + if (selectedSize >= sampleSize) { + break; + } + } + return splits.subList(0, index); + } + /** * Counts the scan ranges read by BE's native (ORC/Parquet) reader (vs JNI), via the generic * {@link ConnectorScanRange#isNativeReadRange()} (default false). Drives the EXPLAIN @@ -1089,6 +1151,14 @@ private boolean computeBatchMode() { if (scanProvider == null) { return false; } + // TABLESAMPLE (FIX-M1) is applied in the synchronous getSplits() path (sampleSplits); the batch + // path (startSplit) never samples. Force sync when the sample WILL be applied (connector opted in), + // so a sampled scan never silently skips sampling on the batch/streaming path. Sampling shuffles the + // whole split set anyway, so batch generation offers no benefit here. Gated on supportsTableSample + // so a non-sampling connector's TABLESAMPLE no-op does not lose its batch path. + if (tableSample != null && onPluginClassLoader(scanProvider, scanProvider::supportsTableSample)) { + return false; + } boolean hasSlots = !desc.getSlots().isEmpty(); // Streaming (file-count) batch flavor (FIX-M3): the connector owns the whole decision (e.g. // Iceberg's matched-file count vs num_files_in_batch_mode); the engine only requires output slots diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java index 0d0f691da282ce..2141a090d949c5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java @@ -818,6 +818,14 @@ public PlanFragment visitPhysicalFileScan(PhysicalFileScan fileScan, PlanTransla // Forward the pruned partitions so the connector reads only the surviving partitions // (mirrors the legacy MaxCompute / Hive branches below). pluginScanNode.setSelectedPartitions(fileScan.getSelectedPartitions()); + // Forward TABLESAMPLE (mirrors the legacy Hive branch below). Whether it is actually applied + // is decided in PluginDrivenScanNode by the connector's supportsTableSample() capability: a + // connector whose split ranges carry byte lengths (Hive) samples; the others no-op with a + // warning. Without this forward the sample is silently dropped and the query scans the full table. + if (fileScan.getTableSample().isPresent()) { + pluginScanNode.setTableSample(new TableSample(fileScan.getTableSample().get().isPercent, + fileScan.getTableSample().get().sampleValue, fileScan.getTableSample().get().seek)); + } scanNode = pluginScanNode; } else if (table instanceof HMSExternalTable) { if (directoryLister == null) { diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeTableSampleTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeTableSampleTest.java new file mode 100644 index 00000000000000..95f0c84fc990a6 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeTableSampleTest.java @@ -0,0 +1,131 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.analysis.TableSample; +import org.apache.doris.spi.Split; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.ArrayList; +import java.util.List; + +/** + * FIX-M1 — guards {@link PluginDrivenScanNode#sampleSplits}, the pure, connector-agnostic TABLESAMPLE + * split selector that restores the sampling lost when Hive flipped onto the plugin SPI. + * + *

    Why this matters: post-cutover, TABLESAMPLE was silently dropped for plugin-driven external + * tables — {@code SELECT ... TABLESAMPLE(N)} returned the FULL table. The fix re-applies sampling in + * {@code getSplits}, but ONLY for connectors that declare {@code supportsTableSample()} (their ranges carry + * byte lengths); this test pins the size-weighted selection itself. Getting the accumulation wrong + * re-introduces the silent full-table bug (returning every split) or over-truncates (dropping rows the + * sample should keep). The helper is invoked only with positive-length splits (the capability gate lives in + * getSplits and is covered by live e2e — connectors whose ranges report -1 must NOT opt in).

    + */ +public class PluginDrivenScanNodeTableSampleTest { + + /** A split whose byte length is {@code len} (the only property sampleSplits reads). */ + private static Split split(long len) { + Split s = Mockito.mock(Split.class); + Mockito.when(s.getLength()).thenReturn(len); + return s; + } + + /** {@code count} splits, each {@code len} bytes. */ + private static List uniform(int count, long len) { + List splits = new ArrayList<>(); + for (int i = 0; i < count; i++) { + splits.add(split(len)); + } + return splits; + } + + private static long totalLen(List splits) { + long t = 0; + for (Split s : splits) { + t += s.getLength(); + } + return t; + } + + @Test + public void testPercentFullSelectsAll() { + // 100 PERCENT -> sampleSize == totalSize -> every split is selected (no reduction). + List splits = uniform(10, 10L); + List sampled = PluginDrivenScanNode.sampleSplits(splits, new TableSample(true, 100L, 0L), 0L); + Assertions.assertEquals(10, sampled.size()); + } + + @Test + public void testPercentHalfIsMinimalPrefixOverTarget() { + // 50 PERCENT of 10x10-byte splits: sampleSize = 50 -> exactly 5 splits (5*10 >= 50, 4*10 < 50). + // Pins the accumulate-until-(>=sampleSize) boundary: an off-by-one (> vs >=, or pre/post-increment) + // changes the count. Split sizes are uniform, so the count is shuffle-independent. + List splits = uniform(10, 10L); + List sampled = PluginDrivenScanNode.sampleSplits(splits, new TableSample(true, 50L, 0L), 0L); + Assertions.assertEquals(5, sampled.size()); + // The selection is a strict reduction of the full set (this is the property the silent-drop bug violated). + Assertions.assertTrue(sampled.size() < 10); + // Minimal prefix: selected size >= target, and dropping the last selected split would fall under it. + long selected = totalLen(sampled); + Assertions.assertTrue(selected >= 50L); + Assertions.assertTrue(selected - 10L < 50L); + } + + @Test + public void testRowsModeUsesEstimatedRowSize() { + // ROWS sampling: sampleSize = estimatedRowSize * rows = 8 * 3 = 24 -> 3 splits of 10 bytes + // (10,20,30; 30 >= 24 at index 3). Pins that ROWS mode consults estimatedRowSize (not totalSize). + List splits = uniform(10, 10L); + List sampled = PluginDrivenScanNode.sampleSplits(splits, new TableSample(false, 3L, 0L), 8L); + Assertions.assertEquals(3, sampled.size()); + } + + @Test + public void testSampleLargerThanTableSelectsAll() { + // A sample target exceeding the whole table returns every split (never NPEs / over-reads). + List splits = uniform(5, 10L); + List sampled = PluginDrivenScanNode.sampleSplits(splits, new TableSample(false, 1_000_000L, 0L), 1000L); + Assertions.assertEquals(5, sampled.size()); + } + + @Test + public void testEmptySplitsReturnEmpty() { + List sampled = PluginDrivenScanNode.sampleSplits(new ArrayList<>(), new TableSample(true, 10L, 0L), 8L); + Assertions.assertTrue(sampled.isEmpty()); + } + + @Test + public void testRepeatableSeekIsDeterministic() { + // REPEATABLE seeds the shuffle: two runs of the SAME query (same seek) select the SAME subset. + // Distinct split sizes make the selection shuffle-sensitive, so this actually exercises seed determinism + // (uniform sizes would pass trivially). MUTATION: an unseeded / time-seeded shuffle -> flaky subset -> red. + List base = new ArrayList<>(); + for (int i = 1; i <= 10; i++) { + base.add(split(i)); // sizes 1..10, total 55 + } + // Copies preserve element identity + initial order, so same-seed shuffles produce the same permutation. + List run1 = PluginDrivenScanNode.sampleSplits(new ArrayList<>(base), new TableSample(true, 30L, 7L), 0L); + List run2 = PluginDrivenScanNode.sampleSplits(new ArrayList<>(base), new TableSample(true, 30L, 7L), 0L); + Assertions.assertEquals(run1, run2); + // And it is a real sample, not the whole table (30% target). + Assertions.assertTrue(run1.size() < base.size()); + } +} diff --git a/regression-test/suites/external_table_p0/hive/test_hive_tablesample_p0.groovy b/regression-test/suites/external_table_p0/hive/test_hive_tablesample_p0.groovy index bc9a52d146e3f0..701b7cd4d8cdd7 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_tablesample_p0.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_tablesample_p0.groovy @@ -47,6 +47,19 @@ suite("test_hive_tablesample_p0", "p0,external") { sql("select count(*) from student tablesample(10 percent);") contains "count(*)[#7]" } + // FIX-M1: post-cutover TABLESAMPLE was silently dropped (plugin scan returned the FULL table). + // The connector now opts in (HiveScanPlanProvider.supportsTableSample) and PluginDrivenScanNode + // samples the splits. A sample never exceeds the full table — assert that invariant on real + // results (not just an EXPLAIN substring). NOTE: a STRONG reduction assertion (sampled < full) + // needs a multi-file table; on a single-file fixture the sample floor is one file (== full), so + // the strict-reduction check is left to live verification against a large table. + def fullCount = sql """select count(*) from student""" + def sampledRows = sql """select count(*) from student tablesample(10 rows)""" + def sampledPercent = sql """select count(*) from student tablesample(10 percent)""" + assertTrue(sampledRows[0][0] <= fullCount[0][0], + "TABLESAMPLE(rows) count ${sampledRows[0][0]} must not exceed full table ${fullCount[0][0]}") + assertTrue(sampledPercent[0][0] <= fullCount[0][0], + "TABLESAMPLE(percent) count ${sampledPercent[0][0]} must not exceed full table ${fullCount[0][0]}") sql """drop catalog if exists ${catalog_name}""" } finally { } From af1cb7e853a979b13bd5b4876f1008fcfb6d263c Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 12:25:02 +0800 Subject: [PATCH 236/334] [doc](catalog) #65185 batch-2: M1 TABLESAMPLE opt-in fix design + tracking (batch 2 done) - FIX-M1-design.md: scope correction (hive-only regression, not all-four), the design red-team wf_32decfa0-349 that overturned the original "generic all-connector sampling" as UNSOUND (Split.getLength() is -1 / row-count for MaxCompute/Paimon), and the corrected connector opt-in design (supportsTableSample capability) with user signoff (scope = hive-only, 2026-07-11). - task-list + HANDOFF: mark M1 DONE (17b432dc1e1) and batch 2 (M3+M1) complete; next = batch 3. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 9 +- plan-doc/tasks/designs/FIX-M1-design.md | 135 +++++++++++++++++++++ 3 files changed, 141 insertions(+), 5 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-M1-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 8b94b9daffe708..de5fed50d08484 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -42,7 +42,7 @@ PR #65474（hive connector SPI 迁移）build **991951** 50 外表 case 失败 **⭐ 下一步（下个 session）：** 1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`+`fca288424fc`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3；**最终复核纠正 TTL 86400→600s 对齐旧版**) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**最终对抗复核（`wf_542c60b9-001`，5 per-fix + 1 cross-cut）= M5/M6/M7/M2/cross-cut CLEAN、M4 命中 1 medium(TTL)已修 → 全 CLEAN。** - **⭐ 批次 2（M3→M1，fe-core 通用节点）进行中：M3 DONE `6963de4124f`**——`shouldUseBatchMode` 的 `!isPruned`→`== NOT_PRUNED`，无谓词大分区表（MaxCompute + 翻闸 hive）恢复异步 batch split，**顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余**（legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证；git `1da88365e85^` 取证 + 全 producer 枚举证闭合）。设计红队 `wf_811e6242-d8b`（3 lens）命中 1 blocker + 1 major 均已解：**反转**被前次评审特意锁定的 pinning 测试（`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`）、**登记 supersession**（`decisions-log` D-035 / `deviations-log` DV-019 的 LP-1「`!isPruned` 等价」判定已批注 SUPERSEDED）、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`（**用户 2026-07-11 签字：采用 SPI 统一分区数口径**，不给 hive 补 split-count 估算；对齐 MaxCompute/Trino）。`PluginDrivenScanNodeBatchModeTest` 12/12 绿。设计 `designs/FIX-M3-design.md`。**⚠ 本轮踩并行 session 构建污染坑**（`be-java-extensions package -am -T 1C` 污染共享 target 报「cannot access 生成类」假失败，非本码；待其结束干净重跑 12/12）。**下一步 = M1**（TABLESAMPLE 插件路径静默全表扫；translator 转发 `setTableSample` + `PluginDrivenScanNode.getSplits` 通用按 split 大小采样，connector-agnostic）。之后批次 3（M8/L1）… ⏸ 决策类（L2/L10/L12/L20）先问用户。 + **⭐ 批次 2（M3→M1，fe-core 通用节点）全部 DONE。M3 `6963de4124f`**——`shouldUseBatchMode` 的 `!isPruned`→`== NOT_PRUNED`，无谓词大分区表（MaxCompute + 翻闸 hive）恢复异步 batch split，**顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余**（legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证；git `1da88365e85^` 取证 + 全 producer 枚举证闭合）。设计红队 `wf_811e6242-d8b`（3 lens）命中 1 blocker + 1 major 均已解：**反转**被前次评审特意锁定的 pinning 测试（`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`）、**登记 supersession**（`decisions-log` D-035 / `deviations-log` DV-019 的 LP-1「`!isPruned` 等价」判定已批注 SUPERSEDED）、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`（**用户 2026-07-11 签字：采用 SPI 统一分区数口径**，不给 hive 补 split-count 估算；对齐 MaxCompute/Trino）。`PluginDrivenScanNodeBatchModeTest` 12/12 绿。设计 `designs/FIX-M3-design.md`。**⚠ 本轮踩并行 session 构建污染坑**（`be-java-extensions package -am -T 1C` 污染共享 target 报「cannot access 生成类」假失败，非本码；待其结束干净重跑 12/12）。**M1 DONE `17b432dc1e1`**——翻闸 hive 上 `TABLESAMPLE` 被静默丢弃(全表扫)。设计红队 `wf_32decfa0-349` **推翻原"通用采样"方案(UNSOUND)**:`Split.getLength()` 语义因连接器而异(MaxCompute 默认 byte_size/Paimon JNI range 报 -1、MaxCompute row_offset 报行数)→ 盲目按字节采样出乱结果;**scope 更正为 hive-only 回归**(只有 hive 曾采样)。→ 改**连接器 opt-in**(`ConnectorScanPlanProvider.supportsTableSample()` 默认 false、仅 `HiveScanPlanProvider` true;**用户 2026-07-11 签字 scope=hive-only**,对齐 Trino `applySample` + `supportsBatchScan` 先例)。translator 通用转发 + `PluginDrivenScanNode.sampleSplits`(仅 `applySample` 时,legacy `selectFiles` 端口)+ 非支持连接器 no-op+WARN + 两 gate 门(COUNT/batch 抑制,挂 `applySample`)。`PluginDrivenScanNodeTableSampleTest` 6/6 + BatchMode 12/12 + hive 285/285 绿、0 checkstyle、import 门净。设计 `designs/FIX-M1-design.md`。**⭐ 批次 2(M3+M1)全部 DONE。下一步 = 批次 3(M8 发布工具/文档 + L1 import 门禁)**;之后批次 4(低危连接器 L3–L20)… ⏸ 决策类(L2/L10/L12/L20)先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 3. **Phase 3 删除旧代码（最后做）**：~90 类循环单元（`datasource/hive|hudi|iceberg`）+ 死 Nereids 臂 + 删除解锁抽取（HiveUtil/HiveSplit/IcebergUtils）+ 那 3 个 @Disabled 测试。拓扑顺序+清单：execution plan §2.4/§3/§4。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index b537a66342edf3..c686b116fe39ef 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -29,7 +29,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 2 | **H2** | 🔴高 | hudi**+hive** | datetime 分区谓词 ISO→0 行 | ✅ | ✅ | ✅ | ✅ | | 3 | **H3** | 🔴高 | hudi | HMS 名当存储路径→非 hive-style 带 filter 0 split | ✅ | ✅ | ✅ | ✅ | | 4 | **H4** | 🔴高 | hudi | 混大小写 Avro→JNI 崩 | ✅ | ✅ | ✅ | ✅ | -| 5 | **M1** | 🟠中 | fe-core | TABLESAMPLE 插件路径静默全表扫 | ⬜ | ⬜ | ⬜ | ⬜ | +| 5 | **M1** | 🟠中 | fe-core+hive | TABLESAMPLE 插件路径静默全表扫 | ✅ | ✅ | ✅ | ✅ | | 6 | **M2** | 🟠中 | hive | 翻闸 hive 丢批量/异步 split | ✅ | ✅ | ✅ | ✅ | | 7 | **M3** | 🟠中 | fe-core | MC batch 闸门 `!=NOT_PRUNED`→`!isPruned` | ✅ | ✅ | ✅ | ✅ | | 8 | **M4** | 🟠中 | maxcompute | MC 分区值缓存删除→每查询全量 listPartitions | ✅ | ✅ | ✅ | ✅ | @@ -111,7 +111,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) ## 🟠 中危(M1–M8) ### M1 — TABLESAMPLE 插件路径静默全表扫 · reverify §3 M1 -- [ ] **M1** +- [x] **M1** · DONE `17b432dc1e1`（设计 `designs/FIX-M1-design.md`；设计红队 `wf_32decfa0-349` 3 lens **推翻原"通用采样"方案**为 UNSOUND → 改**连接器 opt-in**，**用户 2026-07-11 签字"只修 hive"**）。**scope 更正=hive-only 回归**（只有 hive 曾采样；其它连接器 legacy 从不采样）。原方案缺陷=`Split.getLength()` 语义因连接器而异（MaxCompute 默认 byte_size/Paimon JNI range 报 -1，MaxCompute row_offset 报行数），盲目按字节采样出乱结果。修法=SPI `ConnectorScanPlanProvider.supportsTableSample()` 默认 false + `HiveScanPlanProvider` override true；translator 插件臂通用转发 tableSample；`PluginDrivenScanNode` 仅在 `applySample`(能力开)时 `sampleSplits`（legacy `selectFiles` 端口，通用 `Split#getLength()`），非支持连接器 no-op+WARN（非静默）。两 gate 门（COUNT 抑制、batch 抑制）挂 `applySample`。`PluginDrivenScanNodeTableSampleTest` 6/6 + BatchMode 12/12 + hive 285/285，0 checkstyle，import 门净。e2e live-gated（docker-hive 结果不变式已加；强基数缩减须多文件 fixture 真集群验）。 - **现码**:`PhysicalPlanTranslator.visitPhysicalFileScan:812-821`(只 `setSelectedPartitions`,不转发 `getTableSample()`;legacy 转发臂 `:837-840` 死)+ `PluginDrivenScanNode.getSplits:998-1019`(零 tableSample)+ `ConnectorScanPlanProvider.planScan:119`(无采样参数)。 - **Fix**:(1)translator 插件臂在 `setSelectedPartitions` 后镜像 legacy 转发 `setTableSample`;(2)`PluginDrivenScanNode.getSplits` 实现**通用 connector-agnostic** 按 split 大小采样(仿 `HiveScanNode:448-458`,操作通用 `PluginDrivenSplit` 大小,**不按源分支**)。 - **Files**:`fe-core/.../PhysicalPlanTranslator.java`、`fe-core/.../PluginDrivenScanNode.java`。 @@ -251,7 +251,8 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) --- -**⭐ 批次 2(M3→M1,fe-core 通用节点)进行中** +**⭐ 批次 2(M3→M1,fe-core 通用节点)全部 DONE** - **M3** DONE `6963de4124f`(code) — batch 闸门 `!isPruned`→`== NOT_PRUNED`:无谓词大分区表(MaxCompute + 翻闸 hive)恢复异步 batch split(legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证;git `1da88365e85^` 取证 + 全 producer 枚举证闭合)。**顺带解 M2 的 BATCH-UNPRUNED-SYNC**。设计红队 `wf_811e6242-d8b`(3 lens:BLAST-RADIUS SOUND / LEGACY-PARITY SOUND_WITH_CHANGES / COMPLETENESS SOUND_WITH_CHANGES)命中 1 blocker(docker-hive golden 未 reconcile)+ 1 major(overturn 前次签字 LP-1/D-035「等价」未登记)均已解:**反转** pinning 测试(`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`,assertTrue)、**登记 supersession**(D-035/DV-019 补 SUPERSEDED 批注)、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`(**用户 2026-07-11 签字:SPI 统一分区数口径**,非 hive 专属 split-count 估算;对齐 MaxCompute/Trino「引擎层统一报分片」)。`PluginDrivenScanNodeBatchModeTest` **12/12** 绿、fe-core test-compile BUILD SUCCESS 0 checkstyle。**e2e live-gated**(docker-hive `(approximate)inputSplitNum=6` + MaxCompute 无谓词 ≥阈值分区表进 batch;sweep 确认全 regression 仅此 1 处 `(approximate)` 断言受影响,maxcompute p2 只断言 `partition=N/M`/结果不受影响)。**⚠ 构建坑**:本轮 UT 一度被并行 session 的 `be-java-extensions package -am -T 1C` 构建污染共享 target 报「cannot access 生成类」假失败(非本码);待其结束后干净重跑 12/12(memory `concurrent-sessions-shared-worktree-hazard`)。 -- **下一步 = M1**(TABLESAMPLE 插件路径静默全表扫;translator 转发 `setTableSample` + `PluginDrivenScanNode.getSplits` 通用按 split 大小采样,仿 legacy `HiveScanNode:448-458` 但 connector-agnostic)。 +- **M1** DONE `17b432dc1e1`(code) — TABLESAMPLE 翻闸 hive 静默全表扫修复。**红队 `wf_32decfa0-349` 推翻原"通用采样"方案(UNSOUND)**:`Split.getLength()` 对 MaxCompute 默认 byte_size / Paimon JNI range 报 -1、对 MaxCompute row_offset 报行数 → 盲目按字节采样出乱结果(全表或 1 split)。**scope 更正=hive-only 回归**(只有 hive 曾采样)。→ 改**连接器 opt-in**(`ConnectorScanPlanProvider.supportsTableSample()` 默认 false、仅 `HiveScanPlanProvider` true;**用户 2026-07-11 签字 scope=hive-only**)。translator 插件臂通用转发 + `PluginDrivenScanNode.sampleSplits`(仅 `applySample` 时,legacy `selectFiles` 端口)+ 非支持连接器 no-op+WARN(非静默)+ 两 gate 门(COUNT 抑制/batch 抑制,挂 `applySample`)。对齐 Trino `applySample` + 上一批 `supportsBatchScan` opt-in 先例。`PluginDrivenScanNodeTableSampleTest` 6/6 + BatchMode 12/12 + hive 285/285 绿、0 checkstyle、import 门净。e2e live-gated(docker-hive 结果不变式已加;强基数缩减须多文件 fixture 真集群验)。 +- **⭐ 批次 2(M3→M1)全部 DONE。** 2 条 fe-core 通用节点修复,各配 RED-able 单测 + 独立 code/doc commit + 设计红队(各 3 lens;**M1 红队捕获并推翻 UNSOUND 原方案 → 触发用户 scope 决策**)。**下一步 = 批次 3(M8 发布工具/文档 + L1 import 门禁)**;之后批次 4(低危连接器)… ⏸ 决策类(L2/L10/L12/L20)先问用户。 diff --git a/plan-doc/tasks/designs/FIX-M1-design.md b/plan-doc/tasks/designs/FIX-M1-design.md new file mode 100644 index 00000000000000..e8f69d853be687 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-M1-design.md @@ -0,0 +1,135 @@ +# FIX-M1 — 翻闸 hive 上 TABLESAMPLE 被静默丢弃 → 连接器 opt-in 采样 + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §3 M1（scope = **翻闸 hive**，"新激活"）。 +> 批次 2（fe-core 通用节点）。设计红队 `wf_32decfa0-349`（3 lens）**推翻原"对所有连接器通用采样"方案**（UNSOUND）→ 改 +> **连接器 opt-in**（用户 2026-07-11 签字"只修 hive"）。HEAD 复核（primary source）：translator 两臂、`getSplits`、 +> legacy `HiveScanNode.selectFiles`、各连接器 range length。 + +## Problem（scope 更正：**仅 hive 回归**，非四连接器） + +**采样(`TABLESAMPLE`)迁移前只有 hive 真正支持**——`HiveScanNode.selectFiles:435-467` 按文件字节大小随机挑一部分文件。 +其它连接器（paimon/iceberg/maxcompute）的 legacy scan node **从不** sample tableSample（grep 全 fe-core scan node 仅 +`HiveScanNode` 引用 tableSample）→ 对它们 TABLESAMPLE 一直是**静默 no-op（返回全表）**，非回归。 + +翻闸后 hive 走通用 `PluginDrivenScanNode`，`visitPhysicalFileScan` 插件臂 `:812-821` 只 `setSelectedPartitions`、 +**不转发** `fileScan.getTableSample()`（唯一转发在 legacy hive 臂 `:837-840`，翻闸后死）→ hive 也退化成静默全表扫。 +**这是 hive-only 回归**（reverify "翻闸 hive 新激活"）。 + +## Root Cause + 红队推翻的原方案 + +- **转发缺失**（真 bug）：插件臂无 `setTableSample`。 +- **原方案（红队 UNSOUND，已弃）**：在通用节点对**所有** `PluginDrivenSplit` 按 `getLength()` 统一采样。红队证实 + `Split.getLength()` 语义**因连接器而异**——`ConnectorScanRange.getLength()` 默认 -1；**MaxCompute 默认 byte_size 策略 + 每 range `.length(-1L)`**（`MaxComputeScanPlanProvider:345`），**Paimon JNI-read range 不设 length**（默认 -1， + `PaimonScanRange:300`），MaxCompute row_offset 策略 length 是**行数非字节**。把 -1/行数喂进按字节采样 → totalSize 负 → + ROWS 模式返**全表**、PERCENT 模式返全表或 1 个 split（乱）。故对无字节大小的连接器统一采样=**给它们造新错结果**。 +- **结论**：分片能否按大小采样**只有连接器自己知道**；通用节点不能靠 split 本身判断，也不得按源名分支。 + +## Design（连接器 opt-in；connector-agnostic，无源判别、无属性解析） + +### 1. SPI 能力开关（`ConnectorScanPlanProvider`，镜像 `supportsBatchScan` opt-in） +```java +/** 该连接器的 range 是否携带有意义的字节长度(ConnectorScanRange#getLength())，使引擎可按大小采样 TABLESAMPLE。 + * 返回 false(默认)→ TABLESAMPLE no-op(全表 + 一条 WARN)，即这些连接器 SPI 迁移前的行为。连接器**必须**确保它 + * planScan 的每个 range 都有正字节长度才可返 true(MaxCompute 默认 byte_size/Paimon JNI range 报 -1，须保持 false)。 + * 镜像 supportsBatchScan 的 opt-in 形状 + Trino ConnectorMetadata.applySample。 */ +default boolean supportsTableSample() { return false; } +``` + +### 2. hive 连接器声明支持（`HiveScanPlanProvider`） +```java +@Override public boolean supportsTableSample() { return true; } // hive base/insert 文件 range 有正字节长度 +``` ++ 订正 class-javadoc `:64` 的 "No table sampling"。**仅 hive override**；iceberg/paimon/maxcompute 保持 false（不变）。 + +### 3. translator 转发（`PhysicalPlanTranslator.visitPhysicalFileScan` 插件臂，`:820` 后） +镜像 legacy hive 臂的 Nereids→analysis `TableSample` 转换（**通用转发**，不按源；实际是否采样由节点按连接器能力定）： +```java +if (fileScan.getTableSample().isPresent()) { + pluginScanNode.setTableSample(new TableSample(fileScan.getTableSample().get().isPercent, + fileScan.getTableSample().get().sampleValue, fileScan.getTableSample().get().seek)); +} +``` +**Hudi 不动**：`visitPhysicalHudiScan` legacy 臂 `:932-940` 本就不转发 tableSample（legacy parity；reverify 只点 file scan 臂）。 + +### 4. `PluginDrivenScanNode` 采样（`getSplits`）+ 能力门 +```java +boolean applySample = tableSample != null + && onPluginClassLoader(scanProvider, scanProvider::supportsTableSample); +if (tableSample != null && !applySample) { + LOG.warn("TABLESAMPLE is not supported by connector {}; scanning the full table", ); +} +... +boolean countPushdown = getPushDownAggNoGroupingOp() == TPushAggOp.COUNT && !applySample; // (a) +... +// 建完 splits，return 前： +if (applySample) { + long estimatedRowSize = 0; + for (Column column : desc.getTable().getFullSchema()) { + estimatedRowSize += column.getDataType().getSlotSize(); + } + splits = sampleSplits(splits, tableSample, estimatedRowSize); +} +return splits; +``` +新纯静态 helper（对齐 `HiveScanNode.selectFiles:435-467`，操作通用 `Split#getLength()`——**仅在连接器声明 range 有正 +字节长度时才调**，故对 hive 安全）： +```java +static List sampleSplits(List splits, TableSample tableSample, long estimatedRowSize) { + long totalSize = 0; + for (Split split : splits) { totalSize += split.getLength(); } + long sampleSize = tableSample.isPercent() + ? totalSize * tableSample.getSampleValue() / 100 + : estimatedRowSize * tableSample.getSampleValue(); + Collections.shuffle(splits, new Random(tableSample.getSeek())); // REPEATABLE 种子 → 可复现 + long selectedSize = 0; int index = 0; + for (Split split : splits) { + selectedSize += split.getLength(); index += 1; + if (selectedSize >= sampleSize) { break; } + } + return splits.subList(0, index); +} +``` +imports 加 `java.util.Random`、`analysis.TableSample`（`Column`/`Collections`/`Split`/`List`/`ArrayList` 已在）。 + +### 5. 完整性两门（红队确认 NECESSARY，均 gate 在 `applySample`） +- **(a) COUNT 下推抑制**（`getSplits`）：`countPushdown = COUNT && !applySample`。红队证实 paimon(`:471,515-521`)、 + **iceberg**(`IcebergScanPlanProvider:518-524`) 会把 count-eligible split **塌成一个** carrying 全表 precomputed count + 的 range，BE `CountReader` 直接服务、不读数据 → FE 侧采样裁不掉这个塌缩 range → `count(*) TABLESAMPLE` 返全表计数。 + gate 在 `applySample`：当前仅 hive 采样、hive 不塌缩 count（无此风险），但**将来 opt-in 且塌缩 count 的连接器**须此门； + 且 `!applySample` 使 **plain `count(*)`（无 sample）不受影响**（非回归）。 +- **(b) batch 抑制**（`computeBatchMode` scanProvider null-guard 后）：`if (applySample) return false`。采样仅在同步 + `getSplits`；batch `startSplit`(`:1181-1247`) 不采样。M3 已把 batch 拓到无谓词大分区表 → 不抑制则 hive 大分区表 + `TABLESAMPLE` 走 batch 无采样=静默全扫。采样本须全量枚举 split（shuffle 全集），batch 对采样无收益 → 强制同步无损。 + +## Risk + +- **connector-agnostic**：能力开关是连接器声明（非源名分支）；节点只按通用 `tableSample` 字段 + 通用能力布尔 + 通用 + `Split#getLength()`。无 `if(hive)`/instanceof/属性解析。符合铁律 + 用户 2026-07-11「只修 hive」签字。 +- **非采样路径 = 严格 no-op**（红队 SOUND 核心）：`tableSample` 默认 null，仅新 translator 转发在 `isPresent()` 下 set； + `sampleSplits`/countPushdown 额外子句/computeBatchMode 门全 gate 在 `tableSample!=null`(且 `applySample`)→ 普通查询零变。 +- **非 hive 连接器**：转发了 tableSample 但 `supportsTableSample=false` → `applySample=false` → 不采样 + WARN（显式非静默， + 同迁移前全表结果）。iceberg/paimon/maxcompute 行为不变。 +- **hive 采样正确性**：`HiveScanRange.getLength()` 正字节长度；percent/rows/seed 逐字节对齐 legacy `selectFiles`。 + 注：粒度是 split（连接器已切）非 legacy 的 file-then-split，结果**近似** legacy 非逐行等同——采样本即近似语义，可接受。 +- **subList 视图**：`splits.subList(0,index)`（同 legacy），getSplits 结果只读消费(`FileQueryScanNode:415/419/424`)，安全。 +- **Hudi/其它**：不触碰。 + +## Test Plan + +### Unit（`PluginDrivenScanNodeTableSampleTest`，fe-core 有 Mockito；RED-able） +纯静态 `sampleSplits` 直测（mock `Split.getLength()` 返正长度）： +- percent 100%→全选；percent 50%(均匀)→断言 `selectedSize>=sampleSize` 且去末元 `totalSize→全选。 +- **RED-able**：删采样调用则「采样后 < 全集」断言红。 +（能力门 `applySample`/countPushdown/computeBatchMode 的 wiring：CALLS_REAL_METHODS mock 补测或 helper 直测 + live e2e， +登记同 DV-019。SPI 默认 false + hive override true 由连接器模块测/编译覆盖。） + +### E2E（live-gated，docker-hive；须真集群） +- 强化 `test_hive_tablesample_p0.groovy`：现仅断言 EXPLAIN `count(*)[#7]`（抓不到 bug）。加**结果不变式** + `count(*) TABLESAMPLE(...) <= count(*)`(全表)；**强基数缩减**(sample Date: Sat, 11 Jul 2026 14:17:22 +0800 Subject: [PATCH 237/334] [doc](catalog) #65185 reverify-fixes: standalone task-line HANDOFF for resume Self-contained handoff snapshot for the #65185 review-fix series (separate from the mainline HANDOFF.md), so the task line can be resumed after a context switch: current progress (batches 0/1/2 done, batch 3 next), the batch-2 user signoffs (M3 golden 60->6, M1 hive-only TABLESAMPLE opt-in), start-reading order, iron rules, build/verify gotchas, commit discipline, the concurrent-session hazard, and the live-gated e2e backlog. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 154 +++++++++++++++++++++++ 1 file changed, 154 insertions(+) create mode 100644 plan-doc/HANDOFF-65185-reverify-fixes.md diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md new file mode 100644 index 00000000000000..d56e776b020c07 --- /dev/null +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -0,0 +1,154 @@ +# 🔖 HANDOFF —— #65185 复核修复系列(独立任务线快照) + +> **用途**：这是**「复核修复系列」这一条任务线**的**独立**交接快照，与主线滚动 `plan-doc/HANDOFF.md`（HMS 翻闸主线）**分开**。你切去做别的实现后，回到这条线时**先读本文件**即可续做，不必炒对话历史。 +> **生成**：2026-07-11。**分支** = `catalog-spi-11-hive`（off `branch-catalog-spi`，PR base = `branch-catalog-spi`，squash 合并）。**HEAD 快照** = `af1cb7e853a`。 +> **公开 tracking issue** = apache/doris#65185。 + +--- + +## 0. 这条任务线是什么 + +HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方复核报告 +`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` 判定为**真实/活跃且需改代码**的条目，按批次逐条修。 +**不含**已登记/验收偏差（reverify §5）与不适用/已修/parity（reverify §6）——那些**明确不做**（见跟踪表文末「明确不做」）。 + +**处理纪律（每条一遍）**：起步读本文件 + 跟踪表 → 选一条 → **对 HEAD 复核现码**（reverify 行号可能已漂）→ 设计 +（`plan-doc/tasks/designs/FIX--design.md`）→ **设计红队**（多 agent 对抗，见 memory `clean-room-adversarial-review-pref`） +→ 实现 → build + 靶向 UT → **独立 commit** → 勾跟踪表 → 更新本 HANDOFF + commit。**每条一个独立 commit**，code 与 doc 分开。 + +--- + +## 1. 起步必读（回来先读这几个，行号信 HEAD 不信文档） + +1. **跟踪表（权威进度 + 每条现码/fix/测试意图 + 每轮滚动更新）** = `plan-doc/task-list-65185-reverify-fixes.md`。 +2. **复核报告（证据/失败场景/对抗结论）** = `plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` + （§2 高危 / §3 中危 / §4 设计债 / §5 已登记偏差 / §6 不适用）。 +3. **各条设计** = `plan-doc/tasks/designs/FIX--design.md`（含 recon + 设计红队结论 + 最终改动清单）。 +4. **完成明细** = `git log`（commit message 详尽，勿在文档里重述）。 + +--- + +## 2. 当前进度（截至 HEAD `af1cb7e853a`） + +| 批次 | 内容 | 状态 | +|------|------|------| +| 批次 0 | H1–H4 高危（hudi/hive 分区剪枝：转义/DATETIME/非-hive-style/混大小写列名） | ✅ 全 DONE + 3-skeptic 复审 CLEAN | +| 批次 1 | M5/M7/M6/M4/M2 中危（连接器局部：iceberg×3 + mc + hive） | ✅ 全 DONE + 最终复核 CLEAN | +| **批次 2** | **M3→M1 中危(fe-core 通用节点)** | **✅ 全 DONE(本轮)** | +| 批次 3 | M8(发布工具/文档) + L1(import 门禁) | ⬜ **← 回来从这里续** | +| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | ⬜ | +| 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | +| 设计债 | D-系列 | ⏸ 择机 / 随 P8 | + +**批次 2 明细（本轮完成，含用户签字，勿再擅自翻）**： +- **M3** `6963de4124f`(code) + `97363fc9c33`(doc) —— batch 闸门 `shouldUseBatchMode` 的 `!isPruned` → `== NOT_PRUNED`： + 无谓词大分区表（MaxCompute + 翻闸 hive）恢复异步 batch split，**顺带解 M2 的 BATCH-UNPRUNED-SYNC**。 + 证据 = legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED`（git `1da88365e85^`）+ sibling `displayPartitionCounts` + + 全 producer 枚举证闭合。设计红队 `wf_811e6242-d8b` 命中 1 blocker + 1 major 均已解。 + **⭐ 用户签字（2026-07-11）**：docker-hive golden `test_hive_partitions:200` `(approximate)inputSplitNum` **60→6** + ——采用 **SPI 统一分区数口径**（不给 hive 补 legacy split-count 估算；对齐 MaxCompute + Trino「引擎层统一报分片」）。 + **反转**被前次评审特意锁定的 pinning 测试；**登记 supersession**：`decisions-log` D-035 / `deviations-log` DV-019 + 的 LP-1「`!isPruned` 等价」判定已批注 SUPERSEDED。 +- **M1** `17b432dc1e1`(code) + `af1cb7e853a`(doc) —— 翻闸 hive 上 `TABLESAMPLE` 被静默丢弃（全表扫）。 + 设计红队 `wf_32decfa0-349` **推翻原「对所有连接器通用采样」方案（UNSOUND）**：`Split.getLength()` 语义因连接器而异 + （hive/iceberg=字节；MaxCompute 默认 byte_size / Paimon JNI range = **-1**；MaxCompute row_offset = **行数**）→ + 盲目按字节采样出乱结果。**scope 更正 = hive-only 回归**（只有 hive 曾采样）。 + **⭐ 用户签字（2026-07-11）**：**只修 hive** —— 加 `ConnectorScanPlanProvider.supportsTableSample()` 默认 false 能力 + opt-in、仅 `HiveScanPlanProvider` override true；通用节点仅在连接器声明支持时 `sampleSplits`（legacy `selectFiles` + 端口），不支持连接器 no-op + WARN（非静默）。对齐 Trino `applySample` + `supportsBatchScan` 先例。 + （新记 memory `catalog-spi-split-length-not-universal-byte-size`。） + +--- + +## 3. 下一步 = 批次 3（细节，回来直接开工） + +**M8 — 升级只换 lib 不部署 plugins → 首访抛（reverify §3 M8；无 fe 编译）** +- 现码：`PluginDrivenExternalCatalog.java:135` throw；`CatalogFactory` degraded 只护启动。翻闸把 blast radius 扩到全部 + `type=hms` 目录。 +- Fix（主线）：发布/升级工具把连接器 jar 部署到 `connector_plugin_root`（`build.sh`/部署步骤）+ 响亮 release note。 + 可选防御：replay 后聚合 ERROR 枚举所有 degraded 目录。**保留** first-access throw，**不**加 legacy fallback。 +- 测试：升级文档步骤评审；（可选）degraded 聚合日志单测。 + +**L1 — import-gate 三洞（reverify §1）** +- `tools/check-connector-imports.sh:48,50`：grep 改 `^import (static )?${FORBIDDEN}[.]`；`FORBIDDEN` 补 + `persist|transaction|fs|statistics|mysql|service`；glob 覆盖 `src/test/java`。**保留** `is_vendored()` + （`datasource.hive.HiveVersionUtil` 是 fe-connector-hms vendored 副本，误报非违规；memory + `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 + +之后：批次 4（低危连接器）；决策类（L2/L10/L12/L20）先问用户；设计债 D-系列择机。跟踪表「建议批次」节有全清单。 + +--- + +## 4. 铁律 / 约束（每条修复都受约束） + +- **fe-core 不得**新增 `if(hive/iceberg/hudi)` / `instanceof HMSExternal*` / `switch(dlaType)` / 源名判别；**不解析属性** + （storage→fe-filesystem、meta→fe-connector）；通用 SPI 节点 connector-agnostic + （memory `catalog-spi-plugindriven-no-source-specific-code`、`catalog-spi-no-property-parsing-in-fecore`）。 + **按连接器区分能力用 `supports*()` opt-in**（非源名分支），如 `supportsBatchScan`/`supportsTableSample`。 +- **`Split.getLength()` 语义因连接器而异**（-1 / 行数 / 字节）——通用节点凡按 split 大小处理须能力 opt-in、禁假设字节大小 + （memory `catalog-spi-split-length-not-universal-byte-size`）。 +- 跨插件/跨边界**须 pin TCCL**（memory `catalog-spi-plugin-tccl-classloader-gotcha`，含 4 locus + HiveConf 构造点）。 +- `history_schema_info` 嵌套字段名逐层 lowercase（memory `catalog-spi-history-schema-info-lowercase-nested-names`）。 + +--- + +## 5. 构建 / 验证备忘（复用） + +- fe-core：`mvn -o -f /mnt/disk1/yy/git/wt-catalog-spi/fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false` + （**漏 `-am`→假 `${revision}` 错**）。连接器：`-pl :fe-connector- -am`。SPI 改动（fe-connector-api）会被 + `-am` 连带 rebuild。 +- 靶向 UT 加 `-Dtest= -DfailIfNoTests=false`（`-am` + `-Dtest` 上游无匹配测试会报假「No tests were executed!」）。 +- **⚠ paimon 模块用 `install`/`package`**（shade jar 绑 package 阶段）；hms/hive/hudi/iceberg/mc 无此坑。 +- **信 LOG 不信 exit**：后台 task 通知的 exit 是 wrapper 的；重定向到文件 grep `BUILD SUCCESS|BUILD FAILURE|[ERROR].*\.java:|Tests run:|You have N Checkstyle`。全量编译 ~6min → 后台跑。cwd 会重置 → 绝对路径。**勿用 worktree 隔离编译 agent**（`/mnt/disk1` 盘紧）。 +- 连接器测试**无 Mockito**（真 recording fake）；**fe-core 有 Mockito**。checkstyle 禁 static import、扫 test 源、`UnusedImports` fail build。 +- `bash tools/check-connector-imports.sh` 须 exit 0（连接器不得 import fe-core）。 +- **memory** `doris-build-verify-gotchas`、`catalog-spi-fe-core-test-infra`。 + +--- + +## 6. 提交 / 工作树纪律 + +- **path-whitelist `git add`，严禁 `git add -A`**：工作树大量遗留 scratch（`regression-test/conf/regression-conf.groovy` + 明文 key【本轮它被 build 改动过、勿提交】、`*.bak`、`.audit-scratch/`、`conf.cmy/`、`META-INF/`、`docker/...`、 + `plan-doc/reviews/P5-*`、`.claude/`、`failed-cases.out`——**非本线程产物，勿混入任何 commit**）。 +- commit message：`[fix|doc](catalog) …` + 末尾 `Co-Authored-By: Claude Opus 4.8 (1M context) ` + + `Claude-Session: …`。**每子步/每条 fix = 独立 commit**；设计文档 + HANDOFF 单独 commit（与 code 分开）。 +- 上下文超 30% 找干净节点交接（memory `session-handoff-at-30pct-context`）。 + +--- + +## 7. ⚠ 并行 session 风险（本轮真实踩过） + +本工作树是 linked worktree、**多 session 共享同一分支 + 同一构建 target**。本轮： +- 另一 session（你自己，`morningman`）在批次 2 期间往同分支提交了 2 个 hive 改动（`4dbb8e02056` 回归文档、 + `99e0b4a6ade` HiveConf classloader 修复，改 `fe-connector-hms/HmsConfHelper.java`）——与本线程文件**不重叠**、无覆盖。 +- 另一 session 的 `mvn ... be-java-extensions package -am -T 1C` **污染共享 target**，一度令本地 UT 报 + 「cannot access 生成类」**假失败**（非本码问题；待其结束干净重跑即绿）。 +- **起步/动码前先探**：`git log`/`git status`、运行中 maven（`ps aux | grep plexus.classworlds.launcher`）、近 90s mtime； + 发现活跃即优先只写新文件 + 小步快提交；build 假失败先排查并发污染再怀疑本码（memory + `concurrent-sessions-shared-worktree-hazard`）。 + +--- + +## 8. e2e 欠账（用户自跑，勿丢，非静默） + +**批次 0/1/2 所有 e2e 均 live-gated（真集群）**，回归清单： +- 批次 0：H1–H4（转义值 / DATETIME / 非-hive-style 带 filter / MOR-JNI 混大小写读）。 +- 批次 1：equality-delete 统计 UNKNOWN / vended-region / s3tables 默认凭证链 / mc 分区缓存往返 / hive 大分区异步 split。 +- **批次 2（本轮新增）**：M3 = docker-hive `test_hive_partitions` `(approximate)inputSplitNum=6` + MaxCompute 无谓词 + ≥阈值分区表进 batch（结果与同步逐行一致）；M1 = docker-hive `test_hive_tablesample_p0` 结果不变式（**强基数缩减 + sample Date: Sat, 11 Jul 2026 14:43:48 +0800 Subject: [PATCH 238/334] [feat](catalog) hive-fs: add FileSystem.forLocation default for per-location concrete-FS resolution Add `default FileSystem forLocation(Location) { return this; }` to org.apache.doris.filesystem.FileSystem so a caller can resolve the concrete filesystem behind a scheme-routing facade -- e.g. to test `instanceof ObjFileSystem` before driving an object-store multipart upload, which the routing facade cannot answer without a location. A plain single-backend filesystem is its own per-location filesystem (default returns this); SpiSwitchingFileSystem overrides it (now @Override) to return the per-scheme delegate. Foundational seam for routing hive-connector I/O through the engine-provided Doris FileSystem (fe-filesystem) instead of bare Hadoop, aligning with Trino's TrinoFileSystem. Design: plan-doc/tasks/designs/FIX-HIVEFS-design.md. UT: FileSystemDefaultMethodsTest.forLocationDefaultReturnsThis (9/9 green, 0 checkstyle; fe-core test-compile SUCCESS). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../apache/doris/fs/SpiSwitchingFileSystem.java | 1 + .../org/apache/doris/filesystem/FileSystem.java | 15 +++++++++++++++ .../filesystem/FileSystemDefaultMethodsTest.java | 12 ++++++++++++ 3 files changed, 28 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/fs/SpiSwitchingFileSystem.java b/fe/fe-core/src/main/java/org/apache/doris/fs/SpiSwitchingFileSystem.java index 9257352238caae..fe7828416766e8 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/fs/SpiSwitchingFileSystem.java +++ b/fe/fe-core/src/main/java/org/apache/doris/fs/SpiSwitchingFileSystem.java @@ -104,6 +104,7 @@ public FileSystem forPath(String uri) throws IOException { } /** Resolves the appropriate {@link FileSystem} for the given {@link Location}. */ + @Override public FileSystem forLocation(Location location) throws IOException { return forPath(location.uri()); } diff --git a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystem.java b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystem.java index d4f194b4662a53..9f1225f1f513d4 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystem.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/main/java/org/apache/doris/filesystem/FileSystem.java @@ -56,6 +56,21 @@ default T requireCapability(Class capabilityType) { getClass().getSimpleName() + " does not support " + capabilityType.getSimpleName())); } + /** + * Resolves the concrete {@link FileSystem} that actually serves {@code location}. + * + *

    A plain, single-backend filesystem is the filesystem for every location, so the + * default returns {@code this}. A scheme-routing filesystem (e.g. {@code SpiSwitchingFileSystem}) + * overrides this to return the per-scheme delegate the location maps to. + * + *

    Callers use this when they need the concrete implementation rather than the routing + * facade — for example to test {@code instanceof ObjFileSystem} before driving an object-store + * multipart upload, which a routing facade cannot answer without knowing the location. + */ + default FileSystem forLocation(Location location) throws IOException { + return this; + } + boolean exists(Location location) throws IOException; void mkdirs(Location location) throws IOException; diff --git a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemDefaultMethodsTest.java b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemDefaultMethodsTest.java index 48edadfc56f0b3..272e2735c840ef 100644 --- a/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemDefaultMethodsTest.java +++ b/fe/fe-filesystem/fe-filesystem-api/src/test/java/org/apache/doris/filesystem/FileSystemDefaultMethodsTest.java @@ -220,4 +220,16 @@ void globListWithLimitThrowsUnsupportedByDefault() { Assertions.assertThrows(UnsupportedOperationException.class, () -> fs.globListWithLimit(Location.of("s3://b/*"), null, 0, 0)); } + + // --- forLocation --- + + @Test + void forLocationDefaultReturnsThis() throws IOException { + // A plain (non-routing) filesystem is its own per-location filesystem. This contract lets + // callers resolve the concrete impl (e.g. for an instanceof ObjFileSystem check) uniformly, + // whether the handle is a single-backend FS or a scheme-routing facade. + StubFileSystem fs = new StubFileSystem(List.of()); + Assertions.assertSame(fs, fs.forLocation(Location.of("s3://b/dir/a.csv"))); + Assertions.assertSame(fs, fs.forLocation(Location.of("hdfs://ns/user/a"))); + } } From 3b4f7477d34ef737fc7a41155bc14f39ee201d61 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 14:43:58 +0800 Subject: [PATCH 239/334] [feat](catalog) hive-fs: add ConnectorContext.getFileSystem(session) engine-owned FS accessor Add `default FileSystem getFileSystem(ConnectorSession) { return null; }` to the connector SPI. The engine returns a scheme-routing FileSystem backed by the catalog's parsed StorageProperties and the fe-filesystem providers (hdfs/s3/oss/cos/obs/azure/http/local/broker); the connector borrows it to list/read/write table data without bundling any Hadoop FileSystem impl. Contract (javadoc): engine-owned / connector-borrowed / connector MUST NOT close it. The session parameter mirrors Trino's create(ConnectorSession) and reserves per-user identity via getUser(), while the current impl resolves at catalog granularity (session not yet used to key a per-user filesystem). Default null matches the benign default of getBackendStorageProperties(), so every other connector -- and the no-op default context -- is unaffected. Design: plan-doc/tasks/designs/FIX-HIVEFS-design.md (fe-connector-spi test-compile SUCCESS, 0 checkstyle). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../doris/connector/spi/ConnectorContext.java | 30 +++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java index 082a095960b407..9adaf11d5617c9 100644 --- a/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java +++ b/fe/fe-connector/fe-connector-spi/src/main/java/org/apache/doris/connector/spi/ConnectorContext.java @@ -19,6 +19,8 @@ import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.filesystem.FileSystem; import org.apache.doris.filesystem.properties.StorageProperties; import java.util.Collections; @@ -332,6 +334,34 @@ default List getStorageProperties() { return Collections.emptyList(); } + /** + * Returns the engine's {@link FileSystem} for this catalog — a scheme-routing handle backed by the + * catalog's parsed {@link #getStorageProperties() storage properties} and the registered fe-filesystem + * providers (hdfs/s3/oss/cos/obs/azure/http/local/broker). A connector uses it to list, read, and write + * table data without bundling any Hadoop {@code FileSystem} implementation itself; the engine owns scheme + * routing and per-scheme classloader pinning, exactly as Trino's {@code TrinoFileSystemFactory.create(session)} + * hands the connector a {@code TrinoFileSystem}. + * + *

    Ownership. The returned filesystem is engine-owned and connector-borrowed: the engine + * builds and caches it per catalog and closes it when the catalog/context is torn down. A connector MUST NOT + * call {@link FileSystem#close()} on it. + * + *

    Identity. The {@code session} parameter mirrors Trino's {@code create(ConnectorSession)} shape and + * reserves per-user identity via {@link ConnectorSession#getUser()}. The current implementation resolves the + * filesystem at catalog granularity (the session is not yet used to key a per-user filesystem); when per-user + * identity lands, the engine will key the cache by identity. + * + *

    The default returns {@code null} (no engine-managed filesystem), so connectors that do not use it — and + * the no-op default context — are unaffected, matching the benign default of + * {@link #getBackendStorageProperties()}. + * + * @param session the query/connector session (reserved for per-user identity; may be null for catalog-level use) + * @return the catalog's engine-owned {@link FileSystem}, or {@code null} when the engine manages no storage + */ + default FileSystem getFileSystem(ConnectorSession session) { + return null; + } + /** * Best-effort removal of the EMPTY directory shells left behind after a connector drops a managed * table or database. The data + metadata FILES are already deleted by the connector's own drop (e.g. From cde4470b1bacb3c34982d6f0958f885107e29177 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 14:47:21 +0800 Subject: [PATCH 240/334] [doc](catalog) hive-fs: land design + task list; mark steps 0/1/2 done, repoint to engine impl Land the FIX-HIVEFS design doc and task list (previously untracked) and update HANDOFF: the three foundational SPI seams (revert of the hdfs-client band-aid, FileSystem.forLocation default, ConnectorContext.getFileSystem accessor) are done and committed; the next step is the fe-core engine implementation (DefaultConnectorContext.getFileSystem) followed by the read/ACID/write connector cutover and the hadoop-hdfs-client removal. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 30 ++++++- plan-doc/tasks/designs/FIX-HIVEFS-design.md | 97 +++++++++++++++++++++ plan-doc/tasks/task-list-HIVEFS.md | 87 ++++++++++++++++++ 3 files changed, 213 insertions(+), 1 deletion(-) create mode 100644 plan-doc/tasks/designs/FIX-HIVEFS-design.md create mode 100644 plan-doc/tasks/task-list-HIVEFS.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index de5fed50d08484..4a1faf05e85b42 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,6 +5,34 @@ --- +# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 引擎实现 HIVEFS-3） + +> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。** + +**根因**：连接器扫描/写全程用**裸 `org.apache.hadoop.fs.FileSystem`**（`HiveFileListingCache:162`/`HiveScanPlanProvider:272`/`HiveAcidUtil`/`HiveConnectorTransaction`），但 hive 插件类加载器隔离、lib/ 无 `DistributedFileSystem`（仅 hadoop-common，注册 Local/View/Har/Http、无 hdfs）。老 fe-core 经 `org.apache.doris.filesystem.FileSystem`+`DirectoryLister` 列文件（`HiveExternalMetaCache:392-396`），**新代码裸 Hadoop 是迁移走样**。非环境问题（hostname 可解析、metastore 通、已取到 location）。 + +**已签字决策（2026-07-11）**：① 走**正解 B**——引擎经 `ConnectorContext.getFileSystem(ConnectorSession)` 下发 Doris `FileSystem`（实为 `SpiSwitchingFileSystem`），连接器改调 `listFiles/exists/rename/delete/forLocation`、**删 `hadoop-hdfs-client`**；**对齐 Trino**（`TrinoFileSystemFactory.create(session)`→`TrinoFileSystem`，连接器不 bundle Hadoop、Hadoop 可选仅 HDFS）。② **一步到位**（读+ACID+写全转）。③ SPI 照 Trino 形状留 `session`（identity 经 `getUser()` **预留 per-user**，当前 catalog 级）。④ scope=**hive-only**（paimon/iceberg 经各自 `FileIO`，不动）。 + +**✅ 本轮完成 = 基础 SPI 缝 HIVEFS-0/1/2（均构建+靶向 UT 验证、0 checkstyle、已 commit）**： +- **HIVEFS-0**：撤销上会话未提交的 `hadoop-hdfs-client` 创可贴（pom 回 HEAD，无 commit）。 +- **HIVEFS-1** `0c4e0595f8f`：`FileSystem` 接口加 `default FileSystem forLocation(Location){return this}`（非切换 FS 即自身，`SpiSwitchingFileSystem` override 返回 per-scheme 委派）。UT `FileSystemDefaultMethodsTest` 9/9。用途：写路径 MPU 按 location 取具体 `ObjFileSystem`（HIVEFS-6）。 +- **HIVEFS-2** `3b4f7477d34`：`ConnectorContext` 加 `default FileSystem getFileSystem(ConnectorSession){return null}`+javadoc（**引擎所有/连接器借用/连接器不得 close**；session 对齐 Trino、identity 经 `getUser()` 预留 per-user；默认 null 对齐 `getBackendStorageProperties`）。 + +**⭐ 下一步 = HIVEFS-3（fe-core 引擎实现）**：`DefaultConnectorContext.getFileSystem(session)` 懒建 + 字段缓存 `new SpiSwitchingFileSystem(storagePropertiesSupplier.get())`，随 context 拆除 close。近零新件（已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`；范式 `HMSExternalCatalog:146`/`:348`）。之后：**HIVEFS-4 读扫描（+3+重部署 = 失败用例转绿）→ 5 ACID → 6 写路径（最险，落地前须多 agent 对抗红队）→ 7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 + +**已探明的去风险事实（勿重查）**： +- **引擎侧近零新件**——`DefaultConnectorContext` 已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`（`SpiSwitchingFileSystem` 构造入参），`HMSExternalCatalog:146`/`:348` 已这样用；`DefaultConnectorContext` 按 catalog 长生（`CatalogFactory:110`），有 `Connector.close()` 链可挂缓存 FS 的 close。 +- **TCCL 无需连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs **自钉**到自身插件 loader 再 `FileSystem.get`、finally 还原（注释即描述 “No FileSystem for scheme hdfs” 场景）。连接器去 jar 后任意 TCCL 调用皆安全。 +- 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction:809/1336`）→ `FileSystem.forLocation`（HIVEFS-1 已加：非切换返回 this、切换返回具体 FS）就位待 HIVEFS-6 用。 +- 连接器裸 Hadoop 面**很小**：`listStatus`×3 + `exists`×1 + `FileStatus` 字段（getPath/getLen/isDirectory/getModificationTime），全映射 `FileEntry`。 +- fe-filesystem 全 9 插件（hdfs/s3/oss/cos/obs/azure/http/local/broker）已部署 → hive 表任意后端天然支持（免 bundle hadoop-aws/huaweicloud，架构红利）。 + +**⚠ 待下个 session 核定**（task-list Open §）：`DefaultConnectorContext` close 挂点 · `HiveScanPlanProvider.buildHadoopConf`/conf 去留 · `HiveAcidUtilTest` 真-LocalFS→fake 迁移 · 写路径 commit/abort 时 FS/identity 捕获时机（begin 捕获）。 + +**⚠ 状态**：HIVEFS-0/1/2 已入库（code 2 commit `0c4e0595f8f`/`3b4f7477d34` + 本 doc 1 commit）；设计 doc/task-list 随本轮一并 commit。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。 + +--- + # 🎯 当前状态（2026-07-10） **⭐ 本轮 = Phase 2 原子翻闸完成：catalog 类型 `hms` 已接入插件 SPI，所有生产流程改走新连接器代码（plain-hive + iceberg-on-HMS + hudi-on-HMS）。未删任何旧代码（Phase 3 才删）。** 4 个独立 commit（均构建+靶向单测验证；净室对抗复核见下）： @@ -40,7 +68,7 @@ PR #65474（hive connector SPI 迁移）build **991951** 50 外表 case 失败 - commits：`03f4c12dffa`(H4 lowercase JNI 列名) · `39a279e7c26`(H1 hive+hudi unescape) · `cf540eebc3c`(H2 hive+hudi datetime 渲染) · `9c6fc584eb9`(H3 hudi use_hive_sync 感知源) · `f0ee2ab06d2`(test-hardening) · 各配 doc commit。 - **全量对抗复审(3 skeptic)= CLEAN**：四修正确/复合/无回归、范围完整（其它连接器免疫）、10 新测均可 RED。**登记残余（非本批修，pre-existing legacy parity）**：`use_hive_sync_partition=true`+非-hive-style 表 → hive-sync 臂 0 split；D-PRUNE/相对化时评估（见 `designs/FIX-H3-design.md`）。 -**⭐ 下一步（下个 session）：** +**⭐ 复核修复续（backlog；下个 session 首要任务是顶部 FIX-HIVEFS，本系列在其后/穿插）：** 1. **复核修复续批**（跟踪表）：**⭐ 批次 1（M5/M7/M6/M4/M2 连接器局部）全部 DONE**——5 条全绿，各配 RED-able 单测 + 独立 code/doc commit，设计存 `plan-doc/tasks/designs/FIX-M*-design.md`（一轮 recon+对抗红队 `wf_40498e52-19f` 打底，机制全 HEAD 确认、无 UNSOUND）。commits：`84f580c9075`(M5 表级行数恢复 equality gate，**推翻先前签字「不 gate」决定，用户 2026-07-11 已签字**；3 处 P6.6-FIX-H4 已批注 SUPERSEDED) · `f6de950e5bd`(M7 region 别名 4→10) · `03bd4f58187`(M6 s3tables 无存储回退默认凭证链 + data-plane region companion) · `c553c3c7696`+`fca288424fc`(M4 mc `MaxComputePartitionCache`，插件 zip 已验单 caffeine-2.9.3；**最终复核纠正 TTL 86400→600s 对齐旧版**) · `702153885ab`(M2 hive `supportsBatchScan`+`planScanForPartitionBatch`，**登记 BATCH-ACID-SYNC 永久 + BATCH-UNPRUNED-SYNC 由 M3 解**)。**最终对抗复核（`wf_542c60b9-001`，5 per-fix + 1 cross-cut）= M5/M6/M7/M2/cross-cut CLEAN、M4 命中 1 medium(TTL)已修 → 全 CLEAN。** **⭐ 批次 2（M3→M1，fe-core 通用节点）全部 DONE。M3 `6963de4124f`**——`shouldUseBatchMode` 的 `!isPruned`→`== NOT_PRUNED`，无谓词大分区表（MaxCompute + 翻闸 hive）恢复异步 batch split，**顺带解 M2 的 BATCH-UNPRUNED-SYNC 残余**（legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证；git `1da88365e85^` 取证 + 全 producer 枚举证闭合）。设计红队 `wf_811e6242-d8b`（3 lens）命中 1 blocker + 1 major 均已解：**反转**被前次评审特意锁定的 pinning 测试（`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`）、**登记 supersession**（`decisions-log` D-035 / `deviations-log` DV-019 的 LP-1「`!isPruned` 等价」判定已批注 SUPERSEDED）、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`（**用户 2026-07-11 签字：采用 SPI 统一分区数口径**，不给 hive 补 split-count 估算；对齐 MaxCompute/Trino）。`PluginDrivenScanNodeBatchModeTest` 12/12 绿。设计 `designs/FIX-M3-design.md`。**⚠ 本轮踩并行 session 构建污染坑**（`be-java-extensions package -am -T 1C` 污染共享 target 报「cannot access 生成类」假失败，非本码；待其结束干净重跑 12/12）。**M1 DONE `17b432dc1e1`**——翻闸 hive 上 `TABLESAMPLE` 被静默丢弃(全表扫)。设计红队 `wf_32decfa0-349` **推翻原"通用采样"方案(UNSOUND)**:`Split.getLength()` 语义因连接器而异(MaxCompute 默认 byte_size/Paimon JNI range 报 -1、MaxCompute row_offset 报行数)→ 盲目按字节采样出乱结果;**scope 更正为 hive-only 回归**(只有 hive 曾采样)。→ 改**连接器 opt-in**(`ConnectorScanPlanProvider.supportsTableSample()` 默认 false、仅 `HiveScanPlanProvider` true;**用户 2026-07-11 签字 scope=hive-only**,对齐 Trino `applySample` + `supportsBatchScan` 先例)。translator 通用转发 + `PluginDrivenScanNode.sampleSplits`(仅 `applySample` 时,legacy `selectFiles` 端口)+ 非支持连接器 no-op+WARN + 两 gate 门(COUNT/batch 抑制,挂 `applySample`)。`PluginDrivenScanNodeTableSampleTest` 6/6 + BatchMode 12/12 + hive 285/285 绿、0 checkstyle、import 门净。设计 `designs/FIX-M1-design.md`。**⭐ 批次 2(M3+M1)全部 DONE。下一步 = 批次 3(M8 发布工具/文档 + L1 import 门禁)**;之后批次 4(低危连接器 L3–L20)… ⏸ 决策类(L2/L10/L12/L20)先问用户。 2. **e2e 回归（用户自跑，勿丢，非静默）**：**批次 0 全部 e2e 均 live-gated**（含转义值/DATETIME/非-hive-style 带 filter/MOR-JNI 混大小写读），须真集群回归（memory `hms-iceberg-delegation-needs-e2e`）；连同翻闸原有欠账：异构 `type=hms` 目录读/写/DDL/procedure/MTMV/time-travel/@incr；从库事件同步陈旧；Kerberos-HMS 冒烟；升级 GSON replay；耦合缝行；hive 视图。完整矩阵：`hms-cutover-execution-plan-2026-07-10.md` §4/§5 + `hms-spi-cutover-flip-2026-07-10.md` §5。 diff --git a/plan-doc/tasks/designs/FIX-HIVEFS-design.md b/plan-doc/tasks/designs/FIX-HIVEFS-design.md new file mode 100644 index 00000000000000..7feccd5420dce4 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-HIVEFS-design.md @@ -0,0 +1,97 @@ +# FIX-HIVEFS — hive 连接器改经引擎下发 `FileSystem`（fe-filesystem）替代裸 Hadoop，去 `hadoop-hdfs-client`（对齐 Trino） + +> 触发（2026-07-11）：本地 hive 回归 `external_table_p0/hive/test_string_dict_filter` q01 失败 `Failed to resolve filesystem for hdfs://...`。经取回 fe.log/fe.warn.log + fe-filesystem 拓扑排查 + 类加载器/TCCL 验证 + **Trino 方案对标**，定为架构性缺口而非环境/文案问题。用户拍板走"正解 B、一步到位、SPI 照 Trino 形状留 identity 参"。 +> 范围 = **仅 hive 连接器**（读扫描 + ACID + 写路径全部）。paimon/iceberg 不动。 + +## Problem + +- 失败栈（`fe.log:7657/7677/7698`、`fe.warn.log`）： + ``` + RuntimeException: Failed to resolve filesystem for hdfs://hadoop-master-doris-shared:8320/user/doris/preinstalled_data/parquet_table/test_string_dict_filter_parquet + → DorisConnectorException (HiveFileListingCache.listFromFileSystem:168) + → org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "hdfs" + at FileSystem.getFileSystemClass(:3581) ← HiveFileListingCache.listFromFileSystem:162 (FileSystem.get) + ``` +- **非环境问题**：`hadoop-master-doris-shared` 在 `/etc/hosts` 可解析（`172.20.32.136`）；metastore（thrift）连通、已取到表 location；走的是新 `fe-connector-hive`（`HiveFileListingCache:168`）。 +- **直接原因（打包）**：`output/fe/plugins/connector/hive/lib/` 无任何 jar 含 `org.apache.hadoop.hdfs.DistributedFileSystem`；两处 `META-INF/services/org.apache.hadoop.fs.FileSystem`（`hadoop-common`、`hive-catalog-shade`）只注册 Local/View/Har/Http/NullScan/ProxyLocal，**无 hdfs**。paimon/iceberg 插件都自带 `hadoop-hdfs-client-3.4.2.jar`，唯 hive 缺。 +- **深层（架构）**：连接器扫描/写全程用裸 `org.apache.hadoop.fs.FileSystem`（`HiveFileListingCache:162/171`、`HiveScanPlanProvider:272`、`HiveAcidUtil:170/255/298`、`HiveConnectorTransaction` 写/删/MPU）。插件类加载器隔离 → 缺 hdfs 实现即 `No FileSystem for scheme "hdfs"`。而**老 fe-core 经 `org.apache.doris.filesystem.FileSystem` + `DirectoryLister` 列文件**（`HiveExternalMetaCache.loadFiles():392-396` → `FileSystemCache.getFileSystem` → `FileSystemDirectoryLister`），从不裸 `FileSystem.get`。**新代码裸 Hadoop 是迁移走样**。 + +## 备选与否决 + +- **A（bundle `hadoop-hdfs-client`）**：能修 hdfs，对齐 paimon/iceberg 形态。但复制 `fe-filesystem-hdfs` 插件**已拥有**的 hdfs 依赖；且对象存储 hive 表会再撞 `No FileSystem for scheme s3a/obs`，要继续 bundle `hadoop-aws`/`hadoop-huaweicloud`（重且版本敏感），连接器越抱越多。**否**（仅作过渡）。 +- **连接器自建 FileSystem（自跑 `ServiceLoader`）**：跨插件不可行——FS provider 在**兄弟**插件 loader（`output/fe/plugins/filesystem/*`），共享类路径（`output/fe/lib`）**无任何 provider**，且 `DirectoryPluginRuntimeManager.ServiceResourceFilteringParentClassLoader:443` 刻意屏蔽父级 service 描述符。写路径 `HiveConnectorTransaction.resolveObjectStoreFileSystem:788` 正是此坏 stub（注释自认 cutover-time concern）。**否**。 + +## Trino 对标（本方案的镜像来源） + +Trino 已走完同一条路（[Decouple Trino from Hadoop #15921](https://github.com/trinodb/trino/issues/15921)、[Out with the old file system, 2025](https://trino.io/blog/2025/02/10/old-file-system.html)）： + +| Trino | 本方案（Doris） | +|---|---| +| 引擎注入 `TrinoFileSystemFactory`；`HiveMetadata.fileSystemFactory` | `ConnectorContext.getFileSystem(session)`（引擎提供，连接器已持 `context`） | +| `factory.create(ConnectorSession)` → `TrinoFileSystem` | `context.getFileSystem(ConnectorSession)` → `org.apache.doris.filesystem.FileSystem` | +| `TrinoFileSystem`：`listFiles/listDirectories/newInputFile/deleteFile/renameFile`（全 `Location`） | `FileSystem`：`list/listFiles/listDirectories/exists/newInputFile/delete/rename`（全 `Location`，近一一对应） | +| scheme 路由 + native 实现在引擎侧；Hadoop 可选仅 HDFS | `SpiSwitchingFileSystem` + `fe-filesystem-{hdfs,s3,oss,cos,obs,azure,http,local,broker}` 插件 | + +因此本方案让连接器对 hdfs/s3/oss/cos/obs/azure **全部免 bundle**（Q1：其它 fs 类型天然支持，为架构红利）。 + +## Design + +### 1. SPI 契约 +- **`ConnectorContext.getFileSystem(ConnectorSession session)`** → `org.apache.doris.filesystem.FileSystem`，`default` 返回 `null`（对齐 `getBackendStorageProperties()` 良性默认；maxcompute 等不用存储的连接器不受影响）。契约：**引擎所有、连接器借用、连接器不得 `close()`**。`session` 形状对齐 Trino `create(session)`，identity 经 `session.getUser()` **预留 per-user**；当前实现按 catalog 级建（session 暂忽略）。 +- **`org.apache.doris.filesystem.FileSystem` 接口加 `default FileSystem forLocation(Location loc) throws IOException { return this; }`**；`SpiSwitchingFileSystem.forLocation:107`（已 public）改 `@Override`。用途：写路径 MPU 需按 location 取**具体** `ObjFileSystem`（`instanceof ObjFileSystem`，`HiveConnectorTransaction:809/1336`）——非切换 FS 返回自己，切换 FS 返回 per-scheme 委派。capability() 无 location 参、不能按位置路由，故用 forLocation。 + +### 2. 引擎实现（fe-core） +- `DefaultConnectorContext.getFileSystem(session)`：**懒建 + 按 context 缓存**一个 `SpiSwitchingFileSystem(storagePropertiesSupplier.get())`（字段），随 context/catalog 拆除时 `close`（沿 `Connector.close()` 链）。近零新件——`DefaultConnectorContext` 已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`，且 `HMSExternalCatalog:146`/`IcebergMetadataOps:472`/`DefaultConnectorContext:348` 已这样用。缓存 = 对齐老 `FileSystemCache` 的 per-catalog 复用（避免每次重建/重认证）。 +- TCCL：**无需引擎/连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs 自钉到**自身**插件 loader（`DFSFileSystem.class.getClassLoader()`）再调 `FileSystem.get`、finally 还原（注释即描述"No FileSystem for scheme hdfs"场景）。故连接器（去 jar 后）任意 TCCL 调用皆安全。 + +### 3. 连接器改造（fe-connector-hive，一次性转全部裸 Hadoop I/O） + +| 现状（裸 Hadoop） | 改为（注入 `FileSystem`） | 位点 | +|---|---|---| +| `FileSystem.get(uri,conf)` + `fs.listStatus` | `injectedFs.listFiles(Location.of(loc))` | `HiveFileListingCache:162/171`、`HiveScanPlanProvider:272` | +| `FileStatus.getPath/getLen/isDirectory/getModificationTime` | `FileEntry.location/length/isDirectory/mtime` | 列文件后处理（`HiveFileStatus` DTO 不变，仅换来源） | +| `fs.exists`、`fs.listStatus` | `injectedFs.exists/list` | `HiveAcidUtil:170/255/298` | +| `resolveObjectStoreFileSystem`（坏 ServiceLoader stub） | `context.getFileSystem(session).forLocation(loc)` | `HiveConnectorTransaction:784` | + +- `HiveFileListingCache` 的 `DirectoryLister` seam 签名 `(location, Configuration)` → `(location, FileSystem)`；两种异常语义保留（systemic `DorisConnectorException` vs per-partition `HiveDirectoryListingException`）。 +- 写路径 `close()` **不再 close** 借用的引擎 FS（仅关自有资源）。 +- **pom**：删 `hadoop-hdfs-client`（撤销本会话工作区加的、**未提交**的那条）；`hadoop-common` **保留**（`Configuration`/`HiveConf`/HMS client 仍需）。改完全局 grep 确认 `org.apache.hadoop.fs.FileSystem` 在连接器 main 源零残留。 + +## Implementation Plan（单次改动，按可独立验证子步推进） + +1. **fe-filesystem-api**：`FileSystem` 加 `forLocation` default；`SpiSwitchingFileSystem` 加 `@Override`（方法体已存在）。 +2. **fe-connector-spi**：`ConnectorContext.getFileSystem(ConnectorSession)` default null + javadoc（所有权/借用/identity 预留）。 +3. **fe-core**：`DefaultConnectorContext.getFileSystem` 实现 + 字段缓存 + close 释放。 +4. **连接器·读**：`HiveFileListingCache` DirectoryLister 换 `FileSystem`；`HiveScanPlanProvider:272` 换；`FileStatus`→`FileEntry` 映射；列文件调用点改用 `context.getFileSystem(session)`（不再 `FileSystem.get`）。**注**：`buildHadoopConf()`/`Configuration` 若仍被格式/split 参数或传 BE 所需则保留（本步只摘除其"建 FileSystem"一职），实际去留由 writing-plans 逐点核定。 +5. **连接器·ACID**：`HiveAcidUtil` `exists`/`listStatus` 换。 +6. **连接器·写**：`HiveConnectorTransaction` `resolveObjectStoreFileSystem`→`getFileSystem(session).forLocation(loc)`；MPU `instanceof ObjFileSystem` 落到具体 FS；`close` 不关借用 FS；begin 时捕获所需 FS/identity。 +7. **pom**：删 `hadoop-hdfs-client`；全局 grep 校验零残留裸 `org.apache.hadoop.fs.FileSystem`。 +8. **构建 + 单测 + e2e**。 + +## Risk Analysis + +- **写路径最险**（事务/MPU/rename/delete 语义）：`forLocation` 精确取老代码期望的具体 FS；`ObjFileSystem` 接口不变；靠写 e2e 兜底。 +- **生命周期误 close**：连接器若 close 借用的引擎 FS → 引擎 FS 提前失效。对策：契约明确（引擎所有）+ `close()` 只关自有 + 审查。 +- **`DirectoryLister` 签名变** → `HiveFileListingCache` 单测适配（seam 本为可注入设计）。 +- **性能**：引擎按 catalog 缓存 `SpiSwitchingFileSystem`（其内再按 `StorageProperties` 身份缓存 per-scheme FS）；对齐老 `FileSystemCache`，非每次重建。 +- **session 暂忽略**：`getFileSystem(session)` 当前不按 user 建 FS（catalog 级）；javadoc 标注；per-user 落地时缓存须按 identity keying（届时 listing 缓存策略需复审，避免跨用户串读）。 +- **铁律核对**：`getFileSystem` 为**通用** SPI（非 source-specific，不在 fe-core 加 hive 分支）；连接器不解析属性（用 `session`/`context`）；对齐 Trino；TCCL 由 `DFSFileSystem` 自钉（memory `catalog-spi-plugin-tccl-classloader-gotcha` 既有 locus，本方案不新增连接器钉点）。 + +## Test Plan + +### Unit（连接器模块无 Mockito；用可注入 fake FileSystem） +- `HiveFileListingCache`：注入 fake `DirectoryLister`/`FileSystem`（seam 已支持），断言列文件过滤（目录、`_`/`.` 前缀）、两种异常语义、缓存/失效（REFRESH）不变。 +- 写路径：`resolveObjectStoreFileSystem` 现为 `protected` 可 override 注入 fake；补 `forLocation` 语义单测（非切换返回 this、切换返回具体 FS）。 +- 引擎：`DefaultConnectorContext.getFileSystem` 返回非空、缓存复用、close 释放。 + +### E2E（用户自跑，勿丢——新能力必配 e2e） +- 重打包 `fe-connector-hive` + fe-core + fe-filesystem-api/spi → 重部署 → 跑： + - `external_table_p0/hive/test_string_dict_filter`（读 hdfs，本失败用例）； + - hive insert/写套件（`external_table_p0/hive` 中 37 个含 INSERT）验证写路径转换（rename/delete/MPU）； + - 若有对象存储环境，抽查 s3/oss 后端 hive 表读（验证 scheme 路由红利）。 +- 断言与老实现逐位一致（读结果、写落盘、事务提交/回滚）。 + +## Open / Future（不属本次） +- per-user identity（`session.getUser()`）真正落地 + FS/listing 缓存按 identity keying。 +- paimon/iceberg 保持自 bundle `hadoop-hdfs-client`（它们经各自 `FileIO` 做 I/O，非 Doris `FileSystem`；本方案不动）。 +- 其它连接器（maxcompute 等）`getFileSystem` 默认 null，不受影响。 diff --git a/plan-doc/tasks/task-list-HIVEFS.md b/plan-doc/tasks/task-list-HIVEFS.md new file mode 100644 index 00000000000000..deb315719608ea --- /dev/null +++ b/plan-doc/tasks/task-list-HIVEFS.md @@ -0,0 +1,87 @@ +# Task List — FIX-HIVEFS：hive 连接器改经引擎下发 `FileSystem`（fe-filesystem），去 `hadoop-hdfs-client`（对齐 Trino） + +设计 + RCA：`plan-doc/tasks/designs/FIX-HIVEFS-design.md` +触发：本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"`（fe.log:7657/7698）。 +用户签字（2026-07-11）：走正解 B（引擎下发 FileSystem，对齐 Trino）· **一步到位**（读+ACID+写全转）· SPI 照 Trino 形状 `getFileSystem(ConnectorSession)`（identity 预留）· scope = **hive-only**（paimon/iceberg 不动）。 + +> **性质**：单一逻辑改动，但按"每子步 = 独立 commit + 靶向 UT"推进（`AGENT-PLAYBOOK` 纪律）。编译/依赖序：fe-filesystem-api → fe-connector-spi → fe-core + fe-connector-hive。 +> **失败用例何时转绿**：HIVEFS-3（引擎）+ HIVEFS-4（读路径）落地 + 重部署即绿（不必等 HIVEFS-7 删 jar）；HIVEFS-7 是达成"去依赖"终点。 + +--- + +## ⚠️ 起步第 0 步（必做） + +- [x] **HIVEFS-0 撤销过渡创可贴**（✅ DONE：pom 已回 HEAD，无 commit）：撤销本会话在 `fe/fe-connector/fe-connector-hive/pom.xml` 加的、**未提交**的 `hadoop-hdfs-client` 依赖（Option A 过渡；B 用引擎下发替代）。`git diff fe/fe-connector/fe-connector-hive/pom.xml` 应回到 HEAD。**先查并行 session**（`git log`/`git status`/运行中 maven/近 90s mtime，memory `concurrent-sessions-shared-worktree-hazard`）。 + +--- + +## 实现子步（各独立 commit） + +- [x] **HIVEFS-1（fe-filesystem-api，基础）** ✅ DONE `0c4e0595f8f`（UT `FileSystemDefaultMethodsTest` 9/9、fe-core test-compile SUCCESS、0 checkstyle）：`org.apache.doris.filesystem.FileSystem` 加 `default FileSystem forLocation(Location loc) throws IOException { return this; }`；`SpiSwitchingFileSystem.forLocation:107`（已 public，返回 `FileSystem`、throws IOException）加 `@Override`。 + - 为何：写路径 MPU 需按 location 取具体 `ObjFileSystem`（HIVEFS-6）。加 default 方法 = 向后兼容（DFSFileSystem/S3 等既有 impl 不破）。 + - UT：`forLocation` default 返回 this；`SpiSwitchingFileSystem.forLocation` 经 fake 返回 per-location 委派。 + - 校验：`mvn -o -pl :fe-filesystem-api -am test-compile`；fe-core 连带编译过。 + +- [x] **HIVEFS-2（fe-connector-spi，基础）** ✅ DONE `3b4f7477d34`（fe-connector-spi test-compile SUCCESS、0 checkstyle）：`ConnectorContext` 加 `default FileSystem getFileSystem(ConnectorSession session) { return null; }` + javadoc。 + - javadoc 写明：**引擎所有 / 连接器借用 / 连接器不得 close**；`session` 对齐 Trino `create(session)`，identity 经 `session.getUser()` **预留 per-user**，当前 catalog 级（session 暂忽略）。默认 null（对齐 `getBackendStorageProperties()` 良性默认）。 + - 校验：`mvn -o -pl :fe-connector-spi -am test-compile`。 + +- [ ] **HIVEFS-3（fe-core，引擎实现）**：`DefaultConnectorContext.getFileSystem(session)` 懒建 + 字段缓存 `new SpiSwitchingFileSystem(storagePropertiesSupplier.get())`，随 context 拆除 close。 + - 复用现件（已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`；范式见 `HMSExternalCatalog:146`/`DefaultConnectorContext:348`）。 + - **待核**：`DefaultConnectorContext` 的 close/teardown 挂点（沿 `Connector.close()` 链，`ConnectorContext.java:133` 提到 caller 转发 close）——把缓存 FS 的 close 挂上。 + - UT（fe-core 有 Mockito）：`getFileSystem` 非空、二次调用返回同一实例（缓存）、close 释放。 + - 校验：`mvn -o -pl fe-core -am test-compile`。 + +- [ ] **HIVEFS-4（连接器·读扫描列文件）**： + - `HiveFileListingCache`：`DirectoryLister` seam 签名 `(String location, Configuration)` → `(String location, FileSystem)`；`listFromFileSystem` 用 `fs.listFiles(Location.of(location))` 替代 `FileSystem.get`+`listStatus`；保留目录/`_`/`.` 前缀过滤 + 两异常语义（systemic `DorisConnectorException` vs per-partition `HiveDirectoryListingException`）。`FileStatus`→`FileEntry`（getPath→location、getLen→length、isDirectory→isDirectory、getModificationTime→mtime）；`HiveFileStatus` DTO 不变。 + - `HiveScanPlanProvider`：`:272` `FileSystem.get(partPath...)` 换；列文件调用点传 `context.getFileSystem(session)`（不再 `FileSystem.get`）。**待核**：`buildHadoopConf()`/`Configuration` 若仍被格式/split 参数或传 BE 所需则保留（本步只摘"建 FileSystem"一职）。 + - UT（连接器无 Mockito，用 recording fake）：注入 fake `FileSystem`/`DirectoryLister`，断言过滤、两异常、缓存/失效（REFRESH）不变，RED-able。 + - **此步 + HIVEFS-3 + 重部署 = 失败用例转绿**。 + - 校验：`mvn -o -pl :fe-connector-hive -am test-compile -Dtest=HiveFileListingCache* -DfailIfNoTests=false`。 + +- [ ] **HIVEFS-5（连接器·ACID）**：`HiveAcidUtil:170/255/298` `fs.exists`/`fs.listStatus` → 注入 `FileSystem` 的 `exists(Location)`/`list`/`listFiles`。 + - **⚠ 测试基建待解**：`HiveAcidUtilTest` 现对**真 Hadoop LocalFileSystem** 跑（见 `fe-connector-hive/pom.xml:128-137` commons-lang test 注释）。转 Doris `FileSystem` 后须换成 fake Doris `FileSystem` 或 `fe-filesystem-local` 的 local 实现——下个 session 定夺注入方式。 + - 校验：`-Dtest=HiveAcidUtil* -DfailIfNoTests=false`。 + +- [ ] **HIVEFS-6（连接器·写路径，最险）**：`HiveConnectorTransaction`： + - `resolveObjectStoreFileSystem`（坏 ServiceLoader stub，:784-793）→ `context.getFileSystem(session).forLocation(loc)`；MPU `instanceof ObjFileSystem`（:809/1336）落到 `forLocation` 返回的具体 FS；`objFs.completeMultipartUpload`/`getObjStorage().abortMultipartUpload` 不变。 + - 事务内 delete/rename/exists 等裸 Hadoop 调用一并换注入 FS。 + - `close()`：**不再 close 借用的引擎 FS**（仅关自有资源）。 + - **待核**：commit/abort 时机 session 是否还在 → 在 `beginWrite`（:207，已捕获 user）时捕获所需 FS/identity 供后续复用。 + - UT：`resolveObjectStoreFileSystem` 为 `protected` 可 override 注入 fake ObjFileSystem；断言 MPU complete/abort 路径 + close 不关借用 FS。 + - 校验：`-Dtest=HiveConnectorTransaction*,HiveWrite* -DfailIfNoTests=false`。 + +- [ ] **HIVEFS-7（pom + 去 jar，达成终点）**：删 `fe-connector-hive/pom.xml` 的 `hadoop-hdfs-client`；**保留** `hadoop-common`（`Configuration`/`HiveConf`/HMS client 仍需）。 + - 全局 grep 校验连接器 main 源 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get` **零残留**（`new Path` 若仅路径拼接可留，但确认不再 FS I/O）。 + - 打包校验：`output/fe/plugins/connector/hive/lib/` 无 `hadoop-hdfs-client`；插件 zip 仍完整。 + - 依赖：HIVEFS-4/5/6 全完成后做。 + +- [ ] **HIVEFS-8（全量构建 + UT + e2e 交接）**：fe-filesystem-api + fe-connector-spi + fe-core + fe-connector-hive 全量 build（后台跑读 LOG，`BUILD SUCCESS`/`Tests run`/checkstyle），全 UT 绿、0 checkstyle、import 门净。 + +--- + +## 设计红队（落地前） + +- [ ] 按 `clean-room-adversarial-review-pref`：实现前对设计做多 agent 对抗红队（重点：写路径 MPU/rename/delete 语义等价、生命周期误 close、`forLocation` 与 SpiSwitchingFileSystem 缓存交互、session-ignore 的 catalog 级正确性）。 + +## e2e（用户自跑，勿丢——新能力必配 e2e，memory `hms-iceberg-delegation-needs-e2e`） + +- [ ] 重打包 + 重部署后跑： + - `external_table_p0/hive/test_string_dict_filter`（读 hdfs，本失败用例）全绿； + - `external_table_p0/hive` 中 37 个含 INSERT 的写套件（验证 rename/delete/MPU 转换）； + - 若有对象存储环境：抽查 s3/oss 后端 hive 表读（验证 scheme 路由红利、免 bundle hadoop-aws/huaweicloud）； + - 断言与老实现逐位一致（读结果、写落盘、事务提交/回滚）。 + +## Open / 待下个 session 核定（勿丢） + +1. `DefaultConnectorContext` 缓存 FS 的 close 挂点（Connector.close 链）。 +2. `HiveScanPlanProvider.buildHadoopConf()`/`Configuration` 在去 FS.get 后的去留（格式/split/传 BE 是否仍需）。 +3. `HiveAcidUtilTest` 从真 LocalFileSystem 迁到 fake/`fe-filesystem-local` 的注入方式。 +4. 写路径 commit/abort 时 FS/identity 的捕获时机（begin 时捕获）。 +5. `forLocation` 加到接口后与现有非切换 impl 的兼容（default `return this` 覆盖）。 + +## Future（不属本次） + +- per-user identity（`session.getUser()`）真正落地 + FS/listing 缓存按 identity keying。 +- paimon/iceberg 维持自 bundle `hadoop-hdfs-client`（经各自 `FileIO`，非 Doris `FileSystem`）。 +- 其它连接器（maxcompute 等）`getFileSystem` 默认 null，不受影响。 From a8ed72f2650000f58cf79a2338a4eb97e07f4f63 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 15:01:52 +0800 Subject: [PATCH 241/334] [feat](catalog) hive-fs: implement DefaultConnectorContext.getFileSystem (lazy per-catalog FS + teardown close) Implement the engine side of the connector filesystem SPI. getFileSystem lazily builds and caches one scheme-routing SpiSwitchingFileSystem over the catalog's storage properties, reused across scans (mirrors the legacy per-catalog FileSystemCache; avoids rebuilding/re-authenticating per call). Returns null when the catalog has no storage (empty storage map), matching the benign defaults of getBackendStorageProperties() / cleanupEmptyManagedLocation(). Lifecycle: the context is engine-owned and SHARED by the connector and any sibling it builds (createSiblingConnector passes the same context), so the CATALOG is the single owner that closes it -- not the connectors, which only borrow the FS. DefaultConnectorContext now implements Closeable: close() forwards to the cached FS, is idempotent, and drops the ref so a post-teardown getFileSystem returns null. PluginDrivenExternalCatalog stashes the context it creates and closes it at the two existing connector-teardown sites (onClose and the connector replacement in initLocalObjectsImpl), after the connector releases its borrowed reference. No-op for non-hive plugin catalogs (they never call getFileSystem), so this is inert until the connector read path is wired next. UT: DefaultConnectorContextFileSystemTest (lazy build, single-instance cache, null-on-no-storage, idempotent close forwarding) 4/4; existing context/catalog suites 24/24 total, 0 checkstyle, fe-core BUILD SUCCESS. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../connector/DefaultConnectorContext.java | 70 ++++++++++- .../PluginDrivenExternalCatalog.java | 43 ++++++- ...DefaultConnectorContextFileSystemTest.java | 113 ++++++++++++++++++ 3 files changed, 220 insertions(+), 6 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java index 61249a02e9f8b4..9041ab432b0533 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/connector/DefaultConnectorContext.java @@ -27,6 +27,7 @@ import org.apache.doris.common.util.LocationPath; import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorHttpSecurityHook; +import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.spi.ConnectorBrokerAddress; import org.apache.doris.connector.spi.ConnectorContext; import org.apache.doris.connector.spi.ConnectorMetaInvalidator; @@ -48,6 +49,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.io.Closeable; import java.io.IOException; import java.util.ArrayList; import java.util.Collections; @@ -66,7 +68,7 @@ *

    Provides the minimal catalog-level context that connector providers need * during creation. Additional context fields can be added here as the SPI evolves. */ -public class DefaultConnectorContext implements ConnectorContext { +public class DefaultConnectorContext implements ConnectorContext, Closeable { private static final Logger LOG = LogManager.getLogger(DefaultConnectorContext.class); @@ -86,6 +88,15 @@ public class DefaultConnectorContext implements ConnectorContext { // not wire it (non-plugin / 2-3-4-arg) — those yield an empty storage list, correct parity. private final Supplier> rawStoragePropsSupplier; + // Engine-owned, per-catalog Doris FileSystem (a scheme-routing SpiSwitchingFileSystem over the catalog's + // storage properties), lazily built on the first getFileSystem() and closed on catalog teardown (close()). + // Connectors BORROW it and must not close it (see ConnectorContext.getFileSystem javadoc); siblings built + // via createSiblingConnector share this same context, so there is exactly one cached FS per catalog. Guarded + // by fsLock; the field is dropped to null on close so a post-teardown getFileSystem() returns null. + private final Object fsLock = new Object(); + private volatile FileSystem catalogFileSystem; + private volatile boolean closed; + private final ConnectorHttpSecurityHook httpSecurityHook = new ConnectorHttpSecurityHook() { @Override public void beforeRequest(String url) throws Exception { @@ -267,6 +278,63 @@ public List getStorage return FileSystemFactory.bindAllStorageProperties(rawCatalogProps); } + @Override + public FileSystem getFileSystem(ConnectorSession session) { + // Engine-owned, per-catalog scheme-routing FileSystem, lazily built and cached so repeated scans reuse + // one instance (avoids rebuilding/re-authenticating per call, mirroring the legacy per-catalog + // FileSystemCache). The session is accepted for the Trino-shaped SPI but not yet used to key a per-user + // filesystem — the current build is catalog-level. Returns null when the catalog has no storage + // machinery (empty storage map: non-plugin ctor / credential-less warehouse), matching the benign + // defaults of getBackendStorageProperties() and cleanupEmptyManagedLocation(). + FileSystem fs = catalogFileSystem; + if (fs != null) { + return fs; + } + synchronized (fsLock) { + if (closed) { + return null; + } + if (catalogFileSystem == null) { + Map storageProps = storagePropertiesSupplier.get(); + if (storageProps == null || storageProps.isEmpty()) { + return null; + } + catalogFileSystem = buildCatalogFileSystem(storageProps); + } + return catalogFileSystem; + } + } + + /** + * Builds the catalog's scheme-routing filesystem from its storage properties. Extracted so tests can + * inject a recording fake without real storage/FS wiring (mirrors the {@code @VisibleForTesting} seams + * used elsewhere in this class). + */ + @VisibleForTesting + FileSystem buildCatalogFileSystem(Map storageProps) { + return new SpiSwitchingFileSystem(storageProps); + } + + /** + * Closes the cached catalog filesystem, if one was built. Idempotent. Called by the engine when the + * catalog/context is torn down (connector replacement or catalog close); connectors must never call it. + */ + @Override + public void close() throws IOException { + FileSystem fs; + synchronized (fsLock) { + if (closed) { + return; + } + closed = true; + fs = catalogFileSystem; + catalogFileSystem = null; + } + if (fs != null) { + fs.close(); + } + } + @Override public String normalizeStorageUri(String rawUri) { // No vended token → normalize against the catalog's static storage map (behavior unchanged). diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java index a9c5c5195c78b7..7d6ec7c5bc8d05 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalCatalog.java @@ -89,6 +89,12 @@ public class PluginDrivenExternalCatalog extends ExternalCatalog { // via makeSureInitialized() → initLocalObjectsImpl(), or resetToUninitialized() → onClose(). private transient volatile Connector connector; + // The engine-owned context shared by the connector (and any sibling it builds via createSiblingConnector). + // Held so the catalog can close the context's cached engine FileSystem (DefaultConnectorContext.getFileSystem) + // on teardown -- connectors only borrow that FS and must not close it. Null until the real connector is built + // (the lightweight CatalogFactory context is not tracked here; its FS is never built). + private transient volatile DefaultConnectorContext connectorContext; + /** No-arg constructor for GSON deserialization. */ public PluginDrivenExternalCatalog() { } @@ -117,6 +123,9 @@ protected void initLocalObjectsImpl() { // The connector created by CatalogFactory used a lightweight context // without auth (the catalog didn't exist yet); we replace it now. Connector oldConnector = connector; + // Capture the old context before createConnectorFromProperties() overwrites connectorContext, so we can + // close its cached FileSystem when the connector is actually replaced. + DefaultConnectorContext oldContext = connectorContext; Connector newConnector = createConnectorFromProperties(); if (newConnector != null) { connector = newConnector; @@ -129,6 +138,10 @@ protected void initLocalObjectsImpl() { LOG.warn("Failed to close old connector during re-initialization " + "for catalog {}", name, e); } + // ...and close the replaced context's cached engine FileSystem (never the live one). + if (oldContext != null && oldContext != connectorContext) { + closeConnectorContextQuietly(oldContext); + } } } if (connector == null) { @@ -166,11 +179,14 @@ protected Connector createConnectorFromProperties() { // This handles image deserialization of old resource-backed catalogs whose // properties never contained "type" (it was derived from the Resource object). String catalogType = getType(); - return ConnectorFactory.createConnector(catalogType, - catalogProperty.getProperties(), - new DefaultConnectorContext(name, id, this::getExecutionAuthenticator, - () -> catalogProperty.getStoragePropertiesMap(), - catalogProperty::getEffectiveRawStorageProperties)); + // Build the context up front and stash it so the catalog can close its cached engine FileSystem on + // teardown (onClose / connector replacement). The connector — and any sibling it builds — shares this + // one context instance, so there is a single cached FS per catalog. + DefaultConnectorContext context = new DefaultConnectorContext(name, id, this::getExecutionAuthenticator, + () -> catalogProperty.getStoragePropertiesMap(), + catalogProperty::getEffectiveRawStorageProperties); + this.connectorContext = context; + return ConnectorFactory.createConnector(catalogType, catalogProperty.getProperties(), context); } @Override @@ -1188,6 +1204,17 @@ private static String legacyLogTypeToCatalogType(InitCatalogLog.Type logType) { } } + private void closeConnectorContextQuietly(DefaultConnectorContext context) { + if (context == null) { + return; + } + try { + context.close(); + } catch (IOException e) { + LOG.warn("Failed to close connector context filesystem for catalog {}", name, e); + } + } + @Override public void onClose() { super.onClose(); @@ -1199,5 +1226,11 @@ public void onClose() { } connector = null; } + // Close the shared context's cached engine FileSystem AFTER the connector(s) release their borrowed + // reference to it. No-op when no FS was ever built (e.g. non-hive plugin catalogs never call + // getFileSystem()). + DefaultConnectorContext contextToClose = connectorContext; + connectorContext = null; + closeConnectorContextQuietly(contextToClose); } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java new file mode 100644 index 00000000000000..2f8aa6356afa70 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/connector/DefaultConnectorContextFileSystemTest.java @@ -0,0 +1,113 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector; + +import org.apache.doris.datasource.property.storage.StorageProperties; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.fs.SpiSwitchingFileSystem; +import org.apache.doris.kerberos.ExecutionAuthenticator; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Map; +import java.util.function.Supplier; + +/** + * HIVEFS-3: pins {@link DefaultConnectorContext#getFileSystem} — the engine-owned, per-catalog FileSystem is + * built lazily, cached (one instance reused across scans), returns {@code null} when the catalog has no + * storage, and is closed exactly once on teardown. The connector read/write paths borrow this FS and must + * not close it, so the engine owning close is load-bearing. + */ +public class DefaultConnectorContextFileSystemTest { + + private static final Supplier NOOP_AUTH = () -> new ExecutionAuthenticator() {}; + + /** Records close() so the test can assert the engine forwards teardown to the cached filesystem. */ + private static final class RecordingFileSystem extends SpiSwitchingFileSystem { + private int closeCount; + + private RecordingFileSystem() { + super((FileSystem) null); // test-delegate ctor; no path is ever routed through it + } + + @Override + public void close() { + closeCount++; + } + } + + /** Context that intercepts the FS build with a recording fake (no real storage/FS wiring needed). */ + private static final class RecordingContext extends DefaultConnectorContext { + private final RecordingFileSystem fs = new RecordingFileSystem(); + private int buildCount; + + private RecordingContext(Supplier> storageSupplier) { + super("c", 1L, NOOP_AUTH, storageSupplier); + } + + @Override + FileSystem buildCatalogFileSystem(Map storageProps) { + buildCount++; + return fs; + } + } + + private static Map nonEmptyStorage() { + // The value is irrelevant — RecordingContext overrides the actual FS build; only non-emptiness matters, + // because getFileSystem returns null on an empty storage map. + return Collections.singletonMap(StorageProperties.Type.HDFS, (StorageProperties) null); + } + + @Test + public void returnsNullWhenCatalogHasNoStorage() { + // 2-arg ctor -> empty storage map -> no engine-managed filesystem (parity with + // getBackendStorageProperties). MUTATION: building an FS over the empty map -> non-null -> red. + Assertions.assertNull(new DefaultConnectorContext("c", 1L).getFileSystem(null)); + } + + @Test + public void lazilyBuildsAndReusesSingleInstance() { + RecordingContext ctx = new RecordingContext(DefaultConnectorContextFileSystemTest::nonEmptyStorage); + FileSystem first = ctx.getFileSystem(null); + FileSystem second = ctx.getFileSystem(null); + Assertions.assertNotNull(first); + Assertions.assertSame(first, second, "getFileSystem must cache and reuse one instance"); + Assertions.assertEquals(1, ctx.buildCount, "the catalog filesystem must be built exactly once"); + } + + @Test + public void closeForwardsToCachedFileSystemAndIsIdempotent() throws Exception { + RecordingContext ctx = new RecordingContext(DefaultConnectorContextFileSystemTest::nonEmptyStorage); + ctx.getFileSystem(null); // build + cache + ctx.close(); + ctx.close(); // idempotent + Assertions.assertEquals(1, ctx.fs.closeCount, "close must forward to the cached FS exactly once"); + // After teardown the context yields no filesystem and does not rebuild. + Assertions.assertNull(ctx.getFileSystem(null)); + Assertions.assertEquals(1, ctx.buildCount, "getFileSystem must not rebuild after close"); + } + + @Test + public void closeWithoutGetFileSystemIsNoop() throws Exception { + RecordingContext ctx = new RecordingContext(DefaultConnectorContextFileSystemTest::nonEmptyStorage); + ctx.close(); // never built a filesystem + Assertions.assertEquals(0, ctx.fs.closeCount, "close must not touch a filesystem that was never built"); + } +} From 14169366b7603beb340c3b54db2559b47c4616cf Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 15:04:20 +0800 Subject: [PATCH 242/334] [doc](catalog) hive-fs: mark engine impl done, repoint to connector read path HIVEFS-3 (DefaultConnectorContext.getFileSystem + catalog-owned teardown close) landed and verified; tick it in the task list, resolve the close-hook and forLocation-compat open items, and repoint HANDOFF's next step to the connector read-scan cutover (HiveFileListingCache / HiveScanPlanProvider), which together with the engine impl turns the failing test_string_dict_filter case green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 13 +++++++------ plan-doc/tasks/task-list-HIVEFS.md | 20 ++++++++++---------- 2 files changed, 17 insertions(+), 16 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4a1faf05e85b42..faed21adff22e4 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,7 @@ --- -# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 引擎实现 HIVEFS-3） +# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 读扫描 HIVEFS-4） > 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。** @@ -13,23 +13,24 @@ **已签字决策（2026-07-11）**：① 走**正解 B**——引擎经 `ConnectorContext.getFileSystem(ConnectorSession)` 下发 Doris `FileSystem`（实为 `SpiSwitchingFileSystem`），连接器改调 `listFiles/exists/rename/delete/forLocation`、**删 `hadoop-hdfs-client`**；**对齐 Trino**（`TrinoFileSystemFactory.create(session)`→`TrinoFileSystem`，连接器不 bundle Hadoop、Hadoop 可选仅 HDFS）。② **一步到位**（读+ACID+写全转）。③ SPI 照 Trino 形状留 `session`（identity 经 `getUser()` **预留 per-user**，当前 catalog 级）。④ scope=**hive-only**（paimon/iceberg 经各自 `FileIO`，不动）。 -**✅ 本轮完成 = 基础 SPI 缝 HIVEFS-0/1/2（均构建+靶向 UT 验证、0 checkstyle、已 commit）**： +**✅ 本轮完成 = 基础 SPI 缝 + 引擎实现 HIVEFS-0/1/2/3（均构建+靶向 UT 验证、0 checkstyle、已 commit）**： - **HIVEFS-0**：撤销上会话未提交的 `hadoop-hdfs-client` 创可贴（pom 回 HEAD，无 commit）。 - **HIVEFS-1** `0c4e0595f8f`：`FileSystem` 接口加 `default FileSystem forLocation(Location){return this}`（非切换 FS 即自身，`SpiSwitchingFileSystem` override 返回 per-scheme 委派）。UT `FileSystemDefaultMethodsTest` 9/9。用途：写路径 MPU 按 location 取具体 `ObjFileSystem`（HIVEFS-6）。 - **HIVEFS-2** `3b4f7477d34`：`ConnectorContext` 加 `default FileSystem getFileSystem(ConnectorSession){return null}`+javadoc（**引擎所有/连接器借用/连接器不得 close**；session 对齐 Trino、identity 经 `getUser()` 预留 per-user；默认 null 对齐 `getBackendStorageProperties`）。 +- **HIVEFS-3** `a8ed72f2650`：`DefaultConnectorContext.getFileSystem` 懒建+字段缓存一个 per-catalog `SpiSwitchingFileSystem`（空 storage→null），`implements Closeable`；**close 挂点已定**——context 由 catalog **单一持有**（sibling 经 `createSiblingConnector(this)` 共享同一 context），故 catalog 在 `onClose`/换连接器两处关（**连接器只借不关**）。UT 4/4 + 既有 context/catalog 24/24。**对现有行为惰性**（非 hive catalog 不调 getFileSystem→close no-op；须 HIVEFS-4 接线才活）。 -**⭐ 下一步 = HIVEFS-3（fe-core 引擎实现）**：`DefaultConnectorContext.getFileSystem(session)` 懒建 + 字段缓存 `new SpiSwitchingFileSystem(storagePropertiesSupplier.get())`，随 context 拆除 close。近零新件（已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`；范式 `HMSExternalCatalog:146`/`:348`）。之后：**HIVEFS-4 读扫描（+3+重部署 = 失败用例转绿）→ 5 ACID → 6 写路径（最险，落地前须多 agent 对抗红队）→ 7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 +**⭐ 下一步 = HIVEFS-4（连接器·读扫描列文件）**：`HiveFileListingCache`/`HiveScanPlanProvider:272` 的裸 `FileSystem.get`+`listStatus` 换 `context.getFileSystem(session).listFiles(Location.of(...))`；`FileStatus`→`FileEntry`；保留目录/`_`/`.` 过滤 + 两异常语义（systemic `DorisConnectorException` vs per-partition `HiveDirectoryListingException`）；`DirectoryLister` seam 签名 `(location, Configuration)`→`(location, FileSystem)`。**此步 + HIVEFS-3 + 重部署 = 失败用例 `test_string_dict_filter` 转绿**。之后：**5 ACID → 6 写路径（最险，落地前须多 agent 对抗红队）→ 7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 **已探明的去风险事实（勿重查）**： -- **引擎侧近零新件**——`DefaultConnectorContext` 已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`（`SpiSwitchingFileSystem` 构造入参），`HMSExternalCatalog:146`/`:348` 已这样用；`DefaultConnectorContext` 按 catalog 长生（`CatalogFactory:110`），有 `Connector.close()` 链可挂缓存 FS 的 close。 +- **引擎侧已落地（HIVEFS-3）**——`DefaultConnectorContext.getFileSystem` 用已持的 `storagePropertiesSupplier` + `SpiSwitchingFileSystem`（范式镜像 `cleanupEmptyManagedLocation:348`）；close 由 catalog 在 `onClose`/换连接器处关（context 单一持有、sibling 共享）。 - **TCCL 无需连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs **自钉**到自身插件 loader 再 `FileSystem.get`、finally 还原（注释即描述 “No FileSystem for scheme hdfs” 场景）。连接器去 jar 后任意 TCCL 调用皆安全。 - 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction:809/1336`）→ `FileSystem.forLocation`（HIVEFS-1 已加：非切换返回 this、切换返回具体 FS）就位待 HIVEFS-6 用。 - 连接器裸 Hadoop 面**很小**：`listStatus`×3 + `exists`×1 + `FileStatus` 字段（getPath/getLen/isDirectory/getModificationTime），全映射 `FileEntry`。 - fe-filesystem 全 9 插件（hdfs/s3/oss/cos/obs/azure/http/local/broker）已部署 → hive 表任意后端天然支持（免 bundle hadoop-aws/huaweicloud，架构红利）。 -**⚠ 待下个 session 核定**（task-list Open §）：`DefaultConnectorContext` close 挂点 · `HiveScanPlanProvider.buildHadoopConf`/conf 去留 · `HiveAcidUtilTest` 真-LocalFS→fake 迁移 · 写路径 commit/abort 时 FS/identity 捕获时机（begin 捕获）。 +**⚠ 待下个 session 核定**（task-list Open §）：`HiveScanPlanProvider.buildHadoopConf`/conf 去留（HIVEFS-4）· `HiveAcidUtilTest` 真-LocalFS→fake 迁移（HIVEFS-5）· 写路径 commit/abort 时 FS/identity 捕获时机（HIVEFS-6，begin 捕获）。 -**⚠ 状态**：HIVEFS-0/1/2 已入库（code 2 commit `0c4e0595f8f`/`3b4f7477d34` + 本 doc 1 commit）；设计 doc/task-list 随本轮一并 commit。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。 +**⚠ 状态**：HIVEFS-0/1/2/3 已入库（code 3 commit `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650` + doc commit）。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。 --- diff --git a/plan-doc/tasks/task-list-HIVEFS.md b/plan-doc/tasks/task-list-HIVEFS.md index deb315719608ea..d017292f343422 100644 --- a/plan-doc/tasks/task-list-HIVEFS.md +++ b/plan-doc/tasks/task-list-HIVEFS.md @@ -26,11 +26,11 @@ - javadoc 写明：**引擎所有 / 连接器借用 / 连接器不得 close**；`session` 对齐 Trino `create(session)`，identity 经 `session.getUser()` **预留 per-user**，当前 catalog 级（session 暂忽略）。默认 null（对齐 `getBackendStorageProperties()` 良性默认）。 - 校验：`mvn -o -pl :fe-connector-spi -am test-compile`。 -- [ ] **HIVEFS-3（fe-core，引擎实现）**：`DefaultConnectorContext.getFileSystem(session)` 懒建 + 字段缓存 `new SpiSwitchingFileSystem(storagePropertiesSupplier.get())`，随 context 拆除 close。 - - 复用现件（已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`；范式见 `HMSExternalCatalog:146`/`DefaultConnectorContext:348`）。 - - **待核**：`DefaultConnectorContext` 的 close/teardown 挂点（沿 `Connector.close()` 链，`ConnectorContext.java:133` 提到 caller 转发 close）——把缓存 FS 的 close 挂上。 - - UT（fe-core 有 Mockito）：`getFileSystem` 非空、二次调用返回同一实例（缓存）、close 释放。 - - 校验：`mvn -o -pl fe-core -am test-compile`。 +- [x] **HIVEFS-3（fe-core，引擎实现）** ✅ DONE `a8ed72f2650`（fe-core BUILD SUCCESS、UT 4/4 + 既有 context/catalog 24/24、0 checkstyle）：`DefaultConnectorContext.getFileSystem(session)` 懒建 + 字段缓存 `new SpiSwitchingFileSystem(storagePropertiesSupplier.get())`，空 storage→null（对齐 `getBackendStorageProperties`/`cleanupEmptyManagedLocation`）；随 context 拆除 close。 + - 复用现件（已 import `SpiSwitchingFileSystem`/`FileSystemFactory`、已持 `storagePropertiesSupplier`；范式见 `cleanupEmptyManagedLocation:348`）。 + - **close 挂点已定（原「待核」已解）**：context 由**引擎/catalog 单一持有**（sibling 经 `createSiblingConnector(this)` 共享同一 context → 每 catalog 一个缓存 FS），故由 **catalog 关**非连接器关（连接器只借）。`DefaultConnectorContext implements Closeable`（close 幂等、转发缓存 FS、置 null 使 teardown 后 `getFileSystem` 返 null）；`PluginDrivenExternalCatalog` 存 context 引用，在既有两处连接器拆除点关（`onClose` + `initLocalObjectsImpl` 换连接器），**在连接器释放借用引用之后**。非 hive 插件 catalog 从不调 `getFileSystem` → close 为 no-op（本步对现有行为**惰性**，须 HIVEFS-4 接线才活）。 + - UT（fe-core 有 Mockito；实际用 recording-fake seam `buildCatalogFileSystem`）：`getFileSystem` 懒建、二次返回同一实例（缓存）、空 storage→null、close 幂等转发缓存 FS。 + - 校验：`mvn -o -pl fe-core -am test-compile`（已过）。 - [ ] **HIVEFS-4（连接器·读扫描列文件）**： - `HiveFileListingCache`：`DirectoryLister` seam 签名 `(String location, Configuration)` → `(String location, FileSystem)`；`listFromFileSystem` 用 `fs.listFiles(Location.of(location))` 替代 `FileSystem.get`+`listStatus`；保留目录/`_`/`.` 前缀过滤 + 两异常语义（systemic `DorisConnectorException` vs per-partition `HiveDirectoryListingException`）。`FileStatus`→`FileEntry`（getPath→location、getLen→length、isDirectory→isDirectory、getModificationTime→mtime）；`HiveFileStatus` DTO 不变。 @@ -74,11 +74,11 @@ ## Open / 待下个 session 核定（勿丢） -1. `DefaultConnectorContext` 缓存 FS 的 close 挂点（Connector.close 链）。 -2. `HiveScanPlanProvider.buildHadoopConf()`/`Configuration` 在去 FS.get 后的去留（格式/split/传 BE 是否仍需）。 -3. `HiveAcidUtilTest` 从真 LocalFileSystem 迁到 fake/`fe-filesystem-local` 的注入方式。 -4. 写路径 commit/abort 时 FS/identity 的捕获时机（begin 时捕获）。 -5. `forLocation` 加到接口后与现有非切换 impl 的兼容（default `return this` 覆盖）。 +1. ~~`DefaultConnectorContext` 缓存 FS 的 close 挂点~~ ✅ **已解（HIVEFS-3）**：catalog 单一持有 + `onClose`/换连接器两处关，`DefaultConnectorContext implements Closeable`（见上 HIVEFS-3）。 +2. `HiveScanPlanProvider.buildHadoopConf()`/`Configuration` 在去 FS.get 后的去留（格式/split/传 BE 是否仍需）——HIVEFS-4 定。 +3. `HiveAcidUtilTest` 从真 LocalFileSystem 迁到 fake/`fe-filesystem-local` 的注入方式——HIVEFS-5 定。 +4. 写路径 commit/abort 时 FS/identity 的捕获时机（begin 时捕获）——HIVEFS-6 定。 +5. ~~`forLocation` 加到接口后与现有非切换 impl 的兼容~~ ✅ **已解（HIVEFS-1）**：default `return this`；`SpiSwitchingFileSystem` override 返回 per-scheme 委派；`FileSystemDefaultMethodsTest` 已断言。 ## Future（不属本次） From 7e06c2aa2a97fe193a9a4116a646f15203105696 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 15:56:22 +0800 Subject: [PATCH 243/334] [feat](catalog) hive-fs: route read-scan file listing through engine FileSystem (HIVEFS-4) Replace the hive connector's bare org.apache.hadoop.fs.FileSystem.get + listStatus in the non-ACID directory-listing path with the engine-injected Doris FileSystem (ConnectorContext.getFileSystem(session), a per-catalog SpiSwitchingFileSystem), so a hive scan no longer needs an HDFS impl bundled in the hive plugin. Aligns with Trino (connector borrows the engine filesystem; no bundled Hadoop). This + HIVEFS-3 turns the failing test_string_dict_filter ("No FileSystem for scheme hdfs") green once redeployed. Scope = non-ACID read listing only (ACID = HIVEFS-5, write = HIVEFS-6). - HiveFileListingCache: DirectoryLister/listDataFiles seam Configuration -> FileSystem; listFromFileSystem splits fs.forLocation(loc) (SYSTEMIC scheme/storage resolution, loud DorisConnectorException) from resolved.list(loc) (LOCAL per-dir failure, skippable HiveDirectoryListingException). Uses the LITERAL list() (not the glob-aware listFiles() override) for byte-parity with the old listStatus. Preserves dir + _/. filter, zero-length files, and path/len/mtime byte-parity (FileEntry -> HiveFileStatus). - Adversarial design red-team (GO_WITH_FIXES) folded in: null-FS guard -> loud; forLocation catch broadened to IOException|RuntimeException; isSystemicResolutionFailure re-raises a lazily-surfaced "No FileSystem for scheme" (UnsupportedFileSystemException) loud instead of silently skipping (the migration's own failure class). - HiveScanPlanProvider: +ConnectorContext ctor arg; non-ACID path lists via the injected FS; buildHadoopConf() kept for the dormant ACID planAcidScan (hadoop FileSystem there fully-qualified). HiveConnectorMetadata: listFileSizes/estimate list via the injected FS (estimate resolves it inside the -1-protected region); buildHadoopConf + Configuration import removed. HiveConnector wires context into the scan provider. - Tests: new shared FakeFileSystem recording double (listFiles() throws AssertionError to pin "use literal list()"); HiveFileListingCacheTest reworked (19, incl. scheme-missing -> loud, glob-char literal listing, null-FS -> loud, path/len/mtime parity); ctor/seam updates to HiveScanBatchModeTest / HiveConnectorMetadataFileListStatsTest / HiveReadTransactionTest / HiveConnectorInvalidateTest. fe-connector-hive 289/289 green, 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../doris/connector/hive/HiveConnector.java | 2 +- .../connector/hive/HiveConnectorMetadata.java | 37 +--- .../connector/hive/HiveFileListingCache.java | 143 +++++++++---- .../connector/hive/HiveScanPlanProvider.java | 33 ++- .../doris/connector/hive/FakeFileSystem.java | 155 ++++++++++++++ .../hive/HiveConnectorInvalidateTest.java | 46 ++-- ...iveConnectorMetadataFileListStatsTest.java | 4 +- .../hive/HiveFileListingCacheTest.java | 201 ++++++++++++------ .../hive/HiveReadTransactionTest.java | 4 +- .../connector/hive/HiveScanBatchModeTest.java | 6 +- 10 files changed, 447 insertions(+), 184 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 2caa039a0dbca7..02b8148a5f3908 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -170,7 +170,7 @@ private Connector resolveSiblingOwner(ConnectorTableHandle handle) { @Override public ConnectorScanPlanProvider getScanPlanProvider() { - return new HiveScanPlanProvider(getOrCreateClient(), properties, readTxnManager, fileListingCache); + return new HiveScanPlanProvider(getOrCreateClient(), properties, context, readTxnManager, fileListingCache); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 495eddfbfa0f1b..7bd7f53012777f 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -63,13 +63,13 @@ import org.apache.doris.connector.hms.HmsTableInfo; import org.apache.doris.connector.hms.HmsTypeMapping; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.FileSystem; import org.apache.doris.thrift.THiveTable; import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; import com.google.gson.Gson; import com.google.gson.JsonObject; -import org.apache.hadoop.conf.Configuration; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -806,9 +806,12 @@ public long estimateDataSizeByListingFiles(ConnectorSession session, ConnectorTa ClassLoader previous = Thread.currentThread().getContextClassLoader(); try { Thread.currentThread().setContextClassLoader(getClass().getClassLoader()); - Configuration conf = buildHadoopConf(); + // Resolve the engine FileSystem INSIDE the size lambda so it runs within estimateDataSize's + // catch(RuntimeException)->-1 region: statistics collection must degrade to -1, never fail a query, + // if getFileSystem throws. (context.getFileSystem returns the cached per-catalog FS, so per-location + // calls are cheap.) return estimateDataSize(hiveHandle, STATS_PARTITION_SAMPLE_SIZE, - location -> sumCachedFileSizes(hiveHandle, location, conf)); + location -> sumCachedFileSizes(hiveHandle, location, context.getFileSystem(session))); } finally { Thread.currentThread().setContextClassLoader(previous); } @@ -840,11 +843,11 @@ public List listFileSizes(ConnectorSession session, ConnectorTableHandle h ClassLoader previous = Thread.currentThread().getContextClassLoader(); try { Thread.currentThread().setContextClassLoader(getClass().getClassLoader()); - Configuration conf = buildHadoopConf(); + FileSystem fs = context.getFileSystem(session); List sizes = new ArrayList<>(); for (String location : resolvePartitionLocations(hiveHandle)) { for (HiveFileStatus file : fileListingCache.listDataFiles( - hiveHandle.getDbName(), hiveHandle.getTableName(), location, conf)) { + hiveHandle.getDbName(), hiveHandle.getTableName(), location, fs)) { sizes.add(file.getLength()); } } @@ -958,35 +961,15 @@ private List resolvePartitionLocations(HiveTableHandle handle) { * whole estimate to -1 (legacy's file-list estimate was all-or-nothing best-effort). Routing through the * cache keeps the periodic row-count refresh from re-listing directories a scan already cached. */ - private long sumCachedFileSizes(HiveTableHandle handle, String location, Configuration conf) { + private long sumCachedFileSizes(HiveTableHandle handle, String location, FileSystem fs) { long sum = 0; for (HiveFileStatus file : fileListingCache.listDataFiles( - handle.getDbName(), handle.getTableName(), location, conf)) { + handle.getDbName(), handle.getTableName(), location, fs)) { sum += file.getLength(); } return sum; } - /** - * Builds a Hadoop {@link Configuration} from the catalog properties, mirroring - * {@code HiveScanPlanProvider.buildHadoopConf} (the connector must supply the storage credentials for the - * FileSystem it lists). - */ - private Configuration buildHadoopConf() { - Configuration conf = new Configuration(); - for (Map.Entry entry : properties.entrySet()) { - conf.set(entry.getKey(), entry.getValue()); - } - String defaultFs = properties.get("fs.defaultFS"); - if (defaultFs == null) { - defaultFs = properties.get("hadoop.fs.defaultFS"); - } - if (defaultFs != null) { - conf.set("fs.defaultFS", defaultFs); - } - return conf; - } - // ========== ConnectorPushdownOps: Filter Pushdown ========== @Override diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java index c716edd130e2a9..9050981165a65c 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -20,11 +20,12 @@ import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.cache.MetaCacheEntry; +import org.apache.doris.filesystem.FileEntry; +import org.apache.doris.filesystem.FileIterator; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.Location; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; +import org.apache.hadoop.fs.UnsupportedFileSystemException; import java.io.IOException; import java.util.ArrayList; @@ -59,13 +60,13 @@ * *

    Failures are not cached, and are split by blast radius. The loader never caches a failed load (matching * {@code MetaCacheEntry}'s null-is-a-miss / exception-propagates contract). A SYSTEMIC filesystem-resolution failure - * ({@code FileSystem.get}: unknown scheme, bad credentials/endpoint — it fails for every partition of the table) is - * thrown as a plain {@link DorisConnectorException} and the scan path lets it propagate to fail the query loud. A - * LOCAL per-directory failure ({@code listStatus}: this partition missing/unreadable) is thrown as - * {@link HiveDirectoryListingException} and the scan path skips just that partition with a warning (the pre-cache - * tolerance). The estimate path degrades to {@code -1} on either. This keeps a broken storage config from silently - * returning an empty scan (legacy failed {@code FileSystem.get} loud) while preserving one-bad-partition - * resilience.

    + * ({@link FileSystem#forLocation} unresolvable scheme/storage, or a lazily-surfaced {@code "No FileSystem for + * scheme"} — it fails for every partition of the table) is thrown as a plain {@link DorisConnectorException} and the + * scan path lets it propagate to fail the query loud. A LOCAL per-directory failure ({@link FileSystem#list}: this + * partition missing/unreadable) is thrown as {@link HiveDirectoryListingException} and the scan path skips just that + * partition with a warning (the pre-cache tolerance). The estimate path degrades to {@code -1} on either. This keeps + * a broken storage config from silently returning an empty scan (legacy failed {@code FileSystem.get} loud) while + * preserving one-bad-partition resilience.

    * *

    Dormant. {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog builds a * {@link HiveConnector}, so this cache is never instantiated for a live catalog until the flip. Byte-neutral for @@ -85,13 +86,15 @@ public class HiveFileListingCache { static final long DEFAULT_FILE_CAPACITY = 10000L; /** - * The raw directory lister: the real Hadoop filesystem in production ({@link #listFromFileSystem}), a fake in - * unit tests. Injected so the cache's hit/miss/invalidation behaviour is testable without a live filesystem - * (mirrors {@code HiveConnectorMetadata.estimateDataSize} injecting its {@code ToLongFunction} size source). + * The raw directory lister: the engine-injected Doris {@link FileSystem} in production + * ({@link #listFromFileSystem}), a fake in unit tests. Injected so the cache's hit/miss/invalidation behaviour + * is testable without a live filesystem (mirrors {@code HiveConnectorMetadata.estimateDataSize} injecting its + * {@code ToLongFunction} size source). The {@code fs} is borrowed from {@code ConnectorContext.getFileSystem} + * (engine-owned, per-catalog, must NOT be closed by the connector). */ @FunctionalInterface interface DirectoryLister { - List list(String location, Configuration conf); + List list(String location, FileSystem fs); } private final MetaCacheEntry> cache; @@ -118,9 +121,9 @@ public HiveFileListingCache(Map properties) { * failure throws {@link DorisConnectorException} (and is NOT cached). The returned list is shared by reference * (immutable elements) — callers must treat it as read-only, the codebase-wide metadata-cache convention. */ - public List listDataFiles(String dbName, String tableName, String location, Configuration conf) { + public List listDataFiles(String dbName, String tableName, String location, FileSystem fs) { return cache.get(new FileListingKey(dbName, tableName, location), - key -> lister.list(key.location, conf)); + key -> lister.list(key.location, fs)); } /** Drops every cached listing for one table. Backs {@code REFRESH TABLE}. */ @@ -146,50 +149,98 @@ long size() { } /** - * The production {@link DirectoryLister}: a real, non-recursive {@code FileSystem.listStatus}, filtering out - * directories and {@code _}/{@code .}-prefixed hidden files (byte-parity with the pre-cache filters in - * {@code HiveScanPlanProvider.listAndSplitFiles} and {@code HiveConnectorMetadata.sumFileSizesUnder}). A zero - * -length data file is kept (the scan splitter skips it; the size estimate adds 0) so both consumers keep - * their exact prior behaviour. A {@code FileSystem.get} failure (unresolvable filesystem — unknown scheme, bad - * credentials/endpoint) is SYSTEMIC and wrapped in a plain {@link DorisConnectorException} so the scan fails - * loud; a {@code listStatus} failure (this partition directory missing/unreadable) is LOCAL and wrapped in - * {@link HiveDirectoryListingException} so the scan skips just that partition. Neither is ever cached. + * The production {@link DirectoryLister}: a non-recursive, LITERAL listing through the engine-injected Doris + * {@link FileSystem} (a per-catalog {@code SpiSwitchingFileSystem}), filtering out directories and + * {@code _}/{@code .}-prefixed hidden files (byte-parity with the pre-cache filters in + * {@code HiveScanPlanProvider.listAndSplitFiles} and {@code HiveConnectorMetadata.sumCachedFileSizes}). A zero + * -length data file is kept (the scan splitter skips it; the size estimate adds 0) so both consumers keep their + * exact prior behaviour. + * + *

    Two-boundary failure split (byte-parity with the pre-cache {@code FileSystem.get}/{@code listStatus} + * split): {@link FileSystem#forLocation} does the scheme/storage resolution + concrete-FS construction (no + * I/O) — a failure here (unresolvable scheme, no {@code StorageProperties}, factory error) is SYSTEMIC, wrapped + * loud in a plain {@link DorisConnectorException}. {@link FileSystem#list} then does the actual directory + * listing — a failure here is LOCAL (this partition missing/unreadable), wrapped in the skippable + * {@link HiveDirectoryListingException}, EXCEPT a lazily-surfaced {@code "No FileSystem for scheme"} + * ({@link UnsupportedFileSystemException}, the migration's own failure class — a missing engine-side FS impl, + * affecting every partition) which is re-classified SYSTEMIC/loud by {@link #isSystemicResolutionFailure} so a + * broken deployment fails the query instead of silently returning an empty scan. Neither is ever cached. + * + *

    Literal listing: {@code fs.list(loc)} (not {@code listFiles(loc)}) is used deliberately — the + * per-scheme filesystems ({@code DFSFileSystem}, {@code S3CompatibleFileSystem}) override {@code listFiles} with + * a glob-aware branch that would treat a location containing {@code [}/{@code *}/{@code ?} as a pattern; the old + * {@code listStatus} never glob-expanded, and a hive location can legitimately contain those characters. */ - static List listFromFileSystem(String location, Configuration conf) { - Path path = new Path(location); - FileSystem fs; + static List listFromFileSystem(String location, FileSystem fs) { + if (fs == null) { + // No engine filesystem for this catalog (empty storage): a SYSTEMIC config error affecting every + // partition. Fail loud (the scan path does NOT skip this), never a silent empty scan. + throw new DorisConnectorException("No filesystem configured for " + location); + } + Location loc = Location.of(location); + FileSystem resolved; try { - fs = FileSystem.get(path.toUri(), conf); - } catch (IOException e) { - // Filesystem resolution/initialization failed (unknown scheme, bad credentials/endpoint): a SYSTEMIC - // storage-config error affecting every partition of the table. Fail loud with a plain - // DorisConnectorException (the scan path does NOT skip this), matching the pre-cache behavior where a - // FileSystem.get failure aborted the query instead of silently returning an empty scan. + resolved = fs.forLocation(loc); + } catch (IOException | RuntimeException e) { + // Scheme/storage resolution or concrete-FS construction failed: a SYSTEMIC storage-config error + // affecting every partition of the table. (RuntimeException is also caught: the FileSystem factory may + // report a misconfiguration as an unchecked exception, which must still wrap uniformly as the loud + // systemic type rather than escape untyped.) throw new DorisConnectorException("Failed to resolve filesystem for " + location, e); } try { - FileStatus[] statuses = fs.listStatus(path); - List files = new ArrayList<>(statuses.length); - for (FileStatus status : statuses) { - if (status.isDirectory()) { - continue; + List files = new ArrayList<>(); + try (FileIterator it = resolved.list(loc)) { + while (it.hasNext()) { + FileEntry entry = it.next(); + if (entry.isDirectory()) { + continue; + } + String fileName = entry.name(); + if (fileName.startsWith("_") || fileName.startsWith(".")) { + continue; + } + files.add(new HiveFileStatus(entry.location().uri(), entry.length(), + entry.modificationTime())); } - String fileName = status.getPath().getName(); - if (fileName.startsWith("_") || fileName.startsWith(".")) { - continue; - } - files.add(new HiveFileStatus(status.getPath().toString(), status.getLen(), - status.getModificationTime())); } return files; } catch (IOException e) { + if (isSystemicResolutionFailure(e)) { + // A lazily-surfaced "No FileSystem for scheme X": the engine-side FS impl for this scheme is missing + // (broken packaging) — SYSTEMIC, affecting every partition. Fail loud, matching the pre-cache + // FileSystem.get behavior (this is the exact error class FIX-HIVEFS exists to keep loud). + throw new DorisConnectorException("Failed to resolve filesystem for " + location, e); + } // Listing THIS partition directory failed (missing / unreadable / transient): a LOCAL failure the scan // path tolerates by skipping the partition with a warning (pre-cache parity). Distinct exception type so - // only this is skipped, while the systemic FileSystem.get failure above stays loud. + // only this is skipped, while the systemic failures above stay loud. throw new HiveDirectoryListingException("Failed to list files under " + location, e); } } + /** + * Whether {@code t} (or any exception in its cause chain — robust to {@code authenticator.doAs} wrapping) + * is a scheme-not-registered failure ({@link UnsupportedFileSystemException} / {@code "No FileSystem for + * scheme"}). Such a failure is a deterministic, whole-table storage/packaging error and must stay LOUD, unlike + * a per-directory listing failure. + */ + private static boolean isSystemicResolutionFailure(Throwable t) { + for (Throwable c = t; c != null; c = c.getCause()) { + if (c instanceof UnsupportedFileSystemException) { + return true; + } + String msg = c.getMessage(); + if (msg != null && msg.contains("No FileSystem for scheme")) { + return true; + } + if (c.getCause() == c) { + break; + } + } + return false; + } + /** * Cache key: (db, table, location). db+table let {@link #invalidateTable} select one table's entries; * db alone lets {@link #invalidateDb} select one database's entries. diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index dc62a2aef1139c..448edf99696b60 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -27,10 +27,11 @@ import org.apache.doris.connector.api.scan.ConnectorScanRangeType; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.FileSystem; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -83,6 +84,10 @@ public class HiveScanPlanProvider implements ConnectorScanPlanProvider { private final HmsClient hmsClient; private final Map catalogProperties; + // Engine-owned, per-catalog filesystem accessor. The non-ACID listing path borrows the Doris FileSystem via + // context.getFileSystem(session) (never closes it — the engine owns its lifecycle) to list partition + // directories, replacing bare Hadoop FileSystem.get (FIX-HIVEFS: the hive plugin bundles no HDFS impl). + private final ConnectorContext context; private final HiveReadTransactionManager readTxnManager; // Connector-owned directory-listing cache (the SAME instance HiveConnector shares with HiveConnectorMetadata), // so a repeated scan of the same partition directory is served from the cache instead of re-listing. Only the @@ -90,9 +95,11 @@ public class HiveScanPlanProvider implements ConnectorScanPlanProvider { private final HiveFileListingCache fileListingCache; public HiveScanPlanProvider(HmsClient hmsClient, Map catalogProperties, - HiveReadTransactionManager readTxnManager, HiveFileListingCache fileListingCache) { + ConnectorContext context, HiveReadTransactionManager readTxnManager, + HiveFileListingCache fileListingCache) { this.hmsClient = hmsClient; this.catalogProperties = catalogProperties; + this.context = context; this.readTxnManager = readTxnManager; this.fileListingCache = fileListingCache; } @@ -124,19 +131,20 @@ public List planScan( boolean splittable = fileFormat.isSplittable(); List ranges = new ArrayList<>(); - Configuration hadoopConf = buildHadoopConf(); if (hiveHandle.isTransactional()) { // Transactional (ACID) table: descend into base/delta directories under the query's // write-id snapshot and emit ACID-annotated ranges. (Dormant — see planAcidScan.) - planAcidScan(session, hiveHandle, partitions, hadoopConf, fileFormat, + planAcidScan(session, hiveHandle, partitions, buildHadoopConf(), fileFormat, splittable, targetSplitSize, ranges); } else { + // Borrow the engine's per-catalog Doris FileSystem to list partition directories (see field javadoc). + FileSystem fs = context.getFileSystem(session); for (PartitionScanInfo partition : partitions) { HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; listAndSplitFiles(dbName, tableName, partition, partFormat, - splittable, targetSplitSize, hadoopConf, ranges); + splittable, targetSplitSize, fs, ranges); } } @@ -215,14 +223,16 @@ public List planScanForPartitionBatch( readHiveJsonInOneColumn(session), hiveHandle.isFirstColumnString()); long targetSplitSize = getTargetSplitSize(session); boolean splittable = fileFormat.isSplittable(); - Configuration hadoopConf = buildHadoopConf(); + // Only the non-ACID path is reachable here (supportsBatchScan excludes transactional tables), so this + // borrows the engine's per-catalog Doris FileSystem to list — no Hadoop Configuration is needed. + FileSystem fs = context.getFileSystem(session); List ranges = new ArrayList<>(); for (PartitionScanInfo partition : partitions) { HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; listAndSplitFiles(dbName, tableName, partition, partFormat, - splittable, targetSplitSize, hadoopConf, ranges); + splittable, targetSplitSize, fs, ranges); } return ranges; } @@ -269,7 +279,10 @@ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, ? partition.fileFormat : fileFormat; try { Path partPath = new Path(partition.location); - FileSystem fs = FileSystem.get(partPath.toUri(), hadoopConf); + // ACID path still uses bare Hadoop FileSystem (HIVEFS-5 will migrate HiveAcidUtil); fully-qualified + // here because the unqualified FileSystem now refers to the injected Doris filesystem. + org.apache.hadoop.fs.FileSystem fs = + org.apache.hadoop.fs.FileSystem.get(partPath.toUri(), hadoopConf); HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( fs, partition.location, txnValidIds, isFullAcid); // Only full-ACID reads are marked transactional_hive (delete deltas applied by the BE @@ -442,11 +455,11 @@ private List convertPartitions( */ private void listAndSplitFiles(String dbName, String tableName, PartitionScanInfo partition, HiveFileFormat fileFormat, - boolean splittable, long targetSplitSize, Configuration conf, + boolean splittable, long targetSplitSize, FileSystem fs, List ranges) { List files; try { - files = fileListingCache.listDataFiles(dbName, tableName, partition.location, conf); + files = fileListingCache.listDataFiles(dbName, tableName, partition.location, fs); } catch (HiveDirectoryListingException e) { LOG.warn("Cannot list files in partition: {}", partition.location, e); return; diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java new file mode 100644 index 00000000000000..d514f028e59ef2 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java @@ -0,0 +1,155 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.filesystem.DorisInputFile; +import org.apache.doris.filesystem.DorisOutputFile; +import org.apache.doris.filesystem.FileEntry; +import org.apache.doris.filesystem.FileIterator; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.Location; + +import java.io.IOException; +import java.util.Arrays; +import java.util.Collections; +import java.util.Iterator; +import java.util.List; + +/** + * A recording {@link FileSystem} test double for the hive connector's file-listing tests (the module has no + * Mockito). Configurable to return a canned {@link FileEntry} listing from {@link #list}, or to fail at the + * resolution boundary ({@link #forLocation}) or the listing boundary ({@link #list}) — exercising the two + * failure semantics {@code HiveFileListingCache.listFromFileSystem} keeps distinct. + * + *

    {@link #listFiles} deliberately throws {@link AssertionError}: the production lister MUST list via the + * literal {@link #list} (matching the old {@code FileSystem.listStatus}), never the glob-aware {@code listFiles} + * override that a real per-scheme filesystem provides — so any test that lists through this double also pins that + * contract. + */ +final class FakeFileSystem implements FileSystem { + + private List entries = Collections.emptyList(); + private IOException forLocationError; + private IOException listError; + + FakeFileSystem withEntries(FileEntry... e) { + this.entries = Arrays.asList(e); + return this; + } + + /** Makes {@link #forLocation} throw — the SYSTEMIC (scheme/storage resolution) boundary. */ + FakeFileSystem failForLocation(IOException e) { + this.forLocationError = e; + return this; + } + + /** Makes {@link #list} throw — the LOCAL per-directory boundary (or, with UnsupportedFileSystemException, + * the lazily-surfaced systemic scheme-not-registered case). */ + FakeFileSystem failList(IOException e) { + this.listError = e; + return this; + } + + static FileEntry file(String uri, long length, long modificationTime) { + return new FileEntry(Location.of(uri), length, false, modificationTime, null); + } + + static FileEntry dir(String uri) { + return new FileEntry(Location.of(uri), 0L, true, 0L, null); + } + + @Override + public FileSystem forLocation(Location location) throws IOException { + if (forLocationError != null) { + throw forLocationError; + } + return this; + } + + @Override + public FileIterator list(Location location) throws IOException { + if (listError != null) { + throw listError; + } + return new ListFileIterator(entries.iterator()); + } + + @Override + public List listFiles(Location dir) { + throw new AssertionError( + "listFromFileSystem must list via the literal list(), never the glob-aware listFiles()"); + } + + // ---- unused abstract methods (no listing test drives them) ---- + + @Override + public boolean exists(Location location) { + throw new UnsupportedOperationException(); + } + + @Override + public void mkdirs(Location location) { + throw new UnsupportedOperationException(); + } + + @Override + public void delete(Location location, boolean recursive) { + throw new UnsupportedOperationException(); + } + + @Override + public void rename(Location src, Location dst) { + throw new UnsupportedOperationException(); + } + + @Override + public DorisInputFile newInputFile(Location location) { + throw new UnsupportedOperationException(); + } + + @Override + public DorisOutputFile newOutputFile(Location location) { + throw new UnsupportedOperationException(); + } + + @Override + public void close() { + } + + private static final class ListFileIterator implements FileIterator { + private final Iterator it; + + ListFileIterator(Iterator it) { + this.it = it; + } + + @Override + public boolean hasNext() { + return it.hasNext(); + } + + @Override + public FileEntry next() { + return it.next(); + } + + @Override + public void close() { + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java index 7beaf2f56ad712..94a35c6b465df6 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorInvalidateTest.java @@ -22,13 +22,11 @@ import org.apache.doris.connector.hms.HmsDatabaseInfo; import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.filesystem.FileSystem; -import org.apache.hadoop.conf.Configuration; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; -import java.nio.file.Files; import java.util.Collections; import java.util.HashMap; import java.util.List; @@ -51,7 +49,10 @@ */ public class HiveConnectorInvalidateTest { - private static final Configuration CONF = new Configuration(); + // The file-listing cache above the FileSystem seam: the injected FileSystem lists successfully (empty is fine — + // a successful load still leaves a cache entry), so these size()-based invalidation assertions don't need real + // files. Mirrors the role the old Configuration CONF constant played. + private static final FileSystem FS = new FakeFileSystem(); private static Map props() { Map m = new HashMap<>(); @@ -60,8 +61,7 @@ private static Map props() { } @Test - public void invalidateTableFlushesBothCachesForThatTableOnly(@TempDir java.nio.file.Path dirA, - @TempDir java.nio.file.Path dirB) throws Exception { + public void invalidateTableFlushesBothCachesForThatTableOnly() { HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); RecordingHmsClient raw = new RecordingHmsClient(); CachingHmsClient cachingClient = (CachingHmsClient) connector.wrapWithCache(raw); @@ -71,12 +71,10 @@ public void invalidateTableFlushesBothCachesForThatTableOnly(@TempDir java.nio.f cachingClient.getTable("db", "t2"); Assertions.assertEquals(2, raw.getTableCalls); - // Directory-listing cache: populate t1 (dirA) and t2 (dirB) from real local dirs. + // Directory-listing cache: populate t1 and t2 (each one listing, then cached). HiveFileListingCache fileCache = connector.fileListingCacheForTest(); - Files.write(dirA.resolve("f"), new byte[10]); - Files.write(dirB.resolve("f"), new byte[10]); - fileCache.listDataFiles("db", "t1", dirA.toUri().toString(), CONF); - fileCache.listDataFiles("db", "t2", dirB.toUri().toString(), CONF); + fileCache.listDataFiles("db", "t1", "file:///wh/db/t1", FS); + fileCache.listDataFiles("db", "t2", "file:///wh/db/t2", FS); Assertions.assertEquals(2, fileCache.size()); connector.invalidateTable(cachingClient, "db", "t1"); @@ -92,8 +90,7 @@ public void invalidateTableFlushesBothCachesForThatTableOnly(@TempDir java.nio.f } @Test - public void invalidateDbFlushesBothCachesForThatDbOnly(@TempDir java.nio.file.Path dirA, - @TempDir java.nio.file.Path dirB) throws Exception { + public void invalidateDbFlushesBothCachesForThatDbOnly() { HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); RecordingHmsClient raw = new RecordingHmsClient(); CachingHmsClient cachingClient = (CachingHmsClient) connector.wrapWithCache(raw); @@ -105,12 +102,10 @@ public void invalidateDbFlushesBothCachesForThatDbOnly(@TempDir java.nio.file.Pa cachingClient.getTable("db2", "t1"); Assertions.assertEquals(3, raw.getTableCalls); - // Directory-listing cache: db1.t1 (dirA) and db2.t1 (dirB) from real local dirs. + // Directory-listing cache: db1.t1 and db2.t1 (each one listing, then cached). HiveFileListingCache fileCache = connector.fileListingCacheForTest(); - Files.write(dirA.resolve("f"), new byte[10]); - Files.write(dirB.resolve("f"), new byte[10]); - fileCache.listDataFiles("db1", "t1", dirA.toUri().toString(), CONF); - fileCache.listDataFiles("db2", "t1", dirB.toUri().toString(), CONF); + fileCache.listDataFiles("db1", "t1", "file:///wh/db1/t1", FS); + fileCache.listDataFiles("db2", "t1", "file:///wh/db2/t1", FS); Assertions.assertEquals(2, fileCache.size()); connector.invalidateDb(cachingClient, "db1"); @@ -127,7 +122,7 @@ public void invalidateDbFlushesBothCachesForThatDbOnly(@TempDir java.nio.file.Pa } @Test - public void invalidateAllFlushesBothCachesEntirely(@TempDir java.nio.file.Path dir) throws Exception { + public void invalidateAllFlushesBothCachesEntirely() { HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); RecordingHmsClient raw = new RecordingHmsClient(); CachingHmsClient cachingClient = (CachingHmsClient) connector.wrapWithCache(raw); @@ -135,8 +130,7 @@ public void invalidateAllFlushesBothCachesEntirely(@TempDir java.nio.file.Path d cachingClient.getTable("db", "t1"); Assertions.assertEquals(1, raw.getTableCalls); HiveFileListingCache fileCache = connector.fileListingCacheForTest(); - Files.write(dir.resolve("f"), new byte[10]); - fileCache.listDataFiles("db", "t1", dir.toUri().toString(), CONF); + fileCache.listDataFiles("db", "t1", "file:///wh/db/t1", FS); Assertions.assertEquals(1, fileCache.size()); connector.invalidateAll(cachingClient); @@ -148,26 +142,24 @@ public void invalidateAllFlushesBothCachesEntirely(@TempDir java.nio.file.Path d } @Test - public void publicHooksAreNoThrowAndClearFileCacheWithoutBuildingAClient(@TempDir java.nio.file.Path dir) - throws Exception { + public void publicHooksAreNoThrowAndClearFileCacheWithoutBuildingAClient() { // A fresh connector never built its metastore client (hmsClient == null). The public hooks must not // force-build one (a REFRESH on a never-scanned catalog is a cheap no-op on the metastore side), must not // throw on the null client, and must still clear the file cache. HiveConnector connector = new HiveConnector(props(), new FakeConnectorContext()); HiveFileListingCache fileCache = connector.fileListingCacheForTest(); - Files.write(dir.resolve("f"), new byte[10]); - fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); + fileCache.listDataFiles("db", "t", "file:///wh/db/t", FS); Assertions.assertEquals(1, fileCache.size()); Assertions.assertDoesNotThrow(() -> connector.invalidateTable("db", "t")); Assertions.assertEquals(0, fileCache.size(), "REFRESH TABLE clears the file cache even with no client built"); - fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); + fileCache.listDataFiles("db", "t", "file:///wh/db/t", FS); Assertions.assertEquals(1, fileCache.size()); Assertions.assertDoesNotThrow(() -> connector.invalidateDb("db")); Assertions.assertEquals(0, fileCache.size(), "REFRESH DATABASE clears the file cache with no client built"); - fileCache.listDataFiles("db", "t", dir.toUri().toString(), CONF); + fileCache.listDataFiles("db", "t", "file:///wh/db/t", FS); Assertions.assertEquals(1, fileCache.size()); Assertions.assertDoesNotThrow(() -> connector.invalidateAll()); Assertions.assertEquals(0, fileCache.size(), "REFRESH CATALOG clears the file cache even with no client built"); diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java index 3d7ea3dc8bf8c7..48eee277d1c877 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataFileListStatsTest.java @@ -21,8 +21,8 @@ import org.apache.doris.connector.hms.HmsDatabaseInfo; import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.filesystem.FileSystem; -import org.apache.hadoop.conf.Configuration; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -199,7 +199,7 @@ private static final class ThrowingFileListingCache extends HiveFileListingCache @Override public List listDataFiles(String dbName, String tableName, String location, - Configuration conf) { + FileSystem fs) { throw new RuntimeException("simulated listing failure"); } } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java index b78213fea2478c..ff16de847438f7 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java @@ -22,14 +22,14 @@ import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.filesystem.FileSystem; -import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.fs.UnsupportedFileSystemException; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; -import org.junit.jupiter.api.io.TempDir; +import java.io.FileNotFoundException; import java.io.IOException; -import java.nio.file.Files; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -48,16 +48,20 @@ * cache correct: (1) a directory listing is cached keyed by {@code (db, table, location)} so it loads once; the db * / table / location dimensions never collide; (2) {@code invalidateTable} drops exactly one table's entries and * {@code invalidateAll} clears all (arming REFRESH); (3) an I/O failure propagates and is NOT cached; (4) the - * {@code meta.cache.hive.file.*} knobs turn it off; (5) the real filesystem lister filters directories and - * {@code _}/{@code .}-hidden files. Two integration tests prove BOTH consumers — the scan provider and the - * row-count estimate — are served from the SAME cache, so a repeated scan / refresh does not re-list.

    + * {@code meta.cache.hive.file.*} knobs turn it off; (5) the production lister — now driving the engine-injected + * Doris {@link FileSystem} instead of bare Hadoop — filters directories and {@code _}/{@code .}-hidden files, lists + * literally (never glob-expands), and keeps the two failure semantics (systemic loud vs local skippable). Two + * integration tests prove BOTH consumers — the scan provider and the row-count estimate — are served from the SAME + * cache, so a repeated scan / refresh does not re-list.

    * *

    Dormant: {@code "hms"} is not in {@code SPI_READY_TYPES}, so no live catalog builds a connector; this * exercises the cache directly.

    */ public class HiveFileListingCacheTest { - private static final Configuration CONF = new Configuration(); + // A filesystem placeholder for the CountingLister-based tests: the fake lister ignores it (it lists above the + // FS seam), so any non-null FileSystem suffices — mirrors the role the old Configuration CONF constant played. + private static final FileSystem FS = new FakeFileSystem(); private static Map props(String... kv) { Map m = new HashMap<>(); @@ -74,14 +78,14 @@ public void listingIsCachedPerLocation() { CountingLister lister = new CountingLister(); HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); - List a = cache.listDataFiles("db", "t", "loc1", CONF); - List b = cache.listDataFiles("db", "t", "loc1", CONF); + List a = cache.listDataFiles("db", "t", "loc1", FS); + List b = cache.listDataFiles("db", "t", "loc1", FS); // WHY: a hit must serve the cached listing without re-listing the filesystem. Assertions.assertSame(a, b); Assertions.assertEquals(1, lister.totalCalls); // WHY: a different directory is a different key — must re-list. - cache.listDataFiles("db", "t", "loc2", CONF); + cache.listDataFiles("db", "t", "loc2", FS); Assertions.assertEquals(2, lister.totalCalls); } @@ -92,9 +96,9 @@ public void keyIsScopedByDbTableAndLocation() { // Same location string, different db / table. WHY: (db, table) must be part of the key so invalidateTable // can scope one table — and so two tables that happen to share a path never serve each other's listing. - cache.listDataFiles("db1", "t", "loc", CONF); - cache.listDataFiles("db2", "t", "loc", CONF); - cache.listDataFiles("db1", "t2", "loc", CONF); + cache.listDataFiles("db1", "t", "loc", FS); + cache.listDataFiles("db2", "t", "loc", FS); + cache.listDataFiles("db1", "t2", "loc", FS); Assertions.assertEquals(3, lister.totalCalls); } @@ -105,17 +109,17 @@ public void invalidateTableDropsOnlyThatTablesEntries() { CountingLister lister = new CountingLister(); HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); - cache.listDataFiles("db", "t1", "locA", CONF); - cache.listDataFiles("db", "t2", "locB", CONF); + cache.listDataFiles("db", "t1", "locA", FS); + cache.listDataFiles("db", "t2", "locB", FS); Assertions.assertEquals(2, lister.totalCalls); cache.invalidateTable("db", "t1"); // WHY: t1 must re-list after its invalidation... - cache.listDataFiles("db", "t1", "locA", CONF); + cache.listDataFiles("db", "t1", "locA", FS); Assertions.assertEquals(2, (int) lister.callsPerLocation.get("locA")); // ...while t2's entry (a different table) must survive — invalidateTable is scoped by (db, table). - cache.listDataFiles("db", "t2", "locB", CONF); + cache.listDataFiles("db", "t2", "locB", FS); Assertions.assertEquals(1, (int) lister.callsPerLocation.get("locB")); } @@ -124,15 +128,15 @@ public void invalidateAllDropsEverything() { CountingLister lister = new CountingLister(); HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); - cache.listDataFiles("db", "t1", "locA", CONF); - cache.listDataFiles("db", "t2", "locB", CONF); + cache.listDataFiles("db", "t1", "locA", FS); + cache.listDataFiles("db", "t2", "locB", FS); Assertions.assertEquals(2, lister.totalCalls); cache.invalidateAll(); // WHY: every entry must reload after invalidateAll (REFRESH CATALOG). - cache.listDataFiles("db", "t1", "locA", CONF); - cache.listDataFiles("db", "t2", "locB", CONF); + cache.listDataFiles("db", "t1", "locA", FS); + cache.listDataFiles("db", "t2", "locB", FS); Assertions.assertEquals(4, lister.totalCalls); } @@ -145,14 +149,14 @@ public void listingFailureIsPropagatedAndNotCached() { HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, - () -> cache.listDataFiles("db", "t", "loc", CONF)); + () -> cache.listDataFiles("db", "t", "loc", FS)); Assertions.assertEquals("boom", e.getMessage()); Assertions.assertEquals(1, lister.totalCalls); // WHY: a transient listing failure must NOT be cached — after recovery the next call re-lists and succeeds // (otherwise a momentary storage blip would poison the listing for the whole TTL). lister.error = null; - List ok = cache.listDataFiles("db", "t", "loc", CONF); + List ok = cache.listDataFiles("db", "t", "loc", FS); Assertions.assertEquals(1, ok.size()); Assertions.assertEquals(2, lister.totalCalls); } @@ -165,23 +169,25 @@ public void disablingViaPropsBypassesTheCache() { HiveFileListingCache cache = new HiveFileListingCache( props("meta.cache.hive.file.enable", "false"), lister); - cache.listDataFiles("db", "t", "loc", CONF); - cache.listDataFiles("db", "t", "loc", CONF); + cache.listDataFiles("db", "t", "loc", FS); + cache.listDataFiles("db", "t", "loc", FS); // WHY: meta.cache.hive.file.enable=false must bypass caching entirely — every call re-lists. Assertions.assertEquals(2, lister.totalCalls); } - // ==================== the real filesystem lister: filters dirs + hidden files ==================== + // ==================== the production lister: filters dirs + hidden files, lists literally ==================== @Test - public void listFromFileSystemFiltersDirectoriesAndHiddenFiles(@TempDir java.nio.file.Path dir) throws Exception { - Files.write(dir.resolve("data1"), new byte[100]); - Files.write(dir.resolve("data2"), new byte[200]); - Files.write(dir.resolve("_SUCCESS"), new byte[3]); // hive success marker — must be skipped - Files.write(dir.resolve(".hidden"), new byte[3]); // dot-file — must be skipped - Files.createDirectory(dir.resolve("subdir")); // directory — must be skipped + public void listFromFileSystemFiltersDirectoriesAndHiddenFiles() { + String dir = "file:///wh/db/t/dt=1"; + FakeFileSystem fs = new FakeFileSystem().withEntries( + FakeFileSystem.file(dir + "/data1", 100L, 1L), + FakeFileSystem.file(dir + "/data2", 200L, 1L), + FakeFileSystem.file(dir + "/_SUCCESS", 3L, 1L), // hive success marker — must be skipped + FakeFileSystem.file(dir + "/.hidden", 3L, 1L), // dot-file — must be skipped + FakeFileSystem.dir(dir + "/subdir")); // directory — must be skipped - List files = HiveFileListingCache.listFromFileSystem(dir.toUri().toString(), CONF); + List files = HiveFileListingCache.listFromFileSystem(dir, fs); // WHY: only the two real data files survive; the total size is exactly their sum. This pins the same // dir/_-/.-filter the pre-cache scan and estimate paths applied inline. @@ -194,59 +200,119 @@ public void listFromFileSystemFiltersDirectoriesAndHiddenFiles(@TempDir java.nio } } - // ==================== failure split: systemic (FileSystem.get) is loud, local (listStatus) is skippable ====== + @Test + public void listFromFileSystemPreservesPathLengthAndModTime() { + // WHY (Rule 9): the FileEntry -> HiveFileStatus mapping must be byte-parity — the path string verbatim + // (Location.uri(), no re-encoding of hive's %3A timestamp partitions), the byte length and the mtime the + // BE reads to detect a changed split. A field transposition would flip these. + String dir = "file:///wh/db/t/pt=2024-04-09 12%3A34%3A56"; + FakeFileSystem fs = new FakeFileSystem().withEntries( + FakeFileSystem.file(dir + "/000000_0", 4096L, 1712660096000L)); + + List files = HiveFileListingCache.listFromFileSystem(dir, fs); + + Assertions.assertEquals(1, files.size()); + Assertions.assertEquals(dir + "/000000_0", files.get(0).getPath()); + Assertions.assertEquals(4096L, files.get(0).getLength()); + Assertions.assertEquals(1712660096000L, files.get(0).getModificationTime()); + } + + @Test + public void listFromFileSystemListsLiterallyForGlobCharLocation() { + // WHY (Rule 9): a location containing a glob metachar ('[' '*' '?') — e.g. a custom external SET LOCATION — + // must be listed LITERALLY (old fs.listStatus never glob-expanded). The production lister must use the + // literal list(), not the per-scheme glob-aware listFiles() override. FakeFileSystem.listFiles() throws + // AssertionError, so if the lister ever regressed to listFiles() this test would fail loudly. + String dir = "file:///wh/db/my[table]/dt=1"; + FakeFileSystem fs = new FakeFileSystem().withEntries( + FakeFileSystem.file(dir + "/000000_0", 10L, 1L)); + + List files = HiveFileListingCache.listFromFileSystem(dir, fs); + + Assertions.assertEquals(1, files.size(), "the literal directory must be listed, not glob-matched"); + Assertions.assertEquals(dir + "/000000_0", files.get(0).getPath()); + } + + // ==================== failure split: systemic is loud, local is skippable ==================== + + @Test + public void listFromFileSystemFailsLoudWhenFsIsNull() { + // A null engine filesystem (catalog with no storage) is a SYSTEMIC config error affecting every partition. + // WHY (Rule 9): it must fail the query loud (plain DorisConnectorException), not skip / return empty. + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> HiveFileListingCache.listFromFileSystem("hdfs://host/path", null)); + Assertions.assertFalse(e instanceof HiveDirectoryListingException, + "a missing filesystem must be loud, not skippable"); + } @Test public void listFromFileSystemFailsLoudWhenFilesystemCannotBeResolved() { - // An unknown scheme makes FileSystem.get fail (no FileSystem for scheme) — a SYSTEMIC storage-config error - // that affects every partition of the table. It must throw a plain DorisConnectorException (which the scan - // path does NOT skip), and NOT the skippable HiveDirectoryListingException. + // forLocation failing (unresolvable scheme / no StorageProperties / factory error) is a SYSTEMIC + // storage-config error affecting every partition. It must throw a plain DorisConnectorException (which the + // scan path does NOT skip), and NOT the skippable HiveDirectoryListingException. // WHY (Rule 9): if this were the skippable subtype, a misconfigured storage would silently return an empty // scan instead of failing the query loud — the exact regression this fix prevents. + FakeFileSystem fs = new FakeFileSystem().failForLocation(new IOException("no StorageProperties for scheme")); DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, - () -> HiveFileListingCache.listFromFileSystem("nosuchscheme://host/path", CONF)); + () -> HiveFileListingCache.listFromFileSystem("nosuchscheme://host/path", fs)); Assertions.assertFalse(e instanceof HiveDirectoryListingException, "a filesystem-resolution failure must be loud (plain DorisConnectorException), not skippable"); } @Test - public void listFromFileSystemIsSkippableWhenDirectoryMissing(@TempDir java.nio.file.Path dir) { - // A resolvable filesystem (file://) but a non-existent directory makes listStatus fail — a LOCAL failure of - // one partition directory. It must throw the skippable HiveDirectoryListingException so the scan skips just - // that partition (pre-cache tolerance of a missing/unreadable partition dir). - String missing = dir.resolve("does-not-exist").toUri().toString(); + public void listFromFileSystemFailsLoudWhenSchemeImplMissing() { + // A lazily-surfaced "No FileSystem for scheme X" (the engine-side FS impl for the scheme is absent — broken + // packaging) is thrown from list(), not forLocation(). It is nonetheless SYSTEMIC (affects every partition). + // WHY (Rule 9): it must be re-classified LOUD (plain DorisConnectorException), matching the pre-migration + // FileSystem.get behaviour — this is the exact "No FileSystem for scheme hdfs" error class FIX-HIVEFS keeps + // loud. If it degraded to the skippable subtype, a broken deployment would scan EMPTY silently. + FakeFileSystem fs = new FakeFileSystem() + .failList(new UnsupportedFileSystemException("No FileSystem for scheme \"hdfs\"")); + DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, + () -> HiveFileListingCache.listFromFileSystem("hdfs://host/path", fs)); + Assertions.assertFalse(e instanceof HiveDirectoryListingException, + "a scheme-not-registered failure must stay loud, not skippable"); + } + + @Test + public void listFromFileSystemIsSkippableWhenDirectoryMissing() { + // A resolvable filesystem but a non-existent directory makes list() fail — a LOCAL failure of one partition + // directory. It must throw the skippable HiveDirectoryListingException so the scan skips just that partition + // (pre-cache tolerance of a missing/unreadable partition dir). + FakeFileSystem fs = new FakeFileSystem().failList(new FileNotFoundException("Path does not exist")); Assertions.assertThrows(HiveDirectoryListingException.class, - () -> HiveFileListingCache.listFromFileSystem(missing, CONF)); + () -> HiveFileListingCache.listFromFileSystem("file:///wh/db/t/does-not-exist", fs)); } @Test - public void getFailureThroughCacheFailsLoudAndIsNotCached() { - // Drives the REAL production lister THROUGH the cache lookup with an unresolvable scheme, so - // FileSystem.get genuinely fails. WHY (Rule 9): pins that (1) a systemic storage-config failure propagates - // from listDataFiles as the loud plain DorisConnectorException — MetaCacheEntry's manual-miss load rethrows - // RuntimeException unwrapped, so the type survives the cache boundary — and NOT the skippable subtype; and - // (2) the failure leaves NO cache entry. (2) kills the mutation "catch -> return emptyList" in the loader, - // which would cache a poisoned empty listing and silently turn every later scan into 0 rows for the TTL. + public void resolutionFailureThroughCacheFailsLoudAndIsNotCached() { + // Drives the REAL production lister THROUGH the cache lookup with a forLocation failure. WHY (Rule 9): pins + // that (1) a systemic storage-config failure propagates from listDataFiles as the loud plain + // DorisConnectorException — MetaCacheEntry's manual-miss load rethrows RuntimeException unwrapped, so the + // type survives the cache boundary — and NOT the skippable subtype; and (2) the failure leaves NO cache + // entry. (2) kills the mutation "catch -> return emptyList" in the loader, which would cache a poisoned + // empty listing and silently turn every later scan into 0 rows for the TTL. HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap()); + FakeFileSystem fs = new FakeFileSystem().failForLocation(new IOException("no StorageProperties for scheme")); DorisConnectorException e = Assertions.assertThrows(DorisConnectorException.class, - () -> cache.listDataFiles("db", "t", "nosuchfs://host/path", CONF)); + () -> cache.listDataFiles("db", "t", "nosuchfs://host/path", fs)); Assertions.assertFalse(e instanceof HiveDirectoryListingException, "a filesystem-resolution failure through the cache must be the loud type, not the skippable one"); Assertions.assertEquals(0L, cache.size(), "a failed load must never leave a cache entry"); } @Test - public void listStatusFailureThroughCacheIsSkippableAndNotCached(@TempDir java.nio.file.Path dir) { - // Drives the REAL production lister THROUGH the cache with a resolvable filesystem (file://) but a missing - // directory, so listStatus genuinely fails. WHY (Rule 9): pins that the LOCAL per-directory failure keeps - // its distinct skippable type (HiveDirectoryListingException, exactly what the scan's per-partition skip - // catches) across the cache boundary, and leaves no cache entry either. + public void listFailureThroughCacheIsSkippableAndNotCached() { + // Drives the REAL production lister THROUGH the cache with a resolvable filesystem but a list() failure, so + // the listing genuinely fails. WHY (Rule 9): pins that the LOCAL per-directory failure keeps its distinct + // skippable type (HiveDirectoryListingException, exactly what the scan's per-partition skip catches) across + // the cache boundary, and leaves no cache entry either. HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap()); - String missing = dir.resolve("does-not-exist").toUri().toString(); + FakeFileSystem fs = new FakeFileSystem().failList(new FileNotFoundException("Path does not exist")); Assertions.assertThrows(HiveDirectoryListingException.class, - () -> cache.listDataFiles("db", "t", missing, CONF)); + () -> cache.listDataFiles("db", "t", "file:///wh/db/t/does-not-exist", fs)); Assertions.assertEquals(0L, cache.size(), "a failed load must never leave a cache entry"); } @@ -260,7 +326,8 @@ public void scanSkipsOnlyTheFailedPartitionAndPlansTheRest() { lister.error = new HiveDirectoryListingException("dir gone", new IOException("boom")); lister.errorLocation = "loc/dt=1"; HiveScanPlanProvider provider = new HiveScanPlanProvider(null, Collections.emptyMap(), - new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); + new FakeConnectorContext(), new HiveReadTransactionManager(), + new HiveFileListingCache(Collections.emptyMap(), lister)); List ranges = provider.planScan( new FakeSession(), twoPartitionHandle(), Collections.emptyList(), @@ -282,7 +349,8 @@ public void scanFailsLoudOnSystemicFilesystemFailure() { CountingLister lister = new CountingLister(); lister.error = new DorisConnectorException("bad storage config"); HiveScanPlanProvider provider = new HiveScanPlanProvider(null, Collections.emptyMap(), - new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); + new FakeConnectorContext(), new HiveReadTransactionManager(), + new HiveFileListingCache(Collections.emptyMap(), lister)); Assertions.assertThrows(DorisConnectorException.class, () -> provider.planScan( new FakeSession(), singlePartitionHandle(), Collections.emptyList(), @@ -318,7 +386,7 @@ public void scanProviderServesRepeatedScansFromTheCache() { HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap(), lister); // hmsClient is null: with pruned partitions on the handle the scan never calls the metastore. HiveScanPlanProvider provider = new HiveScanPlanProvider( - null, Collections.emptyMap(), new HiveReadTransactionManager(), cache); + null, Collections.emptyMap(), new FakeConnectorContext(), new HiveReadTransactionManager(), cache); HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) .inputFormat("org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat") @@ -359,8 +427,8 @@ public void estimateDataSizeIsServedFromTheCache() { .location("file:///wh/t") .build(); - long first = md.estimateDataSizeByListingFiles(null, handle); - long second = md.estimateDataSizeByListingFiles(null, handle); + long first = md.estimateDataSizeByListingFiles(new FakeSession(), handle); + long second = md.estimateDataSizeByListingFiles(new FakeSession(), handle); // WHY: the estimate returns the one 10-byte file's size, and the second (periodic ExternalRowCountCache // refresh) call is served from the SAME cache — one listing, not two. This proves the row-count refresh @@ -373,7 +441,8 @@ public void estimateDataSizeIsServedFromTheCache() { /** * A {@link HiveFileListingCache.DirectoryLister} double: counts calls (total + per location) and returns a * fresh single-file listing per call, so reference identity distinguishes a cache hit from a reload. Throws - * {@link #error} when set, to exercise the failure-not-cached path. + * {@link #error} when set, to exercise the failure-not-cached path. Lists above the FileSystem seam, so it + * ignores the injected {@link FileSystem}. */ private static final class CountingLister implements HiveFileListingCache.DirectoryLister { final Map callsPerLocation = new HashMap<>(); @@ -383,7 +452,7 @@ private static final class CountingLister implements HiveFileListingCache.Direct String errorLocation; @Override - public List list(String location, Configuration conf) { + public List list(String location, FileSystem fs) { totalCalls++; callsPerLocation.merge(location, 1, Integer::sum); if (error != null && (errorLocation == null || errorLocation.equals(location))) { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java index 1650c282eb1d23..b7298e4b0a66db 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveReadTransactionTest.java @@ -106,8 +106,8 @@ public void testScanProviderReleaseReadTransactionCommitsViaSharedManager() { // register and the provider's release MUST share the same per-connector manager (HiveConnector injects // one instance into every provider), so the release finds and commits the txn register opened. HiveReadTransactionManager mgr = new HiveReadTransactionManager(); - HiveScanPlanProvider provider = new HiveScanPlanProvider(client, new HashMap<>(), mgr, - new HiveFileListingCache(new HashMap<>())); + HiveScanPlanProvider provider = new HiveScanPlanProvider(client, new HashMap<>(), + new FakeConnectorContext(), mgr, new HiveFileListingCache(new HashMap<>())); // A transactional-hive scan opened a read txn (as planAcidScan does via mgr.register), taking the shared // read lock. diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java index b973a2c26be7d8..754a42c7588877 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java @@ -24,8 +24,8 @@ import org.apache.doris.connector.hms.HmsDatabaseInfo; import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; +import org.apache.doris.filesystem.FileSystem; -import org.apache.hadoop.conf.Configuration; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -143,7 +143,7 @@ public void planScanForPartitionBatchResolvesOnlyTheBatch() { // ===== helpers ===== private static HiveScanPlanProvider provider(HmsClient hmsClient, CountingLister lister) { - return new HiveScanPlanProvider(hmsClient, Collections.emptyMap(), + return new HiveScanPlanProvider(hmsClient, Collections.emptyMap(), new FakeConnectorContext(), new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); } @@ -160,7 +160,7 @@ private static final class CountingLister implements HiveFileListingCache.Direct int totalCalls; @Override - public List list(String location, Configuration conf) { + public List list(String location, FileSystem fs) { totalCalls++; callsPerLocation.merge(location, 1, Integer::sum); return new ArrayList<>(Collections.singletonList(new HiveFileStatus(location + "/000000_0", 10L, 1L))); From f617eb1aac7f7eb92167af2ce72e3c907b122751 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 15:57:31 +0800 Subject: [PATCH 244/334] [doc](catalog) hive-fs: land HIVEFS-4 design (v2, red-team folded), mark done, repoint to ACID HIVEFS-5 - FIX-HIVEFS-4-design.md: the read-scan file-listing substep design, with the adversarial design red-team's GO_WITH_FIXES edits folded in (literal list() vs glob listFiles(), lazy "No FileSystem for scheme" re-raised loud, null-FS guard, forLocation catch broadened, estimate-path FS resolution inside the -1-protected region, corrected TCCL rationale, full 6-file test blast radius). - task-list-HIVEFS.md: HIVEFS-4 checked done with the as-built summary. - HANDOFF.md: next step repointed to HIVEFS-5 (ACID); HIVEFS-4 recorded as landed (code 7e06c2aa2a9, 289/289, 0 checkstyle). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 8 +- plan-doc/tasks/designs/FIX-HIVEFS-4-design.md | 134 ++++++++++++++++++ plan-doc/tasks/task-list-HIVEFS.md | 15 +- 3 files changed, 148 insertions(+), 9 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-HIVEFS-4-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index faed21adff22e4..2c984cbd8c4191 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,7 @@ --- -# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 读扫描 HIVEFS-4） +# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = ACID HIVEFS-5） > 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。** @@ -19,7 +19,9 @@ - **HIVEFS-2** `3b4f7477d34`：`ConnectorContext` 加 `default FileSystem getFileSystem(ConnectorSession){return null}`+javadoc（**引擎所有/连接器借用/连接器不得 close**；session 对齐 Trino、identity 经 `getUser()` 预留 per-user；默认 null 对齐 `getBackendStorageProperties`）。 - **HIVEFS-3** `a8ed72f2650`：`DefaultConnectorContext.getFileSystem` 懒建+字段缓存一个 per-catalog `SpiSwitchingFileSystem`（空 storage→null），`implements Closeable`；**close 挂点已定**——context 由 catalog **单一持有**（sibling 经 `createSiblingConnector(this)` 共享同一 context），故 catalog 在 `onClose`/换连接器两处关（**连接器只借不关**）。UT 4/4 + 既有 context/catalog 24/24。**对现有行为惰性**（非 hive catalog 不调 getFileSystem→close no-op；须 HIVEFS-4 接线才活）。 -**⭐ 下一步 = HIVEFS-4（连接器·读扫描列文件）**：`HiveFileListingCache`/`HiveScanPlanProvider:272` 的裸 `FileSystem.get`+`listStatus` 换 `context.getFileSystem(session).listFiles(Location.of(...))`；`FileStatus`→`FileEntry`；保留目录/`_`/`.` 过滤 + 两异常语义（systemic `DorisConnectorException` vs per-partition `HiveDirectoryListingException`）；`DirectoryLister` seam 签名 `(location, Configuration)`→`(location, FileSystem)`。**此步 + HIVEFS-3 + 重部署 = 失败用例 `test_string_dict_filter` 转绿**。之后：**5 ACID → 6 写路径（最险，落地前须多 agent 对抗红队）→ 7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 +**✅ HIVEFS-4 DONE（code `7e06c2aa2a9`）= 连接器·读扫描列文件已转引擎 FileSystem**：`HiveFileListingCache` seam `Configuration`→`org.apache.doris.filesystem.FileSystem`；`listFromFileSystem` 用 `fs.forLocation(loc)`(SYSTEMIC 边界·loud) + `resolved.list(loc)`(LOCAL 边界·skippable，**字面量 `list()` 非 glob 的 `listFiles()`**) 替代裸 `FileSystem.get`+`listStatus`；3 调用点（scan `listAndSplitFiles`、metadata `listFileSizes`/`estimate`）经 `context.getFileSystem(session)` 下发；`HiveScanPlanProvider` +ctx ctor 参；metadata 删 `buildHadoopConf`+`Configuration` import。设计红队 `wf_f25cc498-2de`(GO_WITH_FIXES 7 条全折入：null-FS→loud、`forLocation` catch 拓宽、`isSystemicResolutionFailure` 让惰性"No FileSystem for scheme"仍 loud、literal `list()`、estimate 保护区内下发、4 连带测试文件、去 orphan import)。**fe-connector-hive 289/289、0 checkstyle**。设计 `designs/FIX-HIVEFS-4-design.md`(v2)。**此步 + HIVEFS-3 + 重部署 = `test_string_dict_filter` 应转绿（e2e 待用户自跑）。ACID `planAcidScan:272` 的裸 Hadoop 未动（HIVEFS-5）。** + +**⭐ 下一步 = HIVEFS-5（连接器·ACID）**：`HiveAcidUtil:170/255/298` 的 `fs.exists`/`fs.listStatus` + `HiveScanPlanProvider.planAcidScan:272`（现全限定的 `org.apache.hadoop.fs.FileSystem.get`，喂 `HiveAcidUtil.getAcidState`）换注入 `FileSystem`。**⚠ 测试基建待定**：`HiveAcidUtilTest` 现对真 Hadoop LocalFileSystem 跑，转 Doris `FileSystem` 后须换 fake（可复用本轮新增的 `FakeFileSystem`）或 `fe-filesystem-local`——下个 session 定夺注入方式。之后：**6 写路径（最险，落地前须多 agent 对抗红队）→ 7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 **已探明的去风险事实（勿重查）**： - **引擎侧已落地（HIVEFS-3）**——`DefaultConnectorContext.getFileSystem` 用已持的 `storagePropertiesSupplier` + `SpiSwitchingFileSystem`（范式镜像 `cleanupEmptyManagedLocation:348`）；close 由 catalog 在 `onClose`/换连接器处关（context 单一持有、sibling 共享）。 @@ -30,7 +32,7 @@ **⚠ 待下个 session 核定**（task-list Open §）：`HiveScanPlanProvider.buildHadoopConf`/conf 去留（HIVEFS-4）· `HiveAcidUtilTest` 真-LocalFS→fake 迁移（HIVEFS-5）· 写路径 commit/abort 时 FS/identity 捕获时机（HIVEFS-6，begin 捕获）。 -**⚠ 状态**：HIVEFS-0/1/2/3 已入库（code 3 commit `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650` + doc commit）。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。 +**⚠ 状态**：HIVEFS-0/1/2/3/4 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9` + doc commit）。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。HIVEFS-4 已按此走完一轮（设计→红队→折入→实现→289 UT→commit）。 --- diff --git a/plan-doc/tasks/designs/FIX-HIVEFS-4-design.md b/plan-doc/tasks/designs/FIX-HIVEFS-4-design.md new file mode 100644 index 00000000000000..7e303f95b1b71c --- /dev/null +++ b/plan-doc/tasks/designs/FIX-HIVEFS-4-design.md @@ -0,0 +1,134 @@ +# FIX-HIVEFS-4 — connector read-scan file listing: bare Hadoop `FileSystem` → engine-injected `org.apache.doris.filesystem.FileSystem` + +> Substep of FIX-HIVEFS (parent design `FIX-HIVEFS-design.md`, task list `task-list-HIVEFS.md`). Scope = the **non-ACID read file-listing path only** (`HiveFileListingCache` + its 3 callers). ACID (`planAcidScan:272` / `HiveAcidUtil`) is HIVEFS-5; write path is HIVEFS-6. +> Line numbers verified against HEAD (`14169366b76`). + +## Goal / success criteria + +- `HiveFileListingCache.listFromFileSystem` no longer calls bare `org.apache.hadoop.fs.FileSystem.get` / `listStatus`; it lists through the engine-injected `org.apache.doris.filesystem.FileSystem` (a per-catalog `SpiSwitchingFileSystem`). +- Byte-parity of the produced `HiveFileStatus` list (path string, length, mtime; dir + `_`/`.` filter; zero-length kept) with the old lister. +- The two failure semantics preserved: SYSTEMIC (unresolvable filesystem/scheme) → loud `DorisConnectorException`; LOCAL (this partition dir missing/unreadable) → skippable `HiveDirectoryListingException`. +- `HIVEFS-4 + HIVEFS-3 + redeploy` turns the failing regression `test_string_dict_filter` green. +- Build + 0 checkstyle + targeted UT (adapted `HiveFileListingCacheTest`) all green, all RED-able. + +## Surface (verified against HEAD) + +`fileListingCache.listDataFiles(db, table, location, Configuration)` has exactly 3 callers, all with `ConnectorSession` in scope: +1. `HiveScanPlanProvider.listAndSplitFiles:449` (non-ACID scan hot path). Provider does NOT hold `ConnectorContext`. +2. `HiveConnectorMetadata.listFileSizes:846` (`ANALYZE ... WITH SAMPLE`; loud on error). Holds `context`. +3. `HiveConnectorMetadata.sumCachedFileSizes:963` ← `estimateDataSizeByListingFiles:798` (row-count estimate; best-effort `-1`). Holds `context`. + +`ConnectorContext.getFileSystem(ConnectorSession)` (HIVEFS-2) returns the per-catalog cached `SpiSwitchingFileSystem`, or `null` when the catalog has no storage (HIVEFS-3). + +## Design decisions + +### D1 — Seam signature: `Configuration` → `org.apache.doris.filesystem.FileSystem` +- `DirectoryLister.list(String location, Configuration)` → `list(String location, FileSystem fs)`. +- `listDataFiles(db, table, location, Configuration)` → `listDataFiles(db, table, location, FileSystem fs)`. +- The `FileSystem` is the per-catalog switching FS; passed by each caller from `context.getFileSystem(session)`. It is NOT part of the cache key (key stays `(db, table, location)`) — there is exactly one FS per catalog, constant across calls, so it never varies for a given key. + +### D2 — `listFromFileSystem` new body (v2, red-team folded in): literal `list()` + split resolution/listing + lazy-systemic re-raise +```java +static List listFromFileSystem(String location, FileSystem fs) { + if (fs == null) { // D4 null-FS guard (systemic, loud) + throw new DorisConnectorException("No filesystem configured for " + location); + } + Location loc = Location.of(location); + FileSystem resolved; + try { + resolved = fs.forLocation(loc); // SYSTEMIC boundary: scheme/storage resolution + FS construction (no I/O) + } catch (IOException | RuntimeException e) { // [Minor-1] broadened: FactoryFactory may throw RuntimeException + throw new DorisConnectorException("Failed to resolve filesystem for " + location, e); + } + try { + List files = new ArrayList<>(); + // resolved.list(loc) — the LITERAL, non-glob listing (== old fs.listStatus). NOT listFiles(loc): + // DFSFileSystem/S3CompatibleFileSystem override listFiles with a glob-aware branch that would treat a + // location containing '[','*','?' as a PATTERN. [Major-1] Old listStatus never glob-expanded. + try (FileIterator it = resolved.list(loc)) { + while (it.hasNext()) { + FileEntry e = it.next(); + if (e.isDirectory()) { // dir filter (== old !status.isDirectory()) + continue; + } + String name = e.name(); + if (name.startsWith("_") || name.startsWith(".")) { // hive hidden/marker filter + continue; + } + files.add(new HiveFileStatus(e.location().uri(), e.length(), e.modificationTime())); + } + } + return files; + } catch (IOException e) { + // [Major-2] A lazily-surfaced "No FileSystem for scheme X" (UnsupportedFileSystemException) is a SYSTEMIC + // storage/packaging error affecting every partition — the migration's OWN failure class — and must stay + // LOUD (old FileSystem.get threw it loud). Everything else (dir missing/unreadable/perm) is LOCAL/skippable. + if (isSystemicResolutionFailure(e)) { + throw new DorisConnectorException("Failed to resolve filesystem for " + location, e); + } + throw new HiveDirectoryListingException("Failed to list files under " + location, e); + } +} + +// Walks the cause chain (robust to authenticator.doAs wrapping) for the scheme-not-registered systemic class. +private static boolean isSystemicResolutionFailure(Throwable t) { + for (Throwable c = t; c != null; c = c.getCause()) { + if (c instanceof UnsupportedFileSystemException) { + return true; + } + String msg = c.getMessage(); + if (msg != null && msg.contains("No FileSystem for scheme")) { + return true; + } + if (c.getCause() == c) { // guard self-referential cause + break; + } + } + return false; +} +``` +**Why `forLocation` then `list` (not a single `listFiles`)**: on `SpiSwitchingFileSystem`, `listFiles(dir)` = `forLocation(dir).listFiles(dir)` — one call, so it cannot distinguish a resolution failure from a listing failure. Splitting maps `forLocation` (= `forPath` → `LocationPath.of` + `FileSystemFactory.getFileSystem`, i.e. resolution + FS *construction*, **no I/O**) to the old `FileSystem.get` SYSTEMIC boundary, and `resolved.list` (= `DFSFileSystem.list` → `getHadoopFs` → the actual Hadoop `FileSystem.get` + `listStatusIterator`) to the old `listStatus` LOCAL boundary. **Residual (accepted, documented)**: bad-namenode-host / connect failures surface at `list` = LOCAL/skippable. Old code's classification of these was Hadoop-version-dependent (connect is lazy at first RPC), so this is not a clear regression; only the *deterministic scheme-not-registered* class is force-classified systemic (via `isSystemicResolutionFailure`). + +### D3 — filter + field parity +- `resolved.listFiles(loc)` = non-recursive, directories excluded (contract; default impl iterates `list()` filtering `isDirectory`) — matches old `listStatus` + `!isDirectory` filter. +- `_`/`.` filter applied on `FileEntry.name()` (last path segment) — matches old `status.getPath().getName()` prefix check. +- Zero-length files kept (splitter skips size==0; estimate adds 0) — `listFiles` returns them. +- Path string: `FileEntry.location().uri()` == `HdfsFileIterator`'s `Location.of(status.getPath().toString())` == old `status.getPath().toString()`. **`Path.toString()`, not `toUri().toString()`** → no `%3A` double-encoding regression for hive timestamp partition dirs. `length()`/`modificationTime()` map 1:1. + +### D4 — null-FS guard (fail loud, systemic) +`context.getFileSystem(session)` returns `null` only for a catalog with no storage (HIVEFS-3). A plain-hive catalog always has storage, but guard defensively: `null` FS → loud `DorisConnectorException` (systemic — affects every partition). For the estimate path this is caught by `estimateDataSize`'s `catch (RuntimeException) → -1` (best-effort, unchanged); for scan / `listFileSizes` it fails the query loud (Rule 12), never a silent empty scan. + +### D5 — plumbing +- `HiveScanPlanProvider`: add `ConnectorContext context` ctor arg (3rd, mirroring `HiveConnectorMetadata`; HiveConnector already holds it). Resolve `FileSystem fs = context.getFileSystem(session)` once in `planScan` / `planScanForPartitionBatch`, thread `fs` down to `listAndSplitFiles` → `listDataFiles`. `buildHadoopConf()` STAYS (still used by `planAcidScan:272`, HIVEFS-5) — so `HiveScanPlanProvider`'s `Configuration`/`FileStatus`/`FileSystem`/`Path` imports also STAY. It is no longer passed to `listAndSplitFiles`. In `planScanForPartitionBatch` (non-ACID only) the local `hadoopConf` becomes unused → drop that line. +- `HiveConnectorMetadata.listFileSizes`: resolve `FileSystem fs = context.getFileSystem(session)` under the EXISTING TCCL pin (ANALYZE path, loud by contract — resolving outside the estimate try is fine), pass to `listDataFiles`. +- `HiveConnectorMetadata.estimateDataSizeByListingFiles`: **[Minor-3]** resolve the FS INSIDE `estimateDataSize`'s protected region — pass `context.getFileSystem(session)` inside the size lambda (`location -> sumCachedFileSizes(hiveHandle, location, context.getFileSystem(session))`) so any throw from `getFileSystem` degrades to `-1` (statistics must not fail a query), not loud. `getFileSystem` is a cached field-return so per-location calls are cheap. +- `buildHadoopConf()` in `HiveConnectorMetadata` is used only by these two methods → remove both usages, the orphan private `buildHadoopConf()` method, AND **[Minor-2]** the now-unused `import org.apache.hadoop.conf.Configuration;` (checkstyle UnusedImports). `sumCachedFileSizes` param type flips `Configuration` → `org.apache.doris.filesystem.FileSystem`. + +### D6 — TCCL: keep the existing metadata-layer pins (rationale corrected per red-team) +`listFileSizes` / `estimateDataSizeByListingFiles` already pin the TCCL to the plugin loader (the stats thread is not pinned by fe-core). KEEP those pins. **Corrected rationale [Minor-4]**: the pin protects TCCL-*reflective* config-class loading (Hadoop `ServiceLoader`/`Configuration` reads the TCCL). The engine `DFSFileSystem`/`UserGroupInformation`/`SecurityUtil` classes are resolved by the **fe-filesystem-hdfs** plugin loader (their *defining* loader), not the hive-plugin loader the caller thread is pinned to — so the pin does not itself pick the hadoop copy. The construction path (first `forLocation` lazily builds a `DFSFileSystem` via `SpiSwitchingFileSystem` → `FileSystemFactory`) is **identical to what paimon/iceberg and every other engine FileSystem consumer already trigger in production today** — HIVEFS-4 adds no new classloader locus. `DFSFileSystem.getHadoopFs` additionally self-pins the fe-filesystem-hdfs loader for hdfs/viewfs around the actual `FileSystem.get` (the scheme-reflection point). Simple-auth `createRemoteUser` does no TCCL class reflection (benign); the kerberos-HDFS / custom `hadoop.security.dns` paths ride the same shared construction path already exercised. No NEW connector pin is added. + +## Test plan (connector module has NO Mockito → hand-written recording fake) + +**[Major-3] Full blast radius — 6 test files touch the changed seam/ctor; ALL must compile+pass for "build green":** +- New shared helper `FakeFileSystem` (package-private test class): a recording `org.apache.doris.filesystem.FileSystem` — `forLocation` returns `this` or throws (configurable); `list(Location)` returns a canned in-memory `FileIterator` over set `FileEntry`s or throws (configurable IOException, incl. an `UnsupportedFileSystemException` for the systemic case); all other methods `throw new UnsupportedOperationException`. Reused across the tests below. +- `HiveConnectorInvalidateTest`: replace the `CONF`/`@TempDir`+`Files.write` real-local-listing with a `FakeFileSystem` (list succeeds → leaves a cache entry; `size()` assertions unchanged). `listDataFiles(...,CONF)` → `listDataFiles(...,fakeFs)`. +- `HiveScanBatchModeTest`: `CountingLister.list(String,Configuration)` → `(String,FileSystem)`; `HiveScanPlanProvider` ctor gains `context` (a `new FakeConnectorContext()` — null FS is fine, the fake lister ignores it). +- `HiveConnectorMetadataFileListStatsTest`: `ThrowingFileListingCache.listDataFiles(...,Configuration)` override → `(...,FileSystem)`. +- `HiveReadTransactionTest`: `HiveScanPlanProvider` ctor gains `context` (ACID path; context unused for HIVEFS-4). + +Adapt `HiveFileListingCacheTest`: +- Replace the `CONF` constant + `CountingLister.list(String, Configuration)` with a `FileSystem`-typed seam; `CountingLister` becomes a `DirectoryLister` returning canned `HiveFileStatus` (unchanged — it is above the FS seam). +- Add a hand-written fake `org.apache.doris.filesystem.FileSystem` (a "RecordingFs") for the `listFromFileSystem`-level tests: override `forLocation` (return this or throw) + `listFiles` (return canned `FileEntry`s / throw `IOException`); all other methods throw `UnsupportedOperationException`. +- Rework the 4 real-lister tests to drive `listFromFileSystem(location, fakeFs)`: + - filter test: fake `listFiles` returns `[data1, data2, _SUCCESS, .hidden]` → assert only `data1,data2` (300 bytes) survive. + - `forLocation` throws IOException → assert loud `DorisConnectorException` (NOT `HiveDirectoryListingException`). + - `listFiles` throws IOException → assert skippable `HiveDirectoryListingException`. + - through-cache variants: assert type survives the cache boundary + `size()==0` (failure not cached). +- The scan-integration tests (`scanSkipsOnlyTheFailedPartition...`, `scanFailsLoudOnSystemicFilesystemFailure`, `scanProviderServesRepeatedScansFromTheCache`) update the `HiveScanPlanProvider` ctor to pass a fake `ConnectorContext` whose `getFileSystem` returns the fake FS (or null for a null-FS test). +- `estimateDataSizeIsServedFromTheCache`: `FakeConnectorContext.getFileSystem` returns the fake FS. +- Every assertion must be RED-able (revert the mutation → red). + +## Residual / deferred +- ACID `planAcidScan:272` + `HiveAcidUtil` bare Hadoop → HIVEFS-5. +- S3/OSS exception-split behavior: their FS construction may connect/validate creds eagerly (→ bad creds surface at `forLocation` = SYSTEMIC loud, which is *more* correct than old lazy behavior, not a regression). Flag for the read red-team. +- per-user identity: `getFileSystem(session)` still catalog-level (session ignored) — unchanged from HIVEFS-2/3. diff --git a/plan-doc/tasks/task-list-HIVEFS.md b/plan-doc/tasks/task-list-HIVEFS.md index d017292f343422..d57535d5d98768 100644 --- a/plan-doc/tasks/task-list-HIVEFS.md +++ b/plan-doc/tasks/task-list-HIVEFS.md @@ -32,12 +32,15 @@ - UT（fe-core 有 Mockito；实际用 recording-fake seam `buildCatalogFileSystem`）：`getFileSystem` 懒建、二次返回同一实例（缓存）、空 storage→null、close 幂等转发缓存 FS。 - 校验：`mvn -o -pl fe-core -am test-compile`（已过）。 -- [ ] **HIVEFS-4（连接器·读扫描列文件）**： - - `HiveFileListingCache`：`DirectoryLister` seam 签名 `(String location, Configuration)` → `(String location, FileSystem)`；`listFromFileSystem` 用 `fs.listFiles(Location.of(location))` 替代 `FileSystem.get`+`listStatus`；保留目录/`_`/`.` 前缀过滤 + 两异常语义（systemic `DorisConnectorException` vs per-partition `HiveDirectoryListingException`）。`FileStatus`→`FileEntry`（getPath→location、getLen→length、isDirectory→isDirectory、getModificationTime→mtime）；`HiveFileStatus` DTO 不变。 - - `HiveScanPlanProvider`：`:272` `FileSystem.get(partPath...)` 换；列文件调用点传 `context.getFileSystem(session)`（不再 `FileSystem.get`）。**待核**：`buildHadoopConf()`/`Configuration` 若仍被格式/split 参数或传 BE 所需则保留（本步只摘"建 FileSystem"一职）。 - - UT（连接器无 Mockito，用 recording fake）：注入 fake `FileSystem`/`DirectoryLister`，断言过滤、两异常、缓存/失效（REFRESH）不变，RED-able。 - - **此步 + HIVEFS-3 + 重部署 = 失败用例转绿**。 - - 校验：`mvn -o -pl :fe-connector-hive -am test-compile -Dtest=HiveFileListingCache* -DfailIfNoTests=false`。 +- [x] **HIVEFS-4（连接器·读扫描列文件）** ✅ DONE（fe-connector-hive test-compile SUCCESS、全量 UT 通过、0 checkstyle；设计红队 `wf_f25cc498-2de` GO_WITH_FIXES 7 条全部折入；设计 `designs/FIX-HIVEFS-4-design.md` v2）： + - `HiveFileListingCache`：`DirectoryLister` seam `(String, Configuration)` → `(String, FileSystem)`；`listDataFiles` 同改；`listFromFileSystem` 用 `fs.forLocation(loc)`(SYSTEMIC 边界) + `resolved.list(loc)`(LOCAL 边界，**用 `list()` 非 `listFiles()` 走字面量、不 glob-展开**) 替代 `FileSystem.get`+`listStatus`；保留目录/`_`/`.` 过滤（`FileEntry.name()`）+ 零长保留；`FileEntry`→`HiveFileStatus`（location().uri()/length/modificationTime，路径逐字节等价 `Path.toString()`）。 + - **红队折入的加固**：① 空 FS 守卫→loud（D4）；② `forLocation` catch 拓宽 `IOException | RuntimeException`（Minor-1）；③ list catch 内 `isSystemicResolutionFailure`（cause-chain 走 `UnsupportedFileSystemException`/"No FileSystem for scheme"）→ 惰性"scheme 缺实现"仍 loud（Major-2，迁移自身失败类）。 + - `HiveScanPlanProvider`：+`ConnectorContext` ctor 参（第3位）；非-ACID 路径 `context.getFileSystem(session)` 下发；`buildHadoopConf()` 保留（仅 ACID `planAcidScan:272` 用，该处 hadoop `FileSystem` 全限定，属 HIVEFS-5）；`planScanForPartitionBatch` 去掉无用 `hadoopConf`。 + - `HiveConnectorMetadata`：`listFileSizes`(ANALYZE，loud) 钉内下发 FS；`estimateDataSizeByListingFiles` 在 `estimateDataSize` 保护区(catch→-1)**内**（size lambda）下发 FS（Minor-3，统计不炸查询）；删 `buildHadoopConf()` + `Configuration` import（Minor-2）；`sumCachedFileSizes` 参 `Configuration`→`FileSystem`。 + - `HiveConnector.getScanPlanProvider()` 传 `context`。 + - UT：新增共享 `FakeFileSystem`（recording fake，`listFiles` 抛 AssertionError 钉"必须走 list()"）；`HiveFileListingCacheTest` 19 项（含新增 scheme-missing→loud、glob-字面量、null-FS→loud、path/len/mtime 逐字节）；连带修 `HiveScanBatchModeTest`/`HiveConnectorMetadataFileListStatsTest`/`HiveReadTransactionTest`/`HiveConnectorInvalidateTest`（seam/ctor 签名）。 + - **此步 + HIVEFS-3 + 重部署 = 失败用例 `test_string_dict_filter` 转绿**（e2e 待用户自跑）。 + - 已核：`buildHadoopConf()` fe-core-metadata 侧删除（仅两法用）；`HiveScanPlanProvider` 侧保留（ACID）。 - [ ] **HIVEFS-5（连接器·ACID）**：`HiveAcidUtil:170/255/298` `fs.exists`/`fs.listStatus` → 注入 `FileSystem` 的 `exists(Location)`/`list`/`listFiles`。 - **⚠ 测试基建待解**：`HiveAcidUtilTest` 现对**真 Hadoop LocalFileSystem** 跑（见 `fe-connector-hive/pom.xml:128-137` commons-lang test 注释）。转 Doris `FileSystem` 后须换成 fake Doris `FileSystem` 或 `fe-filesystem-local` 的 local 实现——下个 session 定夺注入方式。 From 7a2d8714951e4d6dd55c55f6502372d9617aa73b Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 16:38:41 +0800 Subject: [PATCH 245/334] [feat](catalog) hive-fs: route ACID directory descent through engine FileSystem (HIVEFS-5) Migrate the hive connector's transactional (ACID) read path off the bare org.apache.hadoop.fs.FileSystem, mirroring HIVEFS-4's non-ACID listing fix: the hive plugin bundles no HDFS impl, so FileSystem.get throws "No FileSystem for scheme hdfs". This path is LIVE post-cutover (hms in SPI_READY_TYPES -> PluginDrivenScanNode -> planScan -> planAcidScan, no gate), so it was broken today for transactional hdfs hive reads. - HiveAcidUtil.getAcidState now takes org.apache.doris.filesystem.FileSystem: exists(new Path)->exists(Location.of); partition listing + the private file helper list via the LITERAL fs.list(Location.of) (never the glob-aware listFiles that per-scheme FS override); FileStatus->FileEntry (getPath().getName()->name(), location().uri()/length()/modificationTime()); AcidState.dataFiles is List; drop hadoop.fs imports (keep hive-common Valid*). - HiveScanPlanProvider.planAcidScan takes the injected FileSystem (context.getFileSystem(session)); drop Path + FileSystem.get; drop the now-orphaned buildHadoopConf() and Configuration/FileStatus/Path imports; correct the stale "Dormant / never reached on a live query" comments to the live reality (red-team wf_792d1900-cc7). - HiveAcidUtilTest migrated to the real Doris LocalFileSystem (fe-filesystem-local, test scope) over a temp tree, with a listFiles- forbidding subclass pinning literal listing; 9/9. commons-lang test dep kept (empirically still needed by hive-common Valid*.writeToString), comment fixed. Engine FS lifecycle unchanged (connector only borrows, never closes). ACID byte-parity e2e deferred to a docker HMS+hdfs harness (verifies a live prod fix). fe-connector-hive: BUILD SUCCESS, HiveAcidUtilTest 9/9, 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- fe/fe-connector/fe-connector-hive/pom.xml | 21 +++-- .../doris/connector/hive/HiveAcidUtil.java | 83 ++++++++++++------- .../connector/hive/HiveScanPlanProvider.java | 55 ++++-------- .../connector/hive/HiveAcidUtilTest.java | 61 +++++++++++--- 4 files changed, 135 insertions(+), 85 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/pom.xml b/fe/fe-connector/fe-connector-hive/pom.xml index a55e1f9769a204..29721c3a038425 100644 --- a/fe/fe-connector/fe-connector-hive/pom.xml +++ b/fe/fe-connector/fe-connector-hive/pom.xml @@ -125,11 +125,22 @@ under the License. test - + + + ${project.groupId} + fe-filesystem-local + ${project.version} + test + + + commons-lang commons-lang diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java index ed0afd6cb6440c..00b391ee3a18d8 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveAcidUtil.java @@ -18,10 +18,11 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.hms.HmsAcidConstants; +import org.apache.doris.filesystem.FileEntry; +import org.apache.doris.filesystem.FileIterator; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.Location; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.FileSystem; -import org.apache.hadoop.fs.Path; import org.apache.hadoop.hive.common.ValidReadTxnList; import org.apache.hadoop.hive.common.ValidReaderWriteIdList; import org.apache.hadoop.hive.common.ValidTxnList; @@ -45,9 +46,9 @@ * delete deltas.

    * *

    The fe-core version drags in {@code FileCacheValue}/{@code LocationPath}/{@code AcidInfo}/ - * {@code HivePartition}; those all drop out at the plugin boundary because the plugin lists files with a - * raw Hadoop {@link FileSystem} and emits {@link HiveScanRange} directly. Only the pure name-parsing plus - * the {@code hive-common} {@code Valid*} algorithm ports.

    + * {@code HivePartition}; those all drop out at the plugin boundary because the plugin lists files with the + * engine-injected Doris {@link FileSystem} and emits {@link HiveScanRange} directly. Only the pure name-parsing + * plus the {@code hive-common} {@code Valid*} algorithm ports.

    * *

    Ref: hive/ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java#getAcidState (the fe-core copy * exists because hive3 cannot read hive4 transaction tables and using hive4 directly is problematic).

    @@ -63,16 +64,16 @@ private HiveAcidUtil() { /** Resolved ACID state for one partition: surviving data files + delete-delta descriptors. */ public static final class AcidState { - private final List dataFiles; + private final List dataFiles; private final List deleteDeltas; - AcidState(List dataFiles, List deleteDeltas) { + AcidState(List dataFiles, List deleteDeltas) { this.dataFiles = dataFiles; this.deleteDeltas = deleteDeltas; } /** Base + non-delete delta bucket files that survive the snapshot; each becomes a scan split. */ - public List getDataFiles() { + public List getDataFiles() { return dataFiles; } @@ -167,7 +168,7 @@ private static boolean isValidMetaDataFile(FileSystem fileSystem, String baseDir throws IOException { String fileLocation = baseDir + "_metadata_acid"; try { - return fileSystem.exists(new Path(fileLocation)); + return fileSystem.exists(Location.of(fileLocation)); } catch (IOException e) { return false; } @@ -244,19 +245,39 @@ public boolean accept(String fileName) { } } + /** + * Lists the immediate children (files and directories) of {@code dir}, non-recursively, via the + * engine-injected Doris {@link FileSystem#list} — the literal listing that mirrors the old + * {@code FileSystem.listStatus}. + * + *

    Literal, not glob: uses {@code fs.list(loc)}, never {@code fs.listFiles(loc)}. The per-scheme + * filesystems ({@code DFSFileSystem}, {@code S3CompatibleFileSystem}) override {@code listFiles} with a + * glob-aware branch that would treat a location containing {@code [}/{@code *}/{@code ?} as a pattern; a hive + * partition/delta location can legitimately contain those characters, and the old {@code listStatus} never + * glob-expanded. Mirrors {@code HiveFileListingCache.listFromFileSystem}.

    + */ + private static List listEntries(FileSystem fs, String dir) throws IOException { + List entries = new ArrayList<>(); + try (FileIterator it = fs.list(Location.of(dir))) { + while (it.hasNext()) { + entries.add(it.next()); + } + } + return entries; + } + /** * Lists the immediate file children of {@code dir} (directories excluded). * *

    Mirrors fe-core {@code globList(fs, dir, false)} on a bare directory path: with no wildcard the - * glob has a {@code null} pattern, so it returns files only and skips any nested directory. A raw - * {@link FileSystem#listStatus} returns both, so the directory entries are filtered out here.

    + * glob has a {@code null} pattern, so it returns files only and skips any nested directory. The literal + * {@link #listEntries} returns both, so the directory entries are filtered out here.

    */ - private static List listFiles(FileSystem fs, String dir) throws IOException { - FileStatus[] statuses = fs.listStatus(new Path(dir)); - List files = new ArrayList<>(statuses.length); - for (FileStatus status : statuses) { - if (!status.isDirectory()) { - files.add(status); + private static List listFiles(FileSystem fs, String dir) throws IOException { + List files = new ArrayList<>(); + for (FileEntry entry : listEntries(fs, dir)) { + if (!entry.isDirectory()) { + files.add(entry); } } return files; @@ -265,7 +286,7 @@ private static List listFiles(FileSystem fs, String dir) throws IOEx /** * Resolves the ACID state of one partition directory under the given snapshot. * - * @param fs raw Hadoop file system for the partition location + * @param fs engine-injected Doris file system for the partition location * @param partitionPath the partition directory (e.g. {@code hdfs://.../data_id=200103}) * @param txnValidIds the two serialized {@code Valid*} lists keyed by {@link HmsAcidConstants} * @param isFullAcid full-ACID (bucket_ filter + delete deltas) vs insert-only (accept-all) @@ -295,7 +316,7 @@ public static AcidState getAcidState(FileSystem fs, String partitionPath, //hdfs://xxxxx/user/hive/warehouse/username/data_id=200103 // List all files and folders, without recursion. - FileStatus[] partitionEntries = fs.listStatus(new Path(partitionPath)); + List partitionEntries = listEntries(fs, partitionPath); String oldestBase = null; long oldestBaseWriteId = Long.MAX_VALUE; @@ -304,9 +325,9 @@ public static AcidState getAcidState(FileSystem fs, String partitionPath, boolean haveOriginalFiles = false; List workingDeltas = new ArrayList<>(); - for (FileStatus entry : partitionEntries) { + for (FileEntry entry : partitionEntries) { if (entry.isDirectory()) { - String dirName = entry.getPath().getName(); //dirName: base_xxx,delta_xxx,... + String dirName = entry.name(); //dirName: base_xxx,delta_xxx,... String dirPath = partitionPath + "/" + dirName; if (dirName.startsWith("base_")) { @@ -420,7 +441,7 @@ && isValidBase(fs, dirPath, base, validWriteIdList)) { } } - List dataFiles = new ArrayList<>(); + List dataFiles = new ArrayList<>(); List deleteDeltas = new ArrayList<>(); FileFilter fileFilter = isFullAcid ? new FullAcidFileFilter() : new InsertOnlyFileFilter(); @@ -428,11 +449,11 @@ && isValidBase(fs, dirPath, base, validWriteIdList)) { // delta directories for (ParsedDelta delta : deltas) { String location = delta.path; - List entries = listFiles(fs, location); + List entries = listFiles(fs, location); if (delta.deleteDelta) { List deleteDeltaFileNames = new ArrayList<>(); - for (FileStatus entry : entries) { - String name = entry.getPath().getName(); + for (FileEntry entry : entries) { + String name = entry.name(); if (fileFilter.accept(name)) { deleteDeltaFileNames.add(name); } @@ -440,8 +461,8 @@ && isValidBase(fs, dirPath, base, validWriteIdList)) { deleteDeltas.add(new DeleteDelta(location, deleteDeltaFileNames)); continue; } - for (FileStatus entry : entries) { - if (fileFilter.accept(entry.getPath().getName())) { + for (FileEntry entry : entries) { + if (fileFilter.accept(entry.name())) { dataFiles.add(entry); } } @@ -449,9 +470,9 @@ && isValidBase(fs, dirPath, base, validWriteIdList)) { // base if (bestBasePath != null) { - List entries = listFiles(fs, bestBasePath); - for (FileStatus entry : entries) { - if (fileFilter.accept(entry.getPath().getName())) { + List entries = listFiles(fs, bestBasePath); + for (FileEntry entry : entries) { + if (fileFilter.accept(entry.name())) { dataFiles.add(entry); } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 448edf99696b60..6dbc26610901b3 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -28,11 +28,9 @@ import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.FileEntry; import org.apache.doris.filesystem.FileSystem; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.Path; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -133,9 +131,10 @@ public List planScan( List ranges = new ArrayList<>(); if (hiveHandle.isTransactional()) { - // Transactional (ACID) table: descend into base/delta directories under the query's - // write-id snapshot and emit ACID-annotated ranges. (Dormant — see planAcidScan.) - planAcidScan(session, hiveHandle, partitions, buildHadoopConf(), fileFormat, + // Transactional (ACID) table: descend into base/delta directories under the query's write-id + // snapshot and emit ACID-annotated ranges. Borrows the engine's per-catalog Doris FileSystem to + // list (same source as the non-ACID branch below; the engine owns its lifecycle — never closed here). + planAcidScan(session, hiveHandle, partitions, context.getFileSystem(session), fileFormat, splittable, targetSplitSize, ranges); } else { // Borrow the engine's per-catalog Doris FileSystem to list partition directories (see field javadoc). @@ -245,15 +244,15 @@ public List planScanForPartitionBatch( * surviving base/delta data files and delete-delta directories, and emits one ACID-annotated * {@link HiveScanRange} per data-file split. The BE subtracts the delete deltas on read.

    * - *

    Dormant: hive is not yet in {@code SPI_READY_TYPES}, so this path is never reached on a - * live query — transactional reads still flow through fe-core {@code HiveScanNode}. It opens a real - * metastore transaction/lock; the matching commit (lock release) is driven by - * {@link HiveReadTransactionManager#deregister} at query finish, wired only at the read cutover - * (P7.4/P7.5) together with routing hive through the plugin. Do not flip one without the other, or a - * shared read lock would leak.

    + *

    Live path. Post-cutover {@code hms} is in {@code SPI_READY_TYPES}, so an hms-catalog + * transactional table is a {@code PluginDrivenExternalTable} routed through {@code PluginDrivenScanNode} + * straight into this method on a live query (the only read gate is the full-ACID ORC-format check below). + * It opens a real metastore transaction/lock; the matching commit (lock release) is driven by + * {@link HiveReadTransactionManager#deregister} at query finish (via {@link #releaseReadTransaction}), + * without which the shared read lock would leak for the metastore's lifetime.

    */ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, - List partitions, Configuration hadoopConf, HiveFileFormat fileFormat, + List partitions, FileSystem fs, HiveFileFormat fileFormat, boolean splittable, long targetSplitSize, List ranges) { boolean isFullAcid = handle.isFullAcid(); if (isFullAcid && !ORC_ACID_INPUT_FORMAT.equals(handle.getInputFormat())) { @@ -278,11 +277,9 @@ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; try { - Path partPath = new Path(partition.location); - // ACID path still uses bare Hadoop FileSystem (HIVEFS-5 will migrate HiveAcidUtil); fully-qualified - // here because the unqualified FileSystem now refers to the injected Doris filesystem. - org.apache.hadoop.fs.FileSystem fs = - org.apache.hadoop.fs.FileSystem.get(partPath.toUri(), hadoopConf); + // Descend the ACID directory tree through the engine-injected Doris FileSystem. The hive plugin + // bundles no HDFS impl, so a bare Hadoop FileSystem.get here would throw "No FileSystem for + // scheme hdfs" (FIX-HIVEFS); the engine owns this FileSystem's lifecycle — never closed here. HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( fs, partition.location, txnValidIds, isFullAcid); // Only full-ACID reads are marked transactional_hive (delete deltas applied by the BE @@ -292,9 +289,9 @@ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, String acidLocation = isFullAcid ? partition.location : null; List encodedDeltas = isFullAcid ? encodeDeleteDeltas(state.getDeleteDeltas()) : null; - for (FileStatus dataFile : state.getDataFiles()) { - splitFile(dataFile.getPath().toString(), dataFile.getLen(), - dataFile.getModificationTime(), partition, partFormat, splittable, + for (FileEntry dataFile : state.getDataFiles()) { + splitFile(dataFile.location().uri(), dataFile.length(), + dataFile.modificationTime(), partition, partFormat, splittable, targetSplitSize, acidLocation, encodedDeltas, ranges); } } catch (IOException e) { @@ -536,22 +533,6 @@ private long getTargetSplitSize(ConnectorSession session) { return DEFAULT_TARGET_SPLIT_SIZE; } - private Configuration buildHadoopConf() { - Configuration conf = new Configuration(); - for (Map.Entry entry : catalogProperties.entrySet()) { - conf.set(entry.getKey(), entry.getValue()); - } - // Set default FS from location properties if present - String defaultFs = catalogProperties.get("fs.defaultFS"); - if (defaultFs == null) { - defaultFs = catalogProperties.get("hadoop.fs.defaultFS"); - } - if (defaultFs != null) { - conf.set("fs.defaultFS", defaultFs); - } - return conf; - } - private boolean isLocationProperty(String key) { return key.startsWith("fs.") || key.startsWith("hadoop.") diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java index 53e8a89c154b26..7be2319459b6a1 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveAcidUtilTest.java @@ -18,10 +18,11 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.hms.HmsAcidConstants; +import org.apache.doris.filesystem.FileEntry; +import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.filesystem.Location; +import org.apache.doris.filesystem.local.LocalFileSystem; -import org.apache.hadoop.conf.Configuration; -import org.apache.hadoop.fs.FileStatus; -import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.hive.common.ValidReadTxnList; import org.apache.hadoop.hive.common.ValidReaderWriteIdList; import org.apache.hadoop.hive.common.ValidTxnList; @@ -33,27 +34,33 @@ import java.io.IOException; import java.nio.file.Files; import java.util.BitSet; +import java.util.Collections; import java.util.HashMap; import java.util.List; import java.util.Map; /** - * Tests the pure ACID directory descent {@link HiveAcidUtil#getAcidState} against a real Hadoop - * {@link FileSystem} over a local temp tree. + * Tests the pure ACID directory descent {@link HiveAcidUtil#getAcidState} against a real Doris + * {@link FileSystem} (the test-only {@link LocalFileSystem}) over a local temp tree — the Doris equivalent of the + * old Hadoop {@code LocalFileSystem}, now that {@link HiveAcidUtil} lists via the engine-injected Doris filesystem. * *

    WHY: for a transactional Hive table the reader must reconstruct the correct snapshot from the * base/delta/delete-delta directory layout — pick the best valid base, layer on the deltas whose * write-id range is still valid, drop obsolete/out-of-snapshot directories, and hand the BE the * delete-delta file names so it can subtract deleted rows. A wrong base or a dropped delta silently * returns stale or over-/under-deleted data. These tests pin that selection algorithm.

    + * + *

    The filesystem is a {@link LiteralListingLocalFileSystem}: it forbids the glob-aware + * {@link FileSystem#listFiles}, so every test here also pins that {@link HiveAcidUtil} lists via the literal + * {@link FileSystem#list} (matching the old {@code FileSystem.listStatus}).

    */ public class HiveAcidUtilTest { @TempDir java.nio.file.Path tempDir; - private FileSystem localFs() throws IOException { - return FileSystem.getLocal(new Configuration()); + private FileSystem localFs() { + return new LiteralListingLocalFileSystem(); } /** Creates {@code //} with 1 byte of content. */ @@ -91,13 +98,13 @@ public void testBestBaseWithDeltaAndDeleteDelta() throws IOException { HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( localFs(), tempDir.toString(), snapshot(6L), true); - List dataFiles = state.getDataFiles(); + List dataFiles = state.getDataFiles(); Assertions.assertEquals(2, dataFiles.size(), "surviving data files must be exactly base_5 + delta_6 bucket files"); boolean hasBase5 = false; boolean hasDelta6 = false; - for (FileStatus f : dataFiles) { - String p = f.getPath().toString(); + for (FileEntry f : dataFiles) { + String p = f.location().uri(); Assertions.assertFalse(p.contains("base_0000002"), "superseded base must be dropped: " + p); Assertions.assertFalse(p.contains("delta_0000009"), "out-of-snapshot delta dropped: " + p); Assertions.assertFalse(p.endsWith("_flush_length"), "side file excluded: " + p); @@ -179,7 +186,7 @@ public void testBaseWithUncommittedVisibilityTxnIsSkipped() throws IOException { HiveAcidUtil.AcidState state = HiveAcidUtil.getAcidState( localFs(), tempDir.toString(), ids, true); Assertions.assertEquals(1, state.getDataFiles().size()); - Assertions.assertTrue(state.getDataFiles().get(0).getPath().toString().contains("/base_0000003/"), + Assertions.assertTrue(state.getDataFiles().get(0).location().uri().contains("/base_0000003/"), "a base whose visibility txn is uncommitted must be skipped for the committed base"); } @@ -197,7 +204,7 @@ public void testHighestBaseInvalidFallsBackToLowerValidBaseAndDropsObsoleteDelta Assertions.assertEquals(1, state.getDataFiles().size(), "only the best VALID base (base_4) survives; the higher but invalid base_8 is rejected"); - String p = state.getDataFiles().get(0).getPath().toString(); + String p = state.getDataFiles().get(0).location().uri(); Assertions.assertTrue(p.contains("/base_0000004/"), p); Assertions.assertFalse(p.contains("base_0000008"), "base above the write-id watermark is invalid"); } @@ -223,4 +230,34 @@ public void testMissingValidTxnListThrows() throws IOException { () -> HiveAcidUtil.getAcidState(localFs(), tempDir.toString(), ids, true)); Assertions.assertTrue(ex.getMessage().contains("ValidTxnList"), ex.getMessage()); } + + /** + * A Doris {@link LocalFileSystem} that FORBIDS the glob-aware {@link FileSystem#listFiles} / + * {@link FileSystem#listFilesRecursive}. Every test lists through this, so any regression in + * {@link HiveAcidUtil} from the literal {@link FileSystem#list} to {@code listFiles()} fails loud here. + * + *

    The production per-scheme filesystems ({@code DFSFileSystem}, {@code S3CompatibleFileSystem}) override + * {@code listFiles} to treat a location containing {@code [}/{@code *}/{@code ?} as a glob pattern; a real hive + * delta/partition location can contain those, and the old {@code FileSystem.listStatus} never glob-expanded. + * Plain {@link LocalFileSystem#listFiles} is the (literal) interface default, so it alone cannot catch a + * {@code list()->listFiles()} regression — hence this guard. Mirrors {@code FakeFileSystem.listFiles} throwing + * in the non-ACID listing tests. + */ + private static final class LiteralListingLocalFileSystem extends LocalFileSystem { + LiteralListingLocalFileSystem() { + super(Collections.emptyMap()); + } + + @Override + public List listFiles(Location dir) { + throw new AssertionError( + "HiveAcidUtil must list via the literal list(), never the glob-aware listFiles()"); + } + + @Override + public List listFilesRecursive(Location dir) { + throw new AssertionError( + "HiveAcidUtil must list via the literal list(), never the glob-aware listFilesRecursive()"); + } + } } From 3faa60e9f54fde579c32261a5b37c7e8074877aa Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 16:40:16 +0800 Subject: [PATCH 246/334] [doc](catalog) hive-fs: land HIVEFS-5 design (red-team folded), mark done, repoint to write-path HIVEFS-6 FIX-HIVEFS-5 design + adversarial red-team (wf_792d1900-cc7): migrate the hive connector's ACID directory descent off bare Hadoop FileSystem. 4 of 5 lenses rated the migration code SOUND; the one REAL finding (planAcidScan is LIVE, not dormant, post-cutover) is folded into the design's dynamic-state/e2e sections and the shipped code comments. Test-injection choice (fe-filesystem-local real LocalFileSystem + a listFiles-forbidding subclass) recorded; the commons-lang test dep is empirically confirmed still needed (hive-common Valid*), not orphan. task-list-HIVEFS: HIVEFS-5 checked done with the concrete edit list; Open #3 (test injection) resolved. HANDOFF: repoint next step to write-path HIVEFS-6, record that transactional hive reads are live-through-plugin post-cutover. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 14 +-- plan-doc/tasks/designs/FIX-HIVEFS-5-design.md | 90 +++++++++++++++++++ plan-doc/tasks/task-list-HIVEFS.md | 11 ++- 3 files changed, 106 insertions(+), 9 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-HIVEFS-5-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 2c984cbd8c4191..143c1071b6081b 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,7 @@ --- -# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = ACID HIVEFS-5） +# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 写路径 HIVEFS-6，最险） > 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。** @@ -21,18 +21,22 @@ **✅ HIVEFS-4 DONE（code `7e06c2aa2a9`）= 连接器·读扫描列文件已转引擎 FileSystem**：`HiveFileListingCache` seam `Configuration`→`org.apache.doris.filesystem.FileSystem`；`listFromFileSystem` 用 `fs.forLocation(loc)`(SYSTEMIC 边界·loud) + `resolved.list(loc)`(LOCAL 边界·skippable，**字面量 `list()` 非 glob 的 `listFiles()`**) 替代裸 `FileSystem.get`+`listStatus`；3 调用点（scan `listAndSplitFiles`、metadata `listFileSizes`/`estimate`）经 `context.getFileSystem(session)` 下发；`HiveScanPlanProvider` +ctx ctor 参；metadata 删 `buildHadoopConf`+`Configuration` import。设计红队 `wf_f25cc498-2de`(GO_WITH_FIXES 7 条全折入：null-FS→loud、`forLocation` catch 拓宽、`isSystemicResolutionFailure` 让惰性"No FileSystem for scheme"仍 loud、literal `list()`、estimate 保护区内下发、4 连带测试文件、去 orphan import)。**fe-connector-hive 289/289、0 checkstyle**。设计 `designs/FIX-HIVEFS-4-design.md`(v2)。**此步 + HIVEFS-3 + 重部署 = `test_string_dict_filter` 应转绿（e2e 待用户自跑）。ACID `planAcidScan:272` 的裸 Hadoop 未动（HIVEFS-5）。** -**⭐ 下一步 = HIVEFS-5（连接器·ACID）**：`HiveAcidUtil:170/255/298` 的 `fs.exists`/`fs.listStatus` + `HiveScanPlanProvider.planAcidScan:272`（现全限定的 `org.apache.hadoop.fs.FileSystem.get`，喂 `HiveAcidUtil.getAcidState`）换注入 `FileSystem`。**⚠ 测试基建待定**：`HiveAcidUtilTest` 现对真 Hadoop LocalFileSystem 跑，转 Doris `FileSystem` 后须换 fake（可复用本轮新增的 `FakeFileSystem`）或 `fe-filesystem-local`——下个 session 定夺注入方式。之后：**6 写路径（最险，落地前须多 agent 对抗红队）→ 7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 +**✅ HIVEFS-5 DONE（code `7a2d8714951`）= 连接器·ACID 目录下降已转引擎 FileSystem**：`HiveAcidUtil.getAcidState(FileSystem,…)` 换 Doris `FileSystem`——`fs.exists(new Path)`→`exists(Location.of)`；分区列 `fs.listStatus`→`listEntries`(迭代 `fs.list` 收全部含目录)；私有 `listFiles` 助手→`fs.list`+过滤目录（**字面量 `list()` 非 glob 的 `listFiles()`**，镜像 HIVEFS-4）；`FileStatus`→`FileEntry`(`name()`/`location().uri()`/`length()`/`modificationTime()`)；`AcidState.dataFiles`→`List`；删 hadoop.fs import（留 hive-common `Valid*`）。`HiveScanPlanProvider.planAcidScan`：签名 `Configuration`→`FileSystem`、传 `context.getFileSystem(session)`、删 `Path`+`FileSystem.get`、删孤儿 `buildHadoopConf`+3 import。**红队 `wf_792d1900-cc7`（5 lens+逐条对抗）：迁移码 4 lens SOUND，唯一 REAL(major)=`planAcidScan` 非休眠而是 live-broken**（翻闸后 `hms`∈`SPI_READY_TYPES`→`PluginDrivenScanNode`→`planScan`→`planAcidScan` 无门，hdfs 事务读今天即炸；已实测确认+订正 `:137`/`:248-253` 陈旧"Dormant/never reached"注释）。测试注入=`fe-filesystem-local` 真 `LocalFileSystem`+抛-`listFiles` 子类守卫（非 fake）；`commons-lang` test 依赖实测证实仍需（hive-common `Valid*`）。**fe-connector-hive 9/9、0 checkstyle、BUILD SUCCESS**。设计 `designs/FIX-HIVEFS-5-design.md`。 + +**⭐ 下一步 = HIVEFS-6（连接器·写路径，最险，落地前须多 agent 对抗红队）**：`HiveConnectorTransaction` `resolveObjectStoreFileSystem`（坏 ServiceLoader stub :784-793）→ `context.getFileSystem(session).forLocation(loc)`；MPU `instanceof ObjFileSystem`(:809/1336) 落到 `forLocation` 返回的具体 FS；事务内 delete/rename/exists 裸 Hadoop 一并换；`close()` 不关借用引擎 FS；FS/identity 于 `beginWrite`(:207) 捕获。之后：**7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 **已探明的去风险事实（勿重查）**： - **引擎侧已落地（HIVEFS-3）**——`DefaultConnectorContext.getFileSystem` 用已持的 `storagePropertiesSupplier` + `SpiSwitchingFileSystem`（范式镜像 `cleanupEmptyManagedLocation:348`）；close 由 catalog 在 `onClose`/换连接器处关（context 单一持有、sibling 共享）。 - **TCCL 无需连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs **自钉**到自身插件 loader 再 `FileSystem.get`、finally 还原（注释即描述 “No FileSystem for scheme hdfs” 场景）。连接器去 jar 后任意 TCCL 调用皆安全。 - 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction:809/1336`）→ `FileSystem.forLocation`（HIVEFS-1 已加：非切换返回 this、切换返回具体 FS）就位待 HIVEFS-6 用。 -- 连接器裸 Hadoop 面**很小**：`listStatus`×3 + `exists`×1 + `FileStatus` 字段（getPath/getLen/isDirectory/getModificationTime），全映射 `FileEntry`。 +- 连接器裸 Hadoop 面（读扫描 HIVEFS-4 + ACID HIVEFS-5 已清）：仅剩**写路径** `HiveConnectorTransaction`（MPU/rename/delete/exists）待 HIVEFS-6；`FileStatus` 字段全映射 `FileEntry`（读侧已验：`name()`/`location().uri()`/`length()`/`modificationTime()`/`isDirectory()`）。 - fe-filesystem 全 9 插件（hdfs/s3/oss/cos/obs/azure/http/local/broker）已部署 → hive 表任意后端天然支持（免 bundle hadoop-aws/huaweicloud，架构红利）。 -**⚠ 待下个 session 核定**（task-list Open §）：`HiveScanPlanProvider.buildHadoopConf`/conf 去留（HIVEFS-4）· `HiveAcidUtilTest` 真-LocalFS→fake 迁移（HIVEFS-5）· 写路径 commit/abort 时 FS/identity 捕获时机（HIVEFS-6，begin 捕获）。 +**⚠ 待下个 session 核定**（task-list Open §）：写路径 commit/abort 时 FS/identity 捕获时机（HIVEFS-6，`beginWrite` 捕获）。（`buildHadoopConf` 去留=HIVEFS-4/5 已删；`HiveAcidUtilTest` 注入方式=HIVEFS-5 已定 `fe-filesystem-local`。） + +**⚠ 重要事实（红队 HIVEFS-5 确认）**：翻闸后 hive **事务表读已 live 走 plugin**（`planAcidScan` 非休眠）；HIVEFS-5 前它在 hdfs 上就是坏的（`No FileSystem for scheme hdfs`），HIVEFS-5 修的是 live 生产 bug（e2e 因需 docker HMS+hdfs harness 延后，非"不可测"）。 -**⚠ 状态**：HIVEFS-0/1/2/3/4 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9` + doc commit）。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。HIVEFS-4 已按此走完一轮（设计→红队→折入→实现→289 UT→commit）。 +**⚠ 状态**：HIVEFS-0/1/2/3/4/5 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9`/`7a2d8714951` + doc commit）。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。HIVEFS-4/5 已按此各走完一轮（设计→红队→折入→实现→UT→commit）。 --- diff --git a/plan-doc/tasks/designs/FIX-HIVEFS-5-design.md b/plan-doc/tasks/designs/FIX-HIVEFS-5-design.md new file mode 100644 index 00000000000000..9e984bc87c67a3 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-HIVEFS-5-design.md @@ -0,0 +1,90 @@ +# FIX-HIVEFS-5 — 连接器 ACID 目录下降改经引擎注入 `FileSystem`（去 `HiveAcidUtil` 裸 Hadoop） + +> 承接 FIX-HIVEFS（设计 `FIX-HIVEFS-design.md`、清单 `task-list-HIVEFS.md`）。HIVEFS-3（引擎实现）+ HIVEFS-4（读扫描列文件）已入库；本步是**连接器 ACID 路径**——事务表 base/delta/delete-delta 目录下降仍走裸 `org.apache.hadoop.fs.FileSystem`，是同一"hive 插件无 HDFS 实现→`No FileSystem for scheme hdfs`"缺口的最后一条读路径分支。 +> 范围 = 仅 hive 连接器 ACID 读路径（`HiveAcidUtil` + `HiveScanPlanProvider.planAcidScan`）。写路径（HIVEFS-6）、去 jar（HIVEFS-7）不属本步。 + +## Problem + +`HiveAcidUtil`（连接器侧 ACID 目录名解析，fe-core `AcidUtil.getAcidState` 的插件移植）仍用裸 Hadoop： +- 签名 `getAcidState(org.apache.hadoop.fs.FileSystem fs, String partitionPath, …)`； +- `HiveAcidUtil:170` `fileSystem.exists(new Path(fileLocation))`（`isValidMetaDataFile`——base 无 `_v` 后缀时探 `_metadata_acid`）； +- `HiveAcidUtil:255` `fs.listStatus(new Path(dir))`（私有 `listFiles` 助手，列 base/delta 目录内 bucket 文件、过滤子目录）； +- `HiveAcidUtil:298` `fs.listStatus(new Path(partitionPath))`（列分区目录**全部**条目——需同时看到 base_/delta_ 子目录与"原始文件"）； +- 返回 `AcidState.dataFiles : List`。 + +调用方 `HiveScanPlanProvider.planAcidScan:281-285` 用 `org.apache.hadoop.fs.FileSystem.get(partPath.toUri(), buildHadoopConf())` 造 FS 喂进去；`planAcidScan:295-298` 迭代 `state.getDataFiles()`（`FileStatus`）取 `getPath()/getLen()/getModificationTime()` 造 split。 + +根因同 HIVEFS-4：hive 插件类加载器隔离、`lib/` 无 `DistributedFileSystem` → 裸 `FileSystem.get` 对 hdfs 抛 `No FileSystem for scheme "hdfs"`。老 fe-core 经 `org.apache.doris.filesystem.FileSystem` 列文件，从不裸 Hadoop。 + +**动态状态（红队 `wf_792d1900-cc7` 纠正——原"休眠"判定为假）**：`planAcidScan` **已 live**、且当前在 hdfs 上就是坏的。Phase 2 原子翻闸（commit `b25b15c357f`）已把 `hms` 加入 `SPI_READY_TYPES`（`CatalogFactory:55`），故 `type=hms` 目录的每张表都是 `PluginDrivenExternalTable`（hive 非 MVCC 走基类），`PhysicalPlanTranslator.visitPhysicalFileScan:812` **先**匹配它 → `PluginDrivenScanNode` → `planScan` → 事务表 `isTransactional()` **无门**直入 `planAcidScan`（唯一读侧守卫是 full-ACID 的 ORC 格式检查 `:262`，无"事务读不支持"早拒）。故一张 hdfs 事务表**今天 live 查询即命中** `planAcidScan:284-285` 的裸 `FileSystem.get` → `No FileSystem for scheme "hdfs"`——正是本步要修的 bug。**含义**：① `HiveScanPlanProvider:137`（`(Dormant — see planAcidScan.)`）与 `:248-253` 的 javadoc（`hive is not yet in SPI_READY_TYPES, so this path is never reached on a live query`）是翻闸前的**陈旧假注释**，本步须一并订正（否则 shipping 自相矛盾的文件）；② 本步是修**live 生产 bug**（非休眠路径整洁），ACID 读字节等价须真回归（e2e 见下，延后理由是 harness 可用性、非休眠）。 + +## Root Cause + +连接器 ACID 分支在 catalog-SPI 迁移中把老 fe-core 的 `org.apache.doris.filesystem.FileSystem` 列文件走样成裸 `org.apache.hadoop.fs.FileSystem`，与 HIVEFS-4 读扫描路径同源。引擎已经 HIVEFS-3 提供 `ConnectorContext.getFileSystem(session)`（per-catalog `SpiSwitchingFileSystem`，引擎所有、连接器借用不 close），HIVEFS-4 已把读扫描列文件接上；ACID 分支是最后一条未接的读路径。 + +## Design + +镜像 HIVEFS-4：把 `HiveAcidUtil` 与 `planAcidScan` 全部裸 Hadoop I/O 换成引擎注入的 Doris `FileSystem`（`SpiSwitchingFileSystem`）。 + +### 1. `HiveAcidUtil`（连接器） +- 签名 `getAcidState(org.apache.doris.filesystem.FileSystem fs, String partitionPath, Map, boolean)` —— 类型换 Doris `FileSystem`，其余不变。 +- `exists`：`fs.exists(Location.of(fileLocation))`（`isValidMetaDataFile`）。捕获 `IOException`→`false` 保留（原语义）。 +- **分区目录列（:298）**：迭代 `fs.list(Location.of(partitionPath))` 收集**全部** `FileEntry`（含目录），非 `listFiles()`。 +- **私有 `listFiles` 助手（:255）**：迭代 `fs.list(Location.of(dir))`，过滤 `!e.isDirectory()` → 文件 `FileEntry`（**字面量 `list()`，非 glob 的 `listFiles()`**——见下"字面量列"）。 +- `FileStatus` 字段映射：`entry.isDirectory()`→`FileEntry.isDirectory()`；`entry.getPath().getName()`→`FileEntry.name()`；`AcidState.dataFiles`/`getDataFiles()` 类型 `List`→`List`。 +- 删 import `org.apache.hadoop.fs.{FileStatus,FileSystem,Path}`；加 `org.apache.doris.filesystem.{FileEntry,FileIterator,FileSystem,Location}`。**保留** `org.apache.hadoop.hive.common.Valid*`（ACID 快照算法，非 FS I/O，不属去 Hadoop 目标）。 +- 更新 class-javadoc（"raw Hadoop FileSystem"）+ `getAcidState` 的 `@param fs` javadoc。 + +### 2. `HiveScanPlanProvider.planAcidScan`（连接器） +- 调用点（:138）：`buildHadoopConf()` → `context.getFileSystem(session)`（与非-ACID 分支 :142 同源，借引擎 per-catalog FS）。 +- 签名 `Configuration hadoopConf` → `org.apache.doris.filesystem.FileSystem fs`。 +- 删 `Path partPath = new Path(...)` + `org.apache.hadoop.fs.FileSystem.get(...)`（:281-285），直接把注入的 `fs` 喂 `getAcidState`。 +- 数据文件循环（:295-298）：`FileStatus dataFile`→`FileEntry dataFile`；`.getPath().toString()`→`.location().uri()`；`.getLen()`→`.length()`；`.getModificationTime()`→`.modificationTime()`。 +- 删已孤儿的 `buildHadoopConf()`（:539，唯一调用者是 ACID）。**保留** `catalogProperties`/`isLocationProperty`（`getScanNodeProperties:365/367` 仍用）。 +- 删孤儿 import `org.apache.hadoop.conf.Configuration`（:33）、`org.apache.hadoop.fs.FileStatus`（:34）、`org.apache.hadoop.fs.Path`（:35）；加 `org.apache.doris.filesystem.FileEntry`（`FileSystem` :31 已 import）。 +- 更新 :282-283 陈旧注释（"ACID path still uses bare Hadoop"）。 +- **订正陈旧"休眠"注释（红队 folded）**：`:137` 的 `(Dormant — see planAcidScan.)` 与 `:248-253` javadoc 的"hive is not yet in SPI_READY_TYPES / never reached on a live query"——翻闸后已假（见上"动态状态"）。改为如实：此路径 live（`type=hms` 事务表读经 `PluginDrivenScanNode` 直达）；保留仍然正确的事务生命周期说明（开元数据读锁 → 查询结束经 `releaseReadTransaction`/`deregister` 释放）。 + +### 3. 失败语义（与 HIVEFS-4 的关键区别） +ACID 路径**任何列文件失败即 loud**——现有 `planAcidScan` 的 `catch(IOException)` 把整批包成 `DorisConnectorException("Failed to list ACID files for partition: …")`、不跳分区。注入 `SpiSwitchingFileSystem` 后，scheme 解析失败（`No StorageProperties`/`No FileSystem for scheme`/factory 异常）经内部 `forLocation` 以 `IOException` 从 `fs.list`/`fs.exists` 冒出 → 同一 catch → loud。**与老 `FileSystem.get`（scheme 失败也抛 IOException→loud）逐位一致**。故本步**不需要** HIVEFS-4 读扫描的 systemic-vs-local 分级 / `isSystemicResolutionFailure` 重分类（那是为"跳单坏分区"服务，ACID 从不跳）。 + +### 4. 为何直接传 switching FS（不先 `forLocation` 解具体 FS） +`getAcidState` 内所有路径（分区目录、base_/delta_ 子目录、`_metadata_acid` 文件）同属一个分区 location → 同 scheme/authority/`StorageProperties`。`SpiSwitchingFileSystem.list/exists` 内部各自 `forLocation`（按 `StorageProperties` 缓存、首次后命中）→ 路由到具体 FS 的**字面量 `list()`**。传 facade 让 util 签名最小（仅 Doris `FileSystem`，无 forLocation 步），且正确。（备选：在 planAcidScan 里 `forLocation` 解一次传具体 FS——亦可，但为无收益的额外步；ACID 不需要 forLocation 边界带来的失败分级。） + +### 5. 字面量列（list() 非 listFiles()） +per-scheme FS（`DFSFileSystem`/`S3CompatibleFileSystem`）override `listFiles` 为 **glob 感知**分支，会把含 `[`/`*`/`?` 的 location 当模式；老 `listStatus` 从不 glob 展开，而 hive location 可合法含这些字符（分区值）。故 `HiveAcidUtil` 一律 `fs.list(Location.of(x))`（`SpiSwitchingFileSystem.list`→具体 FS 字面量 `list`），过滤目录用 `FileEntry.isDirectory()`。**镜像 HIVEFS-4 `listFromFileSystem` 的既定规则**。 + +## Implementation Plan +1. `HiveAcidUtil.java`：签名/内部 I/O/DTO 类型/import/javadoc 全换（§1）。 +2. `HiveScanPlanProvider.java`：调用点/签名/删 `FileSystem.get`/循环/删 `buildHadoopConf`/import/注释（§2）。 +3. `HiveAcidUtilTest.java`：真 Hadoop LocalFileSystem → Doris `LocalFileSystem`（见 Test Plan）。 +4. `fe-connector-hive/pom.xml`：加 `fe-filesystem-local` **test scope** 依赖。 +5. build + 靶向 UT + 0 checkstyle + import 门净。 + +## Risk Analysis +- **`FileStatus`→`FileEntry` 逐位等价**：`getPath().toString()`↔`location().uri()`（URI 字符串）、`getLen()`↔`length()`、`getModificationTime()`↔`modificationTime()`、`isDirectory()`↔`isDirectory()`、`getPath().getName()`↔`name()`（`FileEntry.name()` 取最后一段、剥尾 `/`）。分区名/文件名解析逻辑（parseBase/parseDelta/filter）只吃 `name()` 字符串，不变。 +- **`list()` 非递归、含目录**：`fs.list` = 单层列（LocalFileSystem `Files.newDirectoryStream`、DFSFileSystem `listStatus`），与老 `listStatus` 语义一致（immediate children，dirs+files）。分区列不过滤（需看子目录）、私有 listFiles 过滤目录。 +- **live 路径（非休眠）**：`planAcidScan` 已 live（翻闸后 `type=hms` 事务表读经 `PluginDrivenScanNode` 直达，见"动态状态"）；本步修 live 生产 bug。字节等价 e2e 因 harness 可用性延后（非休眠），一旦 docker HMS+hdfs 环境可用即须回归。 +- **误 close**：`getFileSystem(session)` 返回引擎所有的借用 FS；`planAcidScan`/`getAcidState` 只 `list`/`exists`、**不 close**（契约：连接器只借）。无生命周期变更。 +- **`buildHadoopConf` 删除**：全局仅 ACID 一处调用；`catalogProperties`/`isLocationProperty` 另有 `getScanNodeProperties` 用户，保留。`Configuration`/`Path`/`FileStatus` import 去 FS.get 后无残留用户，删。 +- **铁律核对**：`HiveAcidUtil` 不解析属性（用注入 FS）；不新增 fe-core 分支；TCCL 由 `DFSFileSystem.getHadoopFs` 自钉（连接器去 jar 后任意 TCCL 安全）；无源判别式。 + +## Test Plan + +### Unit（连接器模块无 Mockito） +**迁移 `HiveAcidUtilTest`（9 个既有）到 Doris `LocalFileSystem`（`fe-filesystem-local`，真 FS 真临时树）**——最小改动、最高保真（真目录遍历，杜绝"fake 自带 bug 掩盖真 bug"）： +- `@TempDir` + `Files.createDirectories/write` 建 base/delta/delete-delta 树**不变**（已证正确的 fixture）。 +- `localFs()`：`new org.apache.doris.filesystem.local.LocalFileSystem(Collections.emptyMap())`（Doris），非 Hadoop `FileSystem.getLocal`。`tempDir.toString()` 是裸 `/tmp/…` 绝对路径 → `Location.of` 被 `LocalFileSystem.toPath` 接受（其显式支持裸绝对路径）。 +- import：删 hadoop `Configuration/FileStatus/FileSystem`；加 Doris `FileEntry/FileSystem/LocalFileSystem`；**保留** hive-common `Valid*`（快照仍需）。 +- 断言：`FileStatus`→`FileEntry`；`f.getPath().toString()`→`f.location().uri()`（`file:///tmp/…/base_x/bucket` 仍 `contains("/base_x/…")`，逐位断言不破）。 +- 9 个用例语义全保留（best-base/delta 选择、insert-only 拒 delete-delta、缺 Valid* 列表、原始文件无 base、可见性 txn 跳过、无效 base 回退、not-enough-history）——RED-able 不变（选错 base/delta 即挂）。 + +**字面量列守卫（list() 非 listFiles()）**：`localFs()` 用 `LocalFileSystem` 子类，override `listFiles(Location)`/`listFilesRecursive(Location)` 抛 `AssertionError`（其余委派 super 的真 `list()`/`exists()`）。**如此每个用例都顺带钉住"必须走字面量 `list()`"**——若 `HiveAcidUtil` 退回 `fs.listFiles()`，9 个测试全 RED（AssertionError）。~10 行子类，复用真 FS 保真 + 统一守卫，无需另建 tree fake（LocalFileSystem 默认 `listFiles` 亦字面量，单靠它区分不出 list/listFiles，故须此守卫；镜像 HIVEFS-4 `FakeFileSystem.listFiles` 抛错的同一目的）。 + +### E2E（用户自跑，勿丢——`hms-iceberg-delegation-needs-e2e`） +`external_table_p0/hive` 事务表（full-ACID ORC + insert-only）读，断言与老 fe-core `AcidUtil` 逐位一致（幸存 base/delta、delete-delta 减除）。**此路径已 live（翻闸后事务读经 plugin，见"动态状态"）且当前 hdfs 上就是坏的——本 e2e 验的是 live 生产 bug 的修复**，延后仅因需 docker HMS+hdfs harness，一旦可用即须回归（非"休眠不可测"）。 + +## Open / 已决 +- **测试注入方式**（HIVEFS 清单 Open #3）**已决**：用 `fe-filesystem-local` 真 `LocalFileSystem`（+ 抛-listFiles 子类守卫），非另建 in-memory tree fake。理由：ACID 选择算法安全攸关（选错→静默数据错），真 FS 遍历保真度最高、改动最小、无 fixture-bug 风险；`LocalFileSystem` 本就"for unit testing only"，是老 Hadoop LocalFileSystem 测试的直接 Doris 对等。 +- **`commons-lang` test 依赖去留**（红队 test-fidelity lens minor：迁移可能孤儿化它）**已实测定夺=保留（改注释）**：删掉后 `HiveAcidUtilTest` 报 `NoClassDefFoundError: org.apache.commons.lang.StringUtils`——`snapshot()` 建快照用的 hive-common `Valid*`（`ValidReaderWriteIdList.writeToString()`）引用 commons-lang 2.x，与 Hadoop LocalFileSystem 无关（旧注释归因 Hadoop 是错的）。故保留该 test 依赖、订正注释指向真正的消费者。 +- 写路径 commit/abort 时 FS/identity 捕获时机 → HIVEFS-6。 diff --git a/plan-doc/tasks/task-list-HIVEFS.md b/plan-doc/tasks/task-list-HIVEFS.md index d57535d5d98768..6b3b6936773218 100644 --- a/plan-doc/tasks/task-list-HIVEFS.md +++ b/plan-doc/tasks/task-list-HIVEFS.md @@ -42,9 +42,11 @@ - **此步 + HIVEFS-3 + 重部署 = 失败用例 `test_string_dict_filter` 转绿**（e2e 待用户自跑）。 - 已核：`buildHadoopConf()` fe-core-metadata 侧删除（仅两法用）；`HiveScanPlanProvider` 侧保留（ACID）。 -- [ ] **HIVEFS-5（连接器·ACID）**：`HiveAcidUtil:170/255/298` `fs.exists`/`fs.listStatus` → 注入 `FileSystem` 的 `exists(Location)`/`list`/`listFiles`。 - - **⚠ 测试基建待解**：`HiveAcidUtilTest` 现对**真 Hadoop LocalFileSystem** 跑（见 `fe-connector-hive/pom.xml:128-137` commons-lang test 注释）。转 Doris `FileSystem` 后须换成 fake Doris `FileSystem` 或 `fe-filesystem-local` 的 local 实现——下个 session 定夺注入方式。 - - 校验：`-Dtest=HiveAcidUtil* -DfailIfNoTests=false`。 +- [x] **HIVEFS-5（连接器·ACID）** ✅ DONE（设计 `designs/FIX-HIVEFS-5-design.md`；设计红队 `wf_792d1900-cc7`：5 lens 中 byte-parity/failure-semantics/routing-literal-list/test-fidelity 全判**迁移码 SOUND**，唯一 REAL(major) 是"planAcidScan 非休眠而是 live-broken"——已实测确认并折入注释订正；build+9/9 UT+0 checkstyle）： + - `HiveAcidUtil`：`getAcidState(FileSystem, …)` 换 Doris `FileSystem`；`fs.exists(new Path)`→`fs.exists(Location.of)`；分区列 `fs.listStatus`→`listEntries`(迭代 `fs.list(Location.of)` 收全部)；私有 `listFiles` 助手→`fs.list`+过滤目录（**字面量 `list()` 非 glob 的 `listFiles()`**，镜像 HIVEFS-4）；`FileStatus`→`FileEntry`（`getPath().getName()`→`name()`、`getPath().toString()`→`location().uri()`、`getLen()`→`length()`、`getModificationTime()`→`modificationTime()`）；`AcidState.dataFiles` 类型换 `List`；删 hadoop.fs import（保留 hive-common `Valid*`）。 + - `HiveScanPlanProvider.planAcidScan`：签名 `Configuration`→`FileSystem`；调用点传 `context.getFileSystem(session)`；删 `Path`+`FileSystem.get`；数据文件循环 `FileEntry`；删孤儿 `buildHadoopConf()`+`Configuration/FileStatus/Path` import；**订正 `:137`/`:248-253` 陈旧"Dormant/never reached on a live query"注释→ live 事实**（翻闸后 `type=hms` 事务表读经 `PluginDrivenScanNode` 直达）。 + - **测试注入方式已决=`fe-filesystem-local` 真 `LocalFileSystem`**（+ 抛-`listFiles` 子类 `LiteralListingLocalFileSystem` 钉字面量列），非 in-memory fake（保真、改动最小）。`HiveAcidUtilTest` 9 用例全迁、全绿。pom 加 `fe-filesystem-local` test 依赖；`commons-lang` test 依赖**实测证实仍需**（hive-common `Valid*.writeToString` 引用，非 Hadoop），保留+订正注释。 + - 校验：`-pl :fe-connector-hive -am -Dtest=HiveAcidUtilTest -DfailIfNoTests=false test` = BUILD SUCCESS + 9/9 + 0 checkstyle。 - [ ] **HIVEFS-6（连接器·写路径，最险）**：`HiveConnectorTransaction`： - `resolveObjectStoreFileSystem`（坏 ServiceLoader stub，:784-793）→ `context.getFileSystem(session).forLocation(loc)`；MPU `instanceof ObjFileSystem`（:809/1336）落到 `forLocation` 返回的具体 FS；`objFs.completeMultipartUpload`/`getObjStorage().abortMultipartUpload` 不变。 @@ -66,6 +68,7 @@ ## 设计红队（落地前） - [ ] 按 `clean-room-adversarial-review-pref`：实现前对设计做多 agent 对抗红队（重点：写路径 MPU/rename/delete 语义等价、生命周期误 close、`forLocation` 与 SpiSwitchingFileSystem 缓存交互、session-ignore 的 catalog 级正确性）。 + - [x] HIVEFS-5 已做（`wf_792d1900-cc7`，5 lens + 逐条对抗验证）：迁移码 4 lens SOUND；唯一 REAL(major)=planAcidScan live 非休眠，已折入注释/文档订正。**HIVEFS-6 写路径红队仍待做（落地前）。** ## e2e（用户自跑，勿丢——新能力必配 e2e，memory `hms-iceberg-delegation-needs-e2e`） @@ -79,7 +82,7 @@ 1. ~~`DefaultConnectorContext` 缓存 FS 的 close 挂点~~ ✅ **已解（HIVEFS-3）**：catalog 单一持有 + `onClose`/换连接器两处关，`DefaultConnectorContext implements Closeable`（见上 HIVEFS-3）。 2. `HiveScanPlanProvider.buildHadoopConf()`/`Configuration` 在去 FS.get 后的去留（格式/split/传 BE 是否仍需）——HIVEFS-4 定。 -3. `HiveAcidUtilTest` 从真 LocalFileSystem 迁到 fake/`fe-filesystem-local` 的注入方式——HIVEFS-5 定。 +3. ~~`HiveAcidUtilTest` 从真 LocalFileSystem 迁到 fake/`fe-filesystem-local` 的注入方式~~ ✅ **已解（HIVEFS-5）**：用 `fe-filesystem-local` 真 `LocalFileSystem`（+ 抛-`listFiles` 子类守卫），非 in-memory fake。`commons-lang` test 依赖实测证实仍需（hive-common `Valid*`）。 4. 写路径 commit/abort 时 FS/identity 的捕获时机（begin 时捕获）——HIVEFS-6 定。 5. ~~`forLocation` 加到接口后与现有非切换 impl 的兼容~~ ✅ **已解（HIVEFS-1）**：default `return this`；`SpiSwitchingFileSystem` override 返回 per-scheme 委派；`FileSystemDefaultMethodsTest` 已断言。 From d31ceb0364ef2b32b25aeaa76e6cf9adca817567 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 17:33:18 +0800 Subject: [PATCH 247/334] [feat](catalog) hive-fs: route write-path staging/MPU through engine FileSystem (HIVEFS-6) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit HiveConnectorTransaction built its FileSystem via a local ServiceLoader restricted to OBJECT_STORAGE: broken cross-plugin (FS providers live in sibling filesystem-plugin classloaders, unreachable from the hive plugin) and throwing outright on HDFS-backed tables (objSp==null). After the HMS cutover this path is LIVE, so a type=hms INSERT is currently broken on both HDFS ("No object-store StorageProperties") and object stores (local ServiceLoader finds no provider) — same "plugin has no FS impl" gap the read (HIVEFS-4) and ACID (HIVEFS-5) paths already closed. Route getFileSystem() through the engine-owned context.getFileSystem(session) (per-catalog SpiSwitchingFileSystem, every scheme), mirroring HIVEFS-4/5 and Trino's TrinoFileSystemFactory. The 19 non-MPU I/O sites are unchanged — they already use the Doris FileSystem API and the switching facade routes each op per-path. The 2 MPU sites narrow to the concrete ObjFileSystem via forLocation. session is captured at beginWrite; close() no longer closes the borrowed engine FS (the catalog owns its lifecycle). Removes the dead local ServiceLoader path + orphaned imports. Design red-team (wf_8fd372d6-10d, GO_WITH_FIXES): classloader/instanceof safety (fe-filesystem-spi provided-scope + parent-first prefix => one shared ObjFileSystem class), session/null-safety, close removal, and byte-equivalence of the 19 non-MPU sites all independently reproduced SOUND. Folded 1 major: MPU abort resolves the FS from the upload's native-scheme write path (not a synthetic s3://, which an Azure-typed catalog cannot resolve), and both MPU catches broaden IOException->Exception (SpiSwitchingFileSystem.forLocation throws StoragePropertiesException, a RuntimeException, on a props miss — the narrow catch would escape and break rollback / leak server-side MPUs). Test: injection seam migrated from the deleted resolveObjectStoreFileSystem override to FakeConnectorContext.getFileSystem behind a non-ObjFileSystem routing facade — forces the connector to call forLocation (a regression that casts getFileSystem() directly fails instanceof) and locks the borrow contract (facade close() throws AssertionError). fe-connector-hive: BUILD SUCCESS, HiveConnectorTransactionTest 14/14, 0 checkstyle. Also corrects the stale "Gate-closed / dormant: hive not in SPI_READY_TYPES" class javadoc (live since the cutover). Bare Hadoop FileSystem is now fully gone from the connector write path (only org.apache.hadoop.fs.Path string manipulation remains). hadoop-hdfs-client jar removal + full build + e2e = HIVEFS-7/8. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../hive/HiveConnectorTransaction.java | 113 ++++++++---------- .../hive/HiveConnectorTransactionTest.java | 95 ++++++++++++--- 2 files changed, 130 insertions(+), 78 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java index d496d9facaef74..89acc1f0070fad 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorTransaction.java @@ -37,9 +37,6 @@ import org.apache.doris.filesystem.FileSystem; import org.apache.doris.filesystem.FileSystemUtil; import org.apache.doris.filesystem.Location; -import org.apache.doris.filesystem.properties.StorageKind; -import org.apache.doris.filesystem.properties.StorageProperties; -import org.apache.doris.filesystem.spi.FileSystemProvider; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.thrift.TFileType; import org.apache.doris.thrift.THiveLocationParams; @@ -70,7 +67,6 @@ import java.util.Objects; import java.util.Optional; import java.util.Queue; -import java.util.ServiceLoader; import java.util.Set; import java.util.concurrent.CompletableFuture; import java.util.concurrent.ConcurrentLinkedQueue; @@ -93,14 +89,16 @@ * multipart-upload complete/abort, {@code addPartitions}, and statistics updates. * *

    fe-core couplings broken per design: query profiling dropped (D4); metastore access via the plugin - * {@link HmsClient} SPI instead of {@code HiveMetadataOps} (D10); object-store multipart uploads via a - * plugin-side {@code fe-filesystem-spi} {@link ObjFileSystem} built from {@code context.getStorageProperties()} - * instead of {@code SpiSwitchingFileSystem} (D6); plugin-owned async pool threads each auth-wrapped - * (D5); full-ACID writes hard-rejected at begin (D7); {@code rollback()} deletes staging + aborts MPUs (D9). + * {@link HmsClient} SPI instead of {@code HiveMetadataOps} (D10); staging renames/deletes and object-store + * multipart-upload complete/abort go through the engine-owned {@link FileSystem} borrowed via + * {@code context.getFileSystem(session)} (a per-catalog {@code SpiSwitchingFileSystem}, all schemes), with the + * concrete {@link ObjFileSystem} resolved per object-store location via {@link FileSystem#forLocation} for the + * MPU narrowing (D6); plugin-owned async pool threads each auth-wrapped (D5); full-ACID writes hard-rejected at + * begin (D7); {@code rollback()} deletes staging + aborts MPUs (D9). * - *

    Gate-closed / dormant: hive is not in {@code SPI_READY_TYPES}, so nothing routes plugin-driven hive - * writes through this class until the P7.4/P7.5 cutover. {@link #beginWrite} is wired by INC-4's - * {@code HiveWritePlanProvider.planWrite}. + *

    Live since the HMS cutover: {@code hms} is in {@code SPI_READY_TYPES}, so a {@code type=hms} table INSERT + * routes through {@code HiveWritePlanProvider.planWrite} → {@link #beginWrite} → this class. The engine + * {@link FileSystem} is borrowed (never closed here — the catalog owns its lifecycle). */ public class HiveConnectorTransaction implements ConnectorTransaction { @@ -116,9 +114,10 @@ public class HiveConnectorTransaction implements ConnectorTransaction { private final ExecutorService fileSystemExecutor; private final Executor authWrappingExecutor; - // Built lazily from the catalog's object-store StorageProperties (D6). Held so staging renames/deletes - // and MPU complete/abort share one FileSystem; closed in close(). - private volatile FileSystem fs; + // Captured at beginWrite (D6). Threaded to context.getFileSystem(session) so the engine hands back the + // per-catalog borrowed FileSystem; the session reserves per-user identity (getUser()) — the current engine + // impl resolves the FS at catalog level and ignores it, so a null session (rollback-before-begin) is safe. + private ConnectorSession session; private NameMapping nameMapping; private volatile HmsTableInfo hmsTableInfo; @@ -186,16 +185,9 @@ public long getUpdateCnt() { @Override public void close() { + // Only the self-owned async pool is closed here. The FileSystem is borrowed from the engine + // (context.getFileSystem) — the catalog owns its lifecycle, so the connector must not close it (D6). shutdownExecutorService(fileSystemExecutor); - FileSystem local = fs; - if (local != null) { - try { - local.close(); - } catch (Exception e) { - LOG.warn("Failed to close filesystem for hive write transaction {}: {}", - transactionId, e.getMessage()); - } - } } /** @@ -205,6 +197,7 @@ public void close() { * only pre-commit point that has the table — so the full-ACID reject (D7) can run here. */ public void beginWrite(ConnectorSession session, String db, String tableName, HiveWriteContext ctx) { + this.session = session; this.queryId = ctx.getQueryId(); this.isOverwrite = ctx.isOverwrite(); this.fileType = ctx.getFileType(); @@ -752,45 +745,20 @@ private synchronized void dropPartition( // ─────────────────────────────── filesystem (D6) ─────────────────────────────── - private FileSystem getFileSystem() { - FileSystem local = fs; - if (local == null) { - synchronized (this) { - local = fs; - if (local == null) { - StorageProperties objSp = context.getStorageProperties().stream() - .filter(sp -> sp.kind() == StorageKind.OBJECT_STORAGE) - .findFirst() - .orElse(null); - try { - local = resolveObjectStoreFileSystem(objSp); - } catch (IOException e) { - throw new DorisConnectorException( - "Failed to build filesystem for hive write: " + e.getMessage(), e); - } - fs = local; - } - } - } - return local; - } - /** - * Builds the plugin-side object-store {@link FileSystem} used for staging renames/deletes and MPU - * complete/abort (D6). Isolated behind this single method so production ServiceLoader discovery — a - * cutover-time concern (the object-store providers may be directory-loaded in a separate classloader) — - * is swappable and unit tests can inject a fake FileSystem by overriding it. + * Returns the engine-owned {@link FileSystem} for this write, borrowed via + * {@code context.getFileSystem(session)} (a per-catalog {@code SpiSwitchingFileSystem} that routes every + * scheme — HDFS and object stores alike). The engine lazily builds and caches it per catalog, so this is a + * cheap lookup; the connector borrows and must never close it (D6). MPU sites narrow to the concrete + * {@link ObjFileSystem} via {@link FileSystem#forLocation}. */ - protected FileSystem resolveObjectStoreFileSystem(StorageProperties objSp) throws IOException { - if (objSp == null) { - throw new DorisConnectorException("No object-store StorageProperties available for hive write"); - } - for (FileSystemProvider provider : ServiceLoader.load(FileSystemProvider.class)) { - if (provider.supports(objSp.rawProperties())) { - return provider.create(objSp.rawProperties()); - } + private FileSystem getFileSystem() { + FileSystem engineFs = context.getFileSystem(session); + if (engineFs == null) { + throw new DorisConnectorException("No engine FileSystem available for hive write transaction " + + transactionId + " (catalog has no storage properties)"); } - throw new DorisConnectorException("No FileSystemProvider supports the object-store properties for hive write"); + return engineFs; } // ─────────────────────────────── commit-time object-store MPU (D6) ─────────────────────────────── @@ -805,7 +773,17 @@ private void objCommit(List> asyncFileSystemTaskFutures, if (isMockedPartitionUpdate) { return; } - FileSystem resolved = getFileSystem(); + // Narrow the borrowed switching FileSystem to the concrete ObjFileSystem for this object-store write. + // path is the native-scheme write target (s3://, oss://, abfss://, …); catch Exception because + // forLocation can throw StoragePropertiesException (a RuntimeException) on a props-resolution miss. + FileSystem resolved; + try { + resolved = getFileSystem().forLocation(Location.of(path)); + } catch (Exception e) { + throw new DorisConnectorException( + "Failed to resolve object-store filesystem for MPU commit at '" + path + "': " + + e.getMessage(), e); + } if (!(resolved instanceof ObjFileSystem)) { throw new RuntimeException("Expected ObjFileSystem for MPU commit at path '" + path + "', got: " + resolved.getClass().getSimpleName() + ". This path does not point to an object-storage " @@ -1331,16 +1309,25 @@ private void abortMultiUploads() { if (uncompletedMpuPendingUploads.isEmpty()) { return; } - FileSystem resolved = getFileSystem(); for (UncompletedMpuPendingUpload uncompletedMpuPendingUpload : uncompletedMpuPendingUploads) { + TS3MPUPendingUpload mpu = uncompletedMpuPendingUpload.s3MPUPendingUpload; + String remotePath = "s3://" + mpu.getBucket() + "/" + mpu.getKey(); + // Resolve the concrete ObjFileSystem from the upload's native-scheme write path (NOT the + // BE-unified s3:// remotePath, which an Azure-typed catalog cannot resolve). Lenient: any + // resolution failure (incl. StoragePropertiesException, a RuntimeException) warns and skips. + FileSystem resolved; + try { + resolved = getFileSystem().forLocation(Location.of(uncompletedMpuPendingUpload.path)); + } catch (Exception e) { + LOG.warn("Failed to resolve filesystem for MPU abort {}: {}", remotePath, e.getMessage()); + continue; + } if (!(resolved instanceof ObjFileSystem)) { LOG.warn("FileSystem {} is not object-storage, skipping MPU abort for {}", resolved.getClass().getSimpleName(), uncompletedMpuPendingUpload.path); continue; } ObjFileSystem objFs = (ObjFileSystem) resolved; - TS3MPUPendingUpload mpu = uncompletedMpuPendingUpload.s3MPUPendingUpload; - String remotePath = "s3://" + mpu.getBucket() + "/" + mpu.getKey(); asyncFileSystemTaskFutures.add(CompletableFuture.runAsync(() -> { try { context.executeAuthenticated(() -> { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java index 817d548d80c99b..ddf2e1a4163455 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorTransactionTest.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorSession; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.DorisConnectorException; import org.apache.doris.connector.api.handle.WriteOperation; @@ -33,7 +34,6 @@ import org.apache.doris.filesystem.FileSystem; import org.apache.doris.filesystem.Location; import org.apache.doris.filesystem.UploadPartResult; -import org.apache.doris.filesystem.properties.StorageProperties; import org.apache.doris.filesystem.spi.ObjFileSystem; import org.apache.doris.filesystem.spi.ObjStorage; import org.apache.doris.filesystem.spi.RemoteObject; @@ -52,6 +52,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.io.IOException; import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; @@ -82,11 +83,12 @@ * * *

    The commit-time object-store work is driven through a fake {@code ObjFileSystem}/{@code ObjStorage} - * injected via the {@code resolveObjectStoreFileSystem} seam: multipart-upload complete on commit - * and abort on rollback (D6/D9), the idempotency of a repeated rollback, and the single - * batched {@code addPartitions} call (GAP-7 — the 20-at-a-time batching lives in the client, so the - * committer adds the whole list once). The staging-directory rename walk and the full multi-partition - * commit remain for the P7.4/P7.5 cutover integration gate (the live write path).

    + * injected via the {@code FakeConnectorContext.getFileSystem(session)} seam behind a non-{@code ObjFileSystem} + * routing facade (mirroring the engine's {@code SpiSwitchingFileSystem}, so the connector must narrow via + * {@code forLocation}): multipart-upload complete on commit and abort on rollback (D6/D9), the + * idempotency of a repeated rollback, and the single batched {@code addPartitions} call (GAP-7 — the + * 20-at-a-time batching lives in the client, so the committer adds the whole list once). The staging-directory + * rename walk and the full multi-partition commit are covered by the e2e write suites.

    */ public class HiveConnectorTransactionTest { @@ -160,16 +162,19 @@ private static THivePartitionUpdate puWithMpu(String name, TUpdateMode mode, Str return u; } - // Builds a transaction whose object-store FileSystem is the injected fake, bypassing the production - // ServiceLoader discovery (a cutover-time concern) — the seam the design carved out for exactly this. - private HiveConnectorTransaction newTxnWithFs(HmsClient client, FileSystem fakeFs) { + // Builds a transaction whose engine FileSystem is the injected fake, borrowed via the context — mirroring + // production, where context.getFileSystem(session) hands back the per-catalog SpiSwitchingFileSystem. The + // concrete fake is wrapped in a non-ObjFileSystem routing facade so the connector MUST call forLocation(...) + // to narrow to the ObjFileSystem: a regression that casts getFileSystem() directly then fails instanceof. + private HiveConnectorTransaction newTxnWithFs(HmsClient client, FileSystem concreteFs) { + FileSystem borrowed = new RoutingFacadeFileSystem(concreteFs); return new HiveConnectorTransaction(42L, client, - new FakeConnectorContext("test_catalog", CATALOG_ID, Collections.emptyMap())) { - @Override - protected FileSystem resolveObjectStoreFileSystem(StorageProperties objSp) { - return fakeFs; - } - }; + new FakeConnectorContext("test_catalog", CATALOG_ID, Collections.emptyMap()) { + @Override + public FileSystem getFileSystem(ConnectorSession session) { + return borrowed; + } + }); } // ─────────────────────────────── SPI identity ─────────────────────────────── @@ -655,4 +660,64 @@ public DorisOutputFile newOutputFile(Location location) { throw new UnsupportedOperationException("newOutputFile"); } } + + /** + * A non-{@link ObjFileSystem} routing facade mirroring the engine's {@code SpiSwitchingFileSystem}: it is + * NOT itself an {@code ObjFileSystem}, so the connector must call {@link FileSystem#forLocation} to narrow + * to the concrete {@code ObjFileSystem} before an MPU (a regression that casts {@code getFileSystem()} + * directly then fails {@code instanceof} here). Base operations delegate to the wrapped concrete FS; + * {@code close()} throws to lock in the borrow contract — the connector must never close the engine FS. + */ + private static final class RoutingFacadeFileSystem implements FileSystem { + private final FileSystem delegate; + + RoutingFacadeFileSystem(FileSystem delegate) { + this.delegate = delegate; + } + + @Override + public FileSystem forLocation(Location location) { + return delegate; + } + + @Override + public boolean exists(Location location) throws IOException { + return delegate.exists(location); + } + + @Override + public void mkdirs(Location location) throws IOException { + delegate.mkdirs(location); + } + + @Override + public void delete(Location location, boolean recursive) throws IOException { + delegate.delete(location, recursive); + } + + @Override + public void rename(Location src, Location dst) throws IOException { + delegate.rename(src, dst); + } + + @Override + public FileIterator list(Location location) throws IOException { + return delegate.list(location); + } + + @Override + public DorisInputFile newInputFile(Location location) throws IOException { + return delegate.newInputFile(location); + } + + @Override + public DorisOutputFile newOutputFile(Location location) throws IOException { + return delegate.newOutputFile(location); + } + + @Override + public void close() { + throw new AssertionError("connector must not close the borrowed engine FileSystem"); + } + } } From 55583f4c8bf5c2e5dcc22a43c33dd026880c2ec7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 17:35:53 +0800 Subject: [PATCH 248/334] [doc](catalog) hive-fs: land HIVEFS-6 design (red-team folded), mark done, repoint to jar-removal HIVEFS-7 Design FIX-HIVEFS-6-design.md for the write-path FileSystem injection, with the wf_8fd372d6-10d red-team (GO_WITH_FIXES) outcome folded in: 1 major (MPU abort native-scheme resolution + broadened catch) + 4 minor (test seam/facade/ close-guard/javadoc). task-list-HIVEFS: HIVEFS-6 + its design red-team marked DONE, Open #4 (FS/identity capture timing) resolved. HANDOFF: repoint next session to HIVEFS-7 (delete hadoop-hdfs-client + grep zero bare Hadoop FS) and HIVEFS-8 (full build + e2e); connector read+ACID+write bare-Hadoop I/O now all routed through the engine FileSystem. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 16 +- plan-doc/tasks/designs/FIX-HIVEFS-6-design.md | 157 ++++++++++++++++++ plan-doc/tasks/task-list-HIVEFS.md | 22 +-- 3 files changed, 178 insertions(+), 17 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-HIVEFS-6-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 143c1071b6081b..ad9cadae85a4f0 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,9 +5,9 @@ --- -# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 写路径 HIVEFS-6，最险） +# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 去 jar HIVEFS-7 + 全量 build/e2e HIVEFS-8，收尾） -> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。** +> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。连接器读+ACID+写三条裸 Hadoop 路径已全部转引擎注入 FileSystem（HIVEFS-4/5/6 DONE），只剩去依赖 jar + 全量验证。** **根因**：连接器扫描/写全程用**裸 `org.apache.hadoop.fs.FileSystem`**（`HiveFileListingCache:162`/`HiveScanPlanProvider:272`/`HiveAcidUtil`/`HiveConnectorTransaction`），但 hive 插件类加载器隔离、lib/ 无 `DistributedFileSystem`（仅 hadoop-common，注册 Local/View/Har/Http、无 hdfs）。老 fe-core 经 `org.apache.doris.filesystem.FileSystem`+`DirectoryLister` 列文件（`HiveExternalMetaCache:392-396`），**新代码裸 Hadoop 是迁移走样**。非环境问题（hostname 可解析、metastore 通、已取到 location）。 @@ -23,20 +23,22 @@ **✅ HIVEFS-5 DONE（code `7a2d8714951`）= 连接器·ACID 目录下降已转引擎 FileSystem**：`HiveAcidUtil.getAcidState(FileSystem,…)` 换 Doris `FileSystem`——`fs.exists(new Path)`→`exists(Location.of)`；分区列 `fs.listStatus`→`listEntries`(迭代 `fs.list` 收全部含目录)；私有 `listFiles` 助手→`fs.list`+过滤目录（**字面量 `list()` 非 glob 的 `listFiles()`**，镜像 HIVEFS-4）；`FileStatus`→`FileEntry`(`name()`/`location().uri()`/`length()`/`modificationTime()`)；`AcidState.dataFiles`→`List`；删 hadoop.fs import（留 hive-common `Valid*`）。`HiveScanPlanProvider.planAcidScan`：签名 `Configuration`→`FileSystem`、传 `context.getFileSystem(session)`、删 `Path`+`FileSystem.get`、删孤儿 `buildHadoopConf`+3 import。**红队 `wf_792d1900-cc7`（5 lens+逐条对抗）：迁移码 4 lens SOUND，唯一 REAL(major)=`planAcidScan` 非休眠而是 live-broken**（翻闸后 `hms`∈`SPI_READY_TYPES`→`PluginDrivenScanNode`→`planScan`→`planAcidScan` 无门，hdfs 事务读今天即炸；已实测确认+订正 `:137`/`:248-253` 陈旧"Dormant/never reached"注释）。测试注入=`fe-filesystem-local` 真 `LocalFileSystem`+抛-`listFiles` 子类守卫（非 fake）；`commons-lang` test 依赖实测证实仍需（hive-common `Valid*`）。**fe-connector-hive 9/9、0 checkstyle、BUILD SUCCESS**。设计 `designs/FIX-HIVEFS-5-design.md`。 -**⭐ 下一步 = HIVEFS-6（连接器·写路径，最险，落地前须多 agent 对抗红队）**：`HiveConnectorTransaction` `resolveObjectStoreFileSystem`（坏 ServiceLoader stub :784-793）→ `context.getFileSystem(session).forLocation(loc)`；MPU `instanceof ObjFileSystem`(:809/1336) 落到 `forLocation` 返回的具体 FS；事务内 delete/rename/exists 裸 Hadoop 一并换；`close()` 不关借用引擎 FS；FS/identity 于 `beginWrite`(:207) 捕获。之后：**7 删 `hadoop-hdfs-client`+grep 零裸 Hadoop → 8 全量 build+e2e**。 +**✅ HIVEFS-6 DONE（code `d31ceb0364e`）= 连接器·写路径已转引擎 FileSystem（最险，已过红队）**：`HiveConnectorTransaction` `getFileSystem()`（:755）本已全程用 Doris `FileSystem` API（19 非-MPU I/O + 2 MPU），**缺陷只在 FS 来源**——`resolveObjectStoreFileSystem` 本地 `ServiceLoader`（跨插件拿不到 provider）+ `.filter(OBJECT_STORAGE)`（HDFS 后端直接抛）。翻闸后 **live**（class-javadoc "dormant" 陈旧），hive INSERT 今天在 HDFS/对象存储上都必炸。改：`getFileSystem()`→`context.getFileSystem(session)`（引擎 per-catalog `SpiSwitchingFileSystem`，全 scheme）；删 `resolveObjectStoreFileSystem`+`fs` 字段+OBJECT_STORAGE 过滤+孤儿 import；19 非-MPU 点不动（facade 逐操作 `forLocation` 委派）；MPU 两处 `forLocation` narrow 具体 `ObjFileSystem`（`objCommit`=native 写目标 strict、`abortMultiUploads`=逐 upload native `u.path` lenient）；`close()` 删 `fs.close()`（借用不关）；`session` 于 `beginWrite` 捕获（null 安全）。**红队 `wf_8fd372d6-10d`（5 lens+裁决，GO_WITH_FIXES）**：classloader/instanceof（provided-scope+parent-first→共享 `ObjFileSystem`）、session/null、close 去除、19 非-MPU 字节等价 均 SOUND；折入 1 major（MPU abort 用 native-scheme `u.path` 非合成 `s3://`，Azure 才可解析；两处 catch `IOException`→`Exception` 兜 `StoragePropertiesException` RuntimeException）+ 4 minor（test import / non-`ObjFileSystem` facade 强制 `forLocation` + `close()` 抛 AssertionError 钉借用契约 / 类 javadoc）。**fe-connector-hive BUILD SUCCESS、`HiveConnectorTransactionTest` 14/14、0 checkstyle**。设计 `designs/FIX-HIVEFS-6-design.md`。 + +**⭐ 下一步 = HIVEFS-7（去 jar，达成终点）→ HIVEFS-8（全量 build+e2e）**：① 删 `fe-connector-hive/pom.xml` 的 `hadoop-hdfs-client`（**保留** `hadoop-common`——`Configuration`/`HiveConf`/HMS client 仍需）；全局 grep 校验连接器 main 源 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get`/`listStatus` **零残留**（`org.apache.hadoop.fs.Path` 若仅路径拼接可留，确认不再 FS I/O）；打包校验 `output/fe/plugins/connector/hive/lib/` 无 `hadoop-hdfs-client`、插件 zip 完整。② 全量 build（fe-filesystem-api+spi + fe-core + fe-connector-hive）+ 全 UT + 0 checkstyle + import 门净。③ e2e（用户自跑）：重部署后 `test_string_dict_filter`（读 hdfs）+ hive INSERT 写套件（rename/delete/MPU complete）+ INSERT 回滚（MPU abort）+ 若有对象存储环境抽查 s3/oss。 **已探明的去风险事实（勿重查）**： - **引擎侧已落地（HIVEFS-3）**——`DefaultConnectorContext.getFileSystem` 用已持的 `storagePropertiesSupplier` + `SpiSwitchingFileSystem`（范式镜像 `cleanupEmptyManagedLocation:348`）；close 由 catalog 在 `onClose`/换连接器处关（context 单一持有、sibling 共享）。 - **TCCL 无需连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs **自钉**到自身插件 loader 再 `FileSystem.get`、finally 还原（注释即描述 “No FileSystem for scheme hdfs” 场景）。连接器去 jar 后任意 TCCL 调用皆安全。 -- 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction:809/1336`）→ `FileSystem.forLocation`（HIVEFS-1 已加：非切换返回 this、切换返回具体 FS）就位待 HIVEFS-6 用。 -- 连接器裸 Hadoop 面（读扫描 HIVEFS-4 + ACID HIVEFS-5 已清）：仅剩**写路径** `HiveConnectorTransaction`（MPU/rename/delete/exists）待 HIVEFS-6；`FileStatus` 字段全映射 `FileEntry`（读侧已验：`name()`/`location().uri()`/`length()`/`modificationTime()`/`isDirectory()`）。 +- 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction` objCommit/abort）→ `FileSystem.forLocation`（HIVEFS-1 加、HIVEFS-6 已用：非切换返回 this、切换返回具体 FS）。**跨 连接器↔fs-plugin `instanceof ObjFileSystem` 安全**（`fe-filesystem-spi` 双 provided-scope + `FileSystemPluginManager:72` parent-first `"org.apache.doris.filesystem."` → 共享同一 `ObjFileSystem` Class；红队 HIVEFS-6 逐点取证）。 +- 连接器裸 Hadoop **FileSystem I/O 面已全清**（读扫描 HIVEFS-4 + ACID HIVEFS-5 + 写路径 HIVEFS-6）：三条路径均经 `context.getFileSystem(session)` 借引擎 `SpiSwitchingFileSystem`；仅剩 **pom 的 `hadoop-hdfs-client` jar 依赖**待 HIVEFS-7 删。`FileStatus` 字段全映射 `FileEntry`（`name()`/`location().uri()`/`length()`/`modificationTime()`/`isDirectory()`）。 - fe-filesystem 全 9 插件（hdfs/s3/oss/cos/obs/azure/http/local/broker）已部署 → hive 表任意后端天然支持（免 bundle hadoop-aws/huaweicloud，架构红利）。 -**⚠ 待下个 session 核定**（task-list Open §）：写路径 commit/abort 时 FS/identity 捕获时机（HIVEFS-6，`beginWrite` 捕获）。（`buildHadoopConf` 去留=HIVEFS-4/5 已删；`HiveAcidUtilTest` 注入方式=HIVEFS-5 已定 `fe-filesystem-local`。） +**⚠ 待下个 session 核定**（task-list Open §）：无阻塞项待决——HIVEFS-7 是纯 pom 删依赖 + grep 校验，HIVEFS-8 是全量 build + e2e。（写路径 FS/identity 捕获=HIVEFS-6 已定 `beginWrite`；`buildHadoopConf` 去留=HIVEFS-4/5 已删；`HiveAcidUtilTest` 注入=HIVEFS-5 定 `fe-filesystem-local`。）**HIVEFS-7 删 `hadoop-hdfs-client` 后须打包实测**（避免误删 `hadoop-common` 传递依赖破 HMS client/`HiveConf`）。 **⚠ 重要事实（红队 HIVEFS-5 确认）**：翻闸后 hive **事务表读已 live 走 plugin**（`planAcidScan` 非休眠）；HIVEFS-5 前它在 hdfs 上就是坏的（`No FileSystem for scheme hdfs`），HIVEFS-5 修的是 live 生产 bug（e2e 因需 docker HMS+hdfs harness 延后，非"不可测"）。 -**⚠ 状态**：HIVEFS-0/1/2/3/4/5 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9`/`7a2d8714951` + doc commit）。落地纪律：写路径（HIVEFS-6）落地前设计红队 → 每子步独立 commit + 靶向 UT（见 task-list）。HIVEFS-4/5 已按此各走完一轮（设计→红队→折入→实现→UT→commit）。 +**⚠ 状态**：HIVEFS-0/1/2/3/4/5/6 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9`/`7a2d8714951`/`d31ceb0364e` + doc commit）。落地纪律：每子步 = 设计→红队→折入→实现→UT→独立 commit（HIVEFS-4/5/6 均已按此走完一轮）。剩 HIVEFS-7（去 jar）/HIVEFS-8（全量 build+e2e）为收尾，无需红队（纯依赖删除 + 验证）。 --- diff --git a/plan-doc/tasks/designs/FIX-HIVEFS-6-design.md b/plan-doc/tasks/designs/FIX-HIVEFS-6-design.md new file mode 100644 index 00000000000000..0f79b71770be1e --- /dev/null +++ b/plan-doc/tasks/designs/FIX-HIVEFS-6-design.md @@ -0,0 +1,157 @@ +# FIX-HIVEFS-6 — 连接器写路径改经引擎注入 `FileSystem`（去 `HiveConnectorTransaction` 的本地 ServiceLoader 造 FS） + +> 承接 FIX-HIVEFS（设计 `FIX-HIVEFS-design.md`、清单 `task-list-HIVEFS.md`）。HIVEFS-3（引擎实现）+ HIVEFS-4（读扫描列文件）+ HIVEFS-5（ACID 目录下降）已入库；本步是**最后一条、也是最险的连接器路径**——非-ACID 写事务（staging→target rename / delete / MPU complete·abort）仍用**本地 `ServiceLoader` 造 FileSystem**，是同一"hive 插件类加载器隔离→跨插件拿不到 FS 实现"缺口的写侧分支。 +> 范围 = 仅 hive 连接器写路径（`HiveConnectorTransaction`）。去 jar（HIVEFS-7）、全量 e2e（HIVEFS-8）不属本步。 + +## Problem + +`HiveConnectorTransaction`（P7.3 移植自老 fe-core `HMSTransaction`，commit `eeae58bbd23`，此后未再动）的所有写侧文件 I/O **已经**走 Doris `org.apache.doris.filesystem.FileSystem` API（`exists`/`mkdirs`/`delete`/`listFiles*`/`listDirectories`/`renameDirectory`/`FileSystemUtil.asyncRename*` + MPU `ObjFileSystem.completeMultipartUpload`/`getObjStorage().abortMultipartUpload`），**统一经私有 `getFileSystem()`（:755）取 FS**。缺陷**只在 `getFileSystem()` 的 FS 来源**： + +```java +// :755 getFileSystem() 懒建 + 字段缓存 +StorageProperties objSp = context.getStorageProperties().stream() + .filter(sp -> sp.kind() == StorageKind.OBJECT_STORAGE) // ← 只挑对象存储 + .findFirst().orElse(null); +local = resolveObjectStoreFileSystem(objSp); // :766 + +// :784 resolveObjectStoreFileSystem +for (FileSystemProvider provider : ServiceLoader.load(FileSystemProvider.class)) { // ← 本地 ServiceLoader + if (provider.supports(objSp.rawProperties())) return provider.create(objSp.rawProperties()); +} +throw new DorisConnectorException("No FileSystemProvider supports ..."); +``` + +两处致命： +1. **本地 `ServiceLoader.load(FileSystemProvider.class)` 跨插件拿不到 provider**——FS provider 在**兄弟** filesystem 插件 loader（`output/fe/plugins/filesystem/*`），共享类路径（`output/fe/lib`）无任何 provider，`DirectoryPluginRuntimeManager` 且刻意屏蔽父级 service 描述符。该方法 javadoc 自认是 "cutover-time concern / production ServiceLoader discovery … swappable"（即从来是占位 stub）。 +2. **`.filter(OBJECT_STORAGE)`**——只挑对象存储 `StorageProperties`。**HDFS 后端 hive 表** `objSp==null` → `resolveObjectStoreFileSystem(null)` 抛 `"No object-store StorageProperties available for hive write"`。 + +**动态状态（与 HIVEFS-5 planAcidScan 同型的陈旧注释）**：class-javadoc（:101-103）称 `"Gate-closed / dormant: hive is not in SPI_READY_TYPES, so nothing routes plugin-driven hive writes through this class until the P7.4/P7.5 cutover"` —— **已假**。Phase 2 原子翻闸（commit `fb1624be757`，`hms`∈`SPI_READY_TYPES`）后，`type=hms` 表 INSERT 经 `HiveWritePlanProvider.planWrite` → `beginWrite` → 本类 live 驱动。故这是**修 live 生产 bug**（HDFS hive INSERT 今天必炸 stub、对象存储 hive INSERT 今天必炸跨插件 ServiceLoader），非休眠整洁。本步须一并订正 class-javadoc（否则 ship 自相矛盾的文件）。 + +老 fe-core `HMSTransaction` 用 `SpiSwitchingFileSystem`（引擎侧、全 scheme）做同样的 rename/delete/MPU；P7.3 移植时换成本地 ServiceLoader 是迁移走样，与 HIVEFS-4/5 读侧同源。 + +## Root Cause + +引擎已经 HIVEFS-3 提供 `ConnectorContext.getFileSystem(session)`（per-catalog `SpiSwitchingFileSystem`，引擎所有、连接器借用不 close、全 scheme 路由），HIVEFS-4/5 已把读扫描 + ACID 接上；写路径是最后一条未接的分支。本步把 `getFileSystem()` 的 FS **来源**从"本地 ServiceLoader 造对象存储 FS"换成"借引擎 `context.getFileSystem(session)`"——**下游 20 个调用点全不动**（它们已用 Doris `FileSystem` API），唯 MPU 两处需从 switching-facade 解出**具体** `ObjFileSystem`。 + +## Design + +### 1. 会话捕获（`beginWrite` 时） +`context.getFileSystem(session)` 需 `ConnectorSession`。`beginWrite(ConnectorSession session, …)`（:207）是唯一预提交入口、已捕获 user。新增字段 `private ConnectorSession session;`，`beginWrite` 首行 `this.session = session;`。`ConnectorSession` 已 import（:25）。 +- **session 可为 null 的两条路径**（均安全）：① 单测 `beginWrite(null, …)`；② rollback-before-commit（`rollback()`:292，`hmsCommitter==null`，仅 abort 悬挂 MPU）——生产中 `beginWrite` 恒先于 `addCommitData/commit/rollback`（`planWrite` 在 BE 执行前调），故此路径 session 实为已捕获；纯 rollback（无 beginWrite）仅测试构造。引擎 `DefaultConnectorContext.getFileSystem`（HIVEFS-3）**忽略 session**（catalog 级），null 安全；契约标注 identity 经 `session.getUser()` **预留** per-user。 + +### 2. `getFileSystem()` 换来源（:755-776） +删懒建 + `fs` 字段 + `resolveObjectStoreFileSystem` + OBJECT_STORAGE 过滤，改为借引擎 FS： +```java +private FileSystem getFileSystem() { + FileSystem engineFs = context.getFileSystem(session); + if (engineFs == null) { // loud（对齐 HIVEFS-4/5 null-FS→loud） + throw new DorisConnectorException("No engine FileSystem available for hive write transaction " + + transactionId + " (catalog has no storage properties)"); + } + return engineFs; +} +``` +- 引擎已 per-catalog 懒建 + 缓存 `SpiSwitchingFileSystem`（HIVEFS-3），故连接器侧**无需**再本地缓存 `fs` 字段（删之，连带删 :119-121 注释）。 +- 空 storage → 引擎返 null → 本处 loud（HDFS/对象存储 hive 表都有 storage，仅无存储的畸形 catalog 命中，即时可诊断优于 NPE）。 + +### 3. MPU 两处解具体 `ObjFileSystem`（`forLocation`） +`getFileSystem()` 现返 `SpiSwitchingFileSystem`（facade，**非** `ObjFileSystem`）。非-MPU 的 19 个调用点用**基类** `FileSystem` 方法（`exists`/`delete`/`renameDirectory`/`listFiles*`…），switching FS 内部**逐操作** `forLocation` 路由到具体 FS（`SpiSwitchingFileSystem:116-160`）→ **不动**。唯 MPU 两处要 narrow 到 `ObjFileSystem`（`completeMultipartUpload`/`getObjStorage()` 不在基类接口上），须先 `forLocation` 解出具体 FS： + +- **`objCommit`（:803-833，complete，strict）**：把 :808 `FileSystem resolved = getFileSystem();` 换为 + ```java + FileSystem resolved; + try { + resolved = getFileSystem().forLocation(Location.of(path)); // path=写目标(native scheme) + } catch (Exception e) { // 宽 catch：forLocation 可抛 StoragePropertiesException(RuntimeException) + throw new DorisConnectorException("Failed to resolve object-store filesystem for MPU commit at " + + path + ": " + e.getMessage(), e); + } + if (!(resolved instanceof ObjFileSystem)) { throw …(:809-813 不变); } + ObjFileSystem objFs = (ObjFileSystem) resolved; // 循环体不变 + ``` + `path` 是该分区/表的写目标（**native HMS scheme**：S3 兼容=`s3://`、Azure=`abfss://`；MPU 只在 `hasPendingUploads` 且对象存储写时触发）；`type`-match 解析对所有后端（含 Azure）成立，与循环内 `remotePath="s3://bucket/key"`（传 complete API 的 BE 统一口径）同 catalog 凭证。fail-fast 语义（非对象存储即抛）保留；catch `Exception`（非 `IOException`）兜住 `forLocation` 的 `StoragePropertiesException`（RuntimeException）→ loud-wrap。 +- **`abortMultiUploads`（:1330-1356，abort，lenient）**：把 :1334 循环外 `getFileSystem()` 移入**循环内逐 upload** 解析 FS，**用 upload 的 native-scheme 路径 `u.path`**（=`pu.getLocation().getWritePath()`，:465——非合成 `s3://`；与 objCommit 用 `path` 对称）`forLocation(Location.of(u.path))`，`forLocation`/instanceof 失败 → `LOG.warn`+`continue`（对齐现有 :1336-1339 warn+skip 的 abort 宽松语义，不抛）： + ```java + for (UncompletedMpuPendingUpload u : uncompletedMpuPendingUploads) { + TS3MPUPendingUpload mpu = u.s3MPUPendingUpload; + String remotePath = "s3://" + mpu.getBucket() + "/" + mpu.getKey(); // BE-unified s3:// 路径（传 abort API） + FileSystem resolved; + try { resolved = getFileSystem().forLocation(Location.of(u.path)); } // ← native-scheme 解析（红队 major） + catch (Exception e) { LOG.warn("… skip MPU abort for {}: {}", remotePath, e.getMessage()); continue; } + if (!(resolved instanceof ObjFileSystem)) { LOG.warn(…); continue; } + ObjFileSystem objFs = (ObjFileSystem) resolved; + … 异步 abort（不变，仍 `objFs.getObjStorage().abortMultipartUpload(remotePath, …)` 在 context.executeAuthenticated 内） + } + uncompletedMpuPendingUploads.clear(); + ``` + - **红队 major 折入**：① 解析源用 `u.path`（native scheme：Azure=`abfss://`、S3 兼容=`s3://`）而非合成 `"s3://"+bucket+"/"+key`——否则 Azure-typed catalog（`getStorageName()="AZURE"≠"s3"`）的 `s3://` 无法解析。② catch **`Exception`**（非 `IOException`）——`SpiSwitchingFileSystem.forLocation` props 解析失败抛 `StoragePropertiesException`（**RuntimeException**，非 IOException），窄 catch 无法兑现 abort 的 "resolve 失败→warn+skip" 宽松契约，会逃逸破坏 rollback、泄漏 server 端 MPU（vs HEAD 直建 Azure FS 成功 abort 的回归）。objCommit 同步 catch `Exception`→loud-wrap（strict）。`remotePath`（`s3://bucket/key`，BE 统一口径）仍传 `abortMultipartUpload` API 不变。 + +### 4. `close()` 不关借用 FS（:188-199） +现 `close()` 关 `fileSystemExecutor`（自有，保留）**并 `fs.close()`**（:190-198，借用引擎 FS，**须删**）。删除 `fs` 字段后 `close()` 仅剩 executor 关闭： +```java +public void close() { shutdownExecutorService(fileSystemExecutor); } +``` +引擎 FS 生命周期由 catalog 在 `onClose`/换连接器处关（HIVEFS-3 定），连接器只借。 + +### 5. class-javadoc 订正（:87-104） +- :96-98 D6 描述 `"object-store multipart uploads via a plugin-side fe-filesystem-spi ObjFileSystem built from context.getStorageProperties() instead of SpiSwitchingFileSystem"` → 改为如实：经 `context.getFileSystem(session)`（引擎 `SpiSwitchingFileSystem`，全 scheme）借用，MPU 经 `forLocation` 解具体 `ObjFileSystem`。 +- :101-103 `"Gate-closed / dormant …"` → 翻闸后 live（`type=hms` INSERT 经 `HiveWritePlanProvider.planWrite`→`beginWrite` 直达本类）。 + +### 6. import 增删 +- **删**：`java.util.ServiceLoader`（:73）、`org.apache.doris.filesystem.properties.StorageKind`（:40）、`org.apache.doris.filesystem.properties.StorageProperties`（:41）、`org.apache.doris.filesystem.spi.FileSystemProvider`（:42）——已核实仅 `getFileSystem`/`resolveObjectStoreFileSystem` 用（class-doc 引用随 §5 订正一并去 `{@link}`）。 +- **保留**：`org.apache.doris.filesystem.spi.ObjFileSystem`（:43，MPU 仍用）、`FileSystem`/`Location`/`FileEntry`/`FileSystemUtil`、`org.apache.hadoop.fs.Path`（:53，`new Path(...)` 纯路径拼接如 :878/1166/1232/1500，非 FS I/O，保留）。 +- **无新增**（`ConnectorSession`/`DorisConnectorException` 已 import）。 + +## 关键正确性核验（写路径独有，读侧 HIVEFS-4/5 无此需求） + +**`instanceof ObjFileSystem` 跨 连接器↔filesystem 插件 classloader 边界成立**（本步最大风险，已逐点取证）： +- `ObjFileSystem` 在 `fe-filesystem/fe-filesystem-spi`（共享 SPI 模块）；`S3CompatibleFileSystem extends ObjFileSystem`，s3 插件具体 FS 是其子类。 +- `fe-connector-hive` 依赖 `fe-filesystem-spi` = **`provided`**（pom :97）→ **不打入插件 lib/**，运行时从 fe-core 宿主共享类路径解析。 +- `fe-filesystem-s3` 依赖 `fe-filesystem-spi` 亦 `provided`（pom 注释 :42-43：`"SPI/API … live on the fe-core host classpath and are loaded parent-first; provided so they are not bundled into the plugin lib/"`）。 +- `FileSystemPluginManager:72` parent-first 前缀含 `"org.apache.doris.filesystem."` → filesystem 插件对 `org.apache.doris.filesystem.spi.ObjFileSystem` **parent-first** 解析。 +- ⇒ 连接器与 s3 插件解析**同一** `ObjFileSystem` Class 对象 → `resolved instanceof ObjFileSystem` 真、`(ObjFileSystem) resolved` 不 ClassCast。生产 `SpiSwitchingFileSystem.forLocation("s3://…")` → 具体 `S3CompatibleFileSystem` 子类（is-a `ObjFileSystem`）。 + +**`forLocation` 默认语义**：`ObjFileSystem` **不** override `forLocation` → 用 `FileSystem` 接口默认 `return this`（HIVEFS-1 加）。故：① 生产具体 S3 FS `forLocation(x)`=自身（already 具体）；② 单测 fake `RecordingObjFileSystem`（extends `ObjFileSystem`）`forLocation(x)`=自身 → instanceof 过。switching FS 的 `forLocation` override 才做 per-scheme 路由（生产入口）。 + +**TCCL 无新增钉点**：MPU complete/abort 仍在 `context.executeAuthenticated(...)`（:822/1346）内 → `TcclPinningConnectorContext` 已钉插件 loader（memory `catalog-spi-plugin-tccl-classloader-gotcha` 既有 locus）；`forLocation` 建 S3 FS 走 AWS SDK（非 Hadoop `getHadoopFs` 的 "No FileSystem for scheme" 类），且 S3 provider 由引擎 `FileSystemFactory` 造，无连接器侧 TCCL 责任。 + +## Implementation Plan +1. `HiveConnectorTransaction.java`：字段（+`session`、−`fs`）、`beginWrite` 捕获 session、`getFileSystem()` 换来源、删 `resolveObjectStoreFileSystem`、MPU 两处 `forLocation`、`close()` 去 `fs.close()`、class-javadoc 订正、import 增删（§1-6）。 +2. `HiveConnectorTransactionTest.java`：注入缝从 `resolveObjectStoreFileSystem`-override 迁到 `FakeConnectorContext.getFileSystem(session)`-override（返回 fake `ObjFileSystem`）。 +3. build（`-pl :fe-connector-hive -am`）+ 靶向 UT（`-Dtest=HiveConnectorTransactionTest`）+ 0 checkstyle + import 门净。 + +## Risk Analysis +- **`instanceof ObjFileSystem` 跨边界**（最险）→ 已逐点取证成立（见上「关键正确性核验」），provided-scope + parent-first 双证。**e2e 兜底**：真集群对象存储 hive INSERT（MPU complete）+ rollback（MPU abort）。 +- **误 close 借用 FS**：§4 删 `fs.close()`；引擎所有、catalog 关。 +- **session=null**：引擎/fake 均忽略；契约标注 per-user 预留。 +- **`forLocation` × switching 缓存**：`SpiSwitchingFileSystem.forLocation`→`forPath`→按 `StorageProperties` 值相等缓存（`:66/94`）；同 catalog 对象存储凭证唯一 → MPU 各 upload 命中同一具体 FS，无重复建连。 +- **rename/delete 语义不变**：19 个非-MPU 点未改一字，仅 FS 实例从"本地 ServiceLoader 造的具体对象存储 FS"变"引擎 switching facade"；facade 逐操作 `forLocation` 委派到**同一类**具体 FS（对象存储场景），字节等价；且新增 HDFS/其它 scheme 支持（架构红利，老 stub 直接抛）。 +- **fail 语义**：complete strict（resolve 失败 loud）；abort lenient（warn+skip）——均对齐现有 objCommit/abortMultiUploads 既定语义。 +- **铁律核对**：`HiveConnectorTransaction` 不解析属性（借注入 FS）；不新增 fe-core 分支；无源判别式；TCCL 无新增 locus。 + +## Test Plan + +### Unit（连接器模块无 Mockito；真 recording fake） +现有 `HiveConnectorTransactionTest`（12 用例）注入 fake FS 靠 override `resolveObjectStoreFileSystem`（:165-173 `newTxnWithFs`）。本步删该方法 → 注入缝迁到 `FakeConnectorContext.getFileSystem(session)`，**返回一个 non-`ObjFileSystem` 的路由 facade**（镜像生产 `SpiSwitchingFileSystem`）： +- 新增测试内 `RoutingFacadeFileSystem implements FileSystem`（**不 extends `ObjFileSystem`**）：`forLocation(loc)` 返回被包裹的 `RecordingObjFileSystem`；base 方法（exists/mkdirs/delete/rename/list/newInputFile/newOutputFile）委派该 delegate；`close()` 抛 `AssertionError`（借用契约守卫）。 +- `newTxnWithFs` 用**匿名 `FakeConnectorContext` 子类**（override `getFileSystem(ConnectorSession)` 返回 `new RoutingFacadeFileSystem(RecordingObjFileSystem)`，忽略 session）。 +- **为何 facade 而非直接返回 `RecordingObjFileSystem`（红队 minor-2）**：直接返回则 fake **本身**是 `ObjFileSystem`，`instanceof` 恒过——测不出"漏调 `forLocation`"的回归（生产 facade 非 ObjFileSystem，漏调即 instanceof 失败/throw）。facade 强制走 `forLocation` narrow：漏调 → `getFileSystem() instanceof ObjFileSystem` 对 facade 为 false → 3 MPU 用例 RED。 +- **借用守卫（红队 minor-4，改必做非可选）**：facade `close()` 抛 `AssertionError`；现有 3 MPU 用例已 `txn.close()`——若 §4 漏删 `fs.close()`（或回归重加 close 借用 FS）→ RED。 +- **3 个 MPU 用例断言不变**：`testCommitCompletesMultipartUploads`（complete 一次、ETag 按 part 排序）、`testRollbackAbortsPendingMultipartUploads`（abort 一次，rollback-before-commit 走 session=null 路径 → **顺带证 null-session 安全**）、`testSecondRollbackIsIdempotent`（幂等 abort 一次）。 +- 其余 9 用例（分类/NEW→APPEND/事务表拒/`getUpdateCnt`/`profileLabel`/dup-partition/addPartitions-once）不碰 FS 或走 FILE_S3 write==target 无 rename → 不受影响；`beginWrite(null,…)` 现已传 null，迁移后仍 null（引擎/fake 忽略）。 +- **import bookkeeping（红队 minor-1，必做否则编译/checkstyle 挂）**：`HiveConnectorTransactionTest` 加 `import org.apache.doris.connector.api.ConnectorSession;` + `import java.io.IOException;`（facade 委派方法声明）；删 `import org.apache.doris.filesystem.properties.StorageProperties;`（仅被删除的 `resolveObjectStoreFileSystem` override 用）。 +- **类 javadoc 订正（红队 minor-5）**：:85 "injected via the `resolveObjectStoreFileSystem` seam" → 改指 `FakeConnectorContext.getFileSystem(session)` 缝。 + +## 红队结论（wf_8fd372d6-10d，GO_WITH_FIXES） +5 lens 并行（classloader-cast / semantic-equivalence / lifecycle-session-concurrency / forlocation-mpu-fidelity / test-fidelity-completeness）+ 1 独立裁决者复核 severe 声明。**classloader/instanceof、session 捕获、null 安全、close 去除、19 非-MPU 点字节等价 均独立复现 SOUND → 无 blocker**。折入：1 major（MPU abort native-path 解析 + 两处 catch 拓宽 `Exception`，见 §3）+ 4 minor（test import / non-ObjFileSystem facade / close 守卫必做 / 类 javadoc）。裁决者纠正 lens-2 对 Azure 失败的"静默 skip"误判为"未捕获 RuntimeException 破坏 rollback"（同缺陷、正确机制）。 + +### E2E（用户自跑，勿丢——`hms-iceberg-delegation-needs-e2e`） +翻闸后写路径 live 且当前必炸（HDFS stub / 对象存储跨插件 ServiceLoader）——本 e2e 验 live 生产 bug 修复，非"休眠不可测"： +- `external_table_p0/hive` 含 INSERT 的写套件（rename/delete + MPU complete）：HDFS 后端（验 stub→引擎 FS 转换 + 新 scheme 支持）+ 若有对象存储环境验 s3/oss MPU。 +- INSERT 失败回滚（验 MPU abort 不泄漏 server 端 upload）。 +- 断言与老 fe-core `HMSTransaction` 逐位一致（落盘、事务提交/回滚、分区 add、统计更新）。 + +## Open / 已决 +- **测试注入缝**（清单 Open #4 的伴生项）**已决**：迁到 `FakeConnectorContext.getFileSystem`-override（非保留 `resolveObjectStoreFileSystem` 死方法）——注入点对齐生产真实来源（引擎给 FS），保真度更高。 +- **FS/identity 捕获时机**（清单 Open #4）**已决**：`beginWrite` 捕获 session；FS 解析惰性（引擎 per-catalog 缓存，等价 begin 时建）。 +- per-user identity 真正落地 → Future（清单 Future §），不属本步。 diff --git a/plan-doc/tasks/task-list-HIVEFS.md b/plan-doc/tasks/task-list-HIVEFS.md index 6b3b6936773218..4f440b6d3caa94 100644 --- a/plan-doc/tasks/task-list-HIVEFS.md +++ b/plan-doc/tasks/task-list-HIVEFS.md @@ -48,13 +48,14 @@ - **测试注入方式已决=`fe-filesystem-local` 真 `LocalFileSystem`**（+ 抛-`listFiles` 子类 `LiteralListingLocalFileSystem` 钉字面量列），非 in-memory fake（保真、改动最小）。`HiveAcidUtilTest` 9 用例全迁、全绿。pom 加 `fe-filesystem-local` test 依赖；`commons-lang` test 依赖**实测证实仍需**（hive-common `Valid*.writeToString` 引用，非 Hadoop），保留+订正注释。 - 校验：`-pl :fe-connector-hive -am -Dtest=HiveAcidUtilTest -DfailIfNoTests=false test` = BUILD SUCCESS + 9/9 + 0 checkstyle。 -- [ ] **HIVEFS-6（连接器·写路径，最险）**：`HiveConnectorTransaction`： - - `resolveObjectStoreFileSystem`（坏 ServiceLoader stub，:784-793）→ `context.getFileSystem(session).forLocation(loc)`；MPU `instanceof ObjFileSystem`（:809/1336）落到 `forLocation` 返回的具体 FS；`objFs.completeMultipartUpload`/`getObjStorage().abortMultipartUpload` 不变。 - - 事务内 delete/rename/exists 等裸 Hadoop 调用一并换注入 FS。 - - `close()`：**不再 close 借用的引擎 FS**（仅关自有资源）。 - - **待核**：commit/abort 时机 session 是否还在 → 在 `beginWrite`（:207，已捕获 user）时捕获所需 FS/identity 供后续复用。 - - UT：`resolveObjectStoreFileSystem` 为 `protected` 可 override 注入 fake ObjFileSystem；断言 MPU complete/abort 路径 + close 不关借用 FS。 - - 校验：`-Dtest=HiveConnectorTransaction*,HiveWrite* -DfailIfNoTests=false`。 +- [x] **HIVEFS-6（连接器·写路径，最险）** ✅ DONE（code `d31ceb0364e`；设计 `designs/FIX-HIVEFS-6-design.md`；设计红队 `wf_8fd372d6-10d` GO_WITH_FIXES 全折入；fe-connector-hive BUILD SUCCESS、`HiveConnectorTransactionTest` 14/14、0 checkstyle）：`HiveConnectorTransaction`： + - **关键发现**：`getFileSystem()`（:755）本已全程用 Doris `FileSystem` API（19 个非-MPU I/O 点 + 2 MPU），缺陷**只在 FS 来源**——`resolveObjectStoreFileSystem`（:784）本地 `ServiceLoader.load(FileSystemProvider)`（跨插件拿不到 provider）+ `.filter(OBJECT_STORAGE)`（HDFS 后端 `objSp==null` 直接抛）。**翻闸后 live**（class-javadoc "dormant" 陈旧），hive INSERT 今天在 HDFS/对象存储上都必炸。 + - `getFileSystem()` 换来源 → `context.getFileSystem(session)`（引擎 per-catalog `SpiSwitchingFileSystem`，全 scheme）；删 `resolveObjectStoreFileSystem` + `fs` 字段 + OBJECT_STORAGE 过滤 + 孤儿 import（ServiceLoader/StorageKind/StorageProperties/FileSystemProvider）。19 非-MPU 点**不动**（facade 逐操作 `forLocation` 委派）。 + - MPU 两处 narrow 具体 `ObjFileSystem`：`objCommit`（complete，strict）`forLocation(Location.of(path))`（native 写目标）；`abortMultiUploads`（abort，lenient）循环内逐 upload `forLocation(Location.of(u.path))`（**红队 major**：native-scheme `u.path`=`pu.getLocation().getWritePath()`，非合成 `s3://`——否则 Azure catalog 不可解析）。 + - **两处 catch 拓宽 `IOException`→`Exception`**（红队 major）：`SpiSwitchingFileSystem.forLocation` props 解析失败抛 `StoragePropertiesException`（RuntimeException）；窄 catch 会逃逸破坏 rollback/泄漏 MPU。 + - `close()`：删 `fs.close()`（借用引擎 FS 不关，仅关自有 executor）。session 于 `beginWrite`（:207）捕获（null 安全：引擎忽略 session）。class-javadoc dormant→live 订正。 + - UT：注入缝迁 `resolveObjectStoreFileSystem`-override → `FakeConnectorContext.getFileSystem`-override + **non-`ObjFileSystem` 路由 facade**（红队 minor：强制走 `forLocation`，漏调即 instanceof RED；facade `close()` 抛 AssertionError 钉借用契约）。import bookkeeping（+`ConnectorSession`/`java.io.IOException`、−`StorageProperties`）。 + - 校验：`-pl :fe-connector-hive -am -Dtest=HiveConnectorTransactionTest -DfailIfNoTests=false test` = BUILD SUCCESS + 14/14 + 0 checkstyle。 - [ ] **HIVEFS-7（pom + 去 jar，达成终点）**：删 `fe-connector-hive/pom.xml` 的 `hadoop-hdfs-client`；**保留** `hadoop-common`（`Configuration`/`HiveConf`/HMS client 仍需）。 - 全局 grep 校验连接器 main 源 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get` **零残留**（`new Path` 若仅路径拼接可留，但确认不再 FS I/O）。 @@ -67,8 +68,9 @@ ## 设计红队（落地前） -- [ ] 按 `clean-room-adversarial-review-pref`：实现前对设计做多 agent 对抗红队（重点：写路径 MPU/rename/delete 语义等价、生命周期误 close、`forLocation` 与 SpiSwitchingFileSystem 缓存交互、session-ignore 的 catalog 级正确性）。 - - [x] HIVEFS-5 已做（`wf_792d1900-cc7`，5 lens + 逐条对抗验证）：迁移码 4 lens SOUND；唯一 REAL(major)=planAcidScan live 非休眠，已折入注释/文档订正。**HIVEFS-6 写路径红队仍待做（落地前）。** +- [x] 按 `clean-room-adversarial-review-pref`：实现前对设计做多 agent 对抗红队（重点：写路径 MPU/rename/delete 语义等价、生命周期误 close、`forLocation` 与 SpiSwitchingFileSystem 缓存交互、session-ignore 的 catalog 级正确性）。 + - [x] HIVEFS-5 已做（`wf_792d1900-cc7`）。 + - [x] **HIVEFS-6 写路径红队已做（`wf_8fd372d6-10d`，GO_WITH_FIXES）**：5 lens（classloader-cast / semantic-equivalence / lifecycle-session-concurrency / forlocation-mpu-fidelity / test-fidelity）+ 1 独立裁决者。classloader/instanceof、session/null、close 去除、19 非-MPU 字节等价 均 SOUND；无 blocker。折入 1 major（MPU abort native-path + 两处 catch 拓宽 Exception）+ 4 minor（test import / non-ObjFileSystem facade / close 守卫必做 / 类 javadoc）。 ## e2e（用户自跑，勿丢——新能力必配 e2e，memory `hms-iceberg-delegation-needs-e2e`） @@ -83,7 +85,7 @@ 1. ~~`DefaultConnectorContext` 缓存 FS 的 close 挂点~~ ✅ **已解（HIVEFS-3）**：catalog 单一持有 + `onClose`/换连接器两处关，`DefaultConnectorContext implements Closeable`（见上 HIVEFS-3）。 2. `HiveScanPlanProvider.buildHadoopConf()`/`Configuration` 在去 FS.get 后的去留（格式/split/传 BE 是否仍需）——HIVEFS-4 定。 3. ~~`HiveAcidUtilTest` 从真 LocalFileSystem 迁到 fake/`fe-filesystem-local` 的注入方式~~ ✅ **已解（HIVEFS-5）**：用 `fe-filesystem-local` 真 `LocalFileSystem`（+ 抛-`listFiles` 子类守卫），非 in-memory fake。`commons-lang` test 依赖实测证实仍需（hive-common `Valid*`）。 -4. 写路径 commit/abort 时 FS/identity 的捕获时机（begin 时捕获）——HIVEFS-6 定。 +4. ~~写路径 commit/abort 时 FS/identity 的捕获时机~~ ✅ **已解（HIVEFS-6）**：`session` 于 `beginWrite` 捕获为字段；FS 解析惰性（引擎 per-catalog 缓存，等价 begin 时建）；null-session（rollback-before-begin/测试）安全（引擎忽略 session）。注入缝迁 `FakeConnectorContext.getFileSystem`-override + non-`ObjFileSystem` facade（非保留死 `resolveObjectStoreFileSystem`）。 5. ~~`forLocation` 加到接口后与现有非切换 impl 的兼容~~ ✅ **已解（HIVEFS-1）**：default `return this`；`SpiSwitchingFileSystem` override 返回 per-scheme 委派；`FileSystemDefaultMethodsTest` 已断言。 ## Future（不属本次） From 23a0a5b7bb33620ca089b3dc307d093b8b712b59 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 17:48:22 +0800 Subject: [PATCH 249/334] [doc](catalog) hive-fs: verify HIVEFS-7 (hadoop-hdfs-client already excluded), repoint to HIVEFS-8 HIVEFS-7 (drop the redundant HDFS-client jar) needs no pom/code change: the connector pom never carried a direct hadoop-hdfs-client (the Option-A band-aid was reverted in HIVEFS-0), and fe-connector-hms -> hadoop-common does NOT pull it transitively -- `mvn dependency:tree -pl :fe-connector-hive` shows zero hadoop-hdfs-client in the whole closure, so there is no pom line to delete and an for a non-existent transitive dep would be dead config. Packaging proof (`-pl :fe-connector-hive -am package` BUILD SUCCESS): the fresh plugin zip lib/ has NO hadoop-hdfs-client, retains hadoop-common (+ shaded protobuf/guava/annotations/auth for Configuration/HiveConf/Path/HMS client), root plugin jar present, 82 lib jars, zip intact. Connector main sources carry zero bare org.apache.hadoop.fs.FileSystem/FileStatus/FileSystem.get/listStatus (only o.a.h.fs.Path string manipulation remains). The stale output/ and pre-HIVEFS-6 target zip still bundle hadoop-hdfs-client -- they are leftovers from the reverted band-aid build (today 12:44/12:48), not the current graph; a full rebuild for HIVEFS-8 removes them. Next: HIVEFS-8 (full build + rebuild output/ + e2e regression). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- plan-doc/HANDOFF.md | 12 +++++++----- plan-doc/tasks/task-list-HIVEFS.md | 10 ++++++---- 2 files changed, 13 insertions(+), 9 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index ad9cadae85a4f0..bb02d96fdded56 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,9 +5,9 @@ --- -# 🆕 下一个 session 起步 = FIX-HIVEFS 续（下一步 = 去 jar HIVEFS-7 + 全量 build/e2e HIVEFS-8，收尾） +# 🆕 下一个 session 起步 = FIX-HIVEFS 收尾（下一步 = 全量 build + e2e 交接 HIVEFS-8） -> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。连接器读+ACID+写三条裸 Hadoop 路径已全部转引擎注入 FileSystem（HIVEFS-4/5/6 DONE），只剩去依赖 jar + 全量验证。** +> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。连接器读+ACID+写三条裸 Hadoop 路径已全部转引擎注入 FileSystem（HIVEFS-4/5/6 DONE），去 jar 已核实达成（HIVEFS-7 DONE，无需改 pom）；只剩全量 build + e2e。** **根因**：连接器扫描/写全程用**裸 `org.apache.hadoop.fs.FileSystem`**（`HiveFileListingCache:162`/`HiveScanPlanProvider:272`/`HiveAcidUtil`/`HiveConnectorTransaction`），但 hive 插件类加载器隔离、lib/ 无 `DistributedFileSystem`（仅 hadoop-common，注册 Local/View/Har/Http、无 hdfs）。老 fe-core 经 `org.apache.doris.filesystem.FileSystem`+`DirectoryLister` 列文件（`HiveExternalMetaCache:392-396`），**新代码裸 Hadoop 是迁移走样**。非环境问题（hostname 可解析、metastore 通、已取到 location）。 @@ -25,20 +25,22 @@ **✅ HIVEFS-6 DONE（code `d31ceb0364e`）= 连接器·写路径已转引擎 FileSystem（最险，已过红队）**：`HiveConnectorTransaction` `getFileSystem()`（:755）本已全程用 Doris `FileSystem` API（19 非-MPU I/O + 2 MPU），**缺陷只在 FS 来源**——`resolveObjectStoreFileSystem` 本地 `ServiceLoader`（跨插件拿不到 provider）+ `.filter(OBJECT_STORAGE)`（HDFS 后端直接抛）。翻闸后 **live**（class-javadoc "dormant" 陈旧），hive INSERT 今天在 HDFS/对象存储上都必炸。改：`getFileSystem()`→`context.getFileSystem(session)`（引擎 per-catalog `SpiSwitchingFileSystem`，全 scheme）；删 `resolveObjectStoreFileSystem`+`fs` 字段+OBJECT_STORAGE 过滤+孤儿 import；19 非-MPU 点不动（facade 逐操作 `forLocation` 委派）；MPU 两处 `forLocation` narrow 具体 `ObjFileSystem`（`objCommit`=native 写目标 strict、`abortMultiUploads`=逐 upload native `u.path` lenient）；`close()` 删 `fs.close()`（借用不关）；`session` 于 `beginWrite` 捕获（null 安全）。**红队 `wf_8fd372d6-10d`（5 lens+裁决，GO_WITH_FIXES）**：classloader/instanceof（provided-scope+parent-first→共享 `ObjFileSystem`）、session/null、close 去除、19 非-MPU 字节等价 均 SOUND；折入 1 major（MPU abort 用 native-scheme `u.path` 非合成 `s3://`，Azure 才可解析；两处 catch `IOException`→`Exception` 兜 `StoragePropertiesException` RuntimeException）+ 4 minor（test import / non-`ObjFileSystem` facade 强制 `forLocation` + `close()` 抛 AssertionError 钉借用契约 / 类 javadoc）。**fe-connector-hive BUILD SUCCESS、`HiveConnectorTransactionTest` 14/14、0 checkstyle**。设计 `designs/FIX-HIVEFS-6-design.md`。 -**⭐ 下一步 = HIVEFS-7（去 jar，达成终点）→ HIVEFS-8（全量 build+e2e）**：① 删 `fe-connector-hive/pom.xml` 的 `hadoop-hdfs-client`（**保留** `hadoop-common`——`Configuration`/`HiveConf`/HMS client 仍需）；全局 grep 校验连接器 main 源 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get`/`listStatus` **零残留**（`org.apache.hadoop.fs.Path` 若仅路径拼接可留，确认不再 FS I/O）；打包校验 `output/fe/plugins/connector/hive/lib/` 无 `hadoop-hdfs-client`、插件 zip 完整。② 全量 build（fe-filesystem-api+spi + fe-core + fe-connector-hive）+ 全 UT + 0 checkstyle + import 门净。③ e2e（用户自跑）：重部署后 `test_string_dict_filter`（读 hdfs）+ hive INSERT 写套件（rename/delete/MPU complete）+ INSERT 回滚（MPU abort）+ 若有对象存储环境抽查 s3/oss。 +**✅ HIVEFS-7 DONE（无 commit——无需改 pom/代码，纯核实+打包实测）= 去 jar 已达成**：task-list 原设想"删 pom 的 `hadoop-hdfs-client`"是 Option A 时代假设；实走 Option B（借引擎 FS）+ HIVEFS-0 撤销创可贴，故**连接器 pom 从无该直接依赖**、`fe-connector-hms→hadoop-common` **不传递** hadoop-hdfs-client（`mvn dependency:tree -pl :fe-connector-hive` 全树零命中）→ **pom 无一行可删、加 `` 是死配置**。打包实测（`-am package` BUILD SUCCESS）：新 zip `lib/` **无 hadoop-hdfs-client**、**留 hadoop-common**（+shaded 供 `Configuration`/`HiveConf`/`Path`/HMS）、root jar 在、82 lib、完整。连接器 main 源裸 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get`/`listStatus` 零残留（仅 `Path` 纯拼接）。**⚠ `output/fe/plugins/connector/hive/lib/` 仍含 hadoop-hdfs-client 是撤销前旧构建残留（今日 12:48 创可贴态），HIVEFS-8 全量重构建后消失；e2e 前须重打包重部署。** + +**⭐ 下一步 = HIVEFS-8（全量 build + e2e，收尾）**：① 全量 build（fe-filesystem-api+spi + fe-core + fe-connector-hive；后台跑读 LOG，`BUILD SUCCESS`/`Tests run`/checkstyle）+ 全 UT 绿 + 0 checkstyle + import 门净 + 重构建 output/（刷掉陈旧 hadoop-hdfs-client）。② e2e（用户自跑）：重部署后 `test_string_dict_filter`（读 hdfs，本失败用例）+ `external_table_p0/hive` 含 INSERT 写套件（验 rename/delete/MPU complete）+ INSERT 回滚（验 MPU abort）+ 若有对象存储环境抽查 s3/oss；断言与老 fe-core 逐位一致。 **已探明的去风险事实（勿重查）**： - **引擎侧已落地（HIVEFS-3）**——`DefaultConnectorContext.getFileSystem` 用已持的 `storagePropertiesSupplier` + `SpiSwitchingFileSystem`（范式镜像 `cleanupEmptyManagedLocation:348`）；close 由 catalog 在 `onClose`/换连接器处关（context 单一持有、sibling 共享）。 - **TCCL 无需连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs **自钉**到自身插件 loader 再 `FileSystem.get`、finally 还原（注释即描述 “No FileSystem for scheme hdfs” 场景）。连接器去 jar 后任意 TCCL 调用皆安全。 - 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction` objCommit/abort）→ `FileSystem.forLocation`（HIVEFS-1 加、HIVEFS-6 已用：非切换返回 this、切换返回具体 FS）。**跨 连接器↔fs-plugin `instanceof ObjFileSystem` 安全**（`fe-filesystem-spi` 双 provided-scope + `FileSystemPluginManager:72` parent-first `"org.apache.doris.filesystem."` → 共享同一 `ObjFileSystem` Class；红队 HIVEFS-6 逐点取证）。 -- 连接器裸 Hadoop **FileSystem I/O 面已全清**（读扫描 HIVEFS-4 + ACID HIVEFS-5 + 写路径 HIVEFS-6）：三条路径均经 `context.getFileSystem(session)` 借引擎 `SpiSwitchingFileSystem`；仅剩 **pom 的 `hadoop-hdfs-client` jar 依赖**待 HIVEFS-7 删。`FileStatus` 字段全映射 `FileEntry`（`name()`/`location().uri()`/`length()`/`modificationTime()`/`isDirectory()`）。 +- 连接器裸 Hadoop **FileSystem I/O 面已全清**（读扫描 HIVEFS-4 + ACID HIVEFS-5 + 写路径 HIVEFS-6）：三条路径均经 `context.getFileSystem(session)` 借引擎 `SpiSwitchingFileSystem`；`hadoop-hdfs-client` 已核实不在依赖图（HIVEFS-7）。`FileStatus` 字段全映射 `FileEntry`（`name()`/`location().uri()`/`length()`/`modificationTime()`/`isDirectory()`）。 - fe-filesystem 全 9 插件（hdfs/s3/oss/cos/obs/azure/http/local/broker）已部署 → hive 表任意后端天然支持（免 bundle hadoop-aws/huaweicloud，架构红利）。 **⚠ 待下个 session 核定**（task-list Open §）：无阻塞项待决——HIVEFS-7 是纯 pom 删依赖 + grep 校验，HIVEFS-8 是全量 build + e2e。（写路径 FS/identity 捕获=HIVEFS-6 已定 `beginWrite`；`buildHadoopConf` 去留=HIVEFS-4/5 已删；`HiveAcidUtilTest` 注入=HIVEFS-5 定 `fe-filesystem-local`。）**HIVEFS-7 删 `hadoop-hdfs-client` 后须打包实测**（避免误删 `hadoop-common` 传递依赖破 HMS client/`HiveConf`）。 **⚠ 重要事实（红队 HIVEFS-5 确认）**：翻闸后 hive **事务表读已 live 走 plugin**（`planAcidScan` 非休眠）；HIVEFS-5 前它在 hdfs 上就是坏的（`No FileSystem for scheme hdfs`），HIVEFS-5 修的是 live 生产 bug（e2e 因需 docker HMS+hdfs harness 延后，非"不可测"）。 -**⚠ 状态**：HIVEFS-0/1/2/3/4/5/6 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9`/`7a2d8714951`/`d31ceb0364e` + doc commit）。落地纪律：每子步 = 设计→红队→折入→实现→UT→独立 commit（HIVEFS-4/5/6 均已按此走完一轮）。剩 HIVEFS-7（去 jar）/HIVEFS-8（全量 build+e2e）为收尾，无需红队（纯依赖删除 + 验证）。 +**⚠ 状态**：HIVEFS-0/1/2/3/4/5/6 已入库（code `0c4e0595f8f`/`3b4f7477d34`/`a8ed72f2650`/`7e06c2aa2a9`/`7a2d8714951`/`d31ceb0364e` + doc commit）；HIVEFS-7 = 无代码变更（核实+打包实测，达成去 jar 目标）。落地纪律：每子步 = 设计→红队→折入→实现→UT→独立 commit（HIVEFS-4/5/6 均已按此走完一轮）。剩 HIVEFS-8（全量 build + 重构建 output/ + e2e）为收尾。 --- diff --git a/plan-doc/tasks/task-list-HIVEFS.md b/plan-doc/tasks/task-list-HIVEFS.md index 4f440b6d3caa94..41efccb53e9955 100644 --- a/plan-doc/tasks/task-list-HIVEFS.md +++ b/plan-doc/tasks/task-list-HIVEFS.md @@ -57,10 +57,12 @@ - UT：注入缝迁 `resolveObjectStoreFileSystem`-override → `FakeConnectorContext.getFileSystem`-override + **non-`ObjFileSystem` 路由 facade**（红队 minor：强制走 `forLocation`，漏调即 instanceof RED；facade `close()` 抛 AssertionError 钉借用契约）。import bookkeeping（+`ConnectorSession`/`java.io.IOException`、−`StorageProperties`）。 - 校验：`-pl :fe-connector-hive -am -Dtest=HiveConnectorTransactionTest -DfailIfNoTests=false test` = BUILD SUCCESS + 14/14 + 0 checkstyle。 -- [ ] **HIVEFS-7（pom + 去 jar，达成终点）**：删 `fe-connector-hive/pom.xml` 的 `hadoop-hdfs-client`；**保留** `hadoop-common`（`Configuration`/`HiveConf`/HMS client 仍需）。 - - 全局 grep 校验连接器 main 源 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get` **零残留**（`new Path` 若仅路径拼接可留，但确认不再 FS I/O）。 - - 打包校验：`output/fe/plugins/connector/hive/lib/` 无 `hadoop-hdfs-client`；插件 zip 仍完整。 - - 依赖：HIVEFS-4/5/6 全完成后做。 +- [x] **HIVEFS-7（去 jar，达成终点）** ✅ DONE（**无需改 pom/代码**——已核实当前依赖图本就不含 `hadoop-hdfs-client`；打包实测确认）： + - **关键纠正**：task-list 原设想"删 `fe-connector-hive/pom.xml` 的 `hadoop-hdfs-client`"是 Option A（bundle hdfs）时代的假设。实际走 Option B（借引擎 FS）+ HIVEFS-0 撤销了未提交的 hdfs-client 创可贴，故**连接器 pom 从无该直接依赖**，`fe-connector-hms→hadoop-common` 也**不传递** hadoop-hdfs-client（`mvn dependency:tree -pl :fe-connector-hive` 全树零 hdfs-client 命中）。→ **pom 无一行可删、无需加 ``**（对不存在的传递依赖加排除是死配置）。 + - **grep 校验**：连接器 main 源 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get`/`listStatus` **零残留**（HIVEFS-4/5/6 已清；仅 `org.apache.hadoop.fs.Path` 纯路径拼接留存，非 FS I/O）。 + - **打包实测**（`-pl :fe-connector-hive -am package -DskipTests` BUILD SUCCESS，17:45 新 zip）：`target/doris-fe-connector-hive.zip` 的 `lib/` **无 `hadoop-hdfs-client`**、**保留 `hadoop-common`**（+shaded protobuf/guava/annotations/auth，供 `Configuration`/`HiveConf`/`Path`/HMS client）、root 插件 jar 在、82 个 lib jar，zip 完整。 + - **⚠ 陈旧产物**：`output/fe/plugins/connector/hive/lib/` 与旧 `target` zip（今日 12:44/12:48 打包，创可贴 pom 状态）**仍含 `hadoop-hdfs-client`**——是撤销前的旧构建残留，非当前图。**HIVEFS-8 全量重构建 output/ 后即消失**；e2e 前须重打包重部署（否则测的是带 hdfs-client 的旧插件）。 + - 依赖：HIVEFS-4/5/6 全完成后做（已满足）。 - [ ] **HIVEFS-8（全量构建 + UT + e2e 交接）**：fe-filesystem-api + fe-connector-spi + fe-core + fe-connector-hive 全量 build（后台跑读 LOG，`BUILD SUCCESS`/`Tests run`/checkstyle），全 UT 绿、0 checkstyle、import 门净。 From 37911f2b3fedd40ad6c832e26dce2174bb7003fb Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 18:31:57 +0800 Subject: [PATCH 250/334] [fix](catalog) hive-fs: pin engine HDFS conf classloader (RpcEngine split-brain, HIVEFS-9) test_string_dict_filter q01 (hdfs read) failed after HIVEFS-4 routed connector listing through the engine FileSystem: class org.apache.hadoop.ipc.ProtobufRpcEngine2 cannot be cast to class org.apache.hadoop.ipc.RpcEngine (ProtobufRpcEngine2 in ChildFirstClassLoader @5eea5627 [hive connector]; RpcEngine in 'app' [engine]) Root cause: HdfsConfigBuilder.build() is the only Hadoop-conf builder in the tree that never pins conf.setClassLoader. new HdfsConfiguration() captures the live TCCL into the conf's own classLoader field. DFSFileSystem is built lazily on first HDFS access during a scan, which runs under the hive connector's plugin loader (PluginDrivenScanNode.onPluginClassLoader), so the conf captured that connector loader. RPC.getProtocolEngine then loads ProtobufRpcEngine2 via Configuration.getClass -- which uses the conf's own classLoader field, NOT the live TCCL -- from the connector's hadoop-common copy, while RPC/RpcEngine resolve to the engine copy -> ClassCast. DFSFileSystem.getHadoopFs pins the live TCCL for FileSystem.get (ServiceLoader discovery) but cannot fix the conf-cached CL, exactly as HmsConfHelper:51-58 already documents for the HMS metastore path. Fix: pin conf.setClassLoader(HdfsConfigBuilder.class.getClassLoader()) in build(), mirroring HmsConfHelper / Paimon / Iceberg / Hive / Hudi conf builders. Scheme-agnostic; touches only fe-filesystem-hdfs. Test: HdfsConfigBuilderTest.buildPinsPluginClassLoaderNotTccl (RED-verified: without the fix, expected but was ). fe-filesystem-hdfs 9/9, BUILD SUCCESS, 0 checkstyle. Design: plan-doc/tasks/designs/FIX-HIVEFS-9-design.md Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- .../filesystem/hdfs/HdfsConfigBuilder.java | 11 +++++++++ .../hdfs/HdfsConfigBuilderTest.java | 23 +++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilder.java b/fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilder.java index 8c100f80e31c02..ab44b65e931596 100644 --- a/fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilder.java +++ b/fe/fe-filesystem/fe-filesystem-hdfs/src/main/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilder.java @@ -41,6 +41,17 @@ private HdfsConfigBuilder() { */ public static Configuration build(Map properties) { Configuration conf = new HdfsConfiguration(); + // Pin the conf classloader to this plugin's loader, mirroring HmsConfHelper.createHiveConf and the + // connector conf builders (Paimon/Iceberg/Hive/Hudi). new HdfsConfiguration() captures the LIVE + // thread-context classloader into the conf's OWN classLoader field. DFSFileSystem is built lazily on + // first HDFS access, which runs under a connector's plugin loader (PluginDrivenScanNode.onPluginClassLoader + // pins the TCCL there for the scan), so the captured CL would be that connector loader. Hadoop then + // resolves impl classes via Configuration.getClass using this field, NOT the live TCCL: e.g. + // RPC.getProtocolEngine loads ProtobufRpcEngine2 through it. Left unpinned that yields the connector's + // hadoop-common copy of ProtobufRpcEngine2 while RPC/RpcEngine come from the engine copy, giving + // "class ProtobufRpcEngine2 cannot be cast to class RpcEngine". DFSFileSystem.getHadoopFs pins the live + // TCCL for FileSystem.get (ServiceLoader discovery) but cannot fix this conf-cached CL. + conf.setClassLoader(HdfsConfigBuilder.class.getClassLoader()); conf.setBoolean("fs.hdfs.impl.disable.cache", true); conf.setBoolean("fs.AbstractFileSystem.hdfs.impl.disable.cache", true); for (String scheme : HdfsFileSystemProvider.SUPPORTED_SCHEMES) { diff --git a/fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilderTest.java b/fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilderTest.java index bd3384d0fe2db7..8d011394ef001a 100644 --- a/fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilderTest.java +++ b/fe/fe-filesystem/fe-filesystem-hdfs/src/test/java/org/apache/doris/filesystem/hdfs/HdfsConfigBuilderTest.java @@ -69,6 +69,29 @@ void buildIgnoresNullAndEmptyValues() { && "empty.key".equals(conf.get("empty.key"))); } + @Test + void buildPinsPluginClassLoaderNotTccl() { + // WHY (test_string_dict_filter, hdfs scan): Hadoop resolves impl classes via Configuration.getClass, + // which uses the conf's OWN classLoader field. new HdfsConfiguration() captures the thread-context CL + // active AT CONSTRUCTION into that field. DFSFileSystem is built under a connector's plugin loader + // during a scan, so unpinned the conf would carry that connector loader; then RPC.getProtocolEngine + // loads ProtobufRpcEngine2 from the connector's hadoop-common copy while RPC/RpcEngine come from the + // engine copy -> "class ProtobufRpcEngine2 cannot be cast to class RpcEngine". The conf MUST be pinned + // to this plugin's loader. MUTATION: drop the setClassLoader in build() -> the conf keeps the foreign + // TCCL below -> red. (A flat-classpath assertion alone cannot repro the real cross-loader cast, so we + // install a distinct TCCL to make the captured-loader bug observable offline.) + ClassLoader foreign = new java.net.URLClassLoader(new java.net.URL[0], null); + ClassLoader prev = Thread.currentThread().getContextClassLoader(); + try { + Thread.currentThread().setContextClassLoader(foreign); + Configuration conf = HdfsConfigBuilder.build(Map.of()); + Assertions.assertSame(HdfsConfigBuilder.class.getClassLoader(), conf.getClassLoader()); + Assertions.assertNotSame(foreign, conf.getClassLoader()); + } finally { + Thread.currentThread().setContextClassLoader(prev); + } + } + @Test void isKerberosEnabledBothPresent() { Assertions.assertTrue(HdfsConfigBuilder.isKerberosEnabled(Map.of( From f16878624c6806e0f171847fc19a84d62238aa68 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 18:33:09 +0800 Subject: [PATCH 251/334] [doc](catalog) hive-fs: land HIVEFS-9 design, correct stale "TCCL de-risked" claim Records FIX-HIVEFS-9 (engine HdfsConfigBuilder conf must pin its own classloader; ProtobufRpcEngine2-vs-RpcEngine split-brain surfaced by test_string_dict_filter q01 during HIVEFS-8 e2e). Corrects the HANDOFF "de-risked facts" line that wrongly claimed DFSFileSystem.getHadoopFs's TCCL self-pin made any TCCL caller safe: the TCCL pin fixes ServiceLoader discovery but not the conf-cached classLoader field that Configuration.getClass (RPC.getProtocolEngine) actually reads. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- plan-doc/HANDOFF.md | 6 +- plan-doc/tasks/designs/FIX-HIVEFS-9-design.md | 74 +++++++++++++++++++ 2 files changed, 78 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-HIVEFS-9-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index bb02d96fdded56..225f758c47936e 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -7,7 +7,7 @@ # 🆕 下一个 session 起步 = FIX-HIVEFS 收尾（下一步 = 全量 build + e2e 交接 HIVEFS-8） -> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。连接器读+ACID+写三条裸 Hadoop 路径已全部转引擎注入 FileSystem（HIVEFS-4/5/6 DONE），去 jar 已核实达成（HIVEFS-7 DONE，无需改 pom）；只剩全量 build + e2e。** +> 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。连接器读+ACID+写三条裸 Hadoop 路径已全部转引擎注入 FileSystem（HIVEFS-4/5/6 DONE），去 jar 已核实达成（HIVEFS-7 DONE，无需改 pom）；HIVEFS-8 e2e 首跑引出引擎 conf 的 classloader 阻断已修（**HIVEFS-9 DONE** `37911f2b3fe`，`test_string_dict_filter` q01 的 `ProtobufRpcEngine2 cannot be cast to RpcEngine`）；只剩全量 build + 重部署 + e2e 自跑。** **根因**：连接器扫描/写全程用**裸 `org.apache.hadoop.fs.FileSystem`**（`HiveFileListingCache:162`/`HiveScanPlanProvider:272`/`HiveAcidUtil`/`HiveConnectorTransaction`），但 hive 插件类加载器隔离、lib/ 无 `DistributedFileSystem`（仅 hadoop-common，注册 Local/View/Har/Http、无 hdfs）。老 fe-core 经 `org.apache.doris.filesystem.FileSystem`+`DirectoryLister` 列文件（`HiveExternalMetaCache:392-396`），**新代码裸 Hadoop 是迁移走样**。非环境问题（hostname 可解析、metastore 通、已取到 location）。 @@ -27,11 +27,13 @@ **✅ HIVEFS-7 DONE（无 commit——无需改 pom/代码，纯核实+打包实测）= 去 jar 已达成**：task-list 原设想"删 pom 的 `hadoop-hdfs-client`"是 Option A 时代假设；实走 Option B（借引擎 FS）+ HIVEFS-0 撤销创可贴，故**连接器 pom 从无该直接依赖**、`fe-connector-hms→hadoop-common` **不传递** hadoop-hdfs-client（`mvn dependency:tree -pl :fe-connector-hive` 全树零命中）→ **pom 无一行可删、加 `` 是死配置**。打包实测（`-am package` BUILD SUCCESS）：新 zip `lib/` **无 hadoop-hdfs-client**、**留 hadoop-common**（+shaded 供 `Configuration`/`HiveConf`/`Path`/HMS）、root jar 在、82 lib、完整。连接器 main 源裸 `org.apache.hadoop.fs.FileSystem`/`FileStatus`/`FileSystem.get`/`listStatus` 零残留（仅 `Path` 纯拼接）。**⚠ `output/fe/plugins/connector/hive/lib/` 仍含 hadoop-hdfs-client 是撤销前旧构建残留（今日 12:48 创可贴态），HIVEFS-8 全量重构建后消失；e2e 前须重打包重部署。** +**✅ HIVEFS-9 DONE（code `37911f2b3fe`）= 引擎 HDFS conf 补钉 classLoader（HIVEFS-8 e2e 首跑引出的 classloader 阻断）**：本地 hive 回归 `test_string_dict_filter` q01（读 hdfs）在 HIVEFS-4 转引擎 FS 后**换了个错**——从原 `No FileSystem for scheme hdfs` 变成 `class ProtobufRpcEngine2 cannot be cast to class RpcEngine`（连接器 loader vs app 引擎 loader）。根因=`HdfsConfigBuilder.build` 是**全树唯一漏钉 conf classLoader 的 Hadoop-conf 构造点**：`new HdfsConfiguration()` 捕获 live TCCL 入 conf 的 `classLoader` 字段；`DFSFileSystem` 在扫描线程（`onPluginClassLoader` 钉 hive 连接器 loader）下**惰性**构造 → conf 捕获连接器 loader → `RPC.getProtocolEngine` 经 `conf.getClass` 从连接器 hadoop-common 副本取 `ProtobufRpcEngine2`、撞 app 引擎的 `RpcEngine`。修=`build()` 补 `conf.setClassLoader(HdfsConfigBuilder.class.getClassLoader())`，**镜像 `HmsConfHelper`/`Paimon`/`Iceberg`/`Hive`/`Hudi` 6 处约定**。RED 实证（抽掉→`expected but was `）、`fe-filesystem-hdfs` **9/9、BUILD SUCCESS、0 checkstyle**。设计 `designs/FIX-HIVEFS-9-design.md`。**scope=引擎一处，连接器均不动；e2e 仍待用户重部署自跑（本 fix 只解此 classloader 阻断，q01 其后断言按 HIVEFS-8 矩阵回归）。** + **⭐ 下一步 = HIVEFS-8（全量 build + e2e，收尾）**：① 全量 build（fe-filesystem-api+spi + fe-core + fe-connector-hive；后台跑读 LOG，`BUILD SUCCESS`/`Tests run`/checkstyle）+ 全 UT 绿 + 0 checkstyle + import 门净 + 重构建 output/（刷掉陈旧 hadoop-hdfs-client）。② e2e（用户自跑）：重部署后 `test_string_dict_filter`（读 hdfs，本失败用例）+ `external_table_p0/hive` 含 INSERT 写套件（验 rename/delete/MPU complete）+ INSERT 回滚（验 MPU abort）+ 若有对象存储环境抽查 s3/oss；断言与老 fe-core 逐位一致。 **已探明的去风险事实（勿重查）**： - **引擎侧已落地（HIVEFS-3）**——`DefaultConnectorContext.getFileSystem` 用已持的 `storagePropertiesSupplier` + `SpiSwitchingFileSystem`（范式镜像 `cleanupEmptyManagedLocation:348`）；close 由 catalog 在 `onClose`/换连接器处关（context 单一持有、sibling 共享）。 -- **TCCL 无需连接器侧处理**——`DFSFileSystem.getHadoopFs:131-144` 对 hdfs/viewfs **自钉**到自身插件 loader 再 `FileSystem.get`、finally 还原（注释即描述 “No FileSystem for scheme hdfs” 场景）。连接器去 jar 后任意 TCCL 调用皆安全。 +- **⚠ 旧论断「TCCL 无需连接器侧处理…任意 TCCL 调用皆安全」已被 HIVEFS-9 证伪并修正**：`DFSFileSystem.getHadoopFs` 自钉 live TCCL 只修好 `FileSystem.get` 的 **ServiceLoader 发现**（挡 hive-exec 的 NullScanFileSystem），**修不了 conf 缓存的 classLoader 字段**——`Configuration.getClass`（如 `RPC.getProtocolEngine` 取 `ProtobufRpcEngine2`）只认 conf 自己的 `classLoader`，不认 live TCCL。引擎 FS 在连接器 TCCL 下惰性构造 conf 时捕获了连接器 loader → `ProtobufRpcEngine2`(连接器 hadoop-common)撞 `RpcEngine`(app引擎) ClassCast。**HIVEFS-9 已修**（`HdfsConfigBuilder.build` 补 `conf.setClassLoader(本模块loader)`，对齐全树 6 处 conf 构造点约定）。 - 写路径 MPU 需**具体** `ObjFileSystem`（`HiveConnectorTransaction` objCommit/abort）→ `FileSystem.forLocation`（HIVEFS-1 加、HIVEFS-6 已用：非切换返回 this、切换返回具体 FS）。**跨 连接器↔fs-plugin `instanceof ObjFileSystem` 安全**（`fe-filesystem-spi` 双 provided-scope + `FileSystemPluginManager:72` parent-first `"org.apache.doris.filesystem."` → 共享同一 `ObjFileSystem` Class；红队 HIVEFS-6 逐点取证）。 - 连接器裸 Hadoop **FileSystem I/O 面已全清**（读扫描 HIVEFS-4 + ACID HIVEFS-5 + 写路径 HIVEFS-6）：三条路径均经 `context.getFileSystem(session)` 借引擎 `SpiSwitchingFileSystem`；`hadoop-hdfs-client` 已核实不在依赖图（HIVEFS-7）。`FileStatus` 字段全映射 `FileEntry`（`name()`/`location().uri()`/`length()`/`modificationTime()`/`isDirectory()`）。 - fe-filesystem 全 9 插件（hdfs/s3/oss/cos/obs/azure/http/local/broker）已部署 → hive 表任意后端天然支持（免 bundle hadoop-aws/huaweicloud，架构红利）。 diff --git a/plan-doc/tasks/designs/FIX-HIVEFS-9-design.md b/plan-doc/tasks/designs/FIX-HIVEFS-9-design.md new file mode 100644 index 00000000000000..d8a0d9034edcac --- /dev/null +++ b/plan-doc/tasks/designs/FIX-HIVEFS-9-design.md @@ -0,0 +1,74 @@ +# FIX-HIVEFS-9 — engine `DFSFileSystem` conf must pin its own classloader (RpcEngine split-brain) + +> 状态：**DONE**（code ``）。发现于 HIVEFS-8 e2e（本地 hive 回归 `test_string_dict_filter` q01）。 +> 一句话：`HdfsConfigBuilder.build()` 是全树唯一**未钉 conf classLoader** 的 Hadoop-conf 构造点；在连接器插件 TCCL 下惰性构造引擎 FS 时，conf 捕获了连接器 loader → `RPC.getProtocolEngine` 经 `conf.getClass` 从连接器的 hadoop-common 副本加载 `ProtobufRpcEngine2`，与引擎侧 `RpcEngine`（app）撞类 → `ClassCastException`。 + +## 症状（HEAD 实测） + +`select * from test_string_dict_filter_parquet where o_orderstatus = 'F'`（读 hdfs 分区）q01 炸： + +``` +class org.apache.hadoop.ipc.ProtobufRpcEngine2 cannot be cast to class org.apache.hadoop.ipc.RpcEngine + (ProtobufRpcEngine2 is in unnamed module of loader ChildFirstClassLoader @5eea5627; + RpcEngine is in unnamed module of loader 'app') +``` + +FE 侧真实栈（`output/fe/log/fe.log`）： + +``` +PluginDrivenScanNode.onPluginClassLoader ← 扫描线程 TCCL 钉到 hive 连接器 loader @5eea5627 + → HiveScanPlanProvider.listAndSplitFiles （fe-connector-hive，插件） + → HiveFileListingCache.listFromFileSystem + → DFSFileSystem.list / getHadoopFs （fe-filesystem-hdfs，引擎 FS） + → FileSystem.get → createFileSystem → DistributedFileSystem.initialize + → RPC.getProtocolEngine(RPC.java:226) ← (RpcEngine) 强转在此 + → ClassCastException +``` + +## 根因 + +1. 翻闸后 hive 扫描全程在 `PluginDrivenScanNode.onPluginClassLoader` 内跑，**TCCL = hive 连接器 ChildFirstClassLoader `@5eea5627`**。 +2. HIVEFS-4 把列文件改走引擎 `context.getFileSystem(session)` → per-catalog `SpiSwitchingFileSystem` → 首次触碰 hdfs 路径时**惰性**构造 `DFSFileSystem`；其构造器 `this.conf = HdfsConfigBuilder.build(properties)` 里 `new HdfsConfiguration()` **捕获当时的 live TCCL** 作为 conf 自己的 `classLoader` 字段 = `@5eea5627`（连接器 loader）。 +3. `HdfsConfigBuilder.build()` **从不** `conf.setClassLoader(...)`（全树唯一漏钉的 conf 构造点）。 +4. `FileSystem.get` → `NameNodeProxiesClient` → `RPC.setProtocolEngine(conf, ClientNamenodeProtocolPB, ProtobufRpcEngine2)` 把引擎类**名**写进 conf；`RPC.getProtocolEngine` 再 `conf.getClass(prop, ProtobufRpcEngine2.class)` 按名回取。`Configuration.getClass` 用的是 **conf 自己的 `classLoader` 字段**（= `@5eea5627`），**不是** live TCCL。→ 从连接器的 hadoop-common 副本加载 `ProtobufRpcEngine2`。 +5. 而 `RPC`/`RpcEngine` 自身：`DFSFileSystem.getHadoopFs` 已把 TCCL 钉到 `DFSFileSystem.class.getClassLoader()`（fe-filesystem-hdfs 插件 loader），该 loader **只 bundle hadoop-hdfs、hadoop-common 委派回父 `app`**（症状里 `RpcEngine` 落在 `app` 即铁证）。→ `RpcEngine` = app 副本。 +6. 连接器副本的 `ProtobufRpcEngine2` 强转 app 副本的 `RpcEngine` → 撞类。 + +**要点**：`getHadoopFs` 钉 live TCCL 只修好了 `FileSystem.get` 的 **ServiceLoader 发现**（挡 hive-exec 的 NullScanFileSystem），**修不了 conf 缓存的 classLoader 字段**——`Configuration.getClass` 只认后者。这正是 `HmsConfHelper:51-58` 早已记载的同一机理（HMS metastore-client 路径），只是漏在了引擎 FS 这一处。 + +> ⚠ 更正 HANDOFF「已探明去风险事实」第 34 行的旧论断「TCCL 无需连接器侧处理…任意 TCCL 调用皆安全」——**该论断错**：`getHadoopFs` 的 TCCL 自钉不足以覆盖 conf 缓存 CL，须在 `build()` 显式钉。 + +## 修法（surgical，对齐既有 6 处约定） + +`HdfsConfigBuilder.build()`：`new HdfsConfiguration()` 后立即 +```java +conf.setClassLoader(HdfsConfigBuilder.class.getClassLoader()); +``` +镜像 `HmsConfHelper.createHiveConf:60` / `PaimonCatalogFactory:257,333` / `IcebergCatalogFactory:641,659,677` / `HiveConnector:639` / `HudiConnector:261`（全树本就是「conf 构造点无条件钉本模块 loader」的约定，`HdfsConfigBuilder` 是唯一漏网）。 + +**为何钉 `HdfsConfigBuilder.class.getClassLoader()` 正确**：它与 `getHadoopFs` 已钉的 TCCL 同一 loader（fe-filesystem-hdfs 插件 loader）；该 loader 对 hadoop-common 委派回 `app`（症状实证），故 `conf.getClass("...ProtobufRpcEngine2")` 经它解析 = app 副本 = 与 `RpcEngine`（app）同类 → 强转成功。conf 与框架类解析**同源**。 + +**为何放 `build()` 而非 `getHadoopFs`（scheme 条件钉）**：① 对齐约定（6 处均在 conf 构造点无条件钉）；② conf 是 per-`DFSFileSystem` 单例、`build()` 一次钉定、确定性（不随「先访问哪个 scheme」漂移）；③ 不触碰 `getHadoopFs` 对 non-hdfs 刻意保留调用方 TCCL 的 **ServiceLoader** 逻辑（本改只动 `conf.getClass` 的解析源，与 ServiceLoader/TCCL 正交）；④ bug 非 hdfs 独有——任何 scheme 走 Hadoop 反射类加载都该从本模块 loader 解析。 + +## 备选与否决 + +- **getHadoopFs 内按 `needPluginCL` 条件钉 conf**：更「外科」但把逻辑劈两处、conf.classLoader 随访问顺序漂移、且 conf 是共享单例并不能真正 scheme 隔离 → 否。 +- **钉 `Configuration.class.getClassLoader()`**：语义也对（直接绑 hadoop-common loader），但偏离全树「钉本类 loader」约定 → 从约定，否。 + +## Scope / 风险 + +- **scope = 引擎 fe-filesystem-hdfs 一处**；hive/paimon/iceberg/hudi/mc 连接器均不动。 +- non-hdfs（viewfs/ofs/jfs/oss）：本改把 `conf.getClass` 解析从「捕获的任意调用方 TCCL」改为「本插件 loader（→app 委派）」——是**严格改善**（DFSFileSystem 服务的 scheme，其 driver 本就在本插件/ app，不在任意连接器 loader）；ServiceLoader 发现路径不受影响（`getHadoopFs` TCCL 逻辑原样保留）。 +- Kerberos/UGI 用同一 conf：一并落到 app 的 hadoop-common（本就应如此），无回归。 + +## 验证 + +- `HdfsConfigBuilderTest.buildPinsPluginClassLoaderNotTccl`（新增，镜像 `PaimonCatalogFactoryTest.assembleHiveConfPinsPluginClassLoaderNotTccl`）：装一个 foreign `URLClassLoader` 当 TCCL → `build()` → 断言 `conf.getClassLoader()` == 本插件 loader、≠ foreign。 + - **RED 实证**（抽掉 `setClassLoader`）：`expected but was ` @ :88。 + - **GREEN**：`fe-filesystem-hdfs` **9/9**、`BUILD SUCCESS`、**0 checkstyle**。 +- ⚠ 单测环境是平铺 classpath，`HdfsConfigBuilder.class.getClassLoader()` = AppClassLoader；真跨 loader 强转只在真 child-first 插件环境复现（e2e）。 + +## 残余 / e2e（用户自跑，勿丢） + +- 重打包 + 重部署后：`external_table_p0/hive/test_string_dict_filter` q01 应越过本 ClassCast（读 hdfs 分区）；本 fix 只解**这一 classloader 阻断**，其后各断言/写套件仍按 HIVEFS-8 e2e 矩阵回归。 +- 本 fix 属 HIVEFS-8 收尾的一部分（引擎 FS 在连接器 TCCL 下的正确性补漏）。 From cd4eecac5e6e79ac9c7d8040a01e567ff7463d89 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 20:22:36 +0800 Subject: [PATCH 252/334] [fix](catalog) hive-text: decode numeric serde delimiters via getByte (read parity) HiveTextProperties returned raw SerDe delimiter strings instead of applying the legacy getByte() numeric-string->byte-char decode. Hive stores the default LazySimpleSerDe field delimiter as serialization.format="1" meaning byte 0x01 (Ctrl-A), not the character '1' (0x31). The connector emitted "1" as hive.text.column_separator, so the BE format_v2 TextReader split rows on '1' and every column collapsed into the first field -> null/merged reads across ~12 hive text/CSV/complex suites (test_hive_to_date, test_prepare_hive_data_in_case, test_hive_basic_type, test_hive_meta_cache, test_hive_text_complex_type, ...). Port getByte + the exact legacy defaults into HiveTextProperties and apply it to the field (non-multichar), line, mapkv, collection and escape delimiters; MultiDelimitSerDe field delimiters stay raw (multi-char). Also recognize the Hive2 misspelled collection key "colelction.delim" and mirror legacy getSerdeProperty's table-params-first lookup. null_format stays a raw literal (no getByte), per legacy. New HiveTextPropertiesTest (9/9) pins serialization.format="1"->0x01 plus numeric, multichar, table-params-precedence and hive2-typo cases. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- .../connector/hive/HiveTextProperties.java | 106 +++++++++++----- .../hive/HiveTextPropertiesTest.java | 114 ++++++++++++++++++ 2 files changed, 187 insertions(+), 33 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java index 4f69493f7c0ec8..f4a544cbc75aec 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java @@ -57,11 +57,23 @@ public final class HiveTextProperties { private static final String SERIALIZATION_FORMAT = "serialization.format"; private static final String LINE_DELIM = "line.delim"; private static final String MAPKEY_DELIM = "mapkey.delim"; + // Hive3 uses "collection.delim"; Hive2 uses the historically misspelled "colelction.delim". Legacy + // fe-core checks both, so parity requires recognizing both. private static final String COLLECTION_DELIM = "collection.delim"; + private static final String COLLECTION_DELIM_HIVE2 = "colelction.delim"; private static final String ESCAPE_DELIM = "escape.delim"; private static final String SERIALIZATION_NULL_FORMAT = "serialization.null.format"; private static final String SKIP_HEADER_LINE_COUNT = "skip.header.line.count"; + // Default delimiters, mirroring legacy HiveProperties. These are byte values (Ctrl-A etc.), not the + // literal digit characters Hive stores them as in serialization.format / *.delim SerDe params. + private static final String DEFAULT_FIELD_DELIM = "\001"; + private static final String DEFAULT_LINE_DELIM = "\n"; + private static final String DEFAULT_MAPKV_DELIM = "\003"; + private static final String DEFAULT_COLLECTION_DELIM = "\002"; + private static final String DEFAULT_ESCAPE_DELIM = "\\"; + private static final String DEFAULT_NULL_FORMAT = "\\N"; + // CSV SerDe property keys private static final String SEPARATOR_CHAR = "separatorChar"; private static final String QUOTE_CHAR = "quoteChar"; @@ -91,7 +103,7 @@ public static Map extract(String serDeLib, boolean multiDelimit = HIVE_MULTI_DELIMIT_SERDE.equals(serDeLib) || HIVE_MULTI_DELIMIT_SERDE_SERDE2.equals(serDeLib); if (HIVE_TEXT_SERDE.equals(serDeLib) || multiDelimit) { - extractTextSerDeProps(sdParams, result, multiDelimit); + extractTextSerDeProps(sdParams, tableParams, result, multiDelimit); } else if (HIVE_OPEN_CSV_SERDE.equals(serDeLib)) { extractCsvSerDeProps(sdParams, result); } else if (HIVE_JSON_SERDE.equals(serDeLib) || LEGACY_HIVE_JSON_SERDE.equals(serDeLib) @@ -108,25 +120,31 @@ public static Map extract(String serDeLib, return result; } - private static void extractTextSerDeProps(Map params, - Map result, boolean supportMultiChar) { - // Column separator - String fieldDelim = getFieldDelimiter(params, supportMultiChar); - result.put(PROP_PREFIX + "column_separator", fieldDelim); + private static void extractTextSerDeProps(Map sdParams, + Map tableParams, Map result, boolean supportMultiChar) { + // Column separator. Hive stores single-char delimiters as their numeric byte value (the default + // LazySimpleSerDe field delimiter is serialization.format="1" == byte 0x01, NOT the character + // '1'), so they must be decoded via getByte(). MultiDelimitSerDe keeps its raw multi-char value. + result.put(PROP_PREFIX + "column_separator", + getFieldDelimiter(sdParams, tableParams, supportMultiChar)); // Line delimiter - result.put(PROP_PREFIX + "line_delimiter", getLineDelimiter(params)); + result.put(PROP_PREFIX + "line_delimiter", + getByte(serdeVal(sdParams, tableParams, LINE_DELIM), DEFAULT_LINE_DELIM)); // MapKV delimiter - result.put(PROP_PREFIX + "mapkv_delimiter", getMapKvDelimiter(params)); - // Collection delimiter - result.put(PROP_PREFIX + "collection_delimiter", getCollectionDelimiter(params)); - // Escape delimiter - String escape = getParamOrDefault(params, ESCAPE_DELIM, null); - if (escape != null && !escape.isEmpty()) { - result.put(PROP_PREFIX + "escape", escape); + result.put(PROP_PREFIX + "mapkv_delimiter", + getByte(serdeVal(sdParams, tableParams, MAPKEY_DELIM), DEFAULT_MAPKV_DELIM)); + // Collection delimiter (Hive2 "colelction.delim" typo first, then Hive3 "collection.delim") + result.put(PROP_PREFIX + "collection_delimiter", + getByte(serdeVal(sdParams, tableParams, COLLECTION_DELIM_HIVE2, COLLECTION_DELIM), + DEFAULT_COLLECTION_DELIM)); + // Escape delimiter: emitted only when the SerDe sets it, decoded via getByte + String escape = serdeVal(sdParams, tableParams, ESCAPE_DELIM); + if (escape != null) { + result.put(PROP_PREFIX + "escape", getByte(escape, DEFAULT_ESCAPE_DELIM)); } - // Null format - result.put(PROP_PREFIX + "null_format", - getParamOrDefault(params, SERIALIZATION_NULL_FORMAT, "\\N")); + // Null format (raw string; NOT byte-decoded) + String nullFormat = serdeVal(sdParams, tableParams, SERIALIZATION_NULL_FORMAT); + result.put(PROP_PREFIX + "null_format", nullFormat != null ? nullFormat : DEFAULT_NULL_FORMAT); } private static void extractCsvSerDeProps(Map params, @@ -149,31 +167,53 @@ private static void extractJsonSerDeProps(String serDeLib, result.put(PROP_PREFIX + "json_serde_lib", serDeLib); } - private static String getFieldDelimiter(Map params, - boolean supportMultiChar) { - String delim = getParamOrDefault(params, FIELD_DELIM, null); - if (delim == null || delim.isEmpty()) { - delim = getParamOrDefault(params, SERIALIZATION_FORMAT, null); + private static String getFieldDelimiter(Map sdParams, + Map tableParams, boolean supportMultiChar) { + String delim = serdeVal(sdParams, tableParams, FIELD_DELIM, SERIALIZATION_FORMAT); + if (delim == null) { + delim = ""; } - if (delim == null || delim.isEmpty()) { - return "\001"; // Default Hive field delimiter (Ctrl-A) - } - if (!supportMultiChar && delim.length() == 1) { - return delim; - } - return delim; + // MultiDelimitSerDe delimiters may be multiple characters; keep them raw (no byte decode). + return supportMultiChar ? delim : getByte(delim, DEFAULT_FIELD_DELIM); } private static String getLineDelimiter(Map params) { return getParamOrDefault(params, LINE_DELIM, "\n"); } - private static String getMapKvDelimiter(Map params) { - return getParamOrDefault(params, MAPKEY_DELIM, "\003"); + /** + * Looks up a SerDe property mirroring legacy {@code HiveMetaStoreClientHelper.getSerdeProperty}: + * table parameters take precedence over StorageDescriptor/SerDeInfo parameters, and the keys are + * tried in order. An empty string counts as present. Returns {@code null} if no key is set. + */ + private static String serdeVal(Map sdParams, Map tableParams, + String... keys) { + for (String key : keys) { + if (tableParams != null && tableParams.get(key) != null) { + return tableParams.get(key); + } + if (sdParams != null && sdParams.get(key) != null) { + return sdParams.get(key); + } + } + return null; } - private static String getCollectionDelimiter(Map params) { - return getParamOrDefault(params, COLLECTION_DELIM, "\002"); + /** + * Decodes a Hive delimiter. Hive stores single-char delimiters as their numeric byte value + * ("1" == 0x01, "9" == 0x09); a non-numeric value is taken literally and truncated to its first + * character; an empty/absent value falls back to {@code defValue}. Mirrors legacy + * {@code HiveMetaStoreClientHelper.getByte}. + */ + private static String getByte(String altValue, String defValue) { + if (altValue != null && !altValue.isEmpty()) { + try { + return Character.toString((char) ((Byte.parseByte(altValue) + 256) % 256)); + } catch (NumberFormatException e) { + return altValue.substring(0, 1); + } + } + return defValue; } private static int getSkipHeaderCount(Map tableParams) { diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java new file mode 100644 index 00000000000000..280db47c8f5c1c --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java @@ -0,0 +1,114 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests {@link HiveTextProperties} delimiter extraction, pinned to legacy fe-core parity + * ({@code HiveProperties} + {@code HiveMetaStoreClientHelper.getByte}). + * + *

    WHY these assertions matter: Hive stores text delimiters as NUMERIC BYTE-VALUE STRINGS + * in SerDe params. The canonical case is a default {@code LazySimpleSerDe} table, whose field + * delimiter is stored as {@code serialization.format=1} — meaning byte {@code 0x01} (Ctrl-A), + * NOT the character {@code '1'} (0x31). Legacy applies {@code getByte()} to decode the numeric + * string into the real delimiter byte; the connector must reproduce this exactly, or BE splits + * text rows on the wrong character and every column collapses to null/merged.

    + */ +public class HiveTextPropertiesTest { + + private static final String TEXT_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; + private static final String MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; + private static final String PREFIX = "hive.text."; + + private static String colSep(String serde, Map sd) { + return HiveTextProperties.extract(serde, sd, new HashMap<>()).get(PREFIX + "column_separator"); + } + + private static Map sd(String... kv) { + Map m = new HashMap<>(); + for (int i = 0; i < kv.length; i += 2) { + m.put(kv[i], kv[i + 1]); + } + return m; + } + + @Test + public void testDefaultSerializationFormatDecodesToCtrlA() { + // Default LazySimpleSerDe: Hive stores serialization.format="1" == byte 0x01, no field.delim. + Assertions.assertEquals("\001", colSep(TEXT_SERDE, sd("serialization.format", "1"))); + } + + @Test + public void testNoDelimiterFallsBackToDefaultCtrlA() { + Assertions.assertEquals("\001", colSep(TEXT_SERDE, sd())); + } + + @Test + public void testLiteralFieldDelimiterPreserved() { + // Explicit "FIELDS TERMINATED BY ','": Hive stores the literal comma; getByte keeps it. + Assertions.assertEquals(",", colSep(TEXT_SERDE, sd("field.delim", ","))); + } + + @Test + public void testNumericFieldDelimiterDecodesToByte() { + // field.delim="9" means byte 0x09 (tab), not the character '9'. + Assertions.assertEquals("\t", colSep(TEXT_SERDE, sd("field.delim", "9"))); + } + + @Test + public void testFieldDelimTakesPrecedenceOverSerializationFormat() { + Assertions.assertEquals(",", colSep(TEXT_SERDE, sd("field.delim", ",", "serialization.format", "1"))); + } + + @Test + public void testMultiDelimitSerDeKeepsRawMultiCharDelimiter() { + // MultiDelimitSerDe supports multi-character delimiters; they must NOT be byte-decoded/truncated. + Assertions.assertEquals("||", colSep(MULTI_DELIMIT_SERDE, sd("field.delim", "||"))); + } + + @Test + public void testCollectionDelimiterDefaultAndHive2Typo() { + Map r = HiveTextProperties.extract(TEXT_SERDE, sd(), new HashMap<>()); + Assertions.assertEquals("\002", r.get(PREFIX + "collection_delimiter")); + // Hive2 uses the famously misspelled key "colelction.delim"; "2" decodes to byte 0x02. + Map r2 = HiveTextProperties.extract(TEXT_SERDE, sd("colelction.delim", "2"), new HashMap<>()); + Assertions.assertEquals("\002", r2.get(PREFIX + "collection_delimiter")); + } + + @Test + public void testMapkvAndLineDefaults() { + Map r = HiveTextProperties.extract(TEXT_SERDE, sd(), new HashMap<>()); + Assertions.assertEquals("\003", r.get(PREFIX + "mapkv_delimiter")); + Assertions.assertEquals("\n", r.get(PREFIX + "line_delimiter")); + } + + @Test + public void testTableParamsTakePrecedenceOverSerdeParams() { + // Legacy getSerdeProperty checks table params first, then serde params. + Map sdParams = sd("serialization.format", "1"); + Map tblParams = new HashMap<>(); + tblParams.put("field.delim", ","); + String sep = HiveTextProperties.extract(TEXT_SERDE, sdParams, tblParams).get(PREFIX + "column_separator"); + Assertions.assertEquals(",", sep); + } +} From 5672d7c02091ae4353b57a7d1a5f7cd71cc59632 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 20:22:36 +0800 Subject: [PATCH 253/334] [fix](catalog) hive: read enable.mapping.varbinary/timestamp_tz catalog keys buildTypeMappingOptions read HiveConnectorProperties.ENABLE_MAPPING_BINARY_AS_STRING = "enable_mapping_binary_as_string" and ENABLE_MAPPING_TIMESTAMP_TZ = "enable_mapping_timestamp_tz" -- underscore keys that ExternalCatalog never writes. The real catalog properties are the dot-form enable.mapping.varbinary / enable.mapping.timestamp_tz (CatalogProperty, matching the iceberg/paimon connectors). Because the underscore keys were always absent, mapBinaryToVarbinary and the timestamptz toggle were permanently false: hive BINARY always mapped to STRING (rendered as a decoded string instead of 0x hex) and hive timestamps never mapped to TIMESTAMPTZ. Rename the constants to the correct dot keys and de-invert the local variable (binaryAsString -> enableMappingVarbinary). Restores parity with old fe-core hiveTypeToDorisType and the iceberg/paimon connectors. Fixes test_hive_orc and test_hive_varbinary_type. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- .../apache/doris/connector/hive/HiveConnectorMetadata.java | 6 +++--- .../doris/connector/hive/HiveConnectorProperties.java | 7 +++++-- 2 files changed, 8 insertions(+), 5 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 7bd7f53012777f..e811904f8720cc 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1971,12 +1971,12 @@ private static boolean firstColumnIsString(HmsTableInfo tableInfo) { } private static HmsTypeMapping.Options buildTypeMappingOptions(Map props) { - boolean binaryAsString = Boolean.parseBoolean( - props.getOrDefault(HiveConnectorProperties.ENABLE_MAPPING_BINARY_AS_STRING, "false")); + boolean enableMappingVarbinary = Boolean.parseBoolean( + props.getOrDefault(HiveConnectorProperties.ENABLE_MAPPING_VARBINARY, "false")); boolean timestampTz = Boolean.parseBoolean( props.getOrDefault(HiveConnectorProperties.ENABLE_MAPPING_TIMESTAMP_TZ, "false")); return new HmsTypeMapping.Options( - HmsTypeMapping.DEFAULT_TIME_SCALE, binaryAsString, timestampTz); + HmsTypeMapping.DEFAULT_TIME_SCALE, enableMappingVarbinary, timestampTz); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java index 07c47a7c90c4ab..ddb2273f6e9bd4 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java @@ -53,8 +53,11 @@ private HiveConnectorProperties() { public static final String FLINK_CONNECTOR = "connector"; // -- type mapping options -- - public static final String ENABLE_MAPPING_BINARY_AS_STRING = "enable_mapping_binary_as_string"; - public static final String ENABLE_MAPPING_TIMESTAMP_TZ = "enable_mapping_timestamp_tz"; + // Catalog-level property keys (dot form), matching CatalogProperty and the iceberg/paimon connectors. + // ExternalCatalog forwards these two keys to the connector; the earlier underscore spellings were never + // populated, so the toggles silently no-op'd (hive BINARY always STRING, timestamp never TIMESTAMPTZ). + public static final String ENABLE_MAPPING_VARBINARY = "enable.mapping.varbinary"; + public static final String ENABLE_MAPPING_TIMESTAMP_TZ = "enable.mapping.timestamp_tz"; // -- CREATE TABLE / DATABASE property keys (legacy HiveMetadataOps) -- public static final String CREATE_FILE_FORMAT = "file_format"; From d8c6eb002a11b55604c206d5ddee8079664219cd Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 20:22:36 +0800 Subject: [PATCH 254/334] [fix](catalog) hive: mark external columns isKey=true (DESC parity) ThriftHmsClient.convertFieldSchemas built each ConnectorColumn with the 5-arg constructor, which defaults isKey=false, so DESC on an external hive table/view reported Key=false where legacy HMSExternalTable (and the iceberg connector) report Key=true for external columns. Use the 6-arg constructor with isKey=true. Fixes test_hive_struct_add_column and test_hive_view. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- .../java/org/apache/doris/connector/hms/ThriftHmsClient.java | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 99e17f118e7408..1c19541e679ac8 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -694,8 +694,10 @@ private List convertFieldSchemas( for (FieldSchema fs : schemas) { ConnectorType type = HmsTypeMapping.toConnectorType( fs.getType(), typeMappingOptions); + // isKey=true: external-table semantics (legacy HMSExternalTable and the iceberg connector both + // mark external columns as key so DESC shows Key=true). The 5-arg ctor defaults isKey=false. result.add(new ConnectorColumn( - fs.getName(), type, fs.getComment(), true, null)); + fs.getName(), type, fs.getComment(), true, null, true)); } return result; } From b036c60465177b3deb6a2673ed2a06a8ff63ee51 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 20:22:36 +0800 Subject: [PATCH 255/334] [test](catalog) hive: align stale DDL/transactional error messages to connector Post-cutover DDL errors are produced by the plugin connector with new (semantically equivalent) wording; update the stale expectations rather than reconstructing the old text in the connector: - create-database conflict: "Can't create database 'X'; database exists" -> "Failed to create Hive database ...: ... already exists". - drop-database non-existent: "Can't drop database 'X'; database doesn't exist" -> "Failed to get database: 'X'" (PluginDrivenExternalCatalog.dropDb; code-traced, confirm on rerun). - transactional-table INSERT reject: "Not supported insert into hive transactional table." -> "Cannot write to a transactional Hive table". Create-table-conflict and other fe-core-level messages are unchanged and left as-is. Suites: test_hive_case_sensibility, ddl/test_hive_write_type, ddl/test_hive_ctas, test_transactional_hive. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- .../hive/ddl/test_hive_ctas.groovy | 8 ++++--- .../hive/ddl/test_hive_write_type.groovy | 3 ++- .../hive/test_hive_case_sensibility.groovy | 23 ++++++++----------- .../hive/test_transactional_hive.groovy | 2 +- 4 files changed, 18 insertions(+), 18 deletions(-) diff --git a/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy b/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy index 64c127c030dab4..7a891df556ba62 100644 --- a/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy +++ b/regression-test/suites/external_table_p0/hive/ddl/test_hive_ctas.groovy @@ -340,7 +340,8 @@ suite("test_hive_ctas", "p0,external") { sql """ create database if not exists `test_hive_ex_ctas` """; test { sql """ create database `test_hive_ex_ctas` """ - exception "errCode = 2, detailMessage = Can't create database 'test_hive_ex_ctas'; database exists" + exception "Failed to create Hive database test_hive_ex_ctas" + exception "already exists" } sql """use `${catalog_name}`.`test_hive_ex_ctas`""" sql """ DROP DATABASE IF EXISTS ${catalog_name}.test_hive_ex_ctas """ @@ -349,7 +350,7 @@ suite("test_hive_ctas", "p0,external") { try { test { sql """ DROP DATABASE ${catalog_name}.test_no_exist """ - exception "errCode = 2, detailMessage = Can't drop database 'test_no_exist'; database doesn't exist" + exception "Failed to get database: 'test_no_exist'" } sql """ DROP DATABASE IF EXISTS ${catalog_name}.test_err """ sql """ CREATE DATABASE ${catalog_name}.test_err """ @@ -360,7 +361,8 @@ suite("test_hive_ctas", "p0,external") { "owner" = "err" ) """; - exception "errCode = 2, detailMessage = Can't create database 'test_err'; database exists" + exception "Failed to create Hive database test_err" + exception "already exists" } sql """ DROP DATABASE IF EXISTS ${catalog_name}.test_err """ diff --git a/regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy b/regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy index 9504f5fca2a16b..da607686cd2394 100644 --- a/regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy +++ b/regression-test/suites/external_table_p0/hive/ddl/test_hive_write_type.groovy @@ -163,7 +163,8 @@ suite("test_hive_write_type", "p0,external") { sql """ create database if not exists `test_hive_ex` """; test { sql """ create database `test_hive_ex` """ - exception "errCode = 2, detailMessage = Can't create database 'test_hive_ex'; database exists" + exception "Failed to create Hive database test_hive_ex" + exception "already exists" } sql """use `${catalog_name}`.`test_hive_ex`""" diff --git a/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy b/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy index ba217b53c3ca50..84679d10fa16f5 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy @@ -46,7 +46,8 @@ suite("test_hive_case_sensibility", "p0,external") { sql """create database case_db1;""" test { sql """create database CASE_DB1;""" // conflict - exception "Can't create database 'CASE_DB1'; database exists" + exception "Failed to create Hive database" + exception "already exists" } sql """create database CASE_DB2;""" sql """create database if not exists CASE_DB1;""" @@ -58,19 +59,18 @@ suite("test_hive_case_sensibility", "p0,external") { qt_sql3 """show databases like "%case_db2%";""" test { sql """create database CASE_DB2;""" // conflict - exception "database exists" - exception "CASE_DB2" + exception "Failed to create Hive database" + exception "already exists" } test { sql """create database case_db2;""" // conflict - exception "database exists" - exception "case_db2" + exception "Failed to create Hive database" + exception "already exists" } // 2. drop database test { sql """drop database CASE_DB1""" - exception "database doesn't exist" - exception "CASE_DB1" + exception "Failed to get database" } sql """drop database if exists CASE_DB1;""" qt_sql4 """show databases like "%case_db1%";""" // still exists @@ -79,19 +79,16 @@ suite("test_hive_case_sensibility", "p0,external") { test { sql """drop database CASE_DB2;""" - exception "database doesn't exist" - exception "CASE_DB2" + exception "Failed to get database" } sql """drop database case_db2;""" test { sql """drop database case_db1""" - exception "database doesn't exist" - exception "case_db1" + exception "Failed to get database" } test { sql """drop database case_db2""" - exception "database doesn't exist" - exception "case_db2" + exception "Failed to get database" } sql """drop database if exists case_db2;""" qt_sql6 """show databases like "%case_db1%";""" // empty diff --git a/regression-test/suites/external_table_p0/hive/test_transactional_hive.groovy b/regression-test/suites/external_table_p0/hive/test_transactional_hive.groovy index c11d9165063ae9..044e90368e6bdf 100644 --- a/regression-test/suites/external_table_p0/hive/test_transactional_hive.groovy +++ b/regression-test/suites/external_table_p0/hive/test_transactional_hive.groovy @@ -123,7 +123,7 @@ suite("test_transactional_hive", "p0,external") { try { sql """ insert into orc_acid_major(id,value) values(1,"a1"); """ }catch (Exception e) { - assertTrue(e.getMessage().contains("Not supported insert into hive transactional table.")); + assertTrue(e.getMessage().contains("Cannot write to a transactional Hive table")); } try { From 20513d35a84aeb82cec4aea8af7014df26acb692 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 20:25:43 +0800 Subject: [PATCH 256/334] [doc](catalog) hive-fs: hand off HIVEFS-8 e2e first-run results (37/111) Record the first external_table_p0/hive e2e run: 4 fixes landed this round (getByte delimiter decode, varbinary/timestamptz mapping keys, external isKey=true, DDL/transactional message alignments; ~20 suites) and a triaged backlog of ~12 remaining real connector/fe-core regressions (R1-R12) with file pointers, plus the environment assessment (no genuine non-idempotency). Demotes the completed HIVEFS-0..9 block to background. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_df3afd31-3aa7-47d8-b09f-aa87db2c8509 --- plan-doc/HANDOFF.md | 36 +++++++++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 225f758c47936e..6f25851d92d0d3 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,41 @@ --- -# 🆕 下一个 session 起步 = FIX-HIVEFS 收尾（下一步 = 全量 build + e2e 交接 HIVEFS-8） +# 🆕 下一个 session 起步 = 修复 hive e2e 剩余 regression（HIVEFS-8 e2e 已首跑） + +> **本轮**：`external_table_p0/hive` 全量回归首跑 = **37/111 失败**。已逐个定位根因；**本轮修复约 20 个 suite（4 commit，下）**；**剩余 ~12 个真实 connector/fe-core regression 待修**（均**非**测试/环境问题，**勿 rebless `.out`**）。 +> **起步必读**：本段 + `git log`（4 fix commit 详尽）。失败日志（勿 commit，非本线程产物）：`nohup.out`（Jul 11）、FE `output/fe/log`、BE `output/be/log`。**BE 走 `format_v2` 扫描器**。**行号信 HEAD 不信文档。** +> **⚠ e2e 欠账**：本轮 4 修均 unit/build 绿（fe-connector-hive+hms 920 测试、0 checkstyle），**真值靠重部署+重跑断言**（尤其 ① drop-nonexistent 新文案 `"Failed to get database"` 是读代码推断非日志实证；② text 读回、binary hex 渲染）。**用户将先自跑一轮验证再开新 session 续修剩余。** + +**✅ 本轮 DONE（4 commit，均编译+全套测试绿）**： +- `cd4eecac5e6` **[fix] 文本读取根因（~12 suite）**：`HiveTextProperties`（fe-connector-hive）未做 `getByte` 数值→字节解码。Hive 默认分隔符存 `serialization.format="1"`=**字节 0x01**(Ctrl-A)，新代码当字符 `'1'`(0x31) 发 BE → `format_v2` TextReader 按 '1' 切 → 整行挤第一列 → null/粘连。移植 getByte + 老默认 + Hive2 拼写 key `colelction.delim` + table-params 优先；MultiDelimitSerDe 多字符保持 raw；null_format 不解码。新单测 `HiveTextPropertiesTest` 9/9。覆盖 to_date/prepare_data/basic_type/other/serde_prop/truncate/get_schema/meta_cache/query_cache + text_complex_type/to_array/text_garbled_file(靠 colelction.delim)。 +- `5672d7c0209` **[fix] binary/timestamp 映射 key**：连接器读走样下划线 key `enable_mapping_binary_as_string`/`enable_mapping_timestamp_tz`（全树从不写入），正确点号 `enable.mapping.varbinary`/`enable.mapping.timestamp_tz`（`CatalogProperty`/iceberg/paimon）→ BINARY 恒退 STRING、timestamp 恒不 TIMESTAMPTZ。修 `test_hive_orc`/`test_hive_varbinary_type`。 +- `d8c6eb002a1` **[fix] desc Key=true**：`ThriftHmsClient.convertFieldSchemas`（fe-connector-hms）用 5 参 `ConnectorColumn`(isKey 默认 false)→改 6 参 isKey=true（外表语义，对齐 iceberg）。修 `test_hive_struct_add_column`/`test_hive_view`。 +- `b036c604651` **[test] DDL/事务文案对齐**：create-db-conflict→`"Failed to create Hive database…already exists"`、drop-db-nonexistent→`"Failed to get database:'X'"`(读代码推断)、事务 insert→`"Cannot write to a transactional Hive table"`；create-table-conflict(fe-core 层)未变、未动。 + +**⭐ 下一步 = 修剩余 ~12 真实 regression**（每条：设计→(红队)→实现→靶向验证→独立 commit；勿 rebless `.out`）： + +| # | 缺陷 | 影响 suite | 修复位置指针 | 规模 | +|---|---|---|---|---| +| R1 | SHOW CREATE DATABASE 丢 LOCATION：`HiveConnectorMetadata` 无 `getDatabase` override | `test_hive_ddl`, `test_hms_event_notification`(+`_multi_catalog`) | 加 `getDatabase` 返 HMS db location 于 `ConnectorDatabaseMetadata.LOCATION_PROPERTY`，镜像 `IcebergConnectorMetadata.getDatabase:187` | 中 | +| R2 | SHOW CREATE TABLE 出通用 DDL：`ShowCreateTableCommand:155-159` 门控 `getType()==HMS_EXTERNAL_TABLE`，插件表是 PLUGIN_EXTERNAL_TABLE | `test_hive_show_create_table`, `test_hive_ddl_text_format`(次) | fe-core 路由 plugin-hive→hive DDL 渲染器（守 connector-agnostic 铁律；参 view 臂 :161-169） | 中 | +| R3 | `$partitions` 系统表未支持（"Unknown sys table"） | `test_hive_partition_values_tvf` | 连接器/`PluginDrivenExternalTable` sys-table dispatch | 大 | +| R4 | explain `partition=N/M`=0/0 | `test_hive_default_partition` | `PluginDrivenScanNode` explain 分区计数 + hive `appendExplainInfo` | 大 | +| R5 | 统计行数未接 cardinality（=1 非 66） | `test_hive_statistics_p0` | 连接器 stats 取数 + PluginDrivenScanNode cardinality 接线 | 大 | +| R6 | DECIMAL 分区谓词裁剪→0 行 | `test_hive_partitions`(q03+) | 连接器分区值归一/比较 | 中 | +| R7 | `use_meta_cache` 下 show partitions 空 | `test_hive_use_meta_cache_true`(sql09) | `fe-connector-hms` CachingHmsClient 分区列表 | 中 | +| R8 | `hive.recursive_directories` 未生效（只读顶层）| `hive_config_test`(qt_2) | 连接器文件列举递归 | 中 | +| R9 | LZO 文本 + `.lzo.index` sidecar 排除 | `test_hive_lzo_text_format` | 连接器 LZO input-format 分类+压缩解码+排 .index | 大 | +| R10 | openx JSON serde 接线缺失（ignore.malformed.json / read_hive_json_in_one_column）| `test_hive_openx_json` | `HiveTextProperties.extractJsonSerDeProps`(现仅收 serDeLib) + PluginDrivenScanNode json 属性 | 大 | +| R11 | 特殊字符分区值路径编解码 | `test_hive_special_char_partition` | 连接器分区值 escape/unescape + 写 | 中 | +| R12 | OpenCSV 另有原因（FE 路径老新等价，待定位）| `test_open_csv_serde`(csv_complex_type) | 待查（enclose/escape/复杂类型）| 待查 | + +**环境结论**：**无真幂等问题**；日志 `already exists`/`is not empty`/`does not exist` = 新连接器 DDL 文案包装 + 故意负测。**卫生**：个别 F suite `DROP DATABASE` 无 CASCADE，中断会留残库 → 整轮间重置外部 hive docker。 +**净室复审底稿**：本轮全 triage 由 7-agent 对抗 workflow 打底（根因 CONFIRMED、逐 suite 归因），结论可复用。 + +--- + +# ✅ HIVEFS 背景（HIVEFS-0..9 全 DONE；本轮已跑 e2e，以下为历史参考） > 本地 hive 回归 `test_string_dict_filter` q01 `No FileSystem for scheme "hdfs"` 引出的**架构性改造**（非环境/文案）。**起步必读：设计 `plan-doc/tasks/designs/FIX-HIVEFS-design.md` + 任务清单 `plan-doc/tasks/task-list-HIVEFS.md`（行号信 HEAD 不信文档）。连接器读+ACID+写三条裸 Hadoop 路径已全部转引擎注入 FileSystem（HIVEFS-4/5/6 DONE），去 jar 已核实达成（HIVEFS-7 DONE，无需改 pom）；HIVEFS-8 e2e 首跑引出引擎 conf 的 classloader 阻断已修（**HIVEFS-9 DONE** `37911f2b3fe`，`test_string_dict_filter` q01 的 `ProtobufRpcEngine2 cannot be cast to RpcEngine`）；只剩全量 build + 重部署 + e2e 自跑。** From 3be286517b5983a6608fbac2a5c380074cb1ff45 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 21:21:04 +0800 Subject: [PATCH 257/334] [fix](catalog) hive: restore db-LOCATION / decimal+special-char prune / binary-mapping read parity Round-2 e2e triage: four fe-connector-hive read-path parity regressions (HiveConnectorMetadata, plus HiveConnector for the client wiring): - getDatabase override (R1): metadata had no getDatabase, so SHOW CREATE DATABASE for an hms catalog fell to the ConnectorSchemaOps default (empty props), emitting no LOCATION -> the test's indexOf("hdfs://")=-1 substring crashed. Mirror IcebergConnectorMetadata: surface hmsClient.getDatabase(dbName).getLocationUri() under LOCATION_PROPERTY (blank omitted). Fixes test_hive_ddl, test_hms_event_notification(+_multi_catalog). - DECIMAL partition prune (R6): extractLiteralValue rendered a BigDecimal literal via String.valueOf -> "1.0000", never string-equal to the Hive-canonical stored dir value "1", pruning every partition to 0 rows. Add a BigDecimal branch -> stripTrailingZeros().toPlainString(). Fixes test_hive_partitions. - special-char partition key (R11): parsePartitionName unescaped the value but stored the still- escaped column NAME as the map key; Hive's makePartName escapes special-char column names too (e.g. `pt2=x!!!! **1+1/&^%3`), so matchesPredicates' lookup by the real name missed and dropped every row. Unescape the key too (no-op for plain names). Fixes test_hive_special_char_partition. - binary mapping (orc): commit 5672d7c0209 read enable.mapping.varbinary into a metadata field that is never used -- the live schema conversion runs on the ThriftHmsClient's own options, and HiveConnector.createClient built it with the 2-arg ctor (Options.DEFAULT), so hive BINARY always mapped to STRING despite the catalog toggle. Build the options at client construction (3-arg ctor) and drop the dead metadata field. Fixes test_hive_orc. fe-connector-hive test-compile + 298 UT green, 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/connector/hive/HiveConnector.java | 8 +++- .../connector/hive/HiveConnectorMetadata.java | 41 ++++++++++++++++--- 2 files changed, 43 insertions(+), 6 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java index 02b8148a5f3908..25011d8d95a96b 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnector.java @@ -532,7 +532,13 @@ public T execute(Callable callable) throws Exception { } else { authAction = context::executeAuthenticated; } - return wrapWithCache(new ThriftHmsClient(config, authAction)); + // Feed the catalog's type-mapping options (enable.mapping.varbinary / enable.mapping.timestamp_tz) to + // the LIVE client: ThriftHmsClient.convertFieldSchemas maps hive column types with the client's own + // options, so the 2-arg ctor (HmsTypeMapping.Options.DEFAULT) would ignore the catalog toggles and + // always map hive BINARY -> STRING / timestamp -> non-TZ. Commit 5672d7c0209 read the dot-keys but only + // into a dead metadata field; the fix is to build the options here where the client is constructed. + return wrapWithCache(new ThriftHmsClient(config, authAction, + HiveConnectorMetadata.buildTypeMappingOptions(properties))); } /** diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index e811904f8720cc..412b00c67a27ff 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.ConnectorCapability; import org.apache.doris.connector.api.ConnectorColumn; import org.apache.doris.connector.api.ConnectorColumnStatistics; +import org.apache.doris.connector.api.ConnectorDatabaseMetadata; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorPartitionInfo; import org.apache.doris.connector.api.ConnectorSession; @@ -73,6 +74,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.math.BigDecimal; import java.nio.charset.StandardCharsets; import java.time.LocalDateTime; import java.time.format.DateTimeFormatter; @@ -194,7 +196,6 @@ public class HiveConnectorMetadata implements ConnectorMetadata { private final HmsClient hmsClient; private final Map properties; - private final HmsTypeMapping.Options typeMappingOptions; // Carries the fe-core-injected environment (getEnvironment()) with the FE-global CREATE TABLE defaults // (hive_default_file_format / enable_create_hive_bucket_table / doris_version) that the plugin cannot // read from FE Config. The default getEnvironment() is an empty map, so direct-construction tests that @@ -242,7 +243,6 @@ public HiveConnectorMetadata(HmsClient hmsClient, Map properties this.hmsClient = hmsClient; this.properties = properties; this.context = context; - this.typeMappingOptions = buildTypeMappingOptions(properties); this.icebergSiblingSupplier = icebergSiblingSupplier; this.hudiSiblingSupplier = hudiSiblingSupplier; this.siblingOwnerResolver = siblingOwnerResolver; @@ -309,6 +309,22 @@ public boolean databaseExists(ConnectorSession session, String dbName) { } } + @Override + public ConnectorDatabaseMetadata getDatabase(ConnectorSession session, String dbName) { + // Surface the HMS database base location for SHOW CREATE DATABASE under the neutral "location" + // property key (Trino-aligned properties-map model). Mirrors IcebergConnectorMetadata.getDatabase + // (and legacy HMSExternalCatalog which emitted LOCATION via db.getLocationUri()). Without this + // override the ConnectorSchemaOps default returns an empty property map, so SHOW CREATE DATABASE + // rendered no LOCATION clause. The key is omitted when blank so a location-less namespace renders + // no LOCATION rather than LOCATION ''. The hmsClient is already auth-wrapped (see databaseExists). + Map props = new HashMap<>(); + String location = hmsClient.getDatabase(dbName).getLocationUri(); + if (location != null && !location.isEmpty()) { + props.put(ConnectorDatabaseMetadata.LOCATION_PROPERTY, location); + } + return new ConnectorDatabaseMetadata(dbName, props); + } + // ========== ConnectorTableOps ========== @Override @@ -1970,7 +1986,11 @@ private static boolean firstColumnIsString(HmsTableInfo tableInfo) { return "STRING".equals(columns.get(0).getType().getTypeName()); } - private static HmsTypeMapping.Options buildTypeMappingOptions(Map props) { + // Package-private: HiveConnector.createClient builds the LIVE ThriftHmsClient's type-mapping options from + // the catalog properties (enable.mapping.varbinary / enable.mapping.timestamp_tz). The client — not this + // metadata — converts hive column types (ThriftHmsClient.convertFieldSchemas), so the options must be fed at + // client construction; a metadata-local copy would be dead (that was the 5672d7c0209 gap). + static HmsTypeMapping.Options buildTypeMappingOptions(Map props) { boolean enableMappingVarbinary = Boolean.parseBoolean( props.getOrDefault(HiveConnectorProperties.ENABLE_MAPPING_VARBINARY, "false")); boolean timestampTz = Boolean.parseBoolean( @@ -2048,6 +2068,14 @@ private String extractLiteralValue(ConnectorExpression expr) { // matchesPredicates -> the whole table prunes to 0 rows. Render Hive-canonical text instead. return hiveDateTimeString((LocalDateTime) val); } + if (val instanceof BigDecimal) { + // A DECIMAL partition literal arrives as a BigDecimal carrying the column's declared scale + // (e.g. decimal(8,4) -> "1.0000"), but the Hive-canonical stored partition value is trailing-zero + // trimmed ("1"). String.valueOf keeps the scale, so matchesPredicates' string-compare misses and + // the table prunes to 0 rows. Render trailing-zero-trimmed plain text to string-equal the stored + // value (toPlainString avoids scientific notation from stripTrailingZeros). + return ((BigDecimal) val).stripTrailingZeros().toPlainString(); + } return String.valueOf(val); } @@ -2101,8 +2129,11 @@ static Map parsePartitionName(String partName, // The predicate literal side (extractLiteralValue) is unescaped, so matchesPredicates' string // compare needs the value unescaped too — otherwise an escaped partition value silently drops // rows. Mirrors the sibling partition-value parse (HiveWriteUtils.toPartitionValues) and legacy - // FileUtils.unescapePathName. The key is a column name (never escaped), left as-is. - values.put(part.substring(0, eq), + // FileUtils.unescapePathName. The KEY must be unescaped too: Hive's makePartName escapes the + // column name as well (a special-char partition column such as `pt2=x!!!! **1+1/&^%3` comes + // back as an escaped key), and matchesPredicates looks it up by the real unescaped column name, + // so an escaped key would silently miss and drop every row. Unescaping a plain name is a no-op. + values.put(HiveWriteUtils.unescapePathName(part.substring(0, eq)), HiveWriteUtils.unescapePathName(part.substring(eq + 1))); } } From 70c1116dad7de017fbcea27a1fd101c20c46b757 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 21:21:23 +0800 Subject: [PATCH 258/334] [fix](catalog) hive: validate meta-cache ttl-second at catalog create After the hms cutover an "hms" catalog is created via HiveConnectorProvider, not HMSExternalCatalog, so the legacy checkProperties fail-fast for file.meta.cache.ttl-second / partition.cache.ttl-second no longer ran and an invalid ttl (e.g. -2) was silently accepted. Add validateProperties (mirror IcebergConnectorProvider) using CacheSpec.checkLongProperty with min 0 (legacy CACHE_TTL_DISABLE_CACHE semantics); it emits the identical "The parameter ... is wrong, value is ..." message. Fixes test_hive_meta_cache. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveConnectorProvider.java | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProvider.java index 924ffa879464d1..c575b695126437 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProvider.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hive; import org.apache.doris.connector.api.Connector; +import org.apache.doris.connector.cache.CacheSpec; import org.apache.doris.connector.spi.ConnectorContext; import org.apache.doris.connector.spi.ConnectorProvider; @@ -40,4 +41,17 @@ public String getType() { public Connector create(Map properties, ConnectorContext context) { return new HiveConnector(properties, context); } + + @Override + public void validateProperties(Map properties) { + // Restore the legacy HMSExternalCatalog.checkProperties fail-fast for the two meta-cache TTL knobs: + // after the hms cutover an "hms" catalog is created via this SPI provider (not HMSExternalCatalog), so + // the old per-property validation no longer runs and an invalid ttl (e.g. -2) was silently accepted. + // Legacy semantics: the value, when present, must be a long >= 0 (CACHE_TTL_DISABLE_CACHE); < 0 is + // rejected. checkLongProperty emits the identical "The parameter ... is wrong, value is ..." message. + CacheSpec.checkLongProperty(properties.get("file.meta.cache.ttl-second"), + 0L, "file.meta.cache.ttl-second"); + CacheSpec.checkLongProperty(properties.get("partition.cache.ttl-second"), + 0L, "partition.cache.ttl-second"); + } } From e0e94eeb20f0b8aec73de7d4e02a6a8048c5b4b2 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 21:21:24 +0800 Subject: [PATCH 259/334] [test](catalog) hive: align case_sensibility truncate errors to connector wording The truncate block asserted the legacy analysis-layer wording "Unknown database/table", but post-cutover truncate resolves the db/table through the connector, which throws the same-meaning "Failed to get database/table: '...'" (case mismatch -> not found). The parallel drop-table block was already migrated to this wording; align the truncate block to match (241/245 db-not-found, 250/262 table-not-found). Fixes test_hive_case_sensibility. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../hive/test_hive_case_sensibility.groovy | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy b/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy index 84679d10fa16f5..a8f7eb2f7da461 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_case_sensibility.groovy @@ -238,16 +238,16 @@ suite("test_hive_case_sensibility", "p0,external") { /// full qualified test { sql """truncate table CASE_DB2.CASE_TBL22""" - exception "Unknown database 'CASE_DB2'" + exception "Failed to get database: 'CASE_DB2' in catalog" } test { sql """truncate table CASE_DB2.case_tbl22""" - exception "Unknown database 'CASE_DB2'" + exception "Failed to get database: 'CASE_DB2'" } if (case_type.equals("0")) { test { sql """truncate table case_db2.CASE_TBL22""" - exception "Unknown table 'CASE_TBL22'" + exception "Failed to get table: 'CASE_TBL22'" } } else { sql """truncate table case_db2.CASE_TBL22""" @@ -259,7 +259,7 @@ suite("test_hive_case_sensibility", "p0,external") { if (case_type.equals("0")) { test { sql """truncate table CASE_TBL12;""" - exception "Unknown table 'CASE_TBL12'" + exception "Failed to get table: 'CASE_TBL12' in database: case_db1" } } else { sql """truncate table CASE_TBL12;""" From 3bc5ed1ef250847325245e8d3f2aa6cd8b3d081c Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 21:21:24 +0800 Subject: [PATCH 260/334] [doc](catalog) hive-e2e round-2 triage + batch-1 handoff Full HEAD-verified triage of the 22 round-2 hive e2e failures (15 CODE_FIX / 1 TEST_ALIGN / 2 ENV / 4 USER_DECISION; user ruled all 4 -> CODE_FIX). Records batch-1 (6 fixes / 8 suites) done and the remaining fix plan with per-suite pointers. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 37 +++++--------- .../reviews/hive-e2e-r2-triage-2026-07-11.md | 51 +++++++++++++++++++ 2 files changed, 64 insertions(+), 24 deletions(-) create mode 100644 plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 6f25851d92d0d3..5d9b26b3a74c55 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,11 +5,18 @@ --- -# 🆕 下一个 session 起步 = 修复 hive e2e 剩余 regression（HIVEFS-8 e2e 已首跑） - -> **本轮**：`external_table_p0/hive` 全量回归首跑 = **37/111 失败**。已逐个定位根因；**本轮修复约 20 个 suite（4 commit，下）**；**剩余 ~12 个真实 connector/fe-core regression 待修**（均**非**测试/环境问题，**勿 rebless `.out`**）。 -> **起步必读**：本段 + `git log`（4 fix commit 详尽）。失败日志（勿 commit，非本线程产物）：`nohup.out`（Jul 11）、FE `output/fe/log`、BE `output/be/log`。**BE 走 `format_v2` 扫描器**。**行号信 HEAD 不信文档。** -> **⚠ e2e 欠账**：本轮 4 修均 unit/build 绿（fe-connector-hive+hms 920 测试、0 checkstyle），**真值靠重部署+重跑断言**（尤其 ① drop-nonexistent 新文案 `"Failed to get database"` 是读代码推断非日志实证；② text 读回、binary hex 渲染）。**用户将先自跑一轮验证再开新 session 续修剩余。** +# 🆕 下一个 session 起步 = 修复 hive e2e 剩余 regression（round-2：22 失败，triage 全定案） + +> **本轮(round-2)**：用户按最新代码重跑 = **37 suite 测试、22 失败**。**全 22 已逐个 HEAD-verified 根因 + 对抗复核**（44-agent workflow `wf_c9def639-775`）。定案：**15 CODE_FIX · 1 TEST_ALIGN · 2 ENV · 4 USER_DECISION（用户已全裁决 → 均转 CODE_FIX）**。**完整三元表 + 逐 suite 修复指针 + 用户裁决** = `plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md`（**起步必读**，比下面的 round-1 R1-R12 表更新更准）。 +> **起步必读**：本段 + triage 文档 + `git log`。失败日志（勿 commit）：`nohup.out`(Jul 11 20:27)、FE `output/fe/log`、BE `output/be/log`。**BE 走 `format_v2`**。**行号信 HEAD 不信文档。** +> **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 +> +> **✅ 本轮 batch-1 已修（6 fix / 8 suite，均 fe-connector-hive test-compile+全套 UT 绿、0 checkstyle；见 git log）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 +> **⭐ 剩余待修（下个 session，按 triage 文档指针；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **fe-connector 中型**：R7 SHOW PARTITIONS 绕缓存(use_meta_cache_true) · R10 openx json ignore.malformed · R12/serde OpenCSV 走 serde 解析成全 STRING(test_open_csv_serde + test_hive_serde_prop **同根因**) · text_write LZ4FRAME→LZ4BLOCK 读修正。 +> - **fe-connector/fe-core 大型**：R2 SHOW CREATE TABLE 原生 DDL(show_create_table + ddl_text_format) · R3 `$partitions` 系统表 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode:150-161 那行，最简，可先做)。 +> - **用户已裁决(A)的 3 个 fe-core/SPI 大改**：query_cache 移植 SQL 结果缓存到 SPI(需连接器稳定失效令牌) · default_partition 连接器传 null 标记(SPI 新字段，零 paimon 影响) · hive_config_test 恢复 `hive.recursive_directories` 递归列目录。 +> - **ENV(告知用户，非代码)**：重置外部 hive docker → test_hive_lzo_text_format(run86.hql 未应用)、test_hive_varbinary_type(残留行翻倍)、hive_config_test tag1(HDFS OUTFILE 累积) 一并解决。 **✅ 本轮 DONE（4 commit，均编译+全套测试绿）**： - `cd4eecac5e6` **[fix] 文本读取根因（~12 suite）**：`HiveTextProperties`（fe-connector-hive）未做 `getByte` 数值→字节解码。Hive 默认分隔符存 `serialization.format="1"`=**字节 0x01**(Ctrl-A)，新代码当字符 `'1'`(0x31) 发 BE → `format_v2` TextReader 按 '1' 切 → 整行挤第一列 → null/粘连。移植 getByte + 老默认 + Hive2 拼写 key `colelction.delim` + table-params 优先；MultiDelimitSerDe 多字符保持 raw；null_format 不解码。新单测 `HiveTextPropertiesTest` 9/9。覆盖 to_date/prepare_data/basic_type/other/serde_prop/truncate/get_schema/meta_cache/query_cache + text_complex_type/to_array/text_garbled_file(靠 colelction.delim)。 @@ -17,25 +24,7 @@ - `d8c6eb002a1` **[fix] desc Key=true**：`ThriftHmsClient.convertFieldSchemas`（fe-connector-hms）用 5 参 `ConnectorColumn`(isKey 默认 false)→改 6 参 isKey=true（外表语义，对齐 iceberg）。修 `test_hive_struct_add_column`/`test_hive_view`。 - `b036c604651` **[test] DDL/事务文案对齐**：create-db-conflict→`"Failed to create Hive database…already exists"`、drop-db-nonexistent→`"Failed to get database:'X'"`(读代码推断)、事务 insert→`"Cannot write to a transactional Hive table"`；create-table-conflict(fe-core 层)未变、未动。 -**⭐ 下一步 = 修剩余 ~12 真实 regression**（每条：设计→(红队)→实现→靶向验证→独立 commit；勿 rebless `.out`）： - -| # | 缺陷 | 影响 suite | 修复位置指针 | 规模 | -|---|---|---|---|---| -| R1 | SHOW CREATE DATABASE 丢 LOCATION：`HiveConnectorMetadata` 无 `getDatabase` override | `test_hive_ddl`, `test_hms_event_notification`(+`_multi_catalog`) | 加 `getDatabase` 返 HMS db location 于 `ConnectorDatabaseMetadata.LOCATION_PROPERTY`，镜像 `IcebergConnectorMetadata.getDatabase:187` | 中 | -| R2 | SHOW CREATE TABLE 出通用 DDL：`ShowCreateTableCommand:155-159` 门控 `getType()==HMS_EXTERNAL_TABLE`，插件表是 PLUGIN_EXTERNAL_TABLE | `test_hive_show_create_table`, `test_hive_ddl_text_format`(次) | fe-core 路由 plugin-hive→hive DDL 渲染器（守 connector-agnostic 铁律；参 view 臂 :161-169） | 中 | -| R3 | `$partitions` 系统表未支持（"Unknown sys table"） | `test_hive_partition_values_tvf` | 连接器/`PluginDrivenExternalTable` sys-table dispatch | 大 | -| R4 | explain `partition=N/M`=0/0 | `test_hive_default_partition` | `PluginDrivenScanNode` explain 分区计数 + hive `appendExplainInfo` | 大 | -| R5 | 统计行数未接 cardinality（=1 非 66） | `test_hive_statistics_p0` | 连接器 stats 取数 + PluginDrivenScanNode cardinality 接线 | 大 | -| R6 | DECIMAL 分区谓词裁剪→0 行 | `test_hive_partitions`(q03+) | 连接器分区值归一/比较 | 中 | -| R7 | `use_meta_cache` 下 show partitions 空 | `test_hive_use_meta_cache_true`(sql09) | `fe-connector-hms` CachingHmsClient 分区列表 | 中 | -| R8 | `hive.recursive_directories` 未生效（只读顶层）| `hive_config_test`(qt_2) | 连接器文件列举递归 | 中 | -| R9 | LZO 文本 + `.lzo.index` sidecar 排除 | `test_hive_lzo_text_format` | 连接器 LZO input-format 分类+压缩解码+排 .index | 大 | -| R10 | openx JSON serde 接线缺失（ignore.malformed.json / read_hive_json_in_one_column）| `test_hive_openx_json` | `HiveTextProperties.extractJsonSerDeProps`(现仅收 serDeLib) + PluginDrivenScanNode json 属性 | 大 | -| R11 | 特殊字符分区值路径编解码 | `test_hive_special_char_partition` | 连接器分区值 escape/unescape + 写 | 中 | -| R12 | OpenCSV 另有原因（FE 路径老新等价，待定位）| `test_open_csv_serde`(csv_complex_type) | 待查（enclose/escape/复杂类型）| 待查 | - -**环境结论**：**无真幂等问题**；日志 `already exists`/`is not empty`/`does not exist` = 新连接器 DDL 文案包装 + 故意负测。**卫生**：个别 F suite `DROP DATABASE` 无 CASCADE，中断会留残库 → 整轮间重置外部 hive docker。 -**净室复审底稿**：本轮全 triage 由 7-agent 对抗 workflow 打底（根因 CONFIRMED、逐 suite 归因），结论可复用。 +> **round-1 的 R1-R12 表已被 round-2 triage 文档取代**（`plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md`，含 HEAD-verified 根因 + 精确指针 + 用户裁决）。round-1 的 4 个 commit（cd4eecac5e6/5672d7c0209/d8c6eb002a1/b036c604651）见 `git log`。 --- diff --git a/plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md b/plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md new file mode 100644 index 00000000000000..06fceadf562d6f --- /dev/null +++ b/plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md @@ -0,0 +1,51 @@ +# Hive e2e Round-2 Triage (2026-07-11) + +Round-2 run (`nohup.out`, 2026-07-11 20:27): **37 suites tested, 22 failed.** All 22 root-caused + adversarially +verified against HEAD via `wf-hive-e2e-r2-recon` (44 agents, 0 errors). Verdicts below. + +**Tally: 15 CODE_FIX · 1 TEST_ALIGN · 2 ENV · 4 USER_DECISION.** +BE scans via `format_v2`. Path is fe-connector-hive / fe-connector-hms (not old fe-core datasource.hive). + +## CODE_FIX (15 suites — genuine connector/fe-core regressions, legacy-parity restorations) + +Grouped by shared fix (several suites share one root): + +| Fix | Suites | Root cause (HEAD) | Fix pointer | Eff | +|---|---|---|---|---| +| **R1 getDatabase LOCATION** | test_hive_ddl, test_hms_event_notification, test_hms_event_notification_multi_catalog | `HiveConnectorMetadata` has no `getDatabase()` override → falls to `ConnectorSchemaOps` default (empty props) → SHOW CREATE DATABASE emits no LOCATION → test `substring(indexOf("hdfs://")=-1)` crashes | Add `@Override getDatabase(session,dbName)` after `databaseExists` (~L310) returning `ConnectorDatabaseMetadata` with `LOCATION_PROPERTY` = `hmsClient.getDatabase(dbName).getLocationUri()` (non-blank only). Mirror `IcebergConnectorMetadata:187-198` | S (fixes 3) | +| **orc binary mapping** | test_hive_orc | `HiveConnector.createClient` (HiveConnector.java:535) builds `ThriftHmsClient` via 2-arg ctor → forwards `HmsTypeMapping.Options.DEFAULT` (mapBinaryToVarbinary=false), ignoring catalog `enable.mapping.varbinary=true`. Commit 5672d7c0209 fixed `buildTypeMappingOptions` but that result is dead (used only by metadata field, not by the client that converts schema) | HiveConnector.java:535 — build Options from `this.properties` and pass 3-arg `new ThriftHmsClient(config, authAction, options)`. Remove now-dead metadata field/buildTypeMappingOptions | S | +| **R6 decimal partition prune** | test_hive_partitions | `extractLiteralValue` (HiveConnectorMetadata.java:2051) `String.valueOf(BigDecimal)` → "1.0000" ≠ stored "1" → all partitions pruned | Add `BigDecimal` branch before `String.valueOf` fallback: `((BigDecimal)val).stripTrailingZeros().toPlainString()` (mirror LocalDateTime case at 2044-2050) | S | +| **R11 special-char partition** | test_hive_special_char_partition | `parsePartitionName` (HiveConnectorMetadata.java:2105) unescapes the partition VALUE but stores the still-escaped column NAME as map key → predicate lookup by real name misses | Wrap key in `HiveWriteUtils.unescapePathName(part.substring(0,eq))` too (L2105) | S | +| **R5 cardinality explain** | test_hive_statistics_p0 | `PluginDrivenScanNode.getNodeExplainString` overrides FileScanNode wholesale, re-added many lines but omitted `cardinality=/avgRowSize=/numNodes=`. Field IS wired (`setCardinality` at PhysicalPlanTranslator:960) | fe-core PluginDrivenScanNode.java ~L372-385 — insert the FileScanNode:150-161 emission block verbatim (connector-agnostic: generic FileScanNode line) | S | +| **meta_cache ttl validation** | test_hive_meta_cache | `HiveConnectorProvider` has no `validateProperties` override → invalid `file.meta.cache.ttl-second=-2` accepted (legacy `HMSExternalCatalog.checkProperties` threw "…is wrong", no longer instantiated) | HiveConnectorProvider.java:32 — add `@Override validateProperties` using `CacheSpec.checkLongProperty(...,0L,...)` for the two ttl keys. Mirror `IcebergConnectorProvider:66-92` | S | +| **R7 SHOW PARTITIONS stale cache** | test_hive_use_meta_cache_true | SHOW PARTITIONS served from `CachingHmsClient.listPartitionNames` 24h-TTL cache (CachingHmsClient.java:148-151); sql08 caches empty list, hive adds partitions externally, sql09 returns stale empty | `HiveConnectorMetadata.collectPartitionNames:1085` — SHOW PARTITIONS listing must bypass the partition-name cache (fresh listing) | M | +| **R10 openx json ignore.malformed** | test_hive_openx_json | `HiveTextProperties.extractJsonSerDeProps:162` never reads serde `ignore.malformed.json` (called with only serDeLib at L111); `PluginDrivenScanNode` never calls `setOpenxJsonIgnoreMalformed`. Legacy HiveScanNode:646-647 did | Thread sdParams/tableParams into extractJsonSerDeProps (call site L111), emit `hive.text.openx_ignore_malformed`; fe-core PluginDrivenScanNode set `TFileAttributes.setOpenxJsonIgnoreMalformed` | M | +| **R12/serde OpenCSV schema** | test_open_csv_serde, **test_hive_serde_prop** (verifier said USER_DECISION but same root) | `ThriftHmsClient.convertTable` (ThriftHmsClient.java:654-656) builds schema from RAW `sd.getCols()` declared types instead of serde-resolved `getSchema()`. OpenCSVSerde legacy forced all columns to STRING (string-passthrough). New typed parsing flips `TRUE`→`true`, raw datetime→ISO, empty-string→NULL (changes IS NULL vs = '' semantics) | ThriftHmsClient.java:654-656 — resolve schema via serde (`getSchema()`/get_fields, split data vs partition cols) like legacy `initHiveSchema`. No serde-name branch in fe-core. **Broad blast radius (all-table schema path) — careful** | M | +| **text_write_insert LZ4 read** | test_hive_text_write_insert | READ-path (not write): write maps lz4→LZ4BLOCK fine; read path lost legacy `getFileCompressType` override that converts extension-inferred LZ4FRAME (`.lz4`→LZ4FRAME) to LZ4BLOCK ("hadoop uses lz4 block"). BE `LZ4F_getFrameInfo ERROR_frameType_unknown` | Restore LZ4FRAME→LZ4BLOCK in `HiveScanPlanProvider` (legacy HiveScanNode.java:670-678); add default compress-type hook on `ConnectorScanPlanProvider` (identity) to keep fe-core agnostic | M | +| **R2 SHOW CREATE TABLE hive DDL** | test_hive_show_create_table, test_hive_ddl_text_format | Plugin hive table is `PLUGIN_EXTERNAL_TABLE`, so `ShowCreateTableCommand:156` (gates on HMS_EXTERNAL_TABLE) is dead → generic Env DDL, no ROW FORMAT SERDE / SERDEPROPERTIES / STORED AS. HiveConnector deliberately does not declare SUPPORTS_SHOW_CREATE_DDL. Generic Env plugin arm double-quotes keys; test asserts single-quote | Implement `SUPPORTS_SHOW_CREATE_DDL` in HiveConnector (L262-283) + hive-native DDL renderer mirroring legacy `HiveMetaStoreClientHelper.showCreateTable:736-826`. Route fe-core by table TYPE not source | L | +| **R3 $partitions sys table** | test_hive_partition_values_tvf | `HiveConnectorMetadata.listSupportedSysTables:1298` returns empty ("Hive exposes no system tables"); legacy `HMSExternalTable.getSupportedSysTables` returned PartitionsSysTable | HiveConnectorMetadata.java:1302/1313 — expose "partitions" sys table for partitioned handle + sys-table scan/column plumbing (PartitionsSysTable/partition_values) | L | + +## TEST_ALIGN (1 — rebless-safe, same meaning) + +- **test_hive_case_sensibility**: expects `"Unknown database 'CASE_DB2'"`, new connector says `"Failed to get database: 'CASE_DB2' in catalog: …"` (case mismatch → DB not found). Same meaning. Edit `.groovy` L241/245 (and any cascading sites) to new wording. reblessSafe=true. + +## ENV (2 — not code bugs; need external hive docker reset) + +- **test_hive_varbinary_type**: `select9` reads write-target `test_hive_binary_orc_write_no_mapping` → 14 rows (every row doubled) vs golden 7. FE audit log proves writes happened ONCE this run (ReturnRows 5/1/1); the extra 7 are round-1 rows persisted in the live external HMS. Tables are docker-init'd empty and the suite appends with **no TRUNCATE** (groovy 59-61). Fix = reset external hive docker (fresh tables) OR add TRUNCATE to the test. **Not a rebless** (14 is contaminated). +- **test_hive_lzo_text_format**: 3 LZO tables never created in the running HMS. `run86.hql` (creates them, dated Jun 9) postdates the HMS bootstrap (run80-85 dated Apr 22) and was never applied. FE log confirms connector correctly reported table absent (metadata-only lookup). Fix = re-init docker HMS to apply run86.hql (ensure HS2 restart after lzo-hadoop jar copy so `LzoTextInputFormat` resolves). R9 connector LZO code can't be validated until tables exist. + +## USER_DECISION (4 — meaning-level / conflicts; user rules) + +1. **test_hive_query_cache** — SPI hive tables lost Nereids SQL result cache: all gates keyed on legacy `HMSExternalTable`/`HiveScanNode` (BindRelation:887-888, CacheAnalyzer:308, NereidsSqlCacheManager:475), which the generic `PluginDrivenExternalTable`/`PluginDrivenScanNode` don't satisfy → always `PhysicalFileScan`, never `PhysicalSqlCache`. **(A)** port cache to SPI [L: ~4 fe-core files + connector must supply a STABLE invalidation token — inherited `ExternalTable.updateTime=currentTimeMillis()` is unsafe → stale/spurious cache] / **(B)** accept no-cache (consistent with @Disabled HmsQueryCacheTest), disable assertHasCache(:151)/assertNoCache(:159). Iron rule OK either way (generic SPI types). +2. **test_hive_default_partition** — `__HIVE_DEFAULT_PARTITION__` on INT partition col: shared `PluginDrivenMvccExternalTable.toListPartitionItem:310` builds `PartitionValue(val,false)` unconditionally → `IntLiteral("__HIVE_DEFAULT_PARTITION__")` throws → NULL partition silently dropped → table misclassified unpartitioned → `partition=0/0`. Legacy `HiveExternalMetaCache:309` used `isNull = HIVE_DEFAULT_PARTITION.equals(val)`. Conflict: paimon P5 signed-off wants isNull=false (col IN semantics). **(A)** connector-supplied per-value null flag via SPI [L, zero paimon impact, IRON-RULE clean — RECOMMENDED] / **(B)** hardcode sentinel→isNull=true [1 line, but changes paimon signed-off semantics]. +3. **hive_config_test** (recursive_directories) — new fe-connector-hive ignores `hive.recursive_directories` entirely; `HiveFileListingCache` always non-recursive (skips subdirs). Legacy `HiveExternalMetaCache:391` defaulted "true"=recursive. On clean HDFS tags 2/21 expect 6 rows but connector returns top-level only → **missing subdir rows**. **(A)** restore recursive listing honoring the property [CODE_FIX — RECOMMENDED, else silent data loss] / **(B)** intentional non-recursive, rebless goldens 6→2. NOTE tag 1's failure is separately ENV (HDFS OUTFILE accumulation at fixed path — needs cleanup regardless). + +## USER DECISIONS (2026-07-11, signed off) +1. **query_cache → (A) port SQL result cache to SPI now.** Recognize generic PluginDrivenExternalTable/PluginDrivenScanNode in BindRelation/CacheAnalyzer/NereidsSqlCacheManager (iron-rule OK) AND surface a connector-provided STABLE invalidation token (not ExternalTable.currentTimeMillis). Effort L, needs BE cache-fill e2e. +2. **default_partition → (A) connector-supplied per-value null flag via SPI.** New SPI field so connector passes null semantics per partition value; fe-core builds partition item from it — hive marks null, paimon stays non-null (zero paimon impact, no if(hive) in fe-core). Effort L across SPI + both connectors. +3. **hive_config_test → (A) restore recursive listing.** HiveFileListingCache honors hive.recursive_directories (default true), tags 2/21 stay 6 rows. (tag 1 HDFS accumulation still needs env cleanup.) + +→ All 22 are now CODE_FIX (18) / TEST_ALIGN (1 case_sensibility) / ENV (2). Nothing left for the user to rule on. + +## Env note +A full external-hive-docker reset between rounds resolves both ENV items (fresh HMS applies run86.hql → LZO tables; fresh HDFS/tables → varbinary 7 rows). It also clears the tag-1 HDFS accumulation in hive_config_test. From db01b9fc07dd85229bafdeaccd4b3a168d839251 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 21:42:03 +0800 Subject: [PATCH 261/334] [fix](catalog) hive: re-emit cardinality/avgRowSize/numNodes in plugin scan explain (R5) PluginDrivenScanNode.getNodeExplainString overrides FileScanNode wholesale and does not call super; it re-added inputSplitNum / partition=N/M / VERBOSE backend detail / nested-columns but omitted the "cardinality=, avgRowSize=, numNodes=" line FileScanNode emits between the VERBOSE block and printNestedColumns. So test_hive_statistics_p0's `cardinality=66` assertion failed even though the cardinality field IS populated (PhysicalPlanTranslator#setCardinality from fileScan.getStats().getRowCount()). Re-emit the block verbatim -- connector-agnostic (universal FileScanNode info, no source-name branch, like the sibling inputSplitNum/partition lines). Fixes test_hive_statistics_p0. fe-core test-compile green (BUILD SUCCESS), 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/datasource/PluginDrivenScanNode.java | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 969d1c375ef3f6..cbf50c9c5fbbb2 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -371,6 +371,20 @@ public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { if (detailLevel == TExplainLevel.VERBOSE && !isBatchMode()) { appendBackendScanRangeDetail(output, prefix); } + // R5 (explain gap): FileScanNode#getNodeExplainString emits the cardinality/avgRowSize/numNodes + // line right after the VERBOSE block; this override drops it by not calling super, so + // test_hive_statistics_p0's `cardinality=66` assertion failed. Re-emit it verbatim -- like the + // sibling inputSplitNum / partition / backend-detail / nested-columns lines -- because row-count + // stats visibility is universal FileScanNode info, not connector-specific (the cardinality field + // is populated for every plugin FileScan via PhysicalPlanTranslator#setCardinality). + output.append(prefix); + if (cardinality > 0) { + output.append(String.format("cardinality=%s, ", cardinality)); + } + if (avgRowSize > 0) { + output.append(String.format("avgRowSize=%s, ", avgRowSize)); + } + output.append(String.format("numNodes=%s", numNodes)).append("\n"); // F6/F7 (explain gap): the parent FileScanNode emits the "nested columns:" block (pruned type / // sub path / all + predicate access paths) via printNestedColumns, which this override drops by // not calling super, so the ENTIRE block vanished for every plugin FileScan connector (broader From 142f285cbaec572f9d856568c3b906c68a766457 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 21:43:44 +0800 Subject: [PATCH 262/334] [doc](catalog) hive-e2e: record R5 done + design notes for remaining 13 suites batch-1+2 = 7 fixes / 9 suites done. Enrich the remaining-work list with the design nuances discovered while reading HEAD: R7 shared collectPartitionNames cache-split, R8 recursive byte-parity across 3 consumers, R10/R12/text_write pointers, and the 3 user-approved SPI/fe-core changes (query_cache token, default_partition null flag, recursive listing). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 5d9b26b3a74c55..593af8e811f3fa 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -11,11 +11,18 @@ > **起步必读**：本段 + triage 文档 + `git log`。失败日志（勿 commit）：`nohup.out`(Jul 11 20:27)、FE `output/fe/log`、BE `output/be/log`。**BE 走 `format_v2`**。**行号信 HEAD 不信文档。** > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > -> **✅ 本轮 batch-1 已修（6 fix / 8 suite，均 fe-connector-hive test-compile+全套 UT 绿、0 checkstyle；见 git log）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **⭐ 剩余待修（下个 session，按 triage 文档指针；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **fe-connector 中型**：R7 SHOW PARTITIONS 绕缓存(use_meta_cache_true) · R10 openx json ignore.malformed · R12/serde OpenCSV 走 serde 解析成全 STRING(test_open_csv_serde + test_hive_serde_prop **同根因**) · text_write LZ4FRAME→LZ4BLOCK 读修正。 -> - **fe-connector/fe-core 大型**：R2 SHOW CREATE TABLE 原生 DDL(show_create_table + ddl_text_format) · R3 `$partitions` 系统表 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode:150-161 那行，最简，可先做)。 -> - **用户已裁决(A)的 3 个 fe-core/SPI 大改**：query_cache 移植 SQL 结果缓存到 SPI(需连接器稳定失效令牌) · default_partition 连接器传 null 标记(SPI 新字段，零 paimon 影响) · hive_config_test 恢复 `hive.recursive_directories` 递归列目录。 +> **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 +> **⭐ 剩余 13 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **fe-connector 中型**： +> - **R7** SHOW PARTITIONS 绕缓存(use_meta_cache_true sql09)：**注意** `HiveConnectorMetadata.collectPartitionNames` 被 SHOW PARTITIONS(`listPartitionNames`，须 fresh) 与查询裁剪(`listPartitions`，`use_meta_cache=true` 下须继续用缓存) **共用**——须只对 SHOW-PARTITIONS 路径绕过 `CachingHmsClient` 缓存(走 delegate)，别一刀切关缓存(否则破坏 use_meta_cache 契约)。测试序列：refresh→sql08(缓存空)→hive 外部 add partition(无 refresh)→sql09 须见新分区。 +> - **R10** openx json ignore.malformed：`HiveTextProperties.extractJsonSerDeProps`(:162, 现只收 serDeLib，调用点 :111 有 sdParams/tableParams 可用)串入 serde `ignore.malformed.json`→发 `hive.text.openx_ignore_malformed`；fe-core `PluginDrivenScanNode`(:633-637)调 `TFileAttributes.setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode:646-647)。 +> - **R12/serde** OpenCSV 走 serde 解析成全 STRING(test_open_csv_serde + **test_hive_serde_prop 同根因**)：`ThriftHmsClient.convertTable`(:654-656)当前用原始 `sd.getCols()` 声明类型；须像 legacy `initHiveSchema` 经 `getSchema()`(SerDe ObjectInspector, OpenCSVSerde 强制全列 string) 解析，拆 data cols vs partition keys。**⚠ blast radius 大**(改所有 hive 表 schema 解析路径)，须红队 + 广回归。别在 fe-core 按 serde 名分支。 +> - **text_write** LZ4FRAME→LZ4BLOCK 读修正：`HiveScanPlanProvider` 恢复 legacy HiveScanNode:670-678 的 `getFileCompressType` override(`.lz4`→LZ4FRAME 需转 LZ4BLOCK)；为守 fe-core connector-agnostic，加 `ConnectorScanPlanProvider` default 压缩类型 hook(identity)。 +> - **fe-connector/fe-core 大型**：R2 SHOW CREATE TABLE 原生 DDL(show_create_table + ddl_text_format；HiveConnector 声明 `SUPPORTS_SHOW_CREATE_DDL` + 原生 DDL 渲染器镜像 legacy HiveMetaStoreClientHelper.showCreateTable，注意测试断言单引号) · R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 +> - **用户已裁决(A)的 3 个 fe-core/SPI 大改**： +> - query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 +> - default_partition 连接器传 null 标记：SPI 新字段让连接器按分区值传 null 语义，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem:310`(现无条件 `PartitionValue(v,false)`) 据此建项——hive 标 null、paimon 保持非空(零 paimon 影响，别在 fe-core if(hive))。 +> - hive_config_test 恢复 `hive.recursive_directories`：`HiveFileListingCache.listFromFileSystem`(现无条件非递归 `fs.list`+跳目录)按属性(默认 true)手动递归；**须保持 3 个消费方 byte-parity**(scan 切分 `listAndSplitFiles`、size 估算 `sumCachedFileSizes`、cache)——须把 recursive flag 从 properties 串到 lister。(tag1 另是 ENV：HDFS OUTFILE 累积，须清路径。) > - **ENV(告知用户，非代码)**：重置外部 hive docker → test_hive_lzo_text_format(run86.hql 未应用)、test_hive_varbinary_type(残留行翻倍)、hive_config_test tag1(HDFS OUTFILE 累积) 一并解决。 **✅ 本轮 DONE（4 commit，均编译+全套测试绿）**： From 4df95ad44acb30d359490342606b98a3ae434d94 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 22:09:28 +0800 Subject: [PATCH 263/334] [fix](catalog) hive: SHOW PARTITIONS lists fresh partition names (bypass connector cache) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit SHOW PARTITIONS on a plugin-driven hive table (and the partitions metadata TVF) served a stale partition-name list from the connector-owned CachingHmsClient (24h TTL): after `refresh` cached an empty list, partitions added externally in Hive stayed invisible until TTL/REFRESH. Legacy read the raw pooled metastore client (ThriftHMSCachedClient — "Cached" = connection pooling, not metadata cache), so legacy SHOW PARTITIONS was always fresh and never touched HiveExternalMetaCache; the cutover to CachingHmsClient inadvertently routed it through the cache. Fix keeps the use_meta_cache contract intact — only the SHOW-PARTITIONS / partitions-TVF path lists fresh; query partition pruning and MTMV freshness stay cached (legacy pruned/probed off the cache too): - HmsClient: new default `listPartitionNamesFresh` (= listPartitionNames for a non-decorating client; any caching decorator MUST override it). - CachingHmsClient: override goes straight to the delegate, neither reading nor writing partitionNamesCache. - HiveConnectorMetadata.collectPartitionNames gains a `bypassCache` flag: listPartitionNames (SHOW/TVF) -> true (fresh); listPartitions (pruning) and getTableFreshness (MTMV) -> false (cached, unchanged). fe-core untouched (connector-internal choice between two HmsClient methods; no if(hive)). Design red-teamed by 4 independent lenses (all SOUND): legacy parity, caller completeness, decorator stack, MTMV parity. Tests: CachingHmsClientTest +3 (fresh always hits delegate; fresh does not populate cache; interface default on a non-caching client) — 19/19 green. fe-connector-hms + fe-connector-hive BUILD SUCCESS, 0 checkstyle. Fixes test_hive_use_meta_cache_true sql09 (live-gated; user reruns e2e). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveConnectorMetadata.java | 29 +++++++++++---- .../doris/connector/hms/CachingHmsClient.java | 9 +++++ .../apache/doris/connector/hms/HmsClient.java | 19 ++++++++++ .../connector/hms/CachingHmsClientTest.java | 37 +++++++++++++++++++ 4 files changed, 86 insertions(+), 8 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 412b00c67a27ff..5a51e6467d3fde 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1051,7 +1051,9 @@ public List listPartitionNames(ConnectorSession session, ConnectorTableH if (!(handle instanceof HiveTableHandle)) { return siblingMetadata(session, handle).listPartitionNames(session, handle); } - return collectPartitionNames((HiveTableHandle) handle); + // SHOW PARTITIONS / partitions metadata TVF: FRESH listing (bypass cache), matching legacy's raw-client + // read — else an externally-added partition stays invisible until TTL/REFRESH (test_hive_use_meta_cache_true). + return collectPartitionNames((HiveTableHandle) handle, true); } /** @@ -1076,7 +1078,8 @@ public List listPartitions(ConnectorSession session, } HiveTableHandle hiveHandle = (HiveTableHandle) handle; List partKeyNames = hiveHandle.getPartitionKeyNames(); - List partitionNames = collectPartitionNames(hiveHandle); + // Query partition pruning: CACHED listing (use_meta_cache contract; legacy pruned off HiveExternalMetaCache). + List partitionNames = collectPartitionNames(hiveHandle, false); List result = new ArrayList<>(partitionNames.size()); for (String partitionName : partitionNames) { result.add(new ConnectorPartitionInfo(partitionName, @@ -1087,18 +1090,27 @@ public List listPartitions(ConnectorSession session, } /** - * Shared partition-name lister backing {@link #listPartitionNames} and {@link #listPartitions}. Returns - * the metastore's rendered partition names ({@code key=value/...}); an unpartitioned table (no partition - * keys) lists nothing without touching the metastore. + * Shared partition-name lister backing {@link #listPartitionNames}, {@link #listPartitions} and + * {@link #getTableFreshness}. Returns the metastore's rendered partition names ({@code key=value/...}); an + * unpartitioned table (no partition keys) lists nothing without touching the metastore. + * + *

    {@code bypassCache} selects the freshness contract: the SHOW-PARTITIONS / partitions-TVF path + * ({@link #listPartitionNames}) lists FRESH (legacy read the raw pooled client, never the metadata cache), + * while the query-pruning path ({@link #listPartitions}) and the MTMV freshness path + * ({@link #getTableFreshness}) stay CACHED (the {@code use_meta_cache} contract; legacy pruning/MTMV both read + * the cached {@code HiveExternalMetaCache}). The {@code CachingHmsClient} decorator owns the two behaviours; + * a non-caching client serves both identically. */ - private List collectPartitionNames(HiveTableHandle handle) { + private List collectPartitionNames(HiveTableHandle handle, boolean bypassCache) { List partKeyNames = handle.getPartitionKeyNames(); if (partKeyNames == null || partKeyNames.isEmpty()) { return Collections.emptyList(); } // -1 = "all partitions": ThriftHmsClient maps it to an unbounded HMS listing (no silent cap), // matching legacy's default (Config.max_hive_list_partition_num = -1). - return hmsClient.listPartitionNames(handle.getDbName(), handle.getTableName(), -1); + return bypassCache + ? hmsClient.listPartitionNamesFresh(handle.getDbName(), handle.getTableName(), -1) + : hmsClient.listPartitionNames(handle.getDbName(), handle.getTableName(), -1); } /** @@ -1174,7 +1186,8 @@ public Optional getTableFreshness(ConnectorSession sess return Optional.of(new ConnectorTableFreshness(hiveHandle.getTableName(), lastDdlMillis(hiveHandle.getTableParameters()))); } - List partitionNames = collectPartitionNames(hiveHandle); + // MTMV whole-table freshness: CACHED listing (legacy HiveDlaTable.getTableSnapshot read cached names too). + List partitionNames = collectPartitionNames(hiveHandle, false); if (partitionNames.isEmpty()) { // Parity: an empty partition list yields MTMVMaxTimestampSnapshot(tableName, 0). return Optional.of(new ConnectorTableFreshness(hiveHandle.getTableName(), 0L)); diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index ce70d268da1ee3..1377aad2b6eb69 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -150,6 +150,15 @@ public List listPartitionNames(String dbName, String tableName, int maxP key -> delegate.listPartitionNames(key.dbName, key.tableName, key.maxParts)); } + @Override + public List listPartitionNamesFresh(String dbName, String tableName, int maxParts) { + // Fresh (cache-bypassing) listing for SHOW PARTITIONS / the partitions metadata TVF — legacy read the raw + // pooled client, never the metadata cache. This neither READS nor WRITES partitionNamesCache: reading would + // serve the stale list this method exists to avoid, and writing would let a non-cache path repopulate the + // cache off-band. The query-pruning path stays on the cached listPartitionNames (use_meta_cache contract). + return delegate.listPartitionNames(dbName, tableName, maxParts); + } + @Override public List getPartitions(String dbName, String tableName, List partNames) { if (partNames == null || partNames.isEmpty()) { diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java index 4810305d7d7db2..c45116df2bab35 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java @@ -114,6 +114,25 @@ public interface HmsClient extends Closeable { List listPartitionNames(String dbName, String tableName, int maxParts); + /** + * Lists partition names bypassing any decorator cache (a FRESH metastore listing). SHOW PARTITIONS and the + * {@code partitions} metadata TVF need a fresh view (legacy read the raw pooled client, never the metadata + * cache); the query-pruning path keeps {@link #listPartitionNames} (cached under {@code use_meta_cache}). + * + *

    Default = the cached path: a non-decorating client (e.g. the raw {@link ThriftHmsClient}) has no cache + * to bypass, so the two are identical for it. Any caching {@code HmsClient} decorator MUST override this + * to reach its delegate directly, or SHOW PARTITIONS regresses to serving a stale cached list. + * + * @param dbName database name + * @param tableName table name + * @param maxParts maximum number of partitions to return + * @return list of partition name strings (e.g. "dt=2024-01-01/region=us") + * @throws HmsClientException if the operation fails + */ + default List listPartitionNamesFresh(String dbName, String tableName, int maxParts) { + return listPartitionNames(dbName, tableName, maxParts); + } + /** * Get partition metadata by partition names. * diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java index 8c90fcc6bf2e77..18c3756f9a92e3 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -113,6 +113,43 @@ public void listPartitionNamesCachesByDbTableAndMaxParts() { Assertions.assertEquals(2, delegate.listPartitionNamesCalls); } + // ---- listPartitionNamesFresh (SHOW PARTITIONS / partitions TVF — must bypass the names cache) ---- + + @Test + public void listPartitionNamesFreshAlwaysHitsDelegate() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // WHY: SHOW PARTITIONS must see partitions added externally after the cache filled. Every fresh call + // goes to the metastore — never served from partitionNamesCache. (test_hive_use_meta_cache_true sql09.) + cache.listPartitionNamesFresh("db", "t", -1); + cache.listPartitionNamesFresh("db", "t", -1); + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + } + + @Test + public void listPartitionNamesFreshDoesNotPopulateCache() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Fresh call must NOT write the names cache: a following cached listPartitionNames must still MISS + // (delegate call #2) and only THEN populate — proving fresh bypasses the cache in both directions. + cache.listPartitionNamesFresh("db", "t", -1); // delegate #1, no populate + cache.listPartitionNames("db", "t", -1); // cache miss -> delegate #2 + populate + cache.listPartitionNames("db", "t", -1); // cache hit -> no delegate call + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + } + + @Test + public void listPartitionNamesFreshDefaultOnNonCachingClientIsPlainListing() { + // A bare HmsClient (no caching decorator) inherits the interface default: fresh == the raw listing. + // Guards the C4 foot-gun — a non-decorating client has nothing to bypass, so the two must be identical. + RecordingHmsClient raw = new RecordingHmsClient(); + raw.listPartitionNamesFresh("db", "t", -1); + raw.listPartitionNames("db", "t", -1); + Assertions.assertEquals(2, raw.listPartitionNamesCalls); + } + // ---- getPartitions ---- @Test From e8f4f00f4177e06b8738a596d24f380155c47dc7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 22:10:25 +0800 Subject: [PATCH 264/334] [doc](catalog) hive-e2e: land FIX-R7 design + red-team, mark R7 done, repoint to R10 R7 (SHOW PARTITIONS cache bypass) design doc with the 4-lens red-team result (all SOUND), new hive-e2e-r2 fix task-list, and HANDOFF update (R7 done, 12 suites remaining, next = R10 openx json ignore.malformed). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 4 +- plan-doc/task-list-hive-e2e-r2-fixes.md | 32 +++++ plan-doc/tasks/designs/FIX-R7-design.md | 171 ++++++++++++++++++++++++ 3 files changed, 205 insertions(+), 2 deletions(-) create mode 100644 plan-doc/task-list-hive-e2e-r2-fixes.md create mode 100644 plan-doc/tasks/designs/FIX-R7-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 593af8e811f3fa..d608719dc91102 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -12,9 +12,9 @@ > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > > **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **⭐ 剩余 13 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> **✅ batch-3（本 session）：R7 DONE `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。设计红队 4 lens 全 SOUND(legacy parity=旧 ThriftHMSCachedClient 就是 fresh 直 RPC；caller completeness；decorator stack 单层；MTMV 旧亦读缓存)。`CachingHmsClientTest` 19/19、两模块 BUILD SUCCESS、0 checkstyle。设计 `designs/FIX-R7-design.md`、清单 `task-list-hive-e2e-r2-fixes.md`。**e2e 待用户重部署自跑 test_hive_use_meta_cache_true sql09。** +> **⭐ 剩余 12 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： > - **fe-connector 中型**： -> - **R7** SHOW PARTITIONS 绕缓存(use_meta_cache_true sql09)：**注意** `HiveConnectorMetadata.collectPartitionNames` 被 SHOW PARTITIONS(`listPartitionNames`，须 fresh) 与查询裁剪(`listPartitions`，`use_meta_cache=true` 下须继续用缓存) **共用**——须只对 SHOW-PARTITIONS 路径绕过 `CachingHmsClient` 缓存(走 delegate)，别一刀切关缓存(否则破坏 use_meta_cache 契约)。测试序列：refresh→sql08(缓存空)→hive 外部 add partition(无 refresh)→sql09 须见新分区。 > - **R10** openx json ignore.malformed：`HiveTextProperties.extractJsonSerDeProps`(:162, 现只收 serDeLib，调用点 :111 有 sdParams/tableParams 可用)串入 serde `ignore.malformed.json`→发 `hive.text.openx_ignore_malformed`；fe-core `PluginDrivenScanNode`(:633-637)调 `TFileAttributes.setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode:646-647)。 > - **R12/serde** OpenCSV 走 serde 解析成全 STRING(test_open_csv_serde + **test_hive_serde_prop 同根因**)：`ThriftHmsClient.convertTable`(:654-656)当前用原始 `sd.getCols()` 声明类型；须像 legacy `initHiveSchema` 经 `getSchema()`(SerDe ObjectInspector, OpenCSVSerde 强制全列 string) 解析，拆 data cols vs partition keys。**⚠ blast radius 大**(改所有 hive 表 schema 解析路径)，须红队 + 广回归。别在 fe-core 按 serde 名分支。 > - **text_write** LZ4FRAME→LZ4BLOCK 读修正：`HiveScanPlanProvider` 恢复 legacy HiveScanNode:670-678 的 `getFileCompressType` override(`.lz4`→LZ4FRAME 需转 LZ4BLOCK)；为守 fe-core connector-agnostic，加 `ConnectorScanPlanProvider` default 压缩类型 hook(identity)。 diff --git a/plan-doc/task-list-hive-e2e-r2-fixes.md b/plan-doc/task-list-hive-e2e-r2-fixes.md new file mode 100644 index 00000000000000..b9be9825733610 --- /dev/null +++ b/plan-doc/task-list-hive-e2e-r2-fixes.md @@ -0,0 +1,32 @@ +# Task List — hive e2e round-2 remaining fixes + +Source triage: `plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md`. +Discipline per task: 设计(`tasks/designs/FIX--design.md`) → 设计红队 → 实现 → build+靶向 UT → 独立 commit → 勾表 → 更新 HANDOFF. + +## Done (batch-1+2 — earlier sessions) +- [x] R1 getDatabase LOCATION (test_hive_ddl, test_hms_event_notification[_multi_catalog]) +- [x] orc binary client options (test_hive_orc) +- [x] R6 decimal partition prune (test_hive_partitions) +- [x] R11 special-char partition key (test_hive_special_char_partition) +- [x] meta_cache ttl validation (test_hive_meta_cache) +- [x] R5 cardinality explain (test_hive_statistics_p0) +- [x] TEST_ALIGN case_sensibility (test_hive_case_sensibility) + +## Remaining code fixes +### fe-connector 中型 +- [x] R7 SHOW PARTITIONS bypass cache (test_hive_use_meta_cache_true) `4df95ad44ac` — design red-teamed 4/4 SOUND, UT 19/19 +- [ ] R10 openx json ignore.malformed (test_hive_openx_json) +- [ ] R12/serde OpenCSV all-STRING schema (test_open_csv_serde, test_hive_serde_prop) +- [ ] text_write LZ4FRAME→LZ4BLOCK read (test_hive_text_write_insert) + +### fe-connector / fe-core 大型 +- [ ] R2 SHOW CREATE TABLE native DDL (test_hive_show_create_table, test_hive_ddl_text_format) +- [ ] R3 $partitions sys table (test_hive_partition_values_tvf) + +### fe-core / SPI 大改 (user signed off → option A) +- [ ] query_cache: port SQL result cache to SPI + connector stable invalidation token (test_hive_query_cache) +- [ ] default_partition: connector-supplied per-value null flag via SPI (test_hive_default_partition) +- [ ] hive_config_test: restore recursive listing honoring hive.recursive_directories + +## ENV (告知用户，非代码) +- [ ] Reset external hive docker → resolves test_hive_lzo_text_format, test_hive_varbinary_type, hive_config_test tag1 diff --git a/plan-doc/tasks/designs/FIX-R7-design.md b/plan-doc/tasks/designs/FIX-R7-design.md new file mode 100644 index 00000000000000..1f1f37b80eecf8 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-R7-design.md @@ -0,0 +1,171 @@ +# FIX-R7 — SHOW PARTITIONS serves stale partition-name cache + +Suite: `test_hive_use_meta_cache_true` (sql09). Effort: M. Scope: fe-connector only. + +## Problem + +Under `use_meta_cache=true`, `SHOW PARTITIONS FROM ` returns a stale (empty) partition +list even after partitions were added externally in Hive: + +``` +refresh catalog -> sql08: SHOW PARTITIONS -> {} (0 partitions, caches empty) +hive: alter table add partition part1 +hive: alter table add partition part2 + -> sql09: SHOW PARTITIONS -> STILL {} (WRONG; expected part1, part2) +``` + +The test's own spec comment (`test_hive_use_meta_cache_true.groovy:130`) states the contract: +**"can see because partition file listing is not cached"**. + +## Root Cause (HEAD-verified) + +`ShowPartitionsCommand:325` → `metadata.listPartitionNames(session, handle)` → +`HiveConnectorMetadata.listPartitionNames` (:1050) → `collectPartitionNames` (:1094) → +`hmsClient.listPartitionNames(db, tbl, -1)`. + +`hmsClient` is the `CachingHmsClient` decorator, whose `listPartitionNames` (:148-151) serves from +the `partitionNamesCache` (default 24h TTL). So `refresh` (sql08) flushes + repopulates the cache +with the empty list; the external `add partition` does not invalidate it (coarse REFRESH + TTL is the +only invalidation, by design); sql09 returns the stale empty list. + +**Legacy parity (the regression):** legacy `ShowPartitionsCommand.handleShowHMSTablePartitions:375` +called `hmsCatalog.getClient().listPartitionNames(...)` — the **raw** metastore client, NOT the +`HiveExternalMetaCache`. So legacy SHOW PARTITIONS always listed **fresh** partition names, bypassing +the metadata cache. The metadata TVF path (`MetadataGenerator:1338`, legacy) likewise used the raw +client. The cutover to `CachingHmsClient` inadvertently routed SHOW PARTITIONS through the cache. + +## The nuance (do NOT one-shot disable the cache) + +`collectPartitionNames` is shared by THREE callers: + +| Caller | Purpose | Legacy source | Required | +|---|---|---|---| +| `listPartitionNames` (:1050) | SHOW PARTITIONS + `partitions` metadata TVF (`MetadataGenerator:1358`) | raw client (fresh) | **FRESH** | +| `listPartitions` (:1072) | query partition pruning | `HiveExternalMetaCache` (cached) | **CACHED** (use_meta_cache contract) | +| `getTableFreshness` (:1177) | MTMV whole-table freshness | cached | **CACHED** (unchanged — surgical) | + +Disabling the cache inside `collectPartitionNames` unconditionally would defeat `use_meta_cache` for +query pruning (a fresh HMS RPC on every partitioned SELECT) — a real regression. The fix must make +only the SHOW-PARTITIONS/TVF path (`listPartitionNames`) bypass the cache. + +## Design + +Add a **fresh (cache-bypassing) partition-name listing** to the `HmsClient` interface as a default +method, overridden by the caching decorator to go straight to the delegate. Route the +`listPartitionNames` SPI path through it; leave `listPartitions` and `getTableFreshness` on the cached +path. + +### 1. `HmsClient` (interface) — new default method + +```java +/** + * Lists partition names bypassing any decorator cache (a fresh metastore listing). SHOW PARTITIONS + * and the partitions metadata TVF need a fresh view (legacy read the raw client, never the cache); + * the query-pruning path keeps {@link #listPartitionNames} (cached under use_meta_cache). + * Default = the cached path: a non-decorating client (e.g. the raw ThriftHmsClient) has no cache to + * bypass, so the two are identical for it. + */ +default List listPartitionNamesFresh(String dbName, String tableName, int maxParts) { + return listPartitionNames(dbName, tableName, maxParts); +} +``` + +### 2. `CachingHmsClient` — override to delegate directly + +```java +@Override +public List listPartitionNamesFresh(String dbName, String tableName, int maxParts) { + // Fresh (cache-bypassing) listing for SHOW PARTITIONS / partitions TVF (legacy raw-client parity). + // Neither reads nor writes partitionNamesCache: reading would serve a stale list; writing would let + // a rare escaped-value edge repopulate the cache off a non-cache path. The query-pruning path keeps + // listPartitionNames (cached). + return delegate.listPartitionNames(dbName, tableName, maxParts); +} +``` + +### 3. `HiveConnectorMetadata.collectPartitionNames` — gain a `bypassCache` flag + +```java +private List collectPartitionNames(HiveTableHandle handle, boolean bypassCache) { + List partKeyNames = handle.getPartitionKeyNames(); + if (partKeyNames == null || partKeyNames.isEmpty()) { + return Collections.emptyList(); + } + return bypassCache + ? hmsClient.listPartitionNamesFresh(handle.getDbName(), handle.getTableName(), -1) + : hmsClient.listPartitionNames(handle.getDbName(), handle.getTableName(), -1); +} +``` + +Route the three callers: +- `listPartitionNames` (:1054) → `collectPartitionNames(handle, true)` **(fresh — the fix)** +- `listPartitions` (:1079) → `collectPartitionNames(hiveHandle, false)` (cached — unchanged) +- `getTableFreshness` (:1177) → `collectPartitionNames(hiveHandle, false)` (cached — unchanged) + +## Risk Analysis + +- **Blast radius**: `HmsClient` interface + `CachingHmsClient` + `HiveConnectorMetadata`. The new + interface method is a `default`, so no other `HmsClient` implementor breaks (ThriftHmsClient inherits + the default = its own `listPartitionNames`). Iceberg/hudi siblings do not call these hive methods. +- **Query pruning perf**: unchanged — `listPartitions` stays cached. `use_meta_cache` contract intact. +- **MTMV**: unchanged — `getTableFreshness` stays cached (deliberately not touched; not part of this + regression, and switching it to fresh would add an RPC per freshness probe). +- **Metadata TVF** (`partitions`, `MetadataGenerator:1358`): now fresh. This matches legacy (raw + client) and the TVF is low-frequency. Not a perf concern. +- **Fresh does not repopulate the names cache**: preserves legacy independence (SHOW PARTITIONS never + updated the value cache). Query pruning still bound by TTL/REFRESH as before. +- **Connector-agnostic**: fe-core unchanged. No `if(hive)` anywhere. The bypass is a connector-internal + choice between two `HmsClient` methods. + +## Test Plan + +### Unit Tests (fe-connector-hms, no Mockito — recording fake) + +`CachingHmsClientTest` — new tests using the existing recording `delegate` (`listPartitionNamesCalls`): +1. `listPartitionNamesFreshAlwaysHitsDelegate`: two `listPartitionNamesFresh("db","t",-1)` calls → + `delegate.listPartitionNamesCalls == 2` (never served from cache). +2. `listPartitionNamesFreshDoesNotPopulateCache`: `listPartitionNamesFresh` then two + `listPartitionNames` → total `delegate.listPartitionNamesCalls == 2` (fresh call #1 delegated and did + NOT populate; cached call #2 delegated+populated; cached call #3 served from cache). Proves bypass + both directions (read + write). +3. `listPartitionNamesStillCached` (regression guard for the cached path): unchanged existing behavior + — two `listPartitionNames` → 1 delegate call. + +Optionally a `HmsClient` default-method test: a bare `HmsClient` (no override) has +`listPartitionNamesFresh == listPartitionNames`. + +### E2E + +`test_hive_use_meta_cache_true` sql09 turns green (the failing assertion). Live-gated (needs external +hive docker); user reruns. No golden change (sql09 golden already expects part1/part2). + +## Verification +- `mvn -o -f fe/pom.xml -pl :fe-connector-hms -am test-compile` + `-Dtest=CachingHmsClientTest`. +- `mvn -o -f fe/pom.xml -pl :fe-connector-hive -am test-compile` (interface change compiles against hive). +- 0 checkstyle; import gate clean. + +## Red-team result (wf_d9882a60-f53, 4 independent lenses — all SOUND, 0 blocker/major) +- **C1 legacy parity — SOUND.** Legacy SHOW PARTITIONS → `HMSExternalCatalog.getClient()` = `ThriftHMSCachedClient` + whose `listPartitionNames` issues a direct thrift RPC ("Cached" = connection pooling, NOT metadata cache); no + caffeine read/write, never touches `HiveExternalMetaCache`. Partitions TVF (`MetadataGenerator:1335-1338`) same + raw client. The name/value cache is populated only by the query-pruning loader `loadPartitionValues` — a separate + path. Confirms: fresh + no repopulation is exact legacy parity. +- **C2 caller completeness — SOUND** (2 doc notes folded). SPI `listPartitionNames(session,handle)` has EXACTLY 2 + fe-core callers (SHOW PARTITIONS single-column `:325`, partitions TVF `:1358`) — both intended-fresh, low-freq. + No pruning/stats/MTMV/cache-fill path calls it (hive pruning routes through `listPartitions`), so making it fresh + cannot regress a hot path. Notes: (a) `listPartitions(session,...)` actually serves FOUR callers, ALL staying + cached, none broken — `PluginDrivenExternalTable:781` (pruning), `:837` (`partition_values` TVF), + `PluginDrivenMvccExternalTable:268` (paimon/iceberg MVCC), `ShowPartitionsCommand:308` (paimon 5-col SHOW; hive + never reaches it — no `SUPPORTS_PARTITION_STATS`). (b) Benign freshness asymmetry: after the fix the `partitions` + TVF is fresh while the `partition_values` TVF stays cached (pruning path). ACCEPTED as intended — it mirrors the + legacy split (SHOW/list-names = fresh raw client; pruning/values = cached), and only the SHOW/list-names path is + the regression under test. Not a code change. +- **C3 MTMV + iron-rule — SOUND.** Legacy `HiveDlaTable.getTableSnapshot` (partitioned) read names from the CACHED + `HiveExternalMetaCache`, so leaving `getTableFreshness` cached is CORRECT parity (not a latent bug). Clarify: + `getPartitionFreshnessMillis` never used `collectPartitionNames` (calls `getPartitions` directly) — it is simply + UNTOUCHED, not "left cached". Fix is connector-internal (no fe-core edit) → iron rule trivially clean. +- **C4 decorator stack — SOUND.** Production stack = single `CachingHmsClient(ThriftHmsClient)`; no outer/auth/TCCL + HmsClient wrapper, no double-wrap. Override returns `delegate.listPartitionNames` = raw pooled RPC (fresh, no cache + re-entry). New `default` safe for all impls: `ThriftHmsClient` inherits → fresh; hudi uses raw `ThriftHmsClient`; + iceberg has no HmsClient impl; test fakes inherit cleanly. Foot-gun note (not a HEAD defect): the Javadoc MUST + keep the "any caching decorator must override this" instruction for future implementors. From 9c70d4acf9aea4ce9781c72dc14814660d79a840 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 22:26:29 +0800 Subject: [PATCH 265/334] [fix](catalog) hive: honor OpenX JSON ignore.malformed.json (skip malformed rows) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A plugin-driven Hive JSON table declared with the OpenX serde property ignore.malformed.json=true silently lost the flag, so BE treated malformed rows as an error (DATA_QUALITY_ERROR) instead of skipping them. Legacy HiveScanNode set TFileAttributes.openx_json_ignore_malformed from that serde/table property (default false, table-param precedence) in its OPENX_JSON_SERDE branch; the plugin path never read it. Fix mirrors legacy exactly, over the existing hive.text.* scan-attribute channel: - HiveTextProperties.extractJsonSerDeProps now takes sd/table params and, ONLY for the OpenX serde, emits hive.text.openx_ignore_malformed (ignore.malformed.json resolved table-param-over-serde, default "false"). hcatalog/hive2 JSON serdes never carried the flag, so the key is scoped to OpenX for exact legacy parity. - PluginDrivenScanNode.getFileAttributes reads that key in the existing is_json block and calls setOpenxJsonIgnoreMalformed(Boolean.parseBoolean(...)) — same posture as the adjacent is_json handling (no new source-specific branch). One-column mode (read_hive_json_in_one_column) is unaffected: OpenX resolves to CSV format there, and BE's CSV reader never consults the JSON-only field. Design red-teamed (3 lenses): the initial all-JSON-serdes draft was refuted (legacy is OpenX-only) and folded to the OpenX gate before implementation; test/regression and iron-rule lenses SOUND. All 5 test tables use OpenX serde and only json_table_ignore_malformed sets the flag, so siblings keep erroring. Tests: HiveTextPropertiesTest +4 (true emitted / defaults false / table-param precedence / hcatalog omits key) — 13/13 green. fe-connector-hive + fe-core BUILD SUCCESS, 0 checkstyle. Fixes test_hive_openx_json order_qt_q1 (live-gated). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveTextProperties.java | 18 +++++++-- .../hive/HiveTextPropertiesTest.java | 38 +++++++++++++++++++ .../datasource/PluginDrivenScanNode.java | 6 +++ 3 files changed, 59 insertions(+), 3 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java index f4a544cbc75aec..f1df5fb065ec36 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java @@ -64,6 +64,10 @@ public final class HiveTextProperties { private static final String ESCAPE_DELIM = "escape.delim"; private static final String SERIALIZATION_NULL_FORMAT = "serialization.null.format"; private static final String SKIP_HEADER_LINE_COUNT = "skip.header.line.count"; + // OpenX JSON serde: skip malformed rows instead of erroring. Mirrors legacy + // HiveProperties.PROP_OPENX_IGNORE_MALFORMED_JSON / DEFAULT_OPENX_IGNORE_MALFORMED_JSON. + private static final String IGNORE_MALFORMED_JSON = "ignore.malformed.json"; + private static final String DEFAULT_IGNORE_MALFORMED_JSON = "false"; // Default delimiters, mirroring legacy HiveProperties. These are byte values (Ctrl-A etc.), not the // literal digit characters Hive stores them as in serialization.format / *.delim SerDe params. @@ -108,7 +112,7 @@ public static Map extract(String serDeLib, extractCsvSerDeProps(sdParams, result); } else if (HIVE_JSON_SERDE.equals(serDeLib) || LEGACY_HIVE_JSON_SERDE.equals(serDeLib) || OPENX_JSON_SERDE.equals(serDeLib)) { - extractJsonSerDeProps(serDeLib, result); + extractJsonSerDeProps(serDeLib, sdParams, tableParams, result); } else { return result; } @@ -159,12 +163,20 @@ private static void extractCsvSerDeProps(Map params, result.put(PROP_PREFIX + "null_format", ""); } - private static void extractJsonSerDeProps(String serDeLib, - Map result) { + private static void extractJsonSerDeProps(String serDeLib, Map sdParams, + Map tableParams, Map result) { result.put(PROP_PREFIX + "column_separator", "\t"); result.put(PROP_PREFIX + "line_delimiter", "\n"); result.put(PROP_PREFIX + "is_json", "true"); result.put(PROP_PREFIX + "json_serde_lib", serDeLib); + // OpenX-only: skip malformed rows when the serde/table sets ignore.malformed.json (table-param over + // sd-param, default false). Mirrors legacy HiveScanNode's OPENX_JSON_SERDE branch — the hcatalog/hive2 + // JSON serdes never carried this flag, so scope it to OpenX to keep exact legacy branch parity. + if (OPENX_JSON_SERDE.equals(serDeLib)) { + String ignoreMalformed = serdeVal(sdParams, tableParams, IGNORE_MALFORMED_JSON); + result.put(PROP_PREFIX + "openx_ignore_malformed", + ignoreMalformed != null ? ignoreMalformed : DEFAULT_IGNORE_MALFORMED_JSON); + } } private static String getFieldDelimiter(Map sdParams, diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java index 280db47c8f5c1c..06df85f9ae0b0a 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveTextPropertiesTest.java @@ -38,6 +38,8 @@ public class HiveTextPropertiesTest { private static final String TEXT_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; private static final String MULTI_DELIMIT_SERDE = "org.apache.hadoop.hive.serde2.MultiDelimitSerDe"; + private static final String OPENX_JSON_SERDE = "org.openx.data.jsonserde.JsonSerDe"; + private static final String HCATALOG_JSON_SERDE = "org.apache.hive.hcatalog.data.JsonSerDe"; private static final String PREFIX = "hive.text."; private static String colSep(String serde, Map sd) { @@ -111,4 +113,40 @@ public void testTableParamsTakePrecedenceOverSerdeParams() { String sep = HiveTextProperties.extract(TEXT_SERDE, sdParams, tblParams).get(PREFIX + "column_separator"); Assertions.assertEquals(",", sep); } + + // ---- OpenX JSON ignore.malformed.json (legacy HiveScanNode OPENX_JSON_SERDE branch parity) ---- + + @Test + public void testOpenxJsonIgnoreMalformedTrueEmitted() { + // OpenX table with SERDEPROPERTIES ignore.malformed.json=true -> BE must skip malformed rows. + Map r = HiveTextProperties.extract( + OPENX_JSON_SERDE, sd("ignore.malformed.json", "true"), new HashMap<>()); + Assertions.assertEquals("true", r.get(PREFIX + "openx_ignore_malformed")); + Assertions.assertEquals("true", r.get(PREFIX + "is_json")); + } + + @Test + public void testOpenxJsonIgnoreMalformedDefaultsFalse() { + // No property -> "false" (== Thrift default): malformed rows still error, matching legacy default. + Map r = HiveTextProperties.extract(OPENX_JSON_SERDE, sd(), new HashMap<>()); + Assertions.assertEquals("false", r.get(PREFIX + "openx_ignore_malformed")); + } + + @Test + public void testOpenxJsonIgnoreMalformedTableParamPrecedence() { + // Table param (true) beats serde param (false), mirroring legacy getSerdeProperty precedence. + Map tblParams = new HashMap<>(); + tblParams.put("ignore.malformed.json", "true"); + Map r = HiveTextProperties.extract( + OPENX_JSON_SERDE, sd("ignore.malformed.json", "false"), tblParams); + Assertions.assertEquals("true", r.get(PREFIX + "openx_ignore_malformed")); + } + + @Test + public void testHcatalogJsonSerdeOmitsOpenxKey() { + // Non-OpenX JSON serde never carried this flag: the key must be absent (legacy set it only for OpenX). + Map r = HiveTextProperties.extract(HCATALOG_JSON_SERDE, sd(), new HashMap<>()); + Assertions.assertFalse(r.containsKey(PREFIX + "openx_ignore_malformed")); + Assertions.assertEquals("true", r.get(PREFIX + "is_json")); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index cbf50c9c5fbbb2..d9895db49d5c93 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -648,6 +648,12 @@ protected TFileAttributes getFileAttributes() throws UserException { if ("true".equals(isJson)) { attrs.setReadJsonByLine(true); attrs.setReadByColumnDef(true); + // OpenX JSON "ignore.malformed.json": skip malformed rows instead of erroring. The connector emits + // this only for the OpenX serde (absent otherwise); mirrors legacy HiveScanNode's openx branch. + String ignoreMalformed = props.get(PROP_HIVE_TEXT_PREFIX + "openx_ignore_malformed"); + if (ignoreMalformed != null) { + attrs.setOpenxJsonIgnoreMalformed(Boolean.parseBoolean(ignoreMalformed)); + } } return attrs; From ca2ac90f1e0a2388082129cf34be9a2e8de534a9 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 22:27:10 +0800 Subject: [PATCH 266/334] [doc](catalog) hive-e2e: land FIX-R10 design + red-team, mark R10 done, repoint to R12/serde R10 (OpenX ignore.malformed.json) design doc with the 3-lens red-team result (C1 all-serdes draft refuted -> folded to OpenX-only gate; C2/C3 SOUND), and HANDOFF/task-list updates (R7+R10 done, 11 suites remaining, next = R12/serde OpenCSV all-STRING schema). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 7 +- plan-doc/task-list-hive-e2e-r2-fixes.md | 2 +- plan-doc/tasks/designs/FIX-R10-design.md | 145 +++++++++++++++++++++++ 3 files changed, 150 insertions(+), 4 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-R10-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index d608719dc91102..508e3f8fa0b639 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -12,10 +12,11 @@ > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > > **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **✅ batch-3（本 session）：R7 DONE `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。设计红队 4 lens 全 SOUND(legacy parity=旧 ThriftHMSCachedClient 就是 fresh 直 RPC；caller completeness；decorator stack 单层；MTMV 旧亦读缓存)。`CachingHmsClientTest` 19/19、两模块 BUILD SUCCESS、0 checkstyle。设计 `designs/FIX-R7-design.md`、清单 `task-list-hive-e2e-r2-fixes.md`。**e2e 待用户重部署自跑 test_hive_use_meta_cache_true sql09。** -> **⭐ 剩余 12 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> **✅ batch-3（本 session，2 fix DONE）**： +> - **R7 `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。红队 4 lens 全 SOUND。`CachingHmsClientTest` 19/19。设计 `designs/FIX-R7-design.md`。**e2e 待用户自跑 test_hive_use_meta_cache_true sql09。** +> - **R10 `9c70d4acf9a`**——OpenX JSON `ignore.malformed.json` 跳坏行。`HiveTextProperties.extractJsonSerDeProps` 串入 sd/table params，**仅 OpenX serde**发 `hive.text.openx_ignore_malformed`(table-param 优先、默认 false；hcatalog/hive2 从不带此旗，钉 OpenX 保 legacy 分支 parity) + fe-core `PluginDrivenScanNode.getFileAttributes` 在既有 `is_json` 块读键调 `setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode OPENX 分支)。one-column 模式 OpenX 解析成 CSV、BE CSV reader 不读该 JSON-only 字段故无碍。红队 3 lens(初版"全 3 JSON serde"被驳→已收敛 OpenX-only；test/iron-rule SOUND)。`HiveTextPropertiesTest` 13/13。设计 `designs/FIX-R10-design.md`。**e2e 待用户自跑 test_hive_openx_json q1。** +> **⭐ 剩余 11 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： > - **fe-connector 中型**： -> - **R10** openx json ignore.malformed：`HiveTextProperties.extractJsonSerDeProps`(:162, 现只收 serDeLib，调用点 :111 有 sdParams/tableParams 可用)串入 serde `ignore.malformed.json`→发 `hive.text.openx_ignore_malformed`；fe-core `PluginDrivenScanNode`(:633-637)调 `TFileAttributes.setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode:646-647)。 > - **R12/serde** OpenCSV 走 serde 解析成全 STRING(test_open_csv_serde + **test_hive_serde_prop 同根因**)：`ThriftHmsClient.convertTable`(:654-656)当前用原始 `sd.getCols()` 声明类型；须像 legacy `initHiveSchema` 经 `getSchema()`(SerDe ObjectInspector, OpenCSVSerde 强制全列 string) 解析，拆 data cols vs partition keys。**⚠ blast radius 大**(改所有 hive 表 schema 解析路径)，须红队 + 广回归。别在 fe-core 按 serde 名分支。 > - **text_write** LZ4FRAME→LZ4BLOCK 读修正：`HiveScanPlanProvider` 恢复 legacy HiveScanNode:670-678 的 `getFileCompressType` override(`.lz4`→LZ4FRAME 需转 LZ4BLOCK)；为守 fe-core connector-agnostic，加 `ConnectorScanPlanProvider` default 压缩类型 hook(identity)。 > - **fe-connector/fe-core 大型**：R2 SHOW CREATE TABLE 原生 DDL(show_create_table + ddl_text_format；HiveConnector 声明 `SUPPORTS_SHOW_CREATE_DDL` + 原生 DDL 渲染器镜像 legacy HiveMetaStoreClientHelper.showCreateTable，注意测试断言单引号) · R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 diff --git a/plan-doc/task-list-hive-e2e-r2-fixes.md b/plan-doc/task-list-hive-e2e-r2-fixes.md index b9be9825733610..013211a5f7caf1 100644 --- a/plan-doc/task-list-hive-e2e-r2-fixes.md +++ b/plan-doc/task-list-hive-e2e-r2-fixes.md @@ -15,7 +15,7 @@ Discipline per task: 设计(`tasks/designs/FIX--design.md`) → 设计红队 ## Remaining code fixes ### fe-connector 中型 - [x] R7 SHOW PARTITIONS bypass cache (test_hive_use_meta_cache_true) `4df95ad44ac` — design red-teamed 4/4 SOUND, UT 19/19 -- [ ] R10 openx json ignore.malformed (test_hive_openx_json) +- [x] R10 openx json ignore.malformed (test_hive_openx_json) `9c70d4acf9a` — openx-only gate, red-teamed 3-lens, UT 13/13 - [ ] R12/serde OpenCSV all-STRING schema (test_open_csv_serde, test_hive_serde_prop) - [ ] text_write LZ4FRAME→LZ4BLOCK read (test_hive_text_write_insert) diff --git a/plan-doc/tasks/designs/FIX-R10-design.md b/plan-doc/tasks/designs/FIX-R10-design.md new file mode 100644 index 00000000000000..d68f21f3a03737 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-R10-design.md @@ -0,0 +1,145 @@ +# FIX-R10 — OpenX JSON `ignore.malformed.json` not honored + +Suite: `test_hive_openx_json` (order_qt_q1). Effort: M. Scope: fe-connector-hive + fe-core (1 mirror line). + +## Problem + +A Hive JSON table declared with OpenX serde property `ignore.malformed.json=true` should skip malformed +rows; a query over it returns the well-formed rows. On the plugin-driven path the flag is dropped, so BE +treats malformed rows as an error and the query fails: + +``` +select * from json_table -> DATA_QUALITY_ERROR (expected — no ignore flag) +select * from json_table_ignore_malformed -> DATA_QUALITY_ERROR (WRONG — flag=true dropped; expected: rows) +``` + +`test_hive_openx_json` `order_qt_q1` (`select * from json_table_ignore_malformed`) errors instead of +returning golden rows. + +## Root Cause (HEAD-verified) + +Legacy `HiveScanNode:646-647` set the BE Thrift attribute — inside the **`OPENX_JSON_SERDE` branch only** +(HiveScanNode:632 `else if serDeLib == OPENX_JSON_SERDE`), and only on the **non-one-column** sub-path +(`if (!isReadHiveJsonInOneColumn())`): +```java +fileAttributes.setOpenxJsonIgnoreMalformed( + Boolean.parseBoolean(HiveProperties.getOpenxJsonIgnoreMalformed(table))); +``` +`HiveProperties.getOpenxJsonIgnoreMalformed` reads serde/table property `ignore.malformed.json` via +`HiveMetaStoreClientHelper.getSerdeProperty` = `firstNonNullable(tableParam, sdParam)` (table-param +precedence), default `"false"`. The plain `HIVE_JSON_SERDE`/`LEGACY_HIVE_JSON_SERDE` branches never set it. + +On the plugin path: +- The connector's `HiveTextProperties.extractJsonSerDeProps` (:162) receives only `serDeLib` — it emits + `is_json` / `json_serde_lib` but **never reads `ignore.malformed.json`**, even though the call site + (`extract`, :111) already holds `sdParams` and `tableParams`. +- fe-core `PluginDrivenScanNode.getFileAttributes` (:647-651) sets `readJsonByLine`/`readByColumnDef` from + `hive.text.is_json` but **never calls `setOpenxJsonIgnoreMalformed`**. + +So `TFileAttributes.openx_json_ignore_malformed` stays at its Thrift default `false` for every plugin hive +JSON table → malformed rows always error. + +## Design + +Mirror legacy exactly, using the existing `hive.text.*` property channel (the established connector→fe-core +scan-attribute transport; not a new mechanism). + +### 1. `HiveTextProperties` (fe-connector-hive) + +- New key constant `IGNORE_MALFORMED_JSON = "ignore.malformed.json"` and `DEFAULT_IGNORE_MALFORMED_JSON = + "false"` (mirrors `HiveProperties.PROP_OPENX_IGNORE_MALFORMED_JSON` / default). +- Thread `sdParams` + `tableParams` into `extractJsonSerDeProps` (call site :111 already has them), and emit + the flag **only for the OpenX serde** (legacy fidelity — the other JSON branches never set it): + ```java + private static void extractJsonSerDeProps(String serDeLib, Map sdParams, + Map tableParams, Map result) { + result.put(PROP_PREFIX + "column_separator", "\t"); + result.put(PROP_PREFIX + "line_delimiter", "\n"); + result.put(PROP_PREFIX + "is_json", "true"); + result.put(PROP_PREFIX + "json_serde_lib", serDeLib); + // OpenX-only: ignore.malformed.json (table-param over sd-param, default false) — mirrors legacy + // HiveScanNode's OPENX_JSON_SERDE branch. hcatalog/hive2 JSON serdes never carried this flag. + if (OPENX_JSON_SERDE.equals(serDeLib)) { + String ignoreMalformed = serdeVal(sdParams, tableParams, IGNORE_MALFORMED_JSON); + result.put(PROP_PREFIX + "openx_ignore_malformed", + ignoreMalformed != null ? ignoreMalformed : DEFAULT_IGNORE_MALFORMED_JSON); + } + } + ``` + +**One-column mode (`read_hive_json_in_one_column=true`) is a non-issue.** For OpenX in one-column mode +`HiveFileFormat.detect` resolves the format to **CSV** (single-column line read), so BE uses the CSV reader, +which never consults `openx_json_ignore_malformed` (a JSON-reader-only field). Legacy likewise did not set it +on that sub-path. So even though the connector still emits the key (extract dispatches on serde), setting the +Thrift field has no BE effect in one-column mode — no explicit fe-core one-column gate needed. The failing +assertion (`order_qt_q1`) runs in the default (non-one-column) JSON mode. + +### 2. `PluginDrivenScanNode.getFileAttributes` (fe-core) + +Inside the existing `if ("true".equals(isJson))` block (:648-651), add the mirror of legacy :646-647: +```java +String ignoreMalformed = props.get(PROP_HIVE_TEXT_PREFIX + "openx_ignore_malformed"); +if (ignoreMalformed != null) { + attrs.setOpenxJsonIgnoreMalformed(Boolean.parseBoolean(ignoreMalformed)); +} +``` +Placed with the other JSON attrs, keyed off the connector-emitted `hive.text.*` prop — the same +connector-agnostic transport fe-core already uses for `serde_lib` / `is_json` / delimiters. No new +source-specific branch (it conforms to the pre-existing `hive.text.*` reader; iron-rule status unchanged). + +## Risk Analysis + +- **Blast radius**: `HiveTextProperties` (private method signature — only caller is `extract:111`, updated in + the same edit), 1 new `PluginDrivenScanNode` read. No SPI/interface change. +- **`json_table` regression guard**: without `ignore.malformed.json`, the emitted value is `"false"` → + `setOpenxJsonIgnoreMalformed(false)` → BE still raises DATA_QUALITY_ERROR on malformed rows (the test's + first assertion stays satisfied). The fix flips behavior ONLY when the serde/table property is truthy. +- **Non-openx JSON serdes** (hcatalog / hive2): the key is not emitted at all (openx-only gate) → fe-core never + calls the setter → Thrift default false → no behavior change (matches legacy, which set it only in the openx + branch). +- **Boolean parse**: `Boolean.parseBoolean` — any non-"true" (case-insensitive) → false, exactly legacy. +- **fe-core iron rule**: reads an existing `hive.text.*`-prefixed key via the established transport; no new + `if(hive)`/source-name branch. Conforms to the current `getFileAttributes` design. + +## Test Plan + +### Unit Tests (fe-connector-hive — `HiveTextPropertiesTest`) +1. `testOpenxJsonIgnoreMalformedTrueEmitted`: OpenX serde + sdParam `ignore.malformed.json=true` → + result `hive.text.openx_ignore_malformed == "true"`, `is_json == "true"`. +2. `testOpenxJsonIgnoreMalformedDefaultsFalse`: OpenX serde, no property → `"false"`. +3. `testOpenxJsonIgnoreMalformedTableParamPrecedence`: table-param `true` beats sdParam `false` → `"true"` + (mirrors serdeVal precedence, matching legacy getSerdeProperty). +4. `testHcatalogJsonSerdeOmitsOpenxKey`: hcatalog JSON serde → result has NO `openx_ignore_malformed` key + (openx-only gate), but still `is_json == "true"`. + +### E2E +`test_hive_openx_json` `order_qt_q1` returns golden rows; `json_table` still throws DATA_QUALITY_ERROR. +Live-gated (external hive docker); user reruns. No golden change (goldens already encode the correct rows). + +fe-core plumbing (prop → `setOpenxJsonIgnoreMalformed`) is a 3-line mirror of the untested-in-isolation +`is_json` block (node not bare-constructable per test-infra notes); covered by e2e + the connector UT. + +## Verification +- `mvn -o -f fe/pom.xml -pl :fe-connector-hive -am test` `-Dtest=HiveTextPropertiesTest`. +- `mvn -o -f fe/pom.xml -pl fe-core -am test-compile` (getFileAttributes change compiles). +- 0 checkstyle; import gate clean. + +## Red-team result (wf_64f26946-e33, 3 lenses) +- **C1 legacy parity — caught + folded.** The reviewer refuted the *original* draft (which emitted the flag for + all three JSON serdes) as UNSOUND: legacy sets `setOpenxJsonIgnoreMalformed` ONLY in the `OPENX_JSON_SERDE` + branch (HiveScanNode:631-647), not hcatalog/hive2. **This design now gates emission on `OPENX_JSON_SERDE` + only** (folded before implementation). The other four dimensions confirmed exact against HEAD: key + `ignore.malformed.json` (HiveProperties:65), default `false` (:66), table-over-sd precedence + (getSerdeProperty `firstNonNullable(tbl,sd)` == `serdeVal`), `Boolean.parseBoolean`→`setOpenxJsonIgnoreMalformed` + (Thrift default false, PlanNodes.thrift:309; setter exists TFileAttributes:550). One-column `!isReadHiveJsonInOneColumn` + gate: not replicated, but harmless — one-column openx resolves to CSV format so BE's CSV reader never reads the + JSON-only field (see design note above). +- **C2 test/regression — SOUND.** All 5 `openx_json` tables use the OpenX serde; only `json_table_ignore_malformed` + carries `ignore.malformed.json=true` (SERDEPROPERTIES → sdParams; `run76.hql:94`). So `serdeVal` emits `true` + only for q1's table, `false` for the rest → they keep raising DATA_QUALITY_ERROR (explicit-false == current + never-set, Thrift default false); `read_hive_json_in_one_column` order_qt_2/3 unaffected. Private + `extractJsonSerDeProps` has exactly one caller (extract:111), public signature unchanged, no test asserts on the + JSON prop-map. q1 golden (11 all-null + 2 data rows) is BE-rendered, live-gated. +- **C3 iron-rule/scope — SOUND.** `getFileAttributes` already reads ~10 source-named `hive.text.*` keys + (serde_lib/is_json/delimiters/…); one more read is the identical established connector→fe-core channel, same + posture as `is_json`. No new source-name branch; property resolution stays connector-side. Iron rule unchanged. From 3936434bc9aa4c20060548e375bd0e6ee3e29754 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 23:03:27 +0800 Subject: [PATCH 267/334] [fix](catalog) hive: flatten OpenCSVSerde table columns to STRING (serde-resolved schema parity) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit OpenCSVSerde reads a delimited file as PLAIN text: its deserializer reports every top-level column as string, so a declared int/date/boolean — or even an array/map/struct — is served verbatim as a string, never parsed. Legacy resolved this via the metastore get_schema RPC (server-side deserializer → all-string); the SPI connector reads raw sd.getCols() declared types instead, so OpenCSV columns reached BE typed and flipped TRUE vs 'true', raw datetime vs ISO, empty-string vs NULL. Fixes test_open_csv_serde and the OpenCSV half of test_hive_serde_prop. Reproduce the RESULT connector-side without the extra RPC: HiveConnectorMetadata .buildColumns flattens every DATA column to STRING when the serde is OpenCSVSerde (coerceOpenCsvColumnsToString), reusing the existing fe-connector-hive HiveTextProperties.HIVE_OPEN_CSV_SERDE constant. Placed in the hive metadata layer (not the shared fe-connector-hms ThriftHmsClient, which also feeds the hudi connector) so the serde-specific typing stays off the shared path — mirroring where Trino applies CSV=all-string. Partition keys are untouched (hive appends them after the deserializer, so they keep declared types); a view is guarded out. Non-OpenCSV tables return byte-identical to the raw sd.getCols() path, so every other serde is unaffected. typeMappingOptions are moot for the flattened columns (matching legacy). fe-core unchanged; no serde-name branch in fe-core. Tests: HiveConnectorMetadataSchemaTest +4 (flatten int/datetime/boolean/array; partition key keeps date; LazySimple gate stays typed; view guard). RED-verified (bypass -> expected STRING but was INT). fe-connector-hive 306/306, 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveConnectorMetadata.java | 60 +++++++++++++---- .../hive/HiveConnectorMetadataSchemaTest.java | 65 +++++++++++++++++++ 2 files changed, 112 insertions(+), 13 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 5a51e6467d3fde..9d346bca6e9210 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1839,22 +1839,23 @@ private List buildColumns(HmsTableInfo tableInfo) { } // HmsTableInfo already returns ConnectorColumn with types mapped by HmsTypeMapping // during ThriftHmsClient.getTable(). Enrich with default values if available. + List columns = spiColumns; Map defaults = getDefaultValues(tableInfo); - if (defaults.isEmpty()) { - return spiColumns; - } - List enriched = new ArrayList<>(spiColumns.size()); - for (ConnectorColumn col : spiColumns) { - String defaultVal = defaults.get(col.getName()); - if (defaultVal != null && col.getDefaultValue() == null) { - enriched.add(new ConnectorColumn( - col.getName(), col.getType(), col.getComment(), - col.isNullable(), defaultVal)); - } else { - enriched.add(col); + if (!defaults.isEmpty()) { + List enriched = new ArrayList<>(spiColumns.size()); + for (ConnectorColumn col : spiColumns) { + String defaultVal = defaults.get(col.getName()); + if (defaultVal != null && col.getDefaultValue() == null) { + enriched.add(new ConnectorColumn( + col.getName(), col.getType(), col.getComment(), + col.isNullable(), defaultVal)); + } else { + enriched.add(col); + } } + columns = enriched; } - return enriched; + return coerceOpenCsvColumnsToString(tableInfo, columns); } private List buildPartitionKeys(HmsTableInfo tableInfo) { @@ -1894,6 +1895,39 @@ private static List coercePartitionKeyStringToVarchar(List coerceOpenCsvColumnsToString( + HmsTableInfo tableInfo, List columns) { + if (isView(tableInfo) + || !HiveTextProperties.HIVE_OPEN_CSV_SERDE.equals(tableInfo.getSerializationLib())) { + return columns; + } + ConnectorType stringType = ConnectorType.of("STRING"); + List forced = new ArrayList<>(columns.size()); + for (ConnectorColumn col : columns) { + forced.add(new ConnectorColumn(col.getName(), stringType, col.getComment(), + col.isNullable(), col.getDefaultValue(), + col.isKey(), col.isAutoInc(), col.isAggregated())); + } + return forced; + } + private Map getDefaultValues(HmsTableInfo tableInfo) { try { return hmsClient.getDefaultColumnValues( diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index ace212ae256d4e..4173ed361b0225 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -58,6 +58,8 @@ public class HiveConnectorMetadataSchemaTest { "org.apache.hadoop.hive.ql.io.orc.OrcInputFormat"; private static final String TEXT_INPUT_FORMAT = "org.apache.hadoop.mapred.TextInputFormat"; + private static final String OPEN_CSV_SERDE = "org.apache.hadoop.hive.serde2.OpenCSVSerde"; + private static final String LAZY_SIMPLE_SERDE = "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe"; private ConnectorTableSchema schemaOf(HmsTableInfo tableInfo) { HiveConnectorMetadata metadata = new HiveConnectorMetadata( @@ -89,6 +91,27 @@ private static HmsTableInfo.Builder unpartitionedTable(String inputFormat) { .parameters(Collections.emptyMap()); } + private static ConnectorColumn arrayCol(String name, String elementTypeName) { + return new ConnectorColumn(name, + ConnectorType.arrayOf(ConnectorType.of(elementTypeName)), null, true, null); + } + + /** + * A delimited-text table with DECLARED TYPED data columns (int/datetime/boolean + a complex array) and a + * non-string DATE partition key, parameterized by serde lib. Under OpenCSVSerde the reader serves every data + * column as plain string; under any other serde the declared types stand. + */ + private static HmsTableInfo.Builder csvTypedTable(String serdeLib) { + return HmsTableInfo.builder() + .dbName("db").tableName("t") + .inputFormat(TEXT_INPUT_FORMAT) + .serializationLib(serdeLib) + .columns(Arrays.asList(col("id", "INT"), col("ts", "DATETIMEV2"), + col("active", "BOOLEAN"), arrayCol("arr_col", "INT"))) + .partitionKeys(Collections.singletonList(col("dt", "DATEV2"))) + .parameters(Collections.emptyMap()); + } + private static String perTableCapabilities(ConnectorTableSchema schema) { return schema.getProperties().get(ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY); } @@ -135,6 +158,48 @@ public void testStringDataColumnIsNotWidened() { Assertions.assertEquals("STRING", columnByName(schema, "name").getType().getTypeName()); } + @Test + public void testOpenCsvSerdeFlattensEveryDataColumnToString() { + // WHY: OpenCSVSerde reads the file as PLAIN text — its deserializer reports every top-level column as + // string, so a declared int/datetime/boolean, and even a complex array/map/struct, is served verbatim as + // a string and never parsed. Legacy resolved this via the metastore get_schema RPC (all-string); the SPI + // reads raw sd.getCols() types, so the connector must reproduce the all-string RESULT. MUTATION: emitting + // the declared typed columns flips TRUE vs 'true', raw datetime vs ISO, empty-string vs NULL — the exact + // regression in test_open_csv_serde / test_hive_serde_prop. + ConnectorTableSchema schema = schemaOf(csvTypedTable(OPEN_CSV_SERDE).build()); + for (String dataCol : Arrays.asList("id", "ts", "active", "arr_col")) { + Assertions.assertEquals("STRING", columnByName(schema, dataCol).getType().getTypeName(), + "OpenCSV data column " + dataCol + " must be flattened to STRING"); + } + } + + @Test + public void testOpenCsvSerdePartitionKeyKeepsDeclaredType() { + // Partition keys are appended by hive AFTER the deserializer, so they keep their declared types: a DATE + // partition key stays a date, NOT string-forced. Guards against flattening the whole schema. + ConnectorTableSchema schema = schemaOf(csvTypedTable(OPEN_CSV_SERDE).build()); + Assertions.assertEquals("DATEV2", columnByName(schema, "dt").getType().getTypeName()); + } + + @Test + public void testNonOpenCsvSerdeKeepsDeclaredDataTypes() { + // The flatten is OpenCSV-gated: an identical table under LazySimpleSerDe (plain text, typed columns) keeps + // its declared int/datetime types — the LazySimple half of test_hive_serde_prop, which must stay typed. + // MUTATION: an ungated flatten would break every typed text/parquet/orc table. + ConnectorTableSchema schema = schemaOf(csvTypedTable(LAZY_SIMPLE_SERDE).build()); + Assertions.assertEquals("INT", columnByName(schema, "id").getType().getTypeName()); + Assertions.assertEquals("DATETIMEV2", columnByName(schema, "ts").getType().getTypeName()); + } + + @Test + public void testOpenCsvViewColumnsAreNotFlattened() { + // buildColumns also serves getViewDefinition; a view's logical columns are never an OpenCSV data payload, + // so the isView guard keeps them at declared types even when the SD carries an OpenCSV serde lib. + ConnectorTableSchema schema = schemaOf( + csvTypedTable(OPEN_CSV_SERDE).tableType("VIRTUAL_VIEW").build()); + Assertions.assertEquals("INT", columnByName(schema, "id").getType().getTypeName()); + } + @Test public void testPartitionColumnsComeAfterDataColumns() { ConnectorTableSchema schema = schemaOf(partitionedTable().build()); From 85a28522610ae7e574e9cea4fdaf6bfba2d55713 Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 23:05:08 +0800 Subject: [PATCH 268/334] [doc](catalog) hive-e2e: land FIX-R12 design + red-team, mark R12 done, repoint to text_write MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Records the OpenCSV serde schema-flatten fix (3936434bc9a): root cause (legacy default get_schema RPC vs new connector raw sd.getCols()), the A-vs-B decision (B, user signed off — connector STRING-force, no RPC, Trino-aligned), the architecture red-team relocation into HiveConnectorMetadata (off the shared hms layer hudi consumes), residual Avro-schema-url gap, and the 4-suite acceptance gate. HANDOFF: R12 moved into batch-3 DONE, remaining 11 -> 9 suites. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 6 +- plan-doc/tasks/designs/FIX-R12-design.md | 133 +++++++++++++++++++++++ 2 files changed, 136 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-R12-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 508e3f8fa0b639..92a517c384efd3 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -12,12 +12,12 @@ > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > > **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **✅ batch-3（本 session，2 fix DONE）**： +> **✅ batch-3（R7/R10 前序 session、R12 本 session，共 3 fix DONE）**： > - **R7 `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。红队 4 lens 全 SOUND。`CachingHmsClientTest` 19/19。设计 `designs/FIX-R7-design.md`。**e2e 待用户自跑 test_hive_use_meta_cache_true sql09。** > - **R10 `9c70d4acf9a`**——OpenX JSON `ignore.malformed.json` 跳坏行。`HiveTextProperties.extractJsonSerDeProps` 串入 sd/table params，**仅 OpenX serde**发 `hive.text.openx_ignore_malformed`(table-param 优先、默认 false；hcatalog/hive2 从不带此旗，钉 OpenX 保 legacy 分支 parity) + fe-core `PluginDrivenScanNode.getFileAttributes` 在既有 `is_json` 块读键调 `setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode OPENX 分支)。one-column 模式 OpenX 解析成 CSV、BE CSV reader 不读该 JSON-only 字段故无碍。红队 3 lens(初版"全 3 JSON serde"被驳→已收敛 OpenX-only；test/iron-rule SOUND)。`HiveTextPropertiesTest` 13/13。设计 `designs/FIX-R10-design.md`。**e2e 待用户自跑 test_hive_openx_json q1。** -> **⭐ 剩余 11 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **R12/serde `3936434bc9a`**——OpenCSV 表列全塌成 STRING(修 test_open_csv_serde + test_hive_serde_prop 的 OpenCSV 半)。根因=legacy 默认走 metastore `get_schema` RPC(服务端反序列化器→OpenCSV 全 string)，新连接器只读原始 `sd.getCols()` 声明类型→typed 列到 BE 翻车(TRUE vs 'true'、datetime→ISO、空串 vs NULL)。**方案定案=B(用户 2026-07-11 签字)**：不走 RPC、连接器就地复刻结果——`HiveConnectorMetadata.buildColumns` 加 `coerceOpenCsvColumnsToString`,serde==OpenCSV 时把**数据列**整型别塌成 `STRING`(复合类型也整列塌)、复用现成 `HiveTextProperties.HIVE_OPEN_CSV_SERDE` 常量。**架构红队关键订正**：放 hive 元数据层(非 hive/hudi 共用的 `ThriftHmsClient`,那会误触 hudi schema 路径 + 重复常量)——对齐 Trino(在 HiveMetadata 层做 CSV=全 string)。分区列不动(hive 在反序列化后追加、保声明类型)、view 有 isView 门。非-OpenCSV 表零改动(与原 `sd.getCols()` 逐位一致)。fe-core 零改、无 serde 名分支。3 路对抗红队(correctness SOUND / architecture 订正放置层 / blast 把验收扩到 4 suite)。`HiveConnectorMetadataSchemaTest` +4(int/datetime/bool/**array**→STRING、分区键留 date、LazySimple 门留 typed、view 门)、RED 实证(旁路→expected STRING but was INT)、306/306、0 checkstyle。设计 `designs/FIX-R12-design.md`。**残余(非本修)**：Avro schema-url 自述表 A/B 唯一分歧,新连接器**现状本就没有**、不在失败套件、以后需要时独立小任务补。**e2e 待用户自跑：test_open_csv_serde + test_hive_serde_prop(+对照 test_drop_expired_table_stats/test_trino_hive_serde_prop)。** +> **⭐ 剩余 9 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： > - **fe-connector 中型**： -> - **R12/serde** OpenCSV 走 serde 解析成全 STRING(test_open_csv_serde + **test_hive_serde_prop 同根因**)：`ThriftHmsClient.convertTable`(:654-656)当前用原始 `sd.getCols()` 声明类型；须像 legacy `initHiveSchema` 经 `getSchema()`(SerDe ObjectInspector, OpenCSVSerde 强制全列 string) 解析，拆 data cols vs partition keys。**⚠ blast radius 大**(改所有 hive 表 schema 解析路径)，须红队 + 广回归。别在 fe-core 按 serde 名分支。 > - **text_write** LZ4FRAME→LZ4BLOCK 读修正：`HiveScanPlanProvider` 恢复 legacy HiveScanNode:670-678 的 `getFileCompressType` override(`.lz4`→LZ4FRAME 需转 LZ4BLOCK)；为守 fe-core connector-agnostic，加 `ConnectorScanPlanProvider` default 压缩类型 hook(identity)。 > - **fe-connector/fe-core 大型**：R2 SHOW CREATE TABLE 原生 DDL(show_create_table + ddl_text_format；HiveConnector 声明 `SUPPORTS_SHOW_CREATE_DDL` + 原生 DDL 渲染器镜像 legacy HiveMetaStoreClientHelper.showCreateTable，注意测试断言单引号) · R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 > - **用户已裁决(A)的 3 个 fe-core/SPI 大改**： diff --git a/plan-doc/tasks/designs/FIX-R12-design.md b/plan-doc/tasks/designs/FIX-R12-design.md new file mode 100644 index 00000000000000..60f3f4ab6cb015 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-R12-design.md @@ -0,0 +1,133 @@ +# FIX-R12 — OpenCSV serde schema flattened to all-STRING (connector-side, Trino-aligned) + +**Suites fixed:** `test_open_csv_serde`, `test_hive_serde_prop` (OpenCSV half). +**Scope:** `fe-connector-hive` only. fe-core untouched; shared `fe-connector-hms` untouched. +**Status:** code DONE (`HiveConnectorMetadata` + unit tests, 306 module UT + 4 new green, 0 checkstyle). e2e awaits user self-run. + +--- + +## 1. Symptom + +Two hive regression suites read wrong values from `OpenCSVSerde` (comma/quote-delimited plain-text) tables: +- `test_open_csv_serde` — 3 pure-OpenCSV tables (`run76.hql`), including a complex-type table whose `array/map/struct` + columns render as flat string leaves in the golden. +- `test_hive_serde_prop` — mixed: an **OpenCSV** half (`open_csv_default_prop`: `is_active` renders `TRUE` as raw + uppercase text, declared-typed columns render **empty string, not `\N`**) plus a **LazySimpleSerDe** half + (`serde_test8` → `\N`, `test_empty_null_format_text3`) that must stay typed + null-format-aware. + +If the FE declares these columns with their raw types (`int/date/boolean/…`), BE parses them → `TRUE` vs `'true'`, +raw datetime vs ISO, empty-string vs `NULL` (`IS NULL` vs `= ''`), complex columns mis-shaped. + +## 2. Root cause (HEAD-verified) + +Legacy `HMSExternalTable.initHiveSchema` (`HMSExternalTable.java:765`) resolved a table's columns **by default** +through the metastore **`get_schema` RPC** (`ThriftHMSCachedClient.getSchema:427` → `get_schema_with_environment_context`). +That RPC runs the table's SerDe **on the HMS server**: for serdes Hive does *not* keep columns in the metastore for +(OpenCSVSerde, RegexSerDe, …) it returns the deserializer's schema — and `OpenCSVSerde`'s ObjectInspector reports +**every top-level column as `string`**. Raw `sd.getCols()` was only the opt-in `get_schema_from_table=true` arm. + +The new connector `ThriftHmsClient.convertTable` (`ThriftHmsClient.java:653`) always reads raw `sd.getCols()` — i.e. it +behaves like legacy's *non-default* arm, so OpenCSV declared types (int/date/bool/complex) reach BE unflattened. That is +the regression. + +For the common serdes (LazySimple text, ORC, Parquet) `get_schema` == `sd.getCols()`, so those were never affected — +this is an OpenCSV-shaped problem, not an all-tables problem. + +## 3. Options considered + +| | **A — restore `get_schema` RPC (full legacy parity)** | **B — connector STRING-force on OpenCSV (chosen)** | +|---|---|---| +| Mechanism | add `getSchema` to `HmsClient` + `ThriftHmsClient`, call `get_schema` by default, wire a `get_schema_from_table` opt-out | in the hive metadata layer, when serde == OpenCSVSerde, flatten every DATA column to `STRING` | +| Fixes both suites | yes | yes | +| Extra RPC / table load | +1 | 0 | +| Partition-key re-split hazard | **yes** (`get_schema` concatenates partition keys; connector re-appends them → must strip) | none (partition keys never touched) | +| Server-side serde-jar dependency | **yes** (OpenX-JSON / contrib-MultiDelimit jar absent on HMS ⇒ `MetaException` breaks whole table — the exact failure `get_schema_from_table=true` exists to bypass) | none | +| Touches hudi-on-HMS schema path | **yes** (shared `HmsTableInfo.getColumns` feeds `HudiConnectorMetadata.getSchemaFromHms:902`) | no | +| Avro schema-url self-describing tables | resolves them (only A) | does not (see §7 — pre-existing gap, not worsened) | +| Trino alignment | against (Trino avoids `get_fields`/`get_schema`) | with (Trino forces CSV=all-VARCHAR in `HiveMetadata`, not the metastore client) | + +**Decision: Option B — user signed off 2026-07-11.** The end-state (OpenCSV → all-STRING, encoded in the goldens) is +already agreed; A and B differ only on Avro-schema-url, a table shape absent from the failing suites and unread by the +connector today. B is smaller, RPC-free, and does not fight the SPI grain. + +## 4. Placement (architecture red-team refinement) + +The first draft put the branch in `ThriftHmsClient.convertTable` (**fe-connector-hms**) — the shared raw-metastore +module that also feeds the **hudi** connector, and which has zero OpenCSV knowledge today (it would have needed a +mirrored FQCN constant). Relocated to **`HiveConnectorMetadata.buildColumns`** (fe-connector-hive), where: +- `HmsTableInfo.getSerializationLib()` is already in hand (already consumed at `HiveConnectorMetadata.java:383`); +- the sibling same-category op lives (`coercePartitionKeyStringToVarchar`, string→varchar coercion); +- the canonical OpenCSV constant already exists (`HiveTextProperties.HIVE_OPEN_CSV_SERDE`, same package — **no duplicate**); +- `buildColumns` is the single choke point for both consumers (`getTableSchema:415`, `getViewDefinition:613`). + +This keeps serde-specific typing off the hms module hudi shares, and mirrors the layer Trino applies CSV typing in. + +## 5. The change (`HiveConnectorMetadata.java`) + +`buildColumns` ends with `return coerceOpenCsvColumnsToString(tableInfo, columns);` (applied after default-value +enrichment). New helper: + +```java +private static List coerceOpenCsvColumnsToString( + HmsTableInfo tableInfo, List columns) { + if (isView(tableInfo) + || !HiveTextProperties.HIVE_OPEN_CSV_SERDE.equals(tableInfo.getSerializationLib())) { + return columns; // non-OpenCSV / view → byte-identical to sd.getCols() + } + ConnectorType stringType = ConnectorType.of("STRING"); + List forced = new ArrayList<>(columns.size()); + for (ConnectorColumn col : columns) { + forced.add(new ConnectorColumn(col.getName(), stringType, col.getComment(), + col.isNullable(), col.getDefaultValue(), + col.isKey(), col.isAutoInc(), col.isAggregated())); // mirrors coercePartitionKey rebuild + } + return forced; +} +``` + +- **Whole-type replacement, not primitive widening**: OpenCSV serves `array/map/struct` as one flat string too; the + golden `test_open_csv_serde.out` complex table confirms. Force to flat `STRING` (= what a declared `string` maps to, + `HmsTypeMapping` `"string"→ConnectorType.of("STRING")`), discarding the declared type entirely. +- **Partition keys untouched** — they come from `buildPartitionKeys` + `coercePartitionKeyStringToVarchar`, never through + this helper (hive appends partition keys after the deserializer, so legacy kept them typed too). No double-count. +- **`typeMappingOptions` (varbinary/timestamptz) moot for OpenCSV data columns** — every one becomes STRING before any + mapping flag could apply, matching legacy (OpenCSV all-string is produced before Doris type-mapping). Partition keys + still honor the options via `convertFieldSchemas`. +- **`firstColumnIsString(tableInfo)` (handle stamp, `:388`) reads raw columns — left as is**: its only consumer is the + OpenX-**JSON** one-column read gate; OpenCSV routes to `FORMAT_CSV_PLAIN` (`HiveFileFormat:160`) and never consults it, + so raw-vs-forced is inert here. (Noted so a future reader doesn't "fix" it.) +- **`isView` guard**: `buildColumns` also serves `getViewDefinition`; a view is never an OpenCSV data payload, guard + prevents flattening a view's logical columns in the pathological OpenCSV-view case. + +## 6. Tests — RED/GREEN + +`HiveConnectorMetadataSchemaTest` (+4): `testOpenCsvSerdeFlattensEveryDataColumnToString` (int/datetime/boolean/**array** +→ STRING), `testOpenCsvSerdePartitionKeyKeepsDeclaredType` (DATE partition key stays date), `testNonOpenCsvSerdeKeepsDeclaredDataTypes` +(LazySimple gate — id INT/ts DATETIMEV2 intact), `testOpenCsvViewColumnsAreNotFlattened` (isView guard). + +- **RED verified**: with the coerce call bypassed, `testOpenCsvSerdeFlattensEveryDataColumnToString` fails + (`expected but was `); the LazySimple gate test stays green (proves the flatten test targets the fix). +- **GREEN**: fe-connector-hive **306/306 module UT** (17→21 in this class), 0 checkstyle, BUILD SUCCESS. + +## 7. Residual risk / notes + +1. **Avro schema-url self-describing tables** — the sole A-vs-B divergence. The connector has zero + `avro.schema.url/literal` handling and resolves from `sd.getCols()`; an Avro table carrying only a schema URL with + empty/stale `sd.cols` reads its columns wrong. This gap is **pre-existing** (B neither creates nor worsens it) and + absent from these suites. If ever needed, it is a separate Avro-scoped connector change — not a reason to route all + loads through `get_schema`. +2. **Latent order-fragility (informational)** — `test_utf8_check` qt_4 (`invalid_utf8_data2`, OpenCSV, `id INT`, `order by id`) + is now string-ordered on `id`; it stays green only because all ids are single-digit (string order == int order). + Not a current failure; noted so a future multi-digit id there isn't a surprise. + +## 8. Acceptance (e2e — user self-run) + +Gate suites (blast red-team widened 2 → 4): +- `external_table_p0/hive/test_open_csv_serde` — target, hive3. +- `external_table_p0/hive/test_hive_serde_prop` — target, hive2 + hive3. +- `external_table_p0/hive/test_drop_expired_table_stats` — runs `analyze`/`show table stats` on OpenCSV `employee_gz` + (already all-string → flatten is a no-op); must stay green. +- `external_table_p0/trino_connector/hive/test_trino_hive_serde_prop` — untouched control (trino-connector path, own + metastore client); confirms it is unaffected. + +All must pass against existing `.out` goldens with **no golden edits** (a needed golden edit signals divergence from legacy). From e1d48045bee1f5a9ce688e34e6184a1f47dba25a Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 23:46:19 +0800 Subject: [PATCH 269/334] [fix](catalog) hive/hudi: remap LZ4FRAME->LZ4BLOCK on read via connector opt-in hook A .lz4 text/OpenCSV table fails to READ under the plugin scan path with BE "LZ4F_getFrameInfo ERROR_frameType_unknown": the generic node infers LZ4FRAME from the .lz4 extension (Util.inferFileCompressTypeByPath) and ships it to BE, but hadoop/hive write .lz4 with the LZ4 BLOCK codec. Legacy HiveScanNode .getFileCompressType remapped LZ4FRAME->LZ4BLOCK; the SPI cutover replaced it with the connector-agnostic PluginDrivenScanNode, which lost the remap. Fixes test_hive_text_write_insert (lz4 iteration). Restore parity as a connector opt-in, keeping fe-core connector-agnostic (mirrors supportsTableSample / supportsBatchScan / classifyColumn): - ConnectorScanPlanProvider: new default adjustFileCompressType(TFileCompressType) = identity (no connector's inferred type is silently changed). - PluginDrivenScanNode.getFileCompressType: run base extension inference, then delegate to the provider's adjustFileCompressType (resolveScanProvider + onPluginClassLoader TCCL pin, byte-identical to classifyColumnByConnector). No source/serde branch in fe-core. - HiveScanPlanProvider overrides it to remap only LZ4FRAME->LZ4BLOCK; every other codec passes through (byte-identical to legacy in every reachable case). - HudiScanPlanProvider gets the same override: legacy HudiScanNode extended HiveScanNode and INHERITED the remap; the new provider is independent, so it re-declares it to preserve that behavior rather than assuming hudi never emits .lz4 (red-team strict-parity call). Other 6 providers keep the identity default. Tests: ConnectorScanPlanProviderCompressTypeTest (default is identity for every codec), HiveScanBatchModeTest +1 (remap only LZ4FRAME), HudiBackendDescriptorTest +1 (inherited-parity remap). The api identity test + hive remap test together encode RED/GREEN. fe-connector-api/hive/hudi + fe-core PluginDrivenScanNode tests green, 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../api/scan/ConnectorScanPlanProvider.java | 18 ++++++ ...ectorScanPlanProviderCompressTypeTest.java | 61 +++++++++++++++++++ .../connector/hive/HiveScanPlanProvider.java | 14 +++++ .../connector/hive/HiveScanBatchModeTest.java | 23 +++++++ .../connector/hudi/HudiScanPlanProvider.java | 13 ++++ .../hudi/HudiBackendDescriptorTest.java | 15 +++++ .../datasource/PluginDrivenScanNode.java | 19 ++++++ 7 files changed, 163 insertions(+) create mode 100644 fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderCompressTypeTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index 418c76d9950e52..7a70b465a02979 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -21,6 +21,7 @@ import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.thrift.TFileCompressType; import org.apache.doris.thrift.TFileScanRangeParams; import org.apache.doris.thrift.TTableFormatFileDesc; @@ -107,6 +108,23 @@ default ConnectorColumnCategory classifyColumn(String columnName) { return ConnectorColumnCategory.DEFAULT; } + /** + * Lets a connector adjust the file compression type the generic node inferred from the file path/extension + * (via {@code Util.inferFileCompressTypeByPath}) before it is shipped to BE on the scan range. The default + * is identity — the inferred type is used as-is. + * + *

    Hive overrides this to remap {@code LZ4FRAME -> LZ4BLOCK}: hadoop/hive write {@code .lz4} files with + * the LZ4 block codec, not the LZ4 frame format the {@code .lz4} extension implies, so the frame + * inferred from the path would make BE's frame decoder fail ({@code LZ4F_getFrameInfo ERROR_frameType_unknown}). + * This keeps that hadoop-specific fact inside the connector; the generic node stays connector-agnostic.

    + * + * @param inferred the compression type the generic node inferred from the file path + * @return the compression type to actually send to BE (identity by default) + */ + default TFileCompressType adjustFileCompressType(TFileCompressType inferred) { + return inferred; + } + /** * Plans the scan for the given table, returning a list of scan ranges. * diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderCompressTypeTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderCompressTypeTest.java new file mode 100644 index 00000000000000..1862224b47f69f --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProviderCompressTypeTest.java @@ -0,0 +1,61 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.scan; + +import org.apache.doris.connector.api.ConnectorSession; +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.thrift.TFileCompressType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Optional; + +/** + * Guards the additive {@code adjustFileCompressType} SPI default on {@link ConnectorScanPlanProvider}. + * + *

    WHY: the default MUST be identity so no connector's inferred file compression type is silently altered + * — only a connector that explicitly opts in (hive/hudi remap {@code LZ4FRAME -> LZ4BLOCK}) changes it. If the + * default ever stopped being identity, every non-opting connector's reads would ship a different compress type + * to BE. This is the zero-break guard for es/jdbc/paimon/iceberg/trino/maxcompute.

    + */ +public class ConnectorScanPlanProviderCompressTypeTest { + + /** Bare provider: only the abstract 4-arg planScan implemented; everything else inherits SPI defaults. */ + private static final class BareProvider implements ConnectorScanPlanProvider { + @Override + public List planScan(ConnectorSession session, ConnectorTableHandle handle, + List columns, Optional filter) { + return Collections.emptyList(); + } + } + + @Test + public void testAdjustFileCompressTypeDefaultsToIdentity() { + ConnectorScanPlanProvider provider = new BareProvider(); + // The default must NOT touch any type — including LZ4FRAME, the one hive/hudi opt in to remap. + for (TFileCompressType type : TFileCompressType.values()) { + Assertions.assertEquals(type, provider.adjustFileCompressType(type), + "default adjustFileCompressType must be identity for " + type); + } + } +} diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index 6dbc26610901b3..c86540cf125dd6 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -30,6 +30,7 @@ import org.apache.doris.connector.spi.ConnectorContext; import org.apache.doris.filesystem.FileEntry; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.thrift.TFileCompressType; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -184,6 +185,19 @@ public boolean supportsTableSample() { return true; } + /** + * Remaps {@code LZ4FRAME -> LZ4BLOCK} for hive splits, restoring legacy {@code HiveScanNode.getFileCompressType} + * parity lost at the SPI cutover. Hadoop/hive write {@code .lz4} files with the LZ4 block codec, but + * the generic node infers {@code LZ4FRAME} from the {@code .lz4} extension; sending frame to BE makes its + * text/CSV reader fail with {@code LZ4F_getFrameInfo ERROR_frameType_unknown} on block-format bytes. Only + * {@code LZ4FRAME} is remapped — every other codec (incl. an actual frame-format non-hive file, which hive + * never produces) passes through, so this is byte-identical to legacy in every reachable case. + */ + @Override + public TFileCompressType adjustFileCompressType(TFileCompressType inferred) { + return inferred == TFileCompressType.LZ4FRAME ? TFileCompressType.LZ4BLOCK : inferred; + } + /** * Plans the scan for a SINGLE partition batch (see * {@link ConnectorScanPlanProvider#planScanForPartitionBatch}). Unlike {@link #planScan} — which resolves diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java index 754a42c7588877..c3832a947e66ab 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.hms.HmsPartitionInfo; import org.apache.doris.connector.hms.HmsTableInfo; import org.apache.doris.filesystem.FileSystem; +import org.apache.doris.thrift.TFileCompressType; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; @@ -142,6 +143,28 @@ public void planScanForPartitionBatchResolvesOnlyTheBatch() { // ===== helpers ===== + @Test + public void testAdjustFileCompressTypeRemapsOnlyLz4Frame() { + // hadoop/hive write .lz4 as the LZ4 BLOCK codec, but the generic node infers LZ4FRAME from the .lz4 + // extension; BE's text reader then fails on block bytes decoded as frame. Restore legacy + // HiveScanNode.getFileCompressType parity: remap ONLY LZ4FRAME -> LZ4BLOCK, pass every other codec + // through unchanged. MUTATION: broadening or dropping the remap would either mis-decode other codecs + // or reintroduce the LZ4F_getFrameInfo failure on .lz4 text tables. + HiveScanPlanProvider provider = provider(null, new CountingLister()); + Assertions.assertEquals(TFileCompressType.LZ4BLOCK, + provider.adjustFileCompressType(TFileCompressType.LZ4FRAME)); + Assertions.assertEquals(TFileCompressType.LZ4BLOCK, + provider.adjustFileCompressType(TFileCompressType.LZ4BLOCK)); + Assertions.assertEquals(TFileCompressType.GZ, + provider.adjustFileCompressType(TFileCompressType.GZ)); + Assertions.assertEquals(TFileCompressType.ZSTD, + provider.adjustFileCompressType(TFileCompressType.ZSTD)); + Assertions.assertEquals(TFileCompressType.SNAPPYBLOCK, + provider.adjustFileCompressType(TFileCompressType.SNAPPYBLOCK)); + Assertions.assertEquals(TFileCompressType.PLAIN, + provider.adjustFileCompressType(TFileCompressType.PLAIN)); + } + private static HiveScanPlanProvider provider(HmsClient hmsClient, CountingLister lister) { return new HiveScanPlanProvider(hmsClient, Collections.emptyMap(), new FakeConnectorContext(), new HiveReadTransactionManager(), new HiveFileListingCache(Collections.emptyMap(), lister)); diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index ecdb1fcd1af8a0..cb3aa39b30b76b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -25,6 +25,7 @@ import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.thrift.TFileCompressType; import org.apache.doris.thrift.TFileScanRangeParams; import org.apache.avro.Schema; @@ -120,6 +121,18 @@ static boolean isForceJniScannerEnabled(ConnectorSession session) { return Boolean.parseBoolean(session.getSessionProperties().get(FORCE_JNI_SCANNER)); } + /** + * Remaps {@code LZ4FRAME -> LZ4BLOCK}, preserving the behavior legacy {@code HudiScanNode} inherited from its + * superclass {@code HiveScanNode.getFileCompressType} (hadoop writes {@code .lz4} as the LZ4 block codec, not + * the frame format the extension implies). The new provider does not extend the hive one, so this override + * restores the inherited legacy remap rather than relying on "hudi never produces a {@code .lz4} split". + * Only {@code LZ4FRAME} is remapped; every other codec passes through. + */ + @Override + public TFileCompressType adjustFileCompressType(TFileCompressType inferred) { + return inferred == TFileCompressType.LZ4FRAME ? TFileCompressType.LZ4BLOCK : inferred; + } + @Override public List planScan( ConnectorSession session, diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java index 13234ecad5d2db..91201e834f6c68 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java @@ -18,6 +18,7 @@ package org.apache.doris.connector.hudi; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.thrift.TFileCompressType; import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; @@ -91,6 +92,20 @@ null, new HudiTableHandle("db", "t", "s3://b/t", "COPY_ON_WRITE"), Assertions.assertEquals("hadoopAK", result.get("location.fs.s3a.access.key")); } + @Test + public void adjustFileCompressTypeRemapsLz4FrameLikeLegacyInheritance() { + // Legacy HudiScanNode extended HiveScanNode and INHERITED its LZ4FRAME -> LZ4BLOCK remap (hadoop writes + // .lz4 as the LZ4 block codec). The new HudiScanPlanProvider does not extend the hive provider, so it + // re-declares the remap to preserve that inherited behavior rather than assuming "hudi never emits .lz4". + HudiScanPlanProvider provider = new HudiScanPlanProvider(Collections.emptyMap(), null); + Assertions.assertEquals(TFileCompressType.LZ4BLOCK, + provider.adjustFileCompressType(TFileCompressType.LZ4FRAME)); + Assertions.assertEquals(TFileCompressType.GZ, + provider.adjustFileCompressType(TFileCompressType.GZ)); + Assertions.assertEquals(TFileCompressType.PLAIN, + provider.adjustFileCompressType(TFileCompressType.PLAIN)); + } + private static ConnectorContext contextWithBackendProps(Map backendProps) { return new ConnectorContext() { @Override diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index d9895db49d5c93..f754ad7ab751f7 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -58,6 +58,7 @@ import org.apache.doris.thrift.TColumnCategory; import org.apache.doris.thrift.TExplainLevel; import org.apache.doris.thrift.TFileAttributes; +import org.apache.doris.thrift.TFileCompressType; import org.apache.doris.thrift.TFileFormatType; import org.apache.doris.thrift.TFileRangeDesc; import org.apache.doris.thrift.TFileTextScanRangeParams; @@ -509,6 +510,24 @@ ConnectorColumnCategory classifyColumnByConnector(String columnName) { return onPluginClassLoader(scanProvider, () -> scanProvider.classifyColumn(columnName)); } + /** + * Lets the owning connector adjust the compression type this node inferred from the split's file path + * before it is shipped to BE, WITHOUT any source-specific code here: the base inference runs first, then + * the connector's {@link ConnectorScanPlanProvider#adjustFileCompressType} (identity by default) gets the + * final say. Hive uses it to remap {@code LZ4FRAME -> LZ4BLOCK} (hadoop writes {@code .lz4} as block codec); + * every other connector inherits the identity default and is byte-unchanged. A connector with no scan + * provider keeps the inferred type. Mirrors {@link #classifyColumnByConnector} (same resolve + TCCL pin). + */ + @Override + protected TFileCompressType getFileCompressType(FileSplit fileSplit) throws UserException { + TFileCompressType inferred = super.getFileCompressType(fileSplit); + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); + if (scanProvider == null) { + return inferred; + } + return onPluginClassLoader(scanProvider, () -> scanProvider.adjustFileCompressType(inferred)); + } + @Override protected TableIf getTargetTable() throws UserException { return desc.getTable(); From ce30d12a55cec9c6e66c208396e07cbb4d0597ad Mon Sep 17 00:00:00 2001 From: morningman Date: Sat, 11 Jul 2026 23:48:06 +0800 Subject: [PATCH 270/334] [doc](catalog) hive-e2e: land text_write(LZ4) design; record SHOW CREATE design refutation MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit text_write(LZ4) marked DONE in batch-3 (e1d48045bee), design doc added. HANDOFF remaining 9 -> 8 suites. Also records that recon+red-team refuted the original SHOW CREATE design (eager show.create-ddl key): test_hive_meta_cache requires a FRESH HMS fetch at command time, declaring SUPPORTS_SHOW_CREATE_DDL disturbs delegated/HTTP callers, and the acceptance set is wider than 2 suites — corrected direction is a lazy typed ConnectorMetadata method intercepted in ShowCreateTableCommand (Option X). Pending second design pass + user decision. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 9 +- .../designs/FIX-text-write-LZ4-design.md | 90 +++++++++++++++++++ 2 files changed, 94 insertions(+), 5 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-text-write-LZ4-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 92a517c384efd3..849bedd783907c 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -12,14 +12,13 @@ > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > > **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **✅ batch-3（R7/R10 前序 session、R12 本 session，共 3 fix DONE）**： +> **✅ batch-3（R7/R10 前序 session；R12 + text_write(LZ4) 本 session，共 4 fix DONE）**： > - **R7 `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。红队 4 lens 全 SOUND。`CachingHmsClientTest` 19/19。设计 `designs/FIX-R7-design.md`。**e2e 待用户自跑 test_hive_use_meta_cache_true sql09。** > - **R10 `9c70d4acf9a`**——OpenX JSON `ignore.malformed.json` 跳坏行。`HiveTextProperties.extractJsonSerDeProps` 串入 sd/table params，**仅 OpenX serde**发 `hive.text.openx_ignore_malformed`(table-param 优先、默认 false；hcatalog/hive2 从不带此旗，钉 OpenX 保 legacy 分支 parity) + fe-core `PluginDrivenScanNode.getFileAttributes` 在既有 `is_json` 块读键调 `setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode OPENX 分支)。one-column 模式 OpenX 解析成 CSV、BE CSV reader 不读该 JSON-only 字段故无碍。红队 3 lens(初版"全 3 JSON serde"被驳→已收敛 OpenX-only；test/iron-rule SOUND)。`HiveTextPropertiesTest` 13/13。设计 `designs/FIX-R10-design.md`。**e2e 待用户自跑 test_hive_openx_json q1。** > - **R12/serde `3936434bc9a`**——OpenCSV 表列全塌成 STRING(修 test_open_csv_serde + test_hive_serde_prop 的 OpenCSV 半)。根因=legacy 默认走 metastore `get_schema` RPC(服务端反序列化器→OpenCSV 全 string)，新连接器只读原始 `sd.getCols()` 声明类型→typed 列到 BE 翻车(TRUE vs 'true'、datetime→ISO、空串 vs NULL)。**方案定案=B(用户 2026-07-11 签字)**：不走 RPC、连接器就地复刻结果——`HiveConnectorMetadata.buildColumns` 加 `coerceOpenCsvColumnsToString`,serde==OpenCSV 时把**数据列**整型别塌成 `STRING`(复合类型也整列塌)、复用现成 `HiveTextProperties.HIVE_OPEN_CSV_SERDE` 常量。**架构红队关键订正**：放 hive 元数据层(非 hive/hudi 共用的 `ThriftHmsClient`,那会误触 hudi schema 路径 + 重复常量)——对齐 Trino(在 HiveMetadata 层做 CSV=全 string)。分区列不动(hive 在反序列化后追加、保声明类型)、view 有 isView 门。非-OpenCSV 表零改动(与原 `sd.getCols()` 逐位一致)。fe-core 零改、无 serde 名分支。3 路对抗红队(correctness SOUND / architecture 订正放置层 / blast 把验收扩到 4 suite)。`HiveConnectorMetadataSchemaTest` +4(int/datetime/bool/**array**→STRING、分区键留 date、LazySimple 门留 typed、view 门)、RED 实证(旁路→expected STRING but was INT)、306/306、0 checkstyle。设计 `designs/FIX-R12-design.md`。**残余(非本修)**：Avro schema-url 自述表 A/B 唯一分歧,新连接器**现状本就没有**、不在失败套件、以后需要时独立小任务补。**e2e 待用户自跑：test_open_csv_serde + test_hive_serde_prop(+对照 test_drop_expired_table_stats/test_trino_hive_serde_prop)。** -> **⭐ 剩余 9 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **fe-connector 中型**： -> - **text_write** LZ4FRAME→LZ4BLOCK 读修正：`HiveScanPlanProvider` 恢复 legacy HiveScanNode:670-678 的 `getFileCompressType` override(`.lz4`→LZ4FRAME 需转 LZ4BLOCK)；为守 fe-core connector-agnostic，加 `ConnectorScanPlanProvider` default 压缩类型 hook(identity)。 -> - **fe-connector/fe-core 大型**：R2 SHOW CREATE TABLE 原生 DDL(show_create_table + ddl_text_format；HiveConnector 声明 `SUPPORTS_SHOW_CREATE_DDL` + 原生 DDL 渲染器镜像 legacy HiveMetaStoreClientHelper.showCreateTable，注意测试断言单引号) · R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 +> - **text_write(LZ4) `e1d48045bee`**——`.lz4` 文本表读炸 BE `LZ4F_getFrameInfo ERROR_frameType_unknown`。根因=`.lz4` 被 `Util.inferFileCompressTypeByPath` 推成 LZ4FRAME 发 BE，但 hadoop/hive 写的是 LZ4 **block** 编码；legacy `HiveScanNode.getFileCompressType` 有 LZ4FRAME→LZ4BLOCK remap，SPI 翻闸后丢了。修=连接器 opt-in（镜像 supportsTableSample/supportsBatchScan）：`ConnectorScanPlanProvider.adjustFileCompressType` 默认 identity → `PluginDrivenScanNode.getFileCompressType` 跑 super 推断后委派（resolveScanProvider+onPluginClassLoader，无 source 分支）→ `HiveScanPlanProvider` override 只 remap LZ4FRAME。**hudi 严格对齐**（红队定：legacy `HudiScanNode extends HiveScanNode` 继承了 remap，新独立 provider 须重声明，别赌"hudi 从不出 .lz4"）；其余 6 provider 留 identity。测试：api identity 1/1 + hive remap 5/5 + hudi parity 4/4（api+hive 对子=RED/GREEN），fe-core 节点 54/54 无回归，0 checkstyle。设计 `designs/FIX-text-write-LZ4-design.md`。**e2e 待用户自跑 test_hive_text_write_insert 的 lz4 迭代。** +> **⭐ 剩余 8 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **fe-connector/fe-core 大型**：**R2 SHOW CREATE TABLE 原生 DDL（show_create_table + ddl_text_format）——⚠ recon+红队(wf_992d12bd-d60)已把原方案证伪、需订正后实现**：SPI 早有 `SUPPORTS_SHOW_CREATE_DDL`+fe-core 路由(iceberg/paimon 用),但 iceberg/paimon 是**数据驱动**(发 show.* 渲染提示键、Env.getDdlStmt 拼 Doris 风格 DDL),**结构上拼不出** hive 原生 `ROW FORMAT SERDE`/单引号 SERDEPROPERTIES/`STORED AS`。且**红队三杀原"eager 缓存键"方案**：(a) `test_hive_meta_cache` L354-357 要求 show create **实时取 HMS**(desc 缓存 5 列、show-create 须 6 列)→eager 缓存键结构上做不到,须**惰性方法命令期新取**;(b) **声明 capability** 连接器级会翻转 `Env.getDdlStmt` 门→误触委派的 hudi/iceberg-on-HMS 表 + HTTP schema 端点;(c) 验收套件被低估(还有 test_hive_meta_cache/test_multi_delimit_serde/test_hive_ddl)。**订正方向=惰性 typed 方法**(`ConnectorMetadata.getShowCreateTableDdl` 默认空、hive 就地新取 HMS 渲染,delegated/view 返空) + `ShowCreateTableCommand` 新臂(非 Env.getDdlStmt、不声明 capability=Option X)。渲染器镜像 legacy `HiveMetaStoreClientHelper.showCreateTable`(单/双引号两套约定)。**下一步：二轮设计+红队锁 Option X → 中文向用户呈请拍板(X vs Y/capability) → 实现。** · R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 > - **用户已裁决(A)的 3 个 fe-core/SPI 大改**： > - query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 > - default_partition 连接器传 null 标记：SPI 新字段让连接器按分区值传 null 语义，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem:310`(现无条件 `PartitionValue(v,false)`) 据此建项——hive 标 null、paimon 保持非空(零 paimon 影响，别在 fe-core if(hive))。 diff --git a/plan-doc/tasks/designs/FIX-text-write-LZ4-design.md b/plan-doc/tasks/designs/FIX-text-write-LZ4-design.md new file mode 100644 index 00000000000000..5c3f30cd37e521 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-text-write-LZ4-design.md @@ -0,0 +1,90 @@ +# FIX-text-write — LZ4FRAME→LZ4BLOCK read remap lost in plugin scan path + +**Suite fixed:** `test_hive_text_write_insert` (the `lz4` compression iteration). +**Scope:** connector opt-in hook on the scan SPI; fe-core stays connector-agnostic. +**Status:** code DONE (`e1d48045bee`). e2e awaits user self-run. + +--- + +## 1. Symptom + +A hive TEXT/OpenCSV table written with LZ4 compression fails to READ under the plugin scan path — BE aborts +with `LZ4F_getFrameInfo ERROR_frameType_unknown`. The write path is fine; only the read is broken. +`test_hive_text_write_insert.groovy` loops `format_compressions = [..., "lz4"]`, writing a text table then +reading it back against a codec-invariant golden; the `lz4` iteration diverges/fails. + +## 2. Root cause (HEAD-verified) + +BE does **not** infer compression from the path — it honors the `compress_type` FE sets on each scan range. +Legacy `HiveScanNode.getFileCompressType` (`HiveScanNode.java:670-678`) overrode the base inference to remap +`LZ4FRAME → LZ4BLOCK`, because hadoop/hive write `.lz4` files with the LZ4 **block** codec, not the LZ4 **frame** +format the `.lz4` extension implies. + +The SPI cutover replaced `HiveScanNode` with the connector-agnostic `PluginDrivenScanNode`, which does **not** +override `getFileCompressType` and inherits `FileQueryScanNode.getFileCompressType` (`:624`) → +`Util.inferFileCompressTypeByPath` → `.lz4` becomes `LZ4FRAME`. FE ships `LZ4FRAME` on the scan range +(`FileQueryScanNode.java:477-478`); BE's frame decoder then fails on block-format bytes. FE-side fix only. + +## 3. Fix — connector opt-in hook (mirrors `supportsTableSample` / `supportsBatchScan` / `classifyColumn`) + +Iron rule: fe-core generic node must not branch on source/serde. The remap is a connector opt-in via a default +hook; only hive (and hudi, see §4) override it. + +**A. SPI default (identity)** — `ConnectorScanPlanProvider`: +```java +default TFileCompressType adjustFileCompressType(TFileCompressType inferred) { return inferred; } +``` + +**B. Generic node delegates** — `PluginDrivenScanNode` (byte-identical shape to `classifyColumnByConnector`): +```java +@Override +protected TFileCompressType getFileCompressType(FileSplit fileSplit) throws UserException { + TFileCompressType inferred = super.getFileCompressType(fileSplit); // base extension inference + ConnectorScanPlanProvider scanProvider = resolveScanProvider(); + if (scanProvider == null) { + return inferred; + } + return onPluginClassLoader(scanProvider, () -> scanProvider.adjustFileCompressType(inferred)); +} +``` +No source/serde branch; `onPluginClassLoader` keeps the TCCL pin convention (streaming split paths run on a +pool thread that doesn't inherit the caller's TCCL). Verified: the plugin's `setScanParams` override never +re-touches `compress_type`, so the value set here survives to BE (red-team confirmed). + +**C. Hive override** — `HiveScanPlanProvider`: +```java +@Override +public TFileCompressType adjustFileCompressType(TFileCompressType inferred) { + return inferred == TFileCompressType.LZ4FRAME ? TFileCompressType.LZ4BLOCK : inferred; +} +``` +Exact legacy parity; the hadoop-specific fact stays inside the connector. Only `LZ4FRAME` flips — every other +codec (and a non-hive real frame file, which hive never produces) passes through. + +## 4. Hudi strict-parity (red-team call) + +Legacy `HudiScanNode extends HiveScanNode` (`HudiScanNode.java:94`) and **inherited** the remap. The new +`HudiScanPlanProvider` is independent, so under this design it would get only the identity default. Both +red-team lenses (correctness, iron-rule) flagged this: leaving it off silently drops a behavior legacy shipped +(Rule 3 / Rule 12), resting on the unverifiable-in-code assumption "hudi never produces a `.lz4` split." Adopted +the **strict-parity option**: `HudiScanPlanProvider` re-declares the identical override. Cost is a few lines of +possibly-unreachable code; benefit is provable parity. The other 6 providers (iceberg/paimon/trino/es/mc/jdbc) +keep the identity default — verified legacy had the remap only on `HiveScanNode`. + +## 5. Tests — RED/GREEN + +- `ConnectorScanPlanProviderCompressTypeTest` (fe-connector-api): the default is identity for **every** + `TFileCompressType` — the zero-break guard for non-opting connectors, and it proves the default returns + `LZ4FRAME` unchanged. +- `HiveScanBatchModeTest` +1: hive remaps only `LZ4FRAME→LZ4BLOCK`, passes `GZ/ZSTD/SNAPPYBLOCK/PLAIN` through. +- `HudiBackendDescriptorTest` +1: hudi does the same (inherited-parity guard). +- **RED/GREEN encoded by the pair**: the api test pins the default = identity (returns `LZ4FRAME`); the hive + test pins hive = `LZ4BLOCK`. The delta is exactly the override — remove it and hive returns `LZ4FRAME` ≠ + expected `LZ4BLOCK`. +- **Regression**: fe-core `PluginDrivenScanNodeBatchModeTest` (12), `PluginDrivenScanNodeTableSampleTest` (6), + `PluginDrivenExternalTableTest` (36) all green with the new override; 0 checkstyle. + +## 6. Acceptance (e2e — user self-run) + +`external_table_p0/hive/write/test_hive_text_write_insert` (hive3): the `lz4` iteration must read back +byte-identical rows to the other codecs against the codec-invariant golden. This is the true RED→GREEN gate. From 17764d03665afc00275123b0e641aacb6c41f600 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 00:20:23 +0800 Subject: [PATCH 271/334] [fix](catalog) hive: native SHOW CREATE TABLE via lazy fresh-fetch connector render MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After the SPI cutover, SHOW CREATE TABLE on a flipped hive table emitted generic Doris-style DDL (double-quoted PROPERTIES) instead of native hive DDL (ROW FORMAT SERDE / WITH SERDEPROPERTIES / STORED AS INPUTFORMAT/OUTPUTFORMAT), which the suites assert. Fixes test_hive_show_create_table + test_hive_ddl_text_format (and covers test_hive_meta_cache / test_multi_delimit_serde / test_hive_ddl). Design (Option X, user signed off 2026-07-12): a lazy per-command connector render + fresh HMS fetch, intercepted in ShowCreateTableCommand, WITHOUT declaring SUPPORTS_SHOW_CREATE_DDL. Two earlier designs were red-teamed out: an eager schema-cache render can't satisfy test_hive_meta_cache (SHOW CREATE must show a freshly-added column while DESC still serves the cached 5-col schema), and declaring the capability would flip the shared Env.getDdlStmt gate for delegated hudi/iceberg-on-HMS tables and the HTTP schema endpoint. - ConnectorMetadata.renderShowCreateTableDdl(session, handle): new default empty; iceberg/paimon/es/jdbc inherit it (byte-inert, fall through to Env.getDdlStmt). - HmsClient.getTableFresh + CachingHmsClient override: cache-bypassing table read (mirrors listPartitionNamesFresh) so SHOW CREATE reflects the latest HMS schema. - HiveShowCreateTableRenderer (fe-connector-hms): byte-for-byte port of legacy HiveMetaStoreClientHelper.showCreateTable — two quoting conventions (SERDEPROPERTIES 'k' = 'v' vs TBLPROPERTIES 'k'='v'), 2-space/1-space indents, comment lifted to COMMENT clause, null-comment guard. Types via HmsTypeMapping.toHiveTypeString. - HiveConnectorMetadata override: guards non-HiveTableHandle (delegated iceberg/hudi-on-HMS) -> empty, mirroring getTableSchema; fresh getTableFresh. - PluginDrivenExternalTable.getShowCreateTableDdl: mirrors getViewText (safe under the command read-lock); empty on unresolved handle. ShowCreateTableCommand new arm after the view arm, gated on the method returning present (not source name) so iceberg/paimon/views are untouched. fe-core stays connector-agnostic. Tests: HiveShowCreateTableRendererTest (5, byte-parity incl. both quoting conventions + null-comment guard), CachingHmsClientTest +3 (getTableFresh bypasses cache both directions). fe-core PluginDrivenExternalTable 36/36 + EnvShowCreate 3/3 unchanged; fe-connector-hive 307/307; 0 checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/api/ConnectorTableOps.java | 18 +++ .../connector/hive/HiveConnectorMetadata.java | 21 +++ .../doris/connector/hms/CachingHmsClient.java | 9 ++ .../hms/HiveShowCreateTableRenderer.java | 140 ++++++++++++++++++ .../apache/doris/connector/hms/HmsClient.java | 19 +++ .../connector/hms/CachingHmsClientTest.java | 36 +++++ .../hms/HiveShowCreateTableRendererTest.java | 111 ++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 25 ++++ .../commands/ShowCreateTableCommand.java | 12 ++ 9 files changed, 391 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HiveShowCreateTableRenderer.java create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HiveShowCreateTableRendererTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java index 26b27341a99abc..5e1f21ab28eabb 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java @@ -92,6 +92,24 @@ default ConnectorTableSchema getTableSchema( return getTableSchema(session, handle); } + /** + * Renders the native {@code SHOW CREATE TABLE} DDL for a table, fetching schema FRESH from the underlying + * metastore at call time (bypassing any connector-side table cache) so the returned statement always + * reflects the latest remote schema. + * + *

    This is a LAZY, per-call interception point used ONLY by {@code ShowCreateTableCommand}. It intentionally + * does NOT participate in the {@code SUPPORTS_SHOW_CREATE_DDL} capability (which gates the engine-assembled + * DDL in {@code Env.getDdlStmt} for every caller, including delegated sibling tables and the HTTP schema + * endpoint). A connector that does not natively render its own SHOW CREATE returns {@link Optional#empty()}, + * and the command falls through to the generic {@code Env.getDdlStmt} path unchanged.

    + * + * @return the full {@code CREATE TABLE} statement, or {@link Optional#empty()} to defer to the engine + */ + default Optional renderShowCreateTableDdl( + ConnectorSession session, ConnectorTableHandle handle) { + return Optional.empty(); + } + /** Returns a name-to-handle map for all columns of the table. */ default Map getColumnHandles( ConnectorSession session, ConnectorTableHandle handle) { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 9d346bca6e9210..7a61083b44a577 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -54,6 +54,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorIn; import org.apache.doris.connector.api.pushdown.ConnectorLiteral; import org.apache.doris.connector.api.pushdown.FilterApplicationResult; +import org.apache.doris.connector.hms.HiveShowCreateTableRenderer; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; import org.apache.doris.connector.hms.HmsClientException; @@ -390,6 +391,26 @@ public Optional getTableHandle( return Optional.of(handle); } + /** + * Renders native hive {@code SHOW CREATE TABLE} DDL from a FRESH metastore read (see + * {@link HiveShowCreateTableRenderer}). Fetches via {@link HmsClient#getTableFresh} — SHOW CREATE must show + * the latest schema even while {@code DESC}, served from the schema cache, is stale (the {@code use_meta_cache} + * freshness contract). A delegated iceberg/hudi-on-HMS table routes through THIS hive gateway metadata + * ({@code getTableHandle} returns the sibling's foreign handle), so guard exactly like {@link #getTableSchema}: + * a non-{@link HiveTableHandle} is not a plain-hive base table — return empty to defer to the engine + * ({@code Env.getDdlStmt}), keeping delegated-table SHOW CREATE at today's behavior. + */ + @Override + public Optional renderShowCreateTableDdl( + ConnectorSession session, ConnectorTableHandle handle) { + if (!(handle instanceof HiveTableHandle)) { + return Optional.empty(); + } + HiveTableHandle hiveHandle = (HiveTableHandle) handle; + HmsTableInfo tableInfo = hmsClient.getTableFresh(hiveHandle.getDbName(), hiveHandle.getTableName()); + return Optional.of(HiveShowCreateTableRenderer.render(tableInfo)); + } + @Override public ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index 1377aad2b6eb69..28a3cd535f936f 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -144,6 +144,15 @@ public HmsTableInfo getTable(String dbName, String tableName) { key -> delegate.getTable(key.dbName, key.tableName)); } + @Override + public HmsTableInfo getTableFresh(String dbName, String tableName) { + // Fresh (cache-bypassing) table read for SHOW CREATE TABLE, which must reflect the latest remote schema + // even while DESC (served from the schema cache backed by this tableCache) still shows a stale one. This + // neither READS nor WRITES tableCache: reading would serve the stale table this method exists to avoid, + // writing would let a non-cache path repopulate off-band (mirrors listPartitionNamesFresh). + return delegate.getTableFresh(dbName, tableName); + } + @Override public List listPartitionNames(String dbName, String tableName, int maxParts) { return partitionNamesCache.get(new PartitionNamesKey(dbName, tableName, maxParts), diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HiveShowCreateTableRenderer.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HiveShowCreateTableRenderer.java new file mode 100644 index 00000000000000..8eebf826880e09 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HiveShowCreateTableRenderer.java @@ -0,0 +1,140 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.api.ConnectorColumn; + +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; + +/** + * Renders the native hive {@code SHOW CREATE TABLE} DDL for a base table — a byte-for-byte connector-side port of + * legacy {@code HiveMetaStoreClientHelper.showCreateTable} (base-table branch, L749-824). It lives connector-side + * (not fe-core) because it emits hive-format specifics — {@code ROW FORMAT SERDE}, {@code WITH SERDEPROPERTIES}, + * {@code STORED AS INPUTFORMAT/OUTPUTFORMAT} — that fe-core must not know about (iron rule). + * + *

    Two quoting conventions are load-bearing and must both be preserved (the acceptance suites discriminate on + * them): {@code WITH SERDEPROPERTIES ('k' = 'v')} has SPACES around {@code =}, while {@code TBLPROPERTIES + * ('k'='v')} has NONE. Column comments are emitted only when non-null (an empty {@code COMMENT ''} would break the + * exact substring the meta-cache suite asserts), and the single {@code comment} table param is lifted out to the + * top-level {@code COMMENT} clause rather than left in {@code TBLPROPERTIES}. + * + *

    Column/partition types are reconstructed from the mapped {@link org.apache.doris.connector.api.ConnectorType} + * via {@link HmsTypeMapping#toHiveTypeString} (the SPI carries the mapped type, not the raw thrift string). That + * round-trip is exact for scalars/decimal/date/timestamp/nested; a raw HMS {@code varchar(n)} would render as + * {@code string} (length dropped), harmless here because such columns are stored as {@code string} in HMS. An + * unmappable type throws {@code IllegalArgumentException} — legacy could not hit this (it echoed the raw string), + * so it is left to fail loud rather than guessed at (Rule 2). This method must stay reflection-free: the caller + * runs it on the fe-core thread AFTER the pinned metastore fetch, so any future name-based reflection here would + * need its own TCCL pin. + */ +public final class HiveShowCreateTableRenderer { + + /** HMS table-param key carrying the table comment (lifted to the top-level COMMENT clause). */ + private static final String COMMENT_KEY = "comment"; + + private HiveShowCreateTableRenderer() { + } + + public static String render(HmsTableInfo table) { + StringBuilder output = new StringBuilder(); + output.append(String.format("CREATE TABLE `%s`(\n", table.getTableName())); + + List columns = table.getColumns(); + for (int i = 0; i < columns.size(); i++) { + ConnectorColumn col = columns.get(i); + // 2-space indent for data columns (partition keys below use 1-space, matching legacy). + output.append(String.format(" `%s` %s", col.getName(), + HmsTypeMapping.toHiveTypeString(col.getType()))); + if (col.getComment() != null) { + output.append(String.format(" COMMENT '%s'", col.getComment())); + } + if (i < columns.size() - 1) { + output.append(",\n"); + } + } + output.append(")\n"); + + Map params = table.getParameters(); + if (params != null && params.containsKey(COMMENT_KEY)) { + output.append(String.format("COMMENT '%s'", params.get(COMMENT_KEY))).append("\n"); + } + + List partitionKeys = table.getPartitionKeys(); + if (partitionKeys != null && !partitionKeys.isEmpty()) { + output.append("PARTITIONED BY (\n") + .append(partitionKeys.stream() + .map(p -> String.format(" `%s` %s", p.getName(), + HmsTypeMapping.toHiveTypeString(p.getType()))) + .collect(Collectors.joining(",\n"))) + .append(")\n"); + } + + List bucketCols = table.getBucketCols(); + if (bucketCols != null && !bucketCols.isEmpty()) { + output.append("CLUSTERED BY (\n") + .append(bucketCols.stream().map(c -> " " + c).collect(Collectors.joining(",\n"))) + .append(")\n") + .append(String.format("INTO %d BUCKETS\n", table.getNumBuckets())); + } + + String serdeLib = table.getSerializationLib(); + if (serdeLib != null && !serdeLib.isEmpty()) { + output.append("ROW FORMAT SERDE\n").append(String.format(" '%s'\n", serdeLib)); + } + + Map serdeParams = table.getSdParameters(); + if (serdeParams != null && !serdeParams.isEmpty()) { + // SERDEPROPERTIES: spaces around '=' (legacy L789). + output.append("WITH SERDEPROPERTIES (\n") + .append(serdeParams.entrySet().stream() + .map(e -> String.format(" '%s' = '%s'", e.getKey(), e.getValue())) + .collect(Collectors.joining(",\n"))) + .append(")\n"); + } + + String inputFormat = table.getInputFormat(); + if (inputFormat != null && !inputFormat.isEmpty()) { + output.append("STORED AS INPUTFORMAT\n").append(String.format(" '%s'\n", inputFormat)); + } + String outputFormat = table.getOutputFormat(); + if (outputFormat != null && !outputFormat.isEmpty()) { + output.append("OUTPUTFORMAT\n").append(String.format(" '%s'\n", outputFormat)); + } + String location = table.getLocation(); + if (location != null && !location.isEmpty()) { + output.append("LOCATION\n").append(String.format(" '%s'\n", location)); + } + + if (params != null && !params.isEmpty()) { + // TBLPROPERTIES: no spaces around '=' (legacy L818); the comment key is lifted to COMMENT above. + Map tblProps = new LinkedHashMap<>(params); + tblProps.remove(COMMENT_KEY); + if (!tblProps.isEmpty()) { + output.append("TBLPROPERTIES (\n") + .append(tblProps.entrySet().stream() + .map(e -> String.format(" '%s'='%s'", e.getKey(), e.getValue())) + .collect(Collectors.joining(",\n"))) + .append(")"); + } + } + return output.toString(); + } +} diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java index c45116df2bab35..1be6692974bd51 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsClient.java @@ -90,6 +90,25 @@ public interface HmsClient extends Closeable { */ HmsTableInfo getTable(String dbName, String tableName); + /** + * Get table metadata bypassing any connector-side table cache — always a fresh metastore read. Used by + * SHOW CREATE TABLE, which must reflect the latest remote schema even while {@code DESC} (served from the + * schema cache) still shows a stale one (the {@code use_meta_cache} freshness contract). + * + *

    Default = the cached path: a non-decorating client (e.g. the raw {@link ThriftHmsClient}) has no cache + * to bypass, so the two are identical for it. Any caching {@code HmsClient} decorator MUST override this + * to reach its delegate directly, or SHOW CREATE regresses to serving a stale cached table (mirrors + * {@link #listPartitionNamesFresh}).

    + * + * @param dbName database name + * @param tableName table name + * @return table info with SPI-typed columns, read fresh from the metastore + * @throws HmsClientException if the table is not found or operation fails + */ + default HmsTableInfo getTableFresh(String dbName, String tableName) { + return getTable(dbName, tableName); + } + /** * Get default column values for a table. * diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java index 18c3756f9a92e3..6c4302d542cdcb 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -113,6 +113,42 @@ public void listPartitionNamesCachesByDbTableAndMaxParts() { Assertions.assertEquals(2, delegate.listPartitionNamesCalls); } + // ---- getTableFresh (SHOW CREATE TABLE — must bypass the table cache) ---- + + @Test + public void getTableFreshAlwaysHitsDelegate() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // WHY: SHOW CREATE TABLE must see the latest schema (a column added externally after the cache filled) + // even while DESC serves the stale cached table. Every fresh call goes to the metastore. (test_hive_meta_cache.) + cache.getTableFresh("db", "t1"); + cache.getTableFresh("db", "t1"); + Assertions.assertEquals(2, delegate.getTableCalls); + } + + @Test + public void getTableFreshDoesNotPopulateCache() { + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, Collections.emptyMap()); + + // Fresh call must NOT write the table cache: a following cached getTable must still MISS (delegate call #2) + // and only THEN populate — proving fresh bypasses the cache in both directions. + cache.getTableFresh("db", "t1"); // delegate #1, no populate + cache.getTable("db", "t1"); // cache miss -> delegate #2 + populate + cache.getTable("db", "t1"); // cache hit -> no delegate call + Assertions.assertEquals(2, delegate.getTableCalls); + } + + @Test + public void getTableFreshDefaultOnNonCachingClientIsPlainGet() { + // A bare HmsClient (no caching decorator) inherits the interface default: fresh == the raw getTable. + RecordingHmsClient raw = new RecordingHmsClient(); + raw.getTableFresh("db", "t1"); + raw.getTable("db", "t1"); + Assertions.assertEquals(2, raw.getTableCalls); + } + // ---- listPartitionNamesFresh (SHOW PARTITIONS / partitions TVF — must bypass the names cache) ---- @Test diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HiveShowCreateTableRendererTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HiveShowCreateTableRendererTest.java new file mode 100644 index 00000000000000..a609ff99fd2ab8 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HiveShowCreateTableRendererTest.java @@ -0,0 +1,111 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.connector.api.ConnectorType; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +/** + * Byte-parity tests for {@link HiveShowCreateTableRenderer} — the connector-side port of legacy + * {@code HiveMetaStoreClientHelper.showCreateTable} that native hive SHOW CREATE TABLE relies on. + * + *

    The load-bearing details the acceptance suites (test_hive_show_create_table / _ddl_text_format / + * _meta_cache / test_multi_delimit_serde / test_hive_ddl) discriminate on: the TWO quoting conventions + * (SERDEPROPERTIES {@code 'k' = 'v'} with spaces vs TBLPROPERTIES {@code 'k'='v'} without), the 2-space data / + * 1-space partition column indents, the null-comment guard (an empty {@code COMMENT ''} would break the + * meta-cache column substring), and lifting the {@code comment} table param to a top-level COMMENT clause.

    + */ +public class HiveShowCreateTableRendererTest { + + private static ConnectorColumn col(String name, String typeName, String comment) { + return new ConnectorColumn(name, ConnectorType.of(typeName), comment, true, null); + } + + /** A partitioned TEXT/LazySimpleSerDe table with a commented + an uncommented column and a comment param. */ + private static HmsTableInfo textTable() { + Map serdeParams = new LinkedHashMap<>(); + serdeParams.put("field.delim", "|"); + Map tableParams = new LinkedHashMap<>(); + tableParams.put("comment", "my table"); + tableParams.put("doris.file_format", "text"); + return HmsTableInfo.builder() + .dbName("db").tableName("t") + .columns(Arrays.asList(col("id", "INT", null), col("name", "STRING", "the name"))) + .partitionKeys(Collections.singletonList(col("dt", "DATEV2", null))) + .serializationLib("org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe") + .sdParameters(serdeParams) + .inputFormat("org.apache.hadoop.mapred.TextInputFormat") + .outputFormat("org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat") + .location("hdfs://ns/wh/t") + .parameters(tableParams) + .build(); + } + + @Test + public void testColumnBlockIndentTypesAndNullCommentGuard() { + // 2-space data-column indent, mapped hive type strings, and the null-comment guard: `id` has NO comment + // token (a spurious COMMENT '' would break test_hive_meta_cache's exact ``k3` string)` substring), + // `name` carries its comment, and the block closes with `)\n`. + String ddl = HiveShowCreateTableRenderer.render(textTable()); + Assertions.assertTrue(ddl.contains("CREATE TABLE `t`(\n `id` int,\n `name` string COMMENT 'the name')\n"), + ddl); + } + + @Test + public void testTableCommentLiftedAndPartitionOneSpaceIndent() { + String ddl = HiveShowCreateTableRenderer.render(textTable()); + // The `comment` table param becomes a top-level COMMENT clause (not a TBLPROPERTY). + Assertions.assertTrue(ddl.contains("COMMENT 'my table'\n"), ddl); + // Partition columns use a ONE-space indent (data columns use two), matching legacy. + Assertions.assertTrue(ddl.contains("PARTITIONED BY (\n `dt` date)\n"), ddl); + } + + @Test + public void testSerdeAndFormatBlocks() { + String ddl = HiveShowCreateTableRenderer.render(textTable()); + Assertions.assertTrue(ddl.contains( + "ROW FORMAT SERDE\n 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'\n"), ddl); + Assertions.assertTrue(ddl.contains( + "STORED AS INPUTFORMAT\n 'org.apache.hadoop.mapred.TextInputFormat'\n" + + "OUTPUTFORMAT\n 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'\n"), ddl); + Assertions.assertTrue(ddl.contains("LOCATION\n 'hdfs://ns/wh/t'\n"), ddl); + } + + @Test + public void testTwoQuotingConventions() { + // The discriminator the suites rely on: SERDEPROPERTIES has SPACES around '=', TBLPROPERTIES has NONE. + String ddl = HiveShowCreateTableRenderer.render(textTable()); + Assertions.assertTrue(ddl.contains("WITH SERDEPROPERTIES (\n 'field.delim' = '|')\n"), ddl); + Assertions.assertTrue(ddl.contains("'doris.file_format'='text'"), ddl); + } + + @Test + public void testCommentParamNotLeakedIntoTblproperties() { + // The comment was lifted to COMMENT '...'; it must NOT also appear as a TBLPROPERTY ('comment'='my table'). + String ddl = HiveShowCreateTableRenderer.render(textTable()); + Assertions.assertFalse(ddl.contains("'comment'='my table'"), ddl); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 7757411d837a66..e01cf75cf455a5 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -589,6 +589,31 @@ public String getViewText() { return definition.getSql(); } + /** + * Renders the connector's native {@code SHOW CREATE TABLE} DDL (a fresh, cache-bypassing metastore read) for + * the SHOW CREATE TABLE command's connector arm. Mirrors {@link #getViewText}'s single connector round-trip + * (and, like it, is safe to call under the command's table read-lock — no {@code makeSureInitialized}). Returns + * {@link Optional#empty()} when the connector supplies no native DDL (iceberg/paimon/es/jdbc inherit the empty + * SPI default {@link ConnectorMetadata#renderShowCreateTableDdl}), or when the handle cannot be resolved — the + * command then falls through to the generic {@code Env.getDdlStmt} rendering unchanged. A native-rendering + * connector (hive) returns the full statement, fetched fresh so it reflects a just-applied external ALTER even + * while DESC serves a cached schema. + */ + public Optional getShowCreateTableDdl() { + PluginDrivenExternalCatalog pluginCatalog = (PluginDrivenExternalCatalog) catalog; + Connector connector = pluginCatalog.getConnector(); + if (connector == null) { + return Optional.empty(); + } + ConnectorSession session = pluginCatalog.buildConnectorSession(); + ConnectorMetadata metadata = connector.getMetadata(session); + Optional handleOpt = resolveConnectorTableHandle(session, metadata); + if (!handleOpt.isPresent()) { + return Optional.empty(); + } + return metadata.renderShowCreateTableDdl(session, handleOpt.get()); + } + @Override public boolean isPartitionedTable() { makeSureInitialized(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateTableCommand.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateTableCommand.java index e833958d2bb04d..ea1b3653e47902 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateTableCommand.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ShowCreateTableCommand.java @@ -168,6 +168,18 @@ public ShowResultSet doRun(ConnectContext ctx, StmtExecutor executor) throws Exc + ((PluginDrivenExternalTable) table).getViewText())); return new ShowResultSet(META_DATA, rows); } + if (table instanceof PluginDrivenExternalTable) { + // Native connector-rendered SHOW CREATE (hive: ROW FORMAT SERDE / STORED AS ..., fetched fresh), + // reached only for a non-view plugin table (the view arm above returns first). The guard is the + // method returning a value — NOT the source name — so iceberg/paimon/es/jdbc (empty SPI default) + // fall through to Env.getDdlStmt below and render exactly as today; only a connector that natively + // renders its DDL short-circuits here. Delegated iceberg/hudi-on-HMS tables also return empty. + Optional nativeDdl = ((PluginDrivenExternalTable) table).getShowCreateTableDdl(); + if (nativeDdl.isPresent()) { + rows.add(Arrays.asList(table.getName(), nativeDdl.get())); + return new ShowResultSet(META_DATA, rows); + } + } List createTableStmt = Lists.newArrayList(); Env.getDdlStmt(null, null, table, createTableStmt, null, null, false, true /* hide password */, false, -1L, isBrief, false); From dbd9b4192e2fddc00df68f2fe86998fd9948bcfc Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 00:22:30 +0800 Subject: [PATCH 272/334] [doc](catalog) hive-e2e: land FIX-R2 (SHOW CREATE) design, mark done, repoint to R3 Records the native SHOW CREATE TABLE fix (17764d03665): Option X (lazy connector render + fresh HMS fetch, intercepted in ShowCreateTableCommand, capability NOT declared), why the two earlier designs were red-teamed out (fresh-fetch contract; capability blast radius), the 7-file change, and the residual fidelity notes. HANDOFF: R2 moved into batch-3 DONE, remaining 8 -> 6 suites. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 7 +- .../tasks/designs/FIX-R2-showcreate-design.md | 90 +++++++++++++++++++ 2 files changed, 94 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-R2-showcreate-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 849bedd783907c..523b6e1b9ab7f5 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -12,13 +12,14 @@ > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > > **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **✅ batch-3（R7/R10 前序 session；R12 + text_write(LZ4) 本 session，共 4 fix DONE）**： +> **✅ batch-3（R7/R10 前序 session；R12 + text_write(LZ4) + R2(SHOW CREATE) 本 session，共 5 fix DONE）**： > - **R7 `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。红队 4 lens 全 SOUND。`CachingHmsClientTest` 19/19。设计 `designs/FIX-R7-design.md`。**e2e 待用户自跑 test_hive_use_meta_cache_true sql09。** > - **R10 `9c70d4acf9a`**——OpenX JSON `ignore.malformed.json` 跳坏行。`HiveTextProperties.extractJsonSerDeProps` 串入 sd/table params，**仅 OpenX serde**发 `hive.text.openx_ignore_malformed`(table-param 优先、默认 false；hcatalog/hive2 从不带此旗，钉 OpenX 保 legacy 分支 parity) + fe-core `PluginDrivenScanNode.getFileAttributes` 在既有 `is_json` 块读键调 `setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode OPENX 分支)。one-column 模式 OpenX 解析成 CSV、BE CSV reader 不读该 JSON-only 字段故无碍。红队 3 lens(初版"全 3 JSON serde"被驳→已收敛 OpenX-only；test/iron-rule SOUND)。`HiveTextPropertiesTest` 13/13。设计 `designs/FIX-R10-design.md`。**e2e 待用户自跑 test_hive_openx_json q1。** > - **R12/serde `3936434bc9a`**——OpenCSV 表列全塌成 STRING(修 test_open_csv_serde + test_hive_serde_prop 的 OpenCSV 半)。根因=legacy 默认走 metastore `get_schema` RPC(服务端反序列化器→OpenCSV 全 string)，新连接器只读原始 `sd.getCols()` 声明类型→typed 列到 BE 翻车(TRUE vs 'true'、datetime→ISO、空串 vs NULL)。**方案定案=B(用户 2026-07-11 签字)**：不走 RPC、连接器就地复刻结果——`HiveConnectorMetadata.buildColumns` 加 `coerceOpenCsvColumnsToString`,serde==OpenCSV 时把**数据列**整型别塌成 `STRING`(复合类型也整列塌)、复用现成 `HiveTextProperties.HIVE_OPEN_CSV_SERDE` 常量。**架构红队关键订正**：放 hive 元数据层(非 hive/hudi 共用的 `ThriftHmsClient`,那会误触 hudi schema 路径 + 重复常量)——对齐 Trino(在 HiveMetadata 层做 CSV=全 string)。分区列不动(hive 在反序列化后追加、保声明类型)、view 有 isView 门。非-OpenCSV 表零改动(与原 `sd.getCols()` 逐位一致)。fe-core 零改、无 serde 名分支。3 路对抗红队(correctness SOUND / architecture 订正放置层 / blast 把验收扩到 4 suite)。`HiveConnectorMetadataSchemaTest` +4(int/datetime/bool/**array**→STRING、分区键留 date、LazySimple 门留 typed、view 门)、RED 实证(旁路→expected STRING but was INT)、306/306、0 checkstyle。设计 `designs/FIX-R12-design.md`。**残余(非本修)**：Avro schema-url 自述表 A/B 唯一分歧,新连接器**现状本就没有**、不在失败套件、以后需要时独立小任务补。**e2e 待用户自跑：test_open_csv_serde + test_hive_serde_prop(+对照 test_drop_expired_table_stats/test_trino_hive_serde_prop)。** > - **text_write(LZ4) `e1d48045bee`**——`.lz4` 文本表读炸 BE `LZ4F_getFrameInfo ERROR_frameType_unknown`。根因=`.lz4` 被 `Util.inferFileCompressTypeByPath` 推成 LZ4FRAME 发 BE，但 hadoop/hive 写的是 LZ4 **block** 编码；legacy `HiveScanNode.getFileCompressType` 有 LZ4FRAME→LZ4BLOCK remap，SPI 翻闸后丢了。修=连接器 opt-in（镜像 supportsTableSample/supportsBatchScan）：`ConnectorScanPlanProvider.adjustFileCompressType` 默认 identity → `PluginDrivenScanNode.getFileCompressType` 跑 super 推断后委派（resolveScanProvider+onPluginClassLoader，无 source 分支）→ `HiveScanPlanProvider` override 只 remap LZ4FRAME。**hudi 严格对齐**（红队定：legacy `HudiScanNode extends HiveScanNode` 继承了 remap，新独立 provider 须重声明，别赌"hudi 从不出 .lz4"）；其余 6 provider 留 identity。测试：api identity 1/1 + hive remap 5/5 + hudi parity 4/4（api+hive 对子=RED/GREEN），fe-core 节点 54/54 无回归，0 checkstyle。设计 `designs/FIX-text-write-LZ4-design.md`。**e2e 待用户自跑 test_hive_text_write_insert 的 lz4 迭代。** -> **⭐ 剩余 8 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **fe-connector/fe-core 大型**：**R2 SHOW CREATE TABLE 原生 DDL（show_create_table + ddl_text_format）——⚠ recon+红队(wf_992d12bd-d60)已把原方案证伪、需订正后实现**：SPI 早有 `SUPPORTS_SHOW_CREATE_DDL`+fe-core 路由(iceberg/paimon 用),但 iceberg/paimon 是**数据驱动**(发 show.* 渲染提示键、Env.getDdlStmt 拼 Doris 风格 DDL),**结构上拼不出** hive 原生 `ROW FORMAT SERDE`/单引号 SERDEPROPERTIES/`STORED AS`。且**红队三杀原"eager 缓存键"方案**：(a) `test_hive_meta_cache` L354-357 要求 show create **实时取 HMS**(desc 缓存 5 列、show-create 须 6 列)→eager 缓存键结构上做不到,须**惰性方法命令期新取**;(b) **声明 capability** 连接器级会翻转 `Env.getDdlStmt` 门→误触委派的 hudi/iceberg-on-HMS 表 + HTTP schema 端点;(c) 验收套件被低估(还有 test_hive_meta_cache/test_multi_delimit_serde/test_hive_ddl)。**订正方向=惰性 typed 方法**(`ConnectorMetadata.getShowCreateTableDdl` 默认空、hive 就地新取 HMS 渲染,delegated/view 返空) + `ShowCreateTableCommand` 新臂(非 Env.getDdlStmt、不声明 capability=Option X)。渲染器镜像 legacy `HiveMetaStoreClientHelper.showCreateTable`(单/双引号两套约定)。**下一步：二轮设计+红队锁 Option X → 中文向用户呈请拍板(X vs Y/capability) → 实现。** · R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 +> - **R2(SHOW CREATE) `17764d03665`**——翻闸 hive 表 SHOW CREATE 出 Doris 通用风格(双引号 PROPERTIES),非 hive 原生 `ROW FORMAT SERDE`/`STORED AS`。**方案=Option X(用户 2026-07-12 签字)**:惰性连接器方法 + **命令期实时取 HMS** + 连接器侧渲染,只在 `ShowCreateTableCommand` 拦截、**不声明 capability**。**两版旧方案被红队证伪**:(1)eager schema-cache 渲染过不了 `test_hive_meta_cache`(desc 缓存 5 列、show create 须新取 6 列);(2)声明 capability 会翻转共享 `Env.getDdlStmt` 门→误触 delegated hudi/iceberg-on-HMS + HTTP 端点。**7 文件**:`ConnectorTableOps.renderShowCreateTableDdl` 默认空(iceberg/paimon/es/jdbc 继承→落 Env.getDdlStmt 不变) · `HmsClient.getTableFresh`+`CachingHmsClient` override(绕 tableCache,镜像 listPartitionNamesFresh) · `HiveShowCreateTableRenderer`(fe-connector-hms,逐字节镜像 legacy `HiveMetaStoreClientHelper.showCreateTable`:两套引号约定 SERDEPROPERTIES `'k' = 'v'`/TBLPROPERTIES `'k'='v'`、2空格/1空格缩进、comment 提到 COMMENT 子句、null-comment 门) · `HiveConnectorMetadata` override(非 HiveTableHandle=delegated→空,镜像 getTableSchema 门;fresh getTableFresh) · `PluginDrivenExternalTable.getShowCreateTableDdl`(镜像 getViewText、读锁安全、handle 未解析→空) · `ShowCreateTableCommand` view 臂后新臂(门=方法返回 present 非 source 名,iceberg/paimon/view 不受影响)。**两红队 lens 均 SOUND_WITH_FIXES→已折入 2 守卫(foreign-handle 门 + 未解析→空)**。fe-core 连接器无关、无 source 分支;TCCL 在 `ThriftHmsClient.doAs` 内钉、renderer reflection-free。测试:`HiveShowCreateTableRendererTest` 5(byte-parity,两套引号+null-comment 门) + `CachingHmsClientTest` +3(getTableFresh 双向绕缓存)、fe-core PluginDrivenExternalTable 36/36 + EnvShowCreate 3/3 不变、hive 307/307、0 checkstyle。设计 `designs/FIX-R2-showcreate-design.md`。**残余**:toHiveTypeString 对 `varchar(n)`→`string`(HMS 本就存 string 故无碍)、不可映射类型抛 IAE(legacy 回显原串、无套件触发);TBLPROPERTIES 逐字发含 volatile 键(同 legacy,勿加 .out golden)。**e2e 待用户自跑:test_hive_show_create_table + test_hive_ddl_text_format(+ test_hive_meta_cache 新取契约 / test_multi_delimit_serde / test_hive_ddl)。** +> **⭐ 剩余 6 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **fe-connector/fe-core 大型**：R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 > - **用户已裁决(A)的 3 个 fe-core/SPI 大改**： > - query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 > - default_partition 连接器传 null 标记：SPI 新字段让连接器按分区值传 null 语义，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem:310`(现无条件 `PartitionValue(v,false)`) 据此建项——hive 标 null、paimon 保持非空(零 paimon 影响，别在 fe-core if(hive))。 diff --git a/plan-doc/tasks/designs/FIX-R2-showcreate-design.md b/plan-doc/tasks/designs/FIX-R2-showcreate-design.md new file mode 100644 index 00000000000000..0daa92312c1200 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-R2-showcreate-design.md @@ -0,0 +1,90 @@ +# FIX-R2 — Native hive SHOW CREATE TABLE on the flipped plugin path (Option X) + +**Suites fixed:** `test_hive_show_create_table`, `test_hive_ddl_text_format` (+ acceptance guards +`test_hive_meta_cache`, `test_multi_delimit_serde`, `test_hive_ddl`). +**Scope:** lazy connector method + fresh HMS fetch + connector-side renderer; fe-core stays connector-agnostic. +**Status:** code DONE (`17764d03665`). e2e awaits user self-run. + +--- + +## 1. Symptom + +`SHOW CREATE TABLE` on a flipped (plugin) hive table emitted generic Doris-style DDL (double-quoted +`PROPERTIES ("k" = "v")`) instead of native hive DDL — no `ROW FORMAT SERDE`, `WITH SERDEPROPERTIES`, +`STORED AS INPUTFORMAT/OUTPUTFORMAT`. The suites substring-assert the native clauses and their exact quoting. + +## 2. Root cause + +Legacy rendered native hive DDL in `ShowCreateTableCommand`'s `HMS_EXTERNAL_TABLE` arm +(`HiveMetaStoreClientHelper.showCreateTable`). Post-cutover a hive table is `PLUGIN_EXTERNAL_TABLE`, so that arm +is dead and it falls to the generic `Env.getDdlStmt` assembler, which structurally cannot emit hive serde/format +clauses and double-quotes property keys. + +## 3. Two earlier designs red-teamed OUT (why Option X) + +- **Eager reserved-key rendered in `getTableSchema`** — REFUTED: `test_hive_meta_cache` (L354-357) requires SHOW + CREATE to fetch **fresh** from HMS at command time (after an external `ADD COLUMNS`, `DESC` shows the cached + 5-col schema but SHOW CREATE must show the fresh 6). An eager render baked into the schema-cache value is as + stale as DESC. +- **Declaring `SUPPORTS_SHOW_CREATE_DDL` + rendering in `Env.getDdlStmt` (Option Y)** — REFUTED: the capability + gate is connector-wide, so it would flip for **delegated hudi/iceberg-on-HMS** tables and every other + `Env.getDdlStmt` caller (the HTTP `StmtExecutionAction.getSchema` endpoint) — coupling hive can't honor. + +**Decision: Option X — user signed off 2026-07-12.** Lazy connector render + fresh fetch, intercepted only in +`ShowCreateTableCommand`, capability NOT declared. (Trino renders hive SHOW CREATE engine-side in Trino syntax, +not native hive DDL, so neither X nor Y mirrors Trino; native parity is a Doris-legacy requirement best served by +a connector-returned verbatim string.) + +## 4. The change (7 files) + +1. **`ConnectorTableOps.renderShowCreateTableDdl(session, handle)`** — new `default Optional` = empty. + iceberg/paimon/es/jdbc inherit it → return empty → fall through to `Env.getDdlStmt` byte-inert. +2. **`HmsClient.getTableFresh` + `CachingHmsClient` override** — cache-bypassing table read (exact mirror of the + `listPartitionNamesFresh` precedent; neither reads nor writes `tableCache`). This is what makes SHOW CREATE + fresh while DESC stays cached. +3. **`HiveShowCreateTableRenderer`** (fe-connector-hms, new) — byte-for-byte port of legacy + `HiveMetaStoreClientHelper.showCreateTable` base-table branch. Load-bearing details: **two quoting + conventions** (`WITH SERDEPROPERTIES ('k' = 'v')` spaces vs `TBLPROPERTIES ('k'='v')` none), 2-space data / + 1-space partition indents, the `comment` param lifted to a top-level `COMMENT` clause, and the **null-comment + guard** (an empty `COMMENT ''` would break meta_cache's exact column substring). Types via + `HmsTypeMapping.toHiveTypeString`. +4. **`HiveConnectorMetadata.renderShowCreateTableDdl`** — guards `!(handle instanceof HiveTableHandle)` → empty + (a delegated iceberg/hudi-on-HMS table routes through THIS hive gateway metadata and carries the sibling's + foreign handle; mirror `getTableSchema`'s guard, whose comment already notes "show-create ... is inert here"). + Fresh `getTableFresh` + render. +5. **`PluginDrivenExternalTable.getShowCreateTableDdl()`** — mirrors `getViewText` (safe under the command's + read-lock; no `makeSureInitialized`); returns empty on unresolved handle (red-team soften, keeps iceberg/paimon + at today's behavior even in edge cases). +6. **`ShowCreateTableCommand`** — new arm after the view arm, before the `Env.getDdlStmt` fallthrough. Guard is + the method returning present (NOT `instanceof`/source name), so iceberg/paimon (empty) and flipped views (view + arm fires first) are untouched. `Env.getDdlStmt` is not modified. + +**Iron rule:** fe-core branches only on the generic `PluginDrivenExternalTable` type + the method result, never on +"hive". All serde/format/DDL formatting lives in `fe-connector-hms`. **TCCL:** the fetch pins inside +`ThriftHmsClient.doAs`; no fe-core pin (matches the `getMetadataTableRows`/`getChunkSizes` precedent). The renderer +is reflection-free, so it is CL-inert on the fe-core thread — noted in its javadoc for future maintainers. + +## 5. Tests — RED/GREEN + +- `HiveShowCreateTableRendererTest` (fe-connector-hms, 5): byte-parity — column block indent/types + null-comment + guard, comment-lift + 1-space partition indent, serde/STORED-AS/LOCATION blocks, the two quoting conventions, + and comment-not-leaked-into-TBLPROPERTIES. RED before the class existed / against any wrong-quoting impl. +- `CachingHmsClientTest` +3: `getTableFresh` always hits the delegate, does not populate the cache, and the + non-caching default equals a plain `getTable`. RED without the `CachingHmsClient` override (would serve cached). +- Regression: fe-core `PluginDrivenExternalTableTest` 36/36 + `EnvShowCreatePluginTableTest` 3/3 (capability arm + unchanged — hive still does not declare it); fe-connector-hive 307/307; 0 checkstyle. + +## 6. Residual / fidelity notes + +- **`toHiveTypeString` fidelity**: a raw HMS `varchar(n)` renders as `string` (length dropped) — harmless here + because such columns are stored as `string` in HMS. `char(n)`/`decimal(p,s)`/date/timestamp/nested round-trip + cleanly. An unmappable type throws `IllegalArgumentException` (legacy echoed the raw string and could not hit + this) — left to fail loud rather than guessed (not fed by any suite). +- **TBLPROPERTIES** emits all non-`comment` params verbatim (incl. volatile `transient_lastDdlTime` etc.), matching + legacy. Suites assert substrings only, so no `.out` golden — do not add one (it would be flaky). + +## 7. Acceptance (e2e — user self-run) + +`external_table_p0/hive/`: `test_hive_show_create_table` (hive2), `test_hive_ddl_text_format` (hive2+hive3), +`test_hive_meta_cache` (the fresh-fetch contract: DESC=5 cached / SHOW CREATE=6 fresh), `test_multi_delimit_serde`, +`test_hive_ddl`. All are inline `contains`/`assertTrue` (no `.out` goldens). From e697f189c591d373034c5ee52f5285d2c680e221 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 01:13:13 +0800 Subject: [PATCH 273/334] [fix](catalog) hive: expose $partitions sys table via partition_values TVF (R3) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A flipped plain-hive table is a PluginDrivenExternalTable, whose getSupportedSysTables() discovers sys-table names from the connector and wrapped every name as a native PluginDrivenSysTable. HiveConnectorMetadata reported none, so `t$partitions` resolved to "Unknown sys table" (and `desc t$partitions` to "sys table not found"). Legacy HMSExternalTable returned PartitionsSysTable — a TVF-backed sys table routed to the generic partition_values TVF (already fully plugin-aware, incl. the BE->FE metadata fetch); only the $partitions suffix -> sys-table -> TVF bridge was lost. fe-core cannot map the bare name "partitions" -> TVF unconditionally: iceberg (MetadataTableType.PARTITIONS) and paimon (SystemTableLoader) also report "partitions" as NATIVE metadata tables, so a name-based map would misroute their t$partitions into the hive TVF. The kind must be connector-declared. Fix (connector-declared kind, supports*-style opt-in, iron-rule clean): - SPI ConnectorTableOps.isPartitionValuesSysTable(session,handle,sysName), default false (native). fe-core cannot infer the kind from an empty getSysTableHandle (kind is needed at discovery; getSysTableHandle is eager/auth-bearing; iceberg legitimately omits position_deletes). - fe-core PluginDrivenExternalTable.getSupportedSysTables() branches on it: TVF-declared -> PartitionsSysTable.INSTANCE (keyed on its own name to avoid the createFunction suffix-strip crash), else native PluginDrivenSysTable. SysTableResolver already routes any TvfSysTable through the TVF path for SELECT/DESCRIBE/validate. - HiveConnectorMetadata: listSupportedSysTables returns ["partitions"] UNCONDITIONALLY for a HiveTableHandle (partitioned or not) so a $partitions query on a non-partitioned table reaches the TVF ctor and throws "... is not a partitioned table", mirroring legacy; isPartitionValuesSysTable returns true for "partitions", delegating a foreign (iceberg/hudi-on-HMS) handle to the sibling so those stay native; getSysTableHandle stays empty (TVF path never consults it). No BE / Nereids change. iceberg/paimon (native + on-HMS) t$partitions unchanged (default-false / sibling delegation). Design red-teamed 3-lens -> GO_WITH_FIXES (all folded). Tests: new HiveConnectorMetadataSysTableTest (5), HiveConnectorMetadataSiblingDelegationTest (delegation + hive-exposes- partitions), fe-core PluginDrivenSysTableTest (+1 TVF-kind wrap). Verified: fe-connector-hive 312/312, fe-core PluginDrivenSysTableTest 11/11, 0 checkstyle. e2e (user-run): test_hive_partition_values_tvf + iceberg-on-HMS/ native iceberg/paimon t$partitions still native. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/api/ConnectorTableOps.java | 18 +++ .../connector/hive/HiveConnectorMetadata.java | 23 +++- ...onnectorMetadataSiblingDelegationTest.java | 28 ++++- .../HiveConnectorMetadataSysTableTest.java | 107 ++++++++++++++++++ .../datasource/PluginDrivenExternalTable.java | 11 +- .../datasource/PluginDrivenSysTableTest.java | 39 +++++++ 6 files changed, 218 insertions(+), 8 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSysTableTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java index 5e1f21ab28eabb..aa6d38ef669983 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableOps.java @@ -71,6 +71,24 @@ default Optional getSysTableHandle(ConnectorSession sessio return Optional.empty(); } + /** + * Whether the named system table of {@code baseTableHandle} is served by the generic + * {@code partition_values} table-valued function (fe-core's {@code PartitionsSysTable}) rather + * than by a native connector scan. Default {@code false} (native, the {@link #getSysTableHandle} + * path). + * + *

    A connector whose partitioned tables expose their partition rows through the generic + * partition-values TVF (e.g. hive) overrides this to return {@code true} for that sys-table name; + * such a name need NOT return a handle from {@link #getSysTableHandle} (the TVF path never consults + * it). fe-core needs the kind at discovery time (before any handle is fetched), so it cannot be + * inferred from an empty {@code getSysTableHandle}. {@code sysName} is the bare name (no + * {@code "$"}).

    + */ + default boolean isPartitionValuesSysTable(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + return false; + } + /** Returns the schema (columns, format, etc.) for the given table. */ default ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 7a61083b44a577..73d9ea6b837f64 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -1349,8 +1349,11 @@ public List listSupportedSysTables(ConnectorSession session, ConnectorTa if (!(baseTableHandle instanceof HiveTableHandle)) { return siblingMetadata(session, baseTableHandle).listSupportedSysTables(session, baseTableHandle); } - // Hive exposes no system tables (SPI default empty). - return Collections.emptyList(); + // Hive exposes the "partitions" system table (t$partitions), served by the generic partition_values + // TVF (see isPartitionValuesSysTable). Exposed UNCONDITIONALLY (partitioned or not), mirroring legacy + // HMSExternalTable.getSupportedSysTables for dlaType==HIVE: a $partitions query on a NON-partitioned + // table must reach the TVF and throw "… is not a partitioned table", not "Unknown sys table". + return List.of("partitions"); } @Override @@ -1360,10 +1363,24 @@ public Optional getSysTableHandle(ConnectorSession session // Return the sibling's sys-table handle UNMODIFIED (a rewrap would poison a downstream scan cast). return siblingMetadata(session, baseTableHandle).getSysTableHandle(session, baseTableHandle, sysName); } - // Hive exposes no system tables (SPI default empty). + // Hive's "partitions" sys table is TVF-backed (isPartitionValuesSysTable), so the native handle path + // never consults this — no hive sys-table handle to return. return Optional.empty(); } + @Override + public boolean isPartitionValuesSysTable(ConnectorSession session, ConnectorTableHandle baseTableHandle, + String sysName) { + if (!(baseTableHandle instanceof HiveTableHandle)) { + // A foreign (iceberg/hudi-on-HMS) handle's sys tables are NATIVE — delegate so fe-core wraps them + // native. Dropping this guard would misroute an iceberg-on-HMS t$partitions into the hive TVF. + return siblingMetadata(session, baseTableHandle).isPartitionValuesSysTable(session, baseTableHandle, + sysName); + } + // Plain hive's only sys table, "partitions", is served by the generic partition_values TVF. + return "partitions".equals(sysName); + } + /** * Renders {@code key1=v1/key2=v2} from partition-key names + values, byte-parity with legacy * {@code HivePartition.getPartitionName} (a raw, unescaped {@code name=value} join). Used only to label diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java index 4798f513f3123e..91089872bfc484 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSiblingDelegationTest.java @@ -140,6 +140,9 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { ConnectorTableHandle afterTopn = md.applyTopnLazyMaterialization(null, foreignHandle); List sysTables = md.listSupportedSysTables(null, foreignHandle); Optional sysHandle = md.getSysTableHandle(null, foreignHandle, "snapshots"); + // "snapshots" (not "partitions"): hive's own logic returns false for it, so a true answer proves the + // reply came from the sibling, not hive. + boolean sysIsTvf = md.isPartitionValuesSysTable(null, foreignHandle, "snapshots"); // Every per-handle method reached the sibling (proves the divert covers the whole surface). Assertions.assertEquals(RecordingSiblingMetadata.EXPECTED_METHODS, siblingMetadata.calls, @@ -153,7 +156,10 @@ public void everyPerHandleMethodForwardsAForeignHandleToTheSibling() { Assertions.assertEquals(Collections.singletonList("sibling-part"), partNames, "listPartitionNames must return the sibling's names"); Assertions.assertEquals(Collections.singletonList("snapshots"), sysTables, - "iceberg-on-HMS system tables must resolve through the sibling (hive exposes none)"); + "iceberg-on-HMS system tables must resolve through the sibling (hive exposes only partitions)"); + Assertions.assertTrue(sysIsTvf, + "isPartitionValuesSysTable must return the sibling's answer for a foreign handle, not hive's — " + + "dropping this delegation would misroute an iceberg-on-HMS t$partitions into the hive TVF"); Assertions.assertSame(RecordingSiblingMetadata.SIBLING_TIMELINE_ROWS, timelineRows, "getMetadataTableRows must return the sibling's timeline rows, not hive's empty default — a " + "hudi-on-HMS hudi_meta()/TIMELINE read gets its rows from the hudi sibling post-flip"); @@ -189,9 +195,14 @@ public void hiveHandleRunsHiveBranchAndNeverConsultsSibling() { "hive applyRewriteFileScope returns the handle"); Assertions.assertSame(hive, md.applyTopnLazyMaterialization(null, hive), "hive applyTopnLazyMaterialization returns the handle"); - Assertions.assertTrue(md.listSupportedSysTables(null, hive).isEmpty(), "hive exposes no system tables"); + Assertions.assertEquals(Collections.singletonList("partitions"), md.listSupportedSysTables(null, hive), + "hive exposes the partitions sys table (t$partitions), served by the partition_values TVF"); + Assertions.assertTrue(md.isPartitionValuesSysTable(null, hive, "partitions"), + "hive's partitions sys table is TVF-backed"); + Assertions.assertFalse(md.isPartitionValuesSysTable(null, hive, "snapshots"), + "hive exposes no sys table other than partitions"); Assertions.assertFalse(md.getSysTableHandle(null, hive, "snapshots").isPresent(), - "hive exposes no system tables"); + "hive's TVF-backed sys table has no native handle"); ConnectorMvccSnapshot pin = md.beginQuerySnapshot(null, hive).orElse(null); Assertions.assertNotNull(pin); Assertions.assertEquals(-1L, pin.getSnapshotId(), "hive's pin is the empty (-1) last-modified pin"); @@ -466,7 +477,8 @@ private static final class RecordingSiblingMetadata implements ConnectorMetadata "beginQuerySnapshot", "getTableFreshness", "getPartitionFreshnessMillis", "dropTable", "truncateTable", "getTableSchemaAtSnapshot", "getMvccPartitionView", "resolveTimeTravel", "applySnapshot", "getSyntheticScanPredicates", "applyRewriteFileScope", - "applyTopnLazyMaterialization", "listSupportedSysTables", "getSysTableHandle")); + "applyTopnLazyMaterialization", "listSupportedSysTables", "getSysTableHandle", + "isPartitionValuesSysTable")); // The exact set + order of ALTER-DDL / validate methods the foreign-handle write test drives (Rule-9 // completeness lock for §4.4 W1: dropping a guard, or adding one that should not forward, fails the test). @@ -639,6 +651,14 @@ public Optional getSysTableHandle(ConnectorSession session return Optional.of(SIBLING_HANDLE); } + @Override + public boolean isPartitionValuesSysTable(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + calls.add("isPartitionValuesSysTable"); + // A distinctive true (hive's own logic would say false for "snapshots") proves the divert. + return true; + } + // ---- §4.4 W1: ALTER-DDL mutators + write validators (the write-delegation surface) ---- @Override diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSysTableTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSysTableTest.java new file mode 100644 index 00000000000000..5a2bf073f4144d --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSysTableTest.java @@ -0,0 +1,107 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.apache.doris.connector.api.handle.ConnectorTableHandle; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Optional; + +/** + * Tests the hive connector's system-table exposure ({@code t$partitions}), the FIX-R3 restoration of + * legacy {@code HMSExternalTable.getSupportedSysTables()} for {@code dlaType==HIVE}. + * + *

    WHY these assertions matter:

    + *
      + *
    • {@code partitions} is advertised, TVF-backed. A flipped hive table is a + * {@code PluginDrivenExternalTable}; fe-core discovers its sys tables from + * {@code listSupportedSysTables} and asks {@code isPartitionValuesSysTable} whether each is served + * by the generic {@code partition_values} TVF (fe-core {@code PartitionsSysTable}) vs a native + * scan. Reporting nothing here left {@code t$partitions} resolving to "Unknown sys table".
      • + *
    • Exposed UNCONDITIONALLY (partitioned or not). Mirrors legacy: a {@code $partitions} + * query on a NON-partitioned table must reach the TVF and throw "… is not a partitioned table", + * not "Unknown sys table" — so a non-partitioned handle must still advertise {@code partitions}.
      • + *
    • No native handle. Because {@code partitions} is TVF-backed, {@code getSysTableHandle} + * stays empty — the native handle path must never be consulted for it.
      • + *
    + * + *

    The hive-handle path never touches the metastore client, so the client is {@code null} + * (mirroring {@code HiveConnectorMetadataSiblingDelegationTest}); a foreign-handle divert is covered + * there.

    + */ +public class HiveConnectorMetadataSysTableTest { + + private HiveConnectorMetadata metadata() { + return new HiveConnectorMetadata(null, Collections.emptyMap(), new FakeConnectorContext()); + } + + private HiveTableHandle partitionedHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .partitionKeyNames(Collections.singletonList("p")) + .build(); + } + + private HiveTableHandle unpartitionedHandle() { + return new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE).build(); + } + + @Test + public void listsOnlyPartitionsForAHiveHandle() { + Assertions.assertEquals(Collections.singletonList("partitions"), + metadata().listSupportedSysTables(null, partitionedHandle()), + "hive exposes the partitions sys table (t$partitions), served by the partition_values TVF"); + } + + @Test + public void exposesPartitionsUnconditionallyEvenWhenUnpartitioned() { + // Load-bearing: a $partitions query on a non-partitioned table must reach the TVF (which throws + // "is not a partitioned table"), not short-circuit to "Unknown sys table". So it must still advertise. + Assertions.assertEquals(Collections.singletonList("partitions"), + metadata().listSupportedSysTables(null, unpartitionedHandle()), + "partitions must be advertised for a non-partitioned hive table too (legacy parity)"); + } + + @Test + public void partitionsIsPartitionValuesTvfBacked() { + Assertions.assertTrue(metadata().isPartitionValuesSysTable(null, partitionedHandle(), "partitions"), + "hive's partitions sys table is served by the generic partition_values TVF"); + } + + @Test + public void onlyPartitionsIsTvfBacked() { + HiveConnectorMetadata md = metadata(); + HiveTableHandle h = partitionedHandle(); + Assertions.assertFalse(md.isPartitionValuesSysTable(null, h, "snapshots"), + "hive exposes no sys table other than partitions"); + Assertions.assertFalse(md.isPartitionValuesSysTable(null, h, "PARTITIONS"), + "the sys-table name is case-sensitive lowercase (findSysTable is exact-match)"); + Assertions.assertFalse(md.isPartitionValuesSysTable(null, h, null), + "a null sys name is simply not exposed (no NPE)"); + } + + @Test + public void tvfBackedSysTableHasNoNativeHandle() { + Optional handle = + metadata().getSysTableHandle(null, partitionedHandle(), "partitions"); + Assertions.assertFalse(handle.isPresent(), + "a TVF-backed sys table is served without a native connector handle"); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index e01cf75cf455a5..11354a304c3b4b 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -41,6 +41,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.write.ConnectorWritePlanProvider; import org.apache.doris.datasource.mvcc.MvccSnapshot; +import org.apache.doris.datasource.systable.PartitionsSysTable; import org.apache.doris.datasource.systable.PluginDrivenSysTable; import org.apache.doris.datasource.systable.SysTable; import org.apache.doris.qe.ConnectContext; @@ -966,7 +967,15 @@ public Map getSupportedSysTables() { // case-sensitive findSysTable exact-match works, mirroring legacy PaimonSysTable keys. Map result = Maps.newHashMapWithExpectedSize(names.size()); for (String sysName : names) { - result.put(sysName, new PluginDrivenSysTable(sysName)); + if (metadata.isPartitionValuesSysTable(session, handleOpt.get(), sysName)) { + // Connector declares this name is served by the generic partition_values TVF (e.g. hive + // t$partitions), not a native scan. Key on the singleton's OWN name (== "partitions"): + // PartitionsSysTable strips its hard-wired "$partitions" suffix in createFunction, so a + // differing key would crash there; identical to sysName for hive today, strictly safer. + result.put(PartitionsSysTable.INSTANCE.getSysTableName(), PartitionsSysTable.INSTANCE); + } else { + result.put(sysName, new PluginDrivenSysTable(sysName)); + } } return Collections.unmodifiableMap(result); } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java index 60622d34e8da91..cbfbc3ad30d88c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenSysTableTest.java @@ -25,8 +25,10 @@ import org.apache.doris.connector.api.ConnectorTableSchema; import org.apache.doris.connector.api.ConnectorType; import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.datasource.systable.PartitionsSysTable; import org.apache.doris.datasource.systable.PluginDrivenSysTable; import org.apache.doris.datasource.systable.SysTable; +import org.apache.doris.datasource.systable.TvfSysTable; import com.google.gson.annotations.SerializedName; import org.junit.jupiter.api.Assertions; @@ -82,6 +84,43 @@ public void testGetSupportedSysTablesDelegatesToConnector() { Mockito.verify(metadata).listSupportedSysTables(session, baseHandle); } + @Test + public void testGetSupportedSysTablesMapsTvfKindToPartitionsSysTable() { + ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); + ConnectorSession session = Mockito.mock(ConnectorSession.class); + ConnectorTableHandle baseHandle = Mockito.mock(ConnectorTableHandle.class); + TestablePluginCatalog catalog = new TestablePluginCatalog("hms", metadata, session); + Mockito.when(metadata.getTableHandle(session, "REMOTE_DB", "REMOTE_TBL")) + .thenReturn(Optional.of(baseHandle)); + Mockito.when(metadata.listSupportedSysTables(session, baseHandle)) + .thenReturn(Arrays.asList("partitions", "snapshots")); + // The connector declares only "partitions" as partition_values-TVF-backed (hive). "snapshots" + // defaults to native. (mockito returns false for the unstubbed boolean call.) + Mockito.when(metadata.isPartitionValuesSysTable(session, baseHandle, "partitions")) + .thenReturn(true); + + PluginDrivenExternalTable table = bareTable(catalog, mockDb("REMOTE_DB"), "REMOTE_TBL"); + Map sysTables = table.getSupportedSysTables(); + + // WHY: a TVF-declared name must become the TVF-backed PartitionsSysTable (routed to the + // partition_values TVF in SysTableResolver), NOT the generic native PluginDrivenSysTable — else + // t$partitions would drive a native scan the hive connector has no BE reader for. MUTATION: + // wrapping every name as PluginDrivenSysTable (ignoring the kind) makes "partitions" native -> red. + SysTable partitions = sysTables.get("partitions"); + Assertions.assertTrue(partitions instanceof PartitionsSysTable, + "a partition_values-TVF-declared name must map to the TVF-backed PartitionsSysTable"); + Assertions.assertTrue(partitions instanceof TvfSysTable, "PartitionsSysTable is a TvfSysTable"); + Assertions.assertFalse(partitions.useNativeTablePath(), + "the partitions sys table must route through the TVF path, not the native scan path"); + // A name the connector did NOT declare TVF stays native. + Assertions.assertTrue(sysTables.get("snapshots") instanceof PluginDrivenSysTable, + "a name not declared TVF-backed stays a generic native PluginDrivenSysTable"); + // findSysTable resolves the TVF-backed entry by its exact suffix. + Optional hit = table.findSysTable("REMOTE_TBL$partitions"); + Assertions.assertTrue(hit.isPresent() && !hit.get().useNativeTablePath(), + "t$partitions must resolve to the TVF-backed sys table"); + } + @Test public void testGetSupportedSysTablesEmptyWhenNoBaseHandle() { ConnectorMetadata metadata = Mockito.mock(ConnectorMetadata.class); From 8c177e36c64e030c2a119d4e56a4e62d7674f97a Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 01:14:30 +0800 Subject: [PATCH 274/334] [doc](catalog) hive-e2e: land FIX-R3 ($partitions sys table) design, mark done Design + 3-lens red-team (GO_WITH_FIXES, 5 folds) for R3: expose hive t$partitions via the partition_values TVF through a connector-declared sys-table kind (isPartitionValuesSysTable SPI opt-in). Mark R3 done in the tracking table (e697f189c59) and correct the stale R2/R12/text_write entries to their landed commits. HANDOFF: R3 -> batch-3 done, remaining 5 suites (3 user-A fe-core/SPI changes + ENV). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 6 +- plan-doc/task-list-hive-e2e-r2-fixes.md | 8 +- .../FIX-R3-partitions-systable-design.md | 168 ++++++++++++++++++ 3 files changed, 175 insertions(+), 7 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-R3-partitions-systable-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 523b6e1b9ab7f5..ee4f5a9723023c 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -12,14 +12,14 @@ > **⚠ round-1 的 4 修有 2 个没接上/被证伪**：① binary 映射(5672d7c)读了 dot-key 但只写进 metadata **死字段**，真正转换用的是 client 的 DEFAULT options → 本轮已在 `HiveConnector.createClient` 修正（见 batch-1）；② `test_hive_varbinary_type` 不是 binary bug 而是 **ENV**(外部 HMS 残留致行翻倍，FE audit 证写入只一次)。 > > **✅ 本轮已修（batch-1+2：7 fix / 9 suite，均 test-compile + fe-connector-hive 298 UT 绿、0 checkstyle；见 git log `3be286517b5`/`70c1116dad7`/`e0e94eeb20f`/`db01b9fc07d`）**：R1 `getDatabase` LOCATION(修 test_hive_ddl + test_hms_event_notification[_multi_catalog]) · orc binary client options · R6 decimal 分区裁剪 · R11 特殊字符分区 key unescape · meta_cache ttl 校验 · **R5 cardinality explain**(fe-core PluginDrivenScanNode 补 FileScanNode 那行) · TEST_ALIGN test_hive_case_sensibility(truncate 块对齐已迁移的 drop 块文案)。 -> **✅ batch-3（R7/R10 前序 session；R12 + text_write(LZ4) + R2(SHOW CREATE) 本 session，共 5 fix DONE）**： +> **✅ batch-3（R7/R10/R12/text_write(LZ4)/R2(SHOW CREATE) 前序 session；R3($partitions) 本 session，共 6 fix DONE）**： > - **R7 `4df95ad44ac`**——SHOW PARTITIONS/partitions-TVF 走 fresh 列分区名(绕 CachingHmsClient 缓存)、查询裁剪+MTMV 保持缓存。`HmsClient` 加 default `listPartitionNamesFresh`(非缓存客户端=普通 list；缓存装饰器**须** override 直连 delegate、不读不写 partitionNamesCache) + `CachingHmsClient` override + `HiveConnectorMetadata.collectPartitionNames` 加 `bypassCache` 旗(listPartitionNames→true、listPartitions/getTableFreshness→false)。fe-core 零改。红队 4 lens 全 SOUND。`CachingHmsClientTest` 19/19。设计 `designs/FIX-R7-design.md`。**e2e 待用户自跑 test_hive_use_meta_cache_true sql09。** > - **R10 `9c70d4acf9a`**——OpenX JSON `ignore.malformed.json` 跳坏行。`HiveTextProperties.extractJsonSerDeProps` 串入 sd/table params，**仅 OpenX serde**发 `hive.text.openx_ignore_malformed`(table-param 优先、默认 false；hcatalog/hive2 从不带此旗，钉 OpenX 保 legacy 分支 parity) + fe-core `PluginDrivenScanNode.getFileAttributes` 在既有 `is_json` 块读键调 `setOpenxJsonIgnoreMalformed`(镜像 legacy HiveScanNode OPENX 分支)。one-column 模式 OpenX 解析成 CSV、BE CSV reader 不读该 JSON-only 字段故无碍。红队 3 lens(初版"全 3 JSON serde"被驳→已收敛 OpenX-only；test/iron-rule SOUND)。`HiveTextPropertiesTest` 13/13。设计 `designs/FIX-R10-design.md`。**e2e 待用户自跑 test_hive_openx_json q1。** > - **R12/serde `3936434bc9a`**——OpenCSV 表列全塌成 STRING(修 test_open_csv_serde + test_hive_serde_prop 的 OpenCSV 半)。根因=legacy 默认走 metastore `get_schema` RPC(服务端反序列化器→OpenCSV 全 string)，新连接器只读原始 `sd.getCols()` 声明类型→typed 列到 BE 翻车(TRUE vs 'true'、datetime→ISO、空串 vs NULL)。**方案定案=B(用户 2026-07-11 签字)**：不走 RPC、连接器就地复刻结果——`HiveConnectorMetadata.buildColumns` 加 `coerceOpenCsvColumnsToString`,serde==OpenCSV 时把**数据列**整型别塌成 `STRING`(复合类型也整列塌)、复用现成 `HiveTextProperties.HIVE_OPEN_CSV_SERDE` 常量。**架构红队关键订正**：放 hive 元数据层(非 hive/hudi 共用的 `ThriftHmsClient`,那会误触 hudi schema 路径 + 重复常量)——对齐 Trino(在 HiveMetadata 层做 CSV=全 string)。分区列不动(hive 在反序列化后追加、保声明类型)、view 有 isView 门。非-OpenCSV 表零改动(与原 `sd.getCols()` 逐位一致)。fe-core 零改、无 serde 名分支。3 路对抗红队(correctness SOUND / architecture 订正放置层 / blast 把验收扩到 4 suite)。`HiveConnectorMetadataSchemaTest` +4(int/datetime/bool/**array**→STRING、分区键留 date、LazySimple 门留 typed、view 门)、RED 实证(旁路→expected STRING but was INT)、306/306、0 checkstyle。设计 `designs/FIX-R12-design.md`。**残余(非本修)**：Avro schema-url 自述表 A/B 唯一分歧,新连接器**现状本就没有**、不在失败套件、以后需要时独立小任务补。**e2e 待用户自跑：test_open_csv_serde + test_hive_serde_prop(+对照 test_drop_expired_table_stats/test_trino_hive_serde_prop)。** > - **text_write(LZ4) `e1d48045bee`**——`.lz4` 文本表读炸 BE `LZ4F_getFrameInfo ERROR_frameType_unknown`。根因=`.lz4` 被 `Util.inferFileCompressTypeByPath` 推成 LZ4FRAME 发 BE，但 hadoop/hive 写的是 LZ4 **block** 编码；legacy `HiveScanNode.getFileCompressType` 有 LZ4FRAME→LZ4BLOCK remap，SPI 翻闸后丢了。修=连接器 opt-in（镜像 supportsTableSample/supportsBatchScan）：`ConnectorScanPlanProvider.adjustFileCompressType` 默认 identity → `PluginDrivenScanNode.getFileCompressType` 跑 super 推断后委派（resolveScanProvider+onPluginClassLoader，无 source 分支）→ `HiveScanPlanProvider` override 只 remap LZ4FRAME。**hudi 严格对齐**（红队定：legacy `HudiScanNode extends HiveScanNode` 继承了 remap，新独立 provider 须重声明，别赌"hudi 从不出 .lz4"）；其余 6 provider 留 identity。测试：api identity 1/1 + hive remap 5/5 + hudi parity 4/4（api+hive 对子=RED/GREEN），fe-core 节点 54/54 无回归，0 checkstyle。设计 `designs/FIX-text-write-LZ4-design.md`。**e2e 待用户自跑 test_hive_text_write_insert 的 lz4 迭代。** > - **R2(SHOW CREATE) `17764d03665`**——翻闸 hive 表 SHOW CREATE 出 Doris 通用风格(双引号 PROPERTIES),非 hive 原生 `ROW FORMAT SERDE`/`STORED AS`。**方案=Option X(用户 2026-07-12 签字)**:惰性连接器方法 + **命令期实时取 HMS** + 连接器侧渲染,只在 `ShowCreateTableCommand` 拦截、**不声明 capability**。**两版旧方案被红队证伪**:(1)eager schema-cache 渲染过不了 `test_hive_meta_cache`(desc 缓存 5 列、show create 须新取 6 列);(2)声明 capability 会翻转共享 `Env.getDdlStmt` 门→误触 delegated hudi/iceberg-on-HMS + HTTP 端点。**7 文件**:`ConnectorTableOps.renderShowCreateTableDdl` 默认空(iceberg/paimon/es/jdbc 继承→落 Env.getDdlStmt 不变) · `HmsClient.getTableFresh`+`CachingHmsClient` override(绕 tableCache,镜像 listPartitionNamesFresh) · `HiveShowCreateTableRenderer`(fe-connector-hms,逐字节镜像 legacy `HiveMetaStoreClientHelper.showCreateTable`:两套引号约定 SERDEPROPERTIES `'k' = 'v'`/TBLPROPERTIES `'k'='v'`、2空格/1空格缩进、comment 提到 COMMENT 子句、null-comment 门) · `HiveConnectorMetadata` override(非 HiveTableHandle=delegated→空,镜像 getTableSchema 门;fresh getTableFresh) · `PluginDrivenExternalTable.getShowCreateTableDdl`(镜像 getViewText、读锁安全、handle 未解析→空) · `ShowCreateTableCommand` view 臂后新臂(门=方法返回 present 非 source 名,iceberg/paimon/view 不受影响)。**两红队 lens 均 SOUND_WITH_FIXES→已折入 2 守卫(foreign-handle 门 + 未解析→空)**。fe-core 连接器无关、无 source 分支;TCCL 在 `ThriftHmsClient.doAs` 内钉、renderer reflection-free。测试:`HiveShowCreateTableRendererTest` 5(byte-parity,两套引号+null-comment 门) + `CachingHmsClientTest` +3(getTableFresh 双向绕缓存)、fe-core PluginDrivenExternalTable 36/36 + EnvShowCreate 3/3 不变、hive 307/307、0 checkstyle。设计 `designs/FIX-R2-showcreate-design.md`。**残余**:toHiveTypeString 对 `varchar(n)`→`string`(HMS 本就存 string 故无碍)、不可映射类型抛 IAE(legacy 回显原串、无套件触发);TBLPROPERTIES 逐字发含 volatile 键(同 legacy,勿加 .out golden)。**e2e 待用户自跑:test_hive_show_create_table + test_hive_ddl_text_format(+ test_hive_meta_cache 新取契约 / test_multi_delimit_serde / test_hive_ddl)。** -> **⭐ 剩余 6 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **fe-connector/fe-core 大型**：R3 `$partitions` 系统表(HiveConnectorMetadata:1302/1313 暴露 + sys-table scan/column plumbing)。 +> - **R3($partitions 系统表) `e697f189c59`**——翻闸 hive 表查 `表$partitions` 报 `"Unknown sys table"`（`desc` 报 `"sys table not found"`）。根因=翻闸 hive=`PluginDrivenExternalTable`，其 `getSupportedSysTables()` 把连接器上报的每个系统表名**无条件包成原生 `PluginDrivenSysTable`**，而 `HiveConnectorMetadata.listSupportedSysTables` 对 hive handle 返回空；旧 `HMSExternalTable` 对 HIVE 返回**走 TVF** 的 `PartitionsSysTable`（`partition_values` TVF 及其 BE→FE 元数据回取**早已全 plugin-aware**，只丢了 `$partitions`→系统表→TVF 这座桥）。**关键陷阱**：fe-core 不能按名字 `"partitions"` 硬映射到 TVF——**iceberg(`MetadataTableType.PARTITIONS`)+paimon(`SystemTableLoader`) 也上报 `"partitions"` 但是原生元数据表**，硬映射会把它俩打断；种类须**连接器声明**（不能从 `getSysTableHandle` 空来推断：种类在 discovery 期就要、`getSysTableHandle` eager+auth、iceberg 合法省略 position_deletes）。修=**连接器声明 kind 的 `supports*`-式 opt-in**（铁律干净）：SPI `ConnectorTableOps.isPartitionValuesSysTable` 默认 false→原生 · fe-core `getSupportedSysTables()` 据此分流(TVF→`PartitionsSysTable.INSTANCE`,键钉 INSTANCE 自身名避免 createFunction 切后缀崩;否则原生) · `HiveConnectorMetadata` 对 hive handle `listSupportedSysTables` 返回 `["partitions"]`(**无条件**,含非分区表——`$partitions` 打非分区表须到 TVF ctor 抛 `"is not a partitioned table"` 而非 `"Unknown sys table"`,镜像旧行为)、`isPartitionValuesSysTable("partitions")=true`(外来 handle 委派兄弟→iceberg/hudi-on-HMS 保原生)、`getSysTableHandle` 仍空(TVF 路不查)。无 BE/Nereids 改。3 路对抗红队→GO_WITH_FIXES(5 折全折入:F1 键钉 INSTANCE 名/F2 外来 handle 守卫 verbatim/F3 iceberg-on-HMS+原生 iceberg/paimon t$partitions e2e 提为必测/F4 行号/F5 驳 getSysTableHandle-空推断)。测试:新 `HiveConnectorMetadataSysTableTest` 5 + `HiveConnectorMetadataSiblingDelegationTest`(委派+hive 暴露 partitions,`EXPECTED_METHODS`+`RecordingSiblingMetadata` 加 `isPartitionValuesSysTable`) + fe-core `PluginDrivenSysTableTest` +1(TVF-kind wrap→`PartitionsSysTable`/非原生);**fe-connector-hive 312/312 + fe-core PluginDrivenSysTable 11/11、0 checkstyle**。设计 `designs/FIX-R3-partitions-systable-design.md`。**e2e 待用户自跑（必测 GREEN 门）：test_hive_partition_values_tvf（sql01-113 全部）+ iceberg-on-HMS/原生 iceberg/paimon `t$partitions` 仍返回原生行（F2 外来-handle 守卫的唯一证明，单测因 iceberg divert 休眠证不了）。** +> **⭐ 剩余 5 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： > - **用户已裁决(A)的 3 个 fe-core/SPI 大改**： > - query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 > - default_partition 连接器传 null 标记：SPI 新字段让连接器按分区值传 null 语义，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem:310`(现无条件 `PartitionValue(v,false)`) 据此建项——hive 标 null、paimon 保持非空(零 paimon 影响，别在 fe-core if(hive))。 diff --git a/plan-doc/task-list-hive-e2e-r2-fixes.md b/plan-doc/task-list-hive-e2e-r2-fixes.md index 013211a5f7caf1..be026116b86f32 100644 --- a/plan-doc/task-list-hive-e2e-r2-fixes.md +++ b/plan-doc/task-list-hive-e2e-r2-fixes.md @@ -16,12 +16,12 @@ Discipline per task: 设计(`tasks/designs/FIX--design.md`) → 设计红队 ### fe-connector 中型 - [x] R7 SHOW PARTITIONS bypass cache (test_hive_use_meta_cache_true) `4df95ad44ac` — design red-teamed 4/4 SOUND, UT 19/19 - [x] R10 openx json ignore.malformed (test_hive_openx_json) `9c70d4acf9a` — openx-only gate, red-teamed 3-lens, UT 13/13 -- [ ] R12/serde OpenCSV all-STRING schema (test_open_csv_serde, test_hive_serde_prop) -- [ ] text_write LZ4FRAME→LZ4BLOCK read (test_hive_text_write_insert) +- [x] R12/serde OpenCSV all-STRING schema (test_open_csv_serde, test_hive_serde_prop) `3936434bc9a` — connector-side coerce, red-teamed 3-lens, UT 306/306 +- [x] text_write LZ4FRAME→LZ4BLOCK read (test_hive_text_write_insert) `e1d48045bee` — connector opt-in adjustFileCompressType, hive+hudi parity ### fe-connector / fe-core 大型 -- [ ] R2 SHOW CREATE TABLE native DDL (test_hive_show_create_table, test_hive_ddl_text_format) -- [ ] R3 $partitions sys table (test_hive_partition_values_tvf) +- [x] R2 SHOW CREATE TABLE native DDL (test_hive_show_create_table, test_hive_ddl_text_format) `17764d03665` — lazy fresh-fetch connector render, red-teamed +- [x] R3 $partitions sys table (test_hive_partition_values_tvf) `e697f189c59` — SPI isPartitionValuesSysTable opt-in → PartitionsSysTable(TVF); red-teamed GO_WITH_FIXES, UT hive 312/312 + fe-core PluginDrivenSysTable 11/11 ### fe-core / SPI 大改 (user signed off → option A) - [ ] query_cache: port SQL result cache to SPI + connector stable invalidation token (test_hive_query_cache) diff --git a/plan-doc/tasks/designs/FIX-R3-partitions-systable-design.md b/plan-doc/tasks/designs/FIX-R3-partitions-systable-design.md new file mode 100644 index 00000000000000..13d7ac221182ab --- /dev/null +++ b/plan-doc/tasks/designs/FIX-R3-partitions-systable-design.md @@ -0,0 +1,168 @@ +# FIX-R3 — `table$partitions` system table for flipped hive (`partition_values` TVF) + +**Failing suite**: `external_table_p0/hive/test_hive_partition_values_tvf` +**Scope**: fe-connector-api (SPI, +1 default method) · fe-connector-hive (`HiveConnectorMetadata`) · fe-core (`PluginDrivenExternalTable.getSupportedSysTables`). **No BE change, no Nereids change.** + +## Root cause (HEAD-verified, recon `wf_05998eee-e1d`) + +A flipped plain-hive table is a `PluginDrivenExternalTable`. Its `getSupportedSysTables()` +(`PluginDrivenExternalTable.java:948-972`) discovers sys-table names from the connector +(`metadata.listSupportedSysTables`) and **wraps every name in a `PluginDrivenSysTable`** — a +`NativeSysTable` (native connector scan path). But `HiveConnectorMetadata.listSupportedSysTables` +(`HiveConnectorMetadata.java:1348-1354`) returns **`emptyList()`** for a real `HiveTableHandle` +("Hive exposes no system tables"). So: + +- `getSupportedSysTables()` → empty map → `findSysTable("t$partitions")` → `Optional.empty`. +- `select … t$partitions` → `RelationUtil.validateForQuery` false → **`"Unknown sys table 't$partitions'"`**. +- `desc t$partitions` → `DescribeCommand.resolveForDescribe` empty → **`"sys table not found: partitions"`**. + +Legacy `HMSExternalTable.getSupportedSysTables()` (`HMSExternalTable.java:1239-1251`) returned, for +`dlaType==HIVE`, `PartitionsSysTable.HIVE_SUPPORTED_SYS_TABLES = {"partitions": PartitionsSysTable.INSTANCE}` +— a **`TvfSysTable`** that routes `t$partitions` to the fe-core `partition_values` TVF +(`PartitionsSysTable.java:38,52,59`). That TVF and its BE→FE metadata fetch are **already fully +plugin-aware** (`PartitionValuesTableValuedFunction.analyzeAndGetTable` accepts +`PluginDrivenExternalTable` at :144-148; `MetadataGenerator.partitionValuesMetadataResultForPluginTable` +at :2131). The **only** gap is the `$partitions` suffix → sys-table → TVF *bridge*. + +### Why fe-core cannot decide native-vs-TVF by name + +The bare name `"partitions"` is **overloaded**: + +| Source | `partitions` sys table | Served by | +|---|---|---| +| hive | TVF | fe-core `partition_values` TVF (`PartitionsSysTable`, a `TvfSysTable`) | +| iceberg (`MetadataTableType.PARTITIONS`) | native | `IcebergSysTableJniScanner` (native metadata scan) | +| paimon (`SystemTableLoader.SYSTEM_TABLES`) | native | native metadata scan | + +Iceberg **and** paimon both report `"partitions"` (recon thread A, javap-confirmed on the SDK jars). +So a fe-core `if (name.equals("partitions")) → TVF` mapping would **misroute iceberg/paimon +`t$partitions` to the hive TVF** and break them. The native-vs-TVF decision is inherently +**connector-specific** and must be declared by the connector. + +Nor can fe-core **infer** the kind from `getSysTableHandle` returning empty: the kind is needed at +*discovery* time in `getSupportedSysTables` (before any handle is fetched), `getSysTableHandle` is +eager and auth-bearing (paimon loads the SDK `Table`; calling it per name at discovery would force an +auth/metadata round-trip), and a native connector may legitimately omit a handle for some names +(iceberg omits `position_deletes`) — so empty ≠ TVF. The kind must be an explicit connector signal. + +The connector cannot build the fe-core `TvfSysTable` itself — `createFunction`/`createFunctionRef` +return Nereids/fe-core types (`TableValuedFunction`, `TableValuedFunctionRefInfo`) that connectors +must not import. So the connector can only **declare the kind**; fe-core maps kind→concrete SysTable. + +## Design — connector-declared kind, `supports*`-style opt-in + +Mirrors the established connector opt-in convention in this migration (`supportsTableSample()`, +`supportsBatchScan()`, `adjustFileCompressType()` — all default-off hooks a connector overrides). +Iron-rule clean: fe-core keys on a **generic capability**, never a source name / `instanceof HMSExternal*`. + +### 1. SPI (`ConnectorTableOps`, fe-connector-api) — +1 default method + +```java +/** + * Whether the named system table of {@code baseTableHandle} is served by the generic + * {@code partition_values} table-valued function (fe-core's {@code PartitionsSysTable}), rather + * than a native connector scan. Default {@code false} (native, the {@code getSysTableHandle} path). + * + *

    A connector whose partitioned tables expose their partition rows through the generic + * partition-values TVF (hive) overrides this to return {@code true} for that sys-table name; the + * connector need NOT return a handle from {@link #getSysTableHandle} for such a name (the TVF path + * does not consult it). {@code sysName} is the bare name (no {@code "$"}).

    + */ +default boolean isPartitionValuesSysTable(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + return false; +} +``` + +### 2. fe-core `PluginDrivenExternalTable.getSupportedSysTables()` — kind-aware wrap + +The single wrap loop (`:968-970`) branches on the connector-declared kind: + +```java +for (String sysName : names) { + if (metadata.isPartitionValuesSysTable(session, handleOpt.get(), sysName)) { + // Served by the generic partition_values TVF (fe-core PartitionsSysTable), not a native scan. + // Key on the singleton's OWN name (== "partitions"): PartitionsSysTable strips its hard-wired + // "$partitions" suffix in createFunction, so a map key that ever differed would crash + // (StringIndexOutOfBounds). Same value as sysName for hive today; strictly safer. (F1) + result.put(PartitionsSysTable.INSTANCE.getSysTableName(), PartitionsSysTable.INSTANCE); + } else { + result.put(sysName, new PluginDrivenSysTable(sysName)); + } +} +``` + +`PartitionsSysTable.INSTANCE` is a stateless singleton keyed `"partitions"` (its `getSysTableName()`), +matching the connector-returned key. `SysTableResolver` already routes any `TvfSysTable` to the TVF +path (`:58-62`) for SELECT / DESCRIBE / validate / SHOW-CREATE — **no new call site**. + +### 3. fe-connector-hive `HiveConnectorMetadata` + +- `listSupportedSysTables` hive branch: `emptyList()` → **`List.of("partitions")`**. + **Unconditional** (every hive handle, partitioned or not) — matches legacy + (`HMSExternalTable` returned the map for all `dlaType==HIVE`) and is **required** by the test: + `select/desc non_partitioned_tbl$partitions` must reach the TVF ctor and throw + `"… is not a partitioned table"` (`PartitionValuesTableValuedFunction:140/146`), not + `"Unknown sys table"`. Gating on partitioned-only would surface the wrong error. +- New `isPartitionValuesSysTable` override — the foreign-handle guard is **load-bearing** (F2): a + naive `return "partitions".equals(sysName)` without it would flip iceberg-on-HMS `t$partitions` to + the hive TVF. Verbatim (mirrors `listSupportedSysTables:1349-1350` / `getSysTableHandle:1359-1361`): + ```java + @Override + public boolean isPartitionValuesSysTable(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + if (!(baseTableHandle instanceof HiveTableHandle)) { + return siblingMetadata(session, baseTableHandle) + .isPartitionValuesSysTable(session, baseTableHandle, sysName); + } + return "partitions".equals(sysName); + } + ``` +- `getSysTableHandle`: **unchanged** (stays `emptyList`/`empty` for hive; TVF path never consults it). + +## Blast radius / non-regression + +- **iceberg / paimon native catalogs**: don't override `isPartitionValuesSysTable` → default false → + `t$partitions` stays native. Unchanged. +- **iceberg-on-HMS / hudi-on-HMS (sibling delegation)**: foreign handle → hive delegates + `listSupportedSysTables` + `isPartitionValuesSysTable` to the sibling → sibling returns its native + names + false → fe-core wraps native. `t$partitions` stays native. Unchanged. +- **Internal OLAP tables** (`pv_inner1$partitions`, test §12): default empty `getSupportedSysTables` + → `"sys table not found"` / `"Unknown sys table"`. Untouched. +- **Direct `partition_values(...)` TVF** (test sql92/94/95): bypasses sys-table resolution entirely. + Already worked; untouched. +- `PluginDrivenMvccExternalTable` (the subclass flipped hive actually instantiates) inherits + `getSupportedSysTables` from the base — one fix covers both. + +## Tests (TDD: RED → GREEN) + +**fe-connector-api** — no test (trivial default; exercised via hive + fe-core). + +**fe-connector-hive** (recording-fake, NO mockito): +- New `HiveConnectorMetadataSysTableTest`: hive handle → + `listSupportedSysTables == ["partitions"]`; `isPartitionValuesSysTable(_, hive, "partitions") == true`, + `== false` for `"snapshots"`/null; `getSysTableHandle(_, hive, "partitions")` empty. +- `HiveConnectorMetadataSiblingDelegationTest` (update, lockstep): + - foreign-handle test: add `md.isPartitionValuesSysTable(null, foreignHandle, "partitions")` call; + add `"isPartitionValuesSysTable"` to `EXPECTED_METHODS` (:462-469); add the recording override to + `RecordingSiblingMetadata` (:630-638); assert the sibling's answer flows through. + - hive-handle test (:192): flip `listSupportedSysTables(_, hive).isEmpty()` → + `== ["partitions"]`; add `isPartitionValuesSysTable(_, hive, "partitions") == true`. + +**fe-core** (mockito): +- `PluginDrivenSysTableTest`: add a case — `metadata.isPartitionValuesSysTable(session, baseHandle, "partitions")` + stubbed true → `getSupportedSysTables().get("partitions")` is a `TvfSysTable` (`PartitionsSysTable`), + and a native name (`"snapshots"`, stub false) stays a `PluginDrivenSysTable`. Assert `findSysTable` + routes the TVF one through `SysTableResolver.resolveForPlan` → `forTvf`. + +## Residual / e2e + +- **e2e (user-run), required GREEN gate (F3)** — not optional residual: + - `test_hive_partition_values_tvf` (all of sql01-113: `$partitions` suffix, desc, agg, view, join, + non-partitioned error, all-types). + - **iceberg-on-HMS + native iceberg/paimon `t$partitions` still return NATIVE rows** — this is the + only proof of the F2 foreign-handle guard: the iceberg divert (`HiveConnectorMetadata:353`) is + DORMANT at HEAD, so the recording-fake unit tests cannot exercise it. memory + `hms-iceberg-delegation-needs-e2e`. +- `@Disabled HmsQueryCacheTest` references `PartitionsSysTable.HIVE_SUPPORTED_SYS_TABLES` (compiles, + not run) — unaffected. From 2bc7ad9bc7e350113bb1cacf817192807660fecf Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 06:32:45 +0800 Subject: [PATCH 275/334] [fix](catalog) hive: honor hive.recursive_directories in partition file listing After the hms flip, a plugin-driven hive catalog lists partition data files through HiveFileListingCache.listFromFileSystem, which was unconditionally non-recursive and never read the catalog property hive.recursive_directories. Legacy fe-core (HiveExternalMetaCache.getFileCache) defaulted it to "true" and recursed into sub-directories, so a table whose data lives in sub-directories now silently loses those rows (returns top-level only) instead of all of them. Fix (connector-local, zero fe-core / SPI change): parse the flag once in the cache ctor (default true) and bake it into the production lister; when set, listFromFileSystem descends into non-hidden sub-directories. Hidden dirs (_temporary / .hive-staging) and _/.-prefixed files are skipped at every level -- exact net parity with legacy's full-path containsHiddenPath filter, since the connector filters only the leaf FileEntry.name(), so descend-all would surface staging files legacy suppresses. The failure loop is factored into a recursive collectFiles helper that throws up to the SAME systemic-vs-local classifier, so a sub-directory failure gets the same verdict as the top. All three shared-cache consumers (scan split, size estimate, ANALYZE...WITH SAMPLE stats) recurse consistently by construction (single injected instance). recursive=false stays byte-identical to today. Tests: HiveFileListingCacheTest +6 (recursive descent / non-recursive top-level-only / hidden-dir+file skip / default-true / property=false / nested-descent failure is skippable); FakeFileSystem gains tree-mode + per-location list error. fe-connector-hive 318/318, 0 checkstyle. Design: plan-doc/tasks/designs/FIX-recursive-directories-design.md Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveFileListingCache.java | 94 +++++++++++---- .../doris/connector/hive/FakeFileSystem.java | 28 +++++ .../hive/HiveFileListingCacheTest.java | 112 ++++++++++++++++++ 3 files changed, 211 insertions(+), 23 deletions(-) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java index 9050981165a65c..0a1a0f97041221 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -85,6 +85,13 @@ public class HiveFileListingCache { static final long DEFAULT_TTL_SECOND = 86400L; static final long DEFAULT_FILE_CAPACITY = 10000L; + /** + * Catalog property controlling whether partition directories are listed recursively (descend into + * sub-directories). Default {@code true} — legacy {@code HiveExternalMetaCache.getFileCache} defaulted the + * same. When {@code false}, a table whose data lives in sub-directories silently loses those rows. + */ + static final String RECURSIVE_DIRECTORIES_PROPERTY = "hive.recursive_directories"; + /** * The raw directory lister: the engine-injected Doris {@link FileSystem} in production * ({@link #listFromFileSystem}), a fake in unit tests. Injected so the cache's hit/miss/invalidation behaviour @@ -101,7 +108,20 @@ interface DirectoryLister { private final DirectoryLister lister; public HiveFileListingCache(Map properties) { - this(properties, HiveFileListingCache::listFromFileSystem); + this(properties, defaultLister(properties)); + } + + /** + * The production {@link DirectoryLister}: {@link #listFromFileSystem} with the catalog's + * {@code hive.recursive_directories} flag (default {@code true}) baked in. The flag is a per-catalog + * constant, so capturing it here makes every consumer of the shared cache (scan, size estimate, stats + * sampling) recurse consistently without a hot-path signature change. + */ + private static DirectoryLister defaultLister(Map properties) { + Map props = properties == null ? Collections.emptyMap() : properties; + boolean recursive = Boolean.parseBoolean( + props.getOrDefault(RECURSIVE_DIRECTORIES_PROPERTY, "true")); + return (location, fs) -> listFromFileSystem(location, fs, recursive); } HiveFileListingCache(Map properties, DirectoryLister lister) { @@ -115,7 +135,8 @@ public HiveFileListingCache(Map properties) { } /** - * Lists the data files directly under {@code location} (non-recursive; directories and {@code _}/{@code .} + * Lists the data files under {@code location} (recursively into non-hidden sub-directories when the catalog's + * {@code hive.recursive_directories} is set, default {@code true}; directories and {@code _}/{@code .} * -prefixed hidden files removed), served from the cache. Keyed by {@code (db, table, location)} so * {@link #invalidateTable} can drop exactly one table's entries. The loader runs on the calling thread; an I/O * failure throws {@link DorisConnectorException} (and is NOT cached). The returned list is shared by reference @@ -149,12 +170,13 @@ long size() { } /** - * The production {@link DirectoryLister}: a non-recursive, LITERAL listing through the engine-injected Doris + * The production {@link DirectoryLister}: a LITERAL listing through the engine-injected Doris * {@link FileSystem} (a per-catalog {@code SpiSwitchingFileSystem}), filtering out directories and * {@code _}/{@code .}-prefixed hidden files (byte-parity with the pre-cache filters in - * {@code HiveScanPlanProvider.listAndSplitFiles} and {@code HiveConnectorMetadata.sumCachedFileSizes}). A zero - * -length data file is kept (the scan splitter skips it; the size estimate adds 0) so both consumers keep their - * exact prior behaviour. + * {@code HiveScanPlanProvider.listAndSplitFiles} and {@code HiveConnectorMetadata.sumCachedFileSizes}). When + * {@code recursive} (from {@code hive.recursive_directories}, default {@code true}) it descends into non-hidden + * sub-directories (see {@link #collectFiles}). A zero-length data file is kept (the scan splitter skips it; the + * size estimate adds 0) so both consumers keep their exact prior behaviour. * *

    Two-boundary failure split (byte-parity with the pre-cache {@code FileSystem.get}/{@code listStatus} * split): {@link FileSystem#forLocation} does the scheme/storage resolution + concrete-FS construction (no @@ -172,6 +194,10 @@ long size() { * {@code listStatus} never glob-expanded, and a hive location can legitimately contain those characters. */ static List listFromFileSystem(String location, FileSystem fs) { + return listFromFileSystem(location, fs, false); + } + + static List listFromFileSystem(String location, FileSystem fs, boolean recursive) { if (fs == null) { // No engine filesystem for this catalog (empty storage): a SYSTEMIC config error affecting every // partition. Fail loud (the scan path does NOT skip this), never a silent empty scan. @@ -190,20 +216,7 @@ static List listFromFileSystem(String location, FileSystem fs) { } try { List files = new ArrayList<>(); - try (FileIterator it = resolved.list(loc)) { - while (it.hasNext()) { - FileEntry entry = it.next(); - if (entry.isDirectory()) { - continue; - } - String fileName = entry.name(); - if (fileName.startsWith("_") || fileName.startsWith(".")) { - continue; - } - files.add(new HiveFileStatus(entry.location().uri(), entry.length(), - entry.modificationTime())); - } - } + collectFiles(resolved, loc, recursive, files); return files; } catch (IOException e) { if (isSystemicResolutionFailure(e)) { @@ -212,13 +225,48 @@ static List listFromFileSystem(String location, FileSystem fs) { // FileSystem.get behavior (this is the exact error class FIX-HIVEFS exists to keep loud). throw new DorisConnectorException("Failed to resolve filesystem for " + location, e); } - // Listing THIS partition directory failed (missing / unreadable / transient): a LOCAL failure the scan - // path tolerates by skipping the partition with a warning (pre-cache parity). Distinct exception type so - // only this is skipped, while the systemic failures above stay loud. + // Listing THIS partition directory (or one of its sub-directories) failed (missing / unreadable / + // transient): a LOCAL failure the scan path tolerates by skipping the partition with a warning + // (pre-cache parity). Distinct exception type so only this is skipped, while systemic failures stay loud. throw new HiveDirectoryListingException("Failed to list files under " + location, e); } } + /** + * Collects the visible data files under {@code dir} into {@code out}, filtering directories and + * {@code _}/{@code .}-prefixed hidden files. When {@code recursive}, descends into every NON-hidden + * sub-directory; a hidden sub-directory ({@code _temporary} / {@code .hive-staging}) is skipped — exact net + * parity with legacy {@code HiveExternalMetaCache}'s full-path {@code containsHiddenPath} filter (the connector + * filters only the leaf {@link FileEntry#name()}, so descending into a hidden dir would surface staging files + * legacy suppresses). A listing failure at any level throws {@link IOException} up to the single classifier in + * {@link #listFromFileSystem}, so a sub-directory failure gets the same systemic/local verdict as the top. + * Descent reuses the already-resolved {@code resolved} (a sub-directory shares scheme/authority). + */ + private static void collectFiles(FileSystem resolved, Location dir, boolean recursive, + List out) throws IOException { + try (FileIterator it = resolved.list(dir)) { + while (it.hasNext()) { + FileEntry entry = it.next(); + String name = entry.name(); + if (entry.isDirectory()) { + if (recursive && !isHidden(name)) { + collectFiles(resolved, entry.location(), true, out); + } + continue; + } + if (isHidden(name)) { + continue; + } + out.add(new HiveFileStatus(entry.location().uri(), entry.length(), + entry.modificationTime())); + } + } + } + + private static boolean isHidden(String name) { + return name.startsWith("_") || name.startsWith("."); + } + /** * Whether {@code t} (or any exception in its cause chain — robust to {@code authenticator.doAs} wrapping) * is a scheme-not-registered failure ({@link UnsupportedFileSystemException} / {@code "No FileSystem for diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java index d514f028e59ef2..27d2ee375588ee 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/FakeFileSystem.java @@ -27,8 +27,10 @@ import java.io.IOException; import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; import java.util.Iterator; import java.util.List; +import java.util.Map; /** * A recording {@link FileSystem} test double for the hive connector's file-listing tests (the module has no @@ -44,14 +46,33 @@ final class FakeFileSystem implements FileSystem { private List entries = Collections.emptyList(); + // Tree mode: a per-location listing, keyed by Location.uri(). Populated => list(loc) returns tree.get(uri) + // (empty if absent); empty (the default) => list() falls back to the flat `entries`, so the existing flat-fake + // tests are untouched. Needed to model recursive descent, where each sub-directory must list its OWN entries. + private Map> tree = Collections.emptyMap(); private IOException forLocationError; private IOException listError; + // Per-location list() failures (uri -> error): models "top-level dir lists fine, one sub-directory fails", + // which the single global listError cannot (it fails every list()). + private final Map listErrorByLocation = new HashMap<>(); FakeFileSystem withEntries(FileEntry... e) { this.entries = Arrays.asList(e); return this; } + /** Tree mode: {@code list(loc)} returns the entries mapped to {@code loc.uri()} (empty if unmapped). */ + FakeFileSystem withTree(Map> t) { + this.tree = t; + return this; + } + + /** Makes {@link #list} throw only for {@code location} (a single failing directory among healthy ones). */ + FakeFileSystem failListAt(String location, IOException e) { + this.listErrorByLocation.put(location, e); + return this; + } + /** Makes {@link #forLocation} throw — the SYSTEMIC (scheme/storage resolution) boundary. */ FakeFileSystem failForLocation(IOException e) { this.forLocationError = e; @@ -83,9 +104,16 @@ public FileSystem forLocation(Location location) throws IOException { @Override public FileIterator list(Location location) throws IOException { + IOException perLocation = listErrorByLocation.get(location.uri()); + if (perLocation != null) { + throw perLocation; + } if (listError != null) { throw listError; } + if (!tree.isEmpty()) { + return new ListFileIterator(tree.getOrDefault(location.uri(), Collections.emptyList()).iterator()); + } return new ListFileIterator(entries.iterator()); } diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java index ff16de847438f7..8d18c2b36664ed 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.hms.HmsPartitionInfo; +import org.apache.doris.filesystem.FileEntry; import org.apache.doris.filesystem.FileSystem; import org.apache.hadoop.fs.UnsupportedFileSystemException; @@ -233,6 +234,117 @@ public void listFromFileSystemListsLiterallyForGlobCharLocation() { Assertions.assertEquals(dir + "/000000_0", files.get(0).getPath()); } + // ==================== recursive listing (hive.recursive_directories, default true) ==================== + + /** A three-level tree: top-level exp_a plus sub-directories 1/ (exp_b) and 2/ (exp_c). */ + private static Map> recursiveTree(String top) { + Map> tree = new HashMap<>(); + tree.put(top, Arrays.asList( + FakeFileSystem.file(top + "/exp_a", 1L, 1L), + FakeFileSystem.dir(top + "/1"), + FakeFileSystem.dir(top + "/2"))); + tree.put(top + "/1", Collections.singletonList(FakeFileSystem.file(top + "/1/exp_b", 1L, 1L))); + tree.put(top + "/2", Collections.singletonList(FakeFileSystem.file(top + "/2/exp_c", 1L, 1L))); + return tree; + } + + @Test + public void recursiveDescendsIntoSubdirectories() { + // WHY (Rule 9): a table whose data lives in sub-directories (top + 1/ + 2/) must contribute ALL its files + // when recursion is on — else those rows are silently lost (the regression this restores). Mirrors + // hive_config_test's hive_recursive_directories_table (tags 2/21 = 6 rows). + String top = "file:///wh/db/t/dt=1"; + FakeFileSystem fs = new FakeFileSystem().withTree(recursiveTree(top)); + + List files = HiveFileListingCache.listFromFileSystem(top, fs, true); + + Assertions.assertEquals(3, files.size(), "recursive listing must include files from every sub-directory"); + } + + @Test + public void nonRecursiveListsTopLevelOnly() { + // WHY (Rule 9): with recursion off, only top-level files are returned — sub-directories are NOT descended + // (byte-identical to today; pins hive_config_test tag 1 = 2 rows). + String top = "file:///wh/db/t/dt=1"; + FakeFileSystem fs = new FakeFileSystem().withTree(recursiveTree(top)); + + List files = HiveFileListingCache.listFromFileSystem(top, fs, false); + + Assertions.assertEquals(1, files.size(), "non-recursive listing must not descend into sub-directories"); + Assertions.assertTrue(files.get(0).getPath().endsWith("/exp_a"), "only the top-level file survives"); + } + + @Test + public void recursiveSkipsHiddenSubdirectoriesAndFiles() { + // WHY (Rule 9): recursion must skip hidden sub-directories (_temporary / .hive-staging write-staging) and + // hidden files at every level — exact net parity with legacy's full-path containsHiddenPath filter. A + // descend-all-then-leaf-filter regression would surface _temporary/part-0 staging files. + String top = "file:///wh/db/t/dt=1"; + Map> tree = new HashMap<>(); + tree.put(top, Arrays.asList( + FakeFileSystem.file(top + "/exp_a", 1L, 1L), + FakeFileSystem.file(top + "/.hidden", 1L, 1L), // hidden file — skipped + FakeFileSystem.dir(top + "/_temporary"), // hidden dir — NOT descended + FakeFileSystem.dir(top + "/1"))); // real sub-dir — descended + tree.put(top + "/_temporary", + Collections.singletonList(FakeFileSystem.file(top + "/_temporary/part-0", 1L, 1L))); + tree.put(top + "/1", Collections.singletonList(FakeFileSystem.file(top + "/1/exp_b", 1L, 1L))); + FakeFileSystem fs = new FakeFileSystem().withTree(tree); + + List files = HiveFileListingCache.listFromFileSystem(top, fs, true); + + Assertions.assertEquals(2, files.size(), "only real data files survive; hidden dir/file are excluded"); + for (HiveFileStatus f : files) { + Assertions.assertFalse(f.getPath().contains("_temporary"), + "must not read a hidden staging dir: " + f.getPath()); + Assertions.assertFalse(f.getPath().endsWith("/.hidden"), "must not read a hidden file"); + } + } + + @Test + public void defaultIsRecursive() { + // WHY (Rule 9): with NO hive.recursive_directories property the catalog defaults to recursive (legacy + // default "true"); pins tag 21. Drives the REAL production lister through listDataFiles — the + // RED-against-literal-HEAD guarantee (today's non-recursive lister returns 1, not 3). + String top = "file:///wh/db/t/dt=1"; + FakeFileSystem fs = new FakeFileSystem().withTree(recursiveTree(top)); + HiveFileListingCache cache = new HiveFileListingCache(Collections.emptyMap()); + + List files = cache.listDataFiles("db", "t", top, fs); + + Assertions.assertEquals(3, files.size(), "default (no property) must be recursive"); + } + + @Test + public void recursiveFlagFalseFromProperty() { + // WHY (Rule 9): hive.recursive_directories=false is honoured through the public ctor / listDataFiles path — + // pins that the tag-1 vs tag-2 divergence is driven by the property, not hardcoded. + String top = "file:///wh/db/t/dt=1"; + FakeFileSystem fs = new FakeFileSystem().withTree(recursiveTree(top)); + HiveFileListingCache cache = new HiveFileListingCache(props("hive.recursive_directories", "false")); + + List files = cache.listDataFiles("db", "t", top, fs); + + Assertions.assertEquals(1, files.size(), "hive.recursive_directories=false must list top-level only"); + } + + @Test + public void recursiveSubdirListingFailureIsSkippable() { + // WHY (Rule 9): a listing failure in a DESCENDED sub-directory must reach the SAME classifier as a + // top-level failure — a local FileNotFound stays the skippable HiveDirectoryListingException (so the scan + // skips just this partition). Guards a future swallowing/reclassifying catch around the recursion. + String top = "file:///wh/db/t/dt=1"; + Map> tree = new HashMap<>(); + tree.put(top, Arrays.asList( + FakeFileSystem.file(top + "/exp_a", 1L, 1L), + FakeFileSystem.dir(top + "/1"))); + FakeFileSystem fs = new FakeFileSystem().withTree(tree) + .failListAt(top + "/1", new FileNotFoundException("Path does not exist")); + + Assertions.assertThrows(HiveDirectoryListingException.class, + () -> HiveFileListingCache.listFromFileSystem(top, fs, true)); + } + // ==================== failure split: systemic is loud, local is skippable ==================== @Test From e2fdd65b954c7042126fc8bce1971082ad6cc90f Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 06:33:52 +0800 Subject: [PATCH 276/334] [doc](catalog) hive-e2e: land recursive-directories design + red-team, mark done, repoint remaining Fold the 5 minor red-team folds into the design (legacy net-parity of skip-hidden-dirs via full-path containsHiddenPath, listFileSizes as 3rd consumer, subdir-failure test, FakeFileSystem invariants, RED-ability labels). HANDOFF: hive_config_test tags 2/21 DONE (2bc7ad9bc7e); remaining fe-core/SPI work narrows to query_cache + default_partition (+ ENV items). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 7 +- .../FIX-recursive-directories-design.md | 215 ++++++++++++++++++ 2 files changed, 219 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-recursive-directories-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index ee4f5a9723023c..af7b791e54bab5 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -19,11 +19,12 @@ > - **text_write(LZ4) `e1d48045bee`**——`.lz4` 文本表读炸 BE `LZ4F_getFrameInfo ERROR_frameType_unknown`。根因=`.lz4` 被 `Util.inferFileCompressTypeByPath` 推成 LZ4FRAME 发 BE，但 hadoop/hive 写的是 LZ4 **block** 编码；legacy `HiveScanNode.getFileCompressType` 有 LZ4FRAME→LZ4BLOCK remap，SPI 翻闸后丢了。修=连接器 opt-in（镜像 supportsTableSample/supportsBatchScan）：`ConnectorScanPlanProvider.adjustFileCompressType` 默认 identity → `PluginDrivenScanNode.getFileCompressType` 跑 super 推断后委派（resolveScanProvider+onPluginClassLoader，无 source 分支）→ `HiveScanPlanProvider` override 只 remap LZ4FRAME。**hudi 严格对齐**（红队定：legacy `HudiScanNode extends HiveScanNode` 继承了 remap，新独立 provider 须重声明，别赌"hudi 从不出 .lz4"）；其余 6 provider 留 identity。测试：api identity 1/1 + hive remap 5/5 + hudi parity 4/4（api+hive 对子=RED/GREEN），fe-core 节点 54/54 无回归，0 checkstyle。设计 `designs/FIX-text-write-LZ4-design.md`。**e2e 待用户自跑 test_hive_text_write_insert 的 lz4 迭代。** > - **R2(SHOW CREATE) `17764d03665`**——翻闸 hive 表 SHOW CREATE 出 Doris 通用风格(双引号 PROPERTIES),非 hive 原生 `ROW FORMAT SERDE`/`STORED AS`。**方案=Option X(用户 2026-07-12 签字)**:惰性连接器方法 + **命令期实时取 HMS** + 连接器侧渲染,只在 `ShowCreateTableCommand` 拦截、**不声明 capability**。**两版旧方案被红队证伪**:(1)eager schema-cache 渲染过不了 `test_hive_meta_cache`(desc 缓存 5 列、show create 须新取 6 列);(2)声明 capability 会翻转共享 `Env.getDdlStmt` 门→误触 delegated hudi/iceberg-on-HMS + HTTP 端点。**7 文件**:`ConnectorTableOps.renderShowCreateTableDdl` 默认空(iceberg/paimon/es/jdbc 继承→落 Env.getDdlStmt 不变) · `HmsClient.getTableFresh`+`CachingHmsClient` override(绕 tableCache,镜像 listPartitionNamesFresh) · `HiveShowCreateTableRenderer`(fe-connector-hms,逐字节镜像 legacy `HiveMetaStoreClientHelper.showCreateTable`:两套引号约定 SERDEPROPERTIES `'k' = 'v'`/TBLPROPERTIES `'k'='v'`、2空格/1空格缩进、comment 提到 COMMENT 子句、null-comment 门) · `HiveConnectorMetadata` override(非 HiveTableHandle=delegated→空,镜像 getTableSchema 门;fresh getTableFresh) · `PluginDrivenExternalTable.getShowCreateTableDdl`(镜像 getViewText、读锁安全、handle 未解析→空) · `ShowCreateTableCommand` view 臂后新臂(门=方法返回 present 非 source 名,iceberg/paimon/view 不受影响)。**两红队 lens 均 SOUND_WITH_FIXES→已折入 2 守卫(foreign-handle 门 + 未解析→空)**。fe-core 连接器无关、无 source 分支;TCCL 在 `ThriftHmsClient.doAs` 内钉、renderer reflection-free。测试:`HiveShowCreateTableRendererTest` 5(byte-parity,两套引号+null-comment 门) + `CachingHmsClientTest` +3(getTableFresh 双向绕缓存)、fe-core PluginDrivenExternalTable 36/36 + EnvShowCreate 3/3 不变、hive 307/307、0 checkstyle。设计 `designs/FIX-R2-showcreate-design.md`。**残余**:toHiveTypeString 对 `varchar(n)`→`string`(HMS 本就存 string 故无碍)、不可映射类型抛 IAE(legacy 回显原串、无套件触发);TBLPROPERTIES 逐字发含 volatile 键(同 legacy,勿加 .out golden)。**e2e 待用户自跑:test_hive_show_create_table + test_hive_ddl_text_format(+ test_hive_meta_cache 新取契约 / test_multi_delimit_serde / test_hive_ddl)。** > - **R3($partitions 系统表) `e697f189c59`**——翻闸 hive 表查 `表$partitions` 报 `"Unknown sys table"`（`desc` 报 `"sys table not found"`）。根因=翻闸 hive=`PluginDrivenExternalTable`，其 `getSupportedSysTables()` 把连接器上报的每个系统表名**无条件包成原生 `PluginDrivenSysTable`**，而 `HiveConnectorMetadata.listSupportedSysTables` 对 hive handle 返回空；旧 `HMSExternalTable` 对 HIVE 返回**走 TVF** 的 `PartitionsSysTable`（`partition_values` TVF 及其 BE→FE 元数据回取**早已全 plugin-aware**，只丢了 `$partitions`→系统表→TVF 这座桥）。**关键陷阱**：fe-core 不能按名字 `"partitions"` 硬映射到 TVF——**iceberg(`MetadataTableType.PARTITIONS`)+paimon(`SystemTableLoader`) 也上报 `"partitions"` 但是原生元数据表**，硬映射会把它俩打断；种类须**连接器声明**（不能从 `getSysTableHandle` 空来推断：种类在 discovery 期就要、`getSysTableHandle` eager+auth、iceberg 合法省略 position_deletes）。修=**连接器声明 kind 的 `supports*`-式 opt-in**（铁律干净）：SPI `ConnectorTableOps.isPartitionValuesSysTable` 默认 false→原生 · fe-core `getSupportedSysTables()` 据此分流(TVF→`PartitionsSysTable.INSTANCE`,键钉 INSTANCE 自身名避免 createFunction 切后缀崩;否则原生) · `HiveConnectorMetadata` 对 hive handle `listSupportedSysTables` 返回 `["partitions"]`(**无条件**,含非分区表——`$partitions` 打非分区表须到 TVF ctor 抛 `"is not a partitioned table"` 而非 `"Unknown sys table"`,镜像旧行为)、`isPartitionValuesSysTable("partitions")=true`(外来 handle 委派兄弟→iceberg/hudi-on-HMS 保原生)、`getSysTableHandle` 仍空(TVF 路不查)。无 BE/Nereids 改。3 路对抗红队→GO_WITH_FIXES(5 折全折入:F1 键钉 INSTANCE 名/F2 外来 handle 守卫 verbatim/F3 iceberg-on-HMS+原生 iceberg/paimon t$partitions e2e 提为必测/F4 行号/F5 驳 getSysTableHandle-空推断)。测试:新 `HiveConnectorMetadataSysTableTest` 5 + `HiveConnectorMetadataSiblingDelegationTest`(委派+hive 暴露 partitions,`EXPECTED_METHODS`+`RecordingSiblingMetadata` 加 `isPartitionValuesSysTable`) + fe-core `PluginDrivenSysTableTest` +1(TVF-kind wrap→`PartitionsSysTable`/非原生);**fe-connector-hive 312/312 + fe-core PluginDrivenSysTable 11/11、0 checkstyle**。设计 `designs/FIX-R3-partitions-systable-design.md`。**e2e 待用户自跑（必测 GREEN 门）：test_hive_partition_values_tvf（sql01-113 全部）+ iceberg-on-HMS/原生 iceberg/paimon `t$partitions` 仍返回原生行（F2 外来-handle 守卫的唯一证明，单测因 iceberg divert 休眠证不了）。** -> **⭐ 剩余 5 suite 待修（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **用户已裁决(A)的 3 个 fe-core/SPI 大改**： +> **✅ 本 session DONE：hive_config_test 递归目录（tags 2/21）`2bc7ad9bc7e`**——翻闸 hive 表列分区文件走 `HiveFileListingCache.listFromFileSystem`，此前**无条件非递归**且从不读 `hive.recursive_directories`（legacy `HiveExternalMetaCache.getFileCache` 默认 "true" 递归）→ 数据在子目录的表静默丢行。修=连接器就地（**fe-core/SPI 零改**）在 cache ctor 解析该旗（默认 true）钉进生产 lister；置真时 `collectFiles` 递归**非隐藏**子目录，隐藏目录(`_temporary`/`.hive-staging`)+`_`/`.` 文件逐层跳——与 legacy 全路径 `containsHiddenPath` 过滤**净等价**（连接器只按叶名 `FileEntry.name()` 过滤，descend-all 会捞出 legacy 抑制的 staging 文件，红队证伪 descend-all）。列举循环下沉为递归助手 `collectFiles`，抛到**同一** systemic-vs-local 分类器（子目录失败与顶层同判）。三消费方（scan 切分 `HiveScanPlanProvider:473`、size 估算 `sumCachedFileSizes:1003`、`ANALYZE…WITH SAMPLE` 统计 `listFileSizes:886`）经单一共享实例天然一致；`recursive=false` 与今日逐字节相同。设计红队 `wf_9dabd9b6-4db`（5 lens）= **GO_WITH_FIXES**（0 blocker/major，5 minor 全折入：§2.3/§3 文档订正 + 子目录失败测试 + FakeFileSystem 不变式/RED 标注）。测试 `HiveFileListingCacheTest` +6（递归下降/非递归只顶层/隐藏跳过/默认 true/属性 false/嵌套失败可跳）+ FakeFileSystem 加 tree-mode+per-location error；**fe-connector-hive 318/318、0 checkstyle**。设计 `designs/FIX-recursive-directories-design.md`。**e2e 待用户自跑：hive_config_test tags 2/21（应各返 6 行）；tag1 仍是 ENV。** +> +> **⭐ 剩余 2 fe-core/SPI 大改（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **用户已裁决(A)**： > - query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 > - default_partition 连接器传 null 标记：SPI 新字段让连接器按分区值传 null 语义，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem:310`(现无条件 `PartitionValue(v,false)`) 据此建项——hive 标 null、paimon 保持非空(零 paimon 影响，别在 fe-core if(hive))。 -> - hive_config_test 恢复 `hive.recursive_directories`：`HiveFileListingCache.listFromFileSystem`(现无条件非递归 `fs.list`+跳目录)按属性(默认 true)手动递归；**须保持 3 个消费方 byte-parity**(scan 切分 `listAndSplitFiles`、size 估算 `sumCachedFileSizes`、cache)——须把 recursive flag 从 properties 串到 lister。(tag1 另是 ENV：HDFS OUTFILE 累积，须清路径。) > - **ENV(告知用户，非代码)**：重置外部 hive docker → test_hive_lzo_text_format(run86.hql 未应用)、test_hive_varbinary_type(残留行翻倍)、hive_config_test tag1(HDFS OUTFILE 累积) 一并解决。 **✅ 本轮 DONE（4 commit，均编译+全套测试绿）**： diff --git a/plan-doc/tasks/designs/FIX-recursive-directories-design.md b/plan-doc/tasks/designs/FIX-recursive-directories-design.md new file mode 100644 index 00000000000000..8b715750038d9d --- /dev/null +++ b/plan-doc/tasks/designs/FIX-recursive-directories-design.md @@ -0,0 +1,215 @@ +# FIX — restore `hive.recursive_directories` (recursive partition-dir listing) + +**Status:** design → (red-team) → implement → UT → commit +**Scope:** connector-local (`fe-connector-hive`), **zero fe-core change**, zero SPI change. +**Suite:** `external_table_p0/hive/hive_config_test` tags `2` and `21` (and the ENV tag `1`, separate). + +--- + +## 1. Problem / root cause (HEAD-verified) + +After the hms flip, a plugin-driven hive catalog lists partition data files through +`HiveFileListingCache.listFromFileSystem` (fe-connector-hive). That loader is **unconditionally +non-recursive**: it iterates `resolved.list(loc)` once, `continue`s on every directory entry +(`HiveFileListingCache.java:196`), and never descends. It also **never reads** the catalog property +`hive.recursive_directories`. + +Legacy fe-core honored it: `HiveExternalMetaCache.getFileCache:390-397` read +`catalog.getProperties().getOrDefault("hive.recursive_directories", "true")` (**default true**) and +passed it to `directoryLister.listFiles(fs, isRecursiveDirectories, …)` → `FileSystemDirectoryLister` +→ `FileSystemTransferUtil.globList(fs, location, recursive)` → `collectEntries(…, recursive, …)` +(`FileSystemTransferUtil.java:109-130`), which recurses into subdirectories when `recursive`. + +Consequence on a table whose data lives in subdirectories: the connector returns only the top-level +files → **silent missing rows** (data loss), not an error. + +### Acceptance criterion (the test) +`hive_recursive_directories_table` (built by `hive_config_test.groovy` via `INTO OUTFILE`) has data at +three levels — top-level `exp_*`, subdir `1/exp_*`, subdir `2/exp_*` (subdir names `1`,`2` and file +names `exp_*` are **non-hidden**). Golden `hive_config_test.out`: + +| tag | property | expected rows | +|---|---|---| +| `1` | `hive.recursive_directories=false` | 2 (top-level only) — **already correct today** | +| `2` | `hive.recursive_directories=true` | 6 (top + `1/` + `2/`) — **broken today, returns 2** | +| `21` | (default, absent → true) | 6 — **broken today, returns 2** | + +So: the fix must make `recursive=true`/default descend into subdirectories; `recursive=false` must stay +byte-identical to today (top-level only). + +--- + +## 2. Design + +All three readers (`HiveScanPlanProvider.listAndSplitFiles`, `HiveConnectorMetadata.sumCachedFileSizes` +for `estimateDataSizeByListingFiles`, and the cache itself) go through the **same** shared +`HiveFileListingCache.listDataFiles` → `DirectoryLister.list` → `listFromFileSystem`. The recursive flag +is a **per-catalog constant**, so bake it into the cache instance and they are consistent by +construction — no per-consumer change, no signature change on the hot path. + +### 2.1 Parse the flag once, in the cache ctor +`HiveFileListingCache.java`: +- Add `static final String RECURSIVE_DIRECTORIES_PROPERTY = "hive.recursive_directories";` +- Public ctor `HiveFileListingCache(Map properties)` builds the production lister as a + lambda that captures the parsed flag: + ```java + public HiveFileListingCache(Map properties) { + this(properties, defaultLister(properties)); + } + private static DirectoryLister defaultLister(Map properties) { + Map props = properties == null ? Collections.emptyMap() : properties; + boolean recursive = Boolean.parseBoolean(props.getOrDefault(RECURSIVE_DIRECTORIES_PROPERTY, "true")); + return (location, fs) -> listFromFileSystem(location, fs, recursive); + } + ``` + `Boolean.parseBoolean` matches legacy `Boolean.valueOf` semantics (only "true" case-insensitive → true; + default string "true"). +- The `DirectoryLister` functional interface stays `(location, fs)` — test injection unchanged. + +### 2.2 Recursive-aware loader +Introduce `listFromFileSystem(String location, FileSystem fs, boolean recursive)` and keep the existing +2-arg `listFromFileSystem(String, FileSystem)` delegating to `(…, false)` (the 5 existing direct-call +tests assert non-recursive dir-skipping and use a flat, non-tree fake — they must keep testing exactly +today's behavior; recursion through that flat fake would loop). + +The top-level failure classification (systemic-loud vs local-skippable, null-fs) is **unchanged**; only +the inner listing loop is factored into a recursive helper so a sub-listing `IOException` bubbles to the +SAME outer classifier (a failure anywhere under the partition → the partition's one systemic/local +verdict, exactly as before): + +```java +static List listFromFileSystem(String location, FileSystem fs, boolean recursive) { + // ... unchanged null-fs guard + forLocation resolution (systemic) ... + try { + List files = new ArrayList<>(); + collectFiles(resolved, loc, recursive, files); + return files; + } catch (IOException e) { + // ... unchanged isSystemicResolutionFailure split ... + } +} + +private static void collectFiles(FileSystem resolved, Location dir, boolean recursive, + List out) throws IOException { + try (FileIterator it = resolved.list(dir)) { + while (it.hasNext()) { + FileEntry entry = it.next(); + String name = entry.name(); + if (entry.isDirectory()) { + if (recursive && !isHidden(name)) { + collectFiles(resolved, entry.location(), true, out); // reuse resolved FS (same scheme) + } + continue; + } + if (isHidden(name)) { + continue; + } + out.add(new HiveFileStatus(entry.location().uri(), entry.length(), entry.modificationTime())); + } + } +} + +private static boolean isHidden(String name) { + return name.startsWith("_") || name.startsWith("."); +} +``` + +Notes: +- `entry.name()` returns the last path component (handles the dir trailing slash) — correct for both + file and dir hidden checks (`FileEntry.java:72`). +- Descent reuses the already-resolved `resolved` FS (the subdir shares scheme/authority) — cheaper and + avoids a second `forLocation`. +- **Non-recursive path is byte-identical**: with `recursive=false`, directories hit `continue` (no + descent), files pass the same `_`/`.` filter — the exact current loop, just extracted. + +### 2.3 Hidden-directory policy (deliberate) — EXACT legacy net parity (red-team verified) +When recursing, **skip hidden subdirectories** (`_`/`.` prefixed — e.g. `_temporary`, `.hive-staging`), +mirroring the existing leaf-file filter. This is **exact net parity with legacy**, not a deviation: +- Legacy `HiveExternalMetaCache.getFileCache` → `globList`/`collectEntries` **does** descend into ALL + directories (including hidden), BUT every returned file then passes the mandatory post-filter + `isFileVisible` → `containsHiddenPath` (`HiveExternalMetaCache.java:1005-1025`), which drops any file + whose **FULL path** contains a `_`/`.`-prefixed component (`startsWith(".")||("_")`, or any `/.`/`/_`). + So legacy's NET file set already excludes every file under a hidden subdirectory. Skipping hidden dirs + up front produces the identical net set. +- **Descend-all is REGRESSED here, not more-legacy.** The connector filters by the **leaf** + `entry.name()` only (`HiveFileListingCache.java:199-202`), never the full path. So "descend into all + dirs + leaf-only filter" would surface `_temporary/part-0` / `.hive-staging/…` files that legacy + suppresses — a real data-correctness regression. Skip-hidden-dirs is the parity-safe choice. +- The test does not pin this either way (its subdirs `1`,`2` are non-hidden), so the choice is + golden-safe; §4 adds a test that pins it. + +--- + +## 3. Consumers / parity (why one change suffices) +`HiveConnector` builds a single `HiveFileListingCache` (`:116`) and injects the SAME instance into the +scan provider (`:173`) and metadata (`:139-141`). Both `HiveConnectorMetadata` ctors parse the flag from +the same catalog `properties`. Therefore: +- **scan split** (`HiveScanPlanProvider:473` `listAndSplitFiles` → `listDataFiles`) — recursive. +- **size estimate** (`HiveConnectorMetadata:1003` `sumCachedFileSizes` → `listDataFiles`) — recursive, + consistent with scan (so cardinality/bytes match the rows actually scanned). +- **stats sampling** (`HiveConnectorMetadata:886` `listFileSizes` → `listDataFiles`, the + `ANALYZE … WITH SAMPLE` / legacy `getChunkSizes` port) — also becomes recursive via the same shared + instance. Legacy parity holds: `getChunkSizes → getFilesForPartitions → getFileCache` honored + `hive.recursive_directories` too, so scan and stats stay consistent. +- **cache** — keyed by `(db, table, location)`; the loader now returns the recursive listing; invalidation + unchanged. + +Out of scope (unchanged): the **ACID** path (`HiveAcidUtil` — its own base/delta descent, legacy +`recursive_directories` never applied to it) and the **write** path (`HiveConnectorTransaction`). + +--- + +## 4. Tests (`HiveFileListingCacheTest`) +Add a **tree-aware** capability to `FakeFileSystem`. Two **load-bearing invariants** (drop either and the +existing flat-fake subdir test recurses INFINITELY, because flat `list()` re-returns the same entries for +any location): +- **(a)** the retained 2-arg `listFromFileSystem(String,FileSystem)` MUST delegate to `(…, false)`. +- **(b)** `FakeFileSystem.list()` MUST consult the tree map ONLY when it is populated; otherwise return + the flat `entries` iterator **unchanged** (existing tests untouched). +Also add a **per-location** list error to the tree fake (the current single `listError` fails EVERY +`list()`, so it cannot model "top succeeds, one subdir fails"). New imports (`Map`/`HashMap`) go in the +`CustomImportOrder` alphabetical slot (`checkstyle.xml`). + +New tests: +1. `recursiveDescendsIntoSubdirectories` — tree: top has `exp_a` + dirs `1`,`2`; `1` has `exp_b`; `2` has + `exp_c`. `listFromFileSystem(top, fs, true)` → 3 files (a,b,c). +2. `nonRecursiveListsTopLevelOnly` — same tree, `recursive=false` → 1 file (`exp_a`). Pins the tag-`1` + behavior and that `false` never descends. +3. `recursiveSkipsHiddenSubdirectoriesAndFiles` — tree with `_temporary/` (containing a file) and a + top-level `.hidden` file → both excluded; only real files survive. Pins the §2.3 policy. +4. `defaultIsRecursive` / `recursiveFlagFalseFromProperty` — via the public ctor: + `new HiveFileListingCache(emptyMap())` and `new HiveFileListingCache(props("hive.recursive_directories","false"))` + drive a tree through `listDataFiles`; assert 3 vs 1 files. **This is the genuine RED-against-literal-HEAD + guarantee**: today's non-recursive lister returns 1 where the default-true assertion expects 3. Pins + default=true (tag `21`) and the property wiring (tag `2` vs tag `1`). +5. `recursiveSubdirListingFailureIsSkippable` — healthy top-level dir containing a subdir whose `list()` + throws `FileNotFoundException` → assert the skippable `HiveDirectoryListingException` propagates, + proving a **nested-descent** failure reaches the single outer classifier + (`HiveFileListingCache.java:208-219`). Pins red-team seed #2 (a future swallowing/reclassifying catch + around the recursion is caught RED). + +RED-ability note: tests #1–#3 and #5 call the NEW 3-arg `listFromFileSystem` overload (absent at HEAD), so +they RED only against a signature-added-but-non-recursive stub — standard TDD for a new overload. Test #4 +is the one that REDs against literal HEAD through the public ctor + `listDataFiles`. + +All existing `HiveFileListingCacheTest` cases remain (the flat-fake, non-recursive direct-call tests keep +pinning current behavior). fe-connector-hive full UT must stay green; 0 checkstyle. + +--- + +## 5. Iron-rule / architecture compliance +- **fe-core untouched**, **SPI untouched** — pure connector-local behavior restoration. +- No `if(hive)` anywhere; the flag is a hive catalog property parsed only inside the hive connector. +- Trino parity: Trino's `CachingDirectoryLister` / `hive.recursive-directories` is likewise a + connector-side listing knob; fe-core stays connector-agnostic. + +## 6. Red-team seeds (challenge these) +- Is skip-hidden-dirs (§2.3) a golden-safe / correct parity choice, or should recursion match legacy + `collectEntries` (descend all dirs)? +- Does factoring the loop into `collectFiles` keep the systemic-vs-local failure split byte-identical for + BOTH the top-level and a sub-directory failure? +- Any consumer of `listFromFileSystem`/`listDataFiles` NOT covered (ACID? write? stats sampling + `listFileSizes`)? Confirm ACID/write are genuinely out of scope for this property. +- `FakeFileSystem` tree-mode must not change the semantics the existing flat-mode tests rely on. +- Does `entry.location()` round-trip through `list()` in production (nested descent target correct for + real per-scheme FS, not just the fake)? From 58f3e367ed67b3add88adf8e759d584317115df3 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 08:10:08 +0800 Subject: [PATCH 277/334] [fix](catalog) hive/paimon: connector-supplied per-value NULL partition flag via SPI Restore genuine-NULL partition semantics for __HIVE_DEFAULT_PARTITION__ on a non-string (INT/DATE) partition column. The shared fe-core partition-item builder hardcoded isNull=false, so the sentinel was parsed as the column type (IntLiteral("__HIVE_DEFAULT_PARTITION__") throws), the partition was log-skipped, and the table mis-reported UNPARTITIONED (partition=0/0; WHERE col IS NULL empty). Add a positional List partitionValueNullFlags to ConnectorPartitionInfo (default empty = not opted in = non-null; unchanged for hudi/iceberg/maxcompute). fe-core PluginDrivenMvccExternalTable.toListPartitionItem zips the flag into PartitionValue(value, isNull) -> a typed NullLiteral, no parse. fe-core never string-compares a sentinel (iron rule): hive and paimon render the identical __HIVE_DEFAULT_PARTITION__ with opposite intent, so nullness is connector-supplied. - hive: HiveConnectorMetadata.listPartitions marks the HMS default-partition sentinel isNull=true (exact HIVE_DEFAULT_PARTITION.equals -> legacy HiveExternalMetaCache:309 parity), built positionally from HiveWriteUtils.toPartitionValues so it zips with the fe-core name re-parse. - paimon: adopts genuine-NULL too (user decision) -> its partition.default-name partition becomes a NullLiteral, so col IS NULL prunes to it and an MTMV refresh materializes the null rows (was col IN ('sentinel'), which dropped them). Realizes the connector's already-stated intent; partition-name identity kept. - fail loud: toListPartitionItem asserts the nullFlags arity (empty or == types). - PluginDrivenScanNode prune opt-out comment refreshed (code unchanged; the col IS NULL example no longer applies now that paimon's null is a NullLiteral). Tests (verified: api 6, fe-core 67, hive 9, paimon 10; 0 failures, 0 checkstyle): ConnectorPartitionInfoTest +3, PluginDrivenMvccExternalTableTest +3 (INT/DATE sentinel+flag -> NullLiteral; no-flag INT still drops), hive HiveConnectorMetadataPartitionListTest +2, paimon PaimonConnectorMetadataPartitionTest +1. Design + red-team (5 lenses, all SOUND_WITH_FIXES, folded): plan-doc/tasks/designs/FIX-default-partition-design.md e2e (user): test_hive_default_partition (INT+DATE) + paimon qt_null_partition_* + test_paimon_mtmv (golden 3->5 rows, regenerate; MV null-partition-creation gate). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/api/ConnectorPartitionInfo.java | 52 ++++++++++- .../api/ConnectorPartitionInfoTest.java | 54 ++++++++++++ .../connector/hive/HiveConnectorMetadata.java | 23 ++++- ...iveConnectorMetadataPartitionListTest.java | 32 +++++++ .../paimon/PaimonConnectorMetadata.java | 23 +++-- .../PaimonConnectorMetadataPartitionTest.java | 32 +++++++ .../PluginDrivenMvccExternalTable.java | 43 +++++---- .../datasource/PluginDrivenScanNode.java | 22 ++--- .../PluginDrivenMvccExternalTableTest.java | 88 +++++++++++++++---- .../suites/mtmv_p0/test_paimon_mtmv.groovy | 5 +- 10 files changed, 319 insertions(+), 55 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java index fe345d0b620bce..6e2f8c19e30750 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorPartitionInfo.java @@ -17,7 +17,9 @@ package org.apache.doris.connector.api; +import java.util.ArrayList; import java.util.Collections; +import java.util.List; import java.util.Map; import java.util.Objects; @@ -36,6 +38,16 @@ public final class ConnectorPartitionInfo { private final long sizeBytes; private final long lastModifiedMillis; private final long fileCount; + /** + * Per-partition-value SQL-NULL flags, positionally aligned to the values parsed out of + * {@link #partitionName} (i.e. flag {@code i} describes the {@code i}-th {@code key=value} segment). + * Empty means "no value is NULL" — a connector that does not opt in leaves it empty and the + * fe-core partition-item builder treats every value as non-null (unchanged behavior). A connector + * that renders a genuine-NULL partition value (e.g. hive's {@code __HIVE_DEFAULT_PARTITION__} or + * paimon's {@code partition.default-name}) sets the corresponding flag {@code true} so fe-core builds + * a typed {@code NullLiteral} instead of parsing the sentinel string into the column type. + */ + private final List partitionValueNullFlags; /** * Backward-compatible constructor. Numeric stats fields are set to @@ -48,10 +60,31 @@ public ConnectorPartitionInfo(String partitionName, UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN); } + /** + * Convenience constructor for a partition with unknown numeric stats but connector-supplied + * per-value NULL flags (e.g. hive, which lists names only). + */ + public ConnectorPartitionInfo(String partitionName, + Map partitionValues, + Map properties, + List partitionValueNullFlags) { + this(partitionName, partitionValues, properties, + UNKNOWN, UNKNOWN, UNKNOWN, UNKNOWN, partitionValueNullFlags); + } + public ConnectorPartitionInfo(String partitionName, Map partitionValues, Map properties, long rowCount, long sizeBytes, long lastModifiedMillis, long fileCount) { + this(partitionName, partitionValues, properties, + rowCount, sizeBytes, lastModifiedMillis, fileCount, Collections.emptyList()); + } + + public ConnectorPartitionInfo(String partitionName, + Map partitionValues, + Map properties, + long rowCount, long sizeBytes, long lastModifiedMillis, long fileCount, + List partitionValueNullFlags) { this.partitionName = Objects.requireNonNull( partitionName, "partitionName"); this.partitionValues = partitionValues == null @@ -64,6 +97,9 @@ public ConnectorPartitionInfo(String partitionName, this.sizeBytes = sizeBytes; this.lastModifiedMillis = lastModifiedMillis; this.fileCount = fileCount; + this.partitionValueNullFlags = partitionValueNullFlags == null + ? Collections.emptyList() + : Collections.unmodifiableList(new ArrayList<>(partitionValueNullFlags)); } public String getPartitionName() { @@ -98,6 +134,14 @@ public long getFileCount() { return fileCount; } + /** + * @return per-value SQL-NULL flags positionally aligned to the {@link #partitionName} value parse; + * empty when the connector did not opt in (no value is NULL). Unmodifiable. + */ + public List getPartitionValueNullFlags() { + return partitionValueNullFlags; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -113,13 +157,14 @@ public boolean equals(Object o) { && fileCount == that.fileCount && partitionName.equals(that.partitionName) && partitionValues.equals(that.partitionValues) - && properties.equals(that.properties); + && properties.equals(that.properties) + && partitionValueNullFlags.equals(that.partitionValueNullFlags); } @Override public int hashCode() { return Objects.hash(partitionName, partitionValues, properties, - rowCount, sizeBytes, lastModifiedMillis, fileCount); + rowCount, sizeBytes, lastModifiedMillis, fileCount, partitionValueNullFlags); } @Override @@ -128,6 +173,7 @@ public String toString() { + "', values=" + partitionValues + ", rowCount=" + rowCount + ", sizeBytes=" + sizeBytes - + ", fileCount=" + fileCount + "}"; + + ", fileCount=" + fileCount + + ", nullFlags=" + partitionValueNullFlags + "}"; } } diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorPartitionInfoTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorPartitionInfoTest.java index fb79235d2ae73c..73538bbcce8a83 100644 --- a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorPartitionInfoTest.java +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorPartitionInfoTest.java @@ -20,6 +20,7 @@ import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.Arrays; import java.util.Collections; /** @@ -74,4 +75,57 @@ public void equalsAndHashCodeIncludeFileCount() { // equals()/hashCode() -> a.equals(differByFileCount) -> red. Assertions.assertNotEquals(a, differByFileCount); } + + @Test + public void nullFlagsCtorsCarryPerValueNullFlags() { + // 4-arg convenience ctor (hive: UNKNOWN stats + connector-supplied per-value NULL flags). + ConnectorPartitionInfo hive = new ConnectorPartitionInfo( + "year=__HIVE_DEFAULT_PARTITION__/month=01", Collections.emptyMap(), Collections.emptyMap(), + Arrays.asList(true, false)); + // WHY: fe-core zips getPartitionValueNullFlags() index-for-index with the parsed values to decide + // NullLiteral vs typed literal, so the order and values must round-trip. MUTATION: dropping the + // flags assignment (-> empty) or reordering -> red. + Assertions.assertEquals(Arrays.asList(true, false), hive.getPartitionValueNullFlags()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, hive.getRowCount()); + + // 8-arg ctor (paimon: real stats + NULL flags). + ConnectorPartitionInfo paimon = new ConnectorPartitionInfo( + "region=__HIVE_DEFAULT_PARTITION__", Collections.emptyMap(), Collections.emptyMap(), + 1L, 2L, 3L, 4L, Collections.singletonList(true)); + Assertions.assertEquals(Collections.singletonList(true), paimon.getPartitionValueNullFlags()); + Assertions.assertEquals(4L, paimon.getFileCount()); + } + + @Test + public void backwardCompatCtorsDefaultNullFlagsEmpty() { + // WHY: connectors that do not opt in (3-arg MaxCompute/iceberg, 7-arg hudi) must default the flags + // to empty so fe-core treats every value as non-null (unchanged behavior). MUTATION: defaulting to + // a non-empty list -> red. A null flags arg must normalize to empty (not NPE). + ConnectorPartitionInfo threeArg = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap()); + ConnectorPartitionInfo sevenArg = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), 1L, 2L, 3L, 4L); + ConnectorPartitionInfo nullArg = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), null); + Assertions.assertTrue(threeArg.getPartitionValueNullFlags().isEmpty()); + Assertions.assertTrue(sevenArg.getPartitionValueNullFlags().isEmpty()); + Assertions.assertTrue(nullArg.getPartitionValueNullFlags().isEmpty()); + } + + @Test + public void equalsAndHashCodeIncludeNullFlags() { + ConnectorPartitionInfo a = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), Arrays.asList(true, false)); + ConnectorPartitionInfo b = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), Arrays.asList(true, false)); + ConnectorPartitionInfo differByFlags = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), Arrays.asList(false, false)); + + Assertions.assertEquals(a, b); + Assertions.assertEquals(a.hashCode(), b.hashCode()); + // WHY: two partitions differing only in per-value nullness (a genuine-NULL value vs a literal + // value that happens to render the same string) must not compare equal. MUTATION: omitting + // nullFlags from equals()/hashCode() -> a.equals(differByFlags) -> red. + Assertions.assertNotEquals(a, differByFlags); + } } diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index 73d9ea6b837f64..b951a95ead4c98 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -54,6 +54,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorIn; import org.apache.doris.connector.api.pushdown.ConnectorLiteral; import org.apache.doris.connector.api.pushdown.FilterApplicationResult; +import org.apache.doris.connector.api.scan.ConnectorPartitionValues; import org.apache.doris.connector.hms.HiveShowCreateTableRenderer; import org.apache.doris.connector.hms.HmsClient; import org.apache.doris.connector.hms.HmsClientConfig; @@ -1105,11 +1106,31 @@ public List listPartitions(ConnectorSession session, for (String partitionName : partitionNames) { result.add(new ConnectorPartitionInfo(partitionName, toPartitionValueMap(partitionName, partKeyNames), - Collections.emptyMap())); + Collections.emptyMap(), + toPartitionValueNullFlags(partitionName))); } return result; } + /** + * Per-value SQL-NULL flags for a rendered partition name, positionally aligned to + * {@link HiveWriteUtils#toPartitionValues} — the SAME parse fe-core re-runs at + * {@code PluginDrivenMvccExternalTable.toListPartitionItem}, so flag {@code i} zips to value {@code i} + * regardless of column casing/order (do NOT derive the order from the value map / partition-key names). + * A value equal to the HMS default-partition sentinel {@code __HIVE_DEFAULT_PARTITION__} is a genuine + * SQL NULL — byte-parity with legacy {@code HiveExternalMetaCache.toListPartitionItem}, which marks the + * sentinel (and only the sentinel) null; the broader {@code isNullPartitionValue} (which also treats + * {@code \N}/null as null) is deliberately not used (HMS partition names never carry {@code \N}). + */ + private static List toPartitionValueNullFlags(String partitionName) { + List values = HiveWriteUtils.toPartitionValues(partitionName); + List flags = new ArrayList<>(values.size()); + for (String value : values) { + flags.add(ConnectorPartitionValues.HIVE_DEFAULT_PARTITION.equals(value)); + } + return flags; + } + /** * Shared partition-name lister backing {@link #listPartitionNames}, {@link #listPartitions} and * {@link #getTableFreshness}. Returns the metastore's rendered partition names ({@code key=value/...}); an diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java index bd890732be956e..1b3b2ed16027ff 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataPartitionListTest.java @@ -162,6 +162,38 @@ public void testUnpartitionedTableListsNothingWithoutMetastoreCall() { "an unpartitioned table must not call the metastore"); } + @Test + public void listPartitionsMarksHiveDefaultSentinelNull() { + // A genuine-NULL partition on the `year` column: HMS renders it as year=__HIVE_DEFAULT_PARTITION__. + // The connector must supply isNull=true for that value (byte-parity with legacy + // HiveExternalMetaCache:309) and false for the ordinary `month` value, positionally aligned to the + // name parse fe-core re-runs -> fe-core builds a typed NullLiteral for `year` (INT/DATE-safe). + FakeHmsClient client = new FakeHmsClient(Collections.singletonList( + "year=__HIVE_DEFAULT_PARTITION__/month=01")); + List parts = + metadata(client).listPartitions(null, partitionedHandle(), Optional.empty()); + // MUTATION: dropping the flag (empty list) or marking the wrong position -> red. + Assertions.assertEquals(Arrays.asList(true, false), + parts.get(0).getPartitionValueNullFlags(), + "year (sentinel) -> null flag true; month -> false"); + // The raw value string is still carried (the flag, not the string, drives nullness downstream). + Assertions.assertEquals("__HIVE_DEFAULT_PARTITION__", parts.get(0).getPartitionValues().get("year")); + // Stats stay UNKNOWN — the opt-in flag must not perturb the names-only contract. + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, parts.get(0).getLastModifiedMillis()); + } + + @Test + public void listPartitionsMarksNoNullForOrdinaryValues() { + // Regression floor: ordinary partitions get all-false flags (no value is the sentinel), so fe-core + // builds plain typed literals exactly as before this fix. + FakeHmsClient client = new FakeHmsClient(PARTITIONS); + List parts = + metadata(client).listPartitions(null, partitionedHandle(), Optional.empty()); + for (ConnectorPartitionInfo part : parts) { + Assertions.assertEquals(Arrays.asList(false, false), part.getPartitionValueNullFlags()); + } + } + /** * Minimal {@link HmsClient} double: {@code listPartitionNames} returns a fixed list and records the * requested {@code maxParts}; {@code getPartitions} fails loud (the per-partition round-trip this path diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java index eeab04694bdb80..26033e16fbbb88 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java @@ -1044,16 +1044,22 @@ private List collectPartitions(PaimonTableHandle paimonH for (Partition partition : paimonPartitions) { Map spec = partition.spec(); StringBuilder sb = new StringBuilder(); + // Per-value SQL-NULL flags, built in this SAME loop so flag i aligns with name segment i (which is + // how fe-core re-parses the rendered name positionally at PluginDrivenMvccExternalTable). + List nullFlags = new ArrayList<>(spec.size()); for (Map.Entry entry : spec.entrySet()) { sb.append(entry.getKey()).append("="); String value = entry.getValue(); - if (defaultPartitionName.equals(value)) { - // Genuine NULL partition value. Normalize the paimon sentinel to the Doris-canonical - // null sentinel so the FE prune bridge (PluginDrivenMvccExternalTable.toListPartitionItem) - // marks the partition isNull and `col IS NULL` selects it — aligning prune with the - // native scan path, which already materializes it as SQL NULL from the typed Java-null. - // Handled before the DATE branch so a null DATE partition renders the sentinel instead - // of crashing on Integer.parseInt("__DEFAULT_PARTITION__"). + boolean isNull = defaultPartitionName.equals(value); + nullFlags.add(isNull); + if (isNull) { + // Genuine NULL partition value. Supply isNull=true so the FE bridge + // (PluginDrivenMvccExternalTable.toListPartitionItem) builds a typed NullLiteral and + // `col IS NULL` selects it (MTMV refresh materializes the null rows) — aligning prune with + // the native scan path, which already materializes it as SQL NULL from the typed Java-null. + // The name is still normalized to the Doris-canonical sentinel (partition-name identity is + // preserved; the value string is ignored once the flag marks it null). Handled before the + // DATE branch so a null DATE partition does not crash on Integer.parseInt("__DEFAULT_PARTITION__"). sb.append(ConnectorPartitionValues.HIVE_DEFAULT_PARTITION).append("/"); } else if (legacyName && dateColumns.contains(entry.getKey())) { // When partition.legacy-name = true (default), Paimon stores DATE as days since @@ -1076,7 +1082,8 @@ private List collectPartitions(PaimonTableHandle paimonH partition.recordCount(), partition.fileSizeInBytes(), partition.lastFileCreationTime(), - partition.fileCount())); + partition.fileCount(), + nullFlags)); } return result; } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataPartitionTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataPartitionTest.java index bfd585fb2d4d73..d439800a15ddf7 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataPartitionTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataPartitionTest.java @@ -297,6 +297,38 @@ public void nullDatePartitionRendersSentinelInsteadOfCrashing() { names); } + @Test + public void nullPartitionSuppliesNullFlagTrue() { + // Variant B: paimon adopts genuine-NULL semantics. Its genuine-NULL partition value (rendered as the + // Doris-canonical sentinel in the NAME) must ALSO carry isNull=true in the connector-supplied flag, so + // the FE bridge builds a typed NullLiteral (not a StringLiteral). This realizes the connector's stated + // intent: `category IS NULL` prunes to the null partition and an MTMV refresh materializes the null rows. + RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps(); + FakePaimonTable table = new FakePaimonTable( + "t1", categoryRowType(), Collections.singletonList("category"), Collections.emptyList()); + table.setOptions(Collections.singletonMap("partition.legacy-name", "true")); + ops.table = table; + Map nullSpec = new LinkedHashMap<>(); + nullSpec.put("category", "__DEFAULT_PARTITION__"); + Map literalSpec = new LinkedHashMap<>(); + literalSpec.put("category", "bj"); + ops.partitions = Arrays.asList( + partition(nullSpec, 1L, 1L, 1L), + partition(literalSpec, 1L, 1L, 1L)); + + List infos = + metadataWith(ops).listPartitions(null, categoryHandle(table), Optional.empty()); + + // MUTATION: leaving paimon's flag false (pre-B parity) -> the null partition stays a StringLiteral -> red. + Assertions.assertEquals(Collections.singletonList(true), + infos.get(0).getPartitionValueNullFlags(), "genuine-null partition -> isNull flag true"); + Assertions.assertEquals(Collections.singletonList(false), + infos.get(1).getPartitionValueNullFlags(), "ordinary value -> isNull flag false"); + // The name is still normalized to the sentinel (partition-name identity preserved). + Assertions.assertEquals("category=" + ConnectorPartitionValues.HIVE_DEFAULT_PARTITION, + infos.get(0).getPartitionName()); + } + @Test public void nonPartitionedHandleReturnsEmptyWithoutSeamCall() { RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps(); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java index f6eae7b901ce0a..13ebd9cd85c157 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenMvccExternalTable.java @@ -273,7 +273,8 @@ private void listLatestPartitions(ConnectorMetadata metadata, ConnectorSession s // The connector already renders values (incl. dates) into getPartitionName(), so // building from the rendered name is byte-parity with legacy. Partition values may be // malformed; catch to avoid affecting the query (parity generatePartitionInfo). - nameToPartitionItem.put(partitionName, toListPartitionItem(partitionName, types)); + nameToPartitionItem.put(partitionName, + toListPartitionItem(partitionName, types, part.getPartitionValueNullFlags())); } catch (Exception e) { LOG.warn("toListPartitionItem failed, partitionColumns: {}, partitionName: {}", partitionColumns, partitionName, e); @@ -286,28 +287,36 @@ private ConnectorMvccSnapshot emptySnapshot() { } /** - * Builds a {@link ListPartitionItem} from a RENDERED partition name (e.g. {@code "dt=2024-01-01"}). - * Copied verbatim from legacy {@code PaimonUtil.toListPartitionItem}; it is source-agnostic. + * Builds a {@link ListPartitionItem} from a RENDERED partition name (e.g. {@code "dt=2024-01-01"}) and the + * connector-supplied per-value SQL-NULL flags. Source-agnostic: the connector — not fe-core — decides which + * values are genuine NULL. + * + *

    Each parsed value {@code i} builds {@code new PartitionValue(value, nullFlags.get(i))}. A connector that + * renders a genuine-NULL partition value (hive's {@code __HIVE_DEFAULT_PARTITION__}, paimon's + * {@code partition.default-name}) supplies {@code true} for that position, so + * {@link PartitionKey#createListPartitionKeyWithTypes} emits a typed {@code NullLiteral} instead of parsing + * the sentinel string into the column type — which for a non-string column (INT/DATE/...) would throw and + * silently drop the partition (table then mis-reported UNPARTITIONED, {@code partition=0/0}). The genuine-null + * partition then prunes on {@code col IS NULL} and an MTMV refresh materializes its null rows. A connector + * that supplies no flags ({@code nullFlags} empty) treats every value as non-null — unchanged behavior for + * connectors that do not opt in. {@code fe-core} never string-compares a sentinel (iron rule): hive and paimon + * render the identical {@code __HIVE_DEFAULT_PARTITION__} string with connector-specific NULL semantics, so + * nullness must be connector-supplied. */ - private static ListPartitionItem toListPartitionItem(String partitionName, List types) - throws AnalysisException { + private static ListPartitionItem toListPartitionItem(String partitionName, List types, + List nullFlags) throws AnalysisException { // Partition name will be in format: nation=cn/city=beijing // parse it to get values "cn" and "beijing" List partitionValues = HiveUtil.toPartitionValues(partitionName); Preconditions.checkState(partitionValues.size() == types.size(), partitionName + " vs. " + types); + // Fail loud: a connector that opts in MUST supply one flag per value; a short list would silently + // default the tail to isNull=false and re-introduce the drop bug. Empty = not opted in = OK. + Preconditions.checkState(nullFlags.isEmpty() || nullFlags.size() == types.size(), + "nullFlags " + nullFlags + " vs. " + types); List values = Lists.newArrayListWithExpectedSize(types.size()); - for (String partitionValue : partitionValues) { - // Master parity (PaimonUtil.toListPartitionItem: `new PartitionValue(value, false)`): every - // partition value — including a genuine-NULL value the connector rendered via its sentinel - // (e.g. paimon's partition.default-name normalized to __HIVE_DEFAULT_PARTITION__) — builds a - // NON-null (isNull=false) partition value. So the genuine-null partition is a plain - // StringLiteral; an MTMV refresh emits `col IN ('')` which the scan's genuine SQL-NULL - // rows never match (the null rows are dropped from the MV, like master), and its MTMV name is - // derived from the sentinel (distinct from a literal 'NULL' partition — no p_NULL collision). - // `col IS NULL` still returns the genuine-null rows: the paimon scan is predicate-driven and the - // connector opts out of the FE prune-to-zero short-circuit (see Connector capability consulted by - // PluginDrivenScanNode.resolveRequiredPartitions), so the SDK re-plans with the pushed predicate. - values.add(new PartitionValue(partitionValue, false)); + for (int i = 0; i < partitionValues.size(); i++) { + boolean isNull = i < nullFlags.size() && nullFlags.get(i); + values.add(new PartitionValue(partitionValues.get(i), isNull)); } PartitionKey key = PartitionKey.createListPartitionKeyWithTypes(values, types, true); return new ListPartitionItem(Lists.newArrayList(key)); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index f754ad7ab751f7..a76d5775f0c2fc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -240,14 +240,16 @@ static List resolveRequiredPartitions(SelectedPartitions selectedPartiti * *

    A connector whose {@link ConnectorScanPlanProvider#ignorePartitionPruneShortCircuit()} is {@code * true} (e.g. paimon, whose {@code planScan} ignores {@code requiredPartitions} and re-plans through the - * SDK with the pushed predicate) must NOT be short-circuited to zero rows when FE pruning empties the - * partition set: with master-parity {@code isNull=false} a genuine-null partition renders as a non-null - * sentinel, so {@code col IS NULL} prunes every partition away, yet the genuine-null rows must still be - * returned via the pushed predicate (regression {@code qt_null_partition_4}). For such a connector a - * prune-to-zero maps to {@code null} (scan-all) instead of the empty list, exactly as the legacy - * {@code PaimonScanNode} (which never consults {@code selectedPartitions}). A non-empty pruned set is - * still forwarded unchanged. For every other connector ({@code ignorePartitionPruneShortCircuit=false}) - * the behavior is identical to {@link #resolveRequiredPartitions(SelectedPartitions)}.

    + * SDK with the pushed predicate) must NOT be short-circuited to zero rows when FE pruning genuinely empties + * the partition set — e.g. {@code col = } prunes every partition away, + * yet planScan must still run and answer from the pushed predicate. For such a connector a prune-to-zero + * maps to {@code null} (scan-all) instead of the empty list, exactly as the legacy {@code PaimonScanNode} + * (which never consults {@code selectedPartitions}). A non-empty pruned set is still forwarded unchanged. + * (Note: {@code col IS NULL} over a connector-supplied genuine-NULL partition now prunes ACCURATELY to that + * {@code NullLiteral} partition — a non-empty set — so it no longer relies on this opt-out; see + * {@code PluginDrivenMvccExternalTable.toListPartitionItem}.) For every other connector + * ({@code ignorePartitionPruneShortCircuit=false}) the behavior is identical to + * {@link #resolveRequiredPartitions(SelectedPartitions)}.

    */ static List resolveRequiredPartitions(SelectedPartitions selectedPartitions, boolean ignorePartitionPruneShortCircuit) { @@ -271,8 +273,8 @@ static List resolveRequiredPartitions(SelectedPartitions selectedPartiti } // A predicate-driven connector re-plans through its SDK with the pushed predicate (its planScan // ignores requiredPartitions), so a GENUINE prune-to-zero must NOT short-circuit the scan — return - // scan-all (null) and let planScan answer from the predicate (e.g. paimon `col IS NULL` over a - // genuine-null partition the FE pruner rendered as a non-null sentinel). Master PaimonScanNode parity. + // scan-all (null) and let planScan answer from the predicate (e.g. paimon `col = `). Master PaimonScanNode parity. if (ignorePartitionPruneShortCircuit && selectedPartitions.selectedPartitions.isEmpty()) { return null; } diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java index 53bb0c0c2da15c..eae7ceb4244c6c 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenMvccExternalTableTest.java @@ -55,6 +55,7 @@ import org.mockito.InOrder; import org.mockito.Mockito; +import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; import java.util.HashMap; @@ -268,18 +269,13 @@ public void testGetNameToPartitionItemsBuildsKeyFromRenderedDateName() { } @Test - public void testHiveDefaultSentinelBuildsNonNullStringKey() { - // Master parity (PaimonUtil.toListPartitionItem uses `new PartitionValue(value, false)` - // unconditionally): a genuine-NULL partition value — which the connector renders via the - // __HIVE_DEFAULT_PARTITION__ sentinel — builds a NON-null partition key (isNull=false), i.e. the - // literal sentinel string, NOT a NullLiteral. An MTMV refresh therefore emits - // `region IN ('__HIVE_DEFAULT_PARTITION__')`, which the scan's genuine SQL-NULL rows never match, so - // the null rows are dropped from the MV exactly like master ("Will lose null data", regression - // test_paimon_mtmv). `WHERE region IS NULL` still returns the genuine-null rows because the paimon - // scan is predicate-driven and does NOT short-circuit on an FE prune-to-zero (the connector's - // ignorePartitionPruningShortCircuit capability), NOT because the partition key is a NullLiteral. - // VARCHAR column: the sentinel parses to a plain StringLiteral (a DATE column can't parse it, so the - // bridge logs+skips that partition — also master parity, generatePartitionInfo's per-partition catch). + public void testDefaultSentinelWithoutFlagBuildsNonNullStringKey() { + // NO-FLAG DEFAULT path: a connector that supplies NO per-value null flags leaves every value non-null + // (isNull=false), so a __HIVE_DEFAULT_PARTITION__ value on a VARCHAR column builds a plain StringLiteral, + // NOT a NullLiteral. This is the unchanged default for connectors that do not opt in (hudi/maxcompute/ + // iceberg). NB: hive and paimon DO opt in now (variant B) and would supply isNull=true here — see the two + // ...BuildsGenuineNullPartition tests below. VARCHAR keeps the sentinel parseable; a non-string column + // without the flag throws+drops (per-partition catch) — see testDefaultSentinelWithoutFlagStillDrops. Fixture f = Fixture.with(Collections.singletonList( cpi("dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION, TS_2024_01_01)), Type.VARCHAR); Map items = f.table.getNameToPartitionItems(Optional.empty()); @@ -288,12 +284,63 @@ public void testHiveDefaultSentinelBuildsNonNullStringKey() { PartitionItem item = items.get("dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION); Assertions.assertTrue(item instanceof ListPartitionItem, "expected a ListPartitionItem"); PartitionKey key = ((ListPartitionItem) item).getItems().get(0); - // MUTATION: marking the sentinel isNull=true -> the key is a NullLiteral -> red. + // MUTATION: defaulting the absent flag to isNull=true -> the key is a NullLiteral -> red. Assertions.assertFalse(key.getKeys().get(0).isNullLiteral(), - "master parity: a __HIVE_DEFAULT_PARTITION__ value must build a NON-null literal key (isNull=false)"); + "no-flag default: a __HIVE_DEFAULT_PARTITION__ value must build a NON-null literal key (isNull=false)"); Assertions.assertEquals(TablePartitionValues.HIVE_DEFAULT_PARTITION, key.getKeys().get(0).getStringValue(), - "the genuine-null partition key must carry the sentinel string verbatim (a plain StringLiteral)"); + "the no-flag partition key must carry the sentinel string verbatim (a plain StringLiteral)"); + } + + @Test + public void testDefaultSentinelWithNullFlagOnIntColumnBuildsGenuineNullPartition() { + // RED before the fix (fe-core hardcoded isNull=false): the sentinel on an INT column parses via + // IntLiteral("__HIVE_DEFAULT_PARTITION__") -> NumberFormatException -> the partition is dropped -> the + // snapshot is invalid -> the table mis-reports UNPARTITIONED (partition=0/0). With the connector-supplied + // isNull=true flag the value builds a typed NullLiteral (no parse), so the table stays LIST-partitioned + // with a genuine-NULL partition (legacy HiveExternalMetaCache:309 parity; hive/paimon variant B). + Fixture f = Fixture.with(Collections.singletonList( + cpiNull("dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION, TS_2024_01_01, true)), Type.INT); + + Assertions.assertEquals(PartitionType.LIST, f.table.getPartitionType(Optional.empty()), + "a genuine-NULL INT partition must NOT collapse the table to UNPARTITIONED"); + Assertions.assertFalse(f.table.getPartitionColumns(Optional.empty()).isEmpty(), + "partition columns must survive (not emptied by an invalid partition set)"); + Map items = f.table.getNameToPartitionItems(Optional.empty()); + Assertions.assertEquals(1, items.size(), "the null partition must be present, not dropped"); + PartitionKey key = ((ListPartitionItem) items.get( + "dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION)).getItems().get(0); + // MUTATION: ignoring the flag (hardcoded false) -> IntLiteral parse throws -> 0 items / UNPARTITIONED -> red. + Assertions.assertTrue(key.getKeys().get(0).isNullLiteral(), + "the connector-supplied NULL flag must build a typed NullLiteral for the INT column"); + } + + @Test + public void testDefaultSentinelWithNullFlagOnDateColumnBuildsGenuineNullPartition() { + // Second non-string family (DATEV2 also throws on the sentinel pre-fix). Same expectation as the INT case. + Fixture f = Fixture.with(Collections.singletonList( + cpiNull("dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION, TS_2024_01_01, true)), Type.DATEV2); + + Assertions.assertEquals(PartitionType.LIST, f.table.getPartitionType(Optional.empty())); + Map items = f.table.getNameToPartitionItems(Optional.empty()); + Assertions.assertEquals(1, items.size()); + PartitionKey key = ((ListPartitionItem) items.get( + "dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION)).getItems().get(0); + Assertions.assertTrue(key.getKeys().get(0).isNullLiteral(), + "the connector-supplied NULL flag must build a typed NullLiteral for the DATE column"); + } + + @Test + public void testDefaultSentinelWithoutFlagStillDrops() { + // Locks the fix as OPT-IN: a connector that does NOT supply the flag keeps the pre-fix behavior on a + // non-string column — the sentinel throws on IntLiteral, the partition is dropped, the table degrades to + // UNPARTITIONED. (Compile-independent guard: uses only the pre-existing no-flag cpi helper.) + Fixture f = Fixture.with(Collections.singletonList( + cpi("dt=" + TablePartitionValues.HIVE_DEFAULT_PARTITION, TS_2024_01_01)), Type.INT); + + Assertions.assertEquals(PartitionType.UNPARTITIONED, f.table.getPartitionType(Optional.empty()), + "without the connector flag, an INT sentinel still drops the partition (UNPARTITIONED)"); + Assertions.assertTrue(f.table.getNameToPartitionItems(Optional.empty()).isEmpty()); } // ==================== no-cache schema: bypass the name-keyed cache and read fresh ==================== @@ -1102,6 +1149,17 @@ private static ConnectorPartitionInfo cpi(String name, long lastModifiedMillis) ConnectorPartitionInfo.UNKNOWN); } + /** Like {@link #cpi} but with connector-supplied per-value SQL-NULL flags (the opt-in path). */ + private static ConnectorPartitionInfo cpiNull(String name, long lastModifiedMillis, boolean... nullFlags) { + List flags = new ArrayList<>(nullFlags.length); + for (boolean b : nullFlags) { + flags.add(b); + } + return new ConnectorPartitionInfo(name, Collections.emptyMap(), Collections.emptyMap(), + ConnectorPartitionInfo.UNKNOWN, ConnectorPartitionInfo.UNKNOWN, lastModifiedMillis, + ConnectorPartitionInfo.UNKNOWN, flags); + } + /** * Runs {@code body} with {@code snapshot} pinned for {@code table} in a thread-local * {@link ConnectContext}'s {@link StatementContext}, then clears the thread-local. diff --git a/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy b/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy index c4ea889f92307a..114e76beeac07f 100644 --- a/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy +++ b/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy @@ -262,7 +262,10 @@ suite("test_paimon_mtmv", "p0,external,mtmv,external_docker,external_docker_dori REFRESH MATERIALIZED VIEW ${mvName} auto; """ waitingMTMVTaskFinishedByMvName(mvName) - // Will lose null data + // Connector-supplied NULL flag (paimon variant B): the genuine-NULL `region` partition is now a + // NullLiteral, so `region IS NULL` refresh MATERIALIZES the null rows (was dropped via + // `region IN ('__HIVE_DEFAULT_PARTITION__')`). The golden below must be regenerated on the e2e run to + // include the genuine-NULL rows; see plan-doc/tasks/designs/FIX-default-partition-design.md. order_qt_null_partition "SELECT * FROM ${mvName} " sql """drop materialized view if exists ${mvName};""" From 394f3f1c2ae02a8a1a64287cbad59040c79c733d Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 08:11:47 +0800 Subject: [PATCH 278/334] [doc](catalog) hive-e2e: land default_partition design (variant B) + red-team, mark DONE Design + red-team record for the connector-supplied per-value NULL partition flag (commit 58f3e367ed6). Records: Option A SPI shape (positional List) chosen over name-keyed map / Java-null-in-map; user variant B (paimon adopts genuine-NULL); 5-lens adversarial red-team (all SOUND_WITH_FIXES) with RT-F1..RT-F6 folded, incl. the major finding that test_paimon_mtmv is an impacted existing suite (golden 3->5). HANDOFF: default_partition DONE; query_cache is the sole remaining code item. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 11 +- .../designs/FIX-default-partition-design.md | 279 ++++++++++++++++++ 2 files changed, 285 insertions(+), 5 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-default-partition-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index af7b791e54bab5..8bee7e344e5dc4 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,7 @@ --- -# 🆕 下一个 session 起步 = 修复 hive e2e 剩余 regression（round-2：22 失败，triage 全定案） +# 🆕 下一个 session 起步 = query_cache 移植到 SPI（hive e2e round-2 仅剩此 1 项代码 + ENV；default_partition 本轮已 DONE `58f3e367ed6`） > **本轮(round-2)**：用户按最新代码重跑 = **37 suite 测试、22 失败**。**全 22 已逐个 HEAD-verified 根因 + 对抗复核**（44-agent workflow `wf_c9def639-775`）。定案：**15 CODE_FIX · 1 TEST_ALIGN · 2 ENV · 4 USER_DECISION（用户已全裁决 → 均转 CODE_FIX）**。**完整三元表 + 逐 suite 修复指针 + 用户裁决** = `plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md`（**起步必读**，比下面的 round-1 R1-R12 表更新更准）。 > **起步必读**：本段 + triage 文档 + `git log`。失败日志（勿 commit）：`nohup.out`(Jul 11 20:27)、FE `output/fe/log`、BE `output/be/log`。**BE 走 `format_v2`**。**行号信 HEAD 不信文档。** @@ -21,11 +21,12 @@ > - **R3($partitions 系统表) `e697f189c59`**——翻闸 hive 表查 `表$partitions` 报 `"Unknown sys table"`（`desc` 报 `"sys table not found"`）。根因=翻闸 hive=`PluginDrivenExternalTable`，其 `getSupportedSysTables()` 把连接器上报的每个系统表名**无条件包成原生 `PluginDrivenSysTable`**，而 `HiveConnectorMetadata.listSupportedSysTables` 对 hive handle 返回空；旧 `HMSExternalTable` 对 HIVE 返回**走 TVF** 的 `PartitionsSysTable`（`partition_values` TVF 及其 BE→FE 元数据回取**早已全 plugin-aware**，只丢了 `$partitions`→系统表→TVF 这座桥）。**关键陷阱**：fe-core 不能按名字 `"partitions"` 硬映射到 TVF——**iceberg(`MetadataTableType.PARTITIONS`)+paimon(`SystemTableLoader`) 也上报 `"partitions"` 但是原生元数据表**，硬映射会把它俩打断；种类须**连接器声明**（不能从 `getSysTableHandle` 空来推断：种类在 discovery 期就要、`getSysTableHandle` eager+auth、iceberg 合法省略 position_deletes）。修=**连接器声明 kind 的 `supports*`-式 opt-in**（铁律干净）：SPI `ConnectorTableOps.isPartitionValuesSysTable` 默认 false→原生 · fe-core `getSupportedSysTables()` 据此分流(TVF→`PartitionsSysTable.INSTANCE`,键钉 INSTANCE 自身名避免 createFunction 切后缀崩;否则原生) · `HiveConnectorMetadata` 对 hive handle `listSupportedSysTables` 返回 `["partitions"]`(**无条件**,含非分区表——`$partitions` 打非分区表须到 TVF ctor 抛 `"is not a partitioned table"` 而非 `"Unknown sys table"`,镜像旧行为)、`isPartitionValuesSysTable("partitions")=true`(外来 handle 委派兄弟→iceberg/hudi-on-HMS 保原生)、`getSysTableHandle` 仍空(TVF 路不查)。无 BE/Nereids 改。3 路对抗红队→GO_WITH_FIXES(5 折全折入:F1 键钉 INSTANCE 名/F2 外来 handle 守卫 verbatim/F3 iceberg-on-HMS+原生 iceberg/paimon t$partitions e2e 提为必测/F4 行号/F5 驳 getSysTableHandle-空推断)。测试:新 `HiveConnectorMetadataSysTableTest` 5 + `HiveConnectorMetadataSiblingDelegationTest`(委派+hive 暴露 partitions,`EXPECTED_METHODS`+`RecordingSiblingMetadata` 加 `isPartitionValuesSysTable`) + fe-core `PluginDrivenSysTableTest` +1(TVF-kind wrap→`PartitionsSysTable`/非原生);**fe-connector-hive 312/312 + fe-core PluginDrivenSysTable 11/11、0 checkstyle**。设计 `designs/FIX-R3-partitions-systable-design.md`。**e2e 待用户自跑（必测 GREEN 门）：test_hive_partition_values_tvf（sql01-113 全部）+ iceberg-on-HMS/原生 iceberg/paimon `t$partitions` 仍返回原生行（F2 外来-handle 守卫的唯一证明，单测因 iceberg divert 休眠证不了）。** > **✅ 本 session DONE：hive_config_test 递归目录（tags 2/21）`2bc7ad9bc7e`**——翻闸 hive 表列分区文件走 `HiveFileListingCache.listFromFileSystem`，此前**无条件非递归**且从不读 `hive.recursive_directories`（legacy `HiveExternalMetaCache.getFileCache` 默认 "true" 递归）→ 数据在子目录的表静默丢行。修=连接器就地（**fe-core/SPI 零改**）在 cache ctor 解析该旗（默认 true）钉进生产 lister；置真时 `collectFiles` 递归**非隐藏**子目录，隐藏目录(`_temporary`/`.hive-staging`)+`_`/`.` 文件逐层跳——与 legacy 全路径 `containsHiddenPath` 过滤**净等价**（连接器只按叶名 `FileEntry.name()` 过滤，descend-all 会捞出 legacy 抑制的 staging 文件，红队证伪 descend-all）。列举循环下沉为递归助手 `collectFiles`，抛到**同一** systemic-vs-local 分类器（子目录失败与顶层同判）。三消费方（scan 切分 `HiveScanPlanProvider:473`、size 估算 `sumCachedFileSizes:1003`、`ANALYZE…WITH SAMPLE` 统计 `listFileSizes:886`）经单一共享实例天然一致；`recursive=false` 与今日逐字节相同。设计红队 `wf_9dabd9b6-4db`（5 lens）= **GO_WITH_FIXES**（0 blocker/major，5 minor 全折入：§2.3/§3 文档订正 + 子目录失败测试 + FakeFileSystem 不变式/RED 标注）。测试 `HiveFileListingCacheTest` +6（递归下降/非递归只顶层/隐藏跳过/默认 true/属性 false/嵌套失败可跳）+ FakeFileSystem 加 tree-mode+per-location error；**fe-connector-hive 318/318、0 checkstyle**。设计 `designs/FIX-recursive-directories-design.md`。**e2e 待用户自跑：hive_config_test tags 2/21（应各返 6 行）；tag1 仍是 ENV。** > -> **⭐ 剩余 2 fe-core/SPI 大改（均已确认是 设计周期 工作，非一行改；每条 设计→(红队)→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **用户已裁决(A)**： -> - query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 -> - default_partition 连接器传 null 标记：SPI 新字段让连接器按分区值传 null 语义，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem:310`(现无条件 `PartitionValue(v,false)`) 据此建项——hive 标 null、paimon 保持非空(零 paimon 影响，别在 fe-core if(hive))。 +> **✅ default_partition DONE（本 session，code `58f3e367ed6`）**——翻闸 hive/paimon 表在**非字符串分区列**(INT/DATE)上遇 `__HIVE_DEFAULT_PARTITION__` 时静默丢分区(桥接层硬编码 `isNull=false`→`IntLiteral(哨兵)` 抛→per-partition 吞→整表误判 UNPARTITIONED、`partition=0/0`、`WHERE col IS NULL` 空)。修=**SPI 加位置对齐的 `List partitionValueNullFlags`**(`ConnectorPartitionInfo`，默认空=未 opt-in=非 null，hudi/iceberg/maxcompute 零改)，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem` zip 成 `PartitionValue(value, isNull)`→typed `NullLiteral`(不解析)。**fe-core 零哨兵比较(铁律)**：hive/paimon 渲染相同字符串却语义相反,故 nullness 连接器侧上报。hive 用 `HIVE_DEFAULT_PARTITION.equals`(旧 `HiveExternalMetaCache:309` 逐字节)、按 `HiveWriteUtils.toPartitionValues` 位置构建。**用户裁决 variant B**：paimon 也改真 NULL 语义(其 `partition.default-name` 分区→NullLiteral,`col IS NULL` 裁剪命中、MTMV 刷新物化 null 行;原是 `col IN(哨兵)` 丢 null 行)——实现连接器既有意图、分区名标识不变。加 fail-loud 的 nullFlags 元数校验;`PluginDrivenScanNode` prune 短路 opt-out 只改注释(代码不变,`col IS NULL` 示例对 paimon 已不适用)。设计红队 `wf_1cdaf44e-325`(5 lens 全 SOUND_WITH_FIXES,6 findings 全折入,其中 major=**`test_paimon_mtmv` 是受影响的既有套件**,非新测)。UT 绿:api 6 / fe-core 67 / hive 9 / paimon 10,0 fail、0 checkstyle。设计 `designs/FIX-default-partition-design.md`。**e2e 待用户自跑（含 golden 再生）**：`test_hive_default_partition`(INT+DATE) · paimon `qt_null_partition_*` · **`test_paimon_mtmv`（golden `null_partition` 3→5 行,须按 e2e 实跑再生;并验 MV 自动建 `VALUES IN (NULL)` 分区不报错——region 列可空应放行）**。 +> +> **⭐ 剩余 1 fe-core/SPI 大改（设计周期工作；设计→红队→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： +> - **用户已裁决(A)**：query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 > - **ENV(告知用户，非代码)**：重置外部 hive docker → test_hive_lzo_text_format(run86.hql 未应用)、test_hive_varbinary_type(残留行翻倍)、hive_config_test tag1(HDFS OUTFILE 累积) 一并解决。 +> - **登记的独立跟进项(非本修,均已裁决延后)**：iceberg identity/bucket/truncate 分区真 NULL 渲染成 `col=null`、非字符串列同样丢分区(可用本 SPI 机制 opt-in) · paimon `partition_values()` TVF/`$partitions` 对非字符串 default-name 分区 `Integer.valueOf(默认名)` 抛(pre-existing,与扫描/裁剪面独立)。 **✅ 本轮 DONE（4 commit，均编译+全套测试绿）**： - `cd4eecac5e6` **[fix] 文本读取根因（~12 suite）**：`HiveTextProperties`（fe-connector-hive）未做 `getByte` 数值→字节解码。Hive 默认分隔符存 `serialization.format="1"`=**字节 0x01**(Ctrl-A)，新代码当字符 `'1'`(0x31) 发 BE → `format_v2` TextReader 按 '1' 切 → 整行挤第一列 → null/粘连。移植 getByte + 老默认 + Hive2 拼写 key `colelction.delim` + table-params 优先；MultiDelimitSerDe 多字符保持 raw；null_format 不解码。新单测 `HiveTextPropertiesTest` 9/9。覆盖 to_date/prepare_data/basic_type/other/serde_prop/truncate/get_schema/meta_cache/query_cache + text_complex_type/to_array/text_garbled_file(靠 colelction.delim)。 diff --git a/plan-doc/tasks/designs/FIX-default-partition-design.md b/plan-doc/tasks/designs/FIX-default-partition-design.md new file mode 100644 index 00000000000000..6f917355e381b6 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-default-partition-design.md @@ -0,0 +1,279 @@ +# FIX: `__HIVE_DEFAULT_PARTITION__` on a non-string partition column drops the partition + +Status: **DONE** (code `58f3e367ed6`) · Branch `catalog-spi-11-hive` · Verified against HEAD `e2fdd65b954` +Result: implemented per §7; UT green — fe-connector-api 6, fe-core 67 (PluginDrivenMvcc + PartitionTest), +fe-connector-hive 9, fe-connector-paimon 10 (install); 0 failures, 0 checkstyle across all four modules. +**e2e still owed by the user** (see §7 step 11 + §8): `test_hive_default_partition` (INT+DATE) · paimon +`qt_null_partition_*` · `test_paimon_mtmv` (golden 3→5 rows regenerate + MV null-partition-creation gate). +Decision: **Approach A + user-signed variant B** (2026-07-12) — connector-supplied per-value NULL flag via SPI, +applied to **both hive AND paimon** (paimon adopts genuine-NULL semantics, not just hive). + +Recon+design workflow: `wf_c9f00bc2-470` (5 HEAD-verified recon angles + synthesis). +Design red-team: `wf_1cdaf44e-325` (5 adversarial lenses, all **SOUND_WITH_FIXES**, 0 blocker/UNSOUND). Findings +RT-F1..RT-F6 folded inline below (fail-loud arity, exact-sentinel hive detection, positional hive flag build, +paimon MTMV golden as impacted suite, RED-baseline wording, paimon TVF scoped out). + +--- + +## 1. Problem + +A partitioned external table whose rows land in the genuine-NULL partition — rendered as the literal +partition name `col=__HIVE_DEFAULT_PARTITION__` — is silently mis-planned when the partition column is +**non-string** (INT, DATE, …). The default partition is dropped from the in-memory partition universe; if +it is the only partition the table reports `UNPARTITIONED` and the scan explain shows `partition=0/0`, and +`WHERE col IS NULL` returns nothing. + +- **hive**: the reported regression (`test_hive_default_partition`, INT partition column). +- **paimon**: the *same* latent bug on a non-string (e.g. DATE) default partition — paimon renders the + sentinel into the name and fe-core log-skips it today (test fixture + `PaimonConnectorMetadataPartitionTest.nullDatePartitionRendersSentinelInsteadOfCrashing`). Additionally, + even on a string column paimon's genuine-null partition is currently a non-null `StringLiteral`, so MTMV + refresh emits `col IN ('__HIVE_DEFAULT_PARTITION__')` and **silently drops the null rows from the MV**. + +Legacy `HiveExternalMetaCache:309` handled hive correctly (`isNull = HIVE_DEFAULT_PARTITION.equals(value)`). +The SPI migration copied the *paimon* builder verbatim into shared fe-core, hardcoding `isNull=false`, +regressing hive and freezing paimon into the drop/`col IN` behavior. + +## 2. Root Cause + +`fe/fe-core/.../PluginDrivenMvccExternalTable.java:299-311` — the shared MVCC live/pruning partition-item +builder marks every value non-null: + +```java +for (String partitionValue : partitionValues) { // values re-parsed positionally from the NAME (L296) + values.add(new PartitionValue(partitionValue, false)); // L310 isNull hardcoded false +} +PartitionKey key = PartitionKey.createListPartitionKeyWithTypes(values, types, true); // L312 +``` + +Throw chain (verified): +1. `PartitionKey.createListPartitionKeyWithTypes` (`catalog/PartitionKey.java:189-202`): `isNullPartition()==false` + and type ≠ TIMESTAMPTZ → L201 `values.get(i).getValue(INT)`. +2. `PartitionValue.getValue(Type)` (`analysis/PartitionValue.java:48-56`): not null → + `LiteralExprUtils.createLiteral("__HIVE_DEFAULT_PARTITION__", INT)`. +3. → `new IntLiteral("__HIVE_DEFAULT_PARTITION__", INT)` (`fe-catalog/.../IntLiteral.java:65-72`) → + `Long.parseLong(...)` throws `NumberFormatException` → wrapped `AnalysisException("Invalid number format: …")`. +4. Caught **per-partition** in `listLatestPartitions` (`PluginDrivenMvccExternalTable.java:272-280`, + log-and-skip). The name already entered `nameToLastModifiedMillis` (L271) but the item map did not. +5. Size mismatch → `PluginDrivenMvccSnapshot.isPartitionInvalid()` (`nameToLastModifiedMillis.size() != + nameToPartitionItem.size()`) → `getPartitionType` returns `UNPARTITIONED` → `partition=0/0`. + +The isNull branch (`PartitionKey.java:190-191`) instead produces a typed `NullLiteral.create(type)` and +never parses the sentinel string — INT/DATE-safe. + +**Why fe-core cannot re-add the string compare (iron rule):** paimon (`PaimonConnectorMetadata.java:1057`) +also renders the *identical* string `__HIVE_DEFAULT_PARTITION__` into its name. fe-core cannot decide +nullness from the string; it must be connector-supplied. (Under variant B both connectors want the same +`isNull=true` answer, but they still detect their own null representation — hive's HMS sentinel vs paimon's +`partition.default-name` option — so the detection must stay connector-side.) + +## 3. Design — connector-supplied per-value NULL flag (positional `List`) + +### 3.1 SPI shape — positional `List` (reject name-keyed map, reject Java-null-in-map) + +The fe-core seam derives its values by **re-parsing the rendered partition name positionally** +(`HiveUtil.toPartitionValues(partitionName)`, L296) — it does not consume `getPartitionValues()`. So the +flag carrier must align to that same positional order. + +- **Reject Option B (map/set keyed by column name):** every existing `partitionValues` map is keyed by + **remote** names, while fe-core would key by the local Doris name → breaks under name-mapping / casing + divergence. +- **Reject Option C (Java-null value in the `partitionValues` map):** forces fe-core to abandon the + name-reparse seam; breaks paimon byte-parity (paimon's map is the RAW un-rendered spec while the NAME + carries rendered dates) and perturbs the two other map consumers (`PluginDrivenExternalTable.java:814,868`). +- **Accept Option A (positional `List`):** zips index-for-index at the seam; casing/order-immune + (both sides derive order from the identical positional name parse — hive's `HiveWriteUtils.toPartitionValues` + is a byte-faithful port of fe-core `HiveUtil.toPartitionValues`; paimon builds the flag in the same + `spec.entrySet()` loop that builds the name). Empty ⇒ all-false ⇒ zero change for connectors that don't + opt in and for both existing ctors. Mirrors the already-shipped precedent + `ConnectorPartitionValues.Normalized{List values, List isNull}` + (`fe-connector-api/.../scan/ConnectorPartitionValues.java:32-44`). + +### 3.2 Exact changes + +**(a) SPI — `fe-connector-api/.../ConnectorPartitionInfo.java`** — add one field + one accessor + one new +ctor; thread through `equals`/`hashCode`/`toString`: +```java +private final List partitionValueNullFlags; // positional, aligned to getPartitionName() parse order + +// new 8-arg ctor: +public ConnectorPartitionInfo(String partitionName, Map partitionValues, + Map properties, long rowCount, long sizeBytes, long lastModifiedMillis, + long fileCount, List partitionValueNullFlags) { ... } + +public List getPartitionValueNullFlags() { return partitionValueNullFlags; } +``` +- Null arg ⇒ `Collections.emptyList()`; else `Collections.unmodifiableList(new ArrayList<>(...))`. Class stays `final`. +- 7-arg ctor delegates to the new 8-arg with `emptyList()`; 3-arg already delegates to 7-arg. Both existing + ctors keep working unchanged. +- Add the field to `equals`/`hashCode`/`toString` (value-based contract; `ConnectorPartitionInfoTest` enforces it). + +**(b) fe-core — `PluginDrivenMvccExternalTable.java`** — thread the flag through `toListPartitionItem`; the +lone caller is `listLatestPartitions` L276: +```java +// L276: +nameToPartitionItem.put(partitionName, toListPartitionItem(partitionName, types, part.getPartitionValueNullFlags())); + +// builder: +private static ListPartitionItem toListPartitionItem(String partitionName, List types, + List nullFlags) throws AnalysisException { + List partitionValues = HiveUtil.toPartitionValues(partitionName); + Preconditions.checkState(partitionValues.size() == types.size(), partitionName + " vs. " + types); + // Fail loud (RT-F1): a connector that supplies flags MUST supply one per value; a short list would + // silently default the tail to isNull=false and re-introduce the drop bug. Empty = non-opted-in = OK. + Preconditions.checkState(nullFlags.isEmpty() || nullFlags.size() == types.size(), + "nullFlags " + nullFlags + " vs. " + types); + List values = Lists.newArrayListWithExpectedSize(types.size()); + for (int i = 0; i < partitionValues.size(); i++) { + boolean isNull = i < nullFlags.size() && nullFlags.get(i); // empty ⇒ false ⇒ non-opted-in connectors unchanged + values.add(new PartitionValue(partitionValues.get(i), isNull)); + } + PartitionKey key = PartitionKey.createListPartitionKeyWithTypes(values, types, true); + return new ListPartitionItem(Lists.newArrayList(key)); +} +``` +- Rewrite the L300-309 comment: it currently rationalizes unconditional `isNull=false` for paimon. Under B it + must describe the connector-supplied flag: a genuine-NULL value (hive's HMS sentinel, paimon's + `partition.default-name`) is marked `isNull=true` by its connector → typed `NullLiteral` → `col IS NULL` + selects it and MTMV refresh materializes the null rows; a connector that supplies no flag defaults to non-null. + +**(c) hive connector — `HiveConnectorMetadata.java` (`listPartitions` ~L1105)** — build the positional flag +list by **iterating `HiveWriteUtils.toPartitionValues(partitionName)` directly** (RT-F5: the same positional +parse fe-core re-runs, so flag[i] zips to value[i] regardless of column casing/order — do NOT iterate the +value Map/partKeyNames). Detection uses an **exact** `ConnectorPartitionValues.HIVE_DEFAULT_PARTITION.equals(value)` +compare, **not** `isNullPartitionValue` (RT-F2: byte-parity with legacy `HiveExternalMetaCache:309`, which +marks null on the sentinel only — `isNullPartitionValue` also widens to `\N`/Java-null, an unwanted parity +divergence; HMS partition names never carry `\N` anyway). Pass the 8-arg ctor. The sentinel constant already +lives connector-side (`ConnectorPartitionValues.HIVE_DEFAULT_PARTITION`, already a hive dependency). + +**(d) paimon connector — `PaimonConnectorMetadata.java` (`collectPartitions` ~L1044-1080)** — build a +positional flag list in the **same `spec.entrySet()` loop** that builds the name; set `true` at the existing +null branch L1050 (`defaultPartitionName.equals(value)`) — paimon detects its own default-name, NOT the hive +sentinel. Pass the flag list to the ctor. **Keep the name normalization to `__HIVE_DEFAULT_PARTITION__` +(L1057) unchanged** — the partition NAME identity must not change (avoids re-keying existing partitions / +MTMV names). Update the L1050-1057 comment: the intent it describes ("bridge marks the partition isNull") is +now *realized* via the supplied flag. + +**(e) fe-core — `PluginDrivenScanNode.java` opt-out comment (L238-280)** — comment-only. The +`ignorePartitionPruneShortCircuit` opt-out **code + capability stay** (still required for genuine predicate +prune-to-zero, e.g. paimon `col = `). But its worked example — "with `isNull=false` a +genuine-null partition renders as a non-null sentinel, so `col IS NULL` prunes every partition away" (L244-246, +L272-275) — is **stale under B**: paimon's null partition is now a real `NullLiteral`, so `col IS NULL` +prunes *accurately* to it (non-empty selection → opt-out at L276 does not fire; planScan still re-plans). +Replace the `col IS NULL` example with a still-valid one (`col = ` → genuine prune-to-zero). + +**(f) other connectors** — hudi (`HudiConnectorMetadata.java:709`), maxcompute +(`MaxComputeConnectorMetadata.java:274`), iceberg (`IcebergPartitionUtils.java:545`): keep their existing +ctors → flags default empty → `isNull=false`. **Zero change.** + +### 3.3 Out of scope (noted follow-ups) + +- **iceberg latent analogous bug**: identity/bucket/truncate LIST branch renders a genuine Java-null as the + literal text `col=null` (`IcebergPartitionUtils.java:620`); on a non-string column fe-core throws and drops + the partition — the same failure mode. The flag mechanism would let iceberg opt in at + `IcebergPartitionUtils.java:542-543` (typed Java-null already in hand), but iceberg is not in any failing + suite and this touches signed-off P6 code. **Deferred** to a separate task (user-agreed). + +## 4. Iron-Rule Compliance + +fe-core has **no** source-specific code after the fix: `toListPartitionItem` reads a connector-supplied +`List` and calls `new PartitionValue(value, isNull)`. No `if (hive/paimon)`, no `instanceof HMS*`, +no `HIVE_DEFAULT_PARTITION.equals(...)` in fe-core. The sentinel/default-name compare lives entirely in each +connector. The new field is a generic per-value nullness carrier, semantically identical to the existing +`ConnectorPartitionValues.Normalized.isNull` list already in the API module. (Pre-existing fe-core sentinel +compares at `TablePartitionValues.java:162` and the non-MVCC base path are **not reached by hive/paimon** — +overridden by the MVCC subclass — and are left untouched.) + +## 5. Parity & Behavior Change + +| | before | after (B) | +|---|---|---| +| **hive** INT/DATE null partition | dropped → `partition=0/0`, `col IS NULL` empty | `NullLiteral` partition; `col IS NULL` returns null rows; count correct (legacy `HiveExternalMetaCache:309` parity restored) | +| **hive** string null partition | `StringLiteral` sentinel | `NullLiteral` (legacy parity) | +| **paimon** string null partition | `StringLiteral` sentinel; MTMV `col IN (sentinel)` **drops null rows** | `NullLiteral`; MTMV `col IS NULL` **materializes null rows** (realizes the connector's stated intent) | +| **paimon** DATE/INT null partition | dropped (log-skip) | `NullLiteral` partition (fixed) | +| **paimon** `col IS NULL` query | prune-away → opt-out scan-all → planScan re-plan | prune *accurately* to null partition → planScan re-plan (same rows, more precise) | +| **paimon** `col = ` query | opt-out scan-all → planScan → 0 rows | unchanged (opt-out still fires) | +| hudi / maxcompute / iceberg | — | unchanged (default-false) | + +`test_hive_default_partition` (hive) and paimon null-partition regressions (`qt_null_partition_*`) are the +e2e gates. `qt_null_partition_4` (paimon `col IS NULL`) returns the same rows via the accurate-prune path; +must be re-run (user). + +## 6. Risk Analysis / Blast Radius + +- **paimon behavior change** (the wider-scope part of B): its genuine-null partition flips from + `StringLiteral` to `NullLiteral`. Downstream touch points to verify in red-team: (1) MTMV refresh + (`UpdateMvByPartitionCommand.java:183-202` — NullLiteral → `col IS NULL`, StringLiteral → `col IN`); + (2) the `PluginDrivenScanNode` prune opt-out (still needed for `col=`, no longer needed for + `col IS NULL`); (3) partition NAME identity unchanged (name normalization kept); (4) partition count + display now counts the paimon null partition (was: string partition, also counted — no `0/0` regression + since paimon string sentinel never threw). +- **`ConnectorPartitionInfo.equals/hashCode/toString`**: field added to all three; existing equal instances + stay equal (both empty lists). +- **ctor compatibility**: both existing ctors preserved; only hive + paimon use the new 8-arg ctor. All other + construction sites and tests compile unchanged. +- **Non-MVCC base path / SHOW PARTITIONS**: untouched — consume the value map or name only, never + `toListPartitionItem`. +- **`partition_values()` TVF / `$partitions`** (RT-F6): the flag is **not** threaded here (separate surface — + `getNameToPartitionValues` → `MetadataGenerator.partitionValuesRows`, string-compares the raw value against + the hive sentinel at `:2166`). **hive** is unaffected (its value map carries `__HIVE_DEFAULT_PARTITION__`, + matched at `:2166`). **paimon** is a **pre-existing** gap (NOT introduced or worsened by this fix): its value + map carries the RAW `partition.default-name` (e.g. `__DEFAULT_PARTITION__`), which `:2166` does not match, so + a non-string paimon default-partition column throws `Integer.valueOf("__DEFAULT_PARTITION__")` at `:2177`. The + §5 correction: this fix restores the LIST/scan/prune surface only; the paimon TVF surface is **out of scope** + (deferred follow-up, same iron-rule reason — fe-core can't string-detect paimon's default-name). +- **Impacted EXISTING e2e suite — `mtmv_p0/test_paimon_mtmv.groovy`** (RT-F3, MAJOR): it builds an MTMV over the + paimon `null_partition` table (`partition by(region)`) with a `// Will lose null data` comment and a committed + golden (`test_paimon_mtmv.out:138-141`) that deliberately **omits** the two genuine-NULL `region` rows. Under + B the null partition becomes a `NullLiteral` → MTMV refresh emits `region IS NULL` → the MV **gains** those + rows → golden flips 3→5 rows. This is an **expected** behavior change, not a new test: (1) update the stale + `// Will lose null data` comment; (2) the `.out` golden must be **regenerated by the e2e run** (do NOT + hand-write — `勿 rebless data-wrong`); (3) **e2e gate**: confirm the MV auto-creates the null list partition + (`VALUES IN (NULL)`) without throwing — the `region` column is nullable (base STRING, no NOT NULL) so it + should be allowed, but a NOT-NULL partition column would be rejected (`test_null_partition.groovy:44`); if it + throws, B breaks paimon MTMV creation and must be gated. + +## 7. Implementation Plan (ordered) + +1. `fe-connector-api/.../ConnectorPartitionInfo.java` — field + 8-arg ctor + getter + equals/hashCode/toString; 7-arg delegates with `emptyList()`. +2. `fe-connector-api/.../ConnectorPartitionInfoTest.java` — ctor/getter, back-compat-default (3-arg & 7-arg → empty), equals/hashCode-with-flags. +3. `fe-core/.../PluginDrivenMvccExternalTable.java` — `toListPartitionItem` gains `List nullFlags`, empty-safe consume, caller L276; rewrite L300-309 comment. +4. `fe-core/.../PluginDrivenScanNode.java` — update the stale `col IS NULL` example in the opt-out comment (code unchanged). +5. `fe-core/.../PluginDrivenMvccExternalTableTest.java` — `cpiNull` helper + RED tests (INT/DATE sentinel+flag → LIST/NullLiteral); keep no-flag default test green; flag-absent-INT-still-drops guard. +6. `fe-connector-hive/.../HiveConnectorMetadata.java` — build positional flag via `ConnectorPartitionValues.isNullPartitionValue`, 8-arg ctor. +7. `fe-connector-hive/.../HiveConnectorMetadataPartitionListTest.java` — sentinel→`[true,false]` + ordinary→all-false. +8. `fe-connector-paimon/.../PaimonConnectorMetadata.java` — build flag in the `spec.entrySet()` loop, set true at L1050; update L1050-1057 comment. +9. `fe-connector-paimon/.../PaimonConnectorMetadataPartitionTest.java` — null partition now flags `true` (behavior change assertion). +10. `regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy` — replace the stale `// Will lose null data` comment (RT-F3); the `.out` golden is regenerated by the e2e run, NOT hand-edited here. +11. e2e (user): `test_hive_default_partition` (INT+DATE) + paimon `qt_null_partition_*` + `test_paimon_mtmv` (golden 3→5 rows + MV null-partition-creation gate). + +## 8. Test Plan + +Constraints (verified): fe-core tests use Mockito (mock `ConnectorMetadata`); connector tests +(`ConnectorPartitionInfoTest`, `HiveConnectorMetadataPartitionListTest`, `PaimonConnectorMetadataPartitionTest`) +use hand-written recording fakes — **no Mockito**; extend the fakes. Checkstyle scans test sources: no static +imports, `AvoidStarImport`, `CustomImportOrder`. + +**Unit (RED before fix, GREEN after):** RED baseline caveat (RT-F4) — the flag-dependent tests reference the +new 8-arg ctor/accessor, so they cannot compile against a full revert; their RED baseline is the *staged +mutation* (SPI + tests applied, fe-core seam still hardcoding `false`). The compile-independent opt-in guard is +`testDefaultSentinelWithoutFlagStillDrops` (no-flag INT → UNPARTITIONED), which is RED against unmodified fe-core. +- fe-core `PluginDrivenMvccExternalTableTest` (`cpiNull(name, lastModified, boolean… flags)`): + - `testHiveDefaultSentinelOnIntColumnBuildsGenuineNullPartition` (Type.INT, flag `[true]`): `getPartitionType==LIST`, `getPartitionColumns` non-empty, `getNameToPartitionItems().size()==1`, single key `isNullLiteral()==true`. RED pre-fix (UNPARTITIONED/0). + - `testHiveDefaultSentinelOnDateColumnBuildsGenuineNullPartition` (Type.DATEV2, flag `[true]`): same shape. + - `testHiveDefaultSentinelBuildsNonNullStringKey` (existing, VARCHAR, **no flag**): keep GREEN — the no-flag default (non-opted-in connector) stays non-null. Re-comment: this is the default path, not "paimon parity". + - `testDefaultSentinelWithoutFlagStillDrops` (Type.INT, no flag): pre-fix behavior preserved (UNPARTITIONED) — locks the fix as opt-in. +- fe-connector-api `ConnectorPartitionInfoTest`: `ctorCarriesPerValueNullFlags`, `backwardCompatCtorsDefaultNullFlagsEmpty`, `equalsAndHashCodeIncludeNullFlags`. +- fe-connector-hive `HiveConnectorMetadataPartitionListTest` (extend `FakeHmsClient`): `listPartitionsMarksHiveDefaultSentinelNull` (`year=__HIVE_DEFAULT_PARTITION__/month=01` → flags `[true,false]`, stats still UNKNOWN); `listPartitionsMarksNoNullForOrdinaryValues` (all false). +- fe-connector-paimon `PaimonConnectorMetadataPartitionTest` (extend `RecordingPaimonCatalogOps`): `nullPartitionFlagsTrue` (`category=__DEFAULT_PARTITION__` → emitted flag `true`, name still normalized to the sentinel) — the paimon behavior-change assertion. + +**e2e (user):** `test_hive_default_partition` (INT+DATE default partition → partitioned, `col IS NULL` +returns null rows, counts match; VARCHAR control) + paimon `qt_null_partition_*` (esp. `_4`, `col IS NULL` +unchanged rows) + a paimon MTMV-over-null-partition check (now materializes null rows). + +## Open Design Decisions + +None. SPI shape (positional `List`), fe-core seam, and both connectors' opt-in points are agreed +across all five recon angles; the paimon-semantics fork was resolved by the user to **variant B** (paimon +adopts genuine-NULL). The iceberg latent bug is a deferred follow-up. From c9a863379063fd214984a71efa2c33907c0403cb Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 09:16:43 +0800 Subject: [PATCH 279/334] [fix](catalog) query-cache: recognize SPI plugin tables via MTMVRelatedTableIf capability MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit After the hms cutover, a flipped hive table is a PluginDrivenMvccExternalTable scanned by a PluginDrivenScanNode, not an HMSExternalTable/HiveScanNode. The Nereids SQL result cache gated eligibility, the invalidation token, and the scan-node recognition on those retired class names, so a flipped hive table silently lost result-cache eligibility. Replace the six source-name branches across four files with the connector-agnostic MTMVRelatedTableIf capability + its stable, data-tied token getNewestUpdateVersionOrTime() (hive: max transient_lastDdlTime; iceberg/paimon: monotonic snapshot version), obeying the fe-core iron rule (no instanceof HMSExternalTable / HiveScanNode). Scope = all lakehouse plugin tables that expose a valid token (hive/iceberg/paimon/hudi), gated by the existing enable_hive_sql_cache switch (default false) so behavior is unchanged unless opted in. - BindRelation: admit any ExternalTable that is MTMVRelatedTableIf. - SqlCacheContext.addUsedTable: capture getNewestUpdateVersionOrTime() as the version token; token <= 0 fails safe (mark unsupported) rather than pin a bogus constant that would serve stale results. - NereidsSqlCacheManager: admit PLUGIN_EXTERNAL_TABLE in the type gate; re-check freshness and the partition-existence skip via MTMVRelatedTableIf. - CacheAnalyzer: recognize an external cacheable scan node by its TARGET TABLE's MTMVRelatedTableIf capability (not the scan-node class) so hudi (HudiScanNode extends HiveScanNode) is not dropped and jdbc-query TVFs (FunctionGenTable) are excluded; source the freshness marker from the connector token. The dead source- specific COUNTER_QUERY_HIVE_TABLE bump is removed. The default (unstable) ExternalTable.getUpdateTime() = System.currentTimeMillis() at schema load would serve stale results within the schema-cache TTL; the connector token is data-tied. Tests (fe-core, all RED on the pre-cutover HEAD): PluginTableCacheAnalyzerTest (4), SqlCacheContextPluginTableTest (2), NereidsSqlCacheManagerPluginTableTest (2) — capability recognition, token capture + <=0 fail-safe, and invalidate-on-token-change. fe-core BUILD SUCCESS, 0 checkstyle. Design + red-team: plan-doc/tasks/designs/FIX-querycache-spi-design.md. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../common/cache/NereidsSqlCacheManager.java | 16 ++- .../apache/doris/nereids/SqlCacheContext.java | 16 ++- .../nereids/rules/analysis/BindRelation.java | 7 +- .../apache/doris/qe/cache/CacheAnalyzer.java | 40 ++++-- ...NereidsSqlCacheManagerPluginTableTest.java | 118 ++++++++++++++++ .../SqlCacheContextPluginTableTest.java | 91 ++++++++++++ .../cache/PluginTableCacheAnalyzerTest.java | 133 ++++++++++++++++++ 7 files changed, 396 insertions(+), 25 deletions(-) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java b/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java index fe8f01b7254a3f..f965bbf0e38d88 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/cache/NereidsSqlCacheManager.java @@ -33,7 +33,7 @@ import org.apache.doris.common.util.DebugUtil; import org.apache.doris.datasource.CatalogIf; import org.apache.doris.datasource.CatalogMgr; -import org.apache.doris.datasource.hive.HMSExternalTable; +import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.mysql.privilege.DataMaskPolicy; import org.apache.doris.mysql.privilege.RowFilterPolicy; import org.apache.doris.nereids.CascadesContext; @@ -439,7 +439,8 @@ private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext) for (Entry scanTable : sqlCacheContext.getUsedTables().entrySet()) { TableVersion tableVersion = scanTable.getValue(); if (tableVersion.type != TableType.OLAP && tableVersion.type != TableType.MATERIALIZED_VIEW - && tableVersion.type != TableType.HMS_EXTERNAL_TABLE) { + && tableVersion.type != TableType.HMS_EXTERNAL_TABLE + && tableVersion.type != TableType.PLUGIN_EXTERNAL_TABLE) { return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } TableIf tableIf = findTableIf(env, scanTable.getKey()); @@ -472,9 +473,10 @@ private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext) return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } } - } else if (tableIf instanceof HMSExternalTable) { - HMSExternalTable hiveTable = (HMSExternalTable) tableIf; - if (tableVersion.version != hiveTable.getUpdateTime()) { + } else if (tableIf instanceof MTMVRelatedTableIf) { + // External MVCC table (flipped hive/iceberg/paimon/hudi): compare the stored data-version + // token against the live one. OlapTable/MTMV are matched by the OlapTable arm above. + if (tableVersion.version != ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime()) { return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } } else { @@ -503,7 +505,9 @@ private IsChanged tablesOrDataChanged(Env env, SqlCacheContext sqlCacheContext) return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } } - } else if (!(tableIf instanceof HMSExternalTable)) { + } else if (!(tableIf instanceof MTMVRelatedTableIf)) { + // External MVCC tables skip per-partition existence tracking (an Olap-only concern); + // their freshness is fully covered by the token compare in the used-tables loop above. return IsChanged.CHANGED_AND_INVALIDATE_CACHE; } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java index e532ce611fe8dc..aeb58b771c6bbf 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/SqlCacheContext.java @@ -26,7 +26,7 @@ import org.apache.doris.catalog.TableIf.TableType; import org.apache.doris.common.Pair; import org.apache.doris.datasource.CatalogIf; -import org.apache.doris.datasource.hive.HMSExternalTable; +import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.mysql.FieldInfo; import org.apache.doris.mysql.privilege.Auth; import org.apache.doris.mysql.privilege.DataMaskPolicy; @@ -193,9 +193,17 @@ public synchronized void addUsedTable(TableIf tableIf) { try { if (tableIf instanceof OlapTable) { version = ((OlapTable) tableIf).getVisibleVersion(); - } else if (tableIf instanceof HMSExternalTable) { - HMSExternalTable hmsExternalTable = (HMSExternalTable) tableIf; - version = hmsExternalTable.getUpdateTime(); + } else if (tableIf instanceof MTMVRelatedTableIf) { + // Connector-agnostic data-version token (hive: max transient_lastDdlTime; iceberg/paimon: + // monotonic snapshot version). OlapTable is handled above; MTMV is an OlapTable too, so this + // arm only catches external MVCC tables (flipped hive/iceberg/paimon/hudi). + version = ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime(); + if (version <= 0) { + // No reliable data-change signal (empty partition set / dropped): fail-safe, do not + // pin a bogus constant that would serve stale results. + setHasUnsupportedTables(true); + return; + } } } catch (Throwable e) { // in cloud, getVisibleVersion throw exception, disable sql cache temporary diff --git a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java index e31dc2ae72de22..3343eaee2b4a6e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java +++ b/fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/BindRelation.java @@ -51,6 +51,7 @@ import org.apache.doris.datasource.hive.HMSExternalTable; import org.apache.doris.datasource.hive.HMSExternalTable.DLAType; import org.apache.doris.datasource.systable.SysTableResolver; +import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.nereids.CTEContext; import org.apache.doris.nereids.CascadesContext; import org.apache.doris.nereids.SqlCacheContext; @@ -884,8 +885,12 @@ private LogicalPlan getLogicalPlan(TableIf table, UnboundRelation unboundRelatio sqlCacheContext.setHasUnsupportedTables(true); } else if (table instanceof OlapTable) { sqlCacheContext.addUsedTable(table); - } else if (table instanceof HMSExternalTable + } else if (table instanceof ExternalTable && table instanceof MTMVRelatedTableIf && cascadesContext.getConnectContext().getSessionVariable().enableHiveSqlCache) { + // Any external lakehouse plugin table that exposes a stable data-version token + // (MTMVRelatedTableIf#getNewestUpdateVersionOrTime) is cacheable; addUsedTable + // records the token and fails safe if it is unavailable. Gated by the (default + // false) enable_hive_sql_cache switch so behavior is unchanged unless opted in. sqlCacheContext.addUsedTable(table); } else { sqlCacheContext.setHasUnsupportedTables(true); diff --git a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java index 1736aeeb1ad5a8..83ce3ef3f1497e 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java +++ b/fe/fe-core/src/main/java/org/apache/doris/qe/cache/CacheAnalyzer.java @@ -34,8 +34,8 @@ import org.apache.doris.common.UserException; import org.apache.doris.common.util.DebugUtil; import org.apache.doris.datasource.CatalogIf; -import org.apache.doris.datasource.hive.source.HiveScanNode; import org.apache.doris.metric.MetricRepo; +import org.apache.doris.mtmv.MTMVRelatedTableIf; import org.apache.doris.nereids.NereidsPlanner; import org.apache.doris.nereids.SqlCacheContext; import org.apache.doris.nereids.SqlCacheContext.FullTableName; @@ -301,24 +301,21 @@ private List buildCacheTableList() { // Check the last version time of the table MetricRepo.COUNTER_QUERY_TABLE.increase(1L); long olapScanNodeSize = 0; - long hiveScanNodeSize = 0; + long externalCacheableSize = 0; for (ScanNode scanNode : scanNodes) { if (scanNode instanceof OlapScanNode) { olapScanNodeSize++; - } else if (scanNode instanceof HiveScanNode) { - hiveScanNodeSize++; + } else if (isExternalCacheableScanNode(scanNode)) { + externalCacheableSize++; } } if (olapScanNodeSize > 0) { MetricRepo.COUNTER_QUERY_OLAP_TABLE.increase(1L); } - if (hiveScanNodeSize > 0) { - MetricRepo.COUNTER_QUERY_HIVE_TABLE.increase(1L); - } - if (!(olapScanNodeSize == scanNodes.size() || hiveScanNodeSize == scanNodes.size())) { + if (!(olapScanNodeSize == scanNodes.size() || externalCacheableSize == scanNodes.size())) { if (LOG.isDebugEnabled()) { - LOG.debug("only support olap/hive table with non-federated query, " + LOG.debug("only support olap/external table with non-federated query, " + "other types are not supported now, queryId {}", DebugUtil.printId(queryId)); } return Collections.emptyList(); @@ -329,7 +326,7 @@ private List buildCacheTableList() { ScanNode node = scanNodes.get(i); CacheTable cTable = node instanceof OlapScanNode ? buildCacheTableForOlapScanNode((OlapScanNode) node) - : buildCacheTableForHiveScanNode((HiveScanNode) node); + : buildCacheTableForExternalScanNode(node); tblTimeList.add(cTable); } Collections.sort(tblTimeList); @@ -469,12 +466,27 @@ private CacheTable buildCacheTableForOlapScanNode(OlapScanNode node) { return cacheTable; } - private CacheTable buildCacheTableForHiveScanNode(HiveScanNode node) { + /** + * A non-Olap scan node is cacheable iff its target table exposes a stable data-version token + * (implements {@link MTMVRelatedTableIf}). Keying on the target-table capability rather than the scan + * node class is connector-agnostic and robust: it recognizes every lakehouse plugin table (hive / + * iceberg / paimon / hudi, whether scanned via {@code PluginDrivenScanNode} or a legacy + * {@code HudiScanNode}) while excluding token-less nodes such as {@code jdbc_query(...)} TVFs (backed by + * a {@code FunctionGenTable}) and system-table scans. + */ + private boolean isExternalCacheableScanNode(ScanNode scanNode) { + return scanNode.getTupleDesc() != null + && scanNode.getTupleDesc().getTable() instanceof MTMVRelatedTableIf; + } + + private CacheTable buildCacheTableForExternalScanNode(ScanNode node) { CacheTable cacheTable = new CacheTable(); - cacheTable.table = node.getTargetTable(); + TableIf tableIf = node.getTupleDesc().getTable(); + cacheTable.table = tableIf; cacheTable.partitionNum = node.getSelectedPartitionNum(); - cacheTable.latestPartitionTime = cacheTable.table.getUpdateTime(); - TableIf tableIf = cacheTable.table; + // Connector-agnostic data-version token (hive: max transient_lastDdlTime; iceberg/paimon: monotonic + // snapshot version). Gated to MTMVRelatedTableIf tables by isExternalCacheableScanNode above. + cacheTable.latestPartitionTime = ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime(); DatabaseIf database = tableIf.getDatabase(); CatalogIf catalog = database.getCatalog(); ScanTable scanTable = new ScanTable(new FullTableName( diff --git a/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java new file mode 100644 index 00000000000000..da1a25a96c5072 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/common/cache/NereidsSqlCacheManagerPluginTableTest.java @@ -0,0 +1,118 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.common.cache; + +import org.apache.doris.analysis.UserIdentity; +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.Env; +import org.apache.doris.catalog.TableIf.TableType; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.CatalogMgr; +import org.apache.doris.datasource.PluginDrivenMvccExternalTable; +import org.apache.doris.nereids.SqlCacheContext; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Optional; + +/** + * Unit tests for the lookup-time freshness re-check the SQL-result-cache migration wired into + * {@link NereidsSqlCacheManager} for flipped lakehouse tables. The stored data-version token + * ({@code getNewestUpdateVersionOrTime()}) is compared against the live one: equal ⇒ NOT_CHANGED + * (cache may be served), different ⇒ CHANGED_AND_INVALIDATE_CACHE. This is the correctness core of the + * feature — a cache must invalidate exactly when the underlying data changes. RED on the pre-cutover HEAD, + * whose gate rejected {@code PLUGIN_EXTERNAL_TABLE} outright (always invalidate, never a hit) and re-checked + * only {@code instanceof HMSExternalTable}. + */ +public class NereidsSqlCacheManagerPluginTableTest { + + private static final String CTL = "hms_ctl"; + private static final String DB = "hms_db"; + private static final String TBL = "t"; + private static final long TABLE_ID = 42L; + + private PluginDrivenMvccExternalTable mockTable(long liveToken) { + PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); + DatabaseIf db = Mockito.mock(DatabaseIf.class); + CatalogIf catalog = Mockito.mock(CatalogIf.class); + Mockito.when(catalog.isInternalCatalog()).thenReturn(false); + Mockito.when(catalog.getName()).thenReturn(CTL); + Mockito.when(catalog.getProperties()).thenReturn(new java.util.HashMap<>()); + Mockito.when(db.getCatalog()).thenReturn(catalog); + Mockito.when(db.getFullName()).thenReturn(DB); + Mockito.when(table.getDatabase()).thenReturn(db); + Mockito.when(table.getId()).thenReturn(TABLE_ID); + Mockito.when(table.getName()).thenReturn(TBL); + Mockito.when(table.isTemporary()).thenReturn(false); + Mockito.when(table.getType()).thenReturn(TableType.PLUGIN_EXTERNAL_TABLE); + Mockito.when(table.getNewestUpdateVersionOrTime()).thenReturn(liveToken); + return table; + } + + /** Wires a mock Env so that findTableIf(env, {CTL,DB,TBL}) resolves to the given live table. */ + private Env mockEnvResolvingTo(PluginDrivenMvccExternalTable liveTable) { + Env env = Mockito.mock(Env.class); + CatalogMgr mgr = Mockito.mock(CatalogMgr.class); + CatalogIf catalog = Mockito.mock(CatalogIf.class); + DatabaseIf db = Mockito.mock(DatabaseIf.class); + Mockito.when(env.getCatalogMgr()).thenReturn(mgr); + Mockito.when(mgr.getCatalog(CTL)).thenReturn(catalog); + Mockito.doReturn(Optional.of(db)).when(catalog).getDb(DB); + Mockito.doReturn(Optional.of(liveTable)).when(db).getTable(TBL); + return env; + } + + private boolean isChangedField(Object verdict, String field) { + return Deencapsulation.getField(verdict, field); + } + + /** Same stored and live token ⇒ the cache is fresh (NOT_CHANGED). */ + @Test + public void testNotChangedWhenTokenUnchanged() { + long token = 1_700_000_000_000L; + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(mockTable(token)); // stores TableVersion(id, token, PLUGIN) + + Env env = mockEnvResolvingTo(mockTable(token)); // live token unchanged + Object verdict = Deencapsulation.invoke(new NereidsSqlCacheManager(), + "tablesOrDataChanged", env, context); + + Assertions.assertFalse(isChangedField(verdict, "changed"), + "an unchanged token must keep the cache (NOT_CHANGED)"); + } + + /** Live token advanced past the stored one ⇒ data changed ⇒ invalidate. */ + @Test + public void testInvalidateWhenTokenChanged() { + long storedToken = 1_700_000_000_000L; + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(mockTable(storedToken)); + + Env env = mockEnvResolvingTo(mockTable(storedToken + 1000L)); // data mutated: newer token + Object verdict = Deencapsulation.invoke(new NereidsSqlCacheManager(), + "tablesOrDataChanged", env, context); + + Assertions.assertTrue(isChangedField(verdict, "changed"), + "a changed token must invalidate the cache"); + Assertions.assertTrue(isChangedField(verdict, "invalidCache"), + "a data change must evict the stale entry, not just miss"); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java new file mode 100644 index 00000000000000..401fea63122fe4 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/nereids/SqlCacheContextPluginTableTest.java @@ -0,0 +1,91 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.nereids; + +import org.apache.doris.analysis.UserIdentity; +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.TableIf.TableType; +import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.PluginDrivenMvccExternalTable; +import org.apache.doris.nereids.SqlCacheContext.TableVersion; + +import com.google.common.collect.Maps; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +/** + * Unit tests for the connector-agnostic invalidation token the SQL-result-cache migration wired into + * {@link SqlCacheContext#addUsedTable}. A flipped lakehouse table is a + * {@code PluginDrivenMvccExternalTable} (implements MTMVRelatedTableIf); its cache token must be the stable, + * data-tied {@code getNewestUpdateVersionOrTime()}, NOT the wall-clock {@code getUpdateTime()} it inherits + * (which changes on every FE schema reload and would serve stale results). A token <= 0 (no reliable + * data-change signal) fails safe: the table is marked unsupported rather than pinned at a bogus constant. + */ +public class SqlCacheContextPluginTableTest { + + private PluginDrivenMvccExternalTable mockTable(long id, long token) { + PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); + DatabaseIf db = Mockito.mock(DatabaseIf.class); + CatalogIf catalog = Mockito.mock(CatalogIf.class); + Mockito.when(catalog.isInternalCatalog()).thenReturn(false); + Mockito.when(catalog.getName()).thenReturn("hms_ctl"); + Mockito.when(catalog.getProperties()).thenReturn(Maps.newHashMap()); + Mockito.when(db.getCatalog()).thenReturn(catalog); + Mockito.when(db.getFullName()).thenReturn("hms_db"); + Mockito.when(table.getDatabase()).thenReturn(db); + Mockito.when(table.getId()).thenReturn(id); + Mockito.when(table.getName()).thenReturn("t"); + Mockito.when(table.getType()).thenReturn(TableType.PLUGIN_EXTERNAL_TABLE); + Mockito.when(table.getNewestUpdateVersionOrTime()).thenReturn(token); + return table; + } + + /** + * A plugin table with a real data-version token is admitted, and the recorded TableVersion carries the + * connector token (not a wall-clock or a constant 0). RED on the pre-cutover HEAD, which only recorded a + * token for {@code instanceof HMSExternalTable} and left a flipped plugin table at version 0. + */ + @Test + public void testAddUsedTableCapturesConnectorToken() { + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + long token = 1_700_000_000_000L; + context.addUsedTable(mockTable(42L, token)); + + Assertions.assertFalse(context.hasUnsupportedTables()); + Assertions.assertEquals(1, context.getUsedTables().size()); + TableVersion recorded = context.getUsedTables().values().iterator().next(); + Assertions.assertEquals(42L, recorded.id); + Assertions.assertEquals(token, recorded.version); + Assertions.assertEquals(TableType.PLUGIN_EXTERNAL_TABLE, recorded.type); + } + + /** + * A non-positive token means the connector has no reliable data-change signal (empty partition set / + * dropped table). The table must be marked unsupported (fail safe) rather than cached against a bogus + * constant that could never invalidate. + */ + @Test + public void testAddUsedTableFailsSafeOnNonPositiveToken() { + SqlCacheContext context = new SqlCacheContext(UserIdentity.ROOT); + context.addUsedTable(mockTable(42L, 0L)); + + Assertions.assertTrue(context.hasUnsupportedTables()); + Assertions.assertTrue(context.getUsedTables().isEmpty()); + } +} diff --git a/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java b/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java new file mode 100644 index 00000000000000..bac74e96a11566 --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/qe/cache/PluginTableCacheAnalyzerTest.java @@ -0,0 +1,133 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.qe.cache; + +import org.apache.doris.analysis.TupleDescriptor; +import org.apache.doris.analysis.TupleId; +import org.apache.doris.catalog.DatabaseIf; +import org.apache.doris.catalog.FunctionGenTable; +import org.apache.doris.catalog.TableIf; +import org.apache.doris.common.jmockit.Deencapsulation; +import org.apache.doris.datasource.CatalogIf; +import org.apache.doris.datasource.PluginDrivenMvccExternalTable; +import org.apache.doris.datasource.PluginDrivenScanNode; +import org.apache.doris.qe.ConnectContext; +import org.apache.doris.qe.SessionVariable; + +import org.junit.Assert; +import org.junit.jupiter.api.BeforeEach; +import org.junit.jupiter.api.Test; +import org.mockito.Mockito; + +import java.util.Collections; + +/** + * Unit tests for the connector-agnostic scan-node recognition the SQL-result-cache migration added to + * {@link CacheAnalyzer} after the hms SPI cutover. A flipped lakehouse table is a + * {@code PluginDrivenMvccExternalTable} scanned by a {@link PluginDrivenScanNode}; the old gate matched + * {@code instanceof HiveScanNode}, which both missed the plugin node AND (via the class hierarchy) would + * wrongly include a jdbc-query TVF. The new gate keys on the TARGET TABLE's {@code MTMVRelatedTableIf} + * capability, so it recognizes any lakehouse plugin table and excludes token-less nodes. + * + *

    Tests the two new private members directly (via {@link Deencapsulation}) to avoid the full + * analyze/MetricRepo bootstrap — the same reason the legacy {@code HmsQueryCacheTest} needed a real + * catalog and is now disabled for the plugin path.

    + */ +public class PluginTableCacheAnalyzerTest { + + private CacheAnalyzer analyzer; + + @BeforeEach + public void setUp() { + ConnectContext ctx = Mockito.mock(ConnectContext.class); + Mockito.when(ctx.getSessionVariable()).thenReturn(new SessionVariable()); + // Unit-test constructor: scanNodes are supplied per-test via Deencapsulation.invoke on the helper. + analyzer = new CacheAnalyzer(ctx, null, Collections.emptyList()); + } + + private PluginDrivenScanNode mockPluginScanNode(TableIf table, long selectedPartitionNum) { + PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class); + TupleDescriptor desc = new TupleDescriptor(new TupleId(1)); + desc.setTable(table); + Mockito.when(node.getTupleDesc()).thenReturn(desc); + Mockito.when(node.getSelectedPartitionNum()).thenReturn(selectedPartitionNum); + return node; + } + + /** + * A flipped lakehouse plugin table (implements MTMVRelatedTableIf) IS recognized as cacheable. RED on + * the pre-cutover HEAD, whose gate was {@code instanceof HiveScanNode} and never matched the plugin node. + */ + @Test + public void testRecognizePluginTableByCapability() { + PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); + PluginDrivenScanNode node = mockPluginScanNode(table, 3L); + boolean recognized = Deencapsulation.invoke(analyzer, "isExternalCacheableScanNode", node); + Assert.assertTrue("a PluginDrivenMvccExternalTable scan must be cacheable", recognized); + } + + /** + * A jdbc-query TVF also emits a PluginDrivenScanNode, but its backing table is a FunctionGenTable with no + * data-version token — the capability gate must EXCLUDE it (a class-based {@code instanceof + * PluginDrivenScanNode} gate would have wrongly admitted it). + */ + @Test + public void testRejectTvfBackedNode() { + FunctionGenTable tvfTable = Mockito.mock(FunctionGenTable.class); + PluginDrivenScanNode node = mockPluginScanNode(tvfTable, 0L); + boolean recognized = Deencapsulation.invoke(analyzer, "isExternalCacheableScanNode", node); + Assert.assertFalse("a jdbc-query TVF (FunctionGenTable) has no token and must not be cacheable", + recognized); + } + + /** A scan node with no tuple descriptor is defensively excluded (no NPE). */ + @Test + public void testRejectNullTupleDesc() { + PluginDrivenScanNode node = Mockito.mock(PluginDrivenScanNode.class); + Mockito.when(node.getTupleDesc()).thenReturn(null); + boolean recognized = Deencapsulation.invoke(analyzer, "isExternalCacheableScanNode", node); + Assert.assertFalse(recognized); + } + + /** + * The cache-freshness marker (latestPartitionTime) is sourced from the connector's stable data-version + * token {@code getNewestUpdateVersionOrTime()}, NOT the wall-clock {@code getUpdateTime()} that a flipped + * plugin table would inherit (which would serve stale results). partitionNum comes from the scan node. + */ + @Test + public void testTokenSourcedFromConnectorFreshness() { + long token = 1_700_000_000_000L; + PluginDrivenMvccExternalTable table = Mockito.mock(PluginDrivenMvccExternalTable.class); + DatabaseIf db = Mockito.mock(DatabaseIf.class); + CatalogIf catalog = Mockito.mock(CatalogIf.class); + Mockito.when(catalog.getName()).thenReturn("hms_ctl"); + Mockito.when(db.getCatalog()).thenReturn(catalog); + Mockito.when(db.getFullName()).thenReturn("hms_db"); + Mockito.when(table.getDatabase()).thenReturn(db); + Mockito.when(table.getName()).thenReturn("t"); + Mockito.when(table.getNewestUpdateVersionOrTime()).thenReturn(token); + + PluginDrivenScanNode node = mockPluginScanNode(table, 5L); + CacheAnalyzer.CacheTable cacheTable = + Deencapsulation.invoke(analyzer, "buildCacheTableForExternalScanNode", node); + + Assert.assertSame(table, cacheTable.table); + Assert.assertEquals(token, cacheTable.latestPartitionTime); + Assert.assertEquals(5L, cacheTable.partitionNum); + } +} From bd1ab5a9465acb8ade5666a192356f6956767a63 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 09:18:03 +0800 Subject: [PATCH 280/334] [doc](catalog) query-cache: land SPI-migration design + red-team, mark DONE Design for migrating the Nereids SQL result cache off source-name branches onto the MTMVRelatedTableIf capability + getNewestUpdateVersionOrTime() token (all four loci, the token cost accounting, blast radius, and the as-implemented tests). Folds the adversarial red-team (wf_fe484bc8-43b): capability-based scan-node recognition (hudi / jdbc-TVF), honest token-cost paragraph + registered follow-up optimization, corrected sys-table safety rationale, concrete test seams. Marks query_cache DONE in HANDOFF and repoints the next step to HIVEFS-8 / reverify backlog. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 7 +- .../designs/FIX-querycache-spi-design.md | 352 ++++++++++++++++++ 2 files changed, 355 insertions(+), 4 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-querycache-spi-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 8bee7e344e5dc4..b1d86fc6cad1a0 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,7 @@ --- -# 🆕 下一个 session 起步 = query_cache 移植到 SPI（hive e2e round-2 仅剩此 1 项代码 + ENV；default_partition 本轮已 DONE `58f3e367ed6`） +# 🆕 下一个 session 起步 = HIVEFS-8（全量 build + e2e 收尾）或复核修复 backlog（batch 3）；**hive e2e round-2 代码已全部 DONE**（query_cache 本轮 `c9a86337906`，仅剩 ENV + 用户自跑 e2e） > **本轮(round-2)**：用户按最新代码重跑 = **37 suite 测试、22 失败**。**全 22 已逐个 HEAD-verified 根因 + 对抗复核**（44-agent workflow `wf_c9def639-775`）。定案：**15 CODE_FIX · 1 TEST_ALIGN · 2 ENV · 4 USER_DECISION（用户已全裁决 → 均转 CODE_FIX）**。**完整三元表 + 逐 suite 修复指针 + 用户裁决** = `plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md`（**起步必读**，比下面的 round-1 R1-R12 表更新更准）。 > **起步必读**：本段 + triage 文档 + `git log`。失败日志（勿 commit）：`nohup.out`(Jul 11 20:27)、FE `output/fe/log`、BE `output/be/log`。**BE 走 `format_v2`**。**行号信 HEAD 不信文档。** @@ -23,10 +23,9 @@ > > **✅ default_partition DONE（本 session，code `58f3e367ed6`）**——翻闸 hive/paimon 表在**非字符串分区列**(INT/DATE)上遇 `__HIVE_DEFAULT_PARTITION__` 时静默丢分区(桥接层硬编码 `isNull=false`→`IntLiteral(哨兵)` 抛→per-partition 吞→整表误判 UNPARTITIONED、`partition=0/0`、`WHERE col IS NULL` 空)。修=**SPI 加位置对齐的 `List partitionValueNullFlags`**(`ConnectorPartitionInfo`，默认空=未 opt-in=非 null，hudi/iceberg/maxcompute 零改)，fe-core `PluginDrivenMvccExternalTable.toListPartitionItem` zip 成 `PartitionValue(value, isNull)`→typed `NullLiteral`(不解析)。**fe-core 零哨兵比较(铁律)**：hive/paimon 渲染相同字符串却语义相反,故 nullness 连接器侧上报。hive 用 `HIVE_DEFAULT_PARTITION.equals`(旧 `HiveExternalMetaCache:309` 逐字节)、按 `HiveWriteUtils.toPartitionValues` 位置构建。**用户裁决 variant B**：paimon 也改真 NULL 语义(其 `partition.default-name` 分区→NullLiteral,`col IS NULL` 裁剪命中、MTMV 刷新物化 null 行;原是 `col IN(哨兵)` 丢 null 行)——实现连接器既有意图、分区名标识不变。加 fail-loud 的 nullFlags 元数校验;`PluginDrivenScanNode` prune 短路 opt-out 只改注释(代码不变,`col IS NULL` 示例对 paimon 已不适用)。设计红队 `wf_1cdaf44e-325`(5 lens 全 SOUND_WITH_FIXES,6 findings 全折入,其中 major=**`test_paimon_mtmv` 是受影响的既有套件**,非新测)。UT 绿:api 6 / fe-core 67 / hive 9 / paimon 10,0 fail、0 checkstyle。设计 `designs/FIX-default-partition-design.md`。**e2e 待用户自跑（含 golden 再生）**：`test_hive_default_partition`(INT+DATE) · paimon `qt_null_partition_*` · **`test_paimon_mtmv`（golden `null_partition` 3→5 行,须按 e2e 实跑再生;并验 MV 自动建 `VALUES IN (NULL)` 分区不报错——region 列可空应放行）**。 > -> **⭐ 剩余 1 fe-core/SPI 大改（设计周期工作；设计→红队→实现→靶向验证→独立 commit，勿 rebless data-wrong 的 .out）**： -> - **用户已裁决(A)**：query_cache 移植 SQL 结果缓存到 SPI：BindRelation:887-888 / CacheAnalyzer:308 / NereidsSqlCacheManager:475 认通用 PluginDrivenExternalTable/PluginDrivenScanNode；**且**连接器须提供**稳定失效令牌**(不能用 ExternalTable 默认 `updateTime=currentTimeMillis()`，否则过期缓存)；需 BE 缓存填充 e2e。 +> **✅ query_cache DONE（本 session，code `c9a86337906`）**——翻闸 hive 表(`PluginDrivenMvccExternalTable`/`PluginDrivenScanNode`)静默丢失 Nereids SQL 结果缓存资格。**方案=通用能力 opt-in（用户裁决 A：所有湖仓表统一开启）**：把缓存路径 **4 文件 6 处** source-name 分支(`instanceof HMSExternalTable`/`HiveScanNode`/`HMS_EXTERNAL_TABLE`)换成 **`MTMVRelatedTableIf` 能力 + 其 `getNewestUpdateVersionOrTime()` 稳定数据版本令牌**(hive=max transient_lastDdlTime，iceberg/paimon=单调快照版本)。铁律干净(无源名分支)。范围=任何暴露有效令牌的湖仓插件表(hive/iceberg/paimon/hudi)，仍受 `enable_hive_sql_cache`(默认 false)门控 → **默认零行为变化**。**关键令牌纠正**：`ExternalTable.getUpdateTime()` 默认=schema-load 墙上时钟(与数据无关→schema-cache TTL 内数据变了缓存却读旧值=过期缓存)，改用连接器数据绑定令牌;`令牌<=0`(空分区/dropped) fail-safe 标 unsupported 不缓存。四处：BindRelation 准入(`instanceof ExternalTable && MTMVRelatedTableIf`) · SqlCacheContext.addUsedTable 令牌捕获 · NereidsSqlCacheManager 类型闸(准入 PLUGIN_EXTERNAL_TABLE)+复查(:475)+分区存在环(:506) · CacheAnalyzer **按目标表能力(非 scan-node 类)识别**——红队关键订正：`HudiScanNode extends HiveScanNode`(裸 `instanceof PluginDrivenScanNode` 会漏 hudi)、`jdbc_query()` TVF 也发 `PluginDrivenScanNode`(FunctionGenTable 无令牌须排除)，故键在 `getTupleDesc().getTable() instanceof MTMVRelatedTableIf`。删死的 source-specific `COUNTER_QUERY_HIVE_TABLE` 计数。设计红队 `wf_fe484bc8-43b`(5 lens+synth，GO_WITH_FIXES：L1 token 成本已 document+登记优化跟进、L3 hudi/TVF 已改能力识别、L2/L4/L5 已折)。UT 8 绿(PluginTableCacheAnalyzerTest 4 / SqlCacheContextPluginTableTest 2 / NereidsSqlCacheManagerPluginTableTest 2，均 RED-on-HEAD)，fe-core BUILD SUCCESS、0 checkstyle。设计 `designs/FIX-querycache-spi-design.md`。**e2e 待用户自跑（BE 缓存填充只端到端可见）**：`set enable_sql_cache=true; set enable_hive_sql_cache=true` → hive 分区表/无分区表同查两次命中、改数据(INSERT/加分区)后 MISS 返回新行(证令牌失效)；iceberg/paimon/hudi 各一表同验(Option A 新增)；负例 `jdbc_query()`/`$partitions` 不缓存不崩。 > - **ENV(告知用户，非代码)**：重置外部 hive docker → test_hive_lzo_text_format(run86.hql 未应用)、test_hive_varbinary_type(残留行翻倍)、hive_config_test tag1(HDFS OUTFILE 累积) 一并解决。 -> - **登记的独立跟进项(非本修,均已裁决延后)**：iceberg identity/bucket/truncate 分区真 NULL 渲染成 `col=null`、非字符串列同样丢分区(可用本 SPI 机制 opt-in) · paimon `partition_values()` TVF/`$partitions` 对非字符串 default-name 分区 `Integer.valueOf(默认名)` 抛(pre-existing,与扫描/裁剪面独立)。 +> - **登记的独立跟进项(非本修,均已裁决延后)**：iceberg identity/bucket/truncate 分区真 NULL 渲染成 `col=null`、非字符串列同样丢分区(可用本 SPI 机制 opt-in) · paimon `partition_values()` TVF/`$partitions` 对非字符串 default-name 分区 `Integer.valueOf(默认名)` 抛(pre-existing,与扫描/裁剪面独立) · **query_cache 令牌成本优化**(纯性能，独立任务)：`getNewestUpdateVersionOrTime()` 在 last-modified(hive)分支上多跑一次被丢弃的 `materializeLatest()` 全分区枚举(查缓存热路径 O(partitions))；优化=从 `beginQuerySnapshot` 单读 freshness kind 后直接 `queryTableFreshness()` 跳过 materialize——但改的是共享敏感 MVCC 方法，须自带 iceberg/paimon 成本中性红队，故本修未含(off-by-default 无生产回归)。 **✅ 本轮 DONE（4 commit，均编译+全套测试绿）**： - `cd4eecac5e6` **[fix] 文本读取根因（~12 suite）**：`HiveTextProperties`（fe-connector-hive）未做 `getByte` 数值→字节解码。Hive 默认分隔符存 `serialization.format="1"`=**字节 0x01**(Ctrl-A)，新代码当字符 `'1'`(0x31) 发 BE → `format_v2` TextReader 按 '1' 切 → 整行挤第一列 → null/粘连。移植 getByte + 老默认 + Hive2 拼写 key `colelction.delim` + table-params 优先；MultiDelimitSerDe 多字符保持 raw；null_format 不解码。新单测 `HiveTextPropertiesTest` 9/9。覆盖 to_date/prepare_data/basic_type/other/serde_prop/truncate/get_schema/meta_cache/query_cache + text_complex_type/to_array/text_garbled_file(靠 colelction.delim)。 diff --git a/plan-doc/tasks/designs/FIX-querycache-spi-design.md b/plan-doc/tasks/designs/FIX-querycache-spi-design.md new file mode 100644 index 00000000000000..353ba323ad9a65 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-querycache-spi-design.md @@ -0,0 +1,352 @@ +# FIX — Migrate Nereids SQL result cache to the generic SPI plugin table/scan-node + +> Restore SQL result-cache eligibility for hms-cutover tables (plain-hive today; iceberg/paimon/hudi +> when they qualify), by replacing the four source-name branches in the cache path with a +> connector-agnostic capability (`MTMVRelatedTableIf`) + a stable, data-tied invalidation token +> (`getNewestUpdateVersionOrTime()`), obeying the fe-core iron rule (no `instanceof HMSExternalTable` +> / `HiveScanNode` / `HMS_EXTERNAL_TABLE` branching). +> +> **Scope decision (user-signed, Option A, 2026-07-12):** the generic mechanism admits **any lakehouse +> plugin table that supplies a valid stable token** (hive + iceberg + paimon + hudi), not a hive-only +> connector opt-in. The user-facing switch stays `enable_hive_sql_cache` (default **false**), so *nothing +> changes by default*; the difference from a hive-only design only manifests once a user turns the switch +> on — then every qualifying lakehouse table caches, not just hive. + +## 1. Problem + +Before the hms cutover, a plain-hive table was `HMSExternalTable`; its query results were eligible for +the Nereids SQL result cache. After the cutover it is a `PluginDrivenMvccExternalTable` +(`extends PluginDrivenExternalTable extends ExternalTable`) and its scan node is `PluginDrivenScanNode` +(not `HiveScanNode`). The cache path gates on the *old class names / old table-type enum*, so a flipped +hive table silently falls into the "unsupported → no cache" arm. The clean-room re-review flagged this +as a by-design fail-safe minor at cutover time; this task restores the capability properly. + +## 2. The four loci (HEAD-verified — trust these, not the stale line numbers in HANDOFF) + +The Nereids SQL result cache has **two store tiers** (FE result-set cache for one-row/empty relations via +`StmtExecutor.tryAddFeSqlCache`; BE sql-cache for general scans via `tryAddBeCache` → `CacheProxy`) and a +**lookup** path. `StmtExecutor`'s store dispatch is already table-type-agnostic — **no change there.** The +four source-name gates are: + +| # | File:method | HEAD line | Current source-name branch | What breaks post-cutover | +|---|---|---|---|---| +| L1 | `BindRelation.bindWithMetaData` finally | `885-892` | `else if (table instanceof HMSExternalTable && enableHiveSqlCache)` → `addUsedTable`; else `setHasUnsupportedTables(true)` | flipped hive → else → cache disabled | +| L2 | `SqlCacheContext.addUsedTable` | `194-199` | `else if (tableIf instanceof HMSExternalTable) version = getUpdateTime()` | flipped hive → version stays 0 | +| L3 | `NereidsSqlCacheManager.tablesOrDataChanged` | `441-443` type gate + `475-479` re-check + `506-508` partition loop | type gate allows only OLAP/MV/HMS_EXTERNAL_TABLE; re-check + partition loop both `instanceof HMSExternalTable` | `PLUGIN_EXTERNAL_TABLE` type → always `CHANGED_AND_INVALIDATE` (never a hit); scan-table loop also rejects it | +| L4 | `CacheAnalyzer.buildCacheTableList` `305-332` + `buildCacheTableForHiveScanNode` `472-484` | scan-node gate `instanceof OlapScanNode / HiveScanNode`; `latestPartitionTime = table.getUpdateTime()` | flipped hive's node unrecognized → not "all-olap or all-hive" → `emptyList` → `CacheMode.None` → BE tier never fills. **Fixed by target-table capability, NOT node class (see L4 below): `HudiScanNode extends HiveScanNode` and `jdbc_query` TVFs both emit `PluginDrivenScanNode`.** | + +`CacheAnalyzer` is confirmed **on the Nereids result-cache hot path** (only reached through +`innerCheckCacheModeForNereids`; partition cache is force-disabled, so of `{NoNeed,None,Sql}` a fresh hive +query used to get `Sql`). It computes `CacheTable.latestPartitionTime`, applies the +"table stable for ≥ `cache_last_version_interval_second`" gate, and via `StmtExecutor.tryAddBeCache` copies +`latestPartition{Id,Version,Time}` + `CacheProxy` into the `SqlCacheContext` that the FE-level +`NereidsSqlCacheManager` later validates. So **all four must change** for end-to-end parity; fixing any +subset leaves the cache broken. + +## 3. The invalidation token — the crux + +**Legacy (data-tied, correct):** `HMSExternalTable.initSchemaAndUpdateTime` (`:700-709`) *overrides* the +`ExternalTable` default and stamps `updateTime = transient_lastDdlTime * 1000` (wall-clock only as a view +fallback). `getUpdateTime()` therefore changed iff the table's DDL/data time changed → cache invalidated +exactly on change. + +**Default (unstable, WRONG for a token):** `PluginDrivenMvccExternalTable` overrides *neither* +`initSchemaAndUpdateTime` nor `getUpdateTime`, so it inherits `ExternalTable.initSchemaAndUpdateTime` +(`:371-372`) = `setUpdateTime(System.currentTimeMillis())` at every FE schema (re)load. `getUpdateTime()` +is then wall-clock-at-last-schema-load, **orthogonal to data**. Concrete failure if used as the token: + +``` +10:00 SELECT * FROM t -> FE loads schema, token = 10:00, result stored in cache +10:05 someone INSERTs new rows into hive table t +10:06 SELECT * FROM t -> schema cache not yet expired, token STILL 10:00 == stored 10:00 + -> "unchanged" -> returns the stale 10:00 result (missing the new rows) +``` + +**Stable token (the fix):** `MTMVRelatedTableIf.getNewestUpdateVersionOrTime()` — the same value the MV / +dictionary auto-refresh machinery already trusts. +- Declared on the **capability interface** `MTMVRelatedTableIf:117` (source-agnostic), implemented by + `OlapTable`, `HMSExternalTable`, and `PluginDrivenMvccExternalTable:707-736`. +- For a **last-modified** connector (hive): returns the cache-backed + `getTableFreshness().getTimestampMillis()` — unpartitioned = table `transient_lastDdlTime`; partitioned = + **max `transient_lastDdlTime` over partitions** (`HiveConnectorMetadata:1204-1217`). This is data-tied + *and* strictly more correct than legacy (legacy's table-level DDL time missed partition-only inserts). +- For a **snapshot-id** connector (iceberg/paimon): returns the monotonic newest snapshot version. +- Plain (non-MVCC) `PluginDrivenExternalTable` does **not** implement `MTMVRelatedTableIf` → automatically + excluded (no data-change signal). + +Equality-compare (`stored != current → invalidate`) is satisfied by both token kinds: the value changes iff +data changes. We store the raw `long` (millis or version); the MTMV "carry the name" nuance +(`MTMVMaxTimestampSnapshot`) is only needed for MV snapshot bookkeeping, not for cache equality. + +### 3.1 Token edge cases (must handle) +- **`token <= 0`** (`getNewestUpdateVersionOrTime()` returns `0` for an empty partition set / dropped + table; no real hive `transient_lastDdlTime`): treat as **not cacheable** — `addUsedTable` records the + table as unsupported (`setHasUnsupportedTables(true)`) instead of pinning a bogus `0`. Fail-safe; mirrors + legacy hive-view behaviour (unstable `currentTimeMillis` fallback → never a stable hit within the + interval). Real snapshot ids / epoch-ms tokens are never `≤ 0`, so this only rejects the "no info" state. +- **snapshot-id token vs the CacheAnalyzer stability interval:** for iceberg/paimon the token is a small + monotonic version, while the interval gate computes `now(epoch-ms) - latestPartitionTime(version)` — a + huge positive → always "stable enough". Benign: equality-compare in `tablesOrDataChanged` is the real + correctness mechanism; the interval gate is only a perf heuristic ("don't cache a table that changed in + the last N seconds"). Hive keeps exact legacy semantics because its token *is* epoch-ms. +- **meta-cache staleness bound:** the freshness token reflects the `CachingHmsClient` meta cache + (TTL-bounded), not the live metastore. This is **the same bound legacy operated under** (legacy read + `transient_lastDdlTime` from the schema-cache-loaded table) and the same bound the scan itself uses, so + the cache is never *more* stale than a fresh (uncached) query would be. Acceptable, unchanged contract. +- **time-travel (`FOR VERSION/TIME AS OF`):** cache key = SQL text (includes the snapshot clause) → keys + never collide; token = *latest* version, so a pinned-snapshot result may be *over*-invalidated when new + data lands (harmless perf loss, never wrong data — the pinned snapshot is immutable). No special-casing. + +### 3.2 Token cost on the lookup hot path (red-team L1, honest accounting) +`getNewestUpdateVersionOrTime()` is **not** the cheap "cached-field read" legacy `getUpdateTime()` was. +For a flipped **partitioned hive** table it costs, per call: `materializeLatest()` (a `listLatestPartitions` +→ builds a `PartitionItem` per partition — the result is then *discarded* on the last-modified branch) **plus** +`queryTableFreshness()` → `getTableFreshness()` (`collectPartitionNames` + a `get_partitions_by_names` to +read each partition's `transient_lastDdlTime` and take the max). Both HMS calls are **meta-cache-backed** +(`CachingHmsClient`), so a warm cache makes them in-memory reads; the residual cost is CPU (building the +partition items + the max scan). Unpartitioned hive is cheap (freshness from the handle, no round-trip); +snapshot-id connectors (iceberg/paimon) read a monotonic version. The token is also computed **twice** per +query (store: L2 `addUsedTable` + L4 `buildCacheTableForExternalScanNode`). + +**Why this is acceptable to ship, not a blocker:** (a) opt-in, `enable_hive_sql_cache` default **false** — zero +production cost by default; (b) a cache HIT pays ≈ the *metadata* portion of planning (which a cache MISS pays +anyway) and **skips the entire BE data scan** — so a hit is always cheaper than a miss, the feature is always +net-positive; (c) the meta-cache amortises repeated lookups; (d) it matches the already-accepted cost of the +MV/dictionary poll that calls the same method. + +**Registered follow-up optimisation (separate task, NOT this change):** short-circuit +`getNewestUpdateVersionOrTime()` to skip `materializeLatest()` on the last-modified branch (read the freshness +kind from `beginQuerySnapshot` alone, then `queryTableFreshness()`), collapsing the double handle-resolution +and dropping the discarded partition-item enumeration. It is a *pure, output-preserving* refactor of a shared, +sensitive MVCC method (`plugindriven-mvcc-table-is-live-not-dormant`), so it needs its **own** cost-neutrality +red-team for iceberg/paimon and is deliberately kept out of the fe-core cache-wiring change here. + +## 4. Design — connector-agnostic edits + +Keep the `OlapTable` branch untouched everywhere (surgical; internal-table cache behaviour must stay +byte-identical). Insert a generic external-MVCC branch **after** it, keyed on `instanceof MTMVRelatedTableIf`. +`OlapTable` also implements `MTMVRelatedTableIf`, but is always caught by the earlier `instanceof OlapTable` +arm, so its path is unchanged. + +### L1 — `BindRelation` (`:885-892`) admission +```java +} else if (table instanceof OlapTable) { + sqlCacheContext.addUsedTable(table); +} else if (table instanceof ExternalTable && table instanceof MTMVRelatedTableIf + && cascadesContext.getConnectContext().getSessionVariable().enableHiveSqlCache) { + sqlCacheContext.addUsedTable(table); // token validity enforced inside addUsedTable +} else { + sqlCacheContext.setHasUnsupportedTables(true); +} +``` +- `instanceof ExternalTable` keeps this to external catalogs (defensive; every `MTMVRelatedTableIf` that + isn't `OlapTable` today is an external table, but the belt-and-suspenders keeps a future internal + `MTMVRelatedTableIf` out of this arm). +- `enableHiveSqlCache` (default false) reused as the external opt-in switch — see §5 naming note. +- Import: add `MTMVRelatedTableIf`; **keep** the existing `HMSExternalTable` import (still used by the + view/hudi cutover arm at `:754-795`). + +### L2 — `SqlCacheContext.addUsedTable` (`:192-213`) token capture +```java +long version = 0; +try { + if (tableIf instanceof OlapTable) { + version = ((OlapTable) tableIf).getVisibleVersion(); + } else if (tableIf instanceof MTMVRelatedTableIf) { + version = ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime(); + if (version <= 0) { // §3.1 fail-safe: no reliable data-change signal + setHasUnsupportedTables(true); + return; + } + } +} catch (Throwable e) { + setHasUnsupportedTables(true); + LOG.warn("table {}, can not get version", tableIf.getName(), e); +} +``` +- Swap import `HMSExternalTable` → `MTMVRelatedTableIf`. +- `TableVersion.type` continues to be `tableIf.getType()` (= `PLUGIN_EXTERNAL_TABLE` for flipped tables). + +### L3 — `NereidsSqlCacheManager.tablesOrDataChanged` (`:441-482`) +Type gate — admit the plugin type: +```java +if (tableVersion.type != TableType.OLAP && tableVersion.type != TableType.MATERIALIZED_VIEW + && tableVersion.type != TableType.HMS_EXTERNAL_TABLE + && tableVersion.type != TableType.PLUGIN_EXTERNAL_TABLE) { + return IsChanged.CHANGED_AND_INVALIDATE_CACHE; +} +``` +Re-check (used-tables loop) — replace the HMS branch (the `else` catch-all at `:480-481` stays for +non-capability tables): +```java +} else if (tableIf instanceof MTMVRelatedTableIf) { + if (tableVersion.version != ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime()) { + return IsChanged.CHANGED_AND_INVALIDATE_CACHE; + } +} else { + return IsChanged.CHANGED_AND_INVALIDATE_CACHE; +} +``` +Partition-existence loop (`:506-508`) — **the branch I initially missed; found by the recon sweep.** +`CacheAnalyzer` adds external tables to `getScanTables()` too, so this loop runs for flipped hive. Legacy +skipped `HMSExternalTable` here (per-partition existence is an Olap-only concern; external freshness is +fully covered by the token compare in the used-tables loop above). Generalize the skip to the capability: +```java +} else if (!(tableIf instanceof MTMVRelatedTableIf)) { // was: !(tableIf instanceof HMSExternalTable) + return IsChanged.CHANGED_AND_INVALIDATE_CACHE; +} +``` +- `OlapTable` is matched by the earlier `instanceof OlapTable` arm (`:453`/`:489`); `MTMV` too (it *is* an + `OlapTable`). So the new arms only catch external MVCC tables. +- `HMSExternalTable:125 implements MTMVRelatedTableIf` — so both swaps are a strict *superset* of the old + HMS behaviour: every legacy HMS table still takes the same path, plus flipped plugin tables now qualify. +- `HMS_EXTERNAL_TABLE` left in the type allow-list for safety (no live producer post-cutover; harmless). +- Swap import `HMSExternalTable` → `MTMVRelatedTableIf`. **NereidsSqlCacheManager has 3 branches to fix** + (type gate `:441`, used-tables re-check `:475`, partition loop `:506`). + +### L4 — `CacheAnalyzer` (recognize by TARGET-TABLE capability, **not** scan-node class — red-team fix) +**Do NOT swap `instanceof HiveScanNode` → `instanceof PluginDrivenScanNode`.** The scan-node class is the +wrong discriminator on two counts the red-team proved: +- `HudiScanNode extends HiveScanNode` (**not** `PluginDrivenScanNode`) — a class-based swap would *drop* + hudi from the gate that `HiveScanNode` used to cover (a silent regression). +- `jdbc_query(...)` TVFs *also* emit a `PluginDrivenScanNode` (`JdbcQueryTableValueFunction.java:62`) whose + backing table is a `FunctionGenTable` (`extends Table`, **not** `MTMVRelatedTableIf`) with no data-version + token — a class-based `instanceof PluginDrivenScanNode` would wrongly admit it. + +Instead, recognize an eligible external scan node by its **target table's capability** — the same +`MTMVRelatedTableIf` gate used everywhere else. This is *more* iron-rule-clean (keys on a capability, not a +node class) and robust to the scan-node hierarchy. `buildCacheTableList` (`:305-332`): +```java +long olapScanNodeSize = 0; +long externalCacheableSize = 0; +for (ScanNode scanNode : scanNodes) { + if (scanNode instanceof OlapScanNode) { + olapScanNodeSize++; + } else if (scanNode.getTupleDesc() != null + && scanNode.getTupleDesc().getTable() instanceof MTMVRelatedTableIf) { // capability, not class + externalCacheableSize++; + } +} +... +if (!(olapScanNodeSize == scanNodes.size() || externalCacheableSize == scanNodes.size())) { + return Collections.emptyList(); // no federated / mixed-source / TVF caching (legacy parity) +} +... +CacheTable cTable = node instanceof OlapScanNode + ? buildCacheTableForOlapScanNode((OlapScanNode) node) + : buildCacheTableForExternalScanNode(node); // takes the base ScanNode +``` +New `buildCacheTableForExternalScanNode` is connector-agnostic; `getTupleDesc()`/`getTable()` are public on +the `ScanNode`/`TupleDescriptor` base (`PluginDrivenScanNode.getTargetTable()` is `protected`+`throws`, so +we deliberately route through the tuple descriptor): +```java +private CacheTable buildCacheTableForExternalScanNode(ScanNode node) { + CacheTable cacheTable = new CacheTable(); + TableIf tableIf = node.getTupleDesc().getTable(); // guaranteed MTMVRelatedTableIf by the gate above + cacheTable.table = tableIf; + cacheTable.partitionNum = node.getSelectedPartitionNum(); // ScanNode base, public, no throw + cacheTable.latestPartitionTime = ((MTMVRelatedTableIf) tableIf).getNewestUpdateVersionOrTime(); + DatabaseIf database = tableIf.getDatabase(); + CatalogIf catalog = database.getCatalog(); + ScanTable scanTable = new ScanTable( + new FullTableName(catalog.getName(), database.getFullName(), tableIf.getName())); + scanTables.add(Pair.of(scanTable, tableIf)); + return cacheTable; +} +``` +- `MetricRepo.COUNTER_QUERY_HIVE_TABLE` bump is dropped (source-specific metric; the generic gate covers all + sources). Keep `COUNTER_QUERY_TABLE`/`COUNTER_QUERY_OLAP_TABLE`. +- **Remove** the `HiveScanNode` import; add `MTMVRelatedTableIf`. No `PluginDrivenScanNode` import needed — + the gate never names it (pure capability check), which is the cleanest iron-rule form. + +## 5. Iron-rule compliance & naming + +- All four gates now key on the **capability interface** `MTMVRelatedTableIf` and the **generic node** + `PluginDrivenScanNode` — no `instanceof HMSExternalTable`, no `HiveScanNode`, no `switch(dlaType)`, no + engine-name test. Matches the project rule (memory `catalog-spi-plugindriven-no-source-specific-code`) + and Trino's "ask the connector for the capability, don't inspect its class" model. +- **Session-var naming:** `enable_hive_sql_cache` is reused as the external opt-in switch (default false, + so no default behaviour change). Under Option A its scope is now "all lakehouse plugin tables", so the + `hive` in the name is legacy. Renaming is deliberately **out of scope** (surgical; avoids breaking + existing user configs). Flagged for a possible future generic alias — NOT this change. + +## 6. Blast radius + +- **OlapTable / internal:** untouched (earlier `instanceof OlapTable` arm everywhere). Zero behaviour change. +- **Default (switch off):** every table hits the same "unsupported" arm as today. Zero behaviour change. +- **Switch on:** flipped hive restored to (better-than-)legacy caching; iceberg/paimon/hudi *newly* eligible + when they expose a valid `getNewestUpdateVersionOrTime()` (Option A, user-signed). Their tokens are already + the monotonic values the MV/dictionary path trusts, so correctness rides on an existing contract — but the + cache-fill path for them is newly exercised → their e2e is in scope (§8). +- **Plain non-MVCC `PluginDrivenExternalTable` / system tables (`$partitions`) / TVFs:** excluded because + their table is **not** `MTMVRelatedTableIf` — enforced by **two** capability gates: L1 admission + (`setHasUnsupportedTables` → `supportSqlCache()` false) and L4 scan-node recognition (target table not + `MTMVRelatedTableIf` → `externalCacheableSize` short of total → `emptyList` → `CacheMode.None`). It is + **NOT** enforced by `latestPartitionTime = 0` (a `0` token would pass the CacheAnalyzer interval gate — the + earlier design draft's claim there was mechanically wrong). The `token <= 0` fail-safe in L2 is a *second* + line for a real MVCC table that momentarily has no freshness (empty partitions), not the sys-table guard. +- **`HMSExternalTable` DLA types (legacy, dead post-cutover):** `HMSExternalTable implements MTMVRelatedTableIf` + and `getNewestUpdateVersionOrTime()` returns a non-zero `transient_lastDdlTime`-derived value for a real + table, so the `token <= 0` fail-safe cannot silently disable a live legacy table (moot anyway — no live + producer after the cutover). +- **BE side:** unchanged. The BE sql-cache was already fed for external (hive) tables pre-cutover via + `tryAddBeCache`; restoring `CacheMode.Sql` for the plugin node reuses that exact path. + +## 7. Unit tests — AS IMPLEMENTED (3 classes, all RED on the pre-cutover HEAD) +- **`qe/cache/PluginTableCacheAnalyzerTest`** (L4): drives the two new private members via `Deencapsulation` + (avoids the `MetricRepo`/analyze bootstrap that keeps `HmsQueryCacheTest` disabled for the plugin path). + `isExternalCacheableScanNode` → true for a `PluginDrivenMvccExternalTable`-backed node, false for a + `FunctionGenTable` (jdbc TVF) node, false for a null tuple desc; `buildCacheTableForExternalScanNode` → + `latestPartitionTime == getNewestUpdateVersionOrTime()`, `partitionNum` from the scan node. +- **`nereids/SqlCacheContextPluginTableTest`** (L2): `addUsedTable` records the connector token for a plugin + table (`TableVersion.version == token`, type `PLUGIN_EXTERNAL_TABLE`); `token <= 0` → `hasUnsupportedTables` + and no entry (fail-safe). +- **`common/cache/NereidsSqlCacheManagerPluginTableTest`** (L3, the correctness core): `tablesOrDataChanged` + via `Deencapsulation.invoke` with a mock `Env` chain → unchanged token = NOT_CHANGED (cache served), + advanced token = CHANGED_AND_INVALIDATE_CACHE (data change evicts). +- Gate: `mvn -pl fe-core -am test` for the 3 classes green, 0 checkstyle. e2e (§8) covers BE fill + the + `:506` partition-loop skip end-to-end. + +### 7.1 Original test-design sketch (superseded by 7) +`tablesOrDataChanged` and `findTableIf` are private → drive them via `Deencapsulation.invoke(manager, ...)` +on a manager instance, passing a mock `Env` whose `getCatalogMgr()`/catalog/db chain resolves the used table +to a mock `MTMVRelatedTableIf` external table (Mockito `mock(PluginDrivenMvccExternalTable.class, +CALLS_REAL_METHODS)` or an interface stub) with a settable `getNewestUpdateVersionOrTime()`. +- **`NereidsSqlCacheManager.tablesOrDataChanged`** (the crux): + - used-tables loop (`:475`): `PLUGIN_EXTERNAL_TABLE` used-table, token unchanged → `NOT_CHANGED`; token + changed → `CHANGED_AND_INVALIDATE_CACHE`. **RED on HEAD** (type gate `:441` returns invalidate for + `PLUGIN_EXTERNAL_TABLE`). + - **partition-existence loop (`:506`)**: a plugin table in `getScanTables()` with an *unchanged* token → + `NOT_CHANGED`. **RED if the `:506` skip is reverted** to `!(instanceof HMSExternalTable)` — this is the + easily-missed branch, so it gets its own assertion. +- **`SqlCacheContext.addUsedTable`**: records `getNewestUpdateVersionOrTime()` for an `MTMVRelatedTableIf` + table; `token ≤ 0` → `hasUnsupportedTables()==true` and no entry; throw → unsupported. Keep `OlapTable` + (`getVisibleVersion`) and any legacy HMS expectation byte-unchanged. +- **`CacheAnalyzer.buildCacheTableList`**: build with a Mockito `PluginDrivenScanNode` whose `getTupleDesc()` + returns a `TupleDescriptor` bound to a mock `MTMVRelatedTableIf` table → all-external plan is recognized + (non-empty list, `latestPartitionTime == getNewestUpdateVersionOrTime()`); a plan mixing that node with an + `OlapScanNode` → `emptyList` (no federated caching); a `PluginDrivenScanNode` whose target table is a + `FunctionGenTable` (jdbc TVF) → `emptyList` (capability gate excludes it). **RED on HEAD** (`instanceof + HiveScanNode` never matches the plugin node). +- **Parity note (corrected):** the (currently `@Disabled`) `HmsQueryCacheTest` only pins the **L4** + CacheAnalyzer decision (`buildCacheTableList` + stability-interval → `CacheMode`/`latestTime`) over a legacy + `HMSExternalTable`; it does **not** exercise L1 admission or the invalidation re-check, and is not directly + portable (it drives a real `HMSExternalCatalog`). It is *not* the invalidation oracle — the dedicated + `NereidsSqlCacheManager` tests above are. +- Gate: `mvn -pl fe-core -am test-compile` + targeted `mvn test` green, 0 checkstyle, import gate clean. + +## 8. e2e (user to run — BE cache-fill is only observable end-to-end) +- **hive PARTITIONED:** query-cache suite with `set enable_sql_cache=true` + `set enable_hive_sql_cache=true`: + same query twice → 2nd is a cache hit (profile / `SqlCacheCounter`); then mutate (INSERT / add partition) → + next query MISSES and returns fresh rows (proves token invalidation on a partition-only change — the exact + stale-cache class this fix targets). Assert result equality vs a no-cache run. +- **hive UNPARTITIONED:** same hit-then-invalidate cycle — this exercises the `queryTableFreshness()` + handle-only token path (distinct from the partition-max path) and confirms the `token ≤ 0` fail-safe does + not spuriously disable a real unpartitioned table (assert a non-zero token / an actual cache hit). +- **iceberg / paimon / hudi (Option A newly-eligible):** the same hit-then-invalidate cycle on one table each, + to validate the snapshot-version token path and the BE fill for non-hive lakehouse tables (memory + `hms-iceberg-delegation-needs-e2e`). **hudi** specifically confirms the target-table-capability gate (not + the dead `HudiScanNode extends HiveScanNode` path). +- **negative:** a `jdbc_query(...)` TVF and a `$partitions` sys-table query must NOT be cached (no crash, no + stale) with the switch on. +``` From 36721c9d0eb909c12b532ab6477dac7a6f162ac5 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 09:20:17 +0800 Subject: [PATCH 281/334] fix test_hive_varbinary_type.groovy --- .../hive/test_hive_varbinary_type.groovy | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/regression-test/suites/external_table_p0/hive/test_hive_varbinary_type.groovy b/regression-test/suites/external_table_p0/hive/test_hive_varbinary_type.groovy index d7975b666db318..b2edc7a1f27d6d 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_varbinary_type.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_varbinary_type.groovy @@ -56,13 +56,13 @@ suite("test_hive_varbinary_type", "p0,external") { qt_select5 """ select * from test_hive_binary_edge_cases order by id; """ // write orc - qt_select6 """ insert into test_hive_binary_orc_write_no_mapping select * from test_hive_binary_orc; """ + qt_select6 """ insert overwrite table test_hive_binary_orc_write_no_mapping select * from test_hive_binary_orc; """ qt_select7 """ insert into test_hive_binary_orc_write_no_mapping values(6,X"ABAB",X"ABAB"); """ qt_select8 """ insert into test_hive_binary_orc_write_no_mapping values(NULL,NULL,NULL); """ qt_select9 """ select * from test_hive_binary_orc_write_no_mapping order by id; """ // write parquet - qt_select10 """ insert into test_hive_binary_parquet_write_no_mapping select * from test_hive_binary_parquet; """ + qt_select10 """ insert overwrite table test_hive_binary_parquet_write_no_mapping select * from test_hive_binary_parquet; """ qt_select11 """ insert into test_hive_binary_parquet_write_no_mapping values(6,X"ABAB",X"ABAB"); """ qt_select12 """ insert into test_hive_binary_parquet_write_no_mapping values(NULL,NULL,NULL); """ qt_select13 """ select * from test_hive_binary_parquet_write_no_mapping order by id; """ @@ -77,13 +77,13 @@ suite("test_hive_varbinary_type", "p0,external") { qt_select18 """ select * from test_hive_binary_edge_cases order by id; """ // write orc - qt_select19 """ insert into test_hive_binary_orc_write_with_mapping select * from test_hive_binary_orc; """ + qt_select19 """ insert overwrite table test_hive_binary_orc_write_with_mapping select * from test_hive_binary_orc; """ qt_select20 """ insert into test_hive_binary_orc_write_with_mapping values(6,X"ABAB",X"ABAB"); """ qt_select21 """ insert into test_hive_binary_orc_write_with_mapping values(NULL,NULL,NULL); """ qt_select22 """ select * from test_hive_binary_orc_write_with_mapping order by id; """ // write parquet - qt_select23 """ insert into test_hive_binary_parquet_write_with_mapping select * from test_hive_binary_parquet; """ + qt_select23 """ insert overwrite table test_hive_binary_parquet_write_with_mapping select * from test_hive_binary_parquet; """ qt_select24 """ insert into test_hive_binary_parquet_write_with_mapping values(6,X"ABAB",X"ABAB"); """ qt_select25 """ insert into test_hive_binary_parquet_write_with_mapping values(NULL,NULL,NULL); """ qt_select26 """ select * from test_hive_binary_parquet_write_with_mapping order by id; """ From 8ba3520c9e90791ac19fa35b736d0841f4fd8f16 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 09:44:24 +0800 Subject: [PATCH 282/334] [fix](catalog) hive: honor hive.ignore_absent_partitions in partition listing fe-connector-hive's HiveScanPlanProvider.listAndSplitFiles unconditionally swallowed a missing-partition-location listing failure, silently dropping the legacy hive.ignore_absent_partitions=false semantics (old HiveExternalMetaCache threw "Partition location does not exist" when the flag was false). Restore it: when the listing failure is a not-found (a FileNotFoundException anywhere in the cause chain) AND hive.ignore_absent_partitions=false, rethrow a loud DorisConnectorException; the default true keeps the tolerant skip-with-warning, and transient/unreadable listing failures still skip. Add the IGNORE_ABSENT_PARTITIONS property constant. Also make hive_config_test idempotent (groovy-only): it wrote OUTFILE ORC files into fixed HDFS dirs that were never cleaned, so reruns accumulated files and inflated the recursive_directories row counts (order_qt_1/2/21). Clear the dirs the suite writes before writing (docker init loads no data into them); the country=India/Delhi absent-partition dir is intentionally left untouched so the final ignore_absent_partitions=false check still throws. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../hive/HiveConnectorProperties.java | 7 ++++++ .../connector/hive/HiveScanPlanProvider.java | 25 +++++++++++++++++++ .../hive/hive_config_test.groovy | 13 ++++++++++ 3 files changed, 45 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java index ddb2273f6e9bd4..dae9f256b0d62b 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorProperties.java @@ -108,6 +108,13 @@ private HiveConnectorProperties() { /** Default batch size, matching the engine's legacy {@code hms_events_batch_size_per_rpc} default. */ public static final int DEFAULT_HMS_EVENTS_BATCH_SIZE = 500; + /** + * When {@code false}, a partition whose storage location does not exist fails the query loud + * ({@code "Partition location does not exist"}); the default {@code true} tolerates it by skipping + * the partition with a warning. Mirrors legacy {@code HiveExternalMetaCache} semantics. + */ + public static final String IGNORE_ABSENT_PARTITIONS = "hive.ignore_absent_partitions"; + /** * Parse an integer property with a default value. */ diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index c86540cf125dd6..c277b2039572ac 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -35,6 +35,7 @@ import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; +import java.io.FileNotFoundException; import java.io.IOException; import java.util.ArrayList; import java.util.Collections; @@ -472,6 +473,15 @@ private void listAndSplitFiles(String dbName, String tableName, try { files = fileListingCache.listDataFiles(dbName, tableName, partition.location, fs); } catch (HiveDirectoryListingException e) { + // hive.ignore_absent_partitions=false (non-default): a partition whose LOCATION does not exist + // must fail the query loud with the legacy message rather than be silently skipped. Only a + // not-found cause counts as "absent"; a transient/unreadable listing failure still follows the + // tolerant skip-with-warning path below. Default true preserves the skip behavior. + if (isLocationNotFound(e) && !HiveConnectorProperties.getBoolean( + catalogProperties, HiveConnectorProperties.IGNORE_ABSENT_PARTITIONS, true)) { + throw new DorisConnectorException( + "Partition location does not exist: " + partition.location, e); + } LOG.warn("Cannot list files in partition: {}", partition.location, e); return; } @@ -481,6 +491,21 @@ private void listAndSplitFiles(String dbName, String tableName, } } + /** + * Whether a listing failure was caused by the directory not existing (a {@link FileNotFoundException} + * anywhere in the cause chain), as opposed to a transient or unreadable-permission failure. Used to + * decide whether {@code hive.ignore_absent_partitions=false} should turn a skipped partition into a + * loud "Partition location does not exist" error. + */ + private static boolean isLocationNotFound(Throwable e) { + for (Throwable t = e; t != null; t = t.getCause()) { + if (t instanceof FileNotFoundException) { + return true; + } + } + return false; + } + /** * Splits a file into scan ranges based on target split size. * diff --git a/regression-test/suites/external_table_p0/hive/hive_config_test.groovy b/regression-test/suites/external_table_p0/hive/hive_config_test.groovy index c371f2c9fd2ce9..7d809f73d0e269 100644 --- a/regression-test/suites/external_table_p0/hive/hive_config_test.groovy +++ b/regression-test/suites/external_table_p0/hive/hive_config_test.groovy @@ -15,6 +15,9 @@ // specific language governing permissions and limitations // under the License. +import org.apache.doris.regression.util.Hdfs +import org.apache.hadoop.fs.Path + suite("hive_config_test", "p0,external") { String db_name = "regression_test_external_table_p0_hive" String internal_table = "hive_config_test" @@ -47,6 +50,16 @@ suite("hive_config_test", "p0,external") { // It's okay to use random `hdfsUser`, but can not be empty. def hdfsUserName = "doris" + // Make this suite idempotent: the OUTFILE writes below drop uniquely-named files into + // fixed HDFS dirs that nothing ever cleans, so reruns accumulate files and inflate the + // row counts asserted by order_qt_1/2/21/3. Clear the dirs this suite writes to before + // writing. Docker init loads no data into them (run.sh only mkdir -p), so this is safe. + // The country=India/Delhi absent-partition dir is intentionally left untouched so the + // final 'hive.ignore_absent_partitions'=false check still throws. + Hdfs hdfs = new Hdfs(defaultFS, hdfsUserName, context.config.dataPath + "/") + def fs = hdfs.fs + fs.delete(new Path("/user/doris/suites/default/hive_recursive_directories_table"), true) + fs.delete(new Path("/user/doris/suites/default/hive_ignore_absent_partitions_table/country=USA/city=NewYork"), true) def test_outfile = {format, uri -> def res = sql """ From 6c1e570f31e91e3f3ef8ab28391a4d97e8ec85b7 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 10:14:33 +0800 Subject: [PATCH 283/334] fix test_hive_ddl.groovy --- .../external_table_p0/hive/ddl/test_hive_ddl.groovy | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/regression-test/suites/external_table_p0/hive/ddl/test_hive_ddl.groovy b/regression-test/suites/external_table_p0/hive/ddl/test_hive_ddl.groovy index f274be0bf92f3a..23e3e37d8646fa 100644 --- a/regression-test/suites/external_table_p0/hive/ddl/test_hive_ddl.groovy +++ b/regression-test/suites/external_table_p0/hive/ddl/test_hive_ddl.groovy @@ -23,7 +23,7 @@ suite("test_hive_ddl", "p0,external") { def test_db = { String catalog_name -> logger.info("Test create/drop database...") sql """switch ${catalog_name}""" - sql """ drop database if exists `test_hive_db` """; + sql """ drop database if exists `test_hive_db` force """; sql """ create database if not exists ${catalog_name}.`test_hive_db` """; def create_db_res = sql """ show create database test_hive_db """ logger.info("${create_db_res}") @@ -36,7 +36,7 @@ suite("test_hive_ddl", "p0,external") { """ test { sql """ drop database `test_hive_db` """; - exception "java.sql.SQLException: errCode = 2, detailMessage = failed to drop database from hms client. reason: org.apache.hadoop.hive.metastore.api.InvalidOperationException: Database test_hive_db is not empty. One or more tables exist." + exception "java.sql.SQLException: errCode = 2, detailMessage = Failed to drop Hive database test_hive_db: HMS operation failed: Database test_hive_db is not empty. One or more tables exist." } sql """ DROP TABLE `test_hive_db_has_tbl` """ @@ -111,7 +111,7 @@ suite("test_hive_ddl", "p0,external") { 'file_format'='${file_format}' ) """ - exception "failed to create table from hms client. reason: java.lang.UnsupportedOperationException: Table with default values is not supported if the hive version is less than 3.0. Can set 'hive.version' to 3.0 in properties." + exception "Table with default values is not supported" } test { @@ -672,7 +672,7 @@ suite("test_hive_ddl", "p0,external") { 'file_format'='${file_format}' ) """ - exception "failed to create table from hms client. reason: org.apache.doris.datasource.hive.HMSClientException: Unsupported primitive type conversion of largeint" + exception "Unsupported type conversion of" } test { @@ -724,12 +724,12 @@ suite("test_hive_ddl", "p0,external") { test { sql """ create table err_tb (id int) engine = iceberg """ - exception "Hms type catalog can only use `hive` engine." + exception "This catalog can only use `hive` engine" } test { sql """ create table err_tb (id int) engine = jdbc """ - exception "Hms type catalog can only use `hive` engine." + exception "This catalog can only use `hive` engine" } sql """ drop database test_hive_db_error_tbl """ From 6a65a64b5f25707f12ada8ce338761965998fc96 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 10:58:03 +0800 Subject: [PATCH 284/334] [fix](catalog) hive: honor legacy meta-cache ttl-second properties in SPI connector After the hms SPI cutover, hive catalogs run through fe-connector-hive, whose cache layers read only the namespaced meta.cache.hive..ttl-second keys. The legacy catalog knobs (file.meta.cache.ttl-second, partition.cache.ttl-second, schema.cache.ttl-second) were validated but never remapped, so setting them to 0 no longer disabled the cache -- it silently kept the default 24h TTL, and newly written files/partitions/columns stayed invisible until REFRESH. fe-core did this remap via CacheSpec.applyCompatibilityMap in AbstractExternalMetaCache.initCatalog; the connector dropped it. Apply the same compatibility remap at each cache-construction site, mirroring HiveExternalMetaCache.catalogPropertyCompatibilityMap: - file.meta.cache.ttl-second -> meta.cache.hive.file.ttl-second (HiveFileListingCache) - schema.cache.ttl-second -> meta.cache.hive.table.ttl-second (CachingHmsClient) - partition.cache.ttl-second -> meta.cache.hive.partition_names.ttl-second (CachingHmsClient) partition.cache maps only to the partition-name list (fe-core's partition_values), not the per-partition objects cache, faithful to the legacy mapping. Add CacheSpec.metaCacheTtlKey(engine, entry) helper and unit tests asserting each legacy key at 0 disables the right cache (and only it). Backward-compatible: a no-op when the legacy keys are absent; the namespaced key wins if both are set. Fixes test_hive_meta_cache. Co-Authored-By: Claude Opus 4.8 Claude-Session: https://claude.ai/code/session_01NoUy5UkBMkyE9pqSH84CR3 --- .../doris/connector/cache/CacheSpec.java | 11 +++++++ .../connector/hive/HiveFileListingCache.java | 14 +++++++++ .../hive/HiveFileListingCacheTest.java | 15 +++++++++ .../doris/connector/hms/CachingHmsClient.java | 31 ++++++++++++++++++- .../connector/hms/CachingHmsClientTest.java | 30 ++++++++++++++++++ 5 files changed, 100 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheSpec.java b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheSpec.java index c950cafaa3ce16..47cd0596b46096 100644 --- a/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheSpec.java +++ b/fe/fe-connector/fe-connector-cache/src/main/java/org/apache/doris/connector/cache/CacheSpec.java @@ -168,6 +168,17 @@ public static String metaCacheKeyPrefix(String engine) { return META_CACHE_PREFIX + engine + "."; } + /** + * Build the standard external meta cache TTL key for one engine+entry. + * Example: {@code meta.cache.hive.file.ttl-second}. + * + *

    Used to translate a legacy catalog TTL knob (e.g. {@code file.meta.cache.ttl-second}) into the + * namespaced key a cache actually reads, via {@link #applyCompatibilityMap}. + */ + public static String metaCacheTtlKey(String engine, String entryName) { + return META_CACHE_PREFIX + engine + "." + entryName + KEY_TTL_SECOND; + } + /** * Returns true when the given property key belongs to one engine's meta cache namespace. */ diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java index 0a1a0f97041221..3964ff1aebf16d 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveFileListingCache.java @@ -79,6 +79,12 @@ public class HiveFileListingCache { /** {@code meta.cache.hive.file.*} — cached directory listings. */ static final String ENTRY_FILE = "file"; + /** + * Legacy fe-core catalog knob ({@code HMSExternalCatalog.FILE_META_CACHE_TTL_SECOND}) remapped onto this + * cache's namespaced {@code meta.cache.hive.file.ttl-second} for backward compatibility. See the constructor. + */ + static final String LEGACY_FILE_META_CACHE_TTL_SECOND = "file.meta.cache.ttl-second"; + // Legacy fe-core Config values, mirrored locally (the connector never touches fe-core Config): // TTL = Config.external_cache_expire_time_seconds_after_access (86400s = 24h) // capacity = Config.max_external_file_cache_num (10000) @@ -126,6 +132,14 @@ private static DirectoryLister defaultLister(Map properties) { HiveFileListingCache(Map properties, DirectoryLister lister) { Map props = properties == null ? Collections.emptyMap() : properties; + // Translate the legacy fe-core catalog knob file.meta.cache.ttl-second into the namespaced key this cache + // reads (mirrors HiveExternalMetaCache.catalogPropertyCompatibilityMap: FILE_META_CACHE_TTL_SECOND -> + // ENTRY_FILE ttl). Without this, an "hms" catalog that set the legacy key silently kept the default 24h + // file cache after the SPI cutover, so e.g. file.meta.cache.ttl-second=0 no longer disabled the listing + // cache and a newly-written file in an already-listed partition stayed invisible until REFRESH. + props = CacheSpec.applyCompatibilityMap(props, + Collections.singletonMap(LEGACY_FILE_META_CACHE_TTL_SECOND, + CacheSpec.metaCacheTtlKey(ENGINE, ENTRY_FILE))); CacheSpec spec = CacheSpec.fromProperties(props, ENGINE, ENTRY_FILE, CacheSpec.of(true, DEFAULT_TTL_SECOND, DEFAULT_FILE_CAPACITY)); // Contextual-only + manual-miss so the slow listStatus runs on the caller (TCCL-pinned) thread outside diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java index 8d18c2b36664ed..ec37a965c09299 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveFileListingCacheTest.java @@ -176,6 +176,21 @@ public void disablingViaPropsBypassesTheCache() { Assertions.assertEquals(2, lister.totalCalls); } + @Test + public void legacyFileMetaCacheTtlZeroBypassesTheCache() { + CountingLister lister = new CountingLister(); + HiveFileListingCache cache = new HiveFileListingCache( + props("file.meta.cache.ttl-second", "0"), lister); + + cache.listDataFiles("db", "t", "loc", FS); + cache.listDataFiles("db", "t", "loc", FS); + // WHY: the legacy fe-core knob file.meta.cache.ttl-second=0 must still disable the listing cache after the + // SPI cutover (it is remapped onto meta.cache.hive.file.ttl-second). Without the compatibility remap the + // key was ignored, the default 24h cache stayed on, and a newly-written file in an already-listed + // partition stayed invisible until REFRESH (regression that failed test_hive_meta_cache's sql_2row). + Assertions.assertEquals(2, lister.totalCalls); + } + // ==================== the production lister: filters dirs + hidden files, lists literally ==================== @Test diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java index 28a3cd535f936f..c981f82932b22a 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/CachingHmsClient.java @@ -25,6 +25,7 @@ import java.io.IOException; import java.util.ArrayList; import java.util.Collections; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.Objects; @@ -118,7 +119,8 @@ public class CachingHmsClient implements HmsClient { public CachingHmsClient(HmsClient delegate, Map properties) { this.delegate = Objects.requireNonNull(delegate, "delegate can not be null"); - Map props = properties == null ? Collections.emptyMap() : properties; + Map props = applyLegacyTtlCompatibility( + properties == null ? Collections.emptyMap() : properties); this.tableCache = newEntry("hive.table", props, ENTRY_TABLE, DEFAULT_TABLE_CAPACITY); this.partitionNamesCache = newEntry("hive.partition_names", props, ENTRY_PARTITION_NAMES, DEFAULT_PARTITION_NAMES_CAPACITY); @@ -136,6 +138,33 @@ private static MetaCacheEntry newEntry(String name, Map(name, null, spec, ForkJoinPool.commonPool(), false, true, 0L, true); } + /** Legacy fe-core catalog knob ({@code ExternalCatalog.SCHEMA_CACHE_TTL_SECOND}) for the table/schema cache. */ + static final String LEGACY_SCHEMA_CACHE_TTL_SECOND = "schema.cache.ttl-second"; + /** Legacy fe-core knob ({@code HMSExternalCatalog.PARTITION_CACHE_TTL_SECOND}) for the partition-list cache. */ + static final String LEGACY_PARTITION_CACHE_TTL_SECOND = "partition.cache.ttl-second"; + + /** + * Translate the legacy fe-core catalog TTL knobs into this client's namespaced entry keys, mirroring + * {@code HiveExternalMetaCache.catalogPropertyCompatibilityMap} so an existing "hms" catalog that set the old + * keys keeps working after the SPI cutover: + *

      + *
    • {@code schema.cache.ttl-second} → {@code meta.cache.hive.table.ttl-second} (schema/table meta, + * backs DESC)
      • + *
    • {@code partition.cache.ttl-second} → {@code meta.cache.hive.partition_names.ttl-second} (the + * partition-name list — legacy's {@code partition_values} entry; disabling it makes a newly-added + * partition visible without REFRESH)
      • + *
    + * Only the TTL is remapped (the sole knob the legacy keys exposed); {@code enable}/{@code capacity} have no + * legacy equivalent. If both the legacy and namespaced keys are present, the namespaced key wins + * ({@link CacheSpec#applyCompatibilityMap} contract). + */ + private static Map applyLegacyTtlCompatibility(Map props) { + Map compat = new HashMap<>(); + compat.put(LEGACY_SCHEMA_CACHE_TTL_SECOND, CacheSpec.metaCacheTtlKey(ENGINE, ENTRY_TABLE)); + compat.put(LEGACY_PARTITION_CACHE_TTL_SECOND, CacheSpec.metaCacheTtlKey(ENGINE, ENTRY_PARTITION_NAMES)); + return CacheSpec.applyCompatibilityMap(props, compat); + } + // ========== Cached reads ========== @Override diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java index 6c4302d542cdcb..8400138aa3ed99 100644 --- a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/CachingHmsClientTest.java @@ -351,6 +351,36 @@ public void perEntryPropertiesControlCaching() { Assertions.assertEquals(1, delegate.getPartitionsCalls); } + @Test + public void legacyTtlPropertiesControlCaching() { + // The legacy fe-core catalog knobs must still work after the SPI cutover: schema.cache.ttl-second maps + // onto the table/schema cache; partition.cache.ttl-second onto the partition-NAME list (legacy's + // partition_values), NOT the per-partition objects cache. Mirrors HiveExternalMetaCache's compat map; + // this is what test_hive_meta_cache's schema-cache and partition-cache sections exercise. + Map properties = props( + "schema.cache.ttl-second", "0", + "partition.cache.ttl-second", "0"); + RecordingHmsClient delegate = new RecordingHmsClient(); + CachingHmsClient cache = new CachingHmsClient(delegate, properties); + + cache.getTable("db", "t"); + cache.getTable("db", "t"); + // WHY: schema.cache.ttl-second=0 must disable the table/schema cache (backs DESC) — every call reloads. + Assertions.assertEquals(2, delegate.getTableCalls); + + cache.listPartitionNames("db", "t", -1); + cache.listPartitionNames("db", "t", -1); + // WHY: partition.cache.ttl-second=0 must disable the partition-name list — a newly-added partition is + // then visible without REFRESH. + Assertions.assertEquals(2, delegate.listPartitionNamesCalls); + + cache.getPartitions("db", "t", Arrays.asList("p=1")); + cache.getPartitions("db", "t", Arrays.asList("p=1")); + // WHY: the per-partition objects cache has NO legacy knob (fe-core mapped partition.cache only to the + // partition-values list), so it stays enabled — pins the faithful legacy mapping. + Assertions.assertEquals(1, delegate.getPartitionsCalls); + } + // ---- flush(db, table) ---- @Test From 9cd07c39eef5b85abcc83af9e723b0bc48185992 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 11:49:21 +0800 Subject: [PATCH 285/334] [fix](test) hive: make partition_values_tvf and ctas_to_doris idempotent Both suites left state behind that broke re-runs against a persistent env: - test_hive_partition_values_tvf created hive db partition_values_db with table partition_values_all_types and never dropped it, so the section-13 `drop database if exists partition_values_db` (no FORCE) failed on re-run with "Database ... is not empty". Add FORCE so the pre-drop cleans the leftover non-empty db. This matches the connector's dropDatabase(force) cascade and is the idiom already used across the hive suites; new fe-connector-hive and legacy HiveMetadataOps.dropDbImpl behave identically here (not a regression). - test_hive_ctas_to_doris created internal tables _4/_5/_6 (and the internal scratch db) without dropping them; the _5 CTAS has no IF NOT EXISTS and no drop-before, so a re-run hit "Table 'tb_test_hive_ctas_to_doris_5' already exists". Drop the internal scratch db at the start (mirroring the partition_values_tvf pattern) to clean all leftover internal tables. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../hive/test_hive_partition_values_tvf.groovy | 2 +- .../external_table_p0/hive/write/test_hive_ctas_to_doris.groovy | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/regression-test/suites/external_table_p0/hive/test_hive_partition_values_tvf.groovy b/regression-test/suites/external_table_p0/hive/test_hive_partition_values_tvf.groovy index bd50167725f549..2d6a202e7d301a 100644 --- a/regression-test/suites/external_table_p0/hive/test_hive_partition_values_tvf.groovy +++ b/regression-test/suites/external_table_p0/hive/test_hive_partition_values_tvf.groovy @@ -109,7 +109,7 @@ suite("test_hive_partition_values_tvf", "p0,external") { // 13. test all types of partition columns sql """switch ${catalog_name}""" - sql """drop database if exists partition_values_db"""; + sql """drop database if exists partition_values_db force"""; sql """create database partition_values_db""" sql """use partition_values_db""" diff --git a/regression-test/suites/external_table_p0/hive/write/test_hive_ctas_to_doris.groovy b/regression-test/suites/external_table_p0/hive/write/test_hive_ctas_to_doris.groovy index 6120fb8f01c104..76a1f0539f0090 100644 --- a/regression-test/suites/external_table_p0/hive/write/test_hive_ctas_to_doris.groovy +++ b/regression-test/suites/external_table_p0/hive/write/test_hive_ctas_to_doris.groovy @@ -54,6 +54,7 @@ suite("test_hive_ctas_to_doris", "p0,external") { qt_q01 """ select length(str1),length(str2) ,length(str3) from ${catalog}.${db_name}.${hive_tb} """ qt_q02 """ desc ${catalog}.${db_name}.${hive_tb} """ + sql """ drop database if exists internal.${db_name} """ sql """ create database if not exists internal.${db_name} """ // ctas for partition From d7d25288b2e2b10943d9d51b0a6bdadf68b970af Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 12:39:26 +0800 Subject: [PATCH 286/334] [fix](catalog) hive: exclude *.lzo.index sidecars and keep LZO non-splittable in SPI scan The SPI hive scan (HiveScanPlanProvider) lost two LZO behaviors that legacy fe-core HiveScanNode/HiveExternalMetaCache had, because HiveFileListingCache only strips _/.-prefixed hidden files and HiveFileFormat maps LZO text to the shared TEXT format: - *.lzo.index sidecar files were scanned as data. For an LZO text table only *.lzo files are data; the Hadoop-LZO *.lzo.index sidecar (starts with neither _ nor .) was read as an extra text row -> test_hive_lzo_text_format got count 6 instead of 5. Port legacy HiveUtil.isLzoDataFile: when the table uses an LZO input format, keep only files ending in .lzo in listAndSplitFiles. - LZO files were treated as splittable. TEXT.isSplittable() is true, but a .lzo stream cannot be decompressed from an arbitrary byte offset (legacy HiveUtil.isSplittable returned false for LZO). Mask splittable with !isLzo in planScan/planScanForPartitionBatch. Latent before (the test file is < one split), but wrong for large LZO files. isLzoInputFormat/isLzoDataFile are connector-local (fe-connector-hive has no fe-core dependency). LZO-ness is table-level (the connector reads every partition with the single table format). ACID is never LZO, so the transactional branch is unchanged. Added HiveScanPlanProviderLzoFilterTest. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveScanPlanProvider.java | 53 +++++++++-- .../HiveScanPlanProviderLzoFilterTest.java | 87 +++++++++++++++++++ 2 files changed, 135 insertions(+), 5 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanPlanProviderLzoFilterTest.java diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index c277b2039572ac..d8b6d1134a9487 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -42,6 +42,7 @@ import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; +import java.util.Locale; import java.util.Map; import java.util.Optional; @@ -128,7 +129,12 @@ public List planScan( hiveHandle.getInputFormat(), hiveHandle.getSerializationLib(), readHiveJsonInOneColumn(session), hiveHandle.isFirstColumnString()); long targetSplitSize = getTargetSplitSize(session); - boolean splittable = fileFormat.isSplittable(); + boolean isLzo = isLzoInputFormat(hiveHandle.getInputFormat()); + // LZO text is NOT splittable: a .lzo stream cannot be decompressed from an arbitrary byte offset. + // Legacy HiveUtil.isSplittable returned false for LZO; HiveFileFormat maps LZO text to TEXT (which + // reports splittable), so the LZO case must be masked out here. ACID is never LZO, so this leaves the + // transactional branch unchanged. + boolean splittable = fileFormat.isSplittable() && !isLzo; List ranges = new ArrayList<>(); @@ -145,7 +151,7 @@ public List planScan( HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; listAndSplitFiles(dbName, tableName, partition, partFormat, - splittable, targetSplitSize, fs, ranges); + splittable, isLzo, targetSplitSize, fs, ranges); } } @@ -236,7 +242,9 @@ public List planScanForPartitionBatch( hiveHandle.getInputFormat(), hiveHandle.getSerializationLib(), readHiveJsonInOneColumn(session), hiveHandle.isFirstColumnString()); long targetSplitSize = getTargetSplitSize(session); - boolean splittable = fileFormat.isSplittable(); + boolean isLzo = isLzoInputFormat(hiveHandle.getInputFormat()); + // LZO text is not splittable (see planScan); mask it out of the TEXT-derived splittable flag. + boolean splittable = fileFormat.isSplittable() && !isLzo; // Only the non-ACID path is reachable here (supportsBatchScan excludes transactional tables), so this // borrows the engine's per-catalog Doris FileSystem to list — no Hadoop Configuration is needed. FileSystem fs = context.getFileSystem(session); @@ -246,7 +254,7 @@ public List planScanForPartitionBatch( HiveFileFormat partFormat = partition.fileFormat != null ? partition.fileFormat : fileFormat; listAndSplitFiles(dbName, tableName, partition, partFormat, - splittable, targetSplitSize, fs, ranges); + splittable, isLzo, targetSplitSize, fs, ranges); } return ranges; } @@ -467,7 +475,7 @@ private List convertPartitions( */ private void listAndSplitFiles(String dbName, String tableName, PartitionScanInfo partition, HiveFileFormat fileFormat, - boolean splittable, long targetSplitSize, FileSystem fs, + boolean splittable, boolean isLzo, long targetSplitSize, FileSystem fs, List ranges) { List files; try { @@ -486,6 +494,13 @@ private void listAndSplitFiles(String dbName, String tableName, return; } for (HiveFileStatus file : files) { + // LZO text tables: only *.lzo files are data. Exclude *.lzo.index sidecars (and any other + // non-*.lzo entry), mirroring Hive's LzoTextInputFormat.listStatus() and legacy + // HiveExternalMetaCache's HiveUtil.isLzoDataFile filter. HiveFileListingCache strips only + // _/.-prefixed hidden files, so without this a *.lzo.index sidecar is read as an extra text row. + if (isLzo && !isLzoDataFile(file.getPath())) { + continue; + } splitFile(file.getPath(), file.getLength(), file.getModificationTime(), partition, fileFormat, splittable, targetSplitSize, null, null, ranges); } @@ -506,6 +521,34 @@ private static boolean isLocationNotFound(Throwable e) { return false; } + /** + * Whether {@code inputFormat} is one of the hadoop-lzo text InputFormat variants (the twitter + * {@code com.hadoop.compression.lzo.LzoTextInputFormat}, the anarres {@code com.hadoop.mapreduce. + * LzoTextInputFormat}, and the legacy {@code com.hadoop.mapred.DeprecatedLzoTextInputFormat}). Such tables + * read only {@code *.lzo} data files and must exclude {@code *.lzo.index} sidecars. Ports legacy + * {@code HiveUtil.isLzoInputFormat}; the {@code contains("lzo")} match (rather than exact class names) + * mirrors legacy and covers all variants alike. The connector cannot import fe-core, hence the local copy. + * Package-private for unit testing. + */ + static boolean isLzoInputFormat(String inputFormat) { + return inputFormat != null && inputFormat.toLowerCase(Locale.ROOT).contains("lzo"); + } + + /** + * For an LZO text InputFormat, only {@code *.lzo} files are data files; {@code *.lzo.index} sidecars and any + * other extension are metadata to exclude, mirroring Hive's {@code LzoTextInputFormat.listStatus()}. Ports + * legacy {@code HiveUtil.isLzoDataFile}. Package-private for unit testing. + */ + static boolean isLzoDataFile(String filePath) { + // Strip any object-store query string/fragment before matching the extension. + String path = filePath; + int q = path.indexOf('?'); + if (q >= 0) { + path = path.substring(0, q); + } + return path.toLowerCase(Locale.ROOT).endsWith(".lzo"); + } + /** * Splits a file into scan ranges based on target split size. * diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanPlanProviderLzoFilterTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanPlanProviderLzoFilterTest.java new file mode 100644 index 00000000000000..f6bc88ff421a51 --- /dev/null +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanPlanProviderLzoFilterTest.java @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hive; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Tests the LZO file-filtering helpers on {@link HiveScanPlanProvider}. These reproduce legacy + * {@code HiveExternalMetaCache}'s {@code HiveUtil.isLzoInputFormat}/{@code isLzoDataFile} filtering, which was + * lost at the SPI cutover: {@link HiveFileListingCache} only strips {@code _}/{@code .}-prefixed hidden files, + * so for an LZO text table the {@code *.lzo.index} sidecar (which starts with neither) would otherwise be read + * as an extra text row — the exact failure {@code test_hive_lzo_text_format} caught (count 6 instead of 5). + */ +public class HiveScanPlanProviderLzoFilterTest { + + @Test + public void testIsLzoInputFormatRecognizesAllVariants() { + // The three hadoop-lzo text InputFormat variants must all be recognized. + Assertions.assertTrue( + HiveScanPlanProvider.isLzoInputFormat("com.hadoop.compression.lzo.LzoTextInputFormat")); + Assertions.assertTrue( + HiveScanPlanProvider.isLzoInputFormat("com.hadoop.mapreduce.LzoTextInputFormat")); + Assertions.assertTrue( + HiveScanPlanProvider.isLzoInputFormat("com.hadoop.mapred.DeprecatedLzoTextInputFormat")); + } + + @Test + public void testIsLzoInputFormatRejectsNonLzo() { + // Plain text / parquet / orc tables are not LZO and must NOT trigger the sidecar filter. + Assertions.assertFalse( + HiveScanPlanProvider.isLzoInputFormat("org.apache.hadoop.mapred.TextInputFormat")); + Assertions.assertFalse(HiveScanPlanProvider.isLzoInputFormat( + "org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat")); + Assertions.assertFalse(HiveScanPlanProvider.isLzoInputFormat(null)); + } + + @Test + public void testIsLzoDataFileExcludesIndexSidecar() { + // Only *.lzo is data; the *.lzo.index sidecar (and any other extension) must be excluded. + Assertions.assertTrue(HiveScanPlanProvider.isLzoDataFile( + "hdfs://ns/user/doris/preinstalled_data/text_lzo/part-m-00000.lzo")); + Assertions.assertFalse(HiveScanPlanProvider.isLzoDataFile( + "hdfs://ns/user/doris/preinstalled_data/text_lzo/part-m-00000.lzo.index")); + // Case-insensitive extension match. + Assertions.assertTrue(HiveScanPlanProvider.isLzoDataFile("hdfs://ns/dir/part-m-00000.LZO")); + Assertions.assertFalse(HiveScanPlanProvider.isLzoDataFile("hdfs://ns/dir/part-m-00000.txt")); + } + + @Test + public void testIsLzoDataFileStripsObjectStoreQueryString() { + // An object-store URI may carry a signed query string; the extension match must ignore it. + Assertions.assertTrue(HiveScanPlanProvider.isLzoDataFile("s3://bucket/dir/part.lzo?sig=abc&exp=123")); + Assertions.assertFalse( + HiveScanPlanProvider.isLzoDataFile("s3://bucket/dir/part.lzo.index?sig=abc&exp=123")); + } + + @Test + public void testLzoTextDetectsAsSplittableTextSoMustBeMasked() { + // A LZO text table's InputFormat resolves to the TEXT format, whose isSplittable() is true — the enum + // alone would (wrongly) split a .lzo stream at byte boundaries, which cannot be decompressed from an + // arbitrary offset. This is exactly why planScan masks splittable with !isLzoInputFormat(...): without + // the mask, a large LZO file would be split and produce garbage. + HiveFileFormat fmt = HiveFileFormat.detect( + "com.hadoop.mapreduce.LzoTextInputFormat", + "org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe", + false, false); + Assertions.assertEquals(HiveFileFormat.TEXT, fmt); + Assertions.assertTrue(fmt.isSplittable()); + Assertions.assertTrue(HiveScanPlanProvider.isLzoInputFormat("com.hadoop.mapreduce.LzoTextInputFormat")); + } +} From 63121ec8ad3e498f6df80aa1c9b94d15f7ed7288 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 14:19:56 +0800 Subject: [PATCH 287/334] [fix](catalog) hudi: wire catalog s3.* credentials into FE metaClient Hadoop conf The FE-side Hudi metaClient built its Hadoop Configuration by copying only hadoop./fs./dfs./hive.-prefixed catalog properties, silently dropping the Doris storage-style object-store credentials (s3.access_key / s3.secret_key / s3.endpoint / use_path_style). S3AFileSystem then fell back to the default AWS provider chain and failed with NoAuthWithAWSException, so every hudi metadata read (schema load, hudi_meta timeline, partition listing) surfaced as "Hudi metadata operation failed for catalog". Mirror the iceberg/paimon connectors: overlay the catalog's bound fe-filesystem StorageProperties.toHadoopProperties().toHadoopConfigurationMap() (which translates s3.access_key -> fs.s3a.access.key, etc.) into both buildHadoopConf() sites, before the existing raw fs./dfs./hadoop. passthrough so inline user keys still win. HudiConnectorMetadata gains an overloaded constructor carrying the translated map (the 3-arg constructor delegates with an empty map, so existing same-loader unit tests are unchanged). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/connector/hudi/HudiConnector.java | 3 ++- .../connector/hudi/HudiConnectorMetadata.java | 14 ++++++++++++ .../connector/hudi/HudiScanPlanProvider.java | 22 +++++++++++++++++++ 3 files changed, 38 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java index 4dc740c174c20f..8474c003010fc7 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnector.java @@ -87,7 +87,8 @@ public HudiConnector(Map properties, ConnectorContext context) { @Override public ConnectorMetadata getMetadata(ConnectorSession session) { - return new HudiConnectorMetadata(getOrCreateClient(), properties, metaClientExecutor()); + return new HudiConnectorMetadata(getOrCreateClient(), properties, metaClientExecutor(), + HudiScanPlanProvider.storageHadoopConfig(context)); } /** diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index b75fa00e365b8e..cd5060fda21dc8 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -150,12 +150,23 @@ public class HudiConnectorMetadata implements ConnectorMetadata { private final Map properties; // Runs the metaClient-touching partition/snapshot work under the plugin UGI doAs + TCCL pin (see R4). private final HudiMetaClientExecutor metaClientExecutor; + // Canonical fs.s3a.*/hadoop.* storage config translated from the catalog's fe-filesystem StorageProperties, + // overlaid into the FE-side metaClient's Hadoop conf so its S3AFileSystem has object-store credentials (the + // raw s3.access_key/… aliases are not Hadoop keys). Empty in same-loader unit tests (which override + // getSchemaFromMetaClient or never touch S3). See HudiScanPlanProvider#storageHadoopConfig. + private final Map storageHadoopConfig; public HudiConnectorMetadata(HmsClient hmsClient, Map properties, HudiMetaClientExecutor metaClientExecutor) { + this(hmsClient, properties, metaClientExecutor, Collections.emptyMap()); + } + + public HudiConnectorMetadata(HmsClient hmsClient, Map properties, + HudiMetaClientExecutor metaClientExecutor, Map storageHadoopConfig) { this.hmsClient = hmsClient; this.properties = properties; this.metaClientExecutor = metaClientExecutor; + this.storageHadoopConfig = storageHadoopConfig; } // ========== ConnectorSchemaOps ========== @@ -985,6 +996,9 @@ private String detectHudiTableType(HmsTableInfo tableInfo) { private Configuration buildHadoopConf() { Configuration conf = new Configuration(); + // Storage credentials (fs.s3a.* …) first so an inline user fs./dfs./hadoop. key still wins below; see + // storageHadoopConfig field. Without it the metaClient's S3AFileSystem cannot read hoodie.properties. + storageHadoopConfig.forEach(conf::set); for (Map.Entry entry : properties.entrySet()) { String key = entry.getKey(); if (key.startsWith("hadoop.") || key.startsWith("fs.") diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index cb3aa39b30b76b..668785ef36a9ac 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -56,6 +56,7 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; @@ -881,6 +882,11 @@ static String detectFileFormat(String filePath) { private Configuration buildHadoopConf() { Configuration conf = new Configuration(); + // Overlay the catalog's bound fe-filesystem storage config (s3.access_key -> fs.s3a.access.key, etc.) + // FIRST so an inline user fs./dfs./hadoop. key still wins in the loop below. Without this the FE-side + // HoodieTableMetaClient's S3AFileSystem gets no object-store credentials and cannot read + // .hoodie/hoodie.properties (NoAuthWithAWSException). Mirrors iceberg/paimon buildStorageHadoopConfig. + storageHadoopConfig(context).forEach(conf::set); for (Map.Entry entry : properties.entrySet()) { String key = entry.getKey(); if (key.startsWith("hadoop.") || key.startsWith("fs.") @@ -890,4 +896,20 @@ private Configuration buildHadoopConf() { } return conf; } + + /** + * The canonical object-store/HDFS Hadoop config (fs.s3a.* / fs.oss.* / hadoop.* …) translated from the + * catalog's bound fe-filesystem {@code StorageProperties}. This is the ONLY source of the S3/OSS credentials + * the FE-side {@link HoodieTableMetaClient} needs — the raw catalog aliases (s3.access_key, …) are NOT Hadoop + * keys and S3AFileSystem never reads them. Empty for a null context (offline unit tests) or a catalog with no + * typed storage. Mirrors {@code IcebergConnector.buildStorageHadoopConfig} / {@code PaimonConnector}. + */ + static Map storageHadoopConfig(ConnectorContext context) { + Map merged = new HashMap<>(); + if (context != null) { + context.getStorageProperties().forEach(sp -> + sp.toHadoopProperties().ifPresent(h -> merged.putAll(h.toHadoopConfigurationMap()))); + } + return merged; + } } From 705d8cba5c0378c944f4cc6dedee298e13ea4082 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 14:20:57 +0800 Subject: [PATCH 288/334] [fix](catalog) hudi: bundle parquet-hadoop/parquet-avro child-first for FE schema read Resolving a Hudi table's Avro schema reads a data file's parquet footer: TableSchemaResolver -> ParquetUtils.readAvroSchema -> readSchema -> ParquetFileReader.readFooter(Configuration, Path), then converts the parquet MessageType to Avro via parquet-avro's AvroSchemaConverter. The hudi plugin bundles no parquet, and hive-catalog-shade carries only a RELOCATED (org.apache.paimon.shade.*) parquet, so the un-relocated ParquetFileReader resolved from the parent 'app' loader while ParquetUtils and Configuration came from the child loader (org.apache.hadoop is child-first, from the bundled hadoop-common). readFooter's Configuration parameter then split across two loaders -> java.lang.LinkageError (loader constraint violation on org.apache.hadoop.conf.Configuration), previously masked by the S3 credential failure fixed in the preceding commit. Co-locate parquet child-first (same FIX-PAIMON-HADOOP-CLASSLOADER pattern the plugin already uses for hadoop-aws), keeping ParquetFileReader + Configuration in one loader. parquet-hadoop transitively pulls column/common/encoding/format-structures/jackson; parquet-avro supplies AvroSchemaConverter (+ parquet-variant); versions are BOM-managed (${parquet.version} = 1.17.0). Bundled via plugin-zip.xml's runtime dependencySet. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- fe/fe-connector/fe-connector-hudi/pom.xml | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/fe/fe-connector/fe-connector-hudi/pom.xml b/fe/fe-connector/fe-connector-hudi/pom.xml index b9d5456d9fde53..8ffe151c3a8705 100644 --- a/fe/fe-connector/fe-connector-hudi/pom.xml +++ b/fe/fe-connector/fe-connector-hudi/pom.xml @@ -93,6 +93,29 @@ under the License. hudi-hadoop-mr + + + org.apache.parquet + parquet-hadoop + + + org.apache.parquet + parquet-avro + + org.apache.logging.log4j log4j-api From 748f6781d81bf0f806a580ec237e495675961fff Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 14:23:46 +0800 Subject: [PATCH 289/334] [fix](catalog) hudi: bundle hadoop-aws + AWS SDK v2 child-first for FE s3a:// timeline reads The hudi connector opens the table base path on the FE to read the .hoodie timeline (HudiScanPlanProvider.buildMetaClient -> HoodieTableMetaClient.getStorage -> FileSystem.get). hadoop-common is bundled child-first (org.apache.hadoop is not in the plugin's parent-first allowlist), so bundling it WITHOUT hadoop-aws left FileSystem in the child loader while S3AFileSystem resolved from the parent 'app' loader, and FileSystem.createFileSystem's cast threw "S3AFileSystem cannot be cast to FileSystem". Co-locate hadoop-aws child-first (same FIX-PAIMON-HADOOP-CLASSLOADER split iceberg/paimon carry) so the cast succeeds, plus the AWS SDK v2 modules S3A needs at runtime (s3, apache-client, s3-transfer-manager) so its ExecutionAttribute static registers against the child sdk-core instead of colliding with fe-core's and poisoning S3 for the whole JVM. Also updates the adjacent parquet comment to note hadoop-aws now sits below it. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- fe/fe-connector/fe-connector-hudi/pom.xml | 43 ++++++++++++++++++++++- 1 file changed, 42 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-hudi/pom.xml b/fe/fe-connector/fe-connector-hudi/pom.xml index 8ffe151c3a8705..0c85c499310180 100644 --- a/fe/fe-connector/fe-connector-hudi/pom.xml +++ b/fe/fe-connector/fe-connector-hudi/pom.xml @@ -104,7 +104,7 @@ under the License. bundled hadoop-common) — readFooter's Configuration parameter then splits across two loaders and the JVM throws a LinkageError (loader constraint violation on org.apache.hadoop.conf.Configuration). Co-locating parquet child-first here — the SAME FIX-PAIMON-HADOOP-CLASSLOADER pattern as hadoop-aws - — keeps ParquetFileReader + Configuration in one loader. parquet-hadoop pulls its + below — keeps ParquetFileReader + Configuration in one loader. parquet-hadoop pulls its parquet-column/common/encoding/format-structures/jackson siblings transitively; versions are BOM-managed (${parquet.version}). --> @@ -116,6 +116,47 @@ under the License. parquet-avro + + + org.apache.hadoop + hadoop-aws + + + + + software.amazon.awssdk + s3 + + + software.amazon.awssdk + apache-client + + + software.amazon.awssdk + s3-transfer-manager + + org.apache.logging.log4j log4j-api From a26eaf46b9f32672c42beaa08cd33b0098033e19 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 15:42:28 +0800 Subject: [PATCH 290/334] [fix](catalog) hudi: normalize s3a:// native scan path to s3:// for BE native reader MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Native COW / MOR-no-log reads over an s3a:// (MinIO) warehouse failed with BE "[INVALID_ARGUMENT]Invalid S3 URI: s3a://..." — BE's S3URI (s3_uri.cpp) accepts only s3/http/https. The new fe-connector hudi path shipped the raw HMS s3a:// location to TFileRangeDesc.path without the s3a->s3 rewrite that legacy HudiScanNode performed via the 2-arg LocationPath.of(path, storagePropertiesMap). Fix: normalize the NATIVE range .path() via context.normalizeStorageUri (mirrors the iceberg/paimon normalizeUri seam; fe-core does no property parsing) at ALL THREE native emitters: - collectCowSplits (snapshot COW) - buildMorRange (snapshot + incremental MOR no-log native slice) - COWIncrementalRelation.collectSplits (COW @incr; found by the design red-team) The JNI THudiFileDesc base/data/delta-log paths stay raw s3a:// (Hadoop S3AFileSystem wants the s3a scheme). Threaded as a UnaryOperator through incrementalRanges so the static package-private test seams (buildMorRange / incrementalRanges / collectSplits) stay intact; a null context (offline UT) preserves the raw URI. Design: plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-design.md (red-team wf_4e4ec1d7-d4f GO_WITH_FIXES: third-site blocker folded in). Tests: HudiIncrementalPlanScanTest +2 (COW @incr + MOR no-log native path s3a->s3), existing call sites threaded with identity; fe-connector-hudi 25/25 green, 0 checkstyle violations. e2e (user-run): the 10 native-signature hudi p2 suites must stop throwing Invalid S3 URI (full green also needs FIX-hudi-s3a-jni-creds). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../hudi/COWIncrementalRelation.java | 7 +- .../hudi/EmptyIncrementalRelation.java | 3 +- .../connector/hudi/HudiScanPlanProvider.java | 36 ++++- .../connector/hudi/IncrementalRelation.java | 9 +- .../hudi/MORIncrementalRelation.java | 4 +- .../hudi/HudiIncrementalPlanScanTest.java | 61 +++++++- .../hudi/HudiIncrementalRelationTest.java | 4 +- plan-doc/task-list.md | 24 ++- .../FIX-hudi-s3a-native-scheme-design.md | 143 ++++++++++++++++++ 9 files changed, 260 insertions(+), 31 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-design.md diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java index 20dc2eeef7b139..8d461cf4b09d8f 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/COWIncrementalRelation.java @@ -43,6 +43,7 @@ import java.util.List; import java.util.Map; import java.util.function.Consumer; +import java.util.function.UnaryOperator; import java.util.stream.Collectors; /** @@ -182,7 +183,7 @@ public List collectFileSlices() { } @Override - public List collectSplits() { + public List collectSplits(UnaryOperator nativePathNormalizer) { IncrementalRelation.checkNotFullTableScan(fullTableScan); if (filteredRegularFullPaths.isEmpty() && filteredMetaBootstrapFullPaths.isEmpty()) { return Collections.emptyList(); @@ -197,7 +198,9 @@ public List collectSplits() { Consumer generatorSplit = baseFile -> { HoodieWriteStat stat = fileToWriteStat.get(baseFile); splits.add(new HudiScanRange.Builder() - .path(baseFile) + // Native COW @incr range: normalize scheme (s3a->s3) for BE's native reader. The raw baseFile + // is a full HMS path anchored on metaClient.getBasePath(); BE's S3URI rejects s3a. + .path(nativePathNormalizer.apply(baseFile)) .start(0) // length + fileSize both from the write stat, matching legacy HudiSplit(0, size, size, ...). .length(stat.getFileSizeInBytes()) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java index 70ea92557db809..e7e2c8dc381d69 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/EmptyIncrementalRelation.java @@ -21,6 +21,7 @@ import java.util.Collections; import java.util.List; +import java.util.function.UnaryOperator; /** * The {@code @incr} relation for an empty completed timeline: it selects nothing. Connector-internal port of @@ -43,7 +44,7 @@ public List collectFileSlices() { } @Override - public List collectSplits() { + public List collectSplits(UnaryOperator nativePathNormalizer) { return Collections.emptyList(); } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 668785ef36a9ac..099ae64ea8d599 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -63,6 +63,7 @@ import java.util.Map; import java.util.Optional; import java.util.function.Function; +import java.util.function.UnaryOperator; import java.util.stream.Collectors; /** @@ -256,7 +257,8 @@ public List planScan( if (hudiHandle.getBeginInstant() != null) { IncrementalRelation relation = buildIncrementalRelation(metaClient, conf, hudiHandle, isCow); Optional> incrementalRanges = incrementalRanges(relation, isCow, forceJni, - basePath, inputFormat, serdeLib, columnNames, columnTypes, partitionFieldNames(metaClient)); + basePath, inputFormat, serdeLib, columnNames, columnTypes, partitionFieldNames(metaClient), + this::normalizeNativeUri); if (incrementalRanges.isPresent()) { LOG.info("Hudi incremental scan planning: {}.{} window=({}, {}] splits={}", hudiHandle.getDbName(), hudiHandle.getTableName(), @@ -399,6 +401,20 @@ public void populateScanLevelParams(TFileScanRangeParams params, Map schemaIdResolver) { fsView.getLatestMergedFileSlicesBeforeOrOn(partitionPath, queryInstant) .forEach(fileSlice -> ranges.add(buildMorRange(fileSlice, partValues, queryInstant, - forceJni, basePath, inputFormat, serdeLib, columnNames, columnTypes, schemaIdResolver))); + forceJni, basePath, inputFormat, serdeLib, columnNames, columnTypes, schemaIdResolver, + this::normalizeNativeUri))); } /** @@ -467,7 +484,7 @@ private void collectMorSplits( static HudiScanRange buildMorRange(FileSlice fileSlice, Map partValues, String jniInstant, boolean forceJni, String basePath, String inputFormat, String serdeLib, List columnNames, List columnTypes, - Function schemaIdResolver) { + Function schemaIdResolver, UnaryOperator nativePathNormalizer) { Optional baseFileOpt = fileSlice.getBaseFile().toJavaOptional(); String filePath = baseFileOpt.map(BaseFile::getPath).orElse(""); long fileSize = baseFileOpt.map(BaseFile::getFileSize).orElse(0L); @@ -487,7 +504,7 @@ static HudiScanRange buildMorRange(FileSlice fileSlice, Map part ? logs.get(0) : filePath; HudiScanRange.Builder builder = new HudiScanRange.Builder() - .path(agencyPath) + .path(nativePathNormalizer.apply(agencyPath)) .start(0) .length(fileSize) .fileSize(fileSize) @@ -568,13 +585,16 @@ private static IncrementalRelation buildIncrementalRelation(HoodieTableMetaClien */ static Optional> incrementalRanges(IncrementalRelation relation, boolean isCow, boolean forceJni, String basePath, String inputFormat, String serdeLib, - List columnNames, List columnTypes, List partitionFieldNames) { + List columnNames, List columnTypes, List partitionFieldNames, + UnaryOperator nativePathNormalizer) { if (relation.fallbackFullTableScan()) { return Optional.empty(); } List ranges = new ArrayList<>(); if (isCow) { - ranges.addAll(relation.collectSplits()); + // COW @incr yields native ranges directly; normalize their scheme (s3a->s3) for BE's native reader + // (COWIncrementalRelation.collectSplits builds .path() from the raw HMS base path). + ranges.addAll(relation.collectSplits(nativePathNormalizer)); return Optional.of(ranges); } String endTs = relation.getEndTs(); @@ -582,7 +602,7 @@ static Optional> incrementalRanges(IncrementalRelation Map partValues = parsePartitionValues(fileSlice.getPartitionPath(), partitionFieldNames); // @incr lists the LATEST schema (no per-file schema_id dict on the incremental path) -> null resolver. ranges.add(buildMorRange(fileSlice, partValues, endTs, forceJni, - basePath, inputFormat, serdeLib, columnNames, columnTypes, null)); + basePath, inputFormat, serdeLib, columnNames, columnTypes, null, nativePathNormalizer)); } return Optional.of(ranges); } diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java index 6c5ffffd08bc5e..169ec386cf017b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/IncrementalRelation.java @@ -24,6 +24,7 @@ import java.util.List; import java.util.Map; +import java.util.function.UnaryOperator; /** * Selects the files an {@code @incr(...)} incremental read must scan over a resolved {@code (begin, end]} @@ -55,8 +56,12 @@ interface IncrementalRelation { /** Merged file slices at {@code endTs} for the MOR path (COW throws {@link UnsupportedOperationException}). */ List collectFileSlices(); - /** Native base-file ranges for the COW path (MOR throws {@link UnsupportedOperationException}). */ - List collectSplits(); + /** + * Native base-file ranges for the COW path (MOR throws {@link UnsupportedOperationException}). + * {@code nativePathNormalizer} rewrites each range's raw storage URI to BE's canonical scheme + * (s3a->s3) for the native reader; implementations that emit no native ranges here ignore it. + */ + List collectSplits(UnaryOperator nativePathNormalizer); /** * Whether the window fell back to a full-table scan (archived instant / missing file). The scan planner diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java index 09503e575cdc22..eb9ddb07f502ba 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/MORIncrementalRelation.java @@ -37,6 +37,7 @@ import java.util.Collections; import java.util.List; import java.util.Map; +import java.util.function.UnaryOperator; import java.util.stream.Collectors; import java.util.stream.Stream; @@ -171,7 +172,8 @@ public List collectFileSlices() { } @Override - public List collectSplits() { + public List collectSplits(UnaryOperator nativePathNormalizer) { + // MOR emits ranges via collectFileSlices()/buildMorRange, not here; the normalizer is irrelevant. throw new UnsupportedOperationException(); } } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java index 11dc67a1d8dc15..8d3a0040da03ec 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalPlanScanTest.java @@ -30,6 +30,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.function.UnaryOperator; /** * Tests the OFFLINE-verifiable core of the INC-3 incremental {@code planScan} wiring: the pure static helpers @@ -66,7 +67,7 @@ public void fallbackFullTableScanDegradesToSnapshotScanWithoutCollecting() { fallback.collectFileSlicesThrows = true; Optional> result = HudiScanPlanProvider.incrementalRanges( fallback, /*isCow*/ true, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, - Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH, UnaryOperator.identity()); Assertions.assertFalse(result.isPresent(), "fallbackFullTableScan must signal degrade to the snapshot scan (Optional.empty)"); } @@ -84,7 +85,7 @@ public void cowRoutesToCollectSplitsAndNeverCallsCollectFileSlices() { cow.collectFileSlicesThrows = true; Optional> result = HudiScanPlanProvider.incrementalRanges( cow, /*isCow*/ true, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, - Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH, UnaryOperator.identity()); Assertions.assertTrue(result.isPresent()); Assertions.assertEquals(1, result.get().size()); Assertions.assertSame(native1, result.get().get(0), "COW ranges must be the relation's collectSplits output"); @@ -101,7 +102,7 @@ public void cowIncrementalIgnoresForceJniAndStaysNative() { cow.collectFileSlicesThrows = true; Optional> result = HudiScanPlanProvider.incrementalRanges( cow, /*isCow*/ true, /*forceJni*/ true, BASE_PATH, INPUT_FORMAT, SERDE, - Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH, UnaryOperator.identity()); Assertions.assertTrue(result.isPresent()); Assertions.assertSame(native1, result.get().get(0), "COW incremental must stay native even under force_jni (never call collectFileSlices)"); @@ -122,7 +123,7 @@ public void morRoutesToCollectFileSlicesMappedAtEndTs() { mor.collectSplitsThrows = true; Optional> result = HudiScanPlanProvider.incrementalRanges( mor, /*isCow*/ false, /*forceJni*/ true, BASE_PATH, INPUT_FORMAT, SERDE, - Arrays.asList("c1"), Arrays.asList("int"), YEAR_MONTH); + Arrays.asList("c1"), Arrays.asList("int"), YEAR_MONTH, UnaryOperator.identity()); Assertions.assertTrue(result.isPresent()); Assertions.assertEquals(1, result.get().size()); HudiScanRange range = (HudiScanRange) result.get().get(0); @@ -144,7 +145,7 @@ public void morEmptySliceListYieldsEmptyRangesButTakesMorBranch() { mor.collectSplitsThrows = true; Optional> result = HudiScanPlanProvider.incrementalRanges( mor, /*isCow*/ false, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, - Collections.emptyList(), Collections.emptyList(), YEAR_MONTH); + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH, UnaryOperator.identity()); Assertions.assertTrue(result.isPresent()); Assertions.assertTrue(result.get().isEmpty()); } @@ -160,7 +161,8 @@ public void buildMorRangeStampsJniInstantAndMetadataForForceJniSlice() { Map partValues = HudiScanPlanProvider.parsePartitionValues( slice.getPartitionPath(), YEAR_MONTH); HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, partValues, END_TS, /*forceJni*/ true, - BASE_PATH, INPUT_FORMAT, SERDE, Arrays.asList("c1"), Arrays.asList("int"), p -> 7L); + BASE_PATH, INPUT_FORMAT, SERDE, Arrays.asList("c1"), Arrays.asList("int"), p -> 7L, + UnaryOperator.identity()); Assertions.assertEquals("jni", range.getFileFormat()); Assertions.assertEquals(END_TS, range.getProperties().get("hudi.instant_time")); Assertions.assertEquals(SERDE, range.getProperties().get("hudi.serde")); @@ -177,13 +179,48 @@ public void buildMorRangeDowngradesNoLogSliceToNativeWithoutForceJni() { "s3://b/t/year=2024/month=01/fileid-1_0_20240101000000.parquet"); HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, Collections.emptyMap(), END_TS, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, - Arrays.asList("c1"), Arrays.asList("int"), p -> 7L); + Arrays.asList("c1"), Arrays.asList("int"), p -> 7L, UnaryOperator.identity()); Assertions.assertEquals("parquet", range.getFileFormat(), "a no-log slice without force_jni must downgrade to the native parquet reader"); // C4c: a native downgraded slice carries the per-file schema_id from the resolver (native field-id path). Assertions.assertEquals("7", range.getProperties().get("hudi.schema_id")); } + // ── native path scheme normalization (FIX-hudi-s3a-native-scheme) ──────────────────────────────────────── + + @Test + public void cowIncrementalNormalizesNativePathScheme() { + // The COW @incr branch must thread incrementalRanges' normalizer into relation.collectSplits so the native + // range path is rewritten s3a->s3 for BE's native S3 reader (which rejects s3a). Kills a mutation that drops + // the normalizer or passes identity: the range path would stay s3a:// and BE would throw "Invalid S3 URI". + FakeRelation cow = new FakeRelation(); + cow.rawCowPath = "s3a://datalake/warehouse/t/f1.parquet"; + cow.collectFileSlicesThrows = true; + Optional> result = HudiScanPlanProvider.incrementalRanges( + cow, /*isCow*/ true, /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, + Collections.emptyList(), Collections.emptyList(), YEAR_MONTH, + s -> s.replace("s3a://", "s3://")); + Assertions.assertTrue(result.isPresent()); + HudiScanRange range = (HudiScanRange) result.get().get(0); + Assertions.assertEquals("s3://datalake/warehouse/t/f1.parquet", range.getPath().orElse(null), + "COW @incr native range path must be scheme-normalized (s3a->s3) via the threaded normalizer"); + } + + @Test + public void buildMorRangeNormalizesNativeNoLogSlicePathScheme() { + // A no-log MOR slice reads native; buildMorRange must apply the normalizer to the base-file path (agencyPath) + // so BE's native reader gets s3://, not the raw s3a:// from the Hudi SDK. The JNI metadata paths are a + // separate concern (this slice reads native, so none are set). + FileSlice slice = baseOnlySlice("year=2024/month=01", + "s3a://datalake/warehouse/t/year=2024/month=01/fileid-1_0_20240101000000.parquet"); + HudiScanRange range = HudiScanPlanProvider.buildMorRange(slice, Collections.emptyMap(), END_TS, + /*forceJni*/ false, BASE_PATH, INPUT_FORMAT, SERDE, + Arrays.asList("c1"), Arrays.asList("int"), null, s -> s.replace("s3a://", "s3://")); + Assertions.assertEquals("s3://datalake/warehouse/t/year=2024/month=01/fileid-1_0_20240101000000.parquet", + range.getPath().orElse(null), + "a native no-log MOR slice's range path must be scheme-normalized (s3a->s3) by buildMorRange"); + } + // ── helpers ──────────────────────────────────────────────────────────────────────────────────────────── /** A MOR file slice with a single base file and no delta logs. */ @@ -201,6 +238,10 @@ private static final class FakeRelation implements IncrementalRelation { private String endTs = END_TS; private boolean collectSplitsThrows; private boolean collectFileSlicesThrows; + // When set, collectSplits builds ONE native range whose path is the normalizer applied to this raw URI — + // stands in for the real COWIncrementalRelation.collectSplits (which needs a live metaClient), letting the + // test verify incrementalRanges THREADS its normalizer into the COW collectSplits call. + private String rawCowPath; @Override public List collectFileSlices() { @@ -211,10 +252,14 @@ public List collectFileSlices() { } @Override - public List collectSplits() { + public List collectSplits(UnaryOperator nativePathNormalizer) { if (collectSplitsThrows) { throw new UnsupportedOperationException("collectSplits must not be called on this route"); } + if (rawCowPath != null) { + return Collections.singletonList(new HudiScanRange.Builder() + .path(nativePathNormalizer.apply(rawCowPath)).fileFormat("parquet").build()); + } return splits; } diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java index ef41bcc74e7ba9..f8183aebf53d16 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiIncrementalRelationTest.java @@ -26,6 +26,7 @@ import java.util.Collections; import java.util.HashMap; import java.util.Map; +import java.util.function.UnaryOperator; /** * Tests the OFFLINE-verifiable surface of the ported {@code @incr} IncrementalRelation family (INC-2): the pure @@ -194,7 +195,8 @@ public void emptyRelationSelectsNothing() { // The empty-timeline relation selects no files on either shape and never falls back. Its end bound is the // legacy "000" sentinel. Guards against a mutation returning non-empty / a non-"000" bound. EmptyIncrementalRelation empty = new EmptyIncrementalRelation(); - Assertions.assertTrue(empty.collectSplits().isEmpty(), "empty relation must select no splits"); + Assertions.assertTrue(empty.collectSplits(UnaryOperator.identity()).isEmpty(), + "empty relation must select no splits"); Assertions.assertTrue(empty.collectFileSlices().isEmpty(), "empty relation must select no file slices"); Assertions.assertFalse(empty.fallbackFullTableScan(), "empty relation never falls back to a full scan"); Assertions.assertEquals("000", empty.getEndTs(), "empty relation end bound is the legacy \"000\""); diff --git a/plan-doc/task-list.md b/plan-doc/task-list.md index b5d1746834de39..e72eaa389f4cab 100644 --- a/plan-doc/task-list.md +++ b/plan-doc/task-list.md @@ -1,11 +1,19 @@ -# Task List — CI 973411 paimon-SPI regression fixes +# Task List — hudi BE-scan s3a:// failures (14 p2 suites) -Source RCA: `memory/catalog-spi-ci-973411-4fails-rca.md` + workflow `wf_e1c3d93c-22c` (adversarially verified). -Build 973411, HEAD e1d6f88. All 4 test files are byte-identical to master ⇒ all 4 are SPI-migration regressions. +Root cause: the new fe-connector hudi BE-scan path under-threads S3 storage config (parity gap vs +legacy `HudiScanNode`). Two independent wiring gaps, fixed independently. RCA: recon workflow +`wf_67162858-b79` + inline verification (this session, 2026-07-12). -- [x] FIX-1 — test_create_paimon_table: paimon-over-HMS create-db classloader split (PaimonCatalogFactory.assembleHiveConf) — DONE (16/16 UT, checkstyle clean) -- [x] FIX-2 — test_mysql_mtmv: connector-null NPE during mv_infos scan (PluginDrivenMvccExternalTable.materializeLatest) — DONE (36/36 UT, RED reproduced exact NPE, checkstyle clean) -- [x] FIX-3 — test_paimon_mtmv: Duplicated p_NULL partition naming (ListPartitionItem.toPartitionKeyDesc) — DONE (2/2 new UT + MTMV 10/10, RED reproduced p_NULL collision, checkstyle clean) -- [x] FIX-4 — test_paimon_table_meta_cache: restore paimon table cache, 2 axes (PaimonConnector snapshot cache + Connector SPI invalidate/schemaTtl + fe-core overlay/refresh wiring) — DONE (cache 5/5, override 4/4, Mvcc 40/40, fe-core compiles + FIX-2/3 guards 36/2, checkstyle clean; e2e docker-gated) +- [ ] FIX-hudi-s3a-native-scheme — native reader `[INVALID_ARGUMENT]Invalid S3 URI: s3a://...`: + connector emits the raw HMS `s3a://` path into the native range `.path()`; must normalize + `s3a://`→`s3://` via `context.normalizeStorageUri` (mirror iceberg/paimon `normalizeUri`). + fe-connector-hudi only. JNI `THudiFileDesc` paths stay raw `s3a://` (S3AFileSystem wants s3a). +- [ ] FIX-hudi-s3a-jni-creds — JNI Hudi scanner `NoAuthWithAWSException: No AWS Credentials`: + `HudiScanPlanProvider.getScanNodeProperties` never emits the Hadoop-canonical `fs.s3a.*` + creds under the `location.` prefix; add `storageHadoopConfig(context)` emission so BE's JNI + Hadoop conf receives `fs.s3a.access.key/secret.key/endpoint`. fe-connector-hudi only. -Order: 1 → 2 → 3 → 4 (smallest/lightest module first; #4 largest). TDD per fix, independent commit each. +E2E = the existing 14 failing suites under `regression-test/suites/external_table_p2/hudi/` +(no new suites needed; they are the regression gate). Run both fixes together, then re-run all 14. + +Order: native-scheme → jni-creds (independent; either order works). TDD per fix, independent commit. diff --git a/plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-design.md b/plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-design.md new file mode 100644 index 00000000000000..88d60df8f558fc --- /dev/null +++ b/plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-design.md @@ -0,0 +1,143 @@ +# FIX-hudi-s3a-native-scheme + +## Problem + +All hudi p2 suites that read COW/native slices from an `s3a://` (MinIO) warehouse fail with: + +``` +[INVALID_ARGUMENT]Invalid S3 URI: s3a://datalake/warehouse/regression_hudi.db/.../xxx.parquet +``` + +10 of the 14 failing suites (catalog, partition_prune, orc_tables, schema_change, +full_schema_change, timestamp, runtime_filter, mtmv, rewrite_mtmv, olap_rewrite_mtmv). + +## Root Cause + +BE's native S3 reader parses the file path via `S3URI::parse()` (`be/src/util/s3_uri.cpp:44-78`), +which accepts ONLY the schemes `s3`, `http`, `https`. The scheme `s3a` falls to the `else` at +line 70 → `Invalid S3 URI`. It fails at URI parse, before any credential/network use. + +The new fe-connector hudi path hands BE the raw HMS location with the `s3a://` scheme intact. +There are **three** native `.path()` emitters in the connector (exhaustive grep), all raw `s3a://`: + +- `HudiScanPlanProvider.collectCowSplits:420` → `.path(filePath)` where `filePath = + baseFile.getPath()` (raw `s3a://`). [snapshot COW] +- `HudiScanPlanProvider.buildMorRange:490` → `.path(agencyPath)` (raw `s3a://` base file / first log). + [snapshot MOR + incremental MOR] +- `COWIncrementalRelation.collectSplits:200` → `.path(baseFile)` (raw `s3a://` absolute path anchored + on `metaClient.getBasePath()`). [**COW `@incr`**] — reached via `incrementalRanges`' COW branch + (`HudiScanPlanProvider.java:576-578`) which returns `relation.collectSplits()` verbatim. Exercised by + `test_hudi_incremental.groovy:89-90` (`set force_jni_scanner=false` → `isCow=true`). Found by the + design red-team (`wf_4e4ec1d7-d4f`); missed in the first draft. + +All three flow to BE identically: + +- `PluginDrivenSplit.buildPath:78` wraps it via the **single-arg** `LocationPath.of(pathStr)`, which + stores the raw string and never calls `validateAndNormalizeUri` (`LocationPath.java:163-169`). +- `FileQueryScanNode.createFileRangeDesc:568` ships that verbatim `s3a://` to `TFileRangeDesc.path`. +- `SchemaTypeMapper:56` maps `s3a`→`FILE_S3`, so BE picks the native S3 reader → rejects `s3a`. + +The `s3a://`→`s3://` rewrite lives only in `S3PropertyUtils.validateAndNormalizeUri` +(`S3PropertyUtils.java:172-190`), reachable only via the *normalizing* `LocationPath.of` overload, +which the plugin split path bypasses. + +Legacy parity: the still-present `HudiScanNode` built its snapshot splits via the normalizing 2-arg +overload `LocationPath.of(filePath, hmsTable.getStoragePropertiesMap())` (`HudiScanNode.java:425,590`), +which DID rewrite `s3a`→`s3`. The COW-incremental site is NOT covered by that parity argument: legacy +`COWIncrementalRelation.java:218` used the **single-arg** (non-normalizing) `LocationPath.of(baseFile)`, +so legacy COW `@incr` over `s3a` was latently broken too (a pre-existing gap, not a regression). Net: +this is a migration parity gap for the snapshot paths + a pre-existing latent gap for COW `@incr`, +both fixed here. + +Sibling connectors already do the connector-side normalization the plugin path (correctly, per the +"fe-core holds no property parsing" rule) does NOT do: +- `IcebergScanPlanProvider:834-836` → `.path(normalizeUri(rawDataPath, vendedToken))`. +- `PaimonScanPlanProvider:547` → `.path(normalizeUri(file.path(), vendedToken))`. +Both delegate to the SPI seam `ConnectorContext.normalizeStorageUri` (`ConnectorContext.java:205`). +Hudi is the only object-store connector missing this call. + +## Design + +Mirror iceberg/paimon: normalize the **native** range `.path()` on the connector side via +`context.normalizeStorageUri`. Hudi is HMS-based (no REST vended credentials), so the single-arg +overload suffices — no `vendedToken`. + +CRITICAL invariant: normalize ONLY the native range path field (`HudiScanRange.path`, which becomes +`TFileRangeDesc.path`, consumed by the native S3 reader). The JNI reader's `THudiFileDesc` paths +(`basePath` / `dataFilePath` / `deltaLogs`) MUST stay raw `s3a://` — Hadoop's `S3AFileSystem` +*wants* the `s3a` scheme. Those are set from separate builder args (`.basePath`, `.dataFilePath`, +`.deltaLogs`) and are untouched by this fix. + +Normalizing the `.path()` field is safe even for a JNI-format slice: BE consumes `TFileRangeDesc.path` +via `S3URI` only for native-format ranges; for `FORMAT_JNI` it reads via `THudiFileDesc` and does not +parse the range path (proven by the JNI suites failing at `NoAuthWithAWSException` from `base_path`, +NOT at `Invalid S3 URI` from the range path). So normalizing the native path field unconditionally is +correct and harmless. + +## Implementation Plan + +`fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java`: + +1. Add private instance helper (mirror paimon `normalizeUri`, single-arg): + ```java + private String normalizeNativeUri(String rawUri) { + return context != null ? context.normalizeStorageUri(rawUri) : rawUri; + } + ``` + Null-context guard preserves offline unit tests (no live context). + +2. `collectCowSplits` (instance): `.path(filePath)` → `.path(normalizeNativeUri(filePath))`. + +3. `buildMorRange` (static) + `incrementalRanges` (static): thread a + `java.util.function.UnaryOperator nativePathNormalizer` param. + - `buildMorRange`: `.path(agencyPath)` → `.path(nativePathNormalizer.apply(agencyPath))`. + - `incrementalRanges`: accept the param, pass it to BOTH branches — the MOR branch's + `buildMorRange` call AND the COW branch's `relation.collectSplits(...)` call. + - Instance callers (`collectMorSplits:449`, `planScan`'s `incrementalRanges:258`) pass + `this::normalizeNativeUri`. + - Unit tests pass `UnaryOperator.identity()`. + + Rationale for a threaded function (not making the methods non-static): the methods are + deliberately `static` + package-private for offline unit testing with hand-built `FileSlice`s + (per their javadoc). Threading a `UnaryOperator` keeps them static and lets tests pass + `identity()`, mirroring how paimon threads `vendedToken` through its build methods. + +4. COW-incremental third site: change the `IncrementalRelation.collectSplits()` interface method to + `collectSplits(UnaryOperator nativePathNormalizer)`. + - `COWIncrementalRelation.collectSplits`: apply at `:200` → `.path(nativePathNormalizer.apply(baseFile))`. + - `MORIncrementalRelation.collectSplits` (throws `UnsupportedOperationException`) and + `EmptyIncrementalRelation.collectSplits` (returns `emptyList`): accept the param, ignore it (they + emit no native ranges via `collectSplits`). + - `incrementalRanges`' COW branch (`:577`): `ranges.addAll(relation.collectSplits(nativePathNormalizer))`. + - Chosen over ctor-injection into `COWIncrementalRelation` because threading through + `incrementalRanges` makes the wiring unit-testable at the existing fake-`IncrementalRelation` seam + (the exact place the bug lived) and keeps the normalizer symmetric across COW + MOR. + +Add import: `java.util.function.UnaryOperator` (to `HudiScanPlanProvider`, `IncrementalRelation`, +`COWIncrementalRelation`, `MORIncrementalRelation`, `EmptyIncrementalRelation`). + +## Risk Analysis + +- **JNI paths accidentally normalized** → would break MOR merge reads. Mitigated: only `.path()` is + normalized; `.basePath/.dataFilePath/.deltaLogs` are distinct builder args, left raw. Guard with a + UT asserting a JNI range's `THudiFileDesc` paths remain `s3a://` while the native `.path()` is `s3://`. +- **Non-s3a schemes** (hdfs://, plain paths) → `normalizeStorageUri` is a no-op / passthrough for + non-S3-compatible schemes (same primitive paths that already work for iceberg/paimon on HDFS). No + regression for the EMR/HDFS hudi config. +- **Null context (offline UT)** → helper returns raw; identity in tests. No NPE. +- **Test call-site churn** (`HudiIncrementalPlanScanTest`, ~8 refs) → mechanical `identity()` add; + compile failure is the safety net if one is missed. + +## Test Plan + +### Unit Tests (fe-connector-hudi) +- New: assert `collectCowSplits` / `buildMorRange` native `.path()` is normalized `s3a://`→`s3://` + when a fake `ConnectorContext.normalizeStorageUri` rewrites the scheme; and that a JNI slice's + `THudiFileDesc.base_path/data_file_path/delta_logs` stay raw `s3a://`. +- Existing `HudiIncrementalPlanScanTest` (updated for the new param) must stay green. +- `mvn -pl fe/fe-connector/fe-connector-hudi test`; checkstyle clean. + +### E2E (user-run, docker-gated) +The 10 native-signature suites under `regression-test/suites/external_table_p2/hudi/` must stop +throwing `Invalid S3 URI`. Full green also requires FIX-hudi-s3a-jni-creds (suites with +`force_jni_scanner=true` sub-queries, e.g. test_hudi_catalog, exercise both paths). From 19a8098b3e7ed9722b6c59f9867598c444f6afbf Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 15:49:01 +0800 Subject: [PATCH 291/334] [fix](catalog) hudi: emit translated fs.s3a.* creds to BE for the JNI Hudi scanner MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MOR-with-log hudi reads via the JNI scanner over an s3a:// (MinIO) warehouse failed with NoAuthWithAWSException "No AWS Credentials provided" (also any force_jni_scanner=true sub-query). The BE JNI reader builds the Java scanner's Hadoop Configuration from TFileScanRangeParams.properties (hadoop_conf.-prefixed, stripped back by HadoopHudiJniScanner). getScanNodeProperties emitted only (1) getBackendStorageProperties() BE-canonical AWS_* (which S3AFileSystem ignores) and (2) a raw passthrough of the catalog's s3. aliases (also ignored) — never the Hadoop-canonical fs.s3a.access.key/secret.key/endpoint. The translation (storageHadoopConfig -> StorageProperties.toHadoopProperties) existed but fed only the FE metaClient conf, not the BE scan; legacy getLocationProperties merged backendStorageProperties + the translated hadoop props, and the new path dropped the second half. Fix: emit storageHadoopConfig(context) under the location. prefix in getScanNodeProperties, after the AWS_* canonical set (1) and before the raw passthrough (2) so a user-inline fs./hadoop. key still wins (mirrors buildHadoopConf precedence). Native FILE_S3 reader is unaffected (reads AWS_* only; the extra fs.s3a.* are ignored, same as the s3. aliases already emitted). storageHadoopConfig null-guards. Design: plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-design.md (red-team wf_4e4ec1d7-d4f SOUND; linchpin — the HMS/hudi ConnectorContext storage IS bound — confirmed). Tests: HudiBackendDescriptorTest +1 (typed StorageProperties -> fs.s3a.* emitted under location., catalog carrying only s3. aliases = the real failing scenario). fe-connector-hudi full suite 175/175 green, 0 checkstyle. Also completes the task list (both hudi s3a fixes) + FIX-A summary. e2e (user-run): the 4 JNI-signature suites must stop throwing NoAuthWithAWSException. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hudi/HudiScanPlanProvider.java | 9 ++ .../hudi/HudiBackendDescriptorTest.java | 77 +++++++++++++ plan-doc/task-list.md | 6 +- .../designs/FIX-hudi-s3a-jni-creds-design.md | 102 ++++++++++++++++++ .../designs/FIX-hudi-s3a-jni-creds-summary.md | 36 +++++++ .../FIX-hudi-s3a-native-scheme-summary.md | 35 ++++++ 6 files changed, 263 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-design.md create mode 100644 plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-summary.md create mode 100644 plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-summary.md diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java index 099ae64ea8d599..9d65299509ea91 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java @@ -330,6 +330,15 @@ public Map getScanNodeProperties( if (context != null) { context.getBackendStorageProperties().forEach((k, v) -> props.put("location." + k, v)); } + // (1b) Hadoop-canonical object-store config (fs.s3a.* / fs.oss.* / resolved hadoop.*/dfs.*) TRANSLATED + // from the catalog's typed StorageProperties, for the Hudi JNI reader's own Hadoop FileSystem. + // S3AFileSystem reads ONLY fs.s3a.* — never the AWS_* canonical keys (1) nor the s3. aliases (2) — so + // without this a private s3a warehouse configured with the Doris s3. aliases throws + // NoAuthWithAWSException in the JNI scanner. This is the "hadoopProperties" half of legacy + // getLocationProperties' merge that the raw passthrough (2) alone does not reconstruct (the catalog + // carries s3. aliases, not fs.s3a. keys). Emitted BEFORE (2) so a user-inline fs./hadoop. key still + // wins (mirrors buildHadoopConf precedence); null context yields an empty map (offline / HDFS-only). + storageHadoopConfig(context).forEach((k, v) -> props.put("location." + k, v)); // (2) Hadoop-format passthrough for the Hudi JNI reader (its own Hadoop FileSystem: fs.s3a.* etc). // Emitted AFTER the canonical set so an overlapping hadoop key resolves to the catalog's explicit // value (legacy putAll order: backendStorageProperties then hadoopProperties). The s3./oss./cos./obs. diff --git a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java index 91201e834f6c68..ce531bdb9f145b 100644 --- a/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java +++ b/fe/fe-connector/fe-connector-hudi/src/test/java/org/apache/doris/connector/hudi/HudiBackendDescriptorTest.java @@ -18,6 +18,10 @@ package org.apache.doris.connector.hudi; import org.apache.doris.connector.spi.ConnectorContext; +import org.apache.doris.filesystem.FileSystemType; +import org.apache.doris.filesystem.properties.HadoopStorageProperties; +import org.apache.doris.filesystem.properties.StorageKind; +import org.apache.doris.filesystem.properties.StorageProperties; import org.apache.doris.thrift.TFileCompressType; import org.apache.doris.thrift.TTableDescriptor; import org.apache.doris.thrift.TTableType; @@ -27,6 +31,7 @@ import java.util.Collections; import java.util.HashMap; +import java.util.List; import java.util.Map; import java.util.Optional; @@ -92,6 +97,27 @@ null, new HudiTableHandle("db", "t", "s3://b/t", "COPY_ON_WRITE"), Assertions.assertEquals("hadoopAK", result.get("location.fs.s3a.access.key")); } + @Test + public void scanNodePropertiesEmitsTranslatedFsS3aFromTypedStorageForJniReader() { + // The real failing scenario (FIX-hudi-s3a-jni-creds): the catalog carries Doris s3. aliases + // (s3.access_key/...), NOT inline fs.s3a.* keys. The Hudi JNI reader's S3AFileSystem reads ONLY fs.s3a.*, + // so getScanNodeProperties must emit the TRANSLATED fs.s3a.* from the context's typed StorageProperties + // (storageHadoopConfig). Without it the JNI scanner throws NoAuthWithAWSException. Kills a mutation that + // drops the storageHadoopConfig emission (the pre-fix behavior: only s3. aliases were emitted, useless to + // S3AFileSystem). + Map catalogProps = new HashMap<>(); // s3. aliases only; NO inline fs.s3a.* key + catalogProps.put("s3.access_key", "aliasAK"); + HudiScanPlanProvider provider = new HudiScanPlanProvider(catalogProps, + contextWithHadoopStorage(Collections.singletonMap("fs.s3a.access.key", "translatedAK"))); + + Map result = provider.getScanNodeProperties( + null, new HudiTableHandle("db", "t", "s3a://b/t", "COPY_ON_WRITE"), + Collections.emptyList(), Optional.empty()); + + Assertions.assertEquals("translatedAK", result.get("location.fs.s3a.access.key"), + "translated fs.s3a.* from the catalog's typed StorageProperties must be emitted for the JNI reader"); + } + @Test public void adjustFileCompressTypeRemapsLz4FrameLikeLegacyInheritance() { // Legacy HudiScanNode extended HiveScanNode and INHERITED its LZ4FRAME -> LZ4BLOCK remap (hadoop writes @@ -124,4 +150,55 @@ public Map getBackendStorageProperties() { } }; } + + /** A context whose typed StorageProperties translate to the given Hadoop config (fs.s3a.* etc). */ + private static ConnectorContext contextWithHadoopStorage(Map hadoopConf) { + StorageProperties sp = new StorageProperties() { + @Override + public String providerName() { + return "s3"; + } + + @Override + public StorageKind kind() { + return null; + } + + @Override + public FileSystemType type() { + return null; + } + + @Override + public Map rawProperties() { + return Collections.emptyMap(); + } + + @Override + public Map matchedProperties() { + return Collections.emptyMap(); + } + + @Override + public Optional toHadoopProperties() { + return Optional.of(() -> hadoopConf); + } + }; + return new ConnectorContext() { + @Override + public String getCatalogName() { + return "c"; + } + + @Override + public long getCatalogId() { + return 0; + } + + @Override + public List getStorageProperties() { + return Collections.singletonList(sp); + } + }; + } } diff --git a/plan-doc/task-list.md b/plan-doc/task-list.md index e72eaa389f4cab..fc662c4577e826 100644 --- a/plan-doc/task-list.md +++ b/plan-doc/task-list.md @@ -4,14 +4,16 @@ Root cause: the new fe-connector hudi BE-scan path under-threads S3 storage conf legacy `HudiScanNode`). Two independent wiring gaps, fixed independently. RCA: recon workflow `wf_67162858-b79` + inline verification (this session, 2026-07-12). -- [ ] FIX-hudi-s3a-native-scheme — native reader `[INVALID_ARGUMENT]Invalid S3 URI: s3a://...`: +- [x] FIX-hudi-s3a-native-scheme — native reader `[INVALID_ARGUMENT]Invalid S3 URI: s3a://...`: connector emits the raw HMS `s3a://` path into the native range `.path()`; must normalize `s3a://`→`s3://` via `context.normalizeStorageUri` (mirror iceberg/paimon `normalizeUri`). fe-connector-hudi only. JNI `THudiFileDesc` paths stay raw `s3a://` (S3AFileSystem wants s3a). -- [ ] FIX-hudi-s3a-jni-creds — JNI Hudi scanner `NoAuthWithAWSException: No AWS Credentials`: + **DONE** commit `a26eaf46b9f` (3 native sites incl. COW @incr; 25/25 UT, 0 checkstyle). +- [x] FIX-hudi-s3a-jni-creds — JNI Hudi scanner `NoAuthWithAWSException: No AWS Credentials`: `HudiScanPlanProvider.getScanNodeProperties` never emits the Hadoop-canonical `fs.s3a.*` creds under the `location.` prefix; add `storageHadoopConfig(context)` emission so BE's JNI Hadoop conf receives `fs.s3a.access.key/secret.key/endpoint`. fe-connector-hudi only. + **DONE** (getScanNodeProperties emits translated fs.s3a.* under location.; +1 UT, 0 checkstyle). E2E = the existing 14 failing suites under `regression-test/suites/external_table_p2/hudi/` (no new suites needed; they are the regression gate). Run both fixes together, then re-run all 14. diff --git a/plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-design.md b/plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-design.md new file mode 100644 index 00000000000000..a4dd761653dd89 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-design.md @@ -0,0 +1,102 @@ +# FIX-hudi-s3a-jni-creds + +## Problem + +The 4 hudi p2 suites that read MOR-with-log slices via the JNI Hudi scanner (incremental, +schema_evolution, timetravel, snapshot) fail with: + +``` +[JNI_ERROR] failed to open hadoop hudi jni scanner: Cannot create a RecordReaderWrapper + | CAUSED BY: AccessDeniedException: s3a://datalake/warehouse/.../xxx.parquet: + org.apache.hadoop.fs.s3a.auth.NoAuthWithAWSException: No AWS Credentials provided by + TemporaryAWSCredentialsProvider SimpleAWSCredentialsProvider EnvironmentVariableCredentialsProvider + IAMInstanceCredentialsProvider +``` + +Also affects any `force_jni_scanner=true` sub-query in the native suites (e.g. test_hudi_catalog). + +## Root Cause + +The BE JNI reader builds the Java scanner's Hadoop `Configuration` from +`TFileScanRangeParams.properties`: for each entry it prefixes the key with `hadoop_conf.` +(`be/src/format/table/hudi_jni_reader.cpp:60-66`, `be/src/format_v2/jni/hudi_jni_reader.cpp:100-106`), +and `HadoopHudiJniScanner` strips `hadoop_conf.` back off into the `Configuration` +(`HadoopHudiJniScanner.java:124-127`). Those `properties` come from the connector's `location.`-prefixed +props (`PluginDrivenScanNode.getLocationProperties:603-613` → FILE_S3 branch of +`FileQueryScanNode`). So a connector property emitted as `location.fs.s3a.access.key` round-trips to a +Hadoop `Configuration` key `fs.s3a.access.key` in the JNI scanner. + +`HudiScanPlanProvider.getScanNodeProperties` emits under `location.` only: +1. `context.getBackendStorageProperties()` — BE-**native**-canonical creds (`AWS_ACCESS_KEY` / + `AWS_SECRET_KEY` / `AWS_ENDPOINT` / ...), which Hadoop's `S3AFileSystem` does NOT read. +2. a raw passthrough of the catalog's Doris aliases (`s3.access_key`, `s3.endpoint`, ...), which + its own comment admits are "harmless ... ignored by both readers". + +It never emits the Hadoop-canonical `fs.s3a.access.key/secret.key/endpoint`. Those are produced by +`storageHadoopConfig(context)` → `StorageProperties.toHadoopProperties()` (`HudiScanPlanProvider` +package-private helper, ~`:903-914`), but that map is used ONLY to build the FE-side +`HoodieTableMetaClient` conf (in `buildHadoopConf`), never emitted to BE. So the JNI `S3AFileSystem` +falls back to the default AWS provider chain with no keys → `NoAuthWithAWSException`. (The error +listing the DEFAULT chain, not a pinned `SimpleAWSCredentialsProvider`, confirms the JNI conf got +none of the catalog's s3 config.) + +Legacy parity: legacy `getLocationProperties` merged `backendStorageProperties` THEN the Hadoop +properties (`fs.s3a.*`) — the new path dropped the second half for the scan (kept it only for the FE +metaClient). + +## Design + +Emit the Hadoop-canonical object-store config (`storageHadoopConfig(context)`, already defined and +already used for the metaClient) under the `location.` prefix in `getScanNodeProperties`, so BE's JNI +Hadoop `Configuration` receives `fs.s3a.access.key/secret.key/endpoint/path.style.access` etc. + +Placement / precedence: emit it AFTER (1) `getBackendStorageProperties` (no key overlap — AWS_* vs +fs.s3a.*) and BEFORE (2) the raw passthrough loop, so a user-inline `fs.*`/`hadoop.*` key in the raw +catalog properties still overrides the translated value — matching the precedence in `buildHadoopConf` +(`storageHadoopConfig` first, inline `fs./dfs./hadoop.` keys override). `Map.put` last-wins gives this +ordering. + +Safety for the native reader: the native FILE_S3 reader consumes AWS_* canonical keys and ignores the +extra `fs.s3a.*` keys — same as it already ignores the `s3.` aliases emitted today. So the addition is +JNI-enabling and native-neutral. + +## Implementation Plan + +`fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiScanPlanProvider.java`, +in `getScanNodeProperties`, between emission (1) and (2): + +```java +// (1b) Hadoop-canonical object-store config (fs.s3a.* / fs.oss.* / ... resolved hadoop.*/dfs.*) +// translated from the catalog's typed StorageProperties, for the Hudi JNI reader's own Hadoop +// FileSystem. S3AFileSystem reads ONLY fs.s3a.* — never the AWS_* canonical keys (native) nor +// the s3. aliases — so without this a private s3a warehouse throws NoAuthWithAWSException in the +// JNI scanner. Emitted BEFORE the inline passthrough (2) so a user-inline fs./hadoop. key still +// wins (matches buildHadoopConf precedence). +storageHadoopConfig(context).forEach((k, v) -> props.put("location." + k, v)); +``` + +`storageHadoopConfig(ConnectorContext)` already null-guards a null context (returns empty). No import +or signature change needed. + +## Risk Analysis + +- **Native reader confusion** by extra `fs.s3a.*` keys → none; native reads AWS_* only, ignores the + rest (already true for the `s3.` aliases emitted today). +- **Key collision / wrong precedence** → resolved by placement before passthrough (2); user-inline + `fs.*` wins, mirroring `buildHadoopConf`. +- **Non-S3 (HDFS/EMR) catalogs** → `storageHadoopConfig` yields resolved `hadoop.*/dfs.*` (not + `fs.s3a.*`); already what the metaClient uses; no object-store keys leak. +- **Null context (offline UT)** → `storageHadoopConfig` returns empty; no-op. + +## Test Plan + +### Unit Tests (fe-connector-hudi) +- New: with a fake `ConnectorContext` whose `getStorageProperties().toHadoopProperties()` yields + `fs.s3a.access.key/secret.key/endpoint`, assert `getScanNodeProperties` emits them under + `location.fs.s3a.*`; and that a user-inline `fs.s3a.access.key` in raw properties overrides the + translated value (precedence). Assert AWS_* canonical (native) keys are still present. +- `mvn -pl fe/fe-connector/fe-connector-hudi test`; checkstyle clean. + +### E2E (user-run, docker-gated) +The 4 JNI-signature suites (incremental, schema_evolution, timetravel, snapshot) must stop throwing +`NoAuthWithAWSException`, and `force_jni_scanner=true` sub-queries in the native suites must succeed. diff --git a/plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-summary.md b/plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-summary.md new file mode 100644 index 00000000000000..c1edb6cf7be386 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-hudi-s3a-jni-creds-summary.md @@ -0,0 +1,36 @@ +# Summary — FIX-hudi-s3a-jni-creds + +## Problem +4 hudi p2 suites reading MOR-with-log slices via the JNI Hudi scanner (incremental, +schema_evolution, timetravel, snapshot) failed with +`NoAuthWithAWSException: No AWS Credentials provided ...` — and any `force_jni_scanner=true` +sub-query in the native suites. + +## Root Cause +The BE JNI reader builds the Java scanner's Hadoop `Configuration` from +`TFileScanRangeParams.properties` (prefixing each key `hadoop_conf.`, stripped back by +`HadoopHudiJniScanner`). `HudiScanPlanProvider.getScanNodeProperties` emitted under `location.` only +(1) `getBackendStorageProperties()` — BE-native-canonical `AWS_*` (which `S3AFileSystem` ignores) and +(2) a raw passthrough of the catalog's `s3.` aliases (also ignored). It never emitted the +Hadoop-canonical `fs.s3a.access.key/secret.key/endpoint`. The translation (`storageHadoopConfig` → +`StorageProperties.toHadoopProperties()`) existed but was used only for the FE-side +`HoodieTableMetaClient` conf, not for the BE scan. Legacy `getLocationProperties` merged +`backendStorageProperties` + the translated hadoop props; the new path dropped the second half. + +## Fix +Emit `storageHadoopConfig(context)` (the translated `fs.s3a.*`) under the `location.` prefix in +`getScanNodeProperties`, placed after the `AWS_*` canonical emission (1) and before the raw +passthrough (2) so a user-inline `fs.*`/`hadoop.*` key still wins (mirrors `buildHadoopConf` +precedence). Native reader is unaffected (it reads `AWS_*` only; extra `fs.s3a.*` are ignored, same as +the `s3.` aliases already emitted). One-line addition; `storageHadoopConfig` already null-guards. + +## Tests +`HudiBackendDescriptorTest` +1: a context whose typed `StorageProperties` translate to `fs.s3a.*`, +with the catalog carrying only `s3.` aliases (the real failing scenario), asserts +`getScanNodeProperties` emits `location.fs.s3a.access.key`. Full fe-connector-hudi suite green, +0 checkstyle. + +## Result +E2E (user-run): the 4 JNI-signature suites must stop throwing `NoAuthWithAWSException`, and +`force_jni_scanner=true` sub-queries in the native suites must succeed. Design red-team +`wf_4e4ec1d7-d4f` verdict SOUND (linchpin — HMS/hudi context storage IS bound — confirmed). diff --git a/plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-summary.md b/plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-summary.md new file mode 100644 index 00000000000000..6340724edbd99d --- /dev/null +++ b/plan-doc/tasks/designs/FIX-hudi-s3a-native-scheme-summary.md @@ -0,0 +1,35 @@ +# Summary — FIX-hudi-s3a-native-scheme + +## Problem +10 hudi p2 suites reading COW / MOR-no-log slices from an `s3a://` (MinIO) warehouse failed with +BE `[INVALID_ARGUMENT]Invalid S3 URI: s3a://...`. + +## Root Cause +BE's native S3 reader (`be/src/util/s3_uri.cpp` `S3URI::parse`) accepts only `s3`/`http`/`https`. +The new fe-connector hudi path shipped the raw HMS `s3a://` location straight to `TFileRangeDesc.path` +without the `s3a`→`s3` rewrite legacy `HudiScanNode` did via the 2-arg +`LocationPath.of(path, storagePropertiesMap)`. `PluginDrivenSplit.buildPath` uses the single-arg +(non-normalizing) `LocationPath.of`. + +## Fix +Normalize the NATIVE range `.path()` on the connector side via +`ConnectorContext.normalizeStorageUri` (mirrors iceberg/paimon `normalizeUri`; fe-core parses no +properties). Applied at all THREE native emitters: +- `HudiScanPlanProvider.collectCowSplits` (snapshot COW) +- `HudiScanPlanProvider.buildMorRange` (snapshot + incremental MOR no-log native slice) +- `COWIncrementalRelation.collectSplits` (COW `@incr` — third site found by the design red-team + `wf_4e4ec1d7-d4f`, missed in the first draft) + +Threaded as a `UnaryOperator` through `incrementalRanges` → `{collectSplits, buildMorRange}` +to keep the static package-private test seams. JNI `THudiFileDesc` base/data/delta-log paths stay raw +`s3a://` (Hadoop `S3AFileSystem` wants `s3a`). Null context (offline UT) preserves the raw URI. + +## Tests +`HudiIncrementalPlanScanTest` +2 (COW `@incr` + MOR no-log native path `s3a`→`s3`); existing static +call sites threaded with `UnaryOperator.identity()`; `HudiIncrementalRelationTest` updated. +fe-connector-hudi **25/25 green, 0 checkstyle**. + +## Result +Commit `a26eaf46b9f`. E2E (user-run): the 10 native-signature hudi p2 suites must stop throwing +`Invalid S3 URI`. Full green of the 14 also needs FIX-hudi-s3a-jni-creds (suites with +`force_jni_scanner=true` sub-queries hit both paths). From b905b7098a7489fe92ceda66a39d339b2f5b1884 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 15:50:13 +0800 Subject: [PATCH 292/334] [doc](catalog) HANDOFF: hudi s3a BE-scan 2 fixes DONE (native scheme + JNI creds), e2e pending Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index b1d86fc6cad1a0..4a3749c8acbeec 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,6 +5,15 @@ --- +# 🆕 本 session（hudi s3a BE-scan 读）= 2 fix DONE，e2e 待用户自跑 + +> 用户按最新代码重跑 = **14 个 hudi p2 suite 全失败**（唯 `test_hudi_meta` 过）。两类错误签名、同一主题：新 fe-connector hudi BE-scan 路径**没把 catalog 的 S3 storageProperties 接到 BE**（相对 legacy `HudiScanNode` 的 parity gap，**非环境/非文案**）。两个独立接线缺口，各独立 commit： +> - **native `Invalid S3 URI: s3a://` (10 suite)** = `a26eaf46b9f`：BE `S3URI` 只认 `s3/http/https`，连接器把原始 `s3a://` 直发 `TFileRangeDesc.path`。修=连接器侧 `context.normalizeStorageUri` 归一化**native** range `.path()`（镜像 iceberg/paimon），**三处** native emitter：`collectCowSplits`/`buildMorRange`/**`COWIncrementalRelation.collectSplits`(COW @incr，红队补的第三处)**；JNI `THudiFileDesc` 路径留原始 `s3a://`。 +> - **JNI `NoAuthWithAWSException` (4 suite)** = `19a8098b3e7`：`getScanNodeProperties` 只发 AWS_*(native)+s3. 别名，漏发翻译后的 `fs.s3a.*`。修=补 `storageHadoopConfig(context)` 以 `location.` 前缀下发。 +> **验证**：fe-connector-hudi 全模块 **175/175 UT 绿、0 checkstyle**（含 3 个新 RED-then-GREEN 单测）。设计+摘要 `designs/FIX-hudi-s3a-{native-scheme,jni-creds}-{design,summary}.md`、task-list、红队 `wf_67162858-b79`(RCA)+`wf_4e4ec1d7-d4f`(设计对抗 GO_WITH_FIXES)。**e2e 待用户自跑（须先重编/重部署 FE 连接器 jar）**：14 个 `external_table_p2/hudi/` suite 全绿门；`test_hudi_catalog` 等含 `force_jni_scanner=true` 子查询者两条路径都要过。 + +--- + # 🆕 下一个 session 起步 = HIVEFS-8（全量 build + e2e 收尾）或复核修复 backlog（batch 3）；**hive e2e round-2 代码已全部 DONE**（query_cache 本轮 `c9a86337906`，仅剩 ENV + 用户自跑 e2e） > **本轮(round-2)**：用户按最新代码重跑 = **37 suite 测试、22 失败**。**全 22 已逐个 HEAD-verified 根因 + 对抗复核**（44-agent workflow `wf_c9def639-775`）。定案：**15 CODE_FIX · 1 TEST_ALIGN · 2 ENV · 4 USER_DECISION（用户已全裁决 → 均转 CODE_FIX）**。**完整三元表 + 逐 suite 修复指针 + 用户裁决** = `plan-doc/reviews/hive-e2e-r2-triage-2026-07-11.md`（**起步必读**，比下面的 round-1 R1-R12 表更新更准）。 From f22f8223033a869539d9fa17cc0e4258d17fb18a Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 16:32:39 +0800 Subject: [PATCH 293/334] =?UTF-8?q?[fix](catalog)=20hive:=20object-store?= =?UTF-8?q?=20native=20read=20=E2=80=94=20normalize=20s3a=20scheme=20+=20e?= =?UTF-8?q?mit=20canonical=20BE=20creds?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Plain-hive tables on an object-store warehouse (s3a://oss://cos://) would fail BE-side native reads — the same latent gap class that broke hudi (found via the hudi design red-team; user asked to fix hive too). Two gaps in fe-connector-hive, both unexercised locally (hive docker regression uses hdfs://; hive-s3 suites are p2/real-cloud): 1. Scheme: splitFile -> newRangeBuilder .path(filePath) shipped the raw s3a:// path to TFileRangeDesc.path; BE's native S3 reader (s3_uri.cpp S3URI::parse) accepts only s3/http/https. Legacy HiveScanNode normalized via the 2-arg LocationPath.of; hive's OWN write path already used context.normalizeStorageUri — the read path never got it. 2. Creds (a legacy regression): getScanNodeProperties emitted only the raw s3./fs. catalog aliases; BE's native FILE_S3 reader reads ONLY canonical AWS_ACCESS_KEY/SECRET/ENDPOINT (s3_util.cpp:146-148), so a private bucket 403s. Legacy getLocationProperties() returned getBackendStorageProperties() (canonical); the new path dropped it. (Hive has no JNI scanner -> no fs.s3a./JNI-creds analog.) Fix (fe-core untouched; no source-specific code): - normalizeNativeUri(raw) = context != null ? context.normalizeStorageUri(raw) : raw. Applied to the native data-file path in splitFile (covers regular + ACID data files) and to the ACID BE-facing paths (acidLocation + encodeDeleteDeltas delete-delta dir, threaded a UnaryOperator). Files are still LISTED with the raw scheme (Hadoop S3AFileSystem wants s3a); only BE-facing copies are normalized. - getScanNodeProperties emits context.getBackendStorageProperties() (canonical AWS_* / resolved hadoop.*/dfs.*) under location. before the raw passthrough (inline fs./hadoop. still wins). Restores legacy behavior; for HDFS this equals what legacy sent (parity → HDFS suites preserved). Design: plan-doc/tasks/designs/FIX-hive-s3a-read-design.md. Tests: HiveScanBatchModeTest +2 (RED-able: range path s3a->s3 via planScanForPartitionBatch; canonical AWS_* emission). Full fe-connector-hive suite 328/328 green, 0 checkstyle. UNIT-VERIFIED ONLY — object-store e2e needs a real s3/oss hive table (docker is HDFS); the HDFS hive suites are the parity regression guard for the creds change. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveScanPlanProvider.java | 46 ++++++++++-- .../connector/hive/HiveScanBatchModeTest.java | 66 +++++++++++++++++ plan-doc/task-list.md | 14 +++- .../tasks/designs/FIX-hive-s3a-read-design.md | 70 +++++++++++++++++++ .../designs/FIX-hive-s3a-read-summary.md | 34 +++++++++ 5 files changed, 223 insertions(+), 7 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-hive-s3a-read-design.md create mode 100644 plan-doc/tasks/designs/FIX-hive-s3a-read-summary.md diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java index d8b6d1134a9487..18557976b7f86d 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveScanPlanProvider.java @@ -45,6 +45,7 @@ import java.util.Locale; import java.util.Map; import java.util.Optional; +import java.util.function.UnaryOperator; /** * Scan plan provider for Hive tables. @@ -309,9 +310,11 @@ private void planAcidScan(ConnectorSession session, HiveTableHandle handle, // merge-on-read reader). Insert-only tables use the write-id snapshot to pick files but // store plain data files, so their splits stay plain hive — matching legacy AcidUtil, // which sets acidInfo only when isFullAcid. - String acidLocation = isFullAcid ? partition.location : null; + // BE-facing ACID paths (delete-delta dir + partition location) also carry the raw HMS scheme; the + // BE transactional reader opens them via the same native S3 factory, so normalize s3a://->s3://. + String acidLocation = isFullAcid ? normalizeNativeUri(partition.location) : null; List encodedDeltas = isFullAcid - ? encodeDeleteDeltas(state.getDeleteDeltas()) : null; + ? encodeDeleteDeltas(state.getDeleteDeltas(), this::normalizeNativeUri) : null; for (FileEntry dataFile : state.getDataFiles()) { splitFile(dataFile.location().uri(), dataFile.length(), dataFile.modificationTime(), partition, partFormat, splittable, @@ -339,10 +342,13 @@ public void releaseReadTransaction(String queryId) { } /** Encodes each delete-delta as {@code "dir|file1,file2"} for {@link HiveScanRange.Builder#acidInfo}. */ - private static List encodeDeleteDeltas(List deltas) { + private static List encodeDeleteDeltas(List deltas, + UnaryOperator nativePathNormalizer) { List encoded = new ArrayList<>(deltas.size()); for (HiveAcidUtil.DeleteDelta delta : deltas) { - encoded.add(delta.getDirectoryLocation() + "|" + // Normalize the delete-delta directory scheme (s3a://->s3://) for BE's native reader; the file names + // are bare names appended after '|'. + encoded.add(nativePathNormalizer.apply(delta.getDirectoryLocation()) + "|" + String.join(",", delta.getFileNames())); } return encoded; @@ -381,7 +387,19 @@ public Map getScanNodeProperties( props.put(PROP_PATH_PARTITION_KEYS, String.join(",", partKeys)); } - // Location properties (Hadoop/S3 config for BE file access) + // Location properties (Hadoop/S3 config for BE file access). + // (1) BE-canonical static credentials (AWS_* for object stores, resolved hadoop.*/dfs.* for HDFS): BE's + // native (FILE_S3) reader understands ONLY these canonical keys — it reads AWS_ACCESS_KEY / + // AWS_SECRET_KEY / AWS_ENDPOINT (s3_util.cpp), NOT the raw s3.access_key/... aliases — so without + // them a private bucket 403s. Legacy HiveScanNode.getLocationProperties() emitted exactly this + // (hmsTable.getBackendStorageProperties()); the new path had dropped it. Empty for a null context + // (offline tests) or a credential-less warehouse. + if (context != null) { + context.getBackendStorageProperties().forEach((k, v) -> props.put(PROP_LOCATION_PREFIX + k, v)); + } + // (2) Raw catalog aliases + inline fs./hadoop./dfs. keys. Emitted AFTER the canonical set so a user-inline + // fs./hadoop. key wins; the s3./oss./cos./obs. aliases are harmless to BE (ignored by the native + // reader) but kept so no configured key is dropped. for (Map.Entry entry : catalogProperties.entrySet()) { String key = entry.getKey(); if (isLocationProperty(key)) { @@ -549,6 +567,20 @@ static boolean isLzoDataFile(String filePath) { return path.toLowerCase(Locale.ROOT).endsWith(".lzo"); } + /** + * Normalizes a raw HMS storage URI into BE's canonical scheme for a BE-facing native reader path + * (e.g. {@code s3a://}/{@code oss://}/{@code cos://} → {@code s3://}), delegating to the engine seam + * {@link ConnectorContext#normalizeStorageUri(String)} — the connector cannot import fe-core's + * {@code LocationPath}. BE's native S3 file factory (S3URI) accepts ONLY {@code s3://}, so an un-normalized + * {@code s3a://} scan path fails the native read with "Invalid S3 URI". Mirrors iceberg/paimon/hudi and hive's + * OWN write path ({@code HiveWritePlanProvider}); legacy {@code HiveScanNode} normalized via the 2-arg + * {@code LocationPath.of(path, storagePropertiesMap)}. Non-object-store schemes (hdfs://, local) pass through + * unchanged. A null context (offline unit tests) preserves the raw URI. + */ + private String normalizeNativeUri(String rawUri) { + return context != null ? context.normalizeStorageUri(rawUri) : rawUri; + } + /** * Splits a file into scan ranges based on target split size. * @@ -562,6 +594,10 @@ private void splitFile(String filePath, long fileSize, long modTime, PartitionSc if (fileSize == 0) { return; } + // Normalize the BE-facing native data-file path scheme (s3a://->s3://): the connector lists files via the + // engine FileSystem with the raw scheme (Hadoop wants s3a), but BE's native S3 reader rejects s3a. ACID + // delete-delta / partition paths are normalized separately at their emit sites. + filePath = normalizeNativeUri(filePath); if (!splittable || targetSplitSize <= 0 || fileSize <= targetSplitSize) { // Single range for the whole file diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java index c3832a947e66ab..148fc11dfaee3b 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveScanBatchModeTest.java @@ -141,6 +141,72 @@ public void planScanForPartitionBatchResolvesOnlyTheBatch() { Assertions.assertNull(lister.callsPerLocation.get("year=2024/month=02")); } + // ===== object-store native read (FIX-hive-s3a: scheme normalization + canonical creds) ===== + + @Test + public void nativeScanRangePathNormalizedS3aToS3() { + // BE's native S3 reader rejects the s3a scheme (S3URI accepts only s3/http/https). The connector lists + // files with the raw scheme (Hadoop wants s3a) but must normalize the BE-facing range path to s3://. + // MUTATION: dropping the splitFile normalization leaves the range path s3a:// -> BE "Invalid S3 URI". + HiveFileListingCache.DirectoryLister s3aLister = (location, fs) -> + new ArrayList<>(Collections.singletonList( + new HiveFileStatus("s3a://bucket/db/t/p/000000_0", 10L, 1L))); + FakeConnectorContext normCtx = new FakeConnectorContext() { + @Override + public String normalizeStorageUri(String rawUri) { + return rawUri == null ? null : rawUri.replace("s3a://", "s3://"); + } + }; + HiveScanPlanProvider provider = new HiveScanPlanProvider(new FakeHmsClient(), + Collections.emptyMap(), normCtx, new HiveReadTransactionManager(), + new HiveFileListingCache(Collections.emptyMap(), s3aLister)); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat(PARQUET_INPUT_FORMAT) + .serializationLib(PARQUET_SERDE) + .partitionKeyNames(PART_KEYS) + .prunedPartitions(Collections.singletonList(part("year=2024/month=01"))) + .build(); + + List ranges = provider.planScanForPartitionBatch( + new FakeSession(), handle, Collections.emptyList(), + Optional.empty(), -1L, Collections.singletonList("year=2024/month=01")); + + Assertions.assertEquals(1, ranges.size()); + Assertions.assertEquals("s3://bucket/db/t/p/000000_0", + ((HiveScanRange) ranges.get(0)).getPath().orElse(null), + "native hive range path must be scheme-normalized s3a->s3 for BE's native reader"); + } + + @Test + public void scanNodePropertiesEmitsCanonicalCredsForNativeReader() { + // BE's native FILE_S3 reader reads ONLY AWS_ACCESS_KEY/AWS_SECRET_KEY/AWS_ENDPOINT (s3_util.cpp), never the + // raw s3.access_key alias. getScanNodeProperties must emit the BE-canonical creds + // (getBackendStorageProperties) like legacy HiveScanNode.getLocationProperties did; without them a private + // bucket 403s. MUTATION: dropping the canonical emission (pre-fix: only raw s3. aliases were emitted). + FakeConnectorContext credCtx = new FakeConnectorContext() { + @Override + public Map getBackendStorageProperties() { + return Collections.singletonMap("AWS_ACCESS_KEY", "canonAK"); + } + }; + HiveScanPlanProvider provider = new HiveScanPlanProvider(new FakeHmsClient(), + Collections.singletonMap("s3.access_key", "aliasAK"), credCtx, + new HiveReadTransactionManager(), + new HiveFileListingCache(Collections.emptyMap(), new CountingLister())); + HiveTableHandle handle = new HiveTableHandle.Builder("db", "t", HiveTableType.HIVE) + .inputFormat(PARQUET_INPUT_FORMAT) + .serializationLib(PARQUET_SERDE) + .build(); + + Map props = provider.getScanNodeProperties( + new FakeSession(), handle, Collections.emptyList(), Optional.empty()); + + Assertions.assertEquals("canonAK", props.get("location.AWS_ACCESS_KEY"), + "BE-canonical AWS_* creds must be emitted for the native reader (legacy parity)"); + // the raw s3. alias is still forwarded (harmless, ignored by BE), so no configured key is dropped + Assertions.assertEquals("aliasAK", props.get("location.s3.access_key")); + } + // ===== helpers ===== @Test diff --git a/plan-doc/task-list.md b/plan-doc/task-list.md index fc662c4577e826..3aae80c0ff3a8a 100644 --- a/plan-doc/task-list.md +++ b/plan-doc/task-list.md @@ -15,7 +15,17 @@ legacy `HudiScanNode`). Two independent wiring gaps, fixed independently. RCA: r Hadoop conf receives `fs.s3a.access.key/secret.key/endpoint`. fe-connector-hudi only. **DONE** (getScanNodeProperties emits translated fs.s3a.* under location.; +1 UT, 0 checkstyle). +- [x] FIX-hive-s3a-read — plain-hive object-store latent gap (found via the hudi red-team; user asked + to fix it too). TWO gaps in fe-connector-hive: (1) native `.path()` not scheme-normalized + (`splitFile`/`newRangeBuilder` + ACID paths) → `Invalid S3 URI`; (2) `getScanNodeProperties` + emitted raw `s3.` aliases instead of BE-canonical `AWS_*` (legacy `getLocationProperties` emitted + `getBackendStorageProperties()`) → private-bucket 403. Hive has no JNI → no fs.s3a. analog. + **DONE** (normalizeNativeUri + canonical creds emission; +2 UT, full hive suite 328/328, 0 checkstyle). + **Unexercised locally** (docker=HDFS; hive-s3 suites are p2/real-cloud) → unit-verified only, + object-store e2e needs user's real s3/oss env. + E2E = the existing 14 failing suites under `regression-test/suites/external_table_p2/hudi/` -(no new suites needed; they are the regression gate). Run both fixes together, then re-run all 14. +(no new suites needed; they are the regression gate). Run both hudi fixes together, then re-run all 14. +Hive: object-store e2e needs a real s3/oss hive table (docker is HDFS); HDFS hive suites are the parity guard. -Order: native-scheme → jni-creds (independent; either order works). TDD per fix, independent commit. +Order: hudi native-scheme → hudi jni-creds → hive-s3a-read. TDD per fix, independent commit. diff --git a/plan-doc/tasks/designs/FIX-hive-s3a-read-design.md b/plan-doc/tasks/designs/FIX-hive-s3a-read-design.md new file mode 100644 index 00000000000000..dd1a72fe953f01 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-hive-s3a-read-design.md @@ -0,0 +1,70 @@ +# FIX-hive-s3a-read (native scheme + canonical creds) + +## Problem +Plain-hive tables on an object-store warehouse (`s3a://`/`oss://`/`cos://`/`obs://`) would fail +BE-side reads — the same class of latent gap that broke hudi. Found while fixing hudi; the design +red-team flagged `HiveScanPlanProvider` passes raw HMS paths with no scheme normalization. +**Unexercised locally**: the hive docker regression uses `hdfs://` (no S3URI, no s3 creds); the only +hive-s3 suites (`test_hive_write_insert_s3`, `hive_on_hms_and_dlf`) are p2/external (real cloud), +not run in local docker. So both gaps below are latent. + +## Root Cause — TWO gaps in `HiveScanPlanProvider` (fe-connector-hive only) + +### Gap 1 — native path scheme not normalized +BE's native S3 reader (`be/src/util/s3_uri.cpp` `S3URI::parse`) accepts only `s3`/`http`/`https`. +`splitFile` → `newRangeBuilder:587` `.path(filePath)` ships the raw `s3a://` (etc.) path verbatim to +`TFileRangeDesc.path`. Legacy `HiveScanNode` normalized via the 2-arg +`LocationPath.of(path, storagePropertiesMap)`. Hive's OWN write path already uses +`context.normalizeStorageUri` (`HiveWritePlanProvider:155`) — the read path just never got it. Also +affects the ACID full-transactional BE-facing paths (partition location + delete-delta directories). + +### Gap 2 — BE-canonical credentials not emitted (worse; a legacy regression) +`getScanNodeProperties` emitted under `location.` ONLY the raw catalog aliases (`s3.`/`fs.`/`hadoop.` +via `isLocationProperty`). BE's native FILE_S3 reader reads ONLY `AWS_ACCESS_KEY`/`AWS_SECRET_KEY`/ +`AWS_ENDPOINT` (`s3_util.cpp:146-148`), never `s3.access_key` — so a private bucket 403s. Legacy +`HiveScanNode.getLocationProperties()` returned `hmsTable.getBackendStorageProperties()` (the +canonical `AWS_*`); the new path dropped it. (Hive has **no JNI scanner** — grep for `FORMAT_JNI` +empty — so there is no `fs.s3a.*`/JNI-creds analog of the hudi Fix B.) + +## Design +Mirror the hudi fixes + hive's own write-path pattern + legacy parity: + +- **Gap 1**: add `normalizeNativeUri(raw) = context != null ? context.normalizeStorageUri(raw) : raw`. + Normalize `filePath` at the top of `splitFile` (covers the regular + ACID data-file paths — both + callers route through it). Normalize the ACID BE-facing paths at their emit sites: `acidLocation` + (partition location) and the delete-delta directory inside `encodeDeleteDeltas` (threaded a + `UnaryOperator`). The connector still LISTS files with the raw scheme (Hadoop + `S3AFileSystem` wants `s3a`); only the BE-facing copies are normalized. +- **Gap 2**: emit `context.getBackendStorageProperties()` (canonical `AWS_*` / resolved + `hadoop.*`/`dfs.*`) under `location.` BEFORE the existing raw passthrough, so a user-inline + `fs.`/`hadoop.` key still wins. Keep the raw passthrough (harmless `s3.` aliases + inline keys). + This restores exactly what legacy emitted. + +fe-core untouched (no property parsing; no source-specific code). Non-object-store schemes +(`hdfs://`) pass through `normalizeStorageUri` unchanged, and for HDFS catalogs +`getBackendStorageProperties()` yields resolved `hadoop.*/dfs.*` (what legacy sent) — so the +HDFS-backed docker suites are parity-preserved. + +## Risk Analysis +- **HDFS regression** (reference connector, 300+ tests on HDFS): the canonical emission adds resolved + `hadoop.*/dfs.*` for HDFS catalogs — identical to what legacy `getLocationProperties()` sent, and + the raw passthrough (already present, already green) is retained. Parity argument → low risk; + final proof is the user's docker HDFS run (unit tests can't drive BE). +- **Cannot e2e-validate object-store locally** (docker = HDFS; s3 suites = p2 real-cloud). Verified at + the connector level with unit tests only; end-to-end needs the user's real s3/oss hive setup. +- **ACID-on-object-store** is a rare untested corner; normalization is safe (s3:// is accepted; if BE + doesn't parse a given acid path via S3URI the rewrite is a harmless no-op). +- Existing tests asserting `getScanNodeProperties` output: full-suite run is the guard. + +## Test Plan +### Unit (fe-connector-hive, `HiveScanBatchModeTest`) +- `nativeScanRangePathNormalizedS3aToS3`: drive `planScanForPartitionBatch` with an `s3a://` file + + a context normalizing `s3a`→`s3`; assert the range `.getPath()` is `s3://`. +- `scanNodePropertiesEmitsCanonicalCredsForNativeReader`: a context whose + `getBackendStorageProperties()` yields `AWS_ACCESS_KEY`, catalog carrying only the `s3.` alias; + assert `location.AWS_ACCESS_KEY` emitted (and the raw alias still forwarded). +- Full fe-connector-hive suite must stay green (regression guard for the creds change). + +### E2E (user-run; needs real s3/oss hive) +A hive table on an `s3a://`/`oss://` warehouse: native read stops throwing `Invalid S3 URI`, and a +private bucket no longer 403s. HDFS hive suites must remain green (parity). diff --git a/plan-doc/tasks/designs/FIX-hive-s3a-read-summary.md b/plan-doc/tasks/designs/FIX-hive-s3a-read-summary.md new file mode 100644 index 00000000000000..bdec5c107e1760 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-hive-s3a-read-summary.md @@ -0,0 +1,34 @@ +# Summary — FIX-hive-s3a-read (native scheme + canonical creds) + +## Problem +Plain-hive tables on an object-store warehouse (`s3a://`/`oss://`/`cos://`) would fail BE reads — the +same latent gap class that broke hudi. Unexercised locally (hive docker = `hdfs://`; hive-s3 suites are +p2/real-cloud). Fixed on user request after the hudi fixes. + +## Root Cause (two gaps, fe-connector-hive only) +1. **Scheme**: `splitFile`→`newRangeBuilder:587` `.path(filePath)` shipped raw `s3a://` to + `TFileRangeDesc.path`; BE `S3URI` accepts only `s3/http/https`. (Legacy normalized via 2-arg + `LocationPath.of`; hive's own WRITE path already used `normalizeStorageUri`.) +2. **Creds (legacy regression)**: `getScanNodeProperties` emitted only raw `s3.`/`fs.` aliases; BE's + native reader reads only canonical `AWS_ACCESS_KEY/SECRET/ENDPOINT` (`s3_util.cpp:146-148`) → private + bucket 403s. Legacy `getLocationProperties()` returned `getBackendStorageProperties()` (canonical); + the new path dropped it. (Hive has no JNI scanner → no `fs.s3a.*`/JNI analog.) + +## Fix +- `normalizeNativeUri(raw) = context != null ? context.normalizeStorageUri(raw) : raw`; applied to the + native data-file path in `splitFile`, and to the ACID BE-facing paths (`acidLocation` + + `encodeDeleteDeltas` delete-delta dir, threaded a `UnaryOperator`). Files are still LISTED with + the raw scheme (Hadoop wants `s3a`); only BE-facing copies normalized. +- `getScanNodeProperties` emits `context.getBackendStorageProperties()` (canonical) under `location.` + before the raw passthrough (inline `fs.`/`hadoop.` still wins). Restores legacy behavior. +fe-core untouched; no source-specific code. + +## Tests +`HiveScanBatchModeTest` +2 (RED-able): range path `s3a`→`s3` via `planScanForPartitionBatch`; canonical +`AWS_*` emission. **Full fe-connector-hive suite 328/328 green, 0 checkstyle** (regression guard for the +creds change over the HDFS-based suite). + +## Result +Design `plan-doc/tasks/designs/FIX-hive-s3a-read-design.md`. **Unit-verified only** — object-store e2e +needs the user's real s3/oss hive env (docker is HDFS). HDFS parity: canonical emission = what legacy +sent, so the HDFS docker hive suites should stay green (final proof = user's docker run). From 13885684b1666fe0405c14cc6a6eed7d66c0b975 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 16:33:21 +0800 Subject: [PATCH 294/334] [doc](catalog) HANDOFF: hive object-store read fix DONE (scheme + canonical creds), e2e pending Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 4a3749c8acbeec..419db16a505426 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,12 +5,14 @@ --- -# 🆕 本 session（hudi s3a BE-scan 读）= 2 fix DONE，e2e 待用户自跑 +# 🆕 本 session（对象存储读修复）= hudi 2 fix + hive 1 fix DONE，e2e 待用户自跑 > 用户按最新代码重跑 = **14 个 hudi p2 suite 全失败**（唯 `test_hudi_meta` 过）。两类错误签名、同一主题：新 fe-connector hudi BE-scan 路径**没把 catalog 的 S3 storageProperties 接到 BE**（相对 legacy `HudiScanNode` 的 parity gap，**非环境/非文案**）。两个独立接线缺口，各独立 commit： > - **native `Invalid S3 URI: s3a://` (10 suite)** = `a26eaf46b9f`：BE `S3URI` 只认 `s3/http/https`，连接器把原始 `s3a://` 直发 `TFileRangeDesc.path`。修=连接器侧 `context.normalizeStorageUri` 归一化**native** range `.path()`（镜像 iceberg/paimon），**三处** native emitter：`collectCowSplits`/`buildMorRange`/**`COWIncrementalRelation.collectSplits`(COW @incr，红队补的第三处)**；JNI `THudiFileDesc` 路径留原始 `s3a://`。 > - **JNI `NoAuthWithAWSException` (4 suite)** = `19a8098b3e7`：`getScanNodeProperties` 只发 AWS_*(native)+s3. 别名，漏发翻译后的 `fs.s3a.*`。修=补 `storageHadoopConfig(context)` 以 `location.` 前缀下发。 > **验证**：fe-connector-hudi 全模块 **175/175 UT 绿、0 checkstyle**（含 3 个新 RED-then-GREEN 单测）。设计+摘要 `designs/FIX-hudi-s3a-{native-scheme,jni-creds}-{design,summary}.md`、task-list、红队 `wf_67162858-b79`(RCA)+`wf_4e4ec1d7-d4f`(设计对抗 GO_WITH_FIXES)。**e2e 待用户自跑（须先重编/重部署 FE 连接器 jar）**：14 个 `external_table_p2/hudi/` suite 全绿门；`test_hudi_catalog` 等含 `force_jni_scanner=true` 子查询者两条路径都要过。 +> +> **hive 同类隐患一并修（用户点单）** = `f22f8223033`：**plain-hive 对象存储读**两个缺口(红队顺带发现)：① native `.path()` 不归一化(`splitFile`/`newRangeBuilder`+ACID 路径)→`Invalid S3 URI`;② `getScanNodeProperties` 只发原始 `s3.` 别名、**漏发 legacy 的 canonical `AWS_*`**(`getBackendStorageProperties`,BE `s3_util.cpp:146-148` 只认 AWS_*)→私有桶 403。修=`normalizeNativeUri`(镜像 hive 自己的写路径+hudi)+补发 canonical creds。hive **无 JNI**故无 fs.s3a 那半。**本地测不到**(hive docker=HDFS;hive-s3 套件是 p2 真云)→**仅单测验证**(fe-connector-hive **328/328 绿**、2 新 RED-able 单测)。设计+摘要 `designs/FIX-hive-s3a-read-{design,summary}.md`。**e2e 待用户自跑**：真实 s3/oss hive 表(native 不再 Invalid S3 URI、私有桶不再 403);**HDFS hive 套件是 creds 改动的 parity 回归守卫**(canonical 发的正是 legacy 所发)。 --- From a96dd8a353f54650e723694610935382d2aedc01 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 19:42:59 +0800 Subject: [PATCH 295/334] [fix](catalog) hms: auto-enable metastore SASL for kerberos catalogs The SPI HmsConfHelper.createHiveConf only copied raw catalog properties, so a kerberos HMS catalog that declared hadoop.security.authentication=kerberos but did not spell out hive.metastore.sasl.enabled opened a plain TSocket that the kerberized metastore dropped with TTransportException (regression test_single_hive_kerberos, test_two_hive_kerberos). Restore the legacy fe-core HMSBaseProperties.initHadoopAuthenticator behavior: inject hive.metastore.sasl.enabled=true when the metastore/hadoop auth is kerberos (hadoop.security.authentication or hive.metastore.authentication.type). Adds HmsConfHelperTest (6 cases): SASL on via either auth key incl. case-insensitive; simple/none auth leaves SASL off; arbitrary props pass through. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/connector/hms/HmsConfHelper.java | 9 ++ .../connector/hms/HmsConfHelperTest.java | 96 +++++++++++++++++++ 2 files changed, 105 insertions(+) create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsConfHelperTest.java diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java index 333fa7f56b55d5..e0c0a3eea49e69 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/HmsConfHelper.java @@ -61,6 +61,15 @@ public static HiveConf createHiveConf(Map properties) { for (Map.Entry entry : properties.entrySet()) { hiveConf.set(entry.getKey(), entry.getValue()); } + // A kerberized HMS requires SASL transport on the metastore Thrift connection. The legacy fe-core + // HMSBaseProperties.initHadoopAuthenticator auto-enabled hive.metastore.sasl.enabled whenever the + // metastore/hadoop auth was kerberos; preserve that here so a catalog that only declares kerberos auth + // (without an explicit hive.metastore.sasl.enabled) still negotiates SASL, instead of opening a plain + // TSocket that a kerberized metastore drops with TTransportException. + if ("kerberos".equalsIgnoreCase(properties.get("hadoop.security.authentication")) + || "kerberos".equalsIgnoreCase(properties.get("hive.metastore.authentication.type"))) { + hiveConf.set("hive.metastore.sasl.enabled", "true"); + } return hiveConf; } diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsConfHelperTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsConfHelperTest.java new file mode 100644 index 00000000000000..c67151234a6422 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/HmsConfHelperTest.java @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.hadoop.hive.conf.HiveConf; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Tests {@link HmsConfHelper#createHiveConf(Map)} — the HiveConf builder for the hms connector's + * metastore Thrift client ({@code ThriftHmsClient} constructs its conf here). + * + *

    WHY: a kerberized HMS only accepts a SASL-negotiated Thrift transport. The legacy fe-core + * {@code HMSBaseProperties.initHadoopAuthenticator} auto-enabled {@code hive.metastore.sasl.enabled} + * whenever the metastore/hadoop auth was kerberos (HMSBaseProperties.java:162,179). The SPI connector's + * builder previously only copied raw properties, so a catalog that declared kerberos auth but did not + * spell out {@code hive.metastore.sasl.enabled} opened a plain TSocket the metastore dropped with + * {@code TTransportException} (test_single_hive_kerberos / test_two_hive_kerberos). These tests pin the + * restored auto-injection and its opposite — non-kerberos auth must NOT flip SASL on (Rule 9: encode WHY + * the injection is gated on kerberos, not just that it happens).

    + */ +public class HmsConfHelperTest { + + private static String saslOf(Map props) { + HiveConf conf = HmsConfHelper.createHiveConf(props); + return conf.get("hive.metastore.sasl.enabled"); + } + + @Test + public void hadoopSecurityAuthKerberosEnablesSasl() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", "thrift://host:9583"); + props.put("hadoop.security.authentication", "kerberos"); + props.put("hive.metastore.kerberos.principal", "hive/hadoop-master@LABS.TERADATA.COM"); + // No explicit hive.metastore.sasl.enabled — the kerberized catalog relies on auto-injection. + Assertions.assertEquals("true", saslOf(props)); + } + + @Test + public void hiveMetastoreAuthTypeKerberosEnablesSasl() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", "thrift://host:9583"); + props.put("hive.metastore.authentication.type", "kerberos"); + Assertions.assertEquals("true", saslOf(props)); + } + + @Test + public void kerberosAuthIsCaseInsensitive() { + Map props = new HashMap<>(); + props.put("hadoop.security.authentication", "KERBEROS"); + Assertions.assertEquals("true", saslOf(props)); + } + + @Test + public void simpleAuthDoesNotEnableSasl() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", "thrift://host:9083"); + props.put("hadoop.security.authentication", "simple"); + Assertions.assertNotEquals("true", saslOf(props)); + } + + @Test + public void noAuthPropertyDoesNotEnableSasl() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", "thrift://host:9083"); + Assertions.assertNotEquals("true", saslOf(props)); + } + + @Test + public void arbitraryPropertiesArePassedThrough() { + Map props = new HashMap<>(); + props.put("hive.metastore.uris", "thrift://host:9083"); + props.put("some.custom.key", "custom-value"); + HiveConf conf = HmsConfHelper.createHiveConf(props); + Assertions.assertEquals("thrift://host:9083", conf.get("hive.metastore.uris")); + Assertions.assertEquals("custom-value", conf.get("some.custom.key")); + } +} From a634c5c3e31edde96d681adcd297c2a4a1cc94bc Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 19:43:24 +0800 Subject: [PATCH 296/334] [fix](mtmv) name genuine-NULL partition pn_NULL to avoid collision with literal 'NULL' string A genuine-NULL list value and a literal 'NULL' string both render to the text "NULL" once PartitionKeyDesc quotes the value and generatePartitionName strips the quotes, so a partition column holding both (paimon null_partition: a real NULL row plus a 'NULL' string row) generated two partitions named p_NULL and failed the uniqueness check on CREATE MATERIALIZED VIEW (regression test_paimon_mtmv: "Duplicated named partition: p_NULL"). This surfaced once variant B (58f3e367ed6) gave paimon/hive genuine-NULL semantics. Prefix a null-bearing MTMV partition with pn_ (read from PartitionValue.isNullPartition()) instead of p_: string values always yield a p_ name (second char '_'), so a pn_ name can never collide, and real-NULL vs 'NULL'-string stay distinct partitions. Both the base-connector item and the MV-OLAP item render pn_NULL symmetrically, so the partition-sync join is unaffected (unlike the reverted FIX-3, which used a one-sided sentinel). Renames the genuine-NULL MTMV partition p_NULL -> pn_NULL globally (hive default partition and internal OLAP null partitions too). Updates the impacted unit tests (ListPartitionItemTest + a new 'NULL'-string -> p_NULL case) and regression suites (test_hive_default_mtmv, test_null_partition_mtmv, test_null_multi_pct_mtmv, test_paimon_mtmv). hudi (isNull=false) and user-defined p_NULL partition names are unaffected. The test_paimon_mtmv order_qt_null_partition golden regenerates on the e2e run (genuine-NULL rows now materialize). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../apache/doris/mtmv/MTMVPartitionUtil.java | 30 +++++++++- .../doris/catalog/ListPartitionItemTest.java | 59 +++++++++++++------ .../mtmv_p0/test_hive_default_mtmv.groovy | 4 +- .../mtmv_p0/test_null_multi_pct_mtmv.groovy | 4 +- .../mtmv_p0/test_null_partition_mtmv.groovy | 4 +- .../suites/mtmv_p0/test_paimon_mtmv.groovy | 5 ++ 6 files changed, 81 insertions(+), 25 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java index aea97a2dc875cc..d2bcfc4ca69760 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java +++ b/fe/fe-core/src/main/java/org/apache/doris/mtmv/MTMVPartitionUtil.java @@ -19,6 +19,7 @@ import org.apache.doris.analysis.AllPartitionDesc; import org.apache.doris.analysis.PartitionKeyDesc; +import org.apache.doris.analysis.PartitionValue; import org.apache.doris.analysis.SinglePartitionDesc; import org.apache.doris.catalog.Column; import org.apache.doris.catalog.Database; @@ -65,6 +66,12 @@ public class MTMVPartitionUtil { private static final Logger LOG = LogManager.getLogger(MTMVPartitionUtil.class); private static final Pattern PARTITION_NAME_PATTERN = Pattern.compile("[^a-zA-Z0-9,]"); private static final String PARTITION_NAME_PREFIX = "p_"; + // A genuine-NULL list value and a literal 'NULL' string both render to the text "NULL" once + // PartitionKeyDesc quotes every value and PARTITION_NAME_PATTERN strips the quotes, so a column holding + // both would generate two partitions named p_NULL and fail the uniqueness check. Null-bearing partitions + // use this "pn_" prefix instead: string values always yield a "p_" name (second char '_'), so a "pn_" + // name (second char 'n') can never collide, keeping the real-NULL and 'NULL'-string partitions distinct. + private static final String PARTITION_NAME_NULL_PREFIX = "pn_"; private static final List partitionDescGenerators = ImmutableList .of( @@ -363,7 +370,8 @@ public static boolean isSyncWithPartitions(MTMVRefreshContext context, String mt */ public static String generatePartitionName(PartitionKeyDesc desc) { Matcher matcher = PARTITION_NAME_PATTERN.matcher(desc.toSql()); - String partitionName = PARTITION_NAME_PREFIX + matcher.replaceAll("").replaceAll("\\,", "_"); + String prefix = hasNullPartitionValue(desc) ? PARTITION_NAME_NULL_PREFIX : PARTITION_NAME_PREFIX; + String partitionName = prefix + matcher.replaceAll("").replaceAll("\\,", "_"); if (partitionName.length() > 50) { partitionName = partitionName.substring(0, 30) + Math.abs(Objects.hash(partitionName)) + "_" + System.currentTimeMillis(); @@ -371,6 +379,26 @@ public static String generatePartitionName(PartitionKeyDesc desc) { return partitionName; } + /** + * Whether the list-partition desc carries a genuine-NULL value ({@link PartitionValue#isNullPartition()}). + * Such a partition must be named with {@link #PARTITION_NAME_NULL_PREFIX} so it never collides with a + * literal 'NULL' string partition (both otherwise render to the same p_NULL name). Range/other descs have + * no in-values and are never null-bearing here. + */ + private static boolean hasNullPartitionValue(PartitionKeyDesc desc) { + if (!desc.hasInValues()) { + return false; + } + for (List values : desc.getInValues()) { + for (PartitionValue value : values) { + if (value.isNullPartition()) { + return true; + } + } + } + return false; + } + /** * drop partition of mtmv * diff --git a/fe/fe-core/src/test/java/org/apache/doris/catalog/ListPartitionItemTest.java b/fe/fe-core/src/test/java/org/apache/doris/catalog/ListPartitionItemTest.java index 42497bf1ba0478..6d57249f76091f 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/catalog/ListPartitionItemTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/catalog/ListPartitionItemTest.java @@ -32,24 +32,30 @@ /** * Tests for {@link ListPartitionItem#toPartitionKeyDesc} null-partition display handling. * - *

    Guards the master-parity rendering of a genuine-NULL partition: its MTMV partition NAME must be the bare - * {@code p_NULL}, the same as on master ({@link PartitionInfo#toPartitionValue} renders a {@code NullLiteral} - * key as the keyword {@code NULL}). FIX-3 (CI 973411, reverted) substituted the {@code originHiveKeys} sentinel - * for the display value, renaming the hive default-partition MV to {@code p_HIVEDEFAULTPARTITION} and breaking - * regression test_hive_default_mtmv (asserts {@code p_NULL}) and test_upgrade_downgrade_mtmv (the related-desc - * vs MV-OLAP-desc partition-mapping join, which only matches when both sides render symmetrically). + *

    Guards the naming of a genuine-NULL partition: its MTMV partition NAME must be {@code pn_NULL} (the + * {@code pn_} prefix marks a null-bearing partition), NOT the bare {@code p_NULL}. A real NULL value and a + * literal {@code 'NULL'} string both render to the text {@code NULL} once {@code PartitionKeyDesc} quotes the + * value and the name pattern strips the quotes, so on a column holding both (e.g. paimon + * {@code null_partition}: a real NULL row plus a {@code 'NULL'} string row) they would BOTH be named + * {@code p_NULL} and fail the partition-uniqueness check (regression test_paimon_mtmv). The {@code pn_} prefix + * keeps them distinct: a string value always yields a {@code p_}-prefixed name, so {@code pn_} can never + * collide. The nullness is read from {@link PartitionValue#isNullPartition()}, which BOTH the connector-side + * genuine-null item and the MV-OLAP item carry, so both render the SAME {@code pn_NULL} — keeping the + * related-desc vs MV-OLAP-desc partition-mapping join symmetric (unlike the reverted FIX-3, which used the + * one-sided {@code originHiveKeys} sentinel and broke the join). */ public class ListPartitionItemTest { /** * A genuine-NULL partition (e.g. a hive {@code __HIVE_DEFAULT_PARTITION__} default partition, built isNull - * with the sentinel preserved as originHiveKeys) must render its MTMV partition name as the bare - * {@code p_NULL} — identical to master — so that (a) test_hive_default_mtmv:93 finds p_NULL and (b) the - * MTMV-OLAP partition (which has no originHiveKeys) renders the SAME name, keeping the sync-compare join - * symmetric. The value must still resolve to a NULL literal so {@code col IS NULL} pruning is unaffected. + * with the sentinel preserved as originHiveKeys) must render its MTMV partition name as {@code pn_NULL} so + * that (a) it never collides with a literal {@code 'NULL'} string partition (which renders {@code p_NULL}) + * and (b) the MV-OLAP partition (which has no originHiveKeys) renders the SAME name, keeping the + * sync-compare join symmetric. The value must still resolve to a NULL literal so {@code col IS NULL} + * pruning is unaffected. */ @Test - public void testGenuineNullPartitionRendersAsPNull() throws AnalysisException { + public void testGenuineNullPartitionRendersAsPnNull() throws AnalysisException { List types = Collections.singletonList(Type.VARCHAR); // Genuine NULL partition as a hive/paimon connector builds it: a NULL literal whose origin-hive key @@ -59,9 +65,9 @@ public void testGenuineNullPartitionRendersAsPNull() throws AnalysisException { types, true); ListPartitionItem nullItem = new ListPartitionItem(Lists.newArrayList(nullKey)); - Assertions.assertEquals("p_NULL", + Assertions.assertEquals("pn_NULL", MTMVPartitionUtil.generatePartitionName(nullItem.toPartitionKeyDesc(0)), - "a genuine-null partition must render as p_NULL (master parity), not p_HIVEDEFAULTPARTITION"); + "a genuine-null partition must render as pn_NULL (distinct from a literal 'NULL' string's p_NULL)"); // The null partition's desc value must still resolve to a NULL literal so `col IS NULL` prunes to it. PartitionValue nullDescValue = nullItem.toPartitionKeyDesc(0).getInValues().get(0).get(0); @@ -72,18 +78,35 @@ public void testGenuineNullPartitionRendersAsPNull() throws AnalysisException { } /** - * An internal OLAP null partition (no originHiveKeys) renders as {@code p_NULL} — unchanged by, and after, - * the FIX-3 revert. Kept as a symmetry anchor for {@link #testGenuineNullPartitionRendersAsPNull}: both the - * connector-side genuine-null item and the MV-OLAP item must produce the SAME p_NULL name. + * An internal OLAP null partition (no originHiveKeys) renders as {@code pn_NULL} — the SAME name the + * connector-side genuine-null item produces. Kept as a symmetry anchor for + * {@link #testGenuineNullPartitionRendersAsPnNull}: both sides must produce the SAME pn_NULL name so the + * partition-mapping join stays symmetric. */ @Test - public void testOlapNullPartitionRendersAsPNull() throws AnalysisException { + public void testOlapNullPartitionRendersAsPnNull() throws AnalysisException { List types = Collections.singletonList(Type.VARCHAR); PartitionKey olapNullKey = PartitionKey.createListPartitionKeyWithTypes( Collections.singletonList(new PartitionValue("NULL", true)), types, false); ListPartitionItem item = new ListPartitionItem(Lists.newArrayList(olapNullKey)); + Assertions.assertEquals("pn_NULL", + MTMVPartitionUtil.generatePartitionName(item.toPartitionKeyDesc(0)), + "an OLAP null partition (no originHiveKeys) must render as pn_NULL"); + } + + /** + * A literal {@code 'NULL'} string partition (NOT a genuine null) must keep the bare {@code p_NULL} name — + * it is ordinary string data and must stay distinct from the real-NULL partition's {@code pn_NULL}. This + * is the collision the {@code pn_} prefix resolves (regression test_paimon_mtmv, which has both). + */ + @Test + public void testLiteralNullStringPartitionRendersAsPNull() throws AnalysisException { + List types = Collections.singletonList(Type.VARCHAR); + PartitionKey strKey = PartitionKey.createListPartitionKeyWithTypes( + Collections.singletonList(new PartitionValue("NULL")), types, false); + ListPartitionItem item = new ListPartitionItem(Lists.newArrayList(strKey)); Assertions.assertEquals("p_NULL", MTMVPartitionUtil.generatePartitionName(item.toPartitionKeyDesc(0)), - "an OLAP null partition (no originHiveKeys) must render as p_NULL"); + "a literal 'NULL' string partition must render as p_NULL, distinct from the real-NULL pn_NULL"); } } diff --git a/regression-test/suites/mtmv_p0/test_hive_default_mtmv.groovy b/regression-test/suites/mtmv_p0/test_hive_default_mtmv.groovy index d3fbb2f262ec58..ca44d5b36a4ef0 100644 --- a/regression-test/suites/mtmv_p0/test_hive_default_mtmv.groovy +++ b/regression-test/suites/mtmv_p0/test_hive_default_mtmv.groovy @@ -90,11 +90,11 @@ suite("test_hive_default_mtmv", "p0,external,hive,external_docker,external_docke """ def showPartitionsResult = sql """show partitions from ${mvName}""" logger.info("showPartitionsResult: " + showPartitionsResult.toString()) - assertTrue(showPartitionsResult.toString().contains("p_NULL")) + assertTrue(showPartitionsResult.toString().contains("pn_NULL")) assertTrue(showPartitionsResult.toString().contains("p_1")) sql """ - REFRESH MATERIALIZED VIEW ${mvName} partitions(p_NULL); + REFRESH MATERIALIZED VIEW ${mvName} partitions(pn_NULL); """ def jobName = getJobName(dbName, mvName); waitingMTMVTaskFinished(jobName) diff --git a/regression-test/suites/mtmv_p0/test_null_multi_pct_mtmv.groovy b/regression-test/suites/mtmv_p0/test_null_multi_pct_mtmv.groovy index 390d9c05aad58a..b3c99c1939c31a 100644 --- a/regression-test/suites/mtmv_p0/test_null_multi_pct_mtmv.groovy +++ b/regression-test/suites/mtmv_p0/test_null_multi_pct_mtmv.groovy @@ -81,14 +81,14 @@ String suiteName = "test_multi_pct_bad_mtmv" logger.info("showPartitionsResult: " + showPartitionsResult.toString()) assertTrue(showPartitionsResult.toString().contains("p_1")) assertTrue(showPartitionsResult.toString().contains("p_2")) - assertTrue(showPartitionsResult.toString().contains("p_NULL")) + assertTrue(showPartitionsResult.toString().contains("pn_NULL")) sql """ REFRESH MATERIALIZED VIEW ${mvName} AUTO """ waitingMTMVTaskFinishedByMvName(mvName) - order_qt_list_null "SELECT * FROM ${mvName} partitions(p_NULL) order by user_id,num" + order_qt_list_null "SELECT * FROM ${mvName} partitions(pn_NULL) order by user_id,num" order_qt_list_1 "SELECT * FROM ${mvName} partitions(p_1) order by user_id,num" order_qt_list_2 "SELECT * FROM ${mvName} partitions(p_2) order by user_id,num" diff --git a/regression-test/suites/mtmv_p0/test_null_partition_mtmv.groovy b/regression-test/suites/mtmv_p0/test_null_partition_mtmv.groovy index a6a7890180ecc2..cd0cb34da4e4ce 100644 --- a/regression-test/suites/mtmv_p0/test_null_partition_mtmv.groovy +++ b/regression-test/suites/mtmv_p0/test_null_partition_mtmv.groovy @@ -56,7 +56,7 @@ suite("test_null_partition_mtmv") { def showPartitionsResult = sql """show partitions from ${mvName}""" logger.info("showPartitionsResult: " + showPartitionsResult.toString()) assertTrue(showPartitionsResult.toString().contains("p_1")) - assertTrue(showPartitionsResult.toString().contains("p_NULL")) + assertTrue(showPartitionsResult.toString().contains("pn_NULL")) sql """ REFRESH MATERIALIZED VIEW ${mvName} AUTO @@ -65,7 +65,7 @@ suite("test_null_partition_mtmv") { log.info(jobName) waitingMTMVTaskFinished(jobName) - order_qt_list_null "SELECT * FROM ${mvName} partitions(p_NULL) order by user_id,num" + order_qt_list_null "SELECT * FROM ${mvName} partitions(pn_NULL) order by user_id,num" order_qt_list_1 "SELECT * FROM ${mvName} partitions(p_1) order by user_id,num" sql """drop table if exists `${tableName}`""" diff --git a/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy b/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy index 114e76beeac07f..69c4118fcd3644 100644 --- a/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy +++ b/regression-test/suites/mtmv_p0/test_paimon_mtmv.groovy @@ -255,9 +255,14 @@ suite("test_paimon_mtmv", "p0,external,mtmv,external_docker,external_docker_dori """ def showNullPartitionsResult = sql """show partitions from ${mvName}""" logger.info("showNullPartitionsResult: " + showNullPartitionsResult.toString()) + // null_partition holds four region variants that must stay four distinct MV partitions: + // 'bj' -> p_bj, 'null' -> p_null, 'NULL' -> p_NULL (literal strings), and the genuine-NULL region + // (paimon variant B) -> pn_NULL. The real NULL and the literal 'NULL' string both stripped to the name + // p_NULL before, colliding on CREATE; the pn_ prefix (see MTMVPartitionUtil) keeps them distinct. assertTrue(showNullPartitionsResult.toString().contains("p_null")) assertTrue(showNullPartitionsResult.toString().contains("p_NULL")) assertTrue(showNullPartitionsResult.toString().contains("p_bj")) + assertTrue(showNullPartitionsResult.toString().contains("pn_NULL")) sql """ REFRESH MATERIALIZED VIEW ${mvName} auto; """ From b9f2e206a3285f52a650d88f60aa520a3aafc704 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 20:37:14 +0800 Subject: [PATCH 297/334] [fix](planner) FileSplit.getSelfSplitWeight: null-safe -1 to fix @Data equals/hashCode NPE FileSplit is Lombok @Data, whose generated equals()/hashCode() invoke the hand-written getSelfSplitWeight() (returns primitive long). That getter unboxed the nullable Long selfSplitWeight field, so any FileSplit that never set a size-based weight (the common case; production reads the field directly in getSplitWeight(), which is null-safe) threw NPE the moment its equals()/hashCode() ran -- e.g. the multimap comparison in FederationBackendPolicyTest.testGenerateRandomly. Return -1 ("not provided") for a null weight, matching the ConnectorScanRange SPI default, PluginDrivenSplit's weight >= 0 gating, and the sibling Iceberg/PaimonScanRange (which use primitive long fields). IcebergSplit inherits the fixed getter. Adds a deterministic regression test. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../org/apache/doris/datasource/FileSplit.java | 5 ++++- .../planner/FederationBackendPolicyTest.java | 16 ++++++++++++++++ 2 files changed, 20 insertions(+), 1 deletion(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileSplit.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileSplit.java index fcb6f9562785ac..c2f4de7715dddc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/FileSplit.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/FileSplit.java @@ -113,6 +113,9 @@ public SplitWeight getSplitWeight() { } public long getSelfSplitWeight() { - return selfSplitWeight; + // selfSplitWeight is null when unset; mirror the ConnectorScanRange SPI "-1 = not provided" + // sentinel. Lombok's @Data generates equals()/hashCode() that invoke this getter, so an + // unboxing null here would NPE for any FileSplit that never set a size-based weight. + return selfSplitWeight == null ? -1 : selfSplitWeight; } } diff --git a/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java b/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java index 9dac86f47b98cf..64202624594ee8 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/planner/FederationBackendPolicyTest.java @@ -672,6 +672,22 @@ public void testSplitWeight() { Assert.assertEquals(50, fileSplit.getSplitWeight().getRawValue()); } + // Regression for the NPE in testGenerateRandomly: FileSplit is Lombok @Data, whose generated + // equals()/hashCode() invoke getSelfSplitWeight(). A split that never sets a size-based weight + // leaves selfSplitWeight null, so the getter must surface the "-1 = not provided" sentinel + // instead of unboxing null (which threw NPE during the multimap comparison). + @Test + public void testFileSplitEqualsHashCodeWithUnsetWeight() { + LocationPath path = LocationPath.of("s1"); + // Two distinct instances that share the same LocationPath are field-equal, so equals() + // proceeds past the identity short-circuit and exercises getSelfSplitWeight(). + FileSplit a = new FileSplit(path, 0, 1000, 1000, 0, null, Collections.emptyList()); + FileSplit b = new FileSplit(path, 0, 1000, 1000, 0, null, Collections.emptyList()); + Assert.assertEquals(-1L, a.getSelfSplitWeight()); + Assert.assertEquals(a, b); + Assert.assertEquals(a.hashCode(), b.hashCode()); + } + @Test public void testBiggerSplit() throws UserException { SystemInfoService service = new SystemInfoService(); From 500472410d57f563f92b8e0570f4c1e676a2fd72 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 20:38:07 +0800 Subject: [PATCH 298/334] fix hudi partition table --- .../connector/hudi/HudiConnectorMetadata.java | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index cd5060fda21dc8..2fcd1b5ff5fbc9 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -91,6 +91,12 @@ public class HudiConnectorMetadata implements ConnectorMetadata { // Catalog property gating the partition-name source (mirrors legacy HMSExternalTable.USE_HIVE_SYNC_PARTITION). private static final String USE_HIVE_SYNC_PARTITION = "use_hive_sync_partition"; + // fe-core SPI schema property marking which emitted columns are partition columns (CSV of RAW partition-key + // names, in declaration order). Mirrors HiveConnectorMetadata.PARTITION_COLUMNS_PROPERTY; fe-core derives the + // partition-column set SOLELY from this key (PluginDrivenExternalTable.toSchemaCacheValue). Without it every + // partitioned Hudi table is read as UNPARTITIONED -> wrong pruning/row-count and MTMV "not partition table". + private static final String PARTITION_COLUMNS_PROPERTY = "partition_columns"; + // Hive-canonical partition text for a DATETIME/TIMESTAMP literal: space separator, full seconds. See // hiveDateTimeString / extractLiteralValue (H2: String.valueOf(LocalDateTime) would yield ISO "…T…" and drop // zero seconds, never matching the stored Hive partition value). @@ -349,8 +355,16 @@ private ConnectorTableSchema buildTableSchema(HudiTableHandle hudiHandle, String columns = getSchemaFromHms(hudiHandle.getDbName(), hudiHandle.getTableName()); } - Map tableProperties = Collections.singletonMap( - "hudi.table.type", hudiHandle.getHudiTableType()); + Map tableProperties = new HashMap<>(); + tableProperties.put("hudi.table.type", hudiHandle.getHudiTableType()); + // Stamp the partition-column marker fe-core needs to model the table as partitioned (parity with the OLD + // HMSExternalTable/HudiDlaTable path, which read the HMS partition keys). The keys already live on the + // handle (getTableHandle -> tableInfo.getPartitionKeys()); emit them as the RAW-name CSV, matching the + // schema column names (the partition columns are appended to `columns` by both schema sources). + List partitionKeyNames = hudiHandle.getPartitionKeyNames(); + if (partitionKeyNames != null && !partitionKeyNames.isEmpty()) { + tableProperties.put(PARTITION_COLUMNS_PROPERTY, String.join(",", partitionKeyNames)); + } return new ConnectorTableSchema( hudiHandle.getTableName(), columns, "HUDI", tableProperties); } From 88aa55b831b1dc17bda827a69e917af6b56dcd4b Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 21:30:10 +0800 Subject: [PATCH 299/334] [fix](build) connector-imports gate: close 4 leaks + add self-test MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The forbidden-import gate (tools/check-connector-imports.sh) that keeps fe-connector modules off fe-core internals had four leaks: 1. `import static ...` was not matched (only plain `import`). 2. FORBIDDEN missed 6 real fe-core internal packages: persist, transaction, fs, statistics, mysql, service. 3. Only src/main/java was scanned; src/test/java was not. 4. (found by design red-team) The four SPI-allow exclusions were plain `grep -v 'org.apache.doris.'` that match the whole `::import ...` line — and the regex dot matches '/', so they also matched the FILE PATH. Every one of the 608 connector sources roots at org.apache.doris.connector.**, so any forbidden import in a connector-namespaced file was silently dropped by its location, blinding the gate to the exact modules it guards. The four allow-packages are siblings of (never subpackages of) any FORBIDDEN package, so the candidate grep never captured a legitimate SPI import — those grep -v lines only ever suppressed by path. Fix: match `import static`, add the 6 packages, scan test sources too, and anchor the SPI-allow exclusion to the import target (":import ...") instead of the raw line. Also strip an optional `static` when extracting the fqn so a static import of a vendored class (e.g. HiveVersionUtil.SOME_CONST) is still correctly skipped rather than falsely reported. The widened gate catches zero on the real tree today (no connector imports any forbidden package, no static-import violations, no test-source violations), so this is preventive hardening — the real tree stays exit 0. Add tools/check-connector-imports.test.sh: a self-cleaning RED/GREEN fixture that seeds one violation per gate property (static import, each new package class, a test-source import, a connector-namespaced import, a static import of a vendored class) and asserts exactly the 8 are reported while the SPI-shared and vendored imports stay silent. It passes on the fixed gate and fails on the old one. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- tools/check-connector-imports.sh | 28 ++++-- tools/check-connector-imports.test.sh | 136 ++++++++++++++++++++++++++ 2 files changed, 155 insertions(+), 9 deletions(-) create mode 100755 tools/check-connector-imports.test.sh diff --git a/tools/check-connector-imports.sh b/tools/check-connector-imports.sh index fd4817707cc5bf..8b70abf306bc91 100755 --- a/tools/check-connector-imports.sh +++ b/tools/check-connector-imports.sh @@ -21,11 +21,16 @@ # See plan-doc/01-spi-extensions-rfc.md §15.4. # # Connector modules MUST NOT import fe-core internals (catalog / common / -# datasource / qe / analysis / nereids / planner). Anything they need from -# fe-core has to be exposed through the SPI in +# datasource / qe / analysis / nereids / planner / persist / transaction / fs / +# statistics / mysql / service). Anything they need from fe-core has to be +# exposed through the SPI in # org.apache.doris.connector.{api,spi,extension,...} # or shared types in org.apache.doris.thrift / org.apache.doris.filesystem. # +# The gate matches both plain and `import static` imports, scans src/main/java +# AND src/test/java, and anchors the SPI-allowed exclusion to the import target +# (not the file path). Self-test: tools/check-connector-imports.test.sh. +# # Usage: # tools/check-connector-imports.sh # search default root # tools/check-connector-imports.sh # search supplied root @@ -45,13 +50,18 @@ if [ ! -d "${ROOT}" ]; then exit 2 fi -FORBIDDEN='org\.apache\.doris\.(catalog|common|datasource|qe|analysis|nereids|planner)' +FORBIDDEN='org\.apache\.doris\.(catalog|common|datasource|qe|analysis|nereids|planner|persist|transaction|fs|statistics|mysql|service)' + +# SPI-shared packages a connector MAY import. Anchor the exclusion to the IMPORT TARGET (":import ..."), NOT +# the whole "::import ..." line: a bare 'grep -v org.apache.doris.connector' matches the FILE +# PATH too (regex '.' matches '/'), so a forbidden import inside a src/.../org/apache/doris/connector/** file +# would be dropped by its location — blinding the gate to the very modules it guards. Import-anchored keeps the +# real allowance (imports OF these packages) without the path-collision false negative. +ALLOWED_IMPORT=':import[[:space:]]+(static[[:space:]]+)?org[.]apache[.]doris[.](connector|thrift|extension|filesystem)[.]' -CANDIDATES=$(grep -rEn "^import ${FORBIDDEN}\." "${ROOT}"/*/src/main/java 2>/dev/null \ - | grep -v 'org.apache.doris.thrift' \ - | grep -v 'org.apache.doris.connector' \ - | grep -v 'org.apache.doris.extension' \ - | grep -v 'org.apache.doris.filesystem' || true) +CANDIDATES=$(grep -rEn "^import[[:space:]]+(static[[:space:]]+)?${FORBIDDEN}[.]" \ + "${ROOT}"/*/src/main/java "${ROOT}"/*/src/test/java 2>/dev/null \ + | grep -vE "${ALLOWED_IMPORT}" || true) # A flagged import is a FALSE POSITIVE when the connector module VENDORS its own self-contained copy of the # class: a real source file inside a fe-connector module whose package happens to match a fe-core prefix (e.g. @@ -82,7 +92,7 @@ if [ -n "${CANDIDATES}" ]; then while IFS= read -r line; do [ -z "${line}" ] && continue # line = ::import ; - fqn=$(printf '%s\n' "${line}" | sed -E 's/.*import[[:space:]]+//; s/;.*//') + fqn=$(printf '%s\n' "${line}" | sed -E 's/.*import[[:space:]]+(static[[:space:]]+)?//; s/;.*//') if is_vendored "${fqn}"; then echo "check-connector-imports: skipping vendored same-module import: ${fqn}" >&2 continue diff --git a/tools/check-connector-imports.test.sh b/tools/check-connector-imports.test.sh new file mode 100755 index 00000000000000..fb4edd63d9e8ca --- /dev/null +++ b/tools/check-connector-imports.test.sh @@ -0,0 +1,136 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# +# Self-test for tools/check-connector-imports.sh. +# +# The forbidden-import gate exits 0 on the real (clean) tree both before and +# after the hardening, so a controlled RED/GREEN fixture is the only way to prove +# the gate actually catches what it must and to lock the behavior against silent +# regression. Each seeded violation below targets one gate property: +# - a static import of a forbidden package (must NOT slip past `import static`) +# - imports of the 6 added packages (persist/transaction/fs/statistics/mysql/service) +# - a forbidden import in a src/test/java file (test sources must be scanned) +# - a forbidden import in an org.apache.doris.connector.** file +# (must be judged by import target, not file path) +# - a static import of a VENDORED class (must be SKIPPED, not falsely reported) +# and the allow-cases (thrift/filesystem/connector SPI + the non-static vendored import) must stay silent. +# +# Usage: bash tools/check-connector-imports.test.sh # exit 0 = pass, 1 = fail + +set -u + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +GATE="${SCRIPT_DIR}/check-connector-imports.sh" + +FX="$(mktemp -d)" +trap 'rm -rf "${FX}"' EXIT + +MOD="${FX}/fe-connector-fake" +mkdir -p "${MOD}/src/main/java/org/apache/doris/fake" \ + "${MOD}/src/main/java/org/apache/doris/connector/fake" \ + "${MOD}/src/main/java/org/apache/doris/datasource/hive" \ + "${MOD}/src/test/java/org/apache/doris/fake" + +# token-free package: holes 1 (static) + 2 (six packages), legit allow-imports, both vendored imports. +cat > "${MOD}/src/main/java/org/apache/doris/fake/FakeConn.java" <<'EOF' +package org.apache.doris.fake; +import static org.apache.doris.catalog.Type.INT; +import org.apache.doris.persist.EditLog; +import org.apache.doris.fs.remote.RemoteFileSystem; +import org.apache.doris.statistics.ColumnStatistic; +import org.apache.doris.mysql.privilege.Auth; +import org.apache.doris.service.FrontendOptions; +import org.apache.doris.thrift.TFoo; +import org.apache.doris.filesystem.Bar; +import org.apache.doris.connector.api.Baz; +import org.apache.doris.datasource.hive.HiveVersionUtil; +import static org.apache.doris.datasource.hive.HiveVersionUtil.SOME_CONST; +public class FakeConn {} +EOF + +# vendored self-defined class: same-module definition => both imports above must be skipped. +cat > "${MOD}/src/main/java/org/apache/doris/datasource/hive/HiveVersionUtil.java" <<'EOF' +package org.apache.doris.datasource.hive; +public class HiveVersionUtil { public static final int SOME_CONST = 1; } +EOF + +# hole 4: a forbidden import inside a connector-namespaced file. The gate must report it by import TARGET; +# a path-matching whitelist would drop it because the path contains org/apache/doris/connector/. +cat > "${MOD}/src/main/java/org/apache/doris/connector/fake/ConnPathConn.java" <<'EOF' +package org.apache.doris.connector.fake; +import org.apache.doris.catalog.Type; +public class ConnPathConn {} +EOF + +# hole 3: forbidden import in a test source. +cat > "${MOD}/src/test/java/org/apache/doris/fake/FakeConnTest.java" <<'EOF' +package org.apache.doris.fake; +import org.apache.doris.transaction.TransactionState; +public class FakeConnTest {} +EOF + +OUT="$(bash "${GATE}" "${FX}" 2>&1)"; EC=$? + +FAILED=0 +fail() { echo "FAIL: $1"; FAILED=1; } + +# 1) The gate must reject the fixture. +[ "${EC}" -eq 1 ] || fail "expected exit 1 (violations present), got ${EC}" + +# 2) The reported-violation lines are those printed as ::import ... +REPORTED="$(printf '%s\n' "${OUT}" | grep -E "^${FX}.*:[0-9]+:import" || true)" +N="$(printf '%s\n' "${REPORTED}" | grep -c ':import' )" +[ "${N}" -eq 8 ] || fail "expected exactly 8 reported violations, got ${N}"$'\n'"${REPORTED}" + +# 3) Every seeded violation must be reported (one property each). +must_report() { + printf '%s\n' "${REPORTED}" | grep -qF "$1" || fail "violation NOT reported: $1" +} +must_report 'import static org.apache.doris.catalog.Type.INT;' # hole 1 (static) +must_report 'import org.apache.doris.persist.EditLog;' # hole 2 +must_report 'import org.apache.doris.fs.remote.RemoteFileSystem;' # hole 2 (fs != filesystem) +must_report 'import org.apache.doris.statistics.ColumnStatistic;' # hole 2 +must_report 'import org.apache.doris.mysql.privilege.Auth;' # hole 2 +must_report 'import org.apache.doris.service.FrontendOptions;' # hole 2 +must_report 'ConnPathConn.java:2:import org.apache.doris.catalog.Type;' # hole 4 (connector-namespaced) +must_report 'FakeConnTest.java:2:import org.apache.doris.transaction.TransactionState;' # hole 3 (test src) + +# 4) Allow-cases and vendored imports must NOT be reported as violations. +must_not_report() { + printf '%s\n' "${REPORTED}" | grep -qF "$1" && fail "should NOT be reported: $1" || true +} +must_not_report 'org.apache.doris.thrift.TFoo' # SPI-shared +must_not_report 'org.apache.doris.filesystem.Bar' # SPI-shared +must_not_report 'org.apache.doris.connector.api.Baz' # SPI +must_not_report 'HiveVersionUtil;' # vendored (non-static) +must_not_report 'HiveVersionUtil.SOME_CONST;' # vendored (static) — E3 + +# 5) Both vendored imports must have been actively skipped (proves is_vendored ran on the static one too). +printf '%s\n' "${OUT}" | grep -q 'skipping vendored .*HiveVersionUtil$' \ + || fail "non-static vendored import was not skipped" +printf '%s\n' "${OUT}" | grep -q 'skipping vendored .*HiveVersionUtil.SOME_CONST$' \ + || fail "static vendored import was not skipped (E3 static-strip regressed)" + +if [ "${FAILED}" -eq 0 ]; then + echo "PASS: check-connector-imports.sh catches all 4 holes; allow-cases and vendored imports stay silent." + exit 0 +fi +echo "---- gate output ----" +printf '%s\n' "${OUT}" +exit 1 From a706f16a4e2bb11facff0d9b15541f8d7854e038 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 21:32:36 +0800 Subject: [PATCH 300/334] [doc](catalog) #65185 reverify: L1 done (import-gate, +4th hole), M8 skipped per user MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Land FIX-L1-design.md (design + red-team wf_643c11b4-3fe folded, incl. the 4th hole the reverify's "three holes" missed: path-based whitelist suppression). - Mark L1 DONE (88aa55b831b), M8 deferred at user's 2026-07-12 request (recon preserved in the tracking table — not silently dropped). - Repoint HANDOFF next-step to batch 4 (low-risk connector L-items). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 44 +++++----- plan-doc/task-list-65185-reverify-fixes.md | 21 ++++- plan-doc/tasks/designs/FIX-L1-design.md | 98 ++++++++++++++++++++++ 3 files changed, 139 insertions(+), 24 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L1-design.md diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md index d56e776b020c07..a7cb8e058c4488 100644 --- a/plan-doc/HANDOFF-65185-reverify-fixes.md +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -1,7 +1,7 @@ # 🔖 HANDOFF —— #65185 复核修复系列(独立任务线快照) > **用途**：这是**「复核修复系列」这一条任务线**的**独立**交接快照，与主线滚动 `plan-doc/HANDOFF.md`（HMS 翻闸主线）**分开**。你切去做别的实现后，回到这条线时**先读本文件**即可续做，不必炒对话历史。 -> **生成**：2026-07-11。**分支** = `catalog-spi-11-hive`（off `branch-catalog-spi`，PR base = `branch-catalog-spi`，squash 合并）。**HEAD 快照** = `af1cb7e853a`。 +> **生成**：2026-07-11（更新 2026-07-12）。**分支** = `catalog-spi-11-hive`（off `branch-catalog-spi`，PR base = `branch-catalog-spi`，squash 合并）。**HEAD 快照** = `88aa55b831b`（本线 L1 提交；主线另有 HMS 翻闸提交穿插,与本线文件不重叠,fast-forward）。 > **公开 tracking issue** = apache/doris#65185。 --- @@ -34,9 +34,9 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 |------|------|------| | 批次 0 | H1–H4 高危（hudi/hive 分区剪枝：转义/DATETIME/非-hive-style/混大小写列名） | ✅ 全 DONE + 3-skeptic 复审 CLEAN | | 批次 1 | M5/M7/M6/M4/M2 中危（连接器局部：iceberg×3 + mc + hive） | ✅ 全 DONE + 最终复核 CLEAN | -| **批次 2** | **M3→M1 中危(fe-core 通用节点)** | **✅ 全 DONE(本轮)** | -| 批次 3 | M8(发布工具/文档) + L1(import 门禁) | ⬜ **← 回来从这里续** | -| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | ⬜ | +| 批次 2 | M3→M1 中危(fe-core 通用节点) | ✅ 全 DONE | +| **批次 3** | **L1(import 门禁) + M8(发布工具/文档)** | **✅ L1 DONE;M8 ⏸ 用户跳过(07-12)** | +| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | ⬜ **← 回来从这里续(用户指向)** | | 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | | 设计债 | D-系列 | ⏸ 择机 / 随 P8 | @@ -60,22 +60,26 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 --- -## 3. 下一步 = 批次 3（细节，回来直接开工） - -**M8 — 升级只换 lib 不部署 plugins → 首访抛（reverify §3 M8；无 fe 编译）** -- 现码：`PluginDrivenExternalCatalog.java:135` throw；`CatalogFactory` degraded 只护启动。翻闸把 blast radius 扩到全部 - `type=hms` 目录。 -- Fix（主线）：发布/升级工具把连接器 jar 部署到 `connector_plugin_root`（`build.sh`/部署步骤）+ 响亮 release note。 - 可选防御：replay 后聚合 ERROR 枚举所有 degraded 目录。**保留** first-access throw，**不**加 legacy fallback。 -- 测试：升级文档步骤评审；（可选）degraded 聚合日志单测。 - -**L1 — import-gate 三洞（reverify §1）** -- `tools/check-connector-imports.sh:48,50`：grep 改 `^import (static )?${FORBIDDEN}[.]`；`FORBIDDEN` 补 - `persist|transaction|fs|statistics|mysql|service`；glob 覆盖 `src/test/java`。**保留** `is_vendored()` - （`datasource.hive.HiveVersionUtil` 是 fe-connector-hms vendored 副本，误报非违规；memory - `catalog-spi-hms-hiveversionutil-gate-false-positive`）。 - -之后：批次 4（低危连接器）；决策类（L2/L10/L12/L20）先问用户；设计债 D-系列择机。跟踪表「建议批次」节有全清单。 +## 3. 批次 3 收尾 + 下一步 = 批次 4（低危连接器） + +**批次 3 结果**： +- **L1 DONE** `88aa55b831b`（code+test）—— import 门禁补 3 洞 + **设计红队 `wf_643c11b4-3fe` 发现的第 4 洞**： + 4 条白名单 `grep -v` 按**整行**匹配（正则 `.`≡`/`）→ 连接器命名空间文件（608 个,全根在 `org.apache.doris.connector.**`） + 里的违规 import 被**按路径抑制**,门禁对其本该守护的模块**结构性失明**（实测:连接器目录下放 `import ...catalog.Type;` + 旧脚本 exit 0 放行）。修法=候选 grep 加宽（static / test glob / +6 包）+ **白名单锚定到 import 目标**（`:import ...` + 非整行）+ fqn sed 剥 `static`（修 static-vendored 误报,红队证 E3=正确性非装饰）+ 新增自测 `check-connector-imports.test.sh` + （8 违规上报/2 vendored skip/3 allow;GREEN 于新、RED 于旧、真树 exit 0）。设计 `designs/FIX-L1-design.md`。**非 live**。 +- **M8 ⏸ 用户 2026-07-12 明确跳过**（转做 L 系列）。侦察留档:`build.sh:1069-1083` 已部署连接器插件到 + `output/fe/plugins/connector/`（**非构建缺口**）;缺口在**升级流程**——只替 `fe/lib/` 漏拷新 `fe/plugins/connector/` 目录 + → replay 时全部 `type=hms` degraded（`CatalogFactory.java:119-127`）→首访抛（`PluginDrivenExternalCatalog.java:148-150`）。 + 修法=升级文档/release-note + (**可选**)replay 收尾聚合 degraded ERROR（触 fe-core,需编译）。**不 silently drop**,表中留待办; + 回来做时先中文讲清「仅文档 vs 文档+可选防御码」让用户拍板。 + +**下一步 = 批次 4（低危连接器,用户指向）**：直接可做=`L3–L9`（trino 事务/单例/去重/NPE、kerberos guard/interrupt、mc 谓词）、 +`L11/L13/L14`（paimon file_format/nullability/ignore_split_type）、`L15–L19`（paimon metrics 悬空、iceberg 缓存/version-blind/ +未知类型、partition_columns 撞名）。逐条走单任务循环（复核现码→设计→红队→实现→build+UT→独立 commit→勾表→更 HANDOFF）。 +⏸ **决策类 `L2/L10/L12/L20` 先中文讲清背景+选项问用户再动**（memory `ask-user-explain-in-chinese-first`）。设计债 D-系列择机。 +跟踪表「建议批次」节有全清单。 --- diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index c686b116fe39ef..116efc6511ebcc 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -36,8 +36,8 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 9 | **M5** | 🟠中 | iceberg | computeRowCount 丢 equality-delete gate | ✅ | ✅ | ✅ | ✅ | | 10 | **M6** | 🟠中 | iceberg | s3tables 无显式凭证硬失败(丢默认链) | ✅ | ✅ | ✅ | ✅ | | 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ✅ | ✅ | ✅ | ✅ | -| 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⬜ | -| 13 | **L1** | 🟡低 | tools | import-gate 三洞 | ⬜ | ⬜ | ⬜ | ⬜ | +| 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⏸ 跳过(用户 07-12;侦察见下) | +| 13 | **L1** | 🟡低 | tools | import-gate 三洞**+第4洞** | ✅ | ✅ | ✅ | ✅ | | 14 | **L2** | 🟡低 | fe-core | 翻闸 hive 丢 SQL 缓存资格 + COUNTER 停增 | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | | 15 | **L3** | 🟡低 | trino | 元数据事务从不 commit/close | ⬜ | ⬜ | ⬜ | ⬜ | | 16 | **L4** | 🟡低 | trino | plugin.dir 首胜单例(fail-loud) | ⬜ | ⬜ | ⬜ | ⬜ | @@ -160,7 +160,12 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **Test intent**:仅 `AWS_REGION` 的 props → `CLIENT_REGION` 被发(RED:null)。 ### M8 — 升级只换 lib 不部署 plugins → 首访抛 · reverify §3 M8 -- [ ] **M8**(无 fe 编译;发布工具 + 文档) +- [ ] **M8** ⏸ **用户 2026-07-12 要求跳过**（运营/文档欠账,非代码 bug;不 silently drop,登记待办）。 + **侦察结论(留档)**:`build.sh:1069-1083` 已把每个连接器 zip 解到 `output/fe/plugins/connector//`,故**非构建缺口**; + 缺口在**升级流程**——运营若只替换 `fe/lib/` 而漏拷新的 `fe/plugins/connector/` 目录,FE 重启 replay 时全部 `type=hms` + 目录进 degraded(`CatalogFactory.java:119-127`)→首访抛(`PluginDrivenExternalCatalog.java:148-150`)。修法=升级文档/release-note + 响亮提示新增 `fe/plugins/connector/` + (**可选**)replay 收尾聚合 ERROR 枚举 degraded 目录(触碰 fe-core,需编译)。 + **保留** first-access throw,不加 legacy fallback。回来做时:先用中文讲清「仅文档」vs「文档+可选防御码」让用户拍板。 - **现码**:`PluginDrivenExternalCatalog.java:135` throw;`CatalogFactory` degraded 只护启动 `:119-127`。翻闸把 blast radius 扩到全部 type=hms 目录。 - **Fix**:主线 = 发布/升级工具把连接器 jar 部署到 `connector_plugin_root`(build.sh/部署步骤)+ 响亮 release note。代码侧可选防御:replay 后聚合 ERROR 枚举所有 degraded 目录。**保留** first-access throw,**不**加 legacy fallback。 - **Files**:`build.sh`/部署脚本、release-note/升级文档;(可选)`fe-core/.../CatalogFactory.java` 或 replay 收尾处聚合日志。 @@ -172,7 +177,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) > 每条一行「现码 → fix」,详见 reverify §1 表 + 对应正文。⏸ 三条(L2/L10/L12)先问用户。 -- [ ] **L1** import-gate 三洞 · `tools/check-connector-imports.sh:48,50`:grep 改 `^import (static )?${FORBIDDEN}[.]`;`FORBIDDEN` 补 `persist|transaction|fs|statistics|mysql|service`;glob 覆盖 `src/test/java`。保留 `is_vendored()`(HiveVersionUtil FP)。 +- [x] **L1** DONE `88aa55b831b`(设计 `designs/FIX-L1-design.md`;设计红队 `wf_643c11b4-3fe` 3 lens 全 SOUND_WITH_CHANGES)。补 3 洞(static / +6 包 / test 源)**+ 红队发现的第 4 洞**:4 条白名单 `grep -v` 按**整行**匹配(`.`≡`/`)→连接器命名空间文件(608 个,全根在 `org.apache.doris.connector.**`)里的违规 import 被**按路径抑制**,门禁对其结构性失明。修法=候选 grep 加宽(static+test glob+6 包)+**白名单锚定到 import 目标**(`:import ...` 非整行)+ fqn sed 剥 `static`(修 static-vendored 误报,红队证 E3 属正确性非装饰)。新增自测 `tools/check-connector-imports.test.sh`(8 违规/2 vendored skip/3 allow;GREEN 于新、RED 于旧,真树 exit 0)。保留 `is_vendored()`(HiveVersionUtil FP)。**观察**:grep-denylist 固有盲区(新增内部包漏登记/FQN-无-import 内联/间距)登记设计债,根治须 allowlist/ArchUnit(Trino 先例)。 - [ ] **L2** ⏸ 翻闸 hive 丢 SQL 缓存资格 · `CacheAnalyzer.java:308/316/319`(instanceof HiveScanNode)+ `BindRelation.java:887`(instanceof HMSExternalTable)。**需决策**:加 `ConnectorCapability` 让 `PluginDrivenScanNode`/`PluginDrivenExternalTable` 被识别并恢复缓存 **vs** 登记为 fail-safe 验收偏差(`enable_hive_sql_cache` 默认关)。 - [ ] **L3** trino 事务从不 commit/close · `TrinoConnectorDorisMetadata.java`(6 处)+ `TrinoScanPlanProvider.java:112`(scan 站有意保持):元数据 6 站 try/finally `commit(txn)`(read-only 廉价)。scan 站不动。 - [ ] **L4** trino plugin.dir 首胜单例 · `TrinoBootstrap.java:136,316`:单例已存在且 pluginDir 不同时 fail-loud 抛(而非静默用旧);或删 per-catalog 分支只认全局 config。 @@ -256,3 +261,11 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **M3** DONE `6963de4124f`(code) — batch 闸门 `!isPruned`→`== NOT_PRUNED`:无谓词大分区表(MaxCompute + 翻闸 hive)恢复异步 batch split(legacy `MaxComputeScanNode:227` 的 `!= NOT_PRUNED` + sibling `displayPartitionCounts` 双证;git `1da88365e85^` 取证 + 全 producer 枚举证闭合)。**顺带解 M2 的 BATCH-UNPRUNED-SYNC**。设计红队 `wf_811e6242-d8b`(3 lens:BLAST-RADIUS SOUND / LEGACY-PARITY SOUND_WITH_CHANGES / COMPLETENESS SOUND_WITH_CHANGES)命中 1 blocker(docker-hive golden 未 reconcile)+ 1 major(overturn 前次签字 LP-1/D-035「等价」未登记)均已解:**反转** pinning 测试(`testUnprocessedPruningNeverBatches`→`testNoPredicatePartitionedTableBatches`,assertTrue)、**登记 supersession**(D-035/DV-019 补 SUPERSEDED 批注)、**docker-hive golden** `test_hive_partitions:200` `(approximate)inputSplitNum` `60→6`(**用户 2026-07-11 签字:SPI 统一分区数口径**,非 hive 专属 split-count 估算;对齐 MaxCompute/Trino「引擎层统一报分片」)。`PluginDrivenScanNodeBatchModeTest` **12/12** 绿、fe-core test-compile BUILD SUCCESS 0 checkstyle。**e2e live-gated**(docker-hive `(approximate)inputSplitNum=6` + MaxCompute 无谓词 ≥阈值分区表进 batch;sweep 确认全 regression 仅此 1 处 `(approximate)` 断言受影响,maxcompute p2 只断言 `partition=N/M`/结果不受影响)。**⚠ 构建坑**:本轮 UT 一度被并行 session 的 `be-java-extensions package -am -T 1C` 构建污染共享 target 报「cannot access 生成类」假失败(非本码);待其结束后干净重跑 12/12(memory `concurrent-sessions-shared-worktree-hazard`)。 - **M1** DONE `17b432dc1e1`(code) — TABLESAMPLE 翻闸 hive 静默全表扫修复。**红队 `wf_32decfa0-349` 推翻原"通用采样"方案(UNSOUND)**:`Split.getLength()` 对 MaxCompute 默认 byte_size / Paimon JNI range 报 -1、对 MaxCompute row_offset 报行数 → 盲目按字节采样出乱结果(全表或 1 split)。**scope 更正=hive-only 回归**(只有 hive 曾采样)。→ 改**连接器 opt-in**(`ConnectorScanPlanProvider.supportsTableSample()` 默认 false、仅 `HiveScanPlanProvider` true;**用户 2026-07-11 签字 scope=hive-only**)。translator 插件臂通用转发 + `PluginDrivenScanNode.sampleSplits`(仅 `applySample` 时,legacy `selectFiles` 端口)+ 非支持连接器 no-op+WARN(非静默)+ 两 gate 门(COUNT 抑制/batch 抑制,挂 `applySample`)。对齐 Trino `applySample` + 上一批 `supportsBatchScan` opt-in 先例。`PluginDrivenScanNodeTableSampleTest` 6/6 + BatchMode 12/12 + hive 285/285 绿、0 checkstyle、import 门净。e2e live-gated(docker-hive 结果不变式已加;强基数缩减须多文件 fixture 真集群验)。 - **⭐ 批次 2(M3→M1)全部 DONE。** 2 条 fe-core 通用节点修复,各配 RED-able 单测 + 独立 code/doc commit + 设计红队(各 3 lens;**M1 红队捕获并推翻 UNSOUND 原方案 → 触发用户 scope 决策**)。**下一步 = 批次 3(M8 发布工具/文档 + L1 import 门禁)**;之后批次 4(低危连接器)… ⏸ 决策类(L2/L10/L12/L20)先问用户。 + +--- + +**⭐ 批次 3 = L1 DONE;M8 用户跳过(2026-07-12)** + +- **L1** DONE `88aa55b831b`(code+test) — import 门禁补 3 洞 + **红队发现的第 4 洞**(白名单按整行匹配→连接器命名空间文件违规 import 被路径抑制,门禁对 608 个连接器实现文件结构性失明)。修法=候选 grep 加宽(static/test/6 包)+ 白名单锚定 import 目标 + fqn 剥 static;新增 `tools/check-connector-imports.test.sh`(RED 于旧/GREEN 于新/真树 exit 0)。设计红队 `wf_643c11b4-3fe` 3 lens 全 SOUND_WITH_CHANGES,发现全部逐条复现并折入。**非 live**(纯工具,无 e2e 欠账)。 +- **M8** ⏸ **用户 2026-07-12 明确要求跳过**(转做 L 系列)。侦察结论已留档(build.sh 已部署插件,缺口在升级流程+文档,含一个「仅文档 vs 文档+可选防御码」的用户决策)。**不 silently drop**——保留在表中待办,回来做时先中文讲清再拍板。 +- **下一步(用户指向 L 系列)**:直接可做的低危连接器 L 条=`L3–L9`(trino/kerberos/mc)、`L11/L13/L14`(paimon)、`L15–L19`(paimon/iceberg 杂项)。⏸ 决策类 `L2/L10/L12/L20` 须先中文讲清背景+选项问用户再动(memory `ask-user-explain-in-chinese-first`)。 diff --git a/plan-doc/tasks/designs/FIX-L1-design.md b/plan-doc/tasks/designs/FIX-L1-design.md new file mode 100644 index 00000000000000..ae37d26581630d --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L1-design.md @@ -0,0 +1,98 @@ +# FIX-L1 — import-gate 漏洞补齐（三洞 + 红队发现的第 4 洞） + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §1 表 L1（原 P0-5）。 +> 严重度：🟡 低（门禁/防御性；无 fe 编译）。范围：`tools/check-connector-imports.sh` + 新增 `.test.sh`。 +> HEAD 复核基线：`500472410d5`。 +> **设计红队**：`wf_643c11b4-3fe`（3 lens：correctness / false-positive / scope，均 SOUND_WITH_CHANGES） +> **发现并已折入原设计漏掉的第 4 洞 + 修正 E3 定性 + 修正测试夹具**。已逐条独立复现验证。 + +## Problem + +`tools/check-connector-imports.sh` 是「连接器模块不得 import fe-core 内部包」的防线（RFC §15.4）。 +reverify §1 记为**三个漏洞**，红队复现时又发现**第 4 个、更致命的结构性漏洞**： + +1. **漏 `import static`**：候选 grep 只匹配 `^import org.apache.doris..`，不匹配 `import static ...`。 +2. **漏 6 个 fe-core 内部包**：`FORBIDDEN` 缺 `persist|transaction|fs|statistics|mysql|service` + （fe-core 下各有 89/35/13/81/95/23 个 `.java`，都是连接器绝不应 reach 的内部包）。 +3. **只扫 `src/main/java`**：glob 不覆盖 `src/test/java`（16 个连接器模块都有 test 源）。 +4. **⭐【红队发现】白名单按「文件路径」而非「import 目标」抑制**：4 条 `grep -v 'org.apache.doris.'` + （thrift/connector/extension/filesystem）匹配的是**整行**（`::import ...`）。BRE/ERE 里 `.` + 匹配任意字符**包括 `/`**，故 `grep -v 'org.apache.doris.connector'` 会命中**文件路径** + `.../src/main/java/org/apache/doris/connector/**` → 把该文件里的**任何**违规 import 整行丢掉。 + 而**全部 608 个连接器实现文件**都根在 `org.apache.doris.connector.**`（hive/paimon/jdbc/es 实测） + → 门禁对**它本该守护的那些文件结构性失明**。实测：连接器命名空间文件里放 `import org.apache.doris.catalog.Type;` + 现版脚本 **exit 0**（放行）。→ 洞 1/2/3 的加宽对连接器文件**基本无效**（都被路径抑制先丢了）。 + +## Root Cause + +denylist 门禁四处都写错：正则不含 `static`、包清单缺 6 项、glob 只有 main、**白名单抑制误用整行匹配（含路径）**。 +关键洞察：4 个白名单包（connector/thrift/extension/filesystem）都是 fe-core 的**兄弟包**，**不是任何 FORBIDDEN 包的子包** +→ 候选 grep（`^import ${FORBIDDEN}[.]`）**从不捕获**合法 SPI import → 这 4 条 `grep -v` **对其本意（放行 SPI import）纯冗余**， +唯一实际效果就是**按路径误伤**。 + +## Design（4 处功能编辑 + 1 处配套 + 1 处文档；均已实测验证） + +铁律对齐：tools 侧门禁强化，不碰任何 fe-core/连接器 Java 代码，无架构风险。 + +- **E1（洞 2）**行 48：`FORBIDDEN` 补 6 包。 +- **E2a（洞 1）**候选 grep 正则：`^import ${FORBIDDEN}\.` → `^import[[:space:]]+(static[[:space:]]+)?${FORBIDDEN}[.]` + （`[.]`≡`\.`；`[[:space:]]+` 顺带兜 tab/多空格——红队 nit，零风险 + 增强）。 +- **E2b（洞 3）**glob：`${ROOT}/*/src/main/java` → 同时列 `${ROOT}/*/src/test/java`。 +- **E2c（洞 4，红队发现）**：删 4 条按整行的 `grep -v`，换为**单条锚定到 import 目标**的排除： + `grep -vE ':import[[:space:]]+(static[[:space:]]+)?org[.]apache[.]doris[.](connector|thrift|extension|filesystem)[.]'` + （`:import` 前缀锁定到冒号后的 import 内容，不再命中文件路径）。 +- **E3（配套；红队修正定性=正确性必需，非"cosmetic"）**行 85 fqn 抽取 sed：补 `(static[[:space:]]+)?`。 + **红队证**：无 E3 时 `import static org.apache.doris.datasource.hive.HiveVersionUtil.X;` 会被**误报** + （is_vendored 拿到带 `static ` 前缀的 fqn → 永远找不到 vendored 文件 → 当违规上报）；有 E3 才正确 skip。 + 当前树零此类 import 故 latent，但一旦出现即真误报 → 属正确性修，非装饰。 +- **E4（文档同步）**头注把包清单、含 static / 含 test / import-anchored 白名单写进注释，保持自说明。 + +**保留不动**：`is_vendored()`（HiveVersionUtil vendored FP，memory +`catalog-spi-hms-hiveversionutil-gate-false-positive`）。其 `[A-Z]*` 段判定在 en_US collation 会连带匹配小写 +（红队 nit）——当前树零 `import static org.apache.doris.*` 故 inert，本条不动，登记观察。 + +## Risk Analysis + +- **不破当前构建**：对 HEAD 用**完整修复版**脚本实测（含 E2c 锚定），真实 `fe/fe-connector` 仍 `exit 0` + （3× 确定性；仅 2 条 vendored HiveVersionUtil skip）。连接器今日 import 的 812 `connector`/143 `thrift`/ + 30 `filesystem`/4 `extension` 全被锚定白名单正确放行，且**零** forbidden import → E2c 不 surface 任何存量违规。 + 故第 4 洞 latent 非 live（红队证），折入安全。 +- **`fs` vs `filesystem` 误伤**：`fs[.]` 需字面 `fs.`；`filesystem` 里 `fs` 后接 `ystem` 不匹配。已实测。 +- **glob 边界**：`*/src/test/java` ≥1 模块匹配 → bash 只展开存在目录；`2>/dev/null`+`|| true`+`set -e` 安全。 +- **denylist 仍非穷举**（`system/load/backup/...` 未列；**FQN-无-import 内联用法**、多空格/tab 等 grep-denylist 固有盲区） + ——根治须 allowlist（Trino 用 ArchUnit 按字节码约束 `classesShouldOnlyDependOnAllowedPackages`，能连 FQN 内联一并拦）。 + **本条不做**，登记为设计观察（见文末）。当前树对这些盲区实测零命中。 + +## Implementation Plan + +改 `tools/check-connector-imports.sh` 行 48 / 50-54 / 85 + 头注；新增 `tools/check-connector-imports.test.sh`。 + +## Test Plan + +### Durable 自测（`tools/check-connector-imports.test.sh`，mktemp -d + trap 自清） + +构造临时 fixture ROOT，一个假连接器模块，覆盖全部 4 洞 + E3 + 白名单 + vendored： +- `src/main/java/org/apache/doris/**fake**/FakeConn.java`（**token-free 包**，避开白名单 token 撞路径）： + 洞 1 `import static org.apache.doris.catalog.Type.INT;`；洞 2 `persist/fs/statistics/mysql/service` 各一； + 合法 `thrift/filesystem/connector.api`（须**不**报）；vendored `import ...HiveVersionUtil;`（须 skip）； + **E3** `import static ...HiveVersionUtil.SOME_CONST;`（无 E3 会误报 → 须 skip）。 +- `src/main/java/org/apache/doris/datasource/hive/HiveVersionUtil.java`：vendored 定义。 +- **洞 4** `src/main/java/org/apache/doris/**connector**/fake/ConnPathConn.java`：`import org.apache.doris.catalog.Type;` + ——文件在连接器命名空间下，**旧脚本按路径丢弃（放行）、新脚本须报**（锁死第 4 洞回归）。 +- **洞 3** `src/test/java/org/apache/doris/fake/FakeConnTest.java`：`import org.apache.doris.transaction.TransactionState;`。 + +断言（已用 scratch 预演实证）： +- **RED**（改前脚本 / 或从 git 取旧版）：对该 fixture **exit 0、零上报**（四洞全漏，含连接器路径那条被路径抑制）。 +- **GREEN**（改后脚本）：**exit 1**，恰报 **8** 条违规（static-catalog / persist / fs / statistics / mysql / + service / **连接器路径-catalog** / test-transaction），**不**含 thrift/filesystem/connector.api，2 条 vendored 均 skip。 +- **真树回归**：改后脚本对真实 `fe/fe-connector` 仍 `exit 0`。 + +### E2E + +不涉及运行时行为，无 e2e。 + +## 观察（不在本条实现，登记设计债） + +grep-denylist 有三类固有盲区，本条只补 4 洞、无法根治：① 新增 fe-core 内部包忘登记；② FQN-无-import 内联用法 +（`^import` grep 抓不到，当前树零命中）；③ 非规范 import 间距（E2a 已兜多空格/tab）。根治=allowlist / ArchUnit +（Trino 先例，字节码级、连内联 FQN 一并拦）。属独立重构，登记观察，本条不做。 From 4cd63c6911a178e26007f58884aa81dc374df370 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 21:52:02 +0800 Subject: [PATCH 301/334] [fix](catalog) trino: release read-only metadata transactions (no more leak) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit TrinoConnectorDorisMetadata's 6 FE-only metadata methods (listDatabaseNames, listTableNames, getTableHandle, getTableSchema, applyFilter, applyProjection) each open a Trino read-only transaction via beginTransaction(READ_UNCOMMITTED, true, true) and never end it. The Trino Connector contract requires exactly one of commit/rollback to end a transaction (even in auto-commit mode), so each metadata call leaked a transaction handle into the connector's transaction manager — accumulating over every plan. (Parity with master; the SPI cutover amplified the volume for trino-connector catalogs.) Wrap each site in try/finally and release via a small releaseQuietly helper that commits and swallows+logs any release failure so it cannot mask the real exception a finally runs after. A read-only READ_UNCOMMITTED transaction has nothing to write, so commit simply frees it. The try starts after beginTransaction, preserving the conditional-open in applyFilter/applyProjection (their early returns open no transaction). The scan site (TrinoScanPlanProvider.planScan) is deliberately left untouched: it serializes the transaction handle to BE for the JNI scanner, so that transaction must stay open. The returned table/column handles are immutable opaque value objects reused later under the scan's own fresh transaction, so committing the metadata transaction does not invalidate them. No behavioral UT: driving these methods requires constructing io.trino.Session (a heavyweight concrete class with no existing test fixture in this module); fabricating the full Trino SPI fakes is disproportionate for this low-severity parity leak. Verified by compile (commit signature, 0 checkstyle), the import gate (exit 0), the existing trino UTs, and an adversarial design review; behavioral validation is e2e live-gated (needs a live trino cluster). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../trino/TrinoConnectorDorisMetadata.java | 275 ++++++++++-------- 1 file changed, 157 insertions(+), 118 deletions(-) diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java index f71f4d9a94d3ab..4b9ec8a4785bd3 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java @@ -78,15 +78,34 @@ public TrinoConnectorDorisMetadata( this.trinoCatalogHandle = trinoCatalogHandle; } + /** + * Releases a Trino metadata transaction. Each read-only metadata method below opens a transaction + * that the Trino {@code Connector} contract requires be ended with exactly one of commit/rollback; + * a read-only READ_UNCOMMITTED transaction has nothing to write, so commit simply frees it from the + * connector's transaction manager. A release failure is logged, never rethrown, so it cannot mask + * the real exception a {@code finally} block runs after. + */ + private void releaseQuietly(io.trino.spi.connector.ConnectorTransactionHandle txn) { + try { + trinoConnector.commit(txn); + } catch (RuntimeException e) { + LOG.warn("Failed to release Trino metadata transaction", e); + } + } + @Override public List listDatabaseNames(ConnectorSession session) { io.trino.spi.connector.ConnectorSession connSession = trinoSession.toConnectorSession(trinoCatalogHandle); io.trino.spi.connector.ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); - io.trino.spi.connector.ConnectorMetadata metadata = - trinoConnector.getMetadata(connSession, txn); - return metadata.listSchemaNames(connSession); + try { + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + return metadata.listSchemaNames(connSession); + } finally { + releaseQuietly(txn); + } } @Override @@ -100,14 +119,18 @@ public List listTableNames(ConnectorSession session, String dbName) { trinoSession.toConnectorSession(trinoCatalogHandle); io.trino.spi.connector.ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); - io.trino.spi.connector.ConnectorMetadata metadata = - trinoConnector.getMetadata(connSession, txn); - - Optional schemaName = Optional.of(dbName); - List tables = metadata.listTables(connSession, schemaName); - return tables.stream() - .map(SchemaTableName::getTableName) - .collect(Collectors.toList()); + try { + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + Optional schemaName = Optional.of(dbName); + List tables = metadata.listTables(connSession, schemaName); + return tables.stream() + .map(SchemaTableName::getTableName) + .collect(Collectors.toList()); + } finally { + releaseQuietly(txn); + } } public boolean tableExists(ConnectorSession session, String dbName, String tableName) { @@ -125,33 +148,37 @@ public Optional getTableHandle( trinoSession.toConnectorSession(trinoCatalogHandle); io.trino.spi.connector.ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); - io.trino.spi.connector.ConnectorMetadata metadata = - trinoConnector.getMetadata(connSession, txn); - - SchemaTableName schemaTableName = new SchemaTableName(dbName, tableName); - io.trino.spi.connector.ConnectorTableHandle trinoHandle = - metadata.getTableHandle(connSession, schemaTableName, - Optional.empty(), Optional.empty()); - if (trinoHandle == null) { - return Optional.empty(); - } + try { + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + SchemaTableName schemaTableName = new SchemaTableName(dbName, tableName); + io.trino.spi.connector.ConnectorTableHandle trinoHandle = + metadata.getTableHandle(connSession, schemaTableName, + Optional.empty(), Optional.empty()); + if (trinoHandle == null) { + return Optional.empty(); + } - // Eagerly resolve column handles + metadata for the table - Map handles = metadata.getColumnHandles(connSession, trinoHandle); - ImmutableMap.Builder columnHandleMapBuilder = ImmutableMap.builder(); - ImmutableMap.Builder columnMetadataMapBuilder = ImmutableMap.builder(); - for (Map.Entry entry : handles.entrySet()) { - String colName = entry.getKey().toLowerCase(Locale.ENGLISH); - columnHandleMapBuilder.put(colName, entry.getValue()); - ColumnMetadata colMeta = metadata.getColumnMetadata( - connSession, trinoHandle, entry.getValue()); - columnMetadataMapBuilder.put(colMeta.getName(), colMeta); - } + // Eagerly resolve column handles + metadata for the table + Map handles = metadata.getColumnHandles(connSession, trinoHandle); + ImmutableMap.Builder columnHandleMapBuilder = ImmutableMap.builder(); + ImmutableMap.Builder columnMetadataMapBuilder = ImmutableMap.builder(); + for (Map.Entry entry : handles.entrySet()) { + String colName = entry.getKey().toLowerCase(Locale.ENGLISH); + columnHandleMapBuilder.put(colName, entry.getValue()); + ColumnMetadata colMeta = metadata.getColumnMetadata( + connSession, trinoHandle, entry.getValue()); + columnMetadataMapBuilder.put(colMeta.getName(), colMeta); + } - return Optional.of(new TrinoTableHandle( - dbName, tableName, trinoHandle, - columnHandleMapBuilder.buildOrThrow(), - columnMetadataMapBuilder.buildOrThrow())); + return Optional.of(new TrinoTableHandle( + dbName, tableName, trinoHandle, + columnHandleMapBuilder.buildOrThrow(), + columnMetadataMapBuilder.buildOrThrow())); + } finally { + releaseQuietly(txn); + } } @Override @@ -163,43 +190,47 @@ public ConnectorTableSchema getTableSchema( trinoSession.toConnectorSession(trinoCatalogHandle); io.trino.spi.connector.ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); - io.trino.spi.connector.ConnectorMetadata metadata = - trinoConnector.getMetadata(connSession, txn); - - Map columnHandles = trinoHandle.getColumnHandleMap(); - if (columnHandles == null || columnHandles.isEmpty()) { - columnHandles = metadata.getColumnHandles( - connSession, trinoHandle.getTrinoTableHandle()); - } + try { + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + Map columnHandles = trinoHandle.getColumnHandleMap(); + if (columnHandles == null || columnHandles.isEmpty()) { + columnHandles = metadata.getColumnHandles( + connSession, trinoHandle.getTrinoTableHandle()); + } - List columns = new ArrayList<>(); - for (ColumnHandle columnHandle : columnHandles.values()) { - ColumnMetadata colMeta = metadata.getColumnMetadata( - connSession, trinoHandle.getTrinoTableHandle(), columnHandle); - if (colMeta.isHidden()) { - continue; + List columns = new ArrayList<>(); + for (ColumnHandle columnHandle : columnHandles.values()) { + ColumnMetadata colMeta = metadata.getColumnMetadata( + connSession, trinoHandle.getTrinoTableHandle(), columnHandle); + if (colMeta.isHidden()) { + continue; + } + ConnectorType connType = TrinoTypeMapping.toConnectorType(colMeta.getType()); + // Mark every column as a key column to match the Doris external-table convention + // (legacy TrinoConnectorExternalTable.initSchema and JdbcClient do the same), so + // `desc

    ` reports Key=true for each column. + columns.add(new ConnectorColumn( + colMeta.getName(), + connType, + colMeta.getComment(), + true, + null, + true)); } - ConnectorType connType = TrinoTypeMapping.toConnectorType(colMeta.getType()); - // Mark every column as a key column to match the Doris external-table convention - // (legacy TrinoConnectorExternalTable.initSchema and JdbcClient do the same), so - // `desc
    ` reports Key=true for each column. - columns.add(new ConnectorColumn( - colMeta.getName(), - connType, - colMeta.getComment(), - true, - null, - true)); - } - Map tableProps = new HashMap<>(); - tableProps.put("trino.connector.table", "true"); + Map tableProps = new HashMap<>(); + tableProps.put("trino.connector.table", "true"); - return new ConnectorTableSchema( - trinoHandle.getTableName(), - columns, - "trino_connector", - Collections.unmodifiableMap(tableProps)); + return new ConnectorTableSchema( + trinoHandle.getTableName(), + columns, + "trino_connector", + Collections.unmodifiableMap(tableProps)); + } finally { + releaseQuietly(txn); + } } @Override @@ -237,28 +268,32 @@ public Optional> applyFilter( trinoSession.toConnectorSession(trinoCatalogHandle); io.trino.spi.connector.ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); - io.trino.spi.connector.ConnectorMetadata metadata = - trinoConnector.getMetadata(connSession, txn); + try { + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + Optional> trinoResult = + metadata.applyFilter(connSession, dorisHandle.getTrinoTableHandle(), + new Constraint(tupleDomain)); + if (!trinoResult.isPresent()) { + return Optional.empty(); + } - Optional> trinoResult = - metadata.applyFilter(connSession, dorisHandle.getTrinoTableHandle(), - new Constraint(tupleDomain)); - if (!trinoResult.isPresent()) { - return Optional.empty(); + TrinoTableHandle newHandle = new TrinoTableHandle( + dorisHandle.getDbName(), + dorisHandle.getTableName(), + trinoResult.get().getHandle(), + dorisHandle.getColumnHandleMap(), + dorisHandle.getColumnMetadataMap()); + + // Trino tracks the remaining filter as a TupleDomain, not as a Doris ConnectorExpression. + // Returning the original expression keeps BE-side re-evaluation, matching the legacy + // fe-core scan-node behavior. A future enhancement could try to map the remaining + // TupleDomain back to a ConnectorExpression and clear fully-pushed conjuncts. + return Optional.of(new FilterApplicationResult<>(newHandle, expression, false)); + } finally { + releaseQuietly(txn); } - - TrinoTableHandle newHandle = new TrinoTableHandle( - dorisHandle.getDbName(), - dorisHandle.getTableName(), - trinoResult.get().getHandle(), - dorisHandle.getColumnHandleMap(), - dorisHandle.getColumnMetadataMap()); - - // Trino tracks the remaining filter as a TupleDomain, not as a Doris ConnectorExpression. - // Returning the original expression keeps BE-side re-evaluation, matching the legacy - // fe-core scan-node behavior. A future enhancement could try to map the remaining - // TupleDomain back to a ConnectorExpression and clear fully-pushed conjuncts. - return Optional.of(new FilterApplicationResult<>(newHandle, expression, false)); } @Override @@ -298,37 +333,41 @@ public Optional> applyProjecti trinoSession.toConnectorSession(trinoCatalogHandle); io.trino.spi.connector.ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); - io.trino.spi.connector.ConnectorMetadata metadata = - trinoConnector.getMetadata(connSession, txn); - - Optional> trinoResult = - metadata.applyProjection(connSession, dorisHandle.getTrinoTableHandle(), - trinoProjections, assignments); - if (!trinoResult.isPresent()) { - return Optional.empty(); - } - - TrinoTableHandle newHandle = new TrinoTableHandle( - dorisHandle.getDbName(), - dorisHandle.getTableName(), - trinoResult.get().getHandle(), - colHandleMap, - colMetaMap); + try { + io.trino.spi.connector.ConnectorMetadata metadata = + trinoConnector.getMetadata(connSession, txn); + + Optional> trinoResult = + metadata.applyProjection(connSession, dorisHandle.getTrinoTableHandle(), + trinoProjections, assignments); + if (!trinoResult.isPresent()) { + return Optional.empty(); + } - List outProjections = new ArrayList<>(projections.size()); - List outAssignments = new ArrayList<>(projections.size()); - for (ConnectorColumnHandle col : projections) { - String colName = ((TrinoColumnHandle) col).getColumnName(); - ColumnMetadata cm = colMetaMap.get(colName); - if (cm == null) { - continue; + TrinoTableHandle newHandle = new TrinoTableHandle( + dorisHandle.getDbName(), + dorisHandle.getTableName(), + trinoResult.get().getHandle(), + colHandleMap, + colMetaMap); + + List outProjections = new ArrayList<>(projections.size()); + List outAssignments = new ArrayList<>(projections.size()); + for (ConnectorColumnHandle col : projections) { + String colName = ((TrinoColumnHandle) col).getColumnName(); + ColumnMetadata cm = colMetaMap.get(colName); + if (cm == null) { + continue; + } + ConnectorType type = TrinoTypeMapping.toConnectorType(cm.getType()); + ConnectorColumnRef ref = new ConnectorColumnRef(colName, type); + outProjections.add(ref); + outAssignments.add(new ConnectorColumnAssignment(col, ref)); } - ConnectorType type = TrinoTypeMapping.toConnectorType(cm.getType()); - ConnectorColumnRef ref = new ConnectorColumnRef(colName, type); - outProjections.add(ref); - outAssignments.add(new ConnectorColumnAssignment(col, ref)); + return Optional.of(new ProjectionApplicationResult<>(newHandle, outProjections, outAssignments)); + } finally { + releaseQuietly(txn); } - return Optional.of(new ProjectionApplicationResult<>(newHandle, outProjections, outAssignments)); } } From be96adf76ba76a4b527cc7f1edc26ee69c549fb1 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:03:32 +0800 Subject: [PATCH 302/334] [fix](catalog) trino: de-duplicate listTableNames (restore LinkedHashSet parity) listTableNames mapped listTables() to names and collected straight to a list, dropping the legacy LinkedHashSet de-duplication. Some Trino connectors list the same table name more than once for a schema (tables+views, multi-source merges). Add an order-preserving .distinct() to restore parity. The legacy prefix filter is intentionally not restored (listTables already filters by schema). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/connector/trino/TrinoConnectorDorisMetadata.java | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java index 4b9ec8a4785bd3..d27e184d92b81a 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoConnectorDorisMetadata.java @@ -125,8 +125,11 @@ public List listTableNames(ConnectorSession session, String dbName) { Optional schemaName = Optional.of(dbName); List tables = metadata.listTables(connSession, schemaName); + // distinct() restores the legacy LinkedHashSet de-dup (order-preserving): some Trino + // connectors list the same table name more than once (tables+views, multi-source merges). return tables.stream() .map(SchemaTableName::getTableName) + .distinct() .collect(Collectors.toList()); } finally { releaseQuietly(txn); From 18048f7f2175e5287f4634397ac54c7309086981 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:03:32 +0800 Subject: [PATCH 303/334] [fix](catalog) trino: publish guard field last to close init NPE window TrinoDorisConnector.ensureInitialized() uses `trinoConnector != null` as the double-checked-lock guard, and readers (getMetadata etc.) then dereference trinoSession / trinoCatalogHandle. doInitialize() assigned the guard field FIRST and its dependencies after, so a concurrent reader could observe `trinoConnector != null` while trinoSession was still null -> transient NPE. Even though every field is volatile, the volatile happens-before only guarantees writes made BEFORE the guard write are visible; the dependency writes came after. Assign trinoConnector LAST (after trinoSession/trinoCatalogHandle/ trinoConnectorName). Now observing the guard non-null guarantees the fully published dependencies. Classic safe-publication; all fields already volatile so the reorder alone suffices. Single-thread semantics unchanged. Verified by an adversarial concurrency review (JMM ordering + no reader bypasses the guard). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../apache/doris/connector/trino/TrinoDorisConnector.java | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java index 710c7a4cd102d7..64f31fd8f68196 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoDorisConnector.java @@ -173,10 +173,15 @@ private void doInitialize() { TrinoBootstrap.TrinoConnectionResult result = bootstrap.createConnection( context.getCatalogName(), connectorNameStr, trinoProperties); - this.trinoConnector = result.getConnector(); + // Publish the guard field (trinoConnector) LAST. ensureInitialized() and the other readers use + // `trinoConnector != null` as the initialized flag and then read trinoSession/trinoCatalogHandle. + // Assigning the guard after its dependencies means a concurrent reader that sees it non-null is + // guaranteed (via the volatile write/read happens-before) to also see the fully-published + // session / catalog handle / name — closing the transient half-initialized NPE window. this.trinoSession = result.getSession(); this.trinoCatalogHandle = result.getCatalogHandle(); this.trinoConnectorName = result.getConnectorName(); + this.trinoConnector = result.getConnector(); LOG.info("Trino connector initialized for catalog '{}', connector: {}", context.getCatalogName(), connectorNameStr); From e27602d4ab658f7fa2d2383dd99fa48f223571ad Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:03:32 +0800 Subject: [PATCH 304/334] [fix](catalog) trino: fail loud when TrinoBootstrap plugin dir differs TrinoBootstrap is a process-wide singleton loaded once with the first catalog's plugin dir. getInstance(pluginDir) was first-wins: a later catalog passing a DIFFERENT trino.plugin.dir was silently ignored and reused the first dir's plugins, so it could fail to find its connector factory or use the wrong plugins. Retain the plugin dir on the singleton and throw IllegalStateException when a later getInstance() requests a different one. Paths are canonicalized best-effort (getCanonicalPath, falling back to the raw string) so two spellings of the same physical directory (trailing slash, relative vs absolute, symlink) are treated as equal and do not trip a spurious mismatch. First-wins init behavior is unchanged; only the silent-wrong-dir case becomes a loud error. Verified by an adversarial concurrency review (write-once monotonic instance, null-safe compare). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/connector/trino/TrinoBootstrap.java | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java index eecb8078cddb56..f0b143d127402e 100644 --- a/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java +++ b/fe/fe-connector/fe-connector-trino/src/main/java/org/apache/doris/connector/trino/TrinoBootstrap.java @@ -69,6 +69,7 @@ import org.apache.logging.log4j.Logger; import java.io.File; +import java.io.IOException; import java.time.ZoneId; import java.util.Arrays; import java.util.HashMap; @@ -104,8 +105,12 @@ public class TrinoBootstrap { private final HandleResolver handleResolver; private final TypeRegistry typeRegistry; private final TrinoPluginManager pluginManager; + // The plugin dir this singleton was initialized with, retained so a later getInstance() with a + // different dir fails loudly instead of silently reusing the first dir's plugins. + private final String pluginDir; private TrinoBootstrap(String pluginDir) { + this.pluginDir = pluginDir; System.setProperty("jdk.attach.allowAttachSelf", "true"); String osName = System.getProperty("os.name").toLowerCase(Locale.ENGLISH); if (osName.contains("mac") || osName.contains("darwin")) { @@ -141,9 +146,31 @@ public static TrinoBootstrap getInstance(String pluginDir) { } } } + if (!Objects.equals(canonicalize(instance.pluginDir), canonicalize(pluginDir))) { + throw new IllegalStateException(String.format( + "TrinoBootstrap already initialized with plugin dir '%s'; cannot reuse it for a " + + "different plugin dir '%s'. All trino-connector catalogs in one FE must share a " + + "single plugin dir.", instance.pluginDir, pluginDir)); + } return instance; } + /** + * Best-effort canonicalization so two spellings of the same physical directory (trailing slash, + * relative vs absolute, symlink) are treated as equal and do not trip the mismatch guard above. + * Falls back to the raw string if the path cannot be resolved. + */ + private static String canonicalize(String dir) { + if (dir == null) { + return null; + } + try { + return new File(dir).getCanonicalPath(); + } catch (IOException e) { + return dir; + } + } + /** * Returns the already-initialized singleton. Callers that run after a catalog has been * created (e.g. scan planning) use this instead of re-resolving the plugin directory. From 367f6d589b5e1b0c2b4cc8b7c7b8c18a20e08300 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:04:33 +0800 Subject: [PATCH 305/334] [doc](catalog) #65185 reverify: land trino cluster designs (L3/L4/L5/L6), mark done Design docs for the trino low-risk cluster (transaction release, plugin-dir fail-loud, listTableNames de-dup, guard-field safe publication), each with its adversarial review folded in. Mark L3/L4/L5/L6 DONE in the tracking table and repoint HANDOFF next-step to the kerberos cluster (L7/L8). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 18 ++++- plan-doc/tasks/designs/FIX-L3-design.md | 94 ++++++++++++++++++++++ plan-doc/tasks/designs/FIX-L4-design.md | 60 ++++++++++++++ plan-doc/tasks/designs/FIX-L5-design.md | 30 +++++++ plan-doc/tasks/designs/FIX-L6-design.md | 46 +++++++++++ 6 files changed, 245 insertions(+), 5 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L3-design.md create mode 100644 plan-doc/tasks/designs/FIX-L4-design.md create mode 100644 plan-doc/tasks/designs/FIX-L5-design.md create mode 100644 plan-doc/tasks/designs/FIX-L6-design.md diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md index a7cb8e058c4488..273ce8d00df449 100644 --- a/plan-doc/HANDOFF-65185-reverify-fixes.md +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -36,7 +36,7 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 | 批次 1 | M5/M7/M6/M4/M2 中危（连接器局部：iceberg×3 + mc + hive） | ✅ 全 DONE + 最终复核 CLEAN | | 批次 2 | M3→M1 中危(fe-core 通用节点) | ✅ 全 DONE | | **批次 3** | **L1(import 门禁) + M8(发布工具/文档)** | **✅ L1 DONE;M8 ⏸ 用户跳过(07-12)** | -| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | ⬜ **← 回来从这里续(用户指向)** | +| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino 子群 L3/L4/L5/L6 ✅;**← 续 kerberos L7/L8** | | 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | | 设计债 | D-系列 | ⏸ 择机 / 随 P8 | diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 116efc6511ebcc..f8b1fc92bcf10e 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -179,10 +179,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L1** DONE `88aa55b831b`(设计 `designs/FIX-L1-design.md`;设计红队 `wf_643c11b4-3fe` 3 lens 全 SOUND_WITH_CHANGES)。补 3 洞(static / +6 包 / test 源)**+ 红队发现的第 4 洞**:4 条白名单 `grep -v` 按**整行**匹配(`.`≡`/`)→连接器命名空间文件(608 个,全根在 `org.apache.doris.connector.**`)里的违规 import 被**按路径抑制**,门禁对其结构性失明。修法=候选 grep 加宽(static+test glob+6 包)+**白名单锚定到 import 目标**(`:import ...` 非整行)+ fqn sed 剥 `static`(修 static-vendored 误报,红队证 E3 属正确性非装饰)。新增自测 `tools/check-connector-imports.test.sh`(8 违规/2 vendored skip/3 allow;GREEN 于新、RED 于旧,真树 exit 0)。保留 `is_vendored()`(HiveVersionUtil FP)。**观察**:grep-denylist 固有盲区(新增内部包漏登记/FQN-无-import 内联/间距)登记设计债,根治须 allowlist/ArchUnit(Trino 先例)。 - [ ] **L2** ⏸ 翻闸 hive 丢 SQL 缓存资格 · `CacheAnalyzer.java:308/316/319`(instanceof HiveScanNode)+ `BindRelation.java:887`(instanceof HMSExternalTable)。**需决策**:加 `ConnectorCapability` 让 `PluginDrivenScanNode`/`PluginDrivenExternalTable` 被识别并恢复缓存 **vs** 登记为 fail-safe 验收偏差(`enable_hive_sql_cache` 默认关)。 -- [ ] **L3** trino 事务从不 commit/close · `TrinoConnectorDorisMetadata.java`(6 处)+ `TrinoScanPlanProvider.java:112`(scan 站有意保持):元数据 6 站 try/finally `commit(txn)`(read-only 廉价)。scan 站不动。 -- [ ] **L4** trino plugin.dir 首胜单例 · `TrinoBootstrap.java:136,316`:单例已存在且 pluginDir 不同时 fail-loud 抛(而非静默用旧);或删 per-catalog 分支只认全局 config。 -- [ ] **L5** trino listTableNames 丢去重 · `TrinoConnectorDorisMetadata.java:98`:收集加 `.distinct()`/LinkedHashSet 保序;不复原 prefix 过滤(冗余)。 -- [ ] **L6** trino guard 字段发布顺序 NPE · `TrinoDorisConnector.java:176`:`doInitialize` 把 guard 字段 `trinoConnector` 赋值移到**最后**(volatile happens-before);或原子发布不可变 holder。 +- [x] **L3** DONE `4cd63c6911a`(设计 `FIX-L3-design.md`;对抗复审 `agent a182a049f` SOUND_WITH_CHANGES）· trino 6 个 FE-only 元数据站 try/finally 经 `releaseQuietly` 释放事务(commit + swallow-log 防 mask);scan 站不动(txnHandle 序列化发 BE 须保持打开)。UT 受 `io.trino.Session` 构造墙阻,登记 e2e live-gated。 +- [x] **L4** DONE `e27602d4ab6`(设计 `FIX-L4-design.md`;并发对抗复审 `agent a28dc47095` SOUND）· `TrinoBootstrap` 存 pluginDir,`getInstance` 不同 dir fail-loud 抛(canonicalize best-effort 兜同物理目录异拼写)。UT 受重构造墙阻,登记。 +- [x] **L5** DONE `be96adf76ba`(设计 `FIX-L5-design.md`）· `listTableNames` 加 `.distinct()` 复原 legacy LinkedHashSet 保序去重;不复原冗余 prefix 过滤。 +- [x] **L6** DONE `18048f7f217`(设计 `FIX-L6-design.md`;并发对抗复审同上 SOUND）· `doInitialize` guard 字段 `trinoConnector` 移到最后赋值(安全发布,闭合半初始化瞬时 NPE 窗口);全字段已 volatile,重排即足。 - [ ] **L7** kerberos UGI.setConfiguration 无 guard · `HadoopKerberosAuthenticator.java:53-59,83,115`:port master first-writer-wins(仅首次 setConfiguration,进程锁序列化,已匹配则 skip+WARN);refresh 分支不重跑 initializeAuthConfig。注:metastore 路 parity,仅 fe-filesystem HDFS 数据路是真变。 - [ ] **L8** kerberos doAs 吞 interrupt · `HadoopAuthenticator.java:34`:`throw new IOException(e)` 前加 `Thread.currentThread().interrupt();`。一行。 - [ ] **L9** MC 谓词下推全有全无 · `MaxComputePredicateConverter.java:87-97,117-123`:`convertFilter` 若根是 `ConnectorAnd` 则逐 top-level conjunct 独立 try/catch(丢+log 失败)再 AND 幸存者;**不**对 OR/嵌套 AND 逐子容错。 @@ -269,3 +269,13 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **L1** DONE `88aa55b831b`(code+test) — import 门禁补 3 洞 + **红队发现的第 4 洞**(白名单按整行匹配→连接器命名空间文件违规 import 被路径抑制,门禁对 608 个连接器实现文件结构性失明)。修法=候选 grep 加宽(static/test/6 包)+ 白名单锚定 import 目标 + fqn 剥 static;新增 `tools/check-connector-imports.test.sh`(RED 于旧/GREEN 于新/真树 exit 0)。设计红队 `wf_643c11b4-3fe` 3 lens 全 SOUND_WITH_CHANGES,发现全部逐条复现并折入。**非 live**(纯工具,无 e2e 欠账)。 - **M8** ⏸ **用户 2026-07-12 明确要求跳过**(转做 L 系列)。侦察结论已留档(build.sh 已部署插件,缺口在升级流程+文档,含一个「仅文档 vs 文档+可选防御码」的用户决策)。**不 silently drop**——保留在表中待办,回来做时先中文讲清再拍板。 - **下一步(用户指向 L 系列)**:直接可做的低危连接器 L 条=`L3–L9`(trino/kerberos/mc)、`L11/L13/L14`(paimon)、`L15–L19`(paimon/iceberg 杂项)。⏸ 决策类 `L2/L10/L12/L20` 须先中文讲清背景+选项问用户再动(memory `ask-user-explain-in-chinese-first`)。 + +--- + +**⭐ 批次 4·trino 子群(L3/L4/L5/L6)全 DONE**（各独立 code commit + 设计文档；trino 局部,不碰 fe-core,import 门净）： +- **L3** `4cd63c6911a` — 6 元数据站释放事务(releaseQuietly,masking-safe);scan 站不动。对抗复审 `a182a049f` SOUND_WITH_CHANGES(折入 masking-safe 守卫)。 +- **L5** `be96adf76ba` — listTableNames `.distinct()` 复原去重。 +- **L6** `18048f7f217` — guard 字段最后发布,闭合并发瞬时 NPE。并发对抗复审 `a28dc47095` SOUND。 +- **L4** `e27602d4ab6` — plugin.dir 不同 fail-loud(canonicalize 兜异拼写)。同上复审 SOUND(折入 canonicalize minor)。 +- **共性**:trino UT 受 `io.trino.Session`/重构造墙阻,均 build-compile + 对抗复审 + e2e live-gated 兜底(非静默;显式登记 UT-wall)。 +- **下一步 = kerberos 子群 L7/L8,再 mc L9**。 diff --git a/plan-doc/tasks/designs/FIX-L3-design.md b/plan-doc/tasks/designs/FIX-L3-design.md new file mode 100644 index 00000000000000..8abd67fccc4104 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L3-design.md @@ -0,0 +1,94 @@ +# FIX-L3 — trino 元数据方法开事务却从不 commit/close + +> 来源：`plan-doc/reviews/catalog-spi-review-65185-reverify-2026-07-11.md` §1 表 L3（原 P2-1）。 +> 严重度：🟡 低（连接器局部资源泄漏；与 master parity,翻闸后量级放大）。范围：`fe-connector-trino` 一文件。 +> HEAD 复核基线：`88aa55b831b`。 + +## Problem + +`TrinoConnectorDorisMetadata` 的 **6 个** FE 侧元数据方法各自 +`trinoConnector.beginTransaction(READ_UNCOMMITTED, true, true)` 开一个 Trino 事务、用它取 `ConnectorMetadata`、 +映射结果为 Doris 类型后**直接 return，从不 commit/rollback**： +- `listDatabaseNames`:86、`listTableNames`:102、`getTableHandle`:127、`getTableSchema`:165、 + `applyFilter`:239、`applyProjection`:300。 + +每次规划都会走 listTableNames/getTableHandle/getTableSchema/applyFilter/applyProjection → Trino connector 的 +transaction manager 里累积未释放的事务句柄（部分连接器 per-txn 持资源/状态）→ 长期内存增长 / 资源泄漏。 + +## Root Cause + +FE 侧把 Trino connector 当「开事务→取 metadata→用完丢」，缺 try/finally 释放。Trino 引擎正常路径由 +`TransactionManager` 负责 commit/rollback;这里直接驱动 connector,释放责任落在调用方,却漏了。 + +## Design（6 站就地 try/finally；scan 站不动） + +**只修 6 个 FE-only 元数据站**。每站把「beginTransaction 之后到所有 return」包进 `try`,`finally` 里经小 helper 释放事务: +```java +ConnectorTransactionHandle txn = trinoConnector.beginTransaction(IsolationLevel.READ_UNCOMMITTED, true, true); +try { + ConnectorMetadata metadata = trinoConnector.getMetadata(connSession, txn); + ... 原方法体(含各 early-return)不变 ... + return result; +} finally { + releaseQuietly(txn); +} +``` +小 helper（集中「masking-safe 释放」,避免 6 站各 4 行 try/catch 重复）: +```java +private void releaseQuietly(io.trino.spi.connector.ConnectorTransactionHandle txn) { + try { + trinoConnector.commit(txn); + } catch (RuntimeException e) { + LOG.warn("Failed to release Trino metadata transaction", e); // 不 mask body 异常(Rule 12) + } +} +``` + +**决策要点(已核 + 对抗复审 `agent a182a049f` SOUND_WITH_CHANGES 折入)**： +- **commit + swallow-log(masking-safe)**：事务 **read-only READ_UNCOMMITTED**,无写入 → commit≡rollback,只为「从 txn + manager 释放」。取 `commit`(对齐 reverify + 成功语义)。**复审命中(minor)**：「read-only commit 不会抛」是**未证的连接器相关假设** + (某连接器 commit 可能关 JDBC 连接/metastore client 而抛),裸 finally-commit 会 mask body 的真异常(Rule 12 fail-loud)。 + → 释放包在 `releaseQuietly` 的 `try/catch(RuntimeException)+LOG.warn`,既释放又不 mask。Trino 引擎本身是 rollback-on-error/ + commit-on-success;严格镜像须每个 return 前插 commit(方法多 early-return→太侵入),`releaseQuietly` 是保留单-finally 简洁的最小安全形。 +- **就地 try/finally 而非抽 lambda helper**：镜像**同文件 scan 站已有的 `try{...}finally{metadata.cleanupQuery(...)}` + per-site 惯用法**(Rule 11 conform)+ 保持 surgical(Rule 3),不重构 6 个方法为 lambda。 +- **条件开事务保持**：`applyFilter`(`tupleDomain.isAll()` early-return 在 beginTransaction 前)、 + `applyProjection`(`projections.isEmpty()`/`trinoProjections.isEmpty()` 在前)——`try` 从 **beginTransaction 之后**起, + 不改「满足前置才开事务」的现状,不新增无谓事务。 +- **commit 后句柄仍可用(安全性核实)**：6 方法返回的 `TrinoTableHandle`/列句柄是 Trino **不可变 opaque 值对象**, + scan 时 `TrinoScanPlanProvider.planScan` 会**另开自己的事务**(:111)复用它们——今日事务虽泄漏但 return 后即无用, + 且句柄跨事务复用本就成立 → **commit 元数据事务不影响任何后续使用**。已核。 + +**scan 站不动**（`TrinoScanPlanProvider.planScan:111`）：它 `beginQuery` 后把 **txnHandle 序列化成 JSON 发给 BE** +（`.transactionHandle(txnHandleJson)`,:223/:252,BE JNI scanner 用它读数据）→ 事务**必须保持打开**,故只 `finally +{ metadata.cleanupQuery }` 不 commit。此为设计,L3 不碰。 + +## Risk Analysis + +- **破坏 scan / 句柄**：无（见上「commit 后句柄仍可用」+「scan 站不动」核实）。 +- **异常 mask**：read-only commit 不抛 → finally-commit 不 mask（见上）。 +- **行为变更**：仅新增事务释放,不改任何返回值/映射逻辑。字节级对返回结果不变。 +- **铁律**：连接器局部,不碰 fe-core,不解析属性,无 source-name 分支。import 门禁不受影响（不新增 fe-core import）。 + +## Implementation Plan + +改 `fe-connector-trino/.../TrinoConnectorDorisMetadata.java` 6 站,各加 try/finally。不动 scan provider。 + +## Test Plan + +### Unit Tests — ⚠ 受阻,登记 + +驱动这 6 个方法需构造 `io.trino.Session`(连接器构造入参,方法内 `trinoSession.toConnectorSession(...)` 必用)。 +`io.trino.Session` 是重量级具体类(大 builder,无既有测试夹具;本模块现有 4 个 UT 均不驱动 `TrinoConnectorDorisMetadata`, +正因此墙)。为一个低危 parity 泄漏 fabricate 全套 Trino SPI(`Connector`+`ConnectorMetadata`+`Session`+`CatalogHandle`) +夹具**不成比例**。→ **本条不加行为 UT**,以 build-compile(验 `commit(txn)` 签名 + 无 checkstyle)+ 对抗复审 + e2e 兜底。 +**不静默**：显式登记 UT-wall（Rule 12）。 + +### E2E — live-gated（trino 真集群） + +trino-connector 目录跑一批查询(listTables / desc / 带谓词 filter / 投影),事务不再累积;功能结果与修前一致。 +需真 trino 插件 + 后端,live-gated(memory `hms-iceberg-delegation-needs-e2e`)。 + +## 备注 + +L4/L5/L6 亦在本连接器,随后各自独立 commit（L5 顺带在 `listTableNames` 加 `.distinct()`,与本条同方法但分开提交）。 diff --git a/plan-doc/tasks/designs/FIX-L4-design.md b/plan-doc/tasks/designs/FIX-L4-design.md new file mode 100644 index 00000000000000..ae1da33415ee20 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L4-design.md @@ -0,0 +1,60 @@ +# FIX-L4 — trino plugin.dir 首胜单例静默用旧 → fail-loud + +> 来源：reverify §1 表 L4（原 P2-2）。🟡 低（多目录不同 plugin.dir 时静默用错插件）。范围：`TrinoBootstrap` 单例。 +> HEAD 复核基线：`4cd63c6911a`。 + +## Problem / Root Cause + +`TrinoBootstrap` 是**进程级单例**（:99 `instance`，一次加载所有插件）。`getInstance(pluginDir)`（:136-145）是 +**first-wins**：第一个建的 trino 目录用它的 `pluginDir` 初始化单例；之后目录若传**不同** `plugin.dir`，被**静默忽略** +（返回旧 instance,用第一个 dir 的插件）→ 后建目录可能找不到自己的 connector factory,或悄悄用错插件集。 + +## Design（存 pluginDir + 不匹配 fail-loud） + +- `TrinoBootstrap` 加 `private final String pluginDir;`,构造函数存下。 +- `getInstance(pluginDir)`：保持 first-wins 初始化不变;返回前若 `!Objects.equals(instance.pluginDir, pluginDir)` + → 抛 `IllegalStateException`（响亮报「已用 dir A 初始化,不能再用 dir B」+ 提示同 FE 内 trino 目录须共享单一 plugin dir）。 +```java +public static TrinoBootstrap getInstance(String pluginDir) { + if (instance == null) { + synchronized (INIT_LOCK) { + if (instance == null) { + instance = new TrinoBootstrap(pluginDir); + } + } + } + if (!java.util.Objects.equals(instance.pluginDir, pluginDir)) { + throw new IllegalStateException(String.format( + "TrinoBootstrap already initialized with plugin dir '%s'; cannot reuse for a different " + + "plugin dir '%s'. All trino-connector catalogs in one FE must share one plugin dir.", + instance.pluginDir, pluginDir)); + } + return instance; +} +``` + +**选 fail-loud（task-list 选项 A）而非删 per-catalog 分支（选项 B）**：A 最小、保留现「first-wins」语义、只把静默变响亮; +B 改功能面（移除 per-catalog `trino.plugin.dir`）属更大设计改动,本条不做。`Objects.equals` 兜 null（虽构造用 `new +File(pluginDir)` 已隐含非空,首次前的比较仍走 null-safe）。 + +**并发对抗复审 `agent a28dc47095` = SOUND，折入其 1 minor**：裸 `String.equals` 会把**同一物理目录的不同拼写** +（尾斜杠 / 相对 vs 绝对 / symlink）误判为不同 → 良性同目录配置被误抛（false positive）。→ 比较前经 `canonicalize(dir)` +（`new File(dir).getCanonicalPath()`，best-effort，IOException 回退原串,null 直通）。仍只对**真不同**目录抛。另修 nit: +`Objects` 已 import → 去掉全限定。 + +## Risk + +- 单目录 / 多目录同 dir：`equals` 恒 true → 行为完全不变。 +- 只有「多目录不同 plugin.dir」这一**本就是误配**的场景从「静默用错」变「启动/建目录即响亮抛」——Rule 12 fail-loud,正向。 +- 不碰 fe-core。 + +## Test Plan + +- build-compile。 +- 行为 UT 受阻：`getInstance` 走真 `new TrinoBootstrap(pluginDir)`（`pluginManager.loadPlugins()` + attach-self,重副作用,无既有夹具; + 现 `TrinoBootstrapTest` 只测纯 `resolvePluginDir`）→ 不加 getInstance 行为 UT,登记 UT-wall（同 FIX-L3 模式）。 +- e2e：两个 trino 目录配不同 `trino.plugin.dir` → 第二个建目录时响亮抛（live）。 + +## 备注 + +与 L6 合并一次并发/单例对抗复审。 diff --git a/plan-doc/tasks/designs/FIX-L5-design.md b/plan-doc/tasks/designs/FIX-L5-design.md new file mode 100644 index 00000000000000..9f175cd1ec3ff5 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L5-design.md @@ -0,0 +1,30 @@ +# FIX-L5 — trino listTableNames 丢 LinkedHashSet 去重 + +> 来源：reverify §1 表 L5（原 P2-4）。🟡 低（连接器局部，防御性 parity）。范围：`TrinoConnectorDorisMetadata.listTableNames` 一行。 +> HEAD 复核基线：`4cd63c6911a`（L3 之后；同文件）。 + +## Problem / Root Cause + +`listTableNames` 把 `metadata.listTables(connSession, Optional.of(dbName))` 的结果 `.map(getTableName)` 后 +直接 `.collect(toList())`，**丢了 legacy 的 LinkedHashSet 去重**。个别 Trino 连接器 `listTables` 会对同一 schema 返回 +重复 `SchemaTableName`（表+视图双列、多源合并等），legacy Doris trino 用 `LinkedHashSet` 去重且保序；SPI 版漏了。 + +## Design + +`.map(SchemaTableName::getTableName)` 之后加 `.distinct()`（保序去重，等价 LinkedHashSet）再 `.collect(toList())`。 +**不复原 legacy 的 prefix 过滤**（`listTables` 已按 schema 过滤，prefix 冗余；task-list 明确不复原）。 + +## Risk + +单 schema 内表名本唯一 → `.distinct()` 只折叠意外重复，不会误并两个真不同的表。保序不变。零其它行为变更。连接器局部， +不碰 fe-core，不新增 import。 + +## Test Plan + +- build-compile（`.distinct()` 编译 + 0 checkstyle）。 +- 行为 UT 受同一 `io.trino.Session` 构造墙阻（见 FIX-L3-design）——不加，登记。 +- e2e live-gated：trino 目录 `SHOW TABLES` 无重复项（需真 trino 连接器返回重复的场景，live）。 + +## 备注 + +无 red-team（一行防御性 parity，非「大改动」；memory `clean-room-adversarial-review-pref`）。 diff --git a/plan-doc/tasks/designs/FIX-L6-design.md b/plan-doc/tasks/designs/FIX-L6-design.md new file mode 100644 index 00000000000000..2ca9d9f09805df --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L6-design.md @@ -0,0 +1,46 @@ +# FIX-L6 — trino guard 字段先于依赖字段发布 → 并发首访瞬时 NPE + +> 来源：reverify §1 表 L6（原 P2-5）。🟡 低（并发时序，volatile 自愈的瞬时窗口）。范围：`TrinoDorisConnector.doInitialize` 字段赋值顺序。 +> HEAD 复核基线：`4cd63c6911a`。 + +## Problem / Root Cause + +`ensureInitialized`（:133-141）用 **`trinoConnector` 作 guard** 做 double-checked locking；`getMetadata`（:67-68）等 +在 guard 通过后立即读 `trinoSession`/`trinoCatalogHandle`。但 `doInitialize`（:176-179）先赋 **guard `trinoConnector`**（:176） +再赋 `trinoSession`/`trinoCatalogHandle`/`trinoConnectorName`（:177-179）。 + +字段虽全 `volatile`，问题是**发布顺序**：线程 B 见 `trinoConnector != null`（A 在 :176 写的）→ 跳过 synchronized → +`new TrinoConnectorDorisMetadata(trinoConnector, trinoSession, trinoCatalogHandle)`，而 A 可能尚未执行 :177/:178 +→ B 读到 `trinoSession==null` → NPE。volatile happens-before 只保证「见 guard 写 ⟹ 见 guard 写**之前**的所有写」; +`trinoSession` 写在 guard **之后**,故不被覆盖。 + +## Design（4 行重排：guard 字段最后赋值） + +把 `this.trinoConnector = result.getConnector();` 从 :176 **移到 4 个赋值的最后**： +```java +this.trinoProperties // 已在 :145 更早赋(guard 前),getTrinoProperties 安全,不动 +... +this.trinoSession = result.getSession(); +this.trinoCatalogHandle = result.getCatalogHandle(); +this.trinoConnectorName = result.getConnectorName(); +this.trinoConnector = result.getConnector(); // guard 最后发布 → 见它非空即见其前所有依赖写 +``` +则 B 见 `trinoConnector != null` ⟹（volatile release/acquire）见 `trinoSession`/`trinoCatalogHandle`/ +`trinoConnectorName` 的写 → 无半初始化。经典「guard 字段最后发布」安全发布法;全字段已 volatile,重排即足,无需引 holder。 + +## Risk + +- 单线程语义完全不变（4 个赋值无相互依赖,顺序对结果无影响,仅影响并发可见性）。 +- 所有 guard 检查点（:88 testConnection、:96 close、:134/:136 ensureInitialized）**统一用 `trinoConnector`** → 移它最后即全覆盖。已核。 +- 连接器局部,不碰 fe-core。 + +## Test Plan + +- build-compile。 +- 并发竞态的行为 UT 不切实际（需精确时序触发瞬时窗口）+ 同 `io.trino.Session`/重构造墙（见 FIX-L3）→ 不加,登记。 + 修法是教科书安全发布惯用法,以设计推理 + 并发对抗复审兜底。 +- e2e：并发首访不再偶发 NPE（难稳定复现,live/压测观察）。 + +## 备注 + +与 L4 同属 trino 并发/单例条,合并一次并发对抗复审。 From 59697ce3fc7b1e2344a2722a2fb6e6c1492177ee Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:19:04 +0800 Subject: [PATCH 306/334] [fix](catalog) kerberos: restore interrupt flag in doAs HadoopAuthenticator.doAs caught InterruptedException and rethrew it as a checked IOException without restoring the thread's interrupt status, so callers up the stack could no longer observe that the thread was interrupted (breaks cooperative cancellation). Call Thread.currentThread().interrupt() before wrapping, per standard practice for swallowing InterruptedException. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../java/org/apache/doris/kerberos/HadoopAuthenticator.java | 3 +++ 1 file changed, 3 insertions(+) diff --git a/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopAuthenticator.java b/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopAuthenticator.java index 2599feeaa60eb3..615e7ae88976f9 100644 --- a/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopAuthenticator.java +++ b/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopAuthenticator.java @@ -32,6 +32,9 @@ default T doAs(PrivilegedExceptionAction action) throws IOException { try { return getUGI().doAs(action); } catch (InterruptedException e) { + // Restore the interrupt flag: we are swallowing InterruptedException by rethrowing it as a + // checked IOException, so callers up the stack can still observe that the thread was interrupted. + Thread.currentThread().interrupt(); throw new IOException(e); } catch (UndeclaredThrowableException e) { if (e.getCause() instanceof RuntimeException) { From 9e4f29923824b483b6c65a415599ca2ea4b85adc Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:19:04 +0800 Subject: [PATCH 307/334] [fix](catalog) kerberos: first-writer-wins guard on UGI.setConfiguration initializeAuthConfig() unconditionally called UserGroupInformation .setConfiguration(hadoopConf), which mutates a single process-wide global. Because it runs from login() on both initial login AND every ticket refresh, every kerberos catalog's login/refresh overwrote the global auth config (last-writer-wins), so HDFS catalogs with differing hadoop.security.authentication could not coexist and refreshes churned the global. The consolidation in #64655 (f7992b0a07e) dropped the first-writer-wins guard master had. Restore first-writer-wins: only the first caller publishes the global config; a static field remembers the first auth method; later callers (and every refresh, which re-enters via login()) skip, and a WARN fires only on a genuine auth-method mismatch. Compare against the remembered method rather than master's UserGroupInformation.getLoginUser().getAuthenticationMethod() on purpose: getLoginUser() would establish a process-wide login user, which Doris deliberately avoids (per-instance getUGIFromSubject only). This class is the only non-test setConfiguration caller, so the remembered method equals the true global auth method for every Doris path. The per-conf HADOOP_SECURITY_AUTHORIZATION flag stays outside the guard (each catalog needs it on its own conf). Verified by compile (SecurityUtil.getAuthenticationMethod / AuthenticationMethod, 0 checkstyle), the fe-kerberos UTs, and an adversarial review that checked the guard, thread-safety, and the getLoginUser deviation against the hadoop-common bytecode. Behavioral validation is e2e live-gated (two kerberos HDFS catalogs). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../kerberos/HadoopKerberosAuthenticator.java | 25 +++++++++++++++++-- 1 file changed, 23 insertions(+), 2 deletions(-) diff --git a/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java b/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java index 3b0672ca310d50..599f3dbb995c44 100644 --- a/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java +++ b/fe/fe-kerberos/src/main/java/org/apache/doris/kerberos/HadoopKerberosAuthenticator.java @@ -22,6 +22,7 @@ import com.google.common.collect.ImmutableSet; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.CommonConfigurationKeysPublic; +import org.apache.hadoop.security.SecurityUtil; import org.apache.hadoop.security.UserGroupInformation; import org.apache.logging.log4j.LogManager; import org.apache.logging.log4j.Logger; @@ -46,15 +47,35 @@ public class HadoopKerberosAuthenticator implements HadoopAuthenticator { private long nextRefreshTime; private UserGroupInformation ugi; + // The authentication method the first caller published to the process-wide UGI configuration. + // Guarded by HadoopKerberosAuthenticator.class; null until the first setConfiguration. + private static UserGroupInformation.AuthenticationMethod configuredAuthMethod = null; + public HadoopKerberosAuthenticator(KerberosAuthenticationConfig config) { this.config = config; } public static void initializeAuthConfig(Configuration hadoopConf) { hadoopConf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, "true"); + // UserGroupInformation.setConfiguration mutates a single process-wide global, so multiple HDFS + // catalogs with differing hadoop.security.authentication cannot truly coexist. First-writer-wins: + // only the first caller publishes the global config; later catalogs (and every ticket refresh, + // which re-enters this method via login()) skip it so they don't stomp on each other. A WARN + // fires only on a genuine auth-method mismatch. We compare against a remembered method rather than + // UserGroupInformation.getLoginUser() on purpose: Doris never establishes a process-wide login user + // (per-instance getUGIFromSubject only), and getLoginUser() would trigger exactly that. + UserGroupInformation.AuthenticationMethod desired = SecurityUtil.getAuthenticationMethod(hadoopConf); synchronized (HadoopKerberosAuthenticator.class) { - // avoid other catalog set conf at the same time - UserGroupInformation.setConfiguration(hadoopConf); + if (configuredAuthMethod == null) { + UserGroupInformation.setConfiguration(hadoopConf); + configuredAuthMethod = desired; + return; + } + if (configuredAuthMethod != desired) { + LOG.warn("UGI already configured with authentication={} but this catalog requests {}; " + + "keeping existing JVM-global setting (first-writer-wins).", + configuredAuthMethod, desired); + } } } From f62ccf25fe919de0ede8ea540d79b228d53e4653 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:19:47 +0800 Subject: [PATCH 308/334] [doc](catalog) #65185 reverify: land kerberos cluster designs (L7/L8), mark done Design docs for the kerberos low-risk cluster (first-writer-wins guard on UGI.setConfiguration with the getLoginUser deviation rationale; doAs interrupt restore). L7's adversarial review verified soundness against the hadoop-common bytecode. Mark L7/L8 DONE and repoint HANDOFF next-step to maxcompute L9. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 11 +++- plan-doc/tasks/designs/FIX-L7-design.md | 66 ++++++++++++++++++++++ plan-doc/tasks/designs/FIX-L8-design.md | 27 +++++++++ 4 files changed, 102 insertions(+), 4 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L7-design.md create mode 100644 plan-doc/tasks/designs/FIX-L8-design.md diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md index 273ce8d00df449..0366b7284db5f1 100644 --- a/plan-doc/HANDOFF-65185-reverify-fixes.md +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -36,7 +36,7 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 | 批次 1 | M5/M7/M6/M4/M2 中危（连接器局部：iceberg×3 + mc + hive） | ✅ 全 DONE + 最终复核 CLEAN | | 批次 2 | M3→M1 中危(fe-core 通用节点) | ✅ 全 DONE | | **批次 3** | **L1(import 门禁) + M8(发布工具/文档)** | **✅ L1 DONE;M8 ⏸ 用户跳过(07-12)** | -| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino 子群 L3/L4/L5/L6 ✅;**← 续 kerberos L7/L8** | +| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino L3-L6 ✅ · kerberos L7/L8 ✅;**← 续 mc L9** | | 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | | 设计债 | D-系列 | ⏸ 择机 / 随 P8 | diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index f8b1fc92bcf10e..887874b5441829 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -183,8 +183,8 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L4** DONE `e27602d4ab6`(设计 `FIX-L4-design.md`;并发对抗复审 `agent a28dc47095` SOUND）· `TrinoBootstrap` 存 pluginDir,`getInstance` 不同 dir fail-loud 抛(canonicalize best-effort 兜同物理目录异拼写)。UT 受重构造墙阻,登记。 - [x] **L5** DONE `be96adf76ba`(设计 `FIX-L5-design.md`）· `listTableNames` 加 `.distinct()` 复原 legacy LinkedHashSet 保序去重;不复原冗余 prefix 过滤。 - [x] **L6** DONE `18048f7f217`(设计 `FIX-L6-design.md`;并发对抗复审同上 SOUND）· `doInitialize` guard 字段 `trinoConnector` 移到最后赋值(安全发布,闭合半初始化瞬时 NPE 窗口);全字段已 volatile,重排即足。 -- [ ] **L7** kerberos UGI.setConfiguration 无 guard · `HadoopKerberosAuthenticator.java:53-59,83,115`:port master first-writer-wins(仅首次 setConfiguration,进程锁序列化,已匹配则 skip+WARN);refresh 分支不重跑 initializeAuthConfig。注:metastore 路 parity,仅 fe-filesystem HDFS 数据路是真变。 -- [ ] **L8** kerberos doAs 吞 interrupt · `HadoopAuthenticator.java:34`:`throw new IOException(e)` 前加 `Thread.currentThread().interrupt();`。一行。 +- [x] **L7** DONE `9e4f2992382`(设计 `FIX-L7-design.md`;对抗复审 `agent a3e2a8a6` SOUND，对照 hadoop-common 3.4.2 字节码验证）· `initializeAuthConfig` 恢复 first-writer-wins(静态字段记首个 auth 方式,首写者设全局、后续+刷票跳过、真不匹配才 WARN)。**刻意不用 master 的 `getLoginUser()`**(会建进程级 login user,Doris 有意规避;本类是唯一非测 setConfiguration 调用者故静态字段=真全局)。刷票跳过=恢复 master parity。 +- [x] **L8** DONE `59697ce3fc7`(设计 `FIX-L8-design.md`）· `doAs` catch(InterruptedException) 内 `throw` 前加 `Thread.currentThread().interrupt();`(恢复中断标志)。 - [ ] **L9** MC 谓词下推全有全无 · `MaxComputePredicateConverter.java:87-97,117-123`:`convertFilter` 若根是 `ConnectorAnd` 则逐 top-level conjunct 独立 try/catch(丢+log 失败)再 AND 幸存者;**不**对 OR/嵌套 AND 逐子容错。 - [ ] **L10** ⏸ EXPLAIN 节点名 · `PluginDrivenScanNode.java:170,320`。**需决策**:登记 display-only 验收偏差(cheapest,`CONNECTOR:` 行已披露)**vs** 加连接器声明的 legacy `*_SCAN_NODE` 名(连 `Connector.getLegacyEngineName` 一起,见 D-ENGINE)。注:报告的 `connectorType.toUpperCase()+"_SCAN_NODE"` 一行修法**不够**(hms 会出 `VHMS_SCAN_NODE` 非 `VHIVE_/VICEBERG_`)。 - [ ] **L11** paimon file_format · `PaimonScanPlanProvider.java:812-813,848-849`:JNI DataSplit + COUNT 路按首数据文件后缀 `getFileFormatBySuffix("/"+dataFiles().get(0).fileName()).orElse(defaultFileFormat)`(仿 native 臂 `:540`/legacy)。注:默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。 @@ -278,4 +278,9 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **L6** `18048f7f217` — guard 字段最后发布,闭合并发瞬时 NPE。并发对抗复审 `a28dc47095` SOUND。 - **L4** `e27602d4ab6` — plugin.dir 不同 fail-loud(canonicalize 兜异拼写)。同上复审 SOUND(折入 canonicalize minor)。 - **共性**:trino UT 受 `io.trino.Session`/重构造墙阻,均 build-compile + 对抗复审 + e2e live-gated 兜底(非静默;显式登记 UT-wall)。 -- **下一步 = kerberos 子群 L7/L8,再 mc L9**。 + +**⭐ 批次 4·kerberos 子群(L7/L8)全 DONE**（`fe-kerberos` 独立模块,非连接器,import 门禁不适用;各独立 commit）： +- **L7** `9e4f2992382` — `UGI.setConfiguration` first-writer-wins guard(恢复 #64655 删掉的 master guard 意图,静态字段跟踪首个 auth 方式,避 `getLoginUser` 进程级登录副作用)。对抗复审 `a3e2a8a6` SOUND(对照 hadoop 字节码)。 +- **L8** `59697ce3fc7` — `doAs` 恢复中断标志(一行)。 +- fe-kerberos UT 11/11 绿(含 AuthenticationTest);L7 e2e live-gated(双 kerberos HDFS 目录)。 +- **下一步 = mc L9(谓词下推全有全无),再 paimon L11/L13/L14,iceberg/杂项 L15–L19**。 diff --git a/plan-doc/tasks/designs/FIX-L7-design.md b/plan-doc/tasks/designs/FIX-L7-design.md new file mode 100644 index 00000000000000..efae55fa261ffa --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L7-design.md @@ -0,0 +1,66 @@ +# FIX-L7 — kerberos 每次 login/refresh 无条件 UGI.setConfiguration（丢 first-writer guard） + +> 来源：reverify §1 表 L7（原 P3b-2）。🟡 低（metastore 路 parity;仅 fe-filesystem HDFS 数据路真变）。范围：`HadoopKerberosAuthenticator.initializeAuthConfig`。 +> HEAD 复核基线：`e27602d4ab6`。 + +## Problem / Root Cause + +`initializeAuthConfig`（:53-59）无条件 `UserGroupInformation.setConfiguration(hadoopConf)`。它由 `login()`（:115）调， +而 `login` 在**初次登录**（:65）和**每次刷票**（:83）都跑 → 每个 kerberos 目录的每次 login/refresh 都**覆写进程级全局 +UGI 配置**（last-writer-wins）。多个 auth 方式不同的 HDFS 目录不能共存,且刷票期无谓 churn。 + +**consolidation commit `f7992b0a07e`(#64655) 删掉了 master 原有的 first-writer-wins guard** +（`shouldSkipSetConfiguration` + `UGI_INIT_LOCK` + 不匹配 WARN）。本条恢复其**意图**。 + +## Design（恢复 first-writer-wins，但不引入 getLoginUser 副作用） + +master 原 guard 用 `UserGroupInformation.getLoginUser().getAuthenticationMethod()` 读全局当前 auth 方式。**但** +`getLoginUser()` 在 loginUser 未设时会**触发进程级登录**——而 Doris **有意从不设进程级 login user**（只 per-instance +`getUGIFromSubject`；见 `IcebergConnectorTransaction.java:233` 注释）。verbatim 移植会引入这个被刻意规避的副作用。 + +→ **等价意图、无副作用**：用静态字段记住**首个** setConfiguration 采用的 auth 方式，锁内 first-writer-wins： +```java +private static UserGroupInformation.AuthenticationMethod configuredAuthMethod = null; + +public static void initializeAuthConfig(Configuration hadoopConf) { + hadoopConf.set(CommonConfigurationKeysPublic.HADOOP_SECURITY_AUTHORIZATION, "true"); + UserGroupInformation.AuthenticationMethod desired = SecurityUtil.getAuthenticationMethod(hadoopConf); + synchronized (HadoopKerberosAuthenticator.class) { + if (configuredAuthMethod == null) { + UserGroupInformation.setConfiguration(hadoopConf); // 首写者定全局 + configuredAuthMethod = desired; + return; + } + if (configuredAuthMethod != desired) { + LOG.warn("UGI already configured with authentication={} but this catalog requests {}; " + + "keeping existing JVM-global setting (first-writer-wins).", configuredAuthMethod, desired); + } + } +} +``` +- `SecurityUtil.getAuthenticationMethod(hadoopConf)` 只读**该目录自己的 conf**(无全局副作用)。 +- 首个 kerberos 目录：设全局 + 记录 auth 方式。 +- 之后目录 / 刷票：`desired==configuredAuthMethod`（同 kerberos）→ 静默跳过（满足「refresh 不重跑」）； + 仅**真不匹配**才 WARN（master 意图）。 +- `hadoopConf.set(HADOOP_SECURITY_AUTHORIZATION,"true")` 留在锁外（每目录须在**自己的 conf** 上置授权,供其自身 RPC）。 + +**只有 kerberos 认证器调 `initializeAuthConfig`**（simple 走 `HadoopSimpleAuthenticator`,不设 UGI 全局）→ 静态字段 +在 kerberos 目录间跟踪首写者正确。 + +## Risk + +- 单 kerberos 目录 / 多同 auth：首次设、之后静默跳 → 行为等价「设一次」,消除刷票 churn 与跨目录 stomp。 +- 不引入 `getLoginUser()` 进程级登录副作用（对齐 Doris 设计）。 +- fe-kerberos 独立模块,非连接器,import 门禁不适用。 +- **偏离 verbatim master**：机制换静态字段（非 getLoginUser），已在设计说明理由;意图（first-writer-wins + 不匹配 WARN）完全一致。 + +## Test Plan + +- build-compile（`SecurityUtil.getAuthenticationMethod` / `AuthenticationMethod` 签名 + 0 checkstyle）。 +- 行为 UT：`fe-kerberos` 现有 UT 仅 `KerberosTicketUtilsTest`;驱动 `initializeAuthConfig` 需真 UGI 全局态 + kerberos conf + （静态进程态,测试间污染）→ 不加,登记。以设计推理 + 对抗复审兜底。 +- e2e live-gated：两 kerberos HDFS 目录,第二个不覆写全局(FE log 无「setConfiguration」churn;auth 不同时 WARN)。 + +## 备注 + +与 L8（doAs interrupt 一行）同 fe-kerberos 模块,合并编译 + 各自独立 commit。 diff --git a/plan-doc/tasks/designs/FIX-L8-design.md b/plan-doc/tasks/designs/FIX-L8-design.md new file mode 100644 index 00000000000000..0783fb00a8f040 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L8-design.md @@ -0,0 +1,27 @@ +# FIX-L8 — kerberos doAs 把 InterruptedException→IOException 不 restore interrupt + +> 来源：reverify §1 表 L8（原 P3b-4）。🟡 低（一行）。范围：`HadoopAuthenticator.doAs`。 +> HEAD 复核基线：`e27602d4ab6`。 + +## Problem / Root Cause + +`HadoopAuthenticator.doAs`（默认方法，:31-43）`catch (InterruptedException e) { throw new IOException(e); }`—— +把 `InterruptedException` 吞成 checked `IOException` 却**不 restore 线程中断标志**。上层调用方因此无法感知线程曾被中断, +破坏协作式取消。 + +## Design（一行） + +`throw new IOException(e);` 前加 `Thread.currentThread().interrupt();`。标准 Java 最佳实践:吞掉 `InterruptedException` +（不原样上抛）时须回置中断标志。 + +## Risk + +零功能变更,仅恢复中断标志。fe-kerberos 独立模块。 + +## Test Plan + +- build-compile。一行标准惯用法,无需 UT/red-team。 + +## 备注 + +与 L7 同模块合并编译。 From 017d1af189441faa56bbcfcc223503cf8aa33652 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:32:06 +0800 Subject: [PATCH 309/334] [fix](catalog) maxcompute: partial predicate pushdown for top-level AND MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit MaxComputePredicateConverter.convert wrapped the whole expression tree in one try/catch, so a single unconvertible sub-expression (unknown column, unsupported type, an unparseable session zone) dropped the ENTIRE filter to NO_PREDICATE and pushed nothing down -> full ODPS scan. This is perf-only (BE re-evaluates the full conjuncts; the pushed predicate only reduces source reads), but it means one awkward conjunct forfeits all pushdown. Special-case a top-level ConnectorAnd: convert each conjunct independently and AND the survivors, dropping (and logging) the unconvertible ones. This is only safe at the root, a positive/monotone position — dropping a conjunct from the root AND yields a superset that BE re-filters. OR (dropping a disjunct is a subset -> loses rows), NOT, and nested AND are still converted whole by the extracted convertOne(), so no semantics change. Since dropping the whole filter was already correct, dropping some conjuncts is too. Add 5 unit tests (the converter is a pure function): a top-level AND keeps its convertible conjunct when a sibling fails (RED before the fix), all-fail still degrades to NO_PREDICATE, a single survivor is returned unwrapped, a nested AND stays all-or-nothing, and a top-level OR is not tolerated. 21/21 green; the 3 behavior tests fail against the pre-fix converter. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../MaxComputePredicateConverter.java | 24 +++++++ .../MaxComputePredicateConverterTest.java | 67 +++++++++++++++++++ 2 files changed, 91 insertions(+) diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java index 02e59cfc33aaae..327889c4d555ee 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java @@ -88,6 +88,30 @@ public Predicate convert(ConnectorExpression expr) { if (expr == null) { return Predicate.NO_PREDICATE; } + // Top-level conjunction: convert each conjunct independently and AND the survivors, so one + // unconvertible conjunct doesn't sink the whole filter. This is only safe at the root (a positive, + // monotone position): dropping a conjunct from the root AND yields a superset that BE re-filters. + // OR (dropping a disjunct is a subset -> loses rows), NOT, and nested AND are converted whole by + // convertOne(); partially dropping inside them could change the predicate's meaning. + if (expr instanceof ConnectorAnd) { + List survivors = new ArrayList<>(); + for (ConnectorExpression conjunct : ((ConnectorAnd) expr).getConjuncts()) { + Predicate p = convertOne(conjunct); + if (p != Predicate.NO_PREDICATE) { + survivors.add(p); + } + } + if (survivors.isEmpty()) { + return Predicate.NO_PREDICATE; + } + return survivors.size() == 1 + ? survivors.get(0) + : new CompoundPredicate(CompoundPredicate.Operator.AND, survivors); + } + return convertOne(expr); + } + + private Predicate convertOne(ConnectorExpression expr) { try { return doConvert(expr); } catch (Exception e) { diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java index eaa1864e2edc6d..acf9297cbdb2a9 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.pushdown.ConnectorIn; import org.apache.doris.connector.api.pushdown.ConnectorLiteral; +import org.apache.doris.connector.api.pushdown.ConnectorOr; import com.aliyun.odps.OdpsType; import com.aliyun.odps.table.optimizer.predicate.Predicate; @@ -71,6 +72,7 @@ private static Map typeMap() { m.put("ts", OdpsType.TIMESTAMP); m.put("ntz", OdpsType.TIMESTAMP_NTZ); m.put("id", OdpsType.INT); + m.put("amount", OdpsType.INT); return m; } @@ -264,4 +266,69 @@ public void testKnownColumnComparisonStillPushed() { Assertions.assertTrue(p.toString().contains("id"), "a known-column predicate must still push down; got: " + p); } + + // ---- L9 (FIX-L9): a top-level AND must push its convertible conjuncts even when one conjunct is + // unconvertible, instead of dropping the whole filter (perf; BE re-filters). Only the ROOT AND + // gets per-conjunct tolerance — OR and nested AND stay all-or-nothing so no rows are lost. + // "ghost"/"ghost2" are not in typeMap(), so they are unconvertible. ---- + + private static ConnectorComparison intEq(String col, long value) { + return new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef(col, ConnectorType.of("INT")), ConnectorLiteral.ofLong(value)); + } + + @Test + public void topLevelAndKeepsConvertibleWhenOneConjunctFails() { + // Before the fix: id=5 AND ghost=3 degraded the whole tree to NO_PREDICATE (full scan); + // now id=5 still pushes down and only ghost falls back to BE. + ConnectorAnd and = new ConnectorAnd(Arrays.asList( + intEq("id", 5), intEq("ghost", 3))); + Predicate p = converter(true, UTC).convert(and); + Assertions.assertNotSame(Predicate.NO_PREDICATE, p, + "a convertible conjunct must survive an unconvertible sibling"); + String s = p.toString(); + Assertions.assertTrue(s.contains("id"), "the id conjunct must be pushed; got: " + s); + Assertions.assertFalse(s.contains("ghost"), "the unconvertible conjunct must be dropped; got: " + s); + } + + @Test + public void topLevelAndAllConjunctsFailDropsToNoPredicate() { + ConnectorAnd and = new ConnectorAnd(Arrays.asList( + intEq("ghost", 3), intEq("ghost2", 4))); + Assertions.assertSame(Predicate.NO_PREDICATE, converter(true, UTC).convert(and), + "when every conjunct is unconvertible the filter degrades to NO_PREDICATE"); + } + + @Test + public void topLevelAndSingleSurvivorReturnedWithoutWrappingAnd() { + // One survivor -> return it directly (not wrapped in CompoundPredicate(AND, [x])). + Predicate single = converter(true, UTC).convert(new ConnectorAnd( + Arrays.asList(intEq("id", 5), intEq("ghost", 3)))); + Predicate bare = converter(true, UTC).convert(intEq("id", 5)); + Assertions.assertEquals(bare.toString(), single.toString(), + "a single surviving conjunct must equal the bare predicate; got: " + single); + } + + @Test + public void nestedAndStaysAllOrNothing() { + // id=5 AND (amount=6 AND ghost=3): the nested AND is converted whole, so its failure drops the + // whole nested conjunct -- amount is lost too. Only top-level conjuncts get per-conjunct tolerance. + ConnectorAnd nested = new ConnectorAnd(Arrays.asList( + intEq("amount", 6), intEq("ghost", 3))); + ConnectorAnd and = new ConnectorAnd(Arrays.asList(intEq("id", 5), nested)); + String s = converter(true, UTC).convert(and).toString(); + Assertions.assertTrue(s.contains("id"), "top-level id conjunct must push; got: " + s); + Assertions.assertFalse(s.contains("amount"), + "a nested AND is all-or-nothing: its convertible sibling is dropped with it; got: " + s); + } + + @Test + public void topLevelOrIsNotTolerated() { + // id=5 OR ghost=3: dropping a disjunct would make the predicate a subset and lose rows, so OR is + // all-or-nothing -- one unconvertible disjunct drops the whole OR. + ConnectorOr or = new ConnectorOr(Arrays.asList( + intEq("id", 5), intEq("ghost", 3))); + Assertions.assertSame(Predicate.NO_PREDICATE, converter(true, UTC).convert(or), + "an unconvertible disjunct must drop the whole OR (no row loss)"); + } } From ca77ea774f752f2d8824c36f1b38f7364586bb7b Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 22:32:47 +0800 Subject: [PATCH 310/334] [doc](catalog) #65185 reverify: land L9 design (mc partial pushdown), mark done Design doc for maxcompute L9 (top-level-AND partial predicate pushdown with the monotone-position correctness argument). Mark L9 DONE and repoint HANDOFF next-step to the paimon cluster (L11/L13/L14). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 7 +- plan-doc/tasks/designs/FIX-L9-design.md | 80 ++++++++++++++++++++++ 3 files changed, 86 insertions(+), 3 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L9-design.md diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md index 0366b7284db5f1..68196222758e74 100644 --- a/plan-doc/HANDOFF-65185-reverify-fixes.md +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -36,7 +36,7 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 | 批次 1 | M5/M7/M6/M4/M2 中危（连接器局部：iceberg×3 + mc + hive） | ✅ 全 DONE + 最终复核 CLEAN | | 批次 2 | M3→M1 中危(fe-core 通用节点) | ✅ 全 DONE | | **批次 3** | **L1(import 门禁) + M8(发布工具/文档)** | **✅ L1 DONE;M8 ⏸ 用户跳过(07-12)** | -| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino L3-L6 ✅ · kerberos L7/L8 ✅;**← 续 mc L9** | +| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino L3-L6 ✅ · kerberos L7/L8 ✅ · mc L9 ✅;**← 续 paimon L11/L13/L14** | | 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | | 设计债 | D-系列 | ⏸ 择机 / 随 P8 | diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 887874b5441829..9c9410bdd8afba 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -185,7 +185,8 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L6** DONE `18048f7f217`(设计 `FIX-L6-design.md`;并发对抗复审同上 SOUND）· `doInitialize` guard 字段 `trinoConnector` 移到最后赋值(安全发布,闭合半初始化瞬时 NPE 窗口);全字段已 volatile,重排即足。 - [x] **L7** DONE `9e4f2992382`(设计 `FIX-L7-design.md`;对抗复审 `agent a3e2a8a6` SOUND，对照 hadoop-common 3.4.2 字节码验证）· `initializeAuthConfig` 恢复 first-writer-wins(静态字段记首个 auth 方式,首写者设全局、后续+刷票跳过、真不匹配才 WARN)。**刻意不用 master 的 `getLoginUser()`**(会建进程级 login user,Doris 有意规避;本类是唯一非测 setConfiguration 调用者故静态字段=真全局)。刷票跳过=恢复 master parity。 - [x] **L8** DONE `59697ce3fc7`(设计 `FIX-L8-design.md`）· `doAs` catch(InterruptedException) 内 `throw` 前加 `Thread.currentThread().interrupt();`(恢复中断标志)。 -- [ ] **L9** MC 谓词下推全有全无 · `MaxComputePredicateConverter.java:87-97,117-123`:`convertFilter` 若根是 `ConnectorAnd` 则逐 top-level conjunct 独立 try/catch(丢+log 失败)再 AND 幸存者;**不**对 OR/嵌套 AND 逐子容错。 +- [x] **L9** DONE `017d1af1894`(设计 `FIX-L9-design.md`;converter 纯函数,**RED-able UT**）· `convert` 特判根 `ConnectorAnd`:逐 top-level conjunct 独立转(抽 `convertOne`),丢+log 失败者、AND 幸存者;OR/NOT/嵌套 AND 保持全有全无(只根 AND 处正/单调位置丢 conjunct=超集,BE 重滤;OR 丢 disjunct=子集会漏行)。加 5 UT(3 RED-able+2 guard),21/21 绿。perf-only。 + - [ ] **L10** ⏸ EXPLAIN 节点名 · `PluginDrivenScanNode.java:170,320`。**需决策**:登记 display-only 验收偏差(cheapest,`CONNECTOR:` 行已披露)**vs** 加连接器声明的 legacy `*_SCAN_NODE` 名(连 `Connector.getLegacyEngineName` 一起,见 D-ENGINE)。注:报告的 `connectorType.toUpperCase()+"_SCAN_NODE"` 一行修法**不够**(hms 会出 `VHMS_SCAN_NODE` 非 `VHIVE_/VICEBERG_`)。 - [ ] **L11** paimon file_format · `PaimonScanPlanProvider.java:812-813,848-849`:JNI DataSplit + COUNT 路按首数据文件后缀 `getFileFormatBySuffix("/"+dataFiles().get(0).fileName()).orElse(defaultFileFormat)`(仿 native 臂 `:540`/legacy)。注:默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。 - [ ] **L12** ⏸ selectedPartitionNum 语义 · `PluginDrivenScanNode.java:297-303`。**需决策**:登记为「paimon/iceberg 对齐 MC/hive 的 Nereids-剪枝数」验收偏差(推荐)**vs** 连接器回报 SDK-distinct(重,不推荐)。同步 paimon/iceberg EXPLAIN `partition=N/M` 回归期望。**勿**在通用节点按源分支。 @@ -283,4 +284,6 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **L7** `9e4f2992382` — `UGI.setConfiguration` first-writer-wins guard(恢复 #64655 删掉的 master guard 意图,静态字段跟踪首个 auth 方式,避 `getLoginUser` 进程级登录副作用)。对抗复审 `a3e2a8a6` SOUND(对照 hadoop 字节码)。 - **L8** `59697ce3fc7` — `doAs` 恢复中断标志(一行)。 - fe-kerberos UT 11/11 绿(含 AuthenticationTest);L7 e2e live-gated(双 kerberos HDFS 目录)。 -- **下一步 = mc L9(谓词下推全有全无),再 paimon L11/L13/L14,iceberg/杂项 L15–L19**。 + +**⭐ 批次 4·maxcompute L9 DONE** `017d1af1894` — 谓词下推从「全有全无」改为顶层 AND 逐 conjunct 容错(超集正确、BE 重滤)。**首个有 RED-able UT 的 L 条**(converter 纯函数):5 新 UT(3 RED-able+2 guard,证 OR/嵌套 AND 不容错),21/21 绿。perf-only。 +- **下一步 = paimon 子群 L11/L13/L14,再 iceberg/杂项 L15–L19**;⏸ 决策类 L2/L10/L12/L20 先问用户。 diff --git a/plan-doc/tasks/designs/FIX-L9-design.md b/plan-doc/tasks/designs/FIX-L9-design.md new file mode 100644 index 00000000000000..3634ced95a67a6 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L9-design.md @@ -0,0 +1,80 @@ +# FIX-L9 — maxcompute 谓词下推「全有全无」（一个不可转 conjunct 丢整个 filter） + +> 来源：reverify §1 表 L9（原 P4-4）。🟡 低（perf-only；BE 重过滤保正确）。范围：`MaxComputePredicateConverter.convert`。 +> HEAD 复核基线：`f62ccf25fe9`。 + +## Problem / Root Cause + +`convert(expr)`（:87-97）把整棵表达式树包在**一个** try/catch 里：树中**任一**子表达式抛（不可转列/类型/会话 TZ） +→ 整个 filter 退化为 `NO_PREDICATE` → **完全不下推** → ODPS 读全表数据。`convertAnd`（:117-123）逐 conjunct 调 +`doConvert`,任一抛即向上冒泡到 `convert` 的 catch。 + +**perf-only**：下推谓词经 `TableReadSessionBuilder.withFilterPredicate`（源端读优化）,BE 仍重评估完整 conjuncts +（现「丢整个 filter」本就靠 BE 重过滤保正确）→ 部分下推与丢全部同样正确,只是少读数据。 + +## Design（仅顶层 AND 逐 conjunct 容错；OR/NOT/嵌套 AND 保持全有全无） + +公有入口 `convert` 特判**根**是 `ConnectorAnd`：逐 top-level conjunct 独立转换,丢失败者、AND 幸存者: +```java +public Predicate convert(ConnectorExpression expr) { + if (expr == null) { + return Predicate.NO_PREDICATE; + } + if (expr instanceof ConnectorAnd) { + List survivors = new ArrayList<>(); + for (ConnectorExpression conjunct : ((ConnectorAnd) expr).getConjuncts()) { + Predicate p = convertOne(conjunct); // all-or-nothing per conjunct + if (p != Predicate.NO_PREDICATE) { + survivors.add(p); + } + } + if (survivors.isEmpty()) { + return Predicate.NO_PREDICATE; + } + return survivors.size() == 1 ? survivors.get(0) + : new CompoundPredicate(CompoundPredicate.Operator.AND, survivors); + } + return convertOne(expr); +} + +private Predicate convertOne(ConnectorExpression expr) { // = 旧 convert 的 try/catch doConvert + try { + return doConvert(expr); + } catch (Exception e) { + LOG.warn("Failed to convert expression to ODPS predicate: {}", e.getMessage()); + return Predicate.NO_PREDICATE; + } +} +``` + +**为何只顶层 AND（正确性核心）**：谓词下推须发**超集**（可多读、BE 再滤;不可漏行）。 +- 顶层根是隐式合取,处**正/单调位置**：丢一个 conjunct = 放松 AND = 超集 → 安全（且既然丢全部已正确,丢部分必正确）。 +- **OR 不容错**：丢一个 disjunct = `a OR b`→`a` = **子集** → 漏行,**不安全** → OR 整体 all-or-nothing。 +- **NOT/嵌套 AND 不逐子容错**：NOT 下放松子式 = 子集不安全;嵌套 AND 一律整体转（其失败使所属顶层 conjunct 整条被丢,仍安全）。 + → 内部递归全走 `doConvert`（`convertAnd`/`convertOr`/`convertNot` 不变,保持整体转换）。`convert` 无内部递归调用,只公有入口 + → 特判只作用于根。已核。 +- `NO_PREDICATE` 是单例哨兵（既有测试用 `assertSame`/`assertNotSame` 佐证）→ `!=` 引用比较正确。`doConvert` 失败抛异常 + （不返回 NO_PREDICATE）→ 幸存者判定无假阴。 + +## Risk + +- **正确性**：与现「丢整个 filter」等价安全（都靠 BE 重过滤）,只多下推幸存者。核实入口非递归、内部保持全有全无。 +- 既有 `testMixedAndTreeNotDropped`（两 conjunct 均转成功）结构不变（2 幸存者→AND(两者)）→ 仍绿。 +- 连接器局部,不碰 fe-core,import 门禁不适用。 + +## Test Plan + +### Unit（`MaxComputePredicateConverterTest` 加 5 例,converter 是纯函数、可直接驱动，RED-able） + +- **RED 主证** `topLevelAndKeepsConvertibleWhenOneConjunctFails`：`id=5 AND ghost=3`（ghost 不在 schema）→ 非 NO_PREDICATE、 + 含 `id`、不含 `ghost`（改前：整条 NO_PREDICATE）。 +- `topLevelAndAllConjunctsFailDropsToNoPredicate`：`ghost1=3 AND ghost2=4` → NO_PREDICATE。 +- `topLevelAndSingleSurvivorReturnedDirectly`：`id=5 AND ghost=3` → 返回单个 `id` 谓词（非包 AND）。 +- `nestedAndStaysAllOrNothing`：`id=5 AND (amount=6 AND ghost=3)` → 仅 `id`；`amount` 也被丢（证嵌套 AND 整体转,不逐子）。 +- `topLevelOrNotTolerated`：`id=5 OR ghost=3` → NO_PREDICATE（证 OR 不容错,避免漏行）。 + +（typeMap 补一个已知列 `amount` 供嵌套测试。） + +### E2E — live-gated（真 ODPS） + +mixed filter（可转 + 不可转 conjunct）下,ODPS 读端仍下推可转部分（少读数据）,结果与全 BE 过滤一致。 From 4a8650bd0629fefc98d70d85ff593c3dc77ab320 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:07:00 +0800 Subject: [PATCH 311/334] [fix](catalog) paimon: JNI/COUNT range file_format from data-file suffix MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The plugin-path paimon scan tagged JNI-serialized DataSplit ranges and the COUNT(*) collapse range with the table-level `file.format` default (parquet when unset) instead of the split's actual first data-file suffix. When the table option diverges from the on-disk files (an altered or mixed-format table), FE emitted the wrong `file_format` to BE. Narrow blast radius: the default JNI reader ignores it; only the opt-in paimon-cpp reader backfills it into FILE_FORMAT/MANIFEST_FORMAT, where a wrong value breaks the manifest read. Restore legacy `PaimonScanNode.getFileFormat(getPathString())` parity: add a package-private `dataSplitFileFormat(DataSplit, default)` helper deriving the format from `"/" + dataFiles().get(0).fileName()` (== legacy PaimonSplit path), falling back to the table default for an unrecognized suffix or empty file list (fail-safe, stricter than legacy's unguarded get(0)). Wire it into buildJniScanRange (isDataSplit arm only; non-DataSplit keeps the table default, legacy DUMMY_PATH parity) and buildCountRange. Also add the `.avro` arm dropped from getFileFormatBySuffix (legacy FileFormatUtils had it), completing exact legacy suffix parity; inert on the native arm (avro never reaches it). Test: new PaimonScanPlanProviderTest.jniAndCountRangesUseFileSuffixNotAltered- TableDefault decouples table-default (overlaid to parquet via Table.copy) from the on-disk .orc files and asserts JNI + COUNT ranges carry "orc" — RED against the pre-fix defaultFileFormat emission (the prior test used default==suffix==orc and could not distinguish). Module 66/66 green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../paimon/PaimonScanPlanProvider.java | 43 +++++++--- .../paimon/PaimonScanPlanProviderTest.java | 81 +++++++++++++++++++ 2 files changed, 115 insertions(+), 9 deletions(-) diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java index 6f00cf328f69c8..0a6fab5cc8b36f 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java @@ -804,13 +804,18 @@ private PaimonScanRange buildJniScanRange(Split split, String tableLocation, String serializedSplit = encodeSplit(split, cppReader); - // FIX-JNI-FILE-FORMAT (P7-1): emit the real data-file format (orc/parquet), NOT "jni". JNI routing - // is gated by the paimon.split property (PaimonScanRange.populateRangeParams), so this string only - // feeds fileDesc.file_format, which BE's paimon_cpp_reader backfills into FILE_FORMAT/MANIFEST_FORMAT - // (an invalid "jni" breaks the manifest read). Mirrors legacy PaimonScanNode.setPaimonParams's - // fileDesc.setFileFormat(getFileFormat(...)). + // FIX-JNI-FILE-FORMAT (P7-1) + FIX-L11: emit the real data-file format (orc/parquet/avro), NOT "jni". + // JNI routing is gated by the paimon.split property (PaimonScanRange.populateRangeParams), so this + // string only feeds fileDesc.file_format, which BE's paimon_cpp_reader backfills into + // FILE_FORMAT/MANIFEST_FORMAT (an invalid "jni" breaks the manifest read). Mirrors legacy + // PaimonScanNode.setPaimonParams's fileDesc.setFileFormat(getFileFormat(getPathString())): for a + // DataSplit the format is the FIRST data-file suffix (falling back to the table default); a + // non-DataSplit has no data file and falls back to the table default (legacy DUMMY_PATH -> orElse). + String fileFormat = isDataSplit + ? dataSplitFileFormat((DataSplit) split, defaultFileFormat) + : defaultFileFormat; return new PaimonScanRange.Builder() - .fileFormat(defaultFileFormat) + .fileFormat(fileFormat) .paimonSplit(serializedSplit) // FIX-READER-TYPE (3645dc94306): lockstep with encodeSplit above — PAIMON_CPP iff we // native-serialized a DataSplit for the paimon-cpp reader, else PAIMON_JNI. @@ -844,9 +849,10 @@ private PaimonScanRange buildCountRange(DataSplit dataSplit, String tableLocatio String defaultFileFormat, Map partitionValues, boolean cppReader, long rowCount, long weightDenominator) { String serializedSplit = encodeSplit(dataSplit, cppReader); - // FIX-JNI-FILE-FORMAT (P7-1): real data-file format, not "jni" (see buildJniScanRange). + // FIX-JNI-FILE-FORMAT (P7-1) + FIX-L11: real data-file format from the first data-file suffix, not + // "jni" and not the bare table default (see buildJniScanRange / dataSplitFileFormat). return new PaimonScanRange.Builder() - .fileFormat(defaultFileFormat) + .fileFormat(dataSplitFileFormat(dataSplit, defaultFileFormat)) .paimonSplit(serializedSplit) // FIX-READER-TYPE (3645dc94306): dataSplit is always a DataSplit here, so cpp-reader // serialization (hence PAIMON_CPP) is chosen iff the flag is on — matches encodeSplit above. @@ -1179,12 +1185,31 @@ Map getBackendPaimonOptions() { return options; } + /** + * The real data-file format of a {@link DataSplit}: the suffix of its FIRST data file (legacy + * {@code PaimonSplit} path = {@code "/" + dataFiles().get(0).fileName()}), falling back to the table + * default when the suffix is unrecognized or the split carries no data file. Ports legacy + * {@code PaimonScanNode.getFileFormat(getPathString())} for the JNI/COUNT arms, where HEAD had regressed + * to the bare table-level {@code file.format} default (wrong when the option differs from the on-disk + * files, e.g. an altered/mixed-format table). Package-private static so the suffix-over-default decision + * is unit-testable, like {@link #isCountPushdownSplit} / {@link #computeFileSplitOffsets}. + */ + static String dataSplitFileFormat(DataSplit dataSplit, String defaultFileFormat) { + List files = dataSplit.dataFiles(); + if (files == null || files.isEmpty()) { + return defaultFileFormat; + } + return getFileFormatBySuffix("/" + files.get(0).fileName()).orElse(defaultFileFormat); + } + private static Optional getFileFormatBySuffix(String path) { if (path == null) { return Optional.empty(); } String lower = path.toLowerCase(); - if (lower.endsWith(".orc")) { + if (lower.endsWith(".avro")) { + return Optional.of("avro"); + } else if (lower.endsWith(".orc")) { return Optional.of("orc"); } else if (lower.endsWith(".parquet") || lower.endsWith(".parq")) { return Optional.of("parquet"); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java index 37e6f493f5b1e8..9b691899cc3e61 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java @@ -962,6 +962,87 @@ public void jniAndCountRangesCarryRealFileFormatNotJni(@TempDir Path warehouse) } } + @Test + public void jniAndCountRangesUseFileSuffixNotAlteredTableDefault(@TempDir Path warehouse) throws Exception { + // FIX-L11: the JNI-serialized split (default reader path) and the COUNT(*) collapse range must derive + // file_format from the split's FIRST data-file SUFFIX (legacy PaimonScanNode.getFileFormat(getPathString) + // -> dataSplitFileFormat), NOT the table-level file.format option. These DIVERGE for an altered / + // mixed-format table: the option is changed to parquet while historical data files remain .orc. HEAD + // regressed to emitting the bare table default, so BE's paimon_cpp_reader would backfill the WRONG + // format for those files. WHY it matters: unlike jniAndCountRangesCarryRealFileFormatNotJni (where the + // table default == the .orc suffix, so it cannot distinguish default from suffix), this test forces a + // mismatch and thus is the one that actually goes RED if the emission points revert to defaultFileFormat. + try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(), + new org.apache.paimon.fs.Path(warehouse.toUri()))) { + catalog.createDatabase("db", false); + Identifier id = Identifier.create("db", "t"); + catalog.createTable(id, Schema.newBuilder() + .column("pt", DataTypes.INT()) + .column("id", DataTypes.INT()) + .column("val", DataTypes.BIGINT()) + .partitionKeys("pt") + .primaryKey("pt", "id") + .option("bucket", "1") + .option("file.format", "orc") // on-disk data files are written as .orc + .build(), false); + Table table = catalog.getTable(id); + BatchWriteBuilder wb = table.newBatchWriteBuilder(); + try (BatchTableWrite write = wb.newWrite()) { + write.write(GenericRow.of(1, 1, 100L)); + write.write(GenericRow.of(1, 2, 200L)); + List messages = write.prepareCommit(); + try (BatchTableCommit commit = wb.newCommit()) { + commit.commit(messages); + } + } + + // Overlay file.format=parquet as a dynamic option: the table now REPORTS parquet as its default + // (the connector reads table.options().file.format at plan time), while the committed data files + // stay .orc. copy() overlays read-time options only; it does NOT rewrite the on-disk files, and + // reads still decode each file by its own recorded format. + Table altered = table.copy(Collections.singletonMap("file.format", "parquet")); + Assertions.assertEquals("parquet", altered.options().get("file.format"), + "precondition: the altered table default must be parquet (distinct from the .orc files)"); + + RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps(); + ops.table = altered; + PaimonScanPlanProvider provider = new PaimonScanPlanProvider(Collections.emptyMap(), ops); + PaimonTableHandle handle = new PaimonTableHandle( + "db", "t", Collections.emptyList(), Collections.emptyList()); + List noColumns = Collections.emptyList(); + + // (a) JNI data range: force_jni routes the .orc split to buildJniScanRange. Its file_format must be + // "orc" (the real data-file suffix), NOT "parquet" (the altered table default). + ConnectorSession forceJni = sessionWithProps( + Collections.singletonMap("force_jni_scanner", "true")); + List jniRanges = provider.planScan( + forceJni, handle, noColumns, Optional.empty(), -1, null, /*countPushdown*/ false); + Assertions.assertFalse(jniRanges.isEmpty(), "force_jni scan must emit >=1 JNI range"); + for (ConnectorScanRange r : jniRanges) { + Assertions.assertTrue(r.getProperties().containsKey("paimon.split"), + "force_jni_scanner=true must route the split to the JNI path"); + // MUTATION: buildJniScanRange .fileFormat(defaultFileFormat) -> "parquet" -> red. + Assertions.assertEquals("orc", ((PaimonScanRange) r).getFileFormat(), + "a JNI range must carry the real data-file suffix (orc), not the altered table default"); + } + + // (b) COUNT(*) collapse range (buildCountRange): same suffix-over-default requirement. + ConnectorSession plain = sessionWithProps(Collections.emptyMap()); + List countRanges = provider.planScan( + plain, handle, noColumns, Optional.empty(), -1, null, /*countPushdown*/ true); + PaimonScanRange countRange = null; + for (ConnectorScanRange r : countRanges) { + if (r.getProperties().containsKey("paimon.row_count")) { + countRange = (PaimonScanRange) r; + } + } + Assertions.assertNotNull(countRange, "count pushdown must emit a collapsed count range"); + // MUTATION: buildCountRange .fileFormat(defaultFileFormat) -> "parquet" -> red. + Assertions.assertEquals("orc", countRange.getFileFormat(), + "the COUNT(*) collapse range must carry the real data-file suffix (orc), not the altered default"); + } + } + // ---- FIX-NATIVE-SUBSPLIT (M-3) ---- private static final long MB = 1024L * 1024L; From ced4775b8445c6b8f5058ce80d3764bf01473347 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:12:01 +0800 Subject: [PATCH 312/334] [fix](catalog) paimon: preserve nested nullability in Doris->Paimon mapping MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CREATE TABLE / CTAS building a paimon table dropped the declared NOT NULL on nested child types: ARRAY was built as ARRAY (nullable element), and MAP values / STRUCT fields likewise lost their nullability, so the on-disk paimon schema did not match the DDL. PaimonTypeMapping.toPaimonType now applies .copy(type.isChildNullable(i)) to the ARRAY element, the MAP value, and each STRUCT DataField (the MAP key stays .copy(false), legacy parity), restoring legacy DorisToPaimonTypeVisitor behavior (array/value/field = childResult.copy(getContainsNull)). Scope is nested nullability only: the field comment stays dropped (accepted display-only deviation DV-035 M10.1) and the field id stays sequential (legacy parity). ConnectorType already carries per-child nullability (isChildNullable), so no SPI change is needed. The default-nullable path is unchanged (.copy(true) is a paimon no-op), so existing parity tests stay green. Test: 3 new PaimonTypeMappingToPaimonTest cases assert element/value/field non-null survives for NOT NULL children (via .type().isNullable(), not DataField equality, since the comment stays dropped) — RED against the pre-fix no-copy. Module type-mapping + schema-builder tests 26/26 green, 0 checkstyle, gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/paimon/PaimonTypeMapping.java | 17 +++++--- .../paimon/PaimonTypeMappingToPaimonTest.java | 43 +++++++++++++++++++ 2 files changed, 55 insertions(+), 5 deletions(-) diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java index 4d00395f75fd9b..f60a505bf20c2e 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonTypeMapping.java @@ -251,12 +251,16 @@ public static DataType toPaimonType(ConnectorType type) { case "VARIANT": return new VariantType(); case "ARRAY": - return new ArrayType(toPaimonType(type.getChildren().get(0))); + // FIX-L13: preserve the declared element nullability (legacy DorisToPaimonTypeVisitor + // array = elementResult.copy(array.getContainsNull())). + return new ArrayType( + toPaimonType(type.getChildren().get(0)).copy(type.isChildNullable(0))); case "MAP": - // Legacy forces the map key non-null via .copy(false). + // Legacy forces the map key non-null via .copy(false); the value preserves the declared + // nullability (FIX-L13: legacy map = valueResult.copy(map.getIsValueContainsNull())). return new MapType( toPaimonType(type.getChildren().get(0)).copy(false), - toPaimonType(type.getChildren().get(1))); + toPaimonType(type.getChildren().get(1)).copy(type.isChildNullable(1))); case "STRUCT": case "ROW": return toPaimonRowType(type); @@ -274,8 +278,11 @@ private static DataType toPaimonRowType(ConnectorType type) { AtomicInteger fieldId = new AtomicInteger(-1); for (int i = 0; i < children.size(); i++) { String fieldName = i < names.size() && names.get(i) != null ? names.get(i) : "col" + i; - fields.add(new DataField( - fieldId.incrementAndGet(), fieldName, toPaimonType(children.get(i)))); + // FIX-L13: preserve the declared field nullability (legacy struct = + // fieldResults.get(i).copy(field.getContainsNull())). The field comment stays dropped + // (accepted display-only deviation DV-035 M10.1) and the field id stays sequential (legacy parity). + fields.add(new DataField(fieldId.incrementAndGet(), fieldName, + toPaimonType(children.get(i)).copy(type.isChildNullable(i)))); } return new RowType(fields); } diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java index 371053bc170199..09b71632e1fd52 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTypeMappingToPaimonTest.java @@ -153,6 +153,49 @@ public void structBuildsSequentialFieldIdsAndNames() { Assertions.assertEquals(1, row.getFields().get(1).id(), "second struct field id must be 1"); } + @Test + public void nestedNullabilityPreservedForArrayElement() { + // WHY (FIX-L13): a declared NOT NULL element (ARRAY) must map to a non-null paimon + // element type (legacy DorisToPaimonTypeVisitor array = elementResult.copy(array.getContainsNull())). + // MUTATION: dropping the .copy(isChildNullable(0)) on the ARRAY element leaves it nullable -> red. + DataType actual = PaimonTypeMapping.toPaimonType( + ConnectorType.arrayOf(ConnectorType.of("INT"), /*elementNullable*/ false)); + Assertions.assertFalse(((ArrayType) actual).getElementType().isNullable(), + "a NOT NULL array element must map to a non-null paimon element type"); + } + + @Test + public void nestedNullabilityPreservedForMapValue() { + // WHY (FIX-L13): a declared NOT NULL map value (MAP) must map to a non-null + // paimon value type (legacy map value = valueResult.copy(map.getIsValueContainsNull())), while the + // key stays non-null. MUTATION: dropping the .copy(isChildNullable(1)) on the MAP value -> red. + DataType actual = PaimonTypeMapping.toPaimonType(ConnectorType.mapOf( + ConnectorType.of("STRING"), ConnectorType.of("INT"), /*valueNullable*/ false)); + Assertions.assertFalse(((MapType) actual).getValueType().isNullable(), + "a NOT NULL map value must map to a non-null paimon value type"); + Assertions.assertFalse(((MapType) actual).getKeyType().isNullable(), + "the map key stays non-null regardless (legacy .copy(false))"); + } + + @Test + public void nestedNullabilityPreservedForStructField() { + // WHY (FIX-L13): declared per-field nullability (STRUCT) must survive to + // the paimon DataField types (legacy struct = fieldResults.get(i).copy(field.getContainsNull())). + // Asserted via field.type().isNullable() — NOT DataField equality: the field comment stays dropped + // (DV-035 M10.1), so a DataField carrying a description would false-differ for an unrelated reason. + // MUTATION: dropping the .copy(isChildNullable(i)) on the struct DataField -> field 0 stays nullable. + ConnectorType struct = ConnectorType.structOf( + Arrays.asList("x", "y"), + Arrays.asList(ConnectorType.of("INT"), ConnectorType.of("STRING")), + Arrays.asList(false, true), + Collections.emptyList()); + RowType row = (RowType) PaimonTypeMapping.toPaimonType(struct); + Assertions.assertFalse(row.getFields().get(0).type().isNullable(), + "a NOT NULL struct field must map to a non-null paimon field type"); + Assertions.assertTrue(row.getFields().get(1).type().isNullable(), + "a nullable struct field stays nullable"); + } + @Test public void unsupportedScalarTypesThrow() { // WHY: the legacy visitor had no branch for these and threw; the connector preserves that From 478718aca6f63671bc65fa92fd9ba73412d074b5 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:19:06 +0800 Subject: [PATCH 313/334] [fix](catalog) paimon: honor ignore_split_type session variable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The `ignore_split_type` debugging escape hatch (IGNORE_JNI / IGNORE_NATIVE, used to isolate reader bugs by dropping one split kind) was a silent no-op on the plugin scan path — PaimonScanPlanProvider never read it, so setting it had no effect. Restore legacy PaimonScanNode.getSplits parity: read it once via a null-tolerant resolveIgnoreSplitType(session) helper (mirroring isCppReaderEnabled), then skip splits at the three legacy continue sites — IGNORE_JNI drops the nonDataSplit arm and the DataSplit-JNI arm, IGNORE_NATIVE drops the native arm. The COUNT(*) arm is never dropped (legacy parity). IGNORE_PAIMON_CPP stays a no-op: legacy getSplits never consulted it (whole-tree grep confirms), so honoring it would be a new divergence, not parity. Default NONE leaves every scan byte-unchanged. Test: 3 new PaimonScanPlanProviderTest cases (live ops.table + planScan) — IGNORE_JNI drops a force_jni split, IGNORE_NATIVE drops an append-only native split (precondition asserts the split really took the native path), IGNORE_PAIMON_CPP == NONE. RED against the pre-fix no-op. The nonDataSplit IGNORE_JNI arm is E2E-only (offline fixtures cannot produce a system-table non-DataSplit through planScan). Module scan tests 69/69 green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../paimon/PaimonScanPlanProvider.java | 44 +++++ .../paimon/PaimonScanPlanProviderTest.java | 150 ++++++++++++++++++ 2 files changed, 194 insertions(+) diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java index 0a6fab5cc8b36f..0d619b7d528b48 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java @@ -152,6 +152,15 @@ public class PaimonScanPlanProvider implements ConnectorScanPlanProvider { // bypassing the native ORC/Parquet readers to dodge native-reader bugs. Default false (legacy default). private static final String FORCE_JNI_SCANNER = "force_jni_scanner"; + // Session variable name (byte-identical to SessionVariable.IGNORE_SPLIT_TYPE) surfaced through the same + // VariableMgr.toMap channel. A debugging escape hatch to isolate reader bugs: IGNORE_JNI drops every JNI + // split, IGNORE_NATIVE drops every native split (legacy PaimonScanNode.getSplits). IGNORE_PAIMON_CPP is a + // documented option but was NEVER consulted by legacy getSplits, so it stays a no-op here (legacy parity). + // Default NONE, so normal reads are unaffected. + private static final String IGNORE_SPLIT_TYPE = "ignore_split_type"; + private static final String IGNORE_SPLIT_TYPE_JNI = "IGNORE_JNI"; + private static final String IGNORE_SPLIT_TYPE_NATIVE = "IGNORE_NATIVE"; + // FIX-NATIVE-SUBSPLIT (M-3): file-split session vars (byte-identical to SessionVariable.{FILE_SPLIT_SIZE, // MAX_INITIAL_FILE_SPLIT_SIZE, MAX_FILE_SPLIT_SIZE, MAX_INITIAL_FILE_SPLIT_NUM, MAX_FILE_SPLIT_NUM}), // read via the same VariableMgr.toMap channel as ENABLE_PAIMON_CPP_READER. They drive the native @@ -250,6 +259,21 @@ static boolean isForceJniScannerEnabled(ConnectorSession session) { return Boolean.parseBoolean(session.getSessionProperties().get(FORCE_JNI_SCANNER)); } + /** + * Reads the {@code ignore_split_type} session variable (same {@code VariableMgr.toMap} channel as + * {@link #isCppReaderEnabled}). Returns {@code "NONE"} when the session is absent (offline unit tests) + * or the variable is unset, matching this file's null-tolerant session-read convention. Only + * {@code IGNORE_JNI} / {@code IGNORE_NATIVE} carry behavior (skip the matching split type, legacy + * {@code PaimonScanNode.getSplits}); every other value (incl. {@code NONE} / {@code IGNORE_PAIMON_CPP}) + * is a no-op. Package-private static for offline unit testing. + */ + static String resolveIgnoreSplitType(ConnectorSession session) { + if (session == null) { + return "NONE"; + } + return session.getSessionProperties().getOrDefault(IGNORE_SPLIT_TYPE, "NONE"); + } + /** * Returns the handle's transient Paimon {@link Table}, reloading it from the catalog seam * when the transient reference is null (e.g. after a serialization round-trip across the @@ -427,6 +451,14 @@ private List planScanInternal( // Read the cpp-reader flag once: it selects the JNI split serialization format (see encodeSplit). boolean cppReader = isCppReaderEnabled(session); + // FIX-L14: honor the ignore_split_type debugging escape hatch (legacy PaimonScanNode.getSplits): + // IGNORE_JNI drops JNI splits (nonDataSplit + DataSplit-JNI arms), IGNORE_NATIVE drops native splits. + // The COUNT(*) arm is never dropped (legacy parity); IGNORE_PAIMON_CPP stays a no-op (legacy getSplits + // never consulted it). Read once here, null-tolerant like the flags above. + String ignoreSplitType = resolveIgnoreSplitType(session); + boolean ignoreJni = IGNORE_SPLIT_TYPE_JNI.equals(ignoreSplitType); + boolean ignoreNative = IGNORE_SPLIT_TYPE_NATIVE.equals(ignoreSplitType); + // FIX-REST-VENDED-URI-NORMALIZE (P9-1): extract the per-table vended token ONCE per scan // (validToken() may refresh; legacy computes its storage map once in doInitialize), threaded into // the native-path URI normalization below so REST object-store reads normalize via the vended @@ -444,6 +476,10 @@ private List planScanInternal( // Non-DataSplit → always JNI for (Split split : nonDataSplits) { + if (ignoreJni) { + // FIX-L14: ignore_split_type=IGNORE_JNI drops JNI splits (legacy getSplits:401). + continue; + } ranges.add(buildJniScanRange(split, tableLocation, defaultFileFormat, Collections.emptyMap(), false, cppReader, weightDenominator)); } @@ -484,6 +520,10 @@ private List planScanInternal( if (shouldUseNativeReader(paimonHandle.isForceJni(), isForceJniScannerEnabled(session), optRawFiles)) { + if (ignoreNative) { + // FIX-L14: ignore_split_type=IGNORE_NATIVE drops native splits (legacy getSplits:443). + continue; + } // Native reader path: sub-split large ORC/Parquet files for read parallelism // (FIX-NATIVE-SUBSPLIT), mirroring legacy fileSplitter.splitFile. Under COUNT(*) pushdown // legacy passes splittable=!applyCountPushdown, so a native split that reaches this arm @@ -506,6 +546,10 @@ private List planScanInternal( } } else { // JNI reader path + if (ignoreJni) { + // FIX-L14: ignore_split_type=IGNORE_JNI drops JNI splits (legacy getSplits:483). + continue; + } ranges.add(buildJniScanRange( dataSplit, tableLocation, defaultFileFormat, partitionValues, true, cppReader, weightDenominator)); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java index 9b691899cc3e61..b8c031eb797625 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderTest.java @@ -1043,6 +1043,156 @@ public void jniAndCountRangesUseFileSuffixNotAlteredTableDefault(@TempDir Path w } } + // ---- FIX-L14: ignore_split_type escape hatch ---- + + @Test + public void ignoreJniDropsForcedJniSplit(@TempDir Path warehouse) throws Exception { + // FIX-L14: force_jni routes the DataSplit to the JNI arm; ignore_split_type=IGNORE_JNI must then drop + // it (legacy PaimonScanNode.getSplits:483). MUTATION: not honoring IGNORE_JNI -> the JNI range is + // still emitted -> red. + try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(), + new org.apache.paimon.fs.Path(warehouse.toUri()))) { + catalog.createDatabase("db", false); + Identifier id = Identifier.create("db", "t"); + catalog.createTable(id, Schema.newBuilder() + .column("id", DataTypes.INT()) + .column("val", DataTypes.BIGINT()) + .primaryKey("id") + .option("bucket", "1") + .build(), false); + Table table = catalog.getTable(id); + BatchWriteBuilder wb = table.newBatchWriteBuilder(); + try (BatchTableWrite write = wb.newWrite()) { + write.write(GenericRow.of(1, 100L)); + write.write(GenericRow.of(2, 200L)); + List messages = write.prepareCommit(); + try (BatchTableCommit commit = wb.newCommit()) { + commit.commit(messages); + } + } + + RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps(); + ops.table = table; + PaimonScanPlanProvider provider = new PaimonScanPlanProvider(Collections.emptyMap(), ops); + PaimonTableHandle handle = new PaimonTableHandle( + "db", "t", Collections.emptyList(), Collections.emptyList()); + List noColumns = Collections.emptyList(); + + // Baseline: force_jni alone emits a JNI range. + List baseline = provider.planScan( + sessionWithProps(Collections.singletonMap("force_jni_scanner", "true")), + handle, noColumns, Optional.empty(), -1, null, /*countPushdown*/ false); + Assertions.assertFalse(baseline.isEmpty(), "force_jni alone must emit >=1 JNI range (baseline)"); + + // force_jni + IGNORE_JNI: the JNI split is dropped (2-entry props map). + Map props = new HashMap<>(); + props.put("force_jni_scanner", "true"); + props.put("ignore_split_type", "IGNORE_JNI"); + List ignored = provider.planScan( + sessionWithProps(props), handle, noColumns, Optional.empty(), -1, null, false); + Assertions.assertTrue(ignored.isEmpty(), + "ignore_split_type=IGNORE_JNI must drop the forced-JNI DataSplit"); + } + } + + @Test + public void ignoreNativeDropsNativeSplit(@TempDir Path warehouse) throws Exception { + // FIX-L14: an append-only table's DataSplit is native-eligible; ignore_split_type=IGNORE_NATIVE must + // drop it (legacy PaimonScanNode.getSplits:443). MUTATION: not honoring IGNORE_NATIVE -> the native + // range is still emitted -> red. + try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(), + new org.apache.paimon.fs.Path(warehouse.toUri()))) { + catalog.createDatabase("db", false); + Identifier id = Identifier.create("db", "t"); + catalog.createTable(id, Schema.newBuilder() + .column("id", DataTypes.INT()) + .column("val", DataTypes.BIGINT()) + .option("bucket", "-1") // append-only (no primary key) -> raw files -> native reader + .build(), false); + Table table = catalog.getTable(id); + BatchWriteBuilder wb = table.newBatchWriteBuilder(); + try (BatchTableWrite write = wb.newWrite()) { + write.write(GenericRow.of(1, 100L)); + write.write(GenericRow.of(2, 200L)); + List messages = write.prepareCommit(); + try (BatchTableCommit commit = wb.newCommit()) { + commit.commit(messages); + } + } + + RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps(); + ops.table = table; + PaimonScanPlanProvider provider = new PaimonScanPlanProvider(Collections.emptyMap(), ops); + PaimonTableHandle handle = new PaimonTableHandle( + "db", "t", Collections.emptyList(), Collections.emptyList()); + List noColumns = Collections.emptyList(); + + // Baseline (NONE): a native range is emitted (no paimon.split marker == native path). + List baseline = provider.planScan( + sessionWithProps(Collections.emptyMap()), + handle, noColumns, Optional.empty(), -1, null, false); + Assertions.assertFalse(baseline.isEmpty(), "append-only scan must emit >=1 range (baseline)"); + boolean anyNative = baseline.stream() + .anyMatch(r -> !r.getProperties().containsKey("paimon.split")); + Assertions.assertTrue(anyNative, + "precondition: the append-only split must take the native path (no paimon.split)"); + + // IGNORE_NATIVE: the native split is dropped. + List ignored = provider.planScan( + sessionWithProps(Collections.singletonMap("ignore_split_type", "IGNORE_NATIVE")), + handle, noColumns, Optional.empty(), -1, null, false); + boolean anyNativeAfter = ignored.stream() + .anyMatch(r -> !r.getProperties().containsKey("paimon.split")); + Assertions.assertFalse(anyNativeAfter, + "ignore_split_type=IGNORE_NATIVE must drop the native split"); + } + } + + @Test + public void ignorePaimonCppIsNoOpParity(@TempDir Path warehouse) throws Exception { + // FIX-L14: IGNORE_PAIMON_CPP is a documented ignore_split_type value that legacy + // PaimonScanNode.getSplits NEVER consulted, so it stays a no-op (legacy parity) — the scan emits the + // same ranges as NONE. Pins that a future change does not add a half-implemented CPP arm. + try (Catalog catalog = new FileSystemCatalog(LocalFileIO.create(), + new org.apache.paimon.fs.Path(warehouse.toUri()))) { + catalog.createDatabase("db", false); + Identifier id = Identifier.create("db", "t"); + catalog.createTable(id, Schema.newBuilder() + .column("id", DataTypes.INT()) + .column("val", DataTypes.BIGINT()) + .primaryKey("id") + .option("bucket", "1") + .build(), false); + Table table = catalog.getTable(id); + BatchWriteBuilder wb = table.newBatchWriteBuilder(); + try (BatchTableWrite write = wb.newWrite()) { + write.write(GenericRow.of(1, 100L)); + write.write(GenericRow.of(2, 200L)); + List messages = write.prepareCommit(); + try (BatchTableCommit commit = wb.newCommit()) { + commit.commit(messages); + } + } + + RecordingPaimonCatalogOps ops = new RecordingPaimonCatalogOps(); + ops.table = table; + PaimonScanPlanProvider provider = new PaimonScanPlanProvider(Collections.emptyMap(), ops); + PaimonTableHandle handle = new PaimonTableHandle( + "db", "t", Collections.emptyList(), Collections.emptyList()); + List noColumns = Collections.emptyList(); + + int noneCount = provider.planScan( + sessionWithProps(Collections.emptyMap()), + handle, noColumns, Optional.empty(), -1, null, false).size(); + int cppCount = provider.planScan( + sessionWithProps(Collections.singletonMap("ignore_split_type", "IGNORE_PAIMON_CPP")), + handle, noColumns, Optional.empty(), -1, null, false).size(); + Assertions.assertTrue(noneCount > 0, "baseline scan must emit >=1 range"); + Assertions.assertEquals(noneCount, cppCount, + "IGNORE_PAIMON_CPP must be a no-op (legacy parity): same range count as NONE"); + } + } + // ---- FIX-NATIVE-SUBSPLIT (M-3) ---- private static final long MB = 1024L * 1024L; From dffd51ab99475c565cbdfd547844157d511afbc9 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:22:33 +0800 Subject: [PATCH 314/334] [doc](catalog) #65185 reverify: land paimon designs (L11/L13/L14), mark done Land the three paimon subgroup fix designs (JNI/COUNT file_format from data-file suffix; nested nullability in Doris->Paimon mapping; honor ignore_split_type), each with its folded-in red-team resolution (workflow wf_05574ccb-bd2, 3 designs x 3 lenses, no UNSOUND). Mark L11/L13/L14 done in the tracking table and roll the HANDOFF: batch 4 paimon subgroup complete, next = iceberg/misc L15-L19. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 19 +++- plan-doc/task-list-65185-reverify-fixes.md | 20 ++-- plan-doc/tasks/designs/FIX-L11-design.md | 108 +++++++++++++++++++++ plan-doc/tasks/designs/FIX-L13-design.md | 102 +++++++++++++++++++ plan-doc/tasks/designs/FIX-L14-design.md | 98 +++++++++++++++++++ 5 files changed, 336 insertions(+), 11 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L11-design.md create mode 100644 plan-doc/tasks/designs/FIX-L13-design.md create mode 100644 plan-doc/tasks/designs/FIX-L14-design.md diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md index 68196222758e74..2012d36012adbb 100644 --- a/plan-doc/HANDOFF-65185-reverify-fixes.md +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -36,7 +36,7 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 | 批次 1 | M5/M7/M6/M4/M2 中危（连接器局部：iceberg×3 + mc + hive） | ✅ 全 DONE + 最终复核 CLEAN | | 批次 2 | M3→M1 中危(fe-core 通用节点) | ✅ 全 DONE | | **批次 3** | **L1(import 门禁) + M8(发布工具/文档)** | **✅ L1 DONE;M8 ⏸ 用户跳过(07-12)** | -| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino L3-L6 ✅ · kerberos L7/L8 ✅ · mc L9 ✅;**← 续 paimon L11/L13/L14** | +| 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino L3-L6 ✅ · kerberos L7/L8 ✅ · mc L9 ✅ · paimon L11/L13/L14 ✅;**← 续 iceberg/杂项 L15–L19** | | 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | | 设计债 | D-系列 | ⏸ 择机 / 随 P8 | @@ -75,9 +75,20 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 修法=升级文档/release-note + (**可选**)replay 收尾聚合 degraded ERROR（触 fe-core,需编译）。**不 silently drop**,表中留待办; 回来做时先中文讲清「仅文档 vs 文档+可选防御码」让用户拍板。 -**下一步 = 批次 4（低危连接器,用户指向）**：直接可做=`L3–L9`（trino 事务/单例/去重/NPE、kerberos guard/interrupt、mc 谓词）、 -`L11/L13/L14`（paimon file_format/nullability/ignore_split_type）、`L15–L19`（paimon metrics 悬空、iceberg 缓存/version-blind/ -未知类型、partition_columns 撞名）。逐条走单任务循环（复核现码→设计→红队→实现→build+UT→独立 commit→勾表→更 HANDOFF）。 +**批次 4 已完成子群**：trino `L3–L6` ✅ · kerberos `L7/L8` ✅ · mc `L9` ✅ · **paimon `L11/L13/L14` ✅（本轮）**。 +- **paimon 子群（L11/L13/L14）**：统一设计红队 `wf_05574ccb-bd2`（3 设计 × 3 lens = 9 agent,无 UNSOUND）。 + - `L11` `4a8650bd062` — JNI/COUNT range file_format 按首数据文件后缀取(legacy `getFileFormat(getPathString())` parity)+补 `.avro` 臂; + call-site RED 测(`Table.copy` 令表默认≠磁盘后缀,红队 MAJOR:原 helper 孤立测不守护接线)。 + - `L13` `ced4775b844` — `toPaimonType` 嵌套 nullability `.copy(isChildNullable)`(ARRAY/MAP-value/STRUCT-field); + **scope 仅 nullability**(comment=DV-035 M10.1 已接受、field-id 顺序 parity);`.copy(true)` 默认恒等,既有 parity 测保持绿。 + - `L14` `478718aca6f` — honor `ignore_split_type`(null-tolerant `resolveIgnoreSplitType` + 三 legacy continue 位; + `IGNORE_PAIMON_CPP` no-op=legacy parity,全树 grep 证);nonDataSplit IGNORE_JNI 位 E2E-only。 + - **⚠ 构建坑复现**：paimon 模块 `mvn test` 因 hive-shade 模块 shade 绑 `package` 阶段→`org.apache.hadoop.hive.conf` NoClassDefFound + **假失败**;改 `package` 阶段即绿。模块靶向 UT 全绿(scan 69/69 + type-mapping/schema-builder 26/26)、0 checkstyle、gate 净。 + +**下一步 = 批次 4 剩余（低危连接器/杂项）**：`L15–L19`（`L15` paimon `PAIMON_SCAN_METRICS` 悬空常量、`L16/L17` iceberg +快照/schema 缓存偏斜 + version-blind schema 绑定、`L18` iceberg 未知/v3 类型静默 UNSUPPORTED、`L19` `partition_columns` 魔法键撞名）。 +逐条走单任务循环（复核现码→设计→红队→实现→build+UT→独立 commit→勾表→更 HANDOFF）。 ⏸ **决策类 `L2/L10/L12/L20` 先中文讲清背景+选项问用户再动**（memory `ask-user-explain-in-chinese-first`）。设计债 D-系列择机。 跟踪表「建议批次」节有全清单。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 9c9410bdd8afba..32b9c69aa0d795 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -47,10 +47,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 20 | **L8** | 🟡低 | kerberos | doAs 吞 interrupt 不 restore | ⬜ | ⬜ | ⬜ | ⬜ | | 21 | **L9** | 🟡低 | maxcompute | 谓词下推全有全无 | ⬜ | ⬜ | ⬜ | ⬜ | | 22 | **L10** | 🟡低 | fe-core | EXPLAIN 节点名 VPluginDrivenScanNode | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | -| 23 | **L11** | 🟡低 | paimon | JNI/COUNT file_format 用表级默认 | ⬜ | ⬜ | ⬜ | ⬜ | +| 23 | **L11** | 🟡低 | paimon | JNI/COUNT file_format 用表级默认 | ✅ | ✅ | ✅ | ✅ | | 24 | **L12** | 🟡低 | fe-core/paimon | selectedPartitionNum 语义(登记或对齐) | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | -| 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ⬜ | ⬜ | ⬜ | ⬜ | -| 26 | **L14** | 🟡低 | paimon | ignore_split_type 静默 no-op | ⬜ | ⬜ | ⬜ | ⬜ | +| 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ✅ | ✅ | ✅ | ✅ | +| 26 | **L14** | 🟡低 | paimon | ignore_split_type 静默 no-op | ✅ | ✅ | ✅ | ✅ | | 27 | **L15** | 🟡低 | fe-core | PAIMON_SCAN_METRICS 悬空常量 | ⬜ | ⬜ | ⬜ | ⬜ | | 28 | **L16** | 🟡低 | iceberg | 快照/schema 缓存偏斜(防御性 union) | ⬜ | ⬜ | ⬜ | ⬜ | | 29 | **L17** | 🟡低 | fe-core/iceberg | 同表多版本 version-blind schema 绑定 | ⬜ | ⬜ | ⬜ | ⬜ | @@ -188,10 +188,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L9** DONE `017d1af1894`(设计 `FIX-L9-design.md`;converter 纯函数,**RED-able UT**）· `convert` 特判根 `ConnectorAnd`:逐 top-level conjunct 独立转(抽 `convertOne`),丢+log 失败者、AND 幸存者;OR/NOT/嵌套 AND 保持全有全无(只根 AND 处正/单调位置丢 conjunct=超集,BE 重滤;OR 丢 disjunct=子集会漏行)。加 5 UT(3 RED-able+2 guard),21/21 绿。perf-only。 - [ ] **L10** ⏸ EXPLAIN 节点名 · `PluginDrivenScanNode.java:170,320`。**需决策**:登记 display-only 验收偏差(cheapest,`CONNECTOR:` 行已披露)**vs** 加连接器声明的 legacy `*_SCAN_NODE` 名(连 `Connector.getLegacyEngineName` 一起,见 D-ENGINE)。注:报告的 `connectorType.toUpperCase()+"_SCAN_NODE"` 一行修法**不够**(hms 会出 `VHMS_SCAN_NODE` 非 `VHIVE_/VICEBERG_`)。 -- [ ] **L11** paimon file_format · `PaimonScanPlanProvider.java:812-813,848-849`:JNI DataSplit + COUNT 路按首数据文件后缀 `getFileFormatBySuffix("/"+dataFiles().get(0).fileName()).orElse(defaultFileFormat)`(仿 native 臂 `:540`/legacy)。注:默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。 +- [x] **L11** DONE `4a8650bd062`(设计 `FIX-L11-design.md`;红队 `wf_05574ccb-bd2` 3 lens SOUND/SOUND_WITH_CHANGES）· JNI DataSplit + COUNT 路改按**首数据文件后缀**取 file_format(新 package-private `dataSplitFileFormat`,legacy `getFileFormat(getPathString())` parity),回退表级默认;顺补 `getFileFormatBySuffix` 的 `.avro` 臂(legacy `FileFormatUtils` parity,inert on native)。**call-site RED 测**(`Table.copy` 令表默认 parquet≠磁盘 .orc,断言 JNI+COUNT range 携 "orc")——原 helper 孤立测不守护接线(红队 MAJOR)。69/69 绿、0 checkstyle、gate 净。默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。e2e live-gated(cpp reader over 混/改格式表)。 - [ ] **L12** ⏸ selectedPartitionNum 语义 · `PluginDrivenScanNode.java:297-303`。**需决策**:登记为「paimon/iceberg 对齐 MC/hive 的 Nereids-剪枝数」验收偏差(推荐)**vs** 连接器回报 SDK-distinct(重,不推荐)。同步 paimon/iceberg EXPLAIN `partition=N/M` 回归期望。**勿**在通用节点按源分支。 -- [ ] **L13** paimon 嵌套 nullability · `PaimonTypeMapping.java:254,257-259,269-281`:发 4-arg `DataField(id,name,type.copy(isChildNullable),comment)`;ARRAY 元素/MAP value `.copy(...)`。仿 `IcebergSchemaBuilder`/legacy。comment 半已登记 DV-035c。 -- [ ] **L14** paimon ignore_split_type no-op · `PaimonScanPlanProvider.java:407-451`:读 `ignore_split_type`(经 `ConnectorSession.getSessionProperties()`)按被忽略 reader 类型跳 split;或退休变量+登记。 +- [x] **L13** DONE `ced4775b844`(设计 `FIX-L13-design.md`;红队 3 lens 全 SOUND）· `toPaimonType` 对 ARRAY 元素/MAP value/STRUCT 字段加 `.copy(type.isChildNullable(i))`(MAP key 保持 `.copy(false)`),恢复 legacy `DorisToPaimonTypeVisitor` 嵌套 nullability parity。**scope=仅 nullability**:comment 丢=DV-035 M10.1 已接受偏差、field-id 顺序 parity 均不动。`.copy(true)` 对默认可空子类型逐字节恒等(既有 parity 测保持绿)。3 新 RED-able 测(经 `.type().isNullable()` 断言,非 DataField equals)。type-mapping+schema-builder 26/26 绿。e2e live-gated(建嵌套 NOT NULL 表 DESCRIBE/SDK 读回)。 +- [x] **L14** DONE `478718aca6f`(设计 `FIX-L14-design.md`;红队 3 lens SOUND/SOUND_WITH_CHANGES）· null-tolerant `resolveIgnoreSplitType(session)`(镜像 `isCppReaderEnabled`,红队 MINOR)+三 legacy `continue` 位:`IGNORE_JNI` drops nonDataSplit+DataSplit-JNI 臂、`IGNORE_NATIVE` drops native 臂、count 臂不检、`IGNORE_PAIMON_CPP` 保持 no-op(=legacy parity,全树 grep 证 legacy 从不引用)。默认 NONE 逐字节不变。3 新 live-planScan RED 测(IGNORE_JNI/IGNORE_NATIVE 各证跳 split+IGNORE_PAIMON_CPP==NONE);nonDataSplit IGNORE_JNI 位离线不可测→留 E2E-only。69/69 绿。e2e live-gated(真集群 SET ignore_split_type 断言跳分片)。 - [ ] **L15** PAIMON_SCAN_METRICS 悬空 · `SummaryProfile.java:158,218,277`:删三处死引用(P5 已验收弃 paimon FE scan metrics);或加 connector-neutral scan-metrics SPI(feature,非必需)。 - [ ] **L16** iceberg 缓存偏斜(部分已修) · `IcebergScanPlanProvider.java:1077-1108`:hasSnapshotPin 臂把 field-id dict 建成「pinned schema ∪ requested latest columns」超集(传 requestedLowerNames 而非 `emptyList`),两向都超集;或 fe-core 侧 query-begin pin 解析 pinnedSchema 到 pinned schemaId(对齐 time-travel 臂 `:377-387`)。 - [ ] **L17** iceberg 同表多版本 version-blind 绑定 · `PluginDrivenMvccExternalTable.java:475-485`:per-reference schema 绑定 version-aware(用 `StatementContext.getSnapshot(table, tableSnapshot, scanParams)` 的 pinnedSchema,fallback latest);与 L16 同根,建议一并。窄触发 + fail-loud。 @@ -286,4 +286,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - fe-kerberos UT 11/11 绿(含 AuthenticationTest);L7 e2e live-gated(双 kerberos HDFS 目录)。 **⭐ 批次 4·maxcompute L9 DONE** `017d1af1894` — 谓词下推从「全有全无」改为顶层 AND 逐 conjunct 容错(超集正确、BE 重滤)。**首个有 RED-able UT 的 L 条**(converter 纯函数):5 新 UT(3 RED-able+2 guard,证 OR/嵌套 AND 不容错),21/21 绿。perf-only。 -- **下一步 = paimon 子群 L11/L13/L14,再 iceberg/杂项 L15–L19**;⏸ 决策类 L2/L10/L12/L20 先问用户。 + +**⭐ 批次 4·paimon 子群(L11/L13/L14)全 DONE**（fe-connector-paimon 局部,各独立 code commit + 统一设计红队 `wf_05574ccb-bd2`：3 设计 × 3 lens = 9 agent,无 UNSOUND;设计文档 `FIX-L11/L13/L14-design.md` 含红队折入）： +- **L11** `4a8650bd062` — JNI/COUNT range file_format 改按首数据文件后缀取(legacy `getFileFormat(getPathString())` parity),补 `.avro` 臂;call-site RED 测(`Table.copy` 令表默认≠磁盘后缀,红队 MAJOR)。 +- **L13** `ced4775b844` — `toPaimonType` 嵌套 nullability `.copy(isChildNullable)`(ARRAY/MAP-value/STRUCT-field),scope 仅 nullability(comment=DV-035 M10.1 已接受);`.copy(true)` 默认恒等。 +- **L14** `478718aca6f` — honor `ignore_split_type`(null-tolerant helper + 三 legacy continue 位;`IGNORE_PAIMON_CPP` no-op=legacy parity);3 live-planScan RED 测,nonDataSplit 位 E2E-only。 +- **共性**:全走单任务循环(复核现码→设计→红队→实现→build+靶向 UT→独立 commit)。**paimon 构建坑复现**:`mvn test` 因 hive-shade 模块 shade 绑 package→`HiveConf` NoClassDefFound 假失败,改 `package` 阶段即绿(memory `doris-build-verify-gotchas`)。模块靶向 UT 全绿(scan 69/69 + type-mapping/schema-builder 26/26)、0 checkstyle、import gate 净。**e2e 全 live-gated**(cpp reader 混格式 / 嵌套 NOT NULL DDL / SET ignore_split_type 跳分片)。 +- **下一步 = iceberg/杂项 L15–L19**(L15 paimon-metrics 悬空、L16/L17 iceberg 缓存/version-blind、L18 iceberg 未知类型、L19 partition_columns 撞名);⏸ 决策类 L2/L10/L12/L20 先中文讲清背景问用户。 diff --git a/plan-doc/tasks/designs/FIX-L11-design.md b/plan-doc/tasks/designs/FIX-L11-design.md new file mode 100644 index 00000000000000..fdcc746c615c8d --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L11-design.md @@ -0,0 +1,108 @@ +# FIX-L11 — paimon JNI/COUNT range uses table-level default file format, not per-file suffix + +> reverify §1 L11 (P5-2). 严重度 🟡低。模块 = fe-connector-paimon（连接器局部，不碰 fe-core）。 + +## Problem + +翻闸后的 paimon 连接器在为 **JNI 序列化的 DataSplit** 和 **COUNT(*) 折叠 range** 打 `file_format` 时， +用的是**表级默认** `table.options().file.format`（缺省 `parquet`），而不是**该 split 首个数据文件的实际后缀**。 +当表的当前 `file.format` 选项与磁盘上历史数据文件的实际格式不一致时（例如表先以 orc 写入、后 `ALTER` 改 +选项为 parquet；或混格式表），FE 会给 BE 发**错误**的 `file_format`。 + +影响面窄：默认 JNI reader 不消费 `file_format`（JNI 路由由 `paimon.split` 属性决定）；只有 opt-in 的 +paimon-cpp reader 会把该字段回填进 `FILE_FORMAT/MANIFEST_FORMAT`，错值会破坏它的 manifest 读。故为 🟡低。 + +## HEAD recon（对 HEAD `ca77ea774f7` 复核，行号已核） + +- `PaimonScanPlanProvider.java:411-412` — `defaultFileFormat = table.options().getOrDefault(FILE_FORMAT.key(), "parquet")`。 +- `:447` — nonDataSplits 臂 `buildJniScanRange(..., defaultFileFormat, ..., isDataSplit=false, ...)`。 +- `:504-505` — native 臂 `buildNativeRanges(...)` → `buildNativeRange:540` 已经用 + `getFileFormatBySuffix(file.path()).orElse(defaultFileFormat)`（**native 臂已正确**）。 +- `:509-511` — DataSplit-forced-JNI 臂 `buildJniScanRange(dataSplit, ..., defaultFileFormat, ..., isDataSplit=true, ...)`。 +- `:520-521` — COUNT 折叠臂 `buildCountRange(countRepresentative, ..., defaultFileFormat, ...)`。 +- `buildJniScanRange:795-823` — `.fileFormat(defaultFileFormat)`（硬用表级默认）。 +- `buildCountRange:843-860` — `.fileFormat(defaultFileFormat)`（硬用表级默认）。 +- `getFileFormatBySuffix:1182-1193` — 私有 static，`.orc`→orc、`.parquet`/`.parq`→parquet、否则 empty。 + +## Legacy parity target（legacy 已删，取 git `dbc38a265e5^`） + +- `PaimonScanNode.setPaimonParams:268` — `String fileFormat = getFileFormat(paimonSplit.getPathString());`， + **所有臂**（JNI `split!=null` 与 native `split==null`）都用它，末尾 `fileDesc.setFileFormat(fileFormat)`。 +- `PaimonScanNode.getFileFormat:645-646` — `FileFormatUtils.getFileFormatBySuffix(path).orElse(source.getFileFormatFromTableProperties())`。 +- `PaimonSplit`（ctor）— DataSplit 的 `path = LocationPath.of("/" + dataFiles().get(0).fileName())`； + 非 DataSplit 无文件路径 → `getFileFormatBySuffix` empty → 回退表级默认。 + +结论：legacy 对 **DataSplit** 按**首数据文件后缀**取格式（回退表级默认）；对**非 DataSplit** 天然回退表级默认。 + +## Design（surgical，仿 native 臂 `:540` + legacy） + +新增一个私有 static 助手（对齐已抽出的 `isCountPushdownSplit`/`computeFileSplitOffsets`/`encodeSplit` 风格， +便于单测）： + +```java +/** DataSplit 的实际数据文件格式：取首个数据文件后缀（legacy PaimonSplit path = "/"+dataFiles().get(0).fileName()）， + * 回退表级默认。空文件列表（不应发生）时 fail-safe 回退默认。 */ +private static String dataSplitFileFormat(DataSplit dataSplit, String defaultFileFormat) { + List files = dataSplit.dataFiles(); + if (files == null || files.isEmpty()) { + return defaultFileFormat; + } + return getFileFormatBySuffix("/" + files.get(0).fileName()).orElse(defaultFileFormat); +} +``` + +改两处发射点，只对 DataSplit 生效： + +1. `buildJniScanRange`：把 `.fileFormat(defaultFileFormat)` 改为 + ```java + String fileFormat = isDataSplit + ? dataSplitFileFormat((DataSplit) split, defaultFileFormat) + : defaultFileFormat; + ... + .fileFormat(fileFormat) + ``` + （非 DataSplit 保持 `defaultFileFormat` = legacy parity；`(DataSplit) split` 的 cast 与本方法既有的 + `computeSplitWeight((DataSplit) split)`（isDataSplit 臂）同一守卫，安全。） + +2. `buildCountRange`（参数恒为 DataSplit）：`.fileFormat(dataSplitFileFormat(dataSplit, defaultFileFormat))`。 + +注释更新：把两处 `FIX-JNI-FILE-FORMAT` 注释从「emit real data-file format ... 目前用 defaultFileFormat」 +更正为「按首数据文件后缀取（legacy getFileFormat(getPathString())），回退表级默认」。 + +**不动**：native 臂（已正确）、nonDataSplits 臂（无文件、legacy 亦回退默认）、`getFileFormatBySuffix`。 + +## Risk + +- Cast 安全：`buildJniScanRange` 仅在 `isDataSplit=true` 时 cast，与既有 weight 逻辑同守卫。 +- 空 `dataFiles()`：DataSplit 恒 ≥1 数据文件；仍加 fail-safe 守卫回退默认（比 legacy 更稳，happy-path 不变）。 +- 多文件格式不一：legacy 只看**首**文件；本 fix 同（同一 DataSplit 内文件同格式是 paimon 不变式）。 +- 行为向后兼容：当表默认 == 文件实际格式（绝大多数表），输出逐字节不变。 + +## Test Plan + +### Unit（RED-able）— `PaimonScanPlanProviderTest` + +- **新** `dataSplitFileFormatUsesFileSuffixOverTableDefault`：用 `buildRealDataSplit` 变体建 **orc** 数据文件的 + DataSplit（`.option("file.format","orc")`），断言 `dataSplitFileFormat(split, "parquet") == "orc"` 且 + `dataSplitFileFormat(split, "zzz") == "orc"`。**MUTATION**：助手直接返回 `defaultFileFormat` → 返回 "parquet"/"zzz" → RED。 + （为此把 `dataSplitFileFormat` 设为 package-private static。） +- 现有 `...RealFileFormat`（~933/960）：表默认与文件同为 orc，本 fix 后仍 == "orc"，**回归守卫**（保持绿）。 + +### E2E（live-gated，登记） + +paimon-cpp reader（`enable_paimon_cpp_reader=true`）over 一张 `file.format` 与历史文件格式不一致的表： +断言 cpp 读不因 `file_format` 错值失败。默认 JNI reader 不受影响。→ 真集群回归（memory `hms-iceberg-delegation-needs-e2e`）。 + +--- + +## 设计红队结论（`wf_05574ccb-bd2`，3 lens · 全 SOUND / SOUND_WITH_CHANGES，无 UNSOUND） + +- **MAJOR（test-build）已折入**：原「helper 孤立单测」不守护**接线**（emission point 仍发 defaultFileFormat 也能过）—— + 现有 `jniAndCountRangesCarryRealFileFormatNotJni` 表默认==后缀==orc,无法区分。**加 call-site RED 测** + `jniAndCountRangesUseFileSuffixNotAlteredTableDefault`：`Table.copy(file.format=parquet)` 令表默认=parquet 而磁盘文件仍 .orc, + 断言 JNI + COUNT range 均携 "orc"(后缀)——对 pre-fix `.fileFormat(defaultFileFormat)`(=parquet)必 RED。 +- **MINOR（legacy-parity）已折入**：连接器 `getFileFormatBySuffix` 缺 legacy `FileFormatUtils` 的 `.avro` 臂→avro 数据文件 + 在表默认≠avro 时偏离 legacy。**加 `.avro` 臂**(仅 native 臂+新 helper 调用;avro 永不到 native 臂,inert)。保留既有 `.parq`(非 L11 范围)。 +- **MINOR（test-build）已折入**：helper 须 **package-private static**(非 `private`,否则同包测试不可见)——已按 package-private 实现。 +- 三 lens 均确认:cast 受 `isDataSplit` 守卫、空 `dataFiles()` fail-safe 回退默认(比 legacy 更稳)、默认路径逐字节不变、 + RED-able 经 paimon 1.3.1 字节码证实(orc/parquet 数据文件名恒以 `.orc`/`.parquet` 结尾)。 diff --git a/plan-doc/tasks/designs/FIX-L13-design.md b/plan-doc/tasks/designs/FIX-L13-design.md new file mode 100644 index 00000000000000..0632641df8dd3a --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L13-design.md @@ -0,0 +1,102 @@ +# FIX-L13 — Doris→Paimon 建表丢嵌套 nullability + +> reverify §1 L13 (P5-4)。严重度 🟡低。模块 = fe-connector-paimon（连接器局部）。 + +## Problem + +`CREATE TABLE` / CTAS 建 paimon 表时，Doris 列类型经 `PaimonTypeMapping.toPaimonType` 反向映射为 paimon +`DataType`。对**嵌套子类型的 nullability**——ARRAY 元素、MAP value、STRUCT 字段——当前**一律丢弃**声明的 +`NOT NULL`，落盘为可空。这让 `ARRAY` 建成 `ARRAY`（元素可空），物理 schema 与 DDL 不符。 + +## Scope 界定（重要） + +reverify 标题写「发 4-arg `DataField(id,name,type.copy(isChildNullable),comment)`」，但**comment 半已登记**： +`deviations-log` **DV-035 §(c) M10.1**（用户 D-057 签字，2026-06-12）已把「CREATE 嵌套 struct comment 丢」 +**接受为 display-only 偏差**。故本条 scope = **仅嵌套 nullability**，**不含 comment**（加 comment 会重开已接受偏差， +且非本 bug）。field-id 亦不改：legacy `toPaimonRowType` 用 `new AtomicInteger(-1).incrementAndGet()` 顺序 id， +现码同（`fieldId.incrementAndGet()`），保持 parity。 + +## HEAD recon（对 HEAD 复核） + +`PaimonTypeMapping.java`： +- `:253-254` ARRAY — `new ArrayType(toPaimonType(children.get(0)))`：**元素 nullability 丢**。 +- `:255-259` MAP — key 已 `.copy(false)`（legacy 强制非空，正确保留）；value + `toPaimonType(children.get(1))`：**value nullability 丢**。 +- `:269-281` `toPaimonRowType` — `new DataField(id, name, toPaimonType(children.get(i)))`：**字段 nullability 丢**。 + +`ConnectorType`（fe-connector-api）**已具备**所需输入（无需 SPI 加法）： +- `isChildNullable(int index)` — 未携带时默认 `true`（`:242-244`）。 +- `getChildrenNullable()`、工厂 `arrayOf(elem, elementNullable)` / `mapOf(k,v,valueNullable)` / + `structOf(names, types, fieldNullable, comments)` 均已按索引编码 nullability。 +- paimon `DataType.copy(boolean isNullable)` 返回同类型改 nullability（现码 MAP key `.copy(false)` 已用）。 + +## Legacy parity target + +legacy `DorisToPaimonTypeVisitor`/`PaimonUtil`（已删）：ARRAY/MAP/STRUCT 子类型经 `.copy(isNullable)` 保留声明 +nullability，MAP key `.copy(false)`。iceberg 兄弟 `IcebergSchemaBuilder` 亦按 required/optional 保留嵌套 +nullability（本条参照其模式）。 + +## Design（surgical，纯 `.copy(isChildNullable)`） + +```java +case "ARRAY": + return new ArrayType( + toPaimonType(type.getChildren().get(0)).copy(type.isChildNullable(0))); +case "MAP": + // key 强制非空（legacy .copy(false)）；value 保留声明 nullability。 + return new MapType( + toPaimonType(type.getChildren().get(0)).copy(false), + toPaimonType(type.getChildren().get(1)).copy(type.isChildNullable(1))); +``` + +`toPaimonRowType`： +```java +fields.add(new DataField( + fieldId.incrementAndGet(), fieldName, + toPaimonType(children.get(i)).copy(type.isChildNullable(i)))); +``` + +- 递归语义正确：子的**深层**嵌套 nullability 由递归调用处理；子在**本层**的 nullability 由本层 + `.copy(isChildNullable(i))` 施加（nullability 是父对子的视图属性，存于父的 `childrenNullable`）。 +- 顶层列 nullability（列 `NOT NULL`）由 `PaimonSchemaBuilder` 顶层 RowType 构造处理，**不在本条 scope**。 +- 默认行为兼容：未携带 nullability 时 `isChildNullable` 默认 `true` → `.copy(true)`；paimon 类型缺省即可空， + 故对未声明 NOT NULL 的旧路径**逐字节不变**。 + +**不改**：comment（DV-035 M10.1 已接受）、field-id（顺序 id parity）、scalar 臂、char/decimal/timestamp 臂。 + +## Risk + +- `.copy(true)` 是否与「原本就可空」逐字节等价？paimon `DataType` 默认 `isNullable=true`，`toPaimonType` + 返回的子类型默认可空 → `.copy(true)` 是恒等。红队须确认无 `equals`/序列化差异（用现有 parity 测兜底）。 +- MAP key 保持 `.copy(false)`：不回退既有正确行为。 + +## Test Plan + +### Unit（RED-able）— `PaimonTypeMappingToPaimonTest` + +- **新** `nestedNullabilityPreservedForArrayElement`：`ConnectorType.arrayOf(ConnectorType.of("INT"), false)` → + 断言结果 `ArrayType` 元素类型 `isNullable()==false`。MUTATION：丢 `.copy` → 元素可空 → RED。 +- **新** `nestedNullabilityPreservedForMapValue`：`ConnectorType.mapOf(key, value, /*valueNullable*/ false)` → + value 非空 + key 恒非空。MUTATION：value 丢 `.copy` → RED。 +- **新** `nestedNullabilityPreservedForStructField`：`ConnectorType.structOf(names, types, [false,true], comments)` → + 第 0 字段非空、第 1 可空。MUTATION：DataField 丢 `.copy` → RED。 +- **回归守卫**：未声明 nullability 的既有嵌套 parity 测（若有）保持绿（默认 `.copy(true)` 恒等）。 + +### E2E（live-gated，登记） + +`CREATE TABLE ... (a ARRAY, m MAP, s STRUCT)` on paimon +catalog → `DESCRIBE`/paimon SDK 读回 schema，断言嵌套非空落盘。→ 真集群回归。 + +--- + +## 设计红队结论（`wf_05574ccb-bd2`，3 lens · 全 SOUND，无 changes-required） + +- **legacy-parity SOUND**：逐一核对 legacy `DorisToPaimonTypeVisitor`:array=`elementResult.copy(array.getContainsNull())`、 + map=`key.copy(false)+value.copy(getIsValueContainsNull())`、struct=`fieldResults.get(i).copy(field.getContainsNull())`—— + 本 fix 经 `type.isChildNullable(idx)` 逐一镜像。CREATE 桥 `ConnectorColumnConverter.toConnectorType` 从**同** Doris 源 + (`getContainsNull`/`getIsValueContainsNull`)填 `childrenNullable`→本 fix **有效非惰性**。comment 丢=DV-035 M10.1 已接受(确认 scope)。 +- **correctness SOUND**（关键风险已证真）：paimon 1.3.1 字节码证 `DataType.copy(boolean)` 每子类型均有;`equals/hashCode/serializeJson` + 均折入 `isNullable`→`.copy(true)` 对默认可空子类型**逐字节等价**(恒等)。顶层列 `PaimonSchemaBuilder:127` `.copy(col.isNullable())` + 不 clobber 嵌套(ArrayType.copy 保留 elementType)。 +- **MINOR（test）已折入**：新 struct 测须经 `field.type().isNullable()` 断言(**非** DataField equals/description——comment 丢致 + description=null,equals 会因无关原因假 RED)。已按 `.type().isNullable()` 写。 diff --git a/plan-doc/tasks/designs/FIX-L14-design.md b/plan-doc/tasks/designs/FIX-L14-design.md new file mode 100644 index 00000000000000..07130d36c8f800 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L14-design.md @@ -0,0 +1,98 @@ +# FIX-L14 — paimon `ignore_split_type` session 变量插件路径静默 no-op + +> reverify §1 L14 (P5-5)。严重度 🟡低。模块 = fe-connector-paimon（连接器局部）。 + +## Problem + +`ignore_split_type`（fe-core `SessionVariable`，选项 `NONE`/`IGNORE_JNI`/`IGNORE_NATIVE`/`IGNORE_PAIMON_CPP`） +是一个**调试/隔离 reader-bug 的逃生阀**：设为 `IGNORE_JNI` 应跳过所有 JNI split、`IGNORE_NATIVE` 应跳过所有 +native split。翻闸后的 paimon 连接器**完全不读**该变量 → 无论设何值都照常发全部 split（静默 no-op）。 + +DV-035 §(h) M1.1 曾把它列为 diagnostic 偏差，但注为「须 fe-core SessionVariable 类型」；实为**可在连接器侧就地 +恢复 parity**（变量已经 `session.getSessionProperties()` 暴露为字符串，无需 import fe-core 类型）。 + +## HEAD recon（对 HEAD 复核） + +`PaimonScanPlanProvider.planScan`： +- `:428` 已读 `cppReader = isCppReaderEnabled(session)`（同一 `session.getSessionProperties()` 通道）。 +- `:446-449` nonDataSplits 臂 → `buildJniScanRange`（这些恒 JNI）。 +- `:470-513` DataSplit for-loop： + - `:471-477` count 臂（`isCountPushdownSplit`）→ `continue`（**count 不受 ignore 影响**，legacy 同）。 + - `:485-506` native 臂（`shouldUseNativeReader`）。 + - `:507-512` else = JNI 臂。 +- 全文件无 `ignore_split_type` 引用（grep 证实）。 + +## Legacy parity target（git `dbc38a265e5^` `PaimonScanNode.getSplits`） + +- `:380-381` 读 `IgnoreSplitType.valueOf(sessionVariable.getIgnoreSplitType())`。 +- `:401` nonDataSplits：`if (IGNORE_JNI) continue;`。 +- `:443` native 臂：`if (IGNORE_NATIVE) continue;`。 +- `:483` DataSplit JNI else 臂：`if (IGNORE_JNI) continue;`。 +- count 臂：**不**检查 ignore（never ignored）。 +- **`IGNORE_PAIMON_CPP` 在 legacy `getSplits` 从不被引用 → 本身即 no-op**（grep 全 legacy 文件确认仅 JNI/NATIVE）。 + +结论：parity = 实现 `IGNORE_JNI`（nonDataSplit + DataSplit-JNI）与 `IGNORE_NATIVE`（native 臂）； +`IGNORE_PAIMON_CPP` 保持 no-op（**= legacy parity**，非新缺口），文档/注释显式登记。 + +## Design（surgical，仿 legacy `continue`） + +常量： +```java +private static final String IGNORE_SPLIT_TYPE = "ignore_split_type"; +private static final String IGNORE_SPLIT_TYPE_JNI = "IGNORE_JNI"; +private static final String IGNORE_SPLIT_TYPE_NATIVE = "IGNORE_NATIVE"; +``` + +`planScan` 内（`cppReader` 读取旁）读一次： +```java +String ignoreSplitType = session.getSessionProperties() + .getOrDefault(IGNORE_SPLIT_TYPE, "NONE"); +boolean ignoreJni = IGNORE_SPLIT_TYPE_JNI.equals(ignoreSplitType); +boolean ignoreNative = IGNORE_SPLIT_TYPE_NATIVE.equals(ignoreSplitType); +// IGNORE_PAIMON_CPP 刻意不实现：legacy PaimonScanNode.getSplits 亦从不引用它（parity no-op）。 +``` + +三处 `continue`（严格对齐 legacy 位置）： +- nonDataSplits 循环顶：`if (ignoreJni) { continue; }`。 +- native 臂（`if (shouldUseNativeReader...)` 内首行）：`if (ignoreNative) { continue; }`。 +- JNI else 臂首行：`if (ignoreJni) { continue; }`。 + +count 臂**不加**（parity）。 + +## Risk + +- 语义 = 用户显式设的调试变量：跳过 split ⇒ 丢行是**故意**的隔离行为（legacy 同）。非默认路径（默认 `NONE` + 无行为变更，逐字节不变）。 +- `getOrDefault` 兜底 `"NONE"`：session 未设时 ignoreJni/ignoreNative 皆 false → 全部 split 照发。 +- 大小写：SessionVariable checker 只允许枚举大写值；`.equals` 精确匹配即可（legacy `valueOf` 亦大小写敏感）。 + +## Test Plan + +### Unit（RED-able）— `PaimonScanPlanProviderTest`（复用 `buildRealDataSplit`/`RecordingPaimonCatalogOps`/`sessionWithProps`） + +- **新** `ignoreJniSkipsForcedJniDataSplit`：`force_jni_scanner=true`（DataSplit→JNI）+ + `ignore_split_type=IGNORE_JNI` → 断言无 `paimon.split` JNI range（数据 range 为空）。MUTATION：不实现 → 仍发 1 → RED。 +- **新** `ignoreNativeSkipsNativeDataSplit`：native 路径（无 force_jni）+ `ignore_split_type=IGNORE_NATIVE` → + 断言无 native 数据 range。MUTATION：不实现 → 仍发 → RED。 +- **新** `ignorePaimonCppIsNoOpParity`：`ignore_split_type=IGNORE_PAIMON_CPP` → range 集与 `NONE` 相同 + （钉 legacy parity no-op，防未来误加半套）。 +- **guard** `ignoreNoneEmitsAllSplits`：`NONE`/未设 → range 照常（baseline）。 + +### E2E（live-gated，登记） + +真集群设 `SET ignore_split_type='IGNORE_JNI'` / `'IGNORE_NATIVE'`，对含两类 split 的 paimon 表 scan， +断言对应类型 split 被跳过（行数变化符合预期）。→ 真集群回归。 + +--- + +## 设计红队结论（`wf_05574ccb-bd2`，3 lens · SOUND / SOUND_WITH_CHANGES，无 UNSOUND） + +- **legacy-parity SOUND**：全 legacy 文件 + 全 legacy 树 grep 证 `IGNORE_PAIMON_CPP` **从不**被任何 scan 路引用 + (仅 enum/checker/错误串)→no-op 是**真 legacy parity**非新缺口。三 `continue` 位 1:1 对齐 legacy(:401/:443/:483)、count 臂不检查。 +- **MINOR（null-guard，legacy-parity+correctness 双 lens 同报）已折入**：内联 `session.getSessionProperties()` 缺 `session==null` + 守卫(本文件所有 session 读取器 `isCppReaderEnabled`/`isForceJniScannerEnabled`/`sessionLong` 均先 null-guard)→纯 nonDataSplit/空 scan + 的 null-session 路径会 NPE(HEAD 现容忍)。**加 null-tolerant `resolveIgnoreSplitType(ConnectorSession)`**(null→"NONE"),镜像 `isCppReaderEnabled`。 +- **MINOR（test-build）已折入**：RED 测**不能**用 detached `buildRealDataSplit`(catalog 关闭后 split 游离,仅静态 helper 可用); + 须走 live `ops.table=table + provider.planScan(...)` 端到端;`ignoreJniDropsForcedJniSplit` 需 2-entry props→用 `new HashMap<>()`。已照办。 +- **MINOR（覆盖缺口，已登记）**：nonDataSplits 的 `IGNORE_JNI continue` 离线不可单测(planScan 枚举造不出 system-table 非-DataSplit); + 经 IGNORE_JNI(DataSplit else 臂)+IGNORE_NATIVE(native 臂)+IGNORE_PAIMON_CPP(no-op) 三测覆盖两 continue 位,nonDataSplit 位留 **E2E-only**。 From a691d0264f58055375faebe0fadf56e2e6bbaefb Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:27:55 +0800 Subject: [PATCH 315/334] [doc](catalog) #65185 reverify: sync progress table for L3-L9 (already done) The detailed L-list already recorded L3-L9 as DONE with commits, but the top progress table still showed them todo. Confirmed all 7 commits exist in the log (trino 4cd63c6911a/e27602d4ab6/be96adf76ba/18048f7f217, kerberos 9e4f2992382/59697ce3fc7, maxcompute 017d1af1894); mark them done in the table. L2 and L10 remain decision-type (no commit yet). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/task-list-65185-reverify-fixes.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 32b9c69aa0d795..969bdfeed51f40 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -39,13 +39,13 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⏸ 跳过(用户 07-12;侦察见下) | | 13 | **L1** | 🟡低 | tools | import-gate 三洞**+第4洞** | ✅ | ✅ | ✅ | ✅ | | 14 | **L2** | 🟡低 | fe-core | 翻闸 hive 丢 SQL 缓存资格 + COUNTER 停增 | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | -| 15 | **L3** | 🟡低 | trino | 元数据事务从不 commit/close | ⬜ | ⬜ | ⬜ | ⬜ | -| 16 | **L4** | 🟡低 | trino | plugin.dir 首胜单例(fail-loud) | ⬜ | ⬜ | ⬜ | ⬜ | -| 17 | **L5** | 🟡低 | trino | listTableNames 丢去重 | ⬜ | ⬜ | ⬜ | ⬜ | -| 18 | **L6** | 🟡低 | trino | guard 字段发布顺序→瞬时 NPE | ⬜ | ⬜ | ⬜ | ⬜ | -| 19 | **L7** | 🟡低 | kerberos | UGI.setConfiguration 无 guard(丢 first-writer) | ⬜ | ⬜ | ⬜ | ⬜ | -| 20 | **L8** | 🟡低 | kerberos | doAs 吞 interrupt 不 restore | ⬜ | ⬜ | ⬜ | ⬜ | -| 21 | **L9** | 🟡低 | maxcompute | 谓词下推全有全无 | ⬜ | ⬜ | ⬜ | ⬜ | +| 15 | **L3** | 🟡低 | trino | 元数据事务从不 commit/close | ✅ | ✅ | ✅ | ✅ | +| 16 | **L4** | 🟡低 | trino | plugin.dir 首胜单例(fail-loud) | ✅ | ✅ | ✅ | ✅ | +| 17 | **L5** | 🟡低 | trino | listTableNames 丢去重 | ✅ | ✅ | ✅ | ✅ | +| 18 | **L6** | 🟡低 | trino | guard 字段发布顺序→瞬时 NPE | ✅ | ✅ | ✅ | ✅ | +| 19 | **L7** | 🟡低 | kerberos | UGI.setConfiguration 无 guard(丢 first-writer) | ✅ | ✅ | ✅ | ✅ | +| 20 | **L8** | 🟡低 | kerberos | doAs 吞 interrupt 不 restore | ✅ | ✅ | ✅ | ✅ | +| 21 | **L9** | 🟡低 | maxcompute | 谓词下推全有全无 | ✅ | ✅ | ✅ | ✅ | | 22 | **L10** | 🟡低 | fe-core | EXPLAIN 节点名 VPluginDrivenScanNode | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | | 23 | **L11** | 🟡低 | paimon | JNI/COUNT file_format 用表级默认 | ✅ | ✅ | ✅ | ✅ | | 24 | **L12** | 🟡低 | fe-core/paimon | selectedPartitionNum 语义(登记或对齐) | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | From 83674a8c1ecdef544a9261d1a47204a6c71ce885 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:33:41 +0800 Subject: [PATCH 316/334] [doc](catalog) #65185 reverify: mark L2 done (query-cache SPI capability) L2 (flipped-hive SQL result-cache eligibility + COUNTER) was already resolved by c9a86337906 on a separate workstream: CacheAnalyzer/BindRelation/ SqlCacheContext/NereidsSqlCacheManager now recognize plugin tables via the connector-agnostic MTMVRelatedTableIf capability + its data-tied version token (the "add capability / restore cache" decision, not the register-deviation alternative), and the dead source-specific COUNTER_QUERY_HIVE_TABLE bump was intentionally removed. Mark done in the tracking table. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/task-list-65185-reverify-fixes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 969bdfeed51f40..fcfed42014a341 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -38,7 +38,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 11 | **M7** | 🟠中 | iceberg | REST vended-cred region 别名收窄 | ✅ | ✅ | ✅ | ✅ | | 12 | **M8** | 🟠中(运营) | build/docs | 升级只换 lib 不部署 plugins→首访抛 | ⬜ | ⬜ | ⬜ | ⏸ 跳过(用户 07-12;侦察见下) | | 13 | **L1** | 🟡低 | tools | import-gate 三洞**+第4洞** | ✅ | ✅ | ✅ | ✅ | -| 14 | **L2** | 🟡低 | fe-core | 翻闸 hive 丢 SQL 缓存资格 + COUNTER 停增 | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | +| 14 | **L2** | 🟡低 | fe-core | 翻闸 hive 丢 SQL 缓存资格 + COUNTER 停增 | ✅ | ✅ | ✅ | ✅ | | 15 | **L3** | 🟡低 | trino | 元数据事务从不 commit/close | ✅ | ✅ | ✅ | ✅ | | 16 | **L4** | 🟡低 | trino | plugin.dir 首胜单例(fail-loud) | ✅ | ✅ | ✅ | ✅ | | 17 | **L5** | 🟡低 | trino | listTableNames 丢去重 | ✅ | ✅ | ✅ | ✅ | @@ -178,7 +178,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) > 每条一行「现码 → fix」,详见 reverify §1 表 + 对应正文。⏸ 三条(L2/L10/L12)先问用户。 - [x] **L1** DONE `88aa55b831b`(设计 `designs/FIX-L1-design.md`;设计红队 `wf_643c11b4-3fe` 3 lens 全 SOUND_WITH_CHANGES)。补 3 洞(static / +6 包 / test 源)**+ 红队发现的第 4 洞**:4 条白名单 `grep -v` 按**整行**匹配(`.`≡`/`)→连接器命名空间文件(608 个,全根在 `org.apache.doris.connector.**`)里的违规 import 被**按路径抑制**,门禁对其结构性失明。修法=候选 grep 加宽(static+test glob+6 包)+**白名单锚定到 import 目标**(`:import ...` 非整行)+ fqn sed 剥 `static`(修 static-vendored 误报,红队证 E3 属正确性非装饰)。新增自测 `tools/check-connector-imports.test.sh`(8 违规/2 vendored skip/3 allow;GREEN 于新、RED 于旧,真树 exit 0)。保留 `is_vendored()`(HiveVersionUtil FP)。**观察**:grep-denylist 固有盲区(新增内部包漏登记/FQN-无-import 内联/间距)登记设计债,根治须 allowlist/ArchUnit(Trino 先例)。 -- [ ] **L2** ⏸ 翻闸 hive 丢 SQL 缓存资格 · `CacheAnalyzer.java:308/316/319`(instanceof HiveScanNode)+ `BindRelation.java:887`(instanceof HMSExternalTable)。**需决策**:加 `ConnectorCapability` 让 `PluginDrivenScanNode`/`PluginDrivenExternalTable` 被识别并恢复缓存 **vs** 登记为 fail-safe 验收偏差(`enable_hive_sql_cache` 默认关)。 +- [x] **L2** DONE `c9a86337906`(独立工作线,设计 `FIX-querycache-spi-design.md`;4 文件 + 3 测试类 8 测,fe-core BUILD SUCCESS 0 checkstyle)· **决策已定=恢复缓存**(选「加能力」路,非登记偏差):把 `CacheAnalyzer`/`BindRelation`/`SqlCacheContext`/`NereidsSqlCacheManager` 四文件六处源名分支换成 **connector-agnostic `MTMVRelatedTableIf` 能力** + 其 data-tied token `getNewestUpdateVersionOrTime()`(hive=max transient_lastDdlTime、iceberg/paimon=单调 snapshot 版本;token≤0 fail-safe 标 unsupported 不 pin 假常量),遵铁律无 `instanceof HMSExternalTable/HiveScanNode`。`enable_hive_sql_cache` 默认关不变。**COUNTER 部分**:dead 的 source-specific `COUNTER_QUERY_HIVE_TABLE` bump **有意移除**(非静默——commit 明载,通用外部缓存路不该 bump hive 专属计数)。e2e live-gated(翻闸 hive 表结果缓存命中/失效)。 - [x] **L3** DONE `4cd63c6911a`(设计 `FIX-L3-design.md`;对抗复审 `agent a182a049f` SOUND_WITH_CHANGES）· trino 6 个 FE-only 元数据站 try/finally 经 `releaseQuietly` 释放事务(commit + swallow-log 防 mask);scan 站不动(txnHandle 序列化发 BE 须保持打开)。UT 受 `io.trino.Session` 构造墙阻,登记 e2e live-gated。 - [x] **L4** DONE `e27602d4ab6`(设计 `FIX-L4-design.md`;并发对抗复审 `agent a28dc47095` SOUND）· `TrinoBootstrap` 存 pluginDir,`getInstance` 不同 dir fail-loud 抛(canonicalize best-effort 兜同物理目录异拼写)。UT 受重构造墙阻,登记。 - [x] **L5** DONE `be96adf76ba`(设计 `FIX-L5-design.md`）· `listTableNames` 加 `.distinct()` 复原 legacy LinkedHashSet 保序去重;不复原冗余 prefix 过滤。 From e3d8838ec255b7957387416fe1d39997f0bb8785 Mon Sep 17 00:00:00 2001 From: morningman Date: Sun, 12 Jul 2026 23:38:33 +0800 Subject: [PATCH 317/334] [doc](catalog) #65185 reverify: L10 accept as display-only deviation (DV-050) Per user signoff (2026-07-12), accept the flipped-external EXPLAIN scan-node name VPluginDrivenScanNode as a display-only deviation (DV-050) rather than add a Connector.getLegacyEngineName SPI to restore the per-connector VHIVE_/ VICEBERG_SCAN_NODE names: the getNodeExplainString CONNECTOR: line already discloses the catalog type, the regression goldens already expect the generic name (only one reference left, already the new name), and this matches Trino (one generic TableScan node with the connector as an attribute). No code change. Also roll the L2-L10 reconciliation: L3-L9 confirmed done in the log, L2 done by c9a86337906 (query-cache SPI capability). L2-L10 are now all closed; only L12/L20 remain as decision-type items. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF-65185-reverify-fixes.md | 2 +- plan-doc/deviations-log.md | 1 + plan-doc/task-list-65185-reverify-fixes.md | 16 ++++++++++++---- 3 files changed, 14 insertions(+), 5 deletions(-) diff --git a/plan-doc/HANDOFF-65185-reverify-fixes.md b/plan-doc/HANDOFF-65185-reverify-fixes.md index 2012d36012adbb..6d3ccea6f6895e 100644 --- a/plan-doc/HANDOFF-65185-reverify-fixes.md +++ b/plan-doc/HANDOFF-65185-reverify-fixes.md @@ -37,7 +37,7 @@ HMS 翻闸（catalog 类型 `hms` 从旧代码切到插件 SPI）后，第三方 | 批次 2 | M3→M1 中危(fe-core 通用节点) | ✅ 全 DONE | | **批次 3** | **L1(import 门禁) + M8(发布工具/文档)** | **✅ L1 DONE;M8 ⏸ 用户跳过(07-12)** | | 批次 4 | 低危连接器 L3–L20（trino/kerberos/mc/paimon/iceberg） | 🔄 trino L3-L6 ✅ · kerberos L7/L8 ✅ · mc L9 ✅ · paimon L11/L13/L14 ✅;**← 续 iceberg/杂项 L15–L19** | -| 决策类 | L2 / L10 / L12 / L20 | ⏸ 先用中文讲清背景问用户再动 | +| 决策类 | ~~L2~~ ✅ `c9a86337906` · ~~L10~~ ✅ accept [DV-050] · **余 L12 / L20** | ⏸ L12/L20 先用中文讲清背景问用户再动 | | 设计债 | D-系列 | ⏸ 择机 / 随 P8 | **批次 2 明细（本轮完成，含用户签字，勿再擅自翻）**： diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md index 188c48af131e37..4fd4e8f379c590 100644 --- a/plan-doc/deviations-log.md +++ b/plan-doc/deviations-log.md @@ -17,6 +17,7 @@ | 编号 | 偏差主题 | 原计划位置 | 日期 | 当前状态 | |---|---|---|---|---| +| DV-050 | **翻闸后 EXPLAIN 外部表扫描节点显示通用名 `VPluginDrivenScanNode`**（display-only；用户 2026-07-12 签字 accept）：翻闸前各源显示 `V_SCAN_NODE`（`VHIVE_SCAN_NODE`/`VICEBERG_SCAN_NODE` 等），翻闸后所有外部表走通用节点〔`PluginDrivenScanNode:173` 传 super label `"PluginDrivenScanNode"`〕→ EXPLAIN 统一 `VPluginDrivenScanNode`，且 `getNodeExplainString:325` 已附 `CONNECTOR: ` 行披露实际对接的数据源。**纯显示，无功能/结果/性能影响**。选 **accept**（非加 `Connector.getLegacyEngineName` SPI 恢复旧名）：`CONNECTOR:` 行信息未丢、regression 黄金文件已适配为 `VPluginDrivenScanNode`（全 regression 树仅 1 处引用扫描节点名且已是新名）、且与 **Trino** 一致（Trino EXPLAIN 对所有连接器统一 `TableScan` 通用节点名 + 连接器/表作属性，而非塞进节点类名）。否决 Option B（加 SPI 声明旧引擎名——不能用 catalog type 拼，hive 的 type=`hms`→会误拼 `VHMS_SCAN_NODE`；改动面大且须把黄金文件改回去，仅为复刻一个 cosmetic 串）。关联设计债 D-ENGINE（引擎名收口）择机随 P8 | reverify §1 L10 / [task 表 §L10](./task-list-65185-reverify-fixes.md) | 2026-07-12 | 🟢 已登记（accept；display-only；EXPLAIN reg 黄金已用新名）| | DV-049 | **P6.5 iceberg sys-table perf-cosmetic/display/internal 批汇总**（结果恒等/展示/内部枚举；镜像 DV-047/044 style）：**①sys split self-weight 丢**〔audit `T05-sys-split-weight`：legacy `IcebergScanNode.createIcebergSysSplit:900`+`IcebergSplit.newSysTableSplit:78` 设 `selfSplitWeight=Math.max(recordCount,1L)`；连接器 `IcebergScanRange` 无 `getSelfSplitWeight()` override → SPI 默认 -1 → `PluginDrivenSplit` `SplitWeight.standard()` 均匀。**result-equivalent**——查询结果/thrift 字段/serialized_split 字节皆不变，仅 BE `FederationBackendPolicy` 调度权重差；镜像 [DV-033] native-subsplit weight 不移植〕·**②内部 `TableIf.TableType=PLUGIN_EXTERNAL_TABLE`**〔legacy `IcebergSysExternalTable:57`=`ICEBERG_EXTERNAL_TABLE`〕——但 **user-visible 全 parity**：`getMysqlType`→`information_schema.tables.TABLE_TYPE` 两枚举皆 fall-through "BASE TABLE"〔`TableIf:319/324-325`〕、engine name 经 [D-066] T06-F1="iceberg"、descriptor 经 T06-C1=HIVE/ICEBERG_TABLE → 残留**仅内部枚举**〔无 user 可观测面〕·**③position_deletes 文案**〔Q2 用户签字：连接器 `listSupportedSysTables` 去 `POSITION_DELETES`+`getSysTableHandle`→empty → 通用 fe-core not-found（"Unknown sys table"）vs legacy bespoke "is not supported yet"（`IcebergSysTable:74`）；两侧 support boolean 同〔皆不可查〕仅文案分叉；**reg-test 已同步 2026-06-30**：`test_iceberg_sys_table.groovy` position_deletes 断言从旧 bespoke 文案改断通用 `Unknown sys table '$position_deletes'`，docker e2e 实跑绿〕。全部**非正确性** | T07 对抗审计 / [task 表 §P6.5](./tasks/P6-iceberg-migration.md) | 2026-06-25 | 🟢 已登记（accept；结果恒等/内部/展示；P6.6 docker/live 真值闸）| | DV-048 | **P6.5 iceberg sys-table correctness-bearing 但 UT 不可见**（parity-by-construction / 用户签字；docker 闸）：**①F2 paimon SHOW CREATE priv loosening（LIVE）**〔audit `F2-paimon-showcreate-priv-loosened-live` + critic：[D-066] T06-F2 给 `ShowCreateTableCommand.validate():120-124` 加 `PluginDrivenSysExternalTable`→`getSourceTable().getName()` 解包；**对 iceberg 是 parity**〔legacy `IcebergSysExternalTable` 分支 `:118-119` 已解包，且 iceberg pre-flip dormant〕，但 **paimon 在 SPI_READY_TYPES** → live 行为变更：sys `$`-表 SHOW CREATE 现按 **base** 表授权（pre-T06 按合成 'tbl$snapshots' 名）；严格**更宽松**、同向，与 `Env.getDdlStmt`/`UserAuthentication` 既有 output 解包一致、破坏风险近零；用户 T06-Q2 签字。**⚠️ 无隔离 UT**〔`validate()` 依赖 `Env`/`ConnectContext`/`AccessManager` 全局单例〕→ P6.8 e2e 兜底〕·**②serialized `FileScanTask` 字节潜伏（T05）**〔FE `SerializationUtil` deserialize-round-trip UT 已在**同进程 iceberg 1.10.1** 核「可消费+asDataTask+meta schema」，但与 BE `IcebergSysTableJniScanner` 的**跨版本/classloader interop** FE 不可及 → P6.8 docker e2e 兜底〕。**别于 DV-041**：本条**已建待 docker 实证**，DV-041 是未接线翻闸阻塞 | T07 对抗审计 / [task 表 §P6.5](./tasks/P6-iceberg-migration.md) | 2026-06-25 | 🟢 已登记（accept；P6.6 docker/Kerberos 真值闸）| | DV-047 | **P6.4 iceberg procedures perf/cosmetic/behaviour-equiv/dispatch-order 批汇总**（结果恒等/dormant/接缝/幂等；镜像 DV-044 style）：cache 失效搬 dispatch+短路多失效〔仅 context!=null〕·executeAction 加 ConnectorSession 参〔内部接缝〕·**DV-T08-loadwrap**〔新 "Failed to load iceberg table" 串,引擎再裹 "Failed to execute action:"〕·**DV-T08-factory-advertise**〔rewrite_data_files 广告-但-`createAction` 拒,dormant,canary UT 钉〕·DV-T06r-{scanpool〔丢 scanManifestsWith〕,zone〔复用 DV-T04-f〕,rollback〔不清列表中性〕}·DV-T07-{where〔WHERE 拒 fail-loud dormant〕,name-order〔priv-first 有意发散〕,exc-contract〔IllegalStateException 逃逸 byte-parity 边界〕}·PARTITION(*) 拒不对称〔low,dormant〕·null-row 编码〔low〕·per-conjunct filter 形状〔结构等价〕。全部**非正确性** | T04/T06/T07 设计 §10 / [task 表 §P6.4](./tasks/P6-iceberg-migration.md) | 2026-06-24 | 🟢 已登记（accept；结果恒等/展示/dormant/接缝；P6.6 docker 真值闸）| diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index fcfed42014a341..0233a11a1f09b5 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -46,7 +46,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 19 | **L7** | 🟡低 | kerberos | UGI.setConfiguration 无 guard(丢 first-writer) | ✅ | ✅ | ✅ | ✅ | | 20 | **L8** | 🟡低 | kerberos | doAs 吞 interrupt 不 restore | ✅ | ✅ | ✅ | ✅ | | 21 | **L9** | 🟡低 | maxcompute | 谓词下推全有全无 | ✅ | ✅ | ✅ | ✅ | -| 22 | **L10** | 🟡低 | fe-core | EXPLAIN 节点名 VPluginDrivenScanNode | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | +| 22 | **L10** | 🟡低 | fe-core | EXPLAIN 节点名 VPluginDrivenScanNode | — | — | — | ✅ 登记 DV-050 | | 23 | **L11** | 🟡低 | paimon | JNI/COUNT file_format 用表级默认 | ✅ | ✅ | ✅ | ✅ | | 24 | **L12** | 🟡低 | fe-core/paimon | selectedPartitionNum 语义(登记或对齐) | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | | 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ✅ | ✅ | ✅ | ✅ | @@ -68,7 +68,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **批次 2(中危·fe-core 通用节点,blast radius 较大,单测充分后再动)**：`M3`、`M1`。 - **批次 3(运营/门禁)**：`M8`(发布工具+文档,无 fe 编译)、`L1`(gate)。 - **批次 4(低危·连接器)**：`L3–L6`(trino)、`L7/L8`(kerberos)、`L9/L20`(mc)、`L11/L13/L14`(paimon)、`L18`(iceberg)。 -- **批次 5(需决策,先问用户再动)**：`L2`(SQL 缓存:恢复 vs 登记)、`L10`(EXPLAIN 名:登记 vs 按连接器声明)、`L12`(selectedPartitionNum:登记 vs 对齐)。 +- **批次 5(需决策,先问用户再动)**：~~`L2`(SQL 缓存)~~ **DONE `c9a86337906` 恢复缓存**、~~`L10`(EXPLAIN 名)~~ **DONE 用户签字 accept [DV-050]**;**余** `L12`(selectedPartitionNum:登记 vs 对齐)、`L20`(MC EQ 写法)。 - **批次 6(设计债/P8)**：`D-系列`,择机或随 P8。`D-PRUNE` 因承载 H1/H3,提前到批次 0。 > 决策类(⏸)条目**先在 session 里用中文讲清背景+选项问用户**(memory `ask-user-explain-in-chinese-first`),别擅自选一路实现。 @@ -187,7 +187,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L8** DONE `59697ce3fc7`(设计 `FIX-L8-design.md`）· `doAs` catch(InterruptedException) 内 `throw` 前加 `Thread.currentThread().interrupt();`(恢复中断标志)。 - [x] **L9** DONE `017d1af1894`(设计 `FIX-L9-design.md`;converter 纯函数,**RED-able UT**）· `convert` 特判根 `ConnectorAnd`:逐 top-level conjunct 独立转(抽 `convertOne`),丢+log 失败者、AND 幸存者;OR/NOT/嵌套 AND 保持全有全无(只根 AND 处正/单调位置丢 conjunct=超集,BE 重滤;OR 丢 disjunct=子集会漏行)。加 5 UT(3 RED-able+2 guard),21/21 绿。perf-only。 -- [ ] **L10** ⏸ EXPLAIN 节点名 · `PluginDrivenScanNode.java:170,320`。**需决策**:登记 display-only 验收偏差(cheapest,`CONNECTOR:` 行已披露)**vs** 加连接器声明的 legacy `*_SCAN_NODE` 名(连 `Connector.getLegacyEngineName` 一起,见 D-ENGINE)。注:报告的 `connectorType.toUpperCase()+"_SCAN_NODE"` 一行修法**不够**(hms 会出 `VHMS_SCAN_NODE` 非 `VHIVE_/VICEBERG_`)。 +- [x] **L10** DONE（**用户 2026-07-12 签字 accept**,登记 [DV-050],**无代码**）· EXPLAIN 外部表扫描节点显示通用名 `VPluginDrivenScanNode`(`PluginDrivenScanNode:173` super label)。**决策=接受为 display-only 偏差**(非加 `Connector.getLegacyEngineName` SPI 恢复旧名):`getNodeExplainString:325` 已附 `CONNECTOR: ` 披露数据源、regression 黄金文件全树仅 1 处引用且已是新名、且与 Trino 一致(统一 `TableScan` 通用名 + 连接器作属性)。否决 Option B(catalog type 拼名会误出 `VHMS_SCAN_NODE`;改动面大)。关联 D-ENGINE 择机随 P8。 - [x] **L11** DONE `4a8650bd062`(设计 `FIX-L11-design.md`;红队 `wf_05574ccb-bd2` 3 lens SOUND/SOUND_WITH_CHANGES）· JNI DataSplit + COUNT 路改按**首数据文件后缀**取 file_format(新 package-private `dataSplitFileFormat`,legacy `getFileFormat(getPathString())` parity),回退表级默认;顺补 `getFileFormatBySuffix` 的 `.avro` 臂(legacy `FileFormatUtils` parity,inert on native)。**call-site RED 测**(`Table.copy` 令表默认 parquet≠磁盘 .orc,断言 JNI+COUNT range 携 "orc")——原 helper 孤立测不守护接线(红队 MAJOR)。69/69 绿、0 checkstyle、gate 净。默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。e2e live-gated(cpp reader over 混/改格式表)。 - [ ] **L12** ⏸ selectedPartitionNum 语义 · `PluginDrivenScanNode.java:297-303`。**需决策**:登记为「paimon/iceberg 对齐 MC/hive 的 Nereids-剪枝数」验收偏差(推荐)**vs** 连接器回报 SDK-distinct(重,不推荐)。同步 paimon/iceberg EXPLAIN `partition=N/M` 回归期望。**勿**在通用节点按源分支。 - [x] **L13** DONE `ced4775b844`(设计 `FIX-L13-design.md`;红队 3 lens 全 SOUND）· `toPaimonType` 对 ARRAY 元素/MAP value/STRUCT 字段加 `.copy(type.isChildNullable(i))`(MAP key 保持 `.copy(false)`),恢复 legacy `DorisToPaimonTypeVisitor` 嵌套 nullability parity。**scope=仅 nullability**:comment 丢=DV-035 M10.1 已接受偏差、field-id 顺序 parity 均不动。`.copy(true)` 对默认可空子类型逐字节恒等(既有 parity 测保持绿)。3 新 RED-able 测(经 `.type().isNullable()` 断言,非 DataField equals)。type-mapping+schema-builder 26/26 绿。e2e live-gated(建嵌套 NOT NULL 表 DESCRIBE/SDK 读回)。 @@ -292,4 +292,12 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **L13** `ced4775b844` — `toPaimonType` 嵌套 nullability `.copy(isChildNullable)`(ARRAY/MAP-value/STRUCT-field),scope 仅 nullability(comment=DV-035 M10.1 已接受);`.copy(true)` 默认恒等。 - **L14** `478718aca6f` — honor `ignore_split_type`(null-tolerant helper + 三 legacy continue 位;`IGNORE_PAIMON_CPP` no-op=legacy parity);3 live-planScan RED 测,nonDataSplit 位 E2E-only。 - **共性**:全走单任务循环(复核现码→设计→红队→实现→build+靶向 UT→独立 commit)。**paimon 构建坑复现**:`mvn test` 因 hive-shade 模块 shade 绑 package→`HiveConf` NoClassDefFound 假失败,改 `package` 阶段即绿(memory `doris-build-verify-gotchas`)。模块靶向 UT 全绿(scan 69/69 + type-mapping/schema-builder 26/26)、0 checkstyle、import gate 净。**e2e 全 live-gated**(cpp reader 混格式 / 嵌套 NOT NULL DDL / SET ignore_split_type 跳分片)。 -- **下一步 = iceberg/杂项 L15–L19**(L15 paimon-metrics 悬空、L16/L17 iceberg 缓存/version-blind、L18 iceberg 未知类型、L19 partition_columns 撞名);⏸ 决策类 L2/L10/L12/L20 先中文讲清背景问用户。 +- **下一步 = iceberg/杂项 L15–L19**(L15 paimon-metrics 悬空、L16/L17 iceberg 缓存/version-blind、L18 iceberg 未知类型、L19 partition_columns 撞名)。 + +--- + +**⭐ L2–L10 复核对账（2026-07-12,按 commit log 逐条核实）**： +- **L3–L9 已完成**（顶部进度表此前漏勾,已同步 `a691d0264f5`）——7 commit 全在分支:trino `4cd63c6911a`/`e27602d4ab6`/`be96adf76ba`/`18048f7f217`、kerberos `9e4f2992382`/`59697ce3fc7`、mc `017d1af1894`。 +- **L2 已完成** `c9a86337906`（独立工作线,今日）——SQL 结果缓存资格用 `MTMVRelatedTableIf` 能力恢复(选「加能力」非登记偏差)、`COUNTER_QUERY_HIVE_TABLE` dead bump 有意移除;4 文件+3 测试类。标 done `83674a8c1ec`。 +- **L10 已定** = **用户 2026-07-12 签字 accept**,登记 [DV-050],**无代码**：EXPLAIN 通用节点名 `VPluginDrivenScanNode` 接受为 display-only 偏差(`CONNECTOR:` 行已披露 + 黄金文件已适配 + 对齐 Trino 通用节点名做法)。 +- **净结果**：L2–L10 全部收口(done 或 accept)。决策类**仅余 L12/L20**,动前先中文讲清问用户。⏸ 决策类先问用户。 From 801d8cd6c86d876b56290a28ef99d2dc9e609a43 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 07:32:14 +0800 Subject: [PATCH 318/334] [fix](catalog) hms: surface root cause in HMS client creation errors The plugin-driven HMS connector wrapped client-creation failures with only the immediate cause's getMessage(). For a kerberos/SASL misconfig that immediate cause is Hive's opaque RuntimeException("Unable to instantiate ...HiveMetaStoreClient"), so the real reason (thrift TTransportException: GSS initiate failed) was dropped from the user-facing message. FE surfaces only the top exception's message, unlike the legacy ThriftHMSCachedClient/HMSClientException which appended Util.getRootCauseMessage(cause). Add ThriftHmsClient.withRootCause() to append the deepest cause (className: message form, matching the legacy format) at the createFreshClient/borrowClient throw sites, with a guard so the pool re-wrap does not duplicate the reason. This restores the thrift/GSS reason that external_table_p0/kerberos/test_single_hive_kerberos asserts (both err1 and err2 cases), and keeps the actionable cause visible to users debugging HMS connection failures. Message-only change; no connector logic altered. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../doris/connector/hms/ThriftHmsClient.java | 40 +++++++- .../ThriftHmsClientRootCauseMessageTest.java | 92 +++++++++++++++++++ 2 files changed, 128 insertions(+), 4 deletions(-) create mode 100644 fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientRootCauseMessageTest.java diff --git a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java index 1c19541e679ac8..7cedab95bcd194 100644 --- a/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java +++ b/fe/fe-connector/fe-connector-hms/src/main/java/org/apache/doris/connector/hms/ThriftHmsClient.java @@ -723,8 +723,8 @@ private PooledHmsClient borrowClient() { try { return clientPool.borrowObject(); } catch (Exception e) { - throw new HmsClientException("Failed to borrow HMS client " - + "from pool: " + e.getMessage(), e); + throw new HmsClientException(withRootCause("Failed to borrow HMS client " + + "from pool: " + e.getMessage(), e), e); } } @@ -733,9 +733,41 @@ private PooledHmsClient createFreshClient() { return doAs(() -> new PooledHmsClient( clientProvider.create(hiveConf))); } catch (Exception e) { - throw new HmsClientException("Failed to create HMS client: " - + e.getMessage(), e); + throw new HmsClientException(withRootCause("Failed to create HMS client: " + + e.getMessage(), e), e); + } + } + + /** + * Appends the deepest cause's message to {@code message} so the actionable reason survives. + * + *

    WHY: Hive wraps the real connection failure (e.g. a thrift {@code TTransportException: + * GSS initiate failed} from a SASL/kerberos misconfiguration) inside a generic + * {@code RuntimeException("Unable to instantiate ...HiveMetaStoreClient")} whose own + * {@link Throwable#getMessage()} drops that cause. FE surfaces only the top exception's message, + * so without this the user (and regression assertions) would see "Unable to instantiate ..." + * with no hint of the SASL/GSS/transport root cause. This mirrors the legacy + * {@code ThriftHMSCachedClient}/{@code HMSClientException}, which appended + * {@code Util.getRootCauseMessage(cause)} in the same {@code className: message} form. + * + *

    The guard avoids duplicating the reason when a fresh-client failure is re-wrapped by the + * pool's {@link #borrowClient()} (the inner message already carries the appended root cause). + */ + static String withRootCause(String message, Throwable cause) { + if (cause == null) { + return message; + } + Throwable root = cause; + while (root.getCause() != null && root.getCause() != root) { + root = root.getCause(); + } + String rootMessage = root.getMessage(); + String rootDescription = root.getClass().getName() + + (rootMessage == null ? "" : ": " + rootMessage); + if (message != null && message.contains(rootDescription)) { + return message; } + return message + ". reason: " + rootDescription; } private GenericObjectPoolConfig createPoolConfig( diff --git a/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientRootCauseMessageTest.java b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientRootCauseMessageTest.java new file mode 100644 index 00000000000000..57347d27deb322 --- /dev/null +++ b/fe/fe-connector/fe-connector-hms/src/test/java/org/apache/doris/connector/hms/ThriftHmsClientRootCauseMessageTest.java @@ -0,0 +1,92 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.hms; + +import org.apache.hadoop.hive.metastore.api.MetaException; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.lang.reflect.InvocationTargetException; + +/** + * Tests {@link ThriftHmsClient#withRootCause}: the client-creation failure message must preserve the deepest + * cause so a SASL/kerberos/thrift transport error stays visible to the user. + * + *

    WHY: Hive buries the real connection failure — e.g. {@code TTransportException: GSS initiate failed} on a + * kerberos misconfiguration — inside a generic {@code RuntimeException("Unable to instantiate + * ...HiveMetaStoreClient")} whose own message drops that cause, and FE surfaces only the top exception's + * message. The legacy {@code ThriftHMSCachedClient} appended {@code Util.getRootCauseMessage(cause)}, so + * {@code external_table_p0/kerberos/test_single_hive_kerberos} asserts the surfaced error contains the thrift + * transport reason. These assertions pin that the connector restores the same behavior — and does not + * duplicate the reason when the pool re-wraps a fresh-client failure. + */ +public class ThriftHmsClientRootCauseMessageTest { + + // The message Hive builds via StringUtils.stringifyException when it cannot open the metastore transport. + private static final String GSS_REASON = + "Could not connect to meta store using any of the URIs provided. Most recent failure: " + + "shade.doris.hive.org.apache.thrift.transport.TTransportException: GSS initiate failed"; + + /** Rebuilds the exact nesting the plugin-driven HMS client produces for a kerberos SASL failure. */ + private static Throwable unableToInstantiate() { + MetaException root = new MetaException(GSS_REASON); + InvocationTargetException reflective = new InvocationTargetException(root); + return new RuntimeException( + "Unable to instantiate org.apache.hadoop.hive.metastore.HiveMetaStoreClient", reflective); + } + + @Test + public void freshClientMessageSurfacesThriftRootCause() { + Throwable cause = unableToInstantiate(); + String message = ThriftHmsClient.withRootCause("Failed to create HMS client: " + cause.getMessage(), cause); + // err1 assertion in test_single_hive_kerberos.groovy. + Assertions.assertTrue(message.contains("thrift.transport.TTransportException"), message); + // err2 assertion in the same suite expects the full "could not connect ... GSS initiate failed" reason. + Assertions.assertTrue(message.contains(GSS_REASON), message); + } + + @Test + public void poolReWrapDoesNotDuplicateReason() { + Throwable cause = unableToInstantiate(); + // First wrap: what createFreshClient() throws. + String createMessage = + ThriftHmsClient.withRootCause("Failed to create HMS client: " + cause.getMessage(), cause); + HmsClientException createFailure = new HmsClientException(createMessage, cause); + // Second wrap: what borrowClient() throws around the pool factory failure. + String borrowMessage = ThriftHmsClient.withRootCause( + "Failed to borrow HMS client from pool: " + createFailure.getMessage(), createFailure); + + Assertions.assertTrue(borrowMessage.contains("thrift.transport.TTransportException"), borrowMessage); + Assertions.assertTrue(borrowMessage.contains(GSS_REASON), borrowMessage); + // The reason must be appended exactly once even though the exception is wrapped twice. + Assertions.assertEquals(1, countOccurrences(borrowMessage, ". reason: "), borrowMessage); + } + + @Test + public void nullCauseLeavesMessageUnchanged() { + Assertions.assertEquals("plain", ThriftHmsClient.withRootCause("plain", null)); + } + + private static int countOccurrences(String haystack, String needle) { + int count = 0; + for (int i = haystack.indexOf(needle); i >= 0; i = haystack.indexOf(needle, i + needle.length())) { + count++; + } + return count; + } +} From 4291a481f678d859016c57ce5dca390e632e651d Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 07:40:34 +0800 Subject: [PATCH 319/334] [fix](mtmv) paimon: regenerate null_partition golden for genuine-NULL rows The connector-supplied per-value NULL partition flag (58f3e367ed6) plus the pn_NULL naming (a634c5c3e31) changed the MV refresh predicate for paimon's genuine-NULL region partition from region IN ('__HIVE_DEFAULT_PARTITION__') -- which matched no rows and silently dropped the null rows -- to region IS NULL, which correctly materializes them. test_paimon_mtmv order_qt_null_partition now returns 5 rows: the two genuine-NULL region rows (id=2, id=3 -> \N) join the existing 'bj'/'null'/'NULL' rows. The golden still encoded the stale 3-row result and failed with "line 2, BIGINT result mismatch" (TeamCity #992648). Update the golden to the corrected baseline. Values taken verbatim from the CI run's realResults; the partition set itself (p_bj/p_null/p_NULL/pn_NULL) was already correct. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- regression-test/data/mtmv_p0/test_paimon_mtmv.out | 2 ++ 1 file changed, 2 insertions(+) diff --git a/regression-test/data/mtmv_p0/test_paimon_mtmv.out b/regression-test/data/mtmv_p0/test_paimon_mtmv.out index 5c7547c0687c86..f1d5af6176e046 100644 --- a/regression-test/data/mtmv_p0/test_paimon_mtmv.out +++ b/regression-test/data/mtmv_p0/test_paimon_mtmv.out @@ -137,6 +137,8 @@ true -- !null_partition -- 1 bj +2 \N +3 \N 4 null 5 NULL From b2cdf971889a664318b045d635eb8f33cd3932c4 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 07:52:49 +0800 Subject: [PATCH 320/334] [fix](catalog) remove dangling PAIMON_SCAN_METRICS profile key - root cause: P5 (dbc38a265e5) dropped the paimon FE subsystem and its scan metrics, but left three dead references to PAIMON_SCAN_METRICS in SummaryProfile (constant + EXECUTION_SUMMARY_KEYS + indentation map). Nothing populates the "Paimon Scan Metrics" profile group anymore (no PaimonMetricsReporter), unlike the still-live ICEBERG_SCAN_METRICS which IcebergMetricsReporter creates/fills. - solution: delete the three dead references; leave ICEBERG_SCAN_METRICS untouched. Zero behavior change (the column never appeared). - tests: none added (dead-code removal, nothing behavioral to assert); fe-core test-compile BUILD SUCCESS, 0 Checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../java/org/apache/doris/common/profile/SummaryProfile.java | 3 --- 1 file changed, 3 deletions(-) diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java index 124be52b05b078..cb051c3d624f20 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java @@ -155,7 +155,6 @@ public class SummaryProfile { public static final String LATENCY_FROM_BE_TO_FE = "RPC Latency From BE To FE"; public static final String SPLITS_ASSIGNMENT_WEIGHT = "Splits Assignment Weight"; public static final String ICEBERG_SCAN_METRICS = "Iceberg Scan Metrics"; - public static final String PAIMON_SCAN_METRICS = "Paimon Scan Metrics"; public static final String WAIT_CHANGE_VISIBLE_TIME = "Wait Change Visible Time"; private boolean isWarmUp = false; @@ -215,7 +214,6 @@ public boolean isWarmup() { SINK_SET_PARTITION_VALUES_TIME, CREATE_SCAN_RANGE_TIME, ICEBERG_SCAN_METRICS, - PAIMON_SCAN_METRICS, NEREIDS_DISTRIBUTE_TIME, GET_META_VERSION_TIME, GET_PARTITION_VERSION_TIME, @@ -274,7 +272,6 @@ public boolean isWarmup() { .put(SINK_SET_PARTITION_VALUES_TIME, 3) .put(CREATE_SCAN_RANGE_TIME, 2) .put(ICEBERG_SCAN_METRICS, 3) - .put(PAIMON_SCAN_METRICS, 3) .put(GET_PARTITION_VERSION_TIME, 1) .put(GET_PARTITION_VERSION_COUNT, 1) .put(GET_PARTITION_VERSION_BY_HAS_DATA_COUNT, 1) From 4e6c08d9beac13689722ce122ce420785ab9600e Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 07:53:48 +0800 Subject: [PATCH 321/334] [doc](catalog) #65185 reverify: L15 done + L12 recon (await decision) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - L15: mark done (b2cdf971889); add FIX-L15 design + summary. - L12: record the 4-agent recon (wf_6c516483-c34) — iceberg is affected (routes through PluginDrivenScanNode), old=SDK-distinct vs new=Nereids- pruned count, divergence only under hidden-partition/data-skipping, 0 golden EXPLAIN blast radius, 3 sql_block_rule tests unaffected by either option, Trino leans Option B. Still awaiting user decision. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/task-list-65185-reverify-fixes.md | 16 +++++- plan-doc/tasks/designs/FIX-L15-design.md | 65 ++++++++++++++++++++++ plan-doc/tasks/designs/FIX-L15-summary.md | 36 ++++++++++++ 3 files changed, 115 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L15-design.md create mode 100644 plan-doc/tasks/designs/FIX-L15-summary.md diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 0233a11a1f09b5..375f1d6df933a9 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -51,7 +51,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 24 | **L12** | 🟡低 | fe-core/paimon | selectedPartitionNum 语义(登记或对齐) | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | | 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ✅ | ✅ | ✅ | ✅ | | 26 | **L14** | 🟡低 | paimon | ignore_split_type 静默 no-op | ✅ | ✅ | ✅ | ✅ | -| 27 | **L15** | 🟡低 | fe-core | PAIMON_SCAN_METRICS 悬空常量 | ⬜ | ⬜ | ⬜ | ⬜ | +| 27 | **L15** | 🟡低 | fe-core | PAIMON_SCAN_METRICS 悬空常量 | ✅ | ✅ | ✅ | ✅ | | 28 | **L16** | 🟡低 | iceberg | 快照/schema 缓存偏斜(防御性 union) | ⬜ | ⬜ | ⬜ | ⬜ | | 29 | **L17** | 🟡低 | fe-core/iceberg | 同表多版本 version-blind schema 绑定 | ⬜ | ⬜ | ⬜ | ⬜ | | 30 | **L18** | 🟡低 | iceberg | 未知/v3 类型静默 UNSUPPORTED | ⬜ | ⬜ | ⬜ | ⬜ | @@ -192,7 +192,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [ ] **L12** ⏸ selectedPartitionNum 语义 · `PluginDrivenScanNode.java:297-303`。**需决策**:登记为「paimon/iceberg 对齐 MC/hive 的 Nereids-剪枝数」验收偏差(推荐)**vs** 连接器回报 SDK-distinct(重,不推荐)。同步 paimon/iceberg EXPLAIN `partition=N/M` 回归期望。**勿**在通用节点按源分支。 - [x] **L13** DONE `ced4775b844`(设计 `FIX-L13-design.md`;红队 3 lens 全 SOUND）· `toPaimonType` 对 ARRAY 元素/MAP value/STRUCT 字段加 `.copy(type.isChildNullable(i))`(MAP key 保持 `.copy(false)`),恢复 legacy `DorisToPaimonTypeVisitor` 嵌套 nullability parity。**scope=仅 nullability**:comment 丢=DV-035 M10.1 已接受偏差、field-id 顺序 parity 均不动。`.copy(true)` 对默认可空子类型逐字节恒等(既有 parity 测保持绿)。3 新 RED-able 测(经 `.type().isNullable()` 断言,非 DataField equals)。type-mapping+schema-builder 26/26 绿。e2e live-gated(建嵌套 NOT NULL 表 DESCRIBE/SDK 读回)。 - [x] **L14** DONE `478718aca6f`(设计 `FIX-L14-design.md`;红队 3 lens SOUND/SOUND_WITH_CHANGES）· null-tolerant `resolveIgnoreSplitType(session)`(镜像 `isCppReaderEnabled`,红队 MINOR)+三 legacy `continue` 位:`IGNORE_JNI` drops nonDataSplit+DataSplit-JNI 臂、`IGNORE_NATIVE` drops native 臂、count 臂不检、`IGNORE_PAIMON_CPP` 保持 no-op(=legacy parity,全树 grep 证 legacy 从不引用)。默认 NONE 逐字节不变。3 新 live-planScan RED 测(IGNORE_JNI/IGNORE_NATIVE 各证跳 split+IGNORE_PAIMON_CPP==NONE);nonDataSplit IGNORE_JNI 位离线不可测→留 E2E-only。69/69 绿。e2e live-gated(真集群 SET ignore_split_type 断言跳分片)。 -- [ ] **L15** PAIMON_SCAN_METRICS 悬空 · `SummaryProfile.java:158,218,277`:删三处死引用(P5 已验收弃 paimon FE scan metrics);或加 connector-neutral scan-metrics SPI(feature,非必需)。 +- [x] **L15** DONE `b2cdf971889`(设计 `designs/FIX-L15-design.md`;summary `FIX-L15-summary.md`)· 删 `SummaryProfile.java` 三处 `PAIMON_SCAN_METRICS` 死引用(常量 + `EXECUTION_SUMMARY_KEYS` + 缩进 map);P5(`dbc38a265e5`)删写入方后无任何 reporter 填充该分组(对照活的 `ICEBERG_SCAN_METRICS` 有 `IcebergMetricsReporter` 保留不动)。**未加** connector-neutral scan-metrics SPI(feature、非必需,超出清死引用 scope)。零行为变更、无新测(死码删除无可断言);fe-core test-compile BUILD SUCCESS + 0 checkstyle。**非 live**(该 profile 列本就从不出现)。 - [ ] **L16** iceberg 缓存偏斜(部分已修) · `IcebergScanPlanProvider.java:1077-1108`:hasSnapshotPin 臂把 field-id dict 建成「pinned schema ∪ requested latest columns」超集(传 requestedLowerNames 而非 `emptyList`),两向都超集;或 fe-core 侧 query-begin pin 解析 pinnedSchema 到 pinned schemaId(对齐 time-travel 臂 `:377-387`)。 - [ ] **L17** iceberg 同表多版本 version-blind 绑定 · `PluginDrivenMvccExternalTable.java:475-485`:per-reference schema 绑定 version-aware(用 `StatementContext.getSnapshot(table, tableSnapshot, scanParams)` 的 pinnedSchema,fallback latest);与 L16 同根,建议一并。窄触发 + fail-loud。 - [ ] **L18** iceberg 未知/v3 类型静默 UNSUPPORTED · `IcebergTypeMapping.java:91,143`:两 default 臂改 `throw DorisConnectorException("Cannot transform unknown type: "+...)`,保留显式 TIME/VARIANT UNSUPPORTED;或接受更松并登记。 @@ -301,3 +301,15 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **L2 已完成** `c9a86337906`（独立工作线,今日）——SQL 结果缓存资格用 `MTMVRelatedTableIf` 能力恢复(选「加能力」非登记偏差)、`COUNTER_QUERY_HIVE_TABLE` dead bump 有意移除;4 文件+3 测试类。标 done `83674a8c1ec`。 - **L10 已定** = **用户 2026-07-12 签字 accept**,登记 [DV-050],**无代码**：EXPLAIN 通用节点名 `VPluginDrivenScanNode` 接受为 display-only 偏差(`CONNECTOR:` 行已披露 + 黄金文件已适配 + 对齐 Trino 通用节点名做法)。 - **净结果**：L2–L10 全部收口(done 或 accept)。决策类**仅余 L12/L20**,动前先中文讲清问用户。⏸ 决策类先问用户。 + +--- + +**⭐ L15 DONE `b2cdf971889`(2026-07-13)** — 删 `SummaryProfile` 三处 `PAIMON_SCAN_METRICS` 死引用(P5 后无 reporter 填充,对照活的 iceberg metrics 保留)。零行为变更、无新测、非 live;fe-core BUILD SUCCESS + 0 checkstyle。设计/summary = `designs/FIX-L15-{design,summary}.md`。 + +**⭐ L12 侦察完成(2026-07-13,workflow `wf_6c516483-c34`,4 agent recon)= 待用户决策**： +- **确认 iceberg 也受影响**:iceberg 读实走 `PluginDrivenScanNode`(legacy `IcebergScanNode.java` 已死;`PhysicalPlanTranslator:812-829` instanceof `PluginDrivenExternalTable` 臂)。故 L12 覆盖 paimon+iceberg 两者。 +- **旧→新语义**:旧 = **SDK-distinct**(连接器 SDK 实际规划 split 后的 distinct 原生分区数;paimon `partitionInfoMaps.size()` keyed `dataSplit.partition()`、iceberg `partitionMapInfos.size()` keyed `file().partition()`);新 = **Nereids-剪枝分区项数**(`selectedPartitions.size()`,仅按声明分区列在 `LogicalFilter` 下剪,见不到 SDK manifest/residual/bucket/transform 剪枝)。新数 **≥** 旧数。 +- **背离场景**:仅当 SDK 能剪而 Nereids 不能——iceberg 隐藏/transform 分区(`days(ts)` + WHERE ts:新 30/30 vs 旧 1/30)、paimon 非分区列 manifest 剪枝(WHERE id=999:新 2/2 vs 旧 1/2)。identity 分区 + 无谓词时两者相等。 +- **爆炸半径 = 0 黄金 EXPLAIN 断言**:paimon/iceberg 无任何 `partition=N/M` 黄金(它们断言 split-count 指标 `inputSplitNum`/`paimonNativeReadSplits`,不受影响)。3 个 `sql_block_rule partition_num` 测(iceberg/paimon/hive)均**identity 分区 + 无谓词**→两选项同值→**两选项都不破测**。 +- **Trino 对照**:Trino EXPLAIN **无**引擎统一的 per-scan 分区计数;分区计量完全 connector-owned(row estimate 由连接器 `getTableStatistics` 报,分区 guard 如 hive `max-partitions-per-scan` 在连接器内执行)。据此 Trino 更贴近 **选项 B**(连接器经能力 SPI 回报 SDK-distinct,不违铁律——能力 SPI 本身即反分支机制,仿现有 `supportsTableSample`/`Split.getLength()` opt-in)。 +- **张力(Rule 7)**:task list 原推荐 A(登记 Nereids 数为验收偏差,零代码);但 Trino 与「该数还喂 `sql_block_rule` 治理」角度支持 B(A 在隐藏分区下**高报**实际扫描分区数=治理 false-positive over-block)。已用中文向用户讲清背景+两选项+推荐,待拍板。 diff --git a/plan-doc/tasks/designs/FIX-L15-design.md b/plan-doc/tasks/designs/FIX-L15-design.md new file mode 100644 index 00000000000000..bfbe687a906ff1 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L15-design.md @@ -0,0 +1,65 @@ +# FIX-L15 — `PAIMON_SCAN_METRICS` 悬空常量(死引用) + +> reverify §1 L15 (P5-?)。严重度 🟡低。模块 = fe-core (`SummaryProfile`)。 + +## Problem + +`SummaryProfile` 里保留了 `PAIMON_SCAN_METRICS = "Paimon Scan Metrics"` 这个 profile 分组键的三处引用, +但 **P5 迁移(commit `dbc38a265e5`,把 legacy paimon 子系统从 fe-core 移除、使 fe-core paimon-SDK-free) +之后再没有任何代码往这个分组写入数据**。它是一个纯粹的死引用:一个永远不会被填充的 profile 列。 + +对照活的 `ICEBERG_SCAN_METRICS`:iceberg 保留了 `IcebergMetricsReporter`,它在 +`IcebergMetricsReporter.java:72,74` 主动 `getChildMap().get(ICEBERG_SCAN_METRICS)` / `new RuntimeProfile(ICEBERG_SCAN_METRICS)` +创建并填充该分组。paimon **没有**对应的 reporter(P5 删除时一并弃用了 paimon FE scan metrics),故其键悬空。 + +## HEAD recon(对 HEAD 复核 + grep 全量取证) + +`PAIMON_SCAN_METRICS` 全代码库仅 3 处引用,**全部在 `SummaryProfile.java` 自身**: +- `:158` 常量声明 `public static final String PAIMON_SCAN_METRICS = "Paimon Scan Metrics";` +- `:218` `EXECUTION_SUMMARY_KEYS` 列表条目(display 顺序表) +- `:277` `EXECUTION_SUMMARY_KEYS_INDENTATION` 缩进 map 条目 `.put(PAIMON_SCAN_METRICS, 3)` + +取证: +- `grep -rn "PAIMON_SCAN_METRICS" fe/` → 只上述 3 行。 +- `grep -rn "Paimon Scan Metrics" fe/ be/` → 只 `:158` 一行(BE 侧亦无写入方)。 +- `grep -rln "PaimonMetricsReporter\|PaimonScanMetrics"` → 无(确认无 paimon metrics 机制)。 +- `fe/fe-core/src/test/` 无任何引用 → 删除不破测试。 +- `git log -S "PAIMON_SCAN_METRICS"` → 写入方在 `dbc38a265e5`(P5)被移除。 + +对照:`ICEBERG_SCAN_METRICS` 有活的 `IcebergMetricsReporter` → **保留不动**。 + +## Design(surgical:删三行死引用) + +P5 已验收弃用 paimon FE scan metrics(task list L15:「删三处死引用」为**推荐**路; +「加 connector-neutral scan-metrics SPI」标注为 feature、**非必需**,不做——本 SPI 迁移铁律是不在 fe-core +加 source-specific 结构,而一个真正通用的 scan-metrics SPI 是独立的 feature 债,超出「清死引用」的最小 scope)。 + +删除: +- `:158` 常量声明整行。 +- `:218` `EXECUTION_SUMMARY_KEYS` 里的 `PAIMON_SCAN_METRICS,` 整行。 +- `:277` 缩进 map 里的 `.put(PAIMON_SCAN_METRICS, 3)` 整行。 + +`ICEBERG_SCAN_METRICS` 三处(`:157`/`:217`/`:276`)全部**保留原样**(活引用)。 + +## Risk Analysis + +- **零行为变更**:该键从未被任何 reporter 填充,故其在 `EXECUTION_SUMMARY_KEYS`(display 顺序) + 与缩进 map 中都是无效条目——删除不改变任何实际 profile 输出。既有含 iceberg scan metrics 的 profile 不受影响 + (独立键)。 +- **无序列化耦合**:`EXECUTION_SUMMARY_KEYS` 只驱动 FE web-ui / profile 文本的展示顺序与集合; + 移除一个从不出现的键不影响其它键的展示。 +- **无跨模块引用**:全库仅 `SummaryProfile` 自引用,连接器/BE/测试均不引用。 + +## Test Plan + +### Unit Tests +- 无需新增单测:这是死引用删除,无行为可断言(强行造测=测「不存在的东西不存在」,无意义)。 +- 回归护栏:`SummaryProfile` 及现有 profile 相关单测须继续编译通过(fe-core test-compile)。 + +### E2E Tests +- 不适用(无运行时行为变更;paimon profile 从不含此列)。 + +## 验证判据 +- `mvn -o -f fe/pom.xml -pl fe-core -am test-compile` BUILD SUCCESS、0 checkstyle。 +- 删后 `grep -rn "PAIMON_SCAN_METRICS" fe/` 返回空。 +- `ICEBERG_SCAN_METRICS` 三处保持不变(diff 只删 paimon 三行)。 diff --git a/plan-doc/tasks/designs/FIX-L15-summary.md b/plan-doc/tasks/designs/FIX-L15-summary.md new file mode 100644 index 00000000000000..7da6056f72e0b9 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L15-summary.md @@ -0,0 +1,36 @@ +# FIX-L15 Summary — `PAIMON_SCAN_METRICS` 悬空常量清除 + +## Problem +`SummaryProfile` 保留了 `PAIMON_SCAN_METRICS`(profile 分组键 `"Paimon Scan Metrics"`)三处引用, +但自 P5(`dbc38a265e5`,paimon 子系统移出 fe-core、fe-core paimon-SDK-free)后再无任何代码填充该分组 +——纯死引用(一个永不出现的 profile 列)。 + +## Root Cause +P5 移除 paimon FE 子系统时弃用了 paimon FE scan metrics,却漏删了 `SummaryProfile` 里对应的常量与两处列表/缩进 +条目。对照活的 `ICEBERG_SCAN_METRICS`:iceberg 保留 `IcebergMetricsReporter`(`IcebergMetricsReporter.java:72,74` +主动创建/填充该分组),故其键活;paimon 无对应 reporter,键悬空。 + +## Fix +删除 `SummaryProfile.java` 三处死引用: +- 常量声明 `public static final String PAIMON_SCAN_METRICS = "Paimon Scan Metrics";` +- `EXECUTION_SUMMARY_KEYS` 列表中的 `PAIMON_SCAN_METRICS,` +- `EXECUTION_SUMMARY_KEYS_INDENTATION` map 中的 `.put(PAIMON_SCAN_METRICS, 3)` + +`ICEBERG_SCAN_METRICS` 三处(常量 + 列表 + 缩进)全部保留(活引用)。 + +**未做**「加 connector-neutral scan-metrics SPI」:task list 标注为 feature、非必需;一个真正通用的 scan-metrics +SPI 是独立 feature 债,超出「清死引用」的最小 scope,且与本系列铁律(fe-core 不加 source-specific 结构)正交。 + +## Tests +- 无新增单测:死引用删除,无运行时行为可断言(强行造测无意义)。 +- 回归护栏:fe-core `test-compile` 须继续通过。 + +## Result +- `mvn -o -f fe/pom.xml -pl fe-core -am test-compile -Dmaven.build.cache.enabled=false` → **BUILD SUCCESS**, + **0 Checkstyle violations**。 +- `grep -rn "PAIMON_SCAN_METRICS\|Paimon Scan Metrics" fe/` → 空(确认已清)。 +- `ICEBERG_SCAN_METRICS` 5 处引用不变(diff 仅删 paimon 三行,`1 file changed, 3 deletions(-)`)。 +- 零行为变更;不 live(无 e2e 欠账——该 profile 列本就从不出现)。 + +## Commit +- code:`SummaryProfile.java`(-3 行)。 From e5de7aedcd503d6a8a39318f727fd6f5036cece2 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 08:41:02 +0800 Subject: [PATCH 322/334] [fix](catalog) report connector-real scanned-partition count for selectedPartitionNum - root cause: migrating to the generic plugin scan node changed selectedPartitionNum (EXPLAIN partition=N/M + sql_block_rule partition_num) from the connector SDK's distinct scanned-partition count to the Nereids declared-partition-column prune count (always >= real). For predicate-driven connectors (iceberg hidden/transform partitioning, paimon non-partition-column manifest pruning) this over-reports the partitions actually scanned and can over-block a governed query that really touches one partition. - solution: opt-in SPI ConnectorScanPlanProvider.scannedPartitionCount (default empty, mirroring supportsTableSample). A pure fe-core helper resolveSelectedPartitionNum prefers the connector's count when present and not under COUNT(*) pushdown (collapsed ranges lose per-partition info), else keeps the conservative Nereids count. paimon counts distinct getPartitionValues(); iceberg counts distinct specId|partitionDataJson (new IcebergScanRange.getScannedPartitionKey). hive/MaxCompute don't override -> keep the Nereids count (they coincide). Generic node stays connector-agnostic (no source branch). - tests: fe-core resolveSelectedPartitionNum helper (8/8), paimon scannedPartitionCount (5/5), iceberg scannedPartitionCount + getScannedPartitionKey (4/4) - all with RED-able distinct-count assertions; full paimon 356/356 + iceberg 978/978 green, 0 Checkstyle. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../api/scan/ConnectorScanPlanProvider.java | 24 +++++ .../iceberg/IcebergScanPlanProvider.java | 29 ++++++ .../connector/iceberg/IcebergScanRange.java | 17 ++++ ...ergScanPlanProviderPartitionCountTest.java | 96 +++++++++++++++++++ .../paimon/PaimonScanPlanProvider.java | 31 ++++++ .../PaimonScanPlanProviderCapabilityTest.java | 52 ++++++++++ .../datasource/PluginDrivenScanNode.java | 37 +++++++ ...luginDrivenScanNodePartitionCountTest.java | 35 +++++++ 8 files changed, 321 insertions(+) create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderPartitionCountTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index 7a70b465a02979..fe2d68e0a875c7 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -29,6 +29,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; /** * Plans the set of scan ranges (splits) needed to read a connector table. @@ -268,6 +269,29 @@ default boolean supportsTableSample() { return false; } + /** + * The number of DISTINCT native partitions among the just-planned scan ranges — the count the + * connector's SDK actually resolved after ITS full manifest/residual/transform/bucket pruning. + * Feeds the scan node's {@code selectedPartitionNum} (EXPLAIN {@code partition=N/M} and the + * {@code sql_block_rule} {@code partition_num} guard), so the reported count reflects what is + * really scanned rather than the engine's declared-partition-column (Nereids) prune count. + * + *

    The default returns {@link OptionalLong#empty()}, so the generic node keeps its Nereids-pruned + * count — correct for directory-partitioned / requiredPartition-driven connectors (hive, MaxCompute), + * where the two coincide. A predicate-driven connector whose SDK prunes beyond the engine (Paimon + * manifest pruning, Iceberg hidden/transform partitioning) overrides this. Mirrors the + * {@link #supportsTableSample} opt-in shape; the connector downcasts its OWN range type (it produced + * these very ranges) to read partition identity, so the generic node stays connector-agnostic. It + * must never over-count (each native partition must map to exactly one identity within a scan), so it + * can only tighten, never loosen, the {@code partition_num} guard relative to the Nereids count.

    + * + * @param scanRanges the ranges this provider just returned from {@code planScan} + * @return the distinct scanned-partition count, or empty to keep the engine's Nereids count (default) + */ + default OptionalLong scannedPartitionCount(List scanRanges) { + return OptionalLong.empty(); + } + /** * Plans the scan for a single batch of partitions (used by batch-mode scans). * diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index badebe18cbd28f..2e3ac2d9814594 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -84,12 +84,14 @@ import java.util.ArrayList; import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; import java.util.Map; import java.util.NoSuchElementException; import java.util.Optional; +import java.util.OptionalLong; import java.util.Set; import java.util.function.Function; @@ -262,6 +264,33 @@ public boolean ignorePartitionPruneShortCircuit() { return true; } + /** + * The distinct scanned partitions among the just-planned ranges (FIX-L12) — restores legacy + * {@code IcebergScanNode}'s {@code selectedPartitionNum = partitionMapInfos.size()} (keyed by + * {@code (PartitionData) file().partition()}) so EXPLAIN {@code partition=N/M} and + * {@code sql_block_rule} reflect the partitions iceberg's manifest/residual evaluation actually + * resolved — including hidden/transform partitioning ({@code days(ts)}, {@code bucket(n,id)}) that the + * engine's declared-column Nereids pruning cannot see. The identity is + * {@link IcebergScanRange#getScannedPartitionKey()} ({@code specId|partitionDataJson}), which is + * distinct-faithful for a single spec's transform partitions. Returns empty when no range carries a + * partition key (unpartitioned table), so the engine keeps its own count. Only counts this provider's + * own {@link IcebergScanRange} instances. + */ + @Override + public OptionalLong scannedPartitionCount(List scanRanges) { + Set distinctPartitions = new HashSet<>(); + for (ConnectorScanRange range : scanRanges) { + if (range instanceof IcebergScanRange) { + String key = ((IcebergScanRange) range).getScannedPartitionKey(); + if (key != null) { + distinctPartitions.add(key); + } + } + } + return distinctPartitions.isEmpty() + ? OptionalLong.empty() : OptionalLong.of(distinctPartitions.size()); + } + /** * Iceberg metadata tables legally time-travel ({@code t$snapshots FOR TIME/VERSION AS OF ...}, * {@code t$files@branch('b')}): legacy {@code IcebergScanNode.createTableScan} honors the pin via diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java index 95ca6637ea973e..492ebc0ff7ed0d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanRange.java @@ -189,6 +189,23 @@ public Map getPartitionValues() { return partitionValues; } + /** + * A distinct-faithful key for the native iceberg partition this file belongs to (FIX-L12), or + * {@code null} when the file carries no {@code PartitionData} (unpartitioned / current spec + * unpartitioned). Combines the partition-spec id with the serialized {@code PartitionData} + * ({@code partitionDataJson}) so that two files are keyed equal iff they share the same spec and + * partition tuple — mirroring legacy {@code IcebergScanNode}'s de-dup by + * {@code (PartitionData) file().partition()}, and disambiguating cross-spec value collisions + * (e.g. {@code identity(id)=2} vs {@code bucket(id)=2}) that the value-only json would merge. + * {@code IcebergScanPlanProvider} counts distinct non-null keys for {@code selectedPartitionNum}. + */ + String getScannedPartitionKey() { + if (partitionDataJson == null) { + return null; + } + return partitionSpecId + "|" + partitionDataJson; + } + /** * Iceberg partition values always come from table/file metadata, never from a Hive-style * {@code key=value} directory layout, so the engine must NEVER fall back to path parsing for an iceberg diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderPartitionCountTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderPartitionCountTest.java new file mode 100644 index 00000000000000..cc37e9feeb821b --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanPlanProviderPartitionCountTest.java @@ -0,0 +1,96 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.api.scan.ConnectorScanRange; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; +import java.util.OptionalLong; + +/** + * FIX-L12 — guards {@link IcebergScanPlanProvider#scannedPartitionCount} and the + * {@link IcebergScanRange#getScannedPartitionKey()} identity it counts, which restore legacy + * {@code IcebergScanNode}'s {@code selectedPartitionNum = partitionMapInfos.size()} (keyed by + * {@code (PartitionData) file().partition()}). + * + *

    Why this matters: for a hidden/transform-partitioned iceberg table ({@code days(ts)}, + * {@code bucket(n,id)}) the engine's Nereids pruning only sees the declared source columns and cannot map + * a predicate on {@code ts} to the {@code days(ts)} partition, so it over-reports the scanned-partition + * count (feeding EXPLAIN {@code partition=N/M} and the {@code sql_block_rule} {@code partition_num} guard). + * The connector, which resolves the real partitions via manifest/residual evaluation, reports the faithful + * distinct count. The key is {@code specId|partitionDataJson} so two files are counted equal iff they share + * the same spec and partition tuple — disambiguating cross-spec value collisions the value-only json merges.

    + */ +public class IcebergScanPlanProviderPartitionCountTest { + + private static IcebergScanRange range(String path, Integer specId, String partitionDataJson) { + IcebergScanRange.Builder builder = new IcebergScanRange.Builder().path(path); + if (specId != null) { + builder.partitionSpecId(specId); + } + if (partitionDataJson != null) { + builder.partitionDataJson(partitionDataJson); + } + return builder.build(); + } + + @Test + public void scannedPartitionKeyCombinesSpecAndData() { + // The identity is specId|partitionDataJson; null when the file carries no PartitionData (unpartitioned). + Assertions.assertEquals("0|[\"1\"]", range("/t/f.parquet", 0, "[\"1\"]").getScannedPartitionKey()); + Assertions.assertNull(range("/t/f.parquet", null, null).getScannedPartitionKey()); + } + + @Test + public void scannedPartitionKeyDisambiguatesCrossSpecCollision() { + // identity(id)=2 (spec 0) vs bucket(id)=2 (spec 1) both render ["2"] but are DISTINCT partitions; + // including the spec id keeps them apart, matching legacy's PartitionData-object de-dup. + Assertions.assertNotEquals( + range("/t/a.parquet", 0, "[\"2\"]").getScannedPartitionKey(), + range("/t/b.parquet", 1, "[\"2\"]").getScannedPartitionKey()); + } + + @Test + public void scannedPartitionCountReturnsDistinctPartitions() { + // FIX-L12 THE load-bearing RED assertion: three files over TWO distinct partitions (two files in + // partition [1] + one in [2]) count 2, not 3. A mutation dropping the override (default + // OptionalLong.empty()) makes this red. + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), null); + List ranges = Arrays.asList( + range("/t/a.parquet", 0, "[\"1\"]"), + range("/t/b.parquet", 0, "[\"1\"]"), + range("/t/c.parquet", 0, "[\"2\"]")); + Assertions.assertEquals(OptionalLong.of(2L), provider.scannedPartitionCount(ranges)); + } + + @Test + public void scannedPartitionCountEmptyForUnpartitionedTable() { + // No range carries a partition key (unpartitioned) -> report nothing so the engine keeps its count. + // (Same value as the un-overridden default; documents the fall-through, not RED-able.) + IcebergScanPlanProvider provider = new IcebergScanPlanProvider(Collections.emptyMap(), null); + List ranges = Arrays.asList( + range("/t/a.parquet", null, null), + range("/t/b.parquet", null, null)); + Assertions.assertEquals(OptionalLong.empty(), provider.scannedPartitionCount(ranges)); + } +} diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java index 0d619b7d528b48..fd1869dd127e7e 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java @@ -86,11 +86,14 @@ import java.util.Base64; import java.util.Collections; import java.util.HashMap; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Locale; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; +import java.util.Set; import java.util.stream.Collectors; /** @@ -346,6 +349,34 @@ public boolean ignorePartitionPruneShortCircuit() { return true; } + /** + * The distinct scanned partitions among the just-planned ranges (FIX-L12) — restores legacy + * {@code PaimonScanNode}'s {@code selectedPartitionNum = partitionInfoMaps.size()} (keyed by + * {@code dataSplit.partition()}) so EXPLAIN {@code partition=N/M} and {@code sql_block_rule} reflect + * the partitions the paimon SDK actually resolved after manifest/file-stat pruning, not the engine's + * declared-column Nereids count. The identity is the rendered {@link PaimonScanRange#getPartitionValues()} + * map — {@code getPartitionInfoMap(table, dataSplit.partition(), tz)}, deterministic and injective per + * partition within a scan, so distinct maps == distinct native partitions (multiple ranges of one + * partition de-dup). Returns empty for an unpartitioned table (every range's partition map is empty), + * so the engine keeps its own count. A partition column whose type is unserializable makes + * {@code getPartitionInfoMap} drop the whole map to empty too → empty here → engine keeps the (safe, + * ≥ real) Nereids count. Only counts this provider's own {@link PaimonScanRange} instances. + */ + @Override + public OptionalLong scannedPartitionCount(List scanRanges) { + Set> distinctPartitions = new HashSet<>(); + for (ConnectorScanRange range : scanRanges) { + if (range instanceof PaimonScanRange) { + Map partitionValues = range.getPartitionValues(); + if (partitionValues != null && !partitionValues.isEmpty()) { + distinctPartitions.add(partitionValues); + } + } + } + return distinctPartitions.isEmpty() + ? OptionalLong.empty() : OptionalLong.of(distinctPartitions.size()); + } + /** * COUNT(*)-pushdown-aware scan entry (FIX-COUNT-PUSHDOWN). The generic {@code PluginDrivenScanNode} * forwards the no-grouping {@code COUNT(*)} signal here via the SPI's count-pushdown overload. diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderCapabilityTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderCapabilityTest.java index 5147af54ec6107..f58bb1c21fbc39 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderCapabilityTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanPlanProviderCapabilityTest.java @@ -18,11 +18,17 @@ package org.apache.doris.connector.paimon; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; +import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.OptionalLong; /** * Guards the predicate-driven scan capability: paimon must opt OUT of the FE prune-to-zero short-circuit @@ -48,4 +54,50 @@ public void spiDefaultKeepsShortCircuit() { Assertions.assertFalse(defaultProvider.ignorePartitionPruneShortCircuit(), "the SPI default must keep the prune-to-zero short-circuit"); } + + private static Map part(String key, String value) { + Map m = new HashMap<>(); + m.put(key, value); + return m; + } + + private static PaimonScanRange rangeWithPartition(String path, Map partitionValues) { + return new PaimonScanRange.Builder() + .path(path) + .partitionValues(partitionValues) + .build(); + } + + @Test + public void scannedPartitionCountReturnsDistinctPartitions() { + // FIX-L12 THE load-bearing RED assertion: paimon restores legacy selectedPartitionNum = + // distinct dataSplit.partition() (== distinct rendered getPartitionValues() maps). Three ranges + // over TWO partitions (two ranges of dt=1 + one of dt=2) must count 2, not 3. A mutation that + // drops the override (default OptionalLong.empty()) makes this red. + PaimonScanPlanProvider provider = new PaimonScanPlanProvider(Collections.emptyMap(), null); + List ranges = Arrays.asList( + rangeWithPartition("/t/dt=1/a.parquet", part("dt", "1")), + rangeWithPartition("/t/dt=1/b.parquet", part("dt", "1")), + rangeWithPartition("/t/dt=2/c.parquet", part("dt", "2"))); + Assertions.assertEquals(OptionalLong.of(2L), provider.scannedPartitionCount(ranges)); + } + + @Test + public void scannedPartitionCountEmptyForUnpartitionedTable() { + // Every range's partition map is empty (unpartitioned table) -> report nothing so the engine keeps + // its own count. (Same value as the un-overridden default; documents the fall-through, not RED-able.) + PaimonScanPlanProvider provider = new PaimonScanPlanProvider(Collections.emptyMap(), null); + List ranges = Arrays.asList( + rangeWithPartition("/t/a.parquet", Collections.emptyMap()), + rangeWithPartition("/t/b.parquet", Collections.emptyMap())); + Assertions.assertEquals(OptionalLong.empty(), provider.scannedPartitionCount(ranges)); + } + + @Test + public void scannedPartitionCountDefaultsToEmpty() { + // The SPI default (non-overriding connector, e.g. hive/MaxCompute) reports nothing. + ConnectorScanPlanProvider defaultProvider = (session, handle, columns, filter) -> Collections.emptyList(); + Assertions.assertEquals(OptionalLong.empty(), + defaultProvider.scannedPartitionCount(Collections.emptyList())); + } } diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index a76d5775f0c2fc..00938e64324c30 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -76,6 +76,7 @@ import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; import java.util.Random; import java.util.Set; import java.util.concurrent.CompletableFuture; @@ -307,6 +308,31 @@ static long[] displayPartitionCounts(SelectedPartitions selectedPartitions) { selectedPartitions.selectedPartitions.size(), selectedPartitions.totalPartitionNum}; } + /** + * The {@code selectedPartitionNum} to surface (EXPLAIN {@code partition=N/M} + {@code sql_block_rule} + * {@code partition_num}): the connector's real scanned-partition count when it reports one + * ({@link ConnectorScanPlanProvider#scannedPartitionCount}), else the engine's Nereids-pruned count + * (the {@code displayPartitionCounts} value already set on the node). + * + *

    A predicate-driven connector (paimon manifest pruning, iceberg hidden/transform partitioning) + * scans fewer distinct partitions than Nereids' declared-partition-column pruning can see, so its + * reported count is the faithful one. Directory-partitioned / requiredPartition-driven connectors + * (hive, MaxCompute) report {@code empty} and keep the Nereids count (the two coincide).

    + * + *

    Suppressed under {@code COUNT(*)} pushdown: the connector collapses its splits into one count + * range (paimon {@code countRepresentative}, iceberg's single count range), so per-partition info is + * lost from the returned ranges — the engine keeps its conservative Nereids count there (Nereids + * ≥ real, so the {@code partition_num} guard only tightens, never under-blocks). Pure so the gate + * is unit-testable.

    + */ + static long resolveSelectedPartitionNum(long nereidsSelectedPartitionNum, boolean countPushdown, + OptionalLong connectorScannedPartitionCount) { + if (!countPushdown && connectorScannedPartitionCount.isPresent()) { + return connectorScannedPartitionCount.getAsLong(); + } + return nereidsSelectedPartitionNum; + } + @Override public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { StringBuilder output = new StringBuilder(); @@ -1067,6 +1093,17 @@ public List getSplits(int numBackends) throws UserException { // ConnectorScanRange getters (default false / -1), so non-paimon connectors are unaffected. this.nativeReadSplitNum = countNativeReadRanges(ranges); this.totalReadSplitNum = ranges.size(); + // FIX-L12: prefer the connector's real scanned-partition count (distinct native partitions after + // its SDK's manifest/residual/transform pruning) over the Nereids declared-column prune count set + // at displayPartitionCounts above, so partition=N/M and sql_block_rule reflect what is actually + // scanned. Opt-in: the default returns empty and the Nereids count stands (correct for + // hive/MaxCompute, where the two coincide). Suppressed under COUNT(*) pushdown (collapsed ranges + // lost per-partition info). connector-agnostic: one uniform SPI call + a pure helper, no source + // branch — the connector downcasts its own range type to read partition identity. + OptionalLong connectorScannedPartitions = onPluginClassLoader(scanProvider, + () -> scanProvider.scannedPartitionCount(ranges)); + this.selectedPartitionNum = resolveSelectedPartitionNum( + this.selectedPartitionNum, countPushdown, connectorScannedPartitions); long pushDownRowCount = resolvePushDownRowCount(countPushdown, ranges); if (pushDownRowCount >= 0) { // Only set when a range actually carries a precomputed count (e.g. paimon's collapsed count diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java index 115f96b378c8ee..41a2ed4a90af3b 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodePartitionCountTest.java @@ -27,6 +27,7 @@ import java.util.Collections; import java.util.LinkedHashMap; import java.util.Map; +import java.util.OptionalLong; /** * FIX-EXPLAIN-PARTITION-COUNT — guards {@link PluginDrivenScanNode#displayPartitionCounts}, which @@ -97,4 +98,38 @@ public void testPrunedToZeroReportsZeroOverTotal() { Assertions.assertArrayEquals(new long[] {0, 2}, PluginDrivenScanNode.displayPartitionCounts(prunedToZero)); } + + // FIX-L12 — guards resolveSelectedPartitionNum, which prefers the connector's real scanned-partition + // count (distinct native partitions after the connector's SDK manifest/residual/transform pruning) over + // the engine's Nereids declared-column prune count, so partition=N/M and sql_block_rule reflect what is + // actually scanned. WHY it matters: for a predicate-driven connector (iceberg days(ts) hidden + // partitioning, paimon non-partition-column manifest pruning) the Nereids count OVER-reports the scanned + // partitions (it can only see declared partition columns) and would over-block a governed query that + // really touches one partition. + + @Test + public void testConnectorCountOverridesNereidsWhenNotCountPushdown() { + // THE load-bearing RED assertion: a connector that reports 1 real scanned partition wins over the + // Nereids count of 30 (e.g. iceberg WHERE ts= over a days(ts) table). A mutation that drops + // the override and keeps the Nereids number makes this red. + Assertions.assertEquals(1L, + PluginDrivenScanNode.resolveSelectedPartitionNum(30L, false, OptionalLong.of(1L))); + } + + @Test + public void testCountPushdownKeepsNereidsCount() { + // Under COUNT(*) pushdown the connector collapses its splits into one count range, so its + // per-partition info is lost — keep the conservative Nereids count (>= real, never under-blocks) + // even though the connector reports a (collapsed, wrong) 1. + Assertions.assertEquals(30L, + PluginDrivenScanNode.resolveSelectedPartitionNum(30L, true, OptionalLong.of(1L))); + } + + @Test + public void testEmptyConnectorCountKeepsNereidsCount() { + // A connector that does not report a scanned-partition count (hive/MaxCompute, where the Nereids + // count already equals the real one) keeps the engine's Nereids count. + Assertions.assertEquals(5L, + PluginDrivenScanNode.resolveSelectedPartitionNum(5L, false, OptionalLong.empty())); + } } From 1b9e0c8d41970cd981f97144f83e12dbaf2b6f7c Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 08:43:18 +0800 Subject: [PATCH 323/334] [doc](catalog) #65185 reverify: L12 done (option B, scannedPartitionCount SPI) - mark L12 done (e5de7aedcd5); add FIX-L12 design + summary. - design red-teamed (wf_f1524868-4b8, 3 lenses all SOUND_WITH_CHANGES); all major/minor findings folded in (iceberg streaming inert = legacy parity, cross-spec key collision fixed via specId, query-cache third consumer registered benign, unregistered under-count edges logged). - update HANDOFF: this session = L15 + L12 done; next = L16-L19, decision-class remaining only L20. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 14 +- plan-doc/task-list-65185-reverify-fixes.md | 15 +- plan-doc/tasks/designs/FIX-L12-design.md | 184 +++++++++++++++++++++ plan-doc/tasks/designs/FIX-L12-summary.md | 53 ++++++ 4 files changed, 260 insertions(+), 6 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L12-design.md create mode 100644 plan-doc/tasks/designs/FIX-L12-summary.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 419db16a505426..2487e8e4fb8ff6 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -5,7 +5,19 @@ --- -# 🆕 本 session（对象存储读修复）= hudi 2 fix + hive 1 fix DONE，e2e 待用户自跑 +# 🆕 本 session（2026-07-13）= #65185 复核修复 L15 + L12 DONE + +> `plan-doc/task-list-65185-reverify-fixes.md` 单任务循环续做。**两条全 DONE**： +> - **L15** `b2cdf971889`(code)+`4e6c08d9bea`(doc) — 删 `SummaryProfile` 三处 `PAIMON_SCAN_METRICS` 死引用(P5 `dbc38a265e5` 删写入方后无 reporter 填充;对照活的 `ICEBERG_SCAN_METRICS` 有 `IcebergMetricsReporter` 保留不动)。零行为变更、无新测、非 live;fe-core BUILD SUCCESS + 0 checkstyle。 +> - **L12** `e5de7aedcd5`(code)+ doc — **用户签字选项 B**:`selectedPartitionNum`(喂 EXPLAIN `partition=N/M`+`sql_block_rule partition_num` 治理)用**连接器 SDK 规划后真实 distinct 分区数**,非迁移后的 Nereids 剪枝数(隐藏/transform 分区高报→治理误拦)。新 opt-in SPI `ConnectorScanPlanProvider.scannedPartitionCount`(默认 empty,仿 `supportsTableSample`)+ fe-core 纯 helper `resolveSelectedPartitionNum`(`!countPushdown && present` 用连接器数,否则 Nereids)+ paimon(distinct `getPartitionValues()`)/iceberg(distinct `specId|partitionDataJson`,新 `IcebergScanRange.getScannedPartitionKey`);hive/MC 不 override 保留 Nereids。**通用节点无源分支**(铁律)。设计 3-lens 对抗红队 `wf_f1524868-4b8` 全 SOUND_WITH_CHANGES,major/minor 全折入。8/8+5/5+4/4 RED-able UT、paimon 356/356+iceberg 978/978 全绿、0 checkstyle、import 门净。memory `catalog-spi-selected-partition-num-sdk-distinct`。 +> +> **L12 e2e live-gated**(须真集群):隐藏/transform 分区 iceberg 表(**<1024 文件 或 `enable_external_table_batch_mode=false`**,否则走 streaming batch 报 `partition=0/0` 假绿)`WHERE ts` 单日谓词 → `partition=1/M`;paimon 非分区列 manifest 剪枝 → 真实数;各配 sql_block_rule partition_num 门。 +> +> **下一步**:#65185 复核系列剩余 = `L16`(iceberg 缓存偏斜)、`L17`(iceberg version-blind schema)、`L18`(iceberg 未知类型)、`L19`(partition_columns 撞名);⏸ 决策类**仅余 `L20`**(MC EQ `==`,须先中文讲清问用户)。⏸ `M8`(升级文档)用户 07-12 跳过、`D-系列`设计债随 P8。 + +--- + +# 上个 session（对象存储读修复）= hudi 2 fix + hive 1 fix DONE，e2e 待用户自跑 > 用户按最新代码重跑 = **14 个 hudi p2 suite 全失败**（唯 `test_hudi_meta` 过）。两类错误签名、同一主题：新 fe-connector hudi BE-scan 路径**没把 catalog 的 S3 storageProperties 接到 BE**（相对 legacy `HudiScanNode` 的 parity gap，**非环境/非文案**）。两个独立接线缺口，各独立 commit： > - **native `Invalid S3 URI: s3a://` (10 suite)** = `a26eaf46b9f`：BE `S3URI` 只认 `s3/http/https`，连接器把原始 `s3a://` 直发 `TFileRangeDesc.path`。修=连接器侧 `context.normalizeStorageUri` 归一化**native** range `.path()`（镜像 iceberg/paimon），**三处** native emitter：`collectCowSplits`/`buildMorRange`/**`COWIncrementalRelation.collectSplits`(COW @incr，红队补的第三处)**；JNI `THudiFileDesc` 路径留原始 `s3a://`。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 375f1d6df933a9..8217898caff08a 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -48,7 +48,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 21 | **L9** | 🟡低 | maxcompute | 谓词下推全有全无 | ✅ | ✅ | ✅ | ✅ | | 22 | **L10** | 🟡低 | fe-core | EXPLAIN 节点名 VPluginDrivenScanNode | — | — | — | ✅ 登记 DV-050 | | 23 | **L11** | 🟡低 | paimon | JNI/COUNT file_format 用表级默认 | ✅ | ✅ | ✅ | ✅ | -| 24 | **L12** | 🟡低 | fe-core/paimon | selectedPartitionNum 语义(登记或对齐) | ⬜ | ⬜ | ⬜ | ⏸ 需决策 | +| 24 | **L12** | 🟡低 | fe-core/paimon/iceberg | selectedPartitionNum 语义(能力 SPI 回报真实数) | ✅ | ✅ | ✅ | ✅ | | 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ✅ | ✅ | ✅ | ✅ | | 26 | **L14** | 🟡低 | paimon | ignore_split_type 静默 no-op | ✅ | ✅ | ✅ | ✅ | | 27 | **L15** | 🟡低 | fe-core | PAIMON_SCAN_METRICS 悬空常量 | ✅ | ✅ | ✅ | ✅ | @@ -68,7 +68,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **批次 2(中危·fe-core 通用节点,blast radius 较大,单测充分后再动)**：`M3`、`M1`。 - **批次 3(运营/门禁)**：`M8`(发布工具+文档,无 fe 编译)、`L1`(gate)。 - **批次 4(低危·连接器)**：`L3–L6`(trino)、`L7/L8`(kerberos)、`L9/L20`(mc)、`L11/L13/L14`(paimon)、`L18`(iceberg)。 -- **批次 5(需决策,先问用户再动)**：~~`L2`(SQL 缓存)~~ **DONE `c9a86337906` 恢复缓存**、~~`L10`(EXPLAIN 名)~~ **DONE 用户签字 accept [DV-050]**;**余** `L12`(selectedPartitionNum:登记 vs 对齐)、`L20`(MC EQ 写法)。 +- **批次 5(需决策,先问用户再动)**：~~`L2`(SQL 缓存)~~ **DONE `c9a86337906` 恢复缓存**、~~`L10`(EXPLAIN 名)~~ **DONE 用户签字 accept [DV-050]**、~~`L12`(selectedPartitionNum)~~ **DONE `e5de7aedcd5` 用户签字选项 B=能力 SPI 回报真实数**;**余** `L20`(MC EQ 写法)。 - **批次 6(设计债/P8)**：`D-系列`,择机或随 P8。`D-PRUNE` 因承载 H1/H3,提前到批次 0。 > 决策类(⏸)条目**先在 session 里用中文讲清背景+选项问用户**(memory `ask-user-explain-in-chinese-first`),别擅自选一路实现。 @@ -189,7 +189,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L10** DONE（**用户 2026-07-12 签字 accept**,登记 [DV-050],**无代码**）· EXPLAIN 外部表扫描节点显示通用名 `VPluginDrivenScanNode`(`PluginDrivenScanNode:173` super label)。**决策=接受为 display-only 偏差**(非加 `Connector.getLegacyEngineName` SPI 恢复旧名):`getNodeExplainString:325` 已附 `CONNECTOR: ` 披露数据源、regression 黄金文件全树仅 1 处引用且已是新名、且与 Trino 一致(统一 `TableScan` 通用名 + 连接器作属性)。否决 Option B(catalog type 拼名会误出 `VHMS_SCAN_NODE`;改动面大)。关联 D-ENGINE 择机随 P8。 - [x] **L11** DONE `4a8650bd062`(设计 `FIX-L11-design.md`;红队 `wf_05574ccb-bd2` 3 lens SOUND/SOUND_WITH_CHANGES）· JNI DataSplit + COUNT 路改按**首数据文件后缀**取 file_format(新 package-private `dataSplitFileFormat`,legacy `getFileFormat(getPathString())` parity),回退表级默认;顺补 `getFileFormatBySuffix` 的 `.avro` 臂(legacy `FileFormatUtils` parity,inert on native)。**call-site RED 测**(`Table.copy` 令表默认 parquet≠磁盘 .orc,断言 JNI+COUNT range 携 "orc")——原 helper 孤立测不守护接线(红队 MAJOR)。69/69 绿、0 checkstyle、gate 净。默认 JNI reader 不消费 file_format,影响窄(仅 opt-in cpp reader)。e2e live-gated(cpp reader over 混/改格式表)。 -- [ ] **L12** ⏸ selectedPartitionNum 语义 · `PluginDrivenScanNode.java:297-303`。**需决策**:登记为「paimon/iceberg 对齐 MC/hive 的 Nereids-剪枝数」验收偏差(推荐)**vs** 连接器回报 SDK-distinct(重,不推荐)。同步 paimon/iceberg EXPLAIN `partition=N/M` 回归期望。**勿**在通用节点按源分支。 +- [x] **L12** DONE `e5de7aedcd5`(设计 `designs/FIX-L12-design.md`;summary `FIX-L12-summary.md`;设计 3-lens 对抗红队 `wf_f1524868-4b8` 全 SOUND_WITH_CHANGES,major/minor 全折入)· **用户 2026-07-13 签字选项 B**(连接器回报 SDK-distinct 真实扫描分区数,非登记偏差)。新 opt-in SPI `ConnectorScanPlanProvider.scannedPartitionCount`(默认 empty,仿 `supportsTableSample`)+ fe-core 纯 helper `resolveSelectedPartitionNum`(`!countPushdown && present` 用连接器数,否则 Nereids;getSplits 覆写)+ paimon(distinct `getPartitionValues()`)/iceberg(distinct `specId|partitionDataJson`,新 `IcebergScanRange.getScannedPartitionKey`)override;hive/MC 不 override 保留 Nereids(本就相等)。**通用节点无源分支**(铁律)。8/8+5/5+4/4 RED-able UT,paimon 356/356+iceberg 978/978 全绿,0 checkstyle,import 门净。登记残余:COUNT(*) 保守 Nereids、iceberg binary/null-bucket undercount、streaming-iceberg(显式开 batch+≥1024 文件)inert=legacy parity、query-cache 消费者 benign。**e2e live-gated**(隐藏/transform 分区 <1024 文件 batch-off `WHERE ts` 单日→`partition=1/M`)。 - [x] **L13** DONE `ced4775b844`(设计 `FIX-L13-design.md`;红队 3 lens 全 SOUND）· `toPaimonType` 对 ARRAY 元素/MAP value/STRUCT 字段加 `.copy(type.isChildNullable(i))`(MAP key 保持 `.copy(false)`),恢复 legacy `DorisToPaimonTypeVisitor` 嵌套 nullability parity。**scope=仅 nullability**:comment 丢=DV-035 M10.1 已接受偏差、field-id 顺序 parity 均不动。`.copy(true)` 对默认可空子类型逐字节恒等(既有 parity 测保持绿)。3 新 RED-able 测(经 `.type().isNullable()` 断言,非 DataField equals)。type-mapping+schema-builder 26/26 绿。e2e live-gated(建嵌套 NOT NULL 表 DESCRIBE/SDK 读回)。 - [x] **L14** DONE `478718aca6f`(设计 `FIX-L14-design.md`;红队 3 lens SOUND/SOUND_WITH_CHANGES）· null-tolerant `resolveIgnoreSplitType(session)`(镜像 `isCppReaderEnabled`,红队 MINOR)+三 legacy `continue` 位:`IGNORE_JNI` drops nonDataSplit+DataSplit-JNI 臂、`IGNORE_NATIVE` drops native 臂、count 臂不检、`IGNORE_PAIMON_CPP` 保持 no-op(=legacy parity,全树 grep 证 legacy 从不引用)。默认 NONE 逐字节不变。3 新 live-planScan RED 测(IGNORE_JNI/IGNORE_NATIVE 各证跳 split+IGNORE_PAIMON_CPP==NONE);nonDataSplit IGNORE_JNI 位离线不可测→留 E2E-only。69/69 绿。e2e live-gated(真集群 SET ignore_split_type 断言跳分片)。 - [x] **L15** DONE `b2cdf971889`(设计 `designs/FIX-L15-design.md`;summary `FIX-L15-summary.md`)· 删 `SummaryProfile.java` 三处 `PAIMON_SCAN_METRICS` 死引用(常量 + `EXECUTION_SUMMARY_KEYS` + 缩进 map);P5(`dbc38a265e5`)删写入方后无任何 reporter 填充该分组(对照活的 `ICEBERG_SCAN_METRICS` 有 `IcebergMetricsReporter` 保留不动)。**未加** connector-neutral scan-metrics SPI(feature、非必需,超出清死引用 scope)。零行为变更、无新测(死码删除无可断言);fe-core test-compile BUILD SUCCESS + 0 checkstyle。**非 live**(该 profile 列本就从不出现)。 @@ -306,10 +306,15 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) **⭐ L15 DONE `b2cdf971889`(2026-07-13)** — 删 `SummaryProfile` 三处 `PAIMON_SCAN_METRICS` 死引用(P5 后无 reporter 填充,对照活的 iceberg metrics 保留)。零行为变更、无新测、非 live;fe-core BUILD SUCCESS + 0 checkstyle。设计/summary = `designs/FIX-L15-{design,summary}.md`。 -**⭐ L12 侦察完成(2026-07-13,workflow `wf_6c516483-c34`,4 agent recon)= 待用户决策**： +**⭐ L12 DONE `e5de7aedcd5`(2026-07-13,用户签字选项 B)** — selectedPartitionNum 用连接器 SDK-distinct 真实扫描分区数(能力 SPI opt-in `scannedPartitionCount` + fe-core 纯 helper `resolveSelectedPartitionNum` + paimon/iceberg override,通用节点无源分支);设计经 3-lens 对抗红队(`wf_f1524868-4b8`)全 SOUND_WITH_CHANGES,major(iceberg streaming inert=legacy parity、iceberg 跨 spec 值撞加 specId 解、query-cache 第三消费者 benign)+ minor 全折入。8/8+5/5+4/4 RED-able UT、paimon 356/356+iceberg 978/978 全绿、0 checkstyle、import 门净。设计/summary=`designs/FIX-L12-{design,summary}.md`。**决策类仅余 L20**。 + +
    L12 侦察原始记录(2026-07-13,workflow `wf_6c516483-c34`,4 agent recon) + - **确认 iceberg 也受影响**:iceberg 读实走 `PluginDrivenScanNode`(legacy `IcebergScanNode.java` 已死;`PhysicalPlanTranslator:812-829` instanceof `PluginDrivenExternalTable` 臂)。故 L12 覆盖 paimon+iceberg 两者。 - **旧→新语义**:旧 = **SDK-distinct**(连接器 SDK 实际规划 split 后的 distinct 原生分区数;paimon `partitionInfoMaps.size()` keyed `dataSplit.partition()`、iceberg `partitionMapInfos.size()` keyed `file().partition()`);新 = **Nereids-剪枝分区项数**(`selectedPartitions.size()`,仅按声明分区列在 `LogicalFilter` 下剪,见不到 SDK manifest/residual/bucket/transform 剪枝)。新数 **≥** 旧数。 - **背离场景**:仅当 SDK 能剪而 Nereids 不能——iceberg 隐藏/transform 分区(`days(ts)` + WHERE ts:新 30/30 vs 旧 1/30)、paimon 非分区列 manifest 剪枝(WHERE id=999:新 2/2 vs 旧 1/2)。identity 分区 + 无谓词时两者相等。 - **爆炸半径 = 0 黄金 EXPLAIN 断言**:paimon/iceberg 无任何 `partition=N/M` 黄金(它们断言 split-count 指标 `inputSplitNum`/`paimonNativeReadSplits`,不受影响)。3 个 `sql_block_rule partition_num` 测(iceberg/paimon/hive)均**identity 分区 + 无谓词**→两选项同值→**两选项都不破测**。 - **Trino 对照**:Trino EXPLAIN **无**引擎统一的 per-scan 分区计数;分区计量完全 connector-owned(row estimate 由连接器 `getTableStatistics` 报,分区 guard 如 hive `max-partitions-per-scan` 在连接器内执行)。据此 Trino 更贴近 **选项 B**(连接器经能力 SPI 回报 SDK-distinct,不违铁律——能力 SPI 本身即反分支机制,仿现有 `supportsTableSample`/`Split.getLength()` opt-in)。 -- **张力(Rule 7)**:task list 原推荐 A(登记 Nereids 数为验收偏差,零代码);但 Trino 与「该数还喂 `sql_block_rule` 治理」角度支持 B(A 在隐藏分区下**高报**实际扫描分区数=治理 false-positive over-block)。已用中文向用户讲清背景+两选项+推荐,待拍板。 +- **张力(Rule 7)**:task list 原推荐 A(登记 Nereids 数为验收偏差,零代码);但 Trino 与「该数还喂 `sql_block_rule` 治理」角度支持 B(A 在隐藏分区下**高报**实际扫描分区数=治理 false-positive over-block)。已用中文向用户讲清背景+两选项+推荐 → **用户选 B**。 + +
    diff --git a/plan-doc/tasks/designs/FIX-L12-design.md b/plan-doc/tasks/designs/FIX-L12-design.md new file mode 100644 index 00000000000000..81bfea6dd583ff --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L12-design.md @@ -0,0 +1,184 @@ +# FIX-L12 — `selectedPartitionNum` 恢复为「连接器真实扫描分区数」(能力 SPI opt-in) + +> reverify §1 L12 / 原始 review P5-3(CONFIRMED medium)。严重度 🟡低(治理相关)。模块 = fe-core 通用节点 + fe-connector-paimon + fe-connector-iceberg。 +> **用户 2026-07-13 签字 = 选项 B**(加能力 SPI 回报 SDK-distinct,非登记偏差)。memory `catalog-spi-selected-partition-num-sdk-distinct`。 + +## Problem + +外部表扫描节点的 `selectedPartitionNum`(喂 EXPLAIN `partition=N/M` 的 N + `sql_block_rule` 的 `partition_num` +治理)在迁移到通用插件节点后,含义从**旧 = 连接器 SDK 规划完 split 后的真实 distinct 原生分区数**变成 +**新 = Nereids 按声明分区列剪枝后的分区项数**。新数**恒 ≥** 旧数;隐藏/transform 分区(iceberg `days(ts)`)与 +非分区列 manifest 剪枝(paimon `WHERE 数据列`)下**高报**实际扫描分区数 → 喂 `sql_block_rule` 可**误拦**本只扫 +1 个分区的查询(治理 false-positive)。 + +## HEAD recon(侦察 workflow `wf_6c516483-c34` + 直读确认) + +**通用节点(现状)**:`PluginDrivenScanNode` +- `displayPartitionCounts(selectedPartitions)`(:302-308)= `{selectedPartitions.size(), totalPartitionNum}`, + `selectedPartitions` 来自 Nereids `PruneFileScanPartition`(仅按声明分区列、仅 `LogicalFilter` 下跑,见不到 + SDK manifest/residual/transform/bucket 剪枝);默认 `NOT_PRUNED`。 +- `this.selectedPartitionNum = partitionCounts[0]` 于 `getSplits:1007-1009`(planScan 之前)、`startSplit:1309-1311` + (batch 路);渲染 `partition=N/M` 于 :357-358;`getSelectedPartitionNum()`(基类)喂 sql_block_rule。 +- **iceberg 也走此节点**(legacy `IcebergScanNode.java` 已死;`PhysicalPlanTranslator:812-829` instanceof + `PluginDrivenExternalTable` 臂);故 paimon+iceberg 两者均受影响。 + +**旧语义(SDK-distinct)**: +- paimon(`dbc38a265e5^` `PaimonScanNode.getSplits`):`Map partitionInfoMaps` keyed + `dataSplit.partition()`(:412,:421-429),遍历**全部** DataSplit(含 count-pushdown 前),末 + `selectedPartitionNum = partitionInfoMaps.size()`(:514);**`totalPartitionNum` 从不赋值**(基类默认 0)。 +- iceberg(`7ffeca95abf^` `IcebergScanNode`):`partitionMapInfos` keyed `(PartitionData) file().partition()` + (:881-884),`selectedPartitionNum = partitionMapInfos.size()`(:934/:969;metadata 路 :987=0)。 + +**分区身份可从 range 复算吗?**(决定机制) +- paimon `PaimonScanRange.getPartitionValues()`(:154)= `getPartitionInfoMap(table, dataSplit.partition(), tz)` + (planScan :515-516)——**直接由 `dataSplit.partition()` 渲染**,identity 分区(paimon 无隐藏 transform)→ + distinct map == distinct BinaryRow → **可用作 paimon 身份**。 +- iceberg `IcebergScanRange.getPartitionValues()` **只含 identity 分区列**(:69-71)→ transform 分区**不忠实**; + 但同类另带 `partitionDataJson`(:66,planScan :787 `getPartitionDataJson(partitionData, spec, zone)`,= 序列化 + 的整个 `PartitionData`)→ **iceberg 身份 = `partitionDataJson`**(null=未分区),忠实于 legacy 的 `file().partition()`。 +- ⇒ **身份逻辑因连接器而异,必须落在连接器侧**,通用节点不得渲染/理解分区(铁律 + `catalog-spi-plugindriven-no-source-specific-code`)。 + +**两处 landmine**: +1. **provider 共享**:`resolveScanProvider()`→`connector.getScanPlanProvider(handle)`(:212-213),疑共享 → + **禁在 provider 上 stash 可变计数**。 +2. **COUNT(*) 折叠**:paimon planScan 在 countPushdown 下发**单个折叠 count range**(`countRepresentative`, + :561-566)→ 返回 ranges 丢失 per-partition 信息 → 基于 ranges 复算会 undercount。iceberg COUNT 走 summary + 元数据(无 per-file range)同理。⇒ **countPushdown 下不 override,保留 Nereids 数**(保守:Nereids≥真实,治理 + 只会更严不会漏拦;且 COUNT+partition_num 是边缘、零黄金覆盖)。 + +## Design + +### SPI(fe-connector-api,additive,仿 `supportsTableSample`/`supportsBatchScan` opt-in) + +`ConnectorScanPlanProvider` 加**默认返回空**的能力方法: +```java +/** + * Distinct native partitions among the just-planned scan ranges — the count the connector's SDK + * actually resolved after ITS full manifest/residual/transform/bucket pruning. Feeds the scan node's + * selectedPartitionNum (EXPLAIN partition=N/M + sql_block_rule partition_num), so it reflects what is + * really scanned, not the engine's declared-partition-column (Nereids) prune count. + * + *

    The default returns empty, so the generic node keeps its Nereids-pruned count — correct for + * directory-partitioned / requiredPartition-driven connectors (hive, MaxCompute) where the two + * coincide. A predicate-driven connector whose SDK prunes beyond the engine (paimon manifest pruning, + * iceberg hidden/transform partitioning) overrides this. Mirrors the supportsTableSample opt-in shape; + * the connector downcasts its own range type (it produced these ranges) to read partition identity.

    + */ +default OptionalLong scannedPartitionCount(List scanRanges) { + return OptionalLong.empty(); +} +``` + +### fe-core 通用节点(`PluginDrivenScanNode`) + +`getSplits` 中 planScan 之后(拿到 `ranges`、已设 Nereids 数于 :1009),加: +```java +// L12: prefer the connector's real scanned-partition count over the Nereids declared-column prune +// count, so partition=N/M and sql_block_rule reflect what is actually scanned. Opt-in (default empty) +// keeps the Nereids count for hive/MaxCompute (where they coincide). Suppressed under COUNT(*) +// pushdown, whose collapsed ranges have lost per-partition info (keep the conservative Nereids count). +OptionalLong connectorScannedPartitions = onPluginClassLoader(scanProvider, + () -> scanProvider.scannedPartitionCount(ranges)); +this.selectedPartitionNum = resolveSelectedPartitionNum( + this.selectedPartitionNum, countPushdown, connectorScannedPartitions); +``` +纯静态 helper(**RED-able 单测**,仿 M1/M3 的 `shouldUseBatchMode`/`displayPartitionCounts` 可测模式): +```java +/** + * selectedPartitionNum to surface: the connector's real scanned-partition count when it reports one + * (non-count scans of predicate-driven connectors), else the engine's Nereids-pruned count. Suppressed + * under COUNT(*) pushdown (collapsed ranges lost per-partition info → keep the conservative, >= real + * Nereids count). Pure so the gate is unit-testable. + */ +static long resolveSelectedPartitionNum(long nereidsSelectedPartitionNum, boolean countPushdown, + OptionalLong connectorScannedPartitionCount) { + if (!countPushdown && connectorScannedPartitionCount.isPresent()) { + return connectorScannedPartitionCount.getAsLong(); + } + return nereidsSelectedPartitionNum; +} +``` +- **不动** `totalPartitionNum`(M):legacy 本就留 0;override 只在连接器回报 partition-bearing ranges 时触发 + (⇒ 声明了分区列 ⇒ `displayPartitionCounts` 已设 total,非 NOT_PRUNED),故不产生 `partition=N/0` 异常; + sql_block_rule 只读 N;零黄金覆盖。 +- **override 只在同步 `getSplits` 路**(红队订正):`selectedPartitionNum` 有**三**处写点——`getSplits:1009`、 + `startSplit:1311`(MC 分区切片 batch)、以及**`startStreamingSplit`(:1396,从不设)**。**iceberg 经 streaming + flavor 进 batch**(override `streamingSplitEstimate`,**非** `supportsBatchScan`;`computeBatchMode:1208-1217`): + 当 `enable_external_table_batch_mode=true`(**默认 false**,`SessionVariable:3037`)且匹配文件 ≥ `num_files_in_batch_mode` + (默认 1024)→ `startStreamingSplit` 不设 selectedPartitionNum(=0,**与 legacy batch parity**——legacy 也只在 + `doGetSplits` 设、batch 路留 0),本 override(仅 getSplits)对其 **inert**。**默认路(batch off)paimon+iceberg + 全尺寸均修复**;仅「显式开 batch mode + ≥1024 文件 iceberg」inert,**登记 streaming-iceberg 残余**(非回归:与 + legacy 同为 0)。paimon **无** streaming override → 永远 getSplits → 全修。 +- **不按源分支**:通用节点只调统一方法 + 纯 helper,无 `instanceof`/源名。 + +### 连接器(身份逻辑落连接器) + +**paimon** `PaimonScanPlanProvider` override:遍历 ranges,对 partition-bearing(`getPartitionValues()` 非空)的 +`PaimonScanRange` 收集 distinct `getPartitionValues()` 到 `Set`;有则返回 `OptionalLong.of(size)`,全无(未分区) +→ `OptionalLong.empty()`(通用节点保留现状)。 + +**iceberg** `IcebergScanRange` 加访问器 `getScannedPartitionKey()`(= `partitionSpecId + "|" + partitionDataJson`, +null=未分区/无 PartitionData);**含 specId**(红队订正:`partitionDataJson` 是 value-only JSON,跨 spec 会撞——如 +identity(id)=2 与 bucket(id)=2 都渲 `["2"]`;legacy 按 `PartitionData` 对象(含分区类型)本就区分 spec,故加 specId += 更贴 legacy)。`IcebergScanPlanProvider` override:遍历 ranges 收集 distinct 非 null key;有则 `of(size)`,无则 +`empty()`。访问器 package-private 即可(测试同包)。 + +## Risk Analysis / 忠实度 + +- **hive/MaxCompute 不变**:不 override → `empty` → 保留 Nereids 数(本就 == SDK-distinct)。零回归。 +- **paimon/iceberg 常态忠实,且从不 over-count**(红队核实):paimon `getPartitionValues()`=`getPartitionInfoMap( + table,dataSplit.partition(),tz)` 对 BinaryRow 确定性、单调注入(同分区→同 map、异分区→异 map);iceberg + `specId|partitionDataJson` 对 `(PartitionData,spec,zone)` 确定性,单 spec 的 transform/hidden(day/bucket/truncate) + 渲染各异→distinct(**正是本修目标场景**)。二者均**不可能 over-count**(同 PartitionData/BinaryRow 在一次扫描内 + 不会映到两个 key)→ **不引入任何超过 Nereids 的新 sql_block_rule false-positive**。 +- **登记残余偏差**(Rule 12 不静默;红队补全): + - `COUNT(*)` pushdown:保留 Nereids 数——保守 over-report(Nereids≥真实,治理只更严不漏拦),零覆盖。 + - `ignore_split_type`(L14 调试阀)丢光某分区全部 split → 该分区无 range 不计;legacy 在 `continue` 前已计 → + 可 undercount。调试态、极边缘。 + - **iceberg BINARY/FIXED identity 分区列**(⚠ undercount,governance-unsafe 方向):`serializePartitionValue` + 对 binary/fixed 抛异常→`getPartitionValues` 吞并 append null→每个 distinct binary 分区渲 `[null]`→塌成 1; + legacy 按 PartitionData 分别计。**极罕见**(binary 作分区列本不常见);加 specId 不解此例(同 spec 内仍塌)。 + - **iceberg null-partition bucket**(⚠ undercount ≤1):partitioned 表某文件 `partition()` 非 `PartitionData` + 实例→key=null→本修不计;legacy 把 null 计为一个 distinct 桶(+1)。窄触发、至多差 1。 + - **paimon BINARY/VARBINARY 分区列**(over-report,安全方向):`getPartitionInfoMap` 遇不可序列化列**整表**返空 + map→override 见空→`empty`→回退 Nereids(≥真实,安全)。非 legacy 的 distinct-BinaryRow 数,但保守。 + - paimon 极端 academic:NaN-bit float 分区值皆渲 `"NaN"` 撞;native raw file 长度 ≤0 产 0 range→该分区不计。 + - `totalPartitionNum`(M)保持现状(见上,不产生 N/0 异常)。 + - **注**:iceberg「当前 unpartitioned spec + 旧分区文件」**非**偏差(红队订正):legacy 亦按当前默认 spec + `spec().isPartitioned()` 门,current-unpartitioned 时 legacy 同样留 0 → 本修与 legacy 一致,不登记。 +- **第三消费者:query-cache**(红队补,Rule 12):`getSelectedPartitionNum()` 亦被 `CacheAnalyzer:486` + (`cacheTable.partitionNum`)→`sumOfPartitionNum:237`→SQL-cache proto `RowBatchBuilder:107 setPartitionNum` 读。 + paimon/iceberg 该存值将从 Nereids 数变 SDK-distinct 数。**核实 benign**:外表 SQL-cache 有效性键在 data-version + token `getNewestUpdateVersionOrTime`(`CacheAnalyzer:489`,非 partitionNum);partitionNum 不 gate cacheMode; + 零 paimon/iceberg query-cache 回归测。登记不阻断。关联 L2(已恢复 query-cache SPI 能力)。 +- **streaming-iceberg 残余**(见 Design §override 只在 getSplits):`enable_external_table_batch_mode=true`(默认关) + + ≥1024 文件 iceberg 走 `startStreamingSplit` → selectedPartitionNum=0(与 legacy batch parity,非回归);本修 inert。 +- **铁律**:身份逻辑全在连接器;通用节点 connector-agnostic(统一方法 + 纯 helper)。SPI 加法向后兼容,其它连接器 + (hive/hudi/mc/trino/es/jdbc)继承默认 `empty` 不受影响。 +- **blast radius**:paimon/iceberg **零** `partition=N/M` 黄金断言;3 个 `sql_block_rule partition_num` 测均 + identity 分区(4 桶)+ 无谓词、小表(<1024 文件走同步路)→ 新旧同值(4)>阈值→仍拦→不破测(红队逐条核实, + strict-`<` 语义 `SqlBlockRuleMgr:322,326`)。**新增 import**:`java.util.OptionalLong`(两文件)。 + +## Test Plan + +### Unit Tests +- **fe-core**(有 Mockito):新 `resolveSelectedPartitionNum` 纯 helper 单测——**载重 RED 断言**=connector 报数 + 且非 count→用 connector 数(RED:旧逻辑忽略,恒返 Nereids);另 guard:报数但 countPushdown→Nereids;`empty`→Nereids。 +- **paimon**(无 Mockito,真 range 直构):**载重 RED 断言**=`scannedPartitionCount` 对含 N 个 distinct + partitionValues(含同分区多 range 去重)的列表返回 `of(N)`(RED:默认返 empty);guard:全空 partitionValues→empty + (**非 RED-able**,与默认同值,仅文档)。 +- **iceberg**(无 Mockito):`getScannedPartitionKey`=`specId|partitionDataJson`;**载重 RED 断言**= + `scannedPartitionCount` 对 distinct key 计数返回 `of(N)`(RED:默认 empty);guard:未分区(null key)→empty。 + +### E2E Tests(live-gated) +- ⚠ **前置(红队订正)**:测试须 `set enable_external_table_batch_mode=false`(默认)**或**小表(<1024 文件), + 否则 iceberg 走 streaming batch → `partition=0/0`、本修 inert → 假绿。 +- 隐藏/transform 分区 iceberg 表(<1024 文件)`WHERE ts` 单日谓词 → EXPLAIN `partition=1/M`(迁移后错为 M/M); + paimon 非分区列 manifest 剪枝 → `partition=<真实>/M`;各配 sql_block_rule partition_num 门验治理。 + 真集群(memory `hms-iceberg-delegation-needs-e2e`),本地登记 gated。 + +## 验证判据 +- fe-core `test-compile` BUILD SUCCESS + 0 checkstyle;paimon `package`(shade)靶向 UT 绿;iceberg 靶向 UT 绿。 +- `bash tools/check-connector-imports.sh` exit 0(连接器不 import fe-core)。 +- 三处新 UT 均 RED-able(去掉 override/helper 即挂)。 diff --git a/plan-doc/tasks/designs/FIX-L12-summary.md b/plan-doc/tasks/designs/FIX-L12-summary.md new file mode 100644 index 00000000000000..f223af0bbbe233 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L12-summary.md @@ -0,0 +1,53 @@ +# FIX-L12 Summary — `selectedPartitionNum` 恢复为连接器真实扫描分区数(能力 SPI) + +> 用户 2026-07-13 签字**选项 B**。memory `catalog-spi-selected-partition-num-sdk-distinct`。 + +## Problem +外部表扫描节点的 `selectedPartitionNum`(喂 EXPLAIN `partition=N/M` + `sql_block_rule partition_num` 治理) +迁移后从**连接器 SDK 规划后的真实 distinct 分区数**变成 **Nereids 按声明分区列剪枝数**(恒 ≥ 真实)。隐藏/transform +分区(iceberg `days(ts)`)、非分区列 manifest 剪枝(paimon)下**高报**→ 喂治理规则可**误拦**本只扫 1 分区的查询。 + +## Root Cause +`PluginDrivenScanNode.displayPartitionCounts` 用 `selectedPartitions.size()`(Nereids `PruneFileScanPartition`, +只认声明分区列、只 `LogicalFilter` 下跑,见不到 SDK manifest/residual/transform/bucket 剪枝)。legacy 连接器 +(paimon `partitionInfoMaps.size()` keyed `dataSplit.partition()`、iceberg `partitionMapInfos.size()` keyed +`file().partition()`)按 SDK 规划后的 distinct 原生分区数报——迁移丢了这层。 + +## Fix(能力 SPI opt-in,仿 `supportsTableSample`;身份逻辑落连接器,通用节点 connector-agnostic) +- **SPI**(`ConnectorScanPlanProvider`):新 `default OptionalLong scannedPartitionCount(List)` + = `empty`(默认保留 Nereids 数)。 +- **fe-core**(`PluginDrivenScanNode`):新纯静态 `resolveSelectedPartitionNum(nereids, countPushdown, connectorCount)` + ——`!countPushdown && present` 时用连接器数,否则 Nereids;`getSplits` 中 planScan 后统一调用覆写 `selectedPartitionNum`。 + 无 `instanceof`/源名分支。 +- **paimon**(`PaimonScanPlanProvider`):override 收集 distinct 非空 `getPartitionValues()`(= 由 `dataSplit.partition()` + 渲染,单调注入),空则 `empty`。 +- **iceberg**(`IcebergScanRange` 加 `getScannedPartitionKey()`=`specId|partitionDataJson`;`IcebergScanPlanProvider` + override 收集 distinct 非 null key,空则 `empty`)。含 specId 以区分跨 spec 值撞(对齐 legacy PartitionData 对象去重)。 +- **不动**:`totalPartitionNum`(M,legacy 本留 0)、batch/streaming 路(paimon 从不 stream;iceberg 仅在显式开 + `enable_external_table_batch_mode`(默认关)+ ≥1024 文件时 stream,那路留 0=legacy batch parity)。 + +## Tests(全 RED-able 载重断言) +- fe-core `PluginDrivenScanNodePartitionCountTest` +3:connector 报数覆写 Nereids(RED)/countPushdown 保留 Nereids/ + empty 保留 Nereids。**8/8**。 +- paimon `PaimonScanPlanProviderCapabilityTest` +3:3 range/2 分区计 2(RED)/未分区 empty/默认 empty。**5/5**。 +- iceberg 新 `IcebergScanPlanProviderPartitionCountTest`:key=specId|json、跨 spec 不撞、3 range/2 分区计 2(RED)、 + 未分区 empty。**4/4**。 + +## 登记残余(Rule 12) +- `COUNT(*)` pushdown 保留 Nereids(collapsed range,保守 over-report,零覆盖)。 +- iceberg BINARY/FIXED identity 分区列(渲 `[null]` 塌成 1,⚠ undercount)、null-partition bucket(⚠ ≤1 undercount); + paimon binary 分区列(整表空 map → 回退 Nereids over-report,安全)、NaN-float/零长 native file(academic)。 +- streaming-iceberg(显式开 batch + ≥1024 文件)本修 inert(=legacy batch parity,留 0,非回归)。 +- 第三消费者 query-cache(`CacheAnalyzer` partitionNum)随之变 SDK-distinct 数,核实 benign(有效性键在 data-version + token 非 partitionNum;零相关回归测)。 + +## Result +- fe-core / paimon / iceberg 三模块 **BUILD SUCCESS + 0 Checkstyle**;靶向 UT 8/8 + 5/5 + 4/4;import 门 exit 0。 +- 3 sql_block_rule 治理测(iceberg/paimon/hive)均 identity 分区 + 无谓词 + 小表(<1024 文件同步路)→ 新旧同值不破测; + paimon/iceberg 零 `partition=N/M` 黄金断言。 +- 设计经 3-lens 对抗红队(`wf_f1524868-4b8`)全 SOUND_WITH_CHANGES,所有 major/minor 已折入设计文档 + 本 summary。 +- **e2e live-gated**:隐藏/transform 分区 iceberg(<1024 文件,batch off)`WHERE ts` 单日 → `partition=1/M`;paimon + 非分区列剪枝 → 真实数;真集群回归。 + +## Commits +- code:SPI + fe-core + paimon + iceberg(4 文件)+ 3 测试。 From 8d4209865a3ce2a7099c195dca0f92dc68ddb438 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 10:37:01 +0800 Subject: [PATCH 324/334] [feature](catalog) restore paimon/iceberg scan metrics via connector-neutral SPI - problem: the plugin migration dropped the paimon AND iceberg SDK scan metrics (manifest cache hit/miss, scan/planning durations, files and manifests scanned vs skipped) from the query profile. The connectors cannot write fe-core's SummaryProfile (iron rule), so the legacy reporters were left behind; iceberg's constant only looked live via the dead legacy IcebergScanNode. - solution: a connector-neutral opt-in SPI. New ConnectorScanProfile value type + ConnectorScanPlanProvider.collectScanProfiles (default empty, mirroring scannedPartitionCount). Connectors harvest their SDK metrics during planScan, stash keyed by queryId, and drain in collectScanProfiles; fe-core transcribes them into the profile via the pure static writeScanProfilesInto (no source branch). paimon PULLS from a ported PaimonMetricRegistry (InnerTableScan.withMetricRegistry); iceberg PUSHES via IcebergScanProfileReporter attached on the synchronous data/count path only (not shared buildScan -> no streaming stash leak). DebugUtil's time/byte formatters are self-ported (connectors can't import fe-core). Revives SummaryProfile.PAIMON_SCAN_METRICS (supersedes the L15 deletion). - tests: fe-core writeScanProfilesInto (4/4), paimon harvest via real ScanMetrics/ScanStats (4/4), iceberg report via real ImmutableScanReport (4/4) - all RED-able; group-name mirror checks. Full paimon 360/360 + iceberg 982/982 + fe-core plugin-scan 94/94 green, 0 Checkstyle, import-gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../api/scan/ConnectorScanPlanProvider.java | 20 ++ .../api/scan/ConnectorScanProfile.java | 66 ++++++ .../iceberg/IcebergScanPlanProvider.java | 39 ++++ .../iceberg/IcebergScanProfileReporter.java | 203 ++++++++++++++++++ .../IcebergScanProfileReporterTest.java | 106 +++++++++ .../paimon/PaimonMetricRegistry.java | 78 +++++++ .../connector/paimon/PaimonScanMetrics.java | 183 ++++++++++++++++ .../paimon/PaimonScanPlanProvider.java | 59 +++++ .../paimon/PaimonScanMetricsTest.java | 82 +++++++ .../doris/common/profile/SummaryProfile.java | 3 + .../datasource/PluginDrivenScanNode.java | 52 +++++ .../PluginDrivenScanNodeScanProfileTest.java | 100 +++++++++ 12 files changed, 991 insertions(+) create mode 100644 fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporter.java create mode 100644 fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporterTest.java create mode 100644 fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonMetricRegistry.java create mode 100644 fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanMetrics.java create mode 100644 fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanMetricsTest.java create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProfileTest.java diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index fe2d68e0a875c7..a0cb2c4e4fcf25 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -292,6 +292,26 @@ default OptionalLong scannedPartitionCount(List scanRanges) return OptionalLong.empty(); } + /** + * Connector SDK scan diagnostics harvested during the just-finished {@link #planScan} — manifest cache + * hit/miss, scan/planning durations, files and manifests scanned vs skipped — as connector-neutral + * {@link ConnectorScanProfile} groups the engine writes into the query's profile execution summary. + * + *

    The default returns an empty list (connector reports nothing). A connector that wants scan + * diagnostics harvests them from its SDK during {@code planScan} (the paimon SDK exposes a metric + * registry, the iceberg SDK a metrics reporter), stashes them keyed by {@link ConnectorSession#getQueryId()}, + * and drains them here — mirroring the per-query queryId stashes this SPI already uses (read-transaction + * release, rewritable-delete supply). The engine calls this immediately after {@code planScan} on the + * same thread, so the harvest is complete; the connector must also drop its stash on + * {@link #releaseReadTransaction} to reclaim any entry a thrown {@code planScan} left behind.

    + * + * @param session the current session (its queryId keys the connector's per-query stash) + * @return this scan's diagnostics, or an empty list (the default) to contribute nothing to the profile + */ + default List collectScanProfiles(ConnectorSession session) { + return Collections.emptyList(); + } + /** * Plans the scan for a single batch of partitions (used by batch-mode scans). * diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java new file mode 100644 index 00000000000000..034941a24261d9 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.scan; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +/** + * A connector-neutral bundle of scan diagnostics for one table scan, produced by a connector from its own + * SDK metrics and written verbatim into the query profile by the generic scan node + * (via {@link ConnectorScanPlanProvider#collectScanProfiles}). + * + *

    The engine treats all three fields opaquely — it get-or-creates a profile group named + * {@link #getGroupName()} under the execution summary, adds a child named {@link #getScanLabel()}, and + * writes each {@link #getMetrics()} entry as an info string. This keeps the engine connector-agnostic: + * it never interprets a metric, only the connector knows what its SDK exposes (paimon manifest cache + * hit/miss, iceberg scanned/skipped manifests, etc.). Values are ALREADY formatted strings because the + * formatting (durations, byte sizes) lives with the SDK-specific harvest in the connector.

    + * + *

    Immutable: the metrics map is copied into an unmodifiable, insertion-ordered view so the profile + * rendering order is stable.

    + */ +public final class ConnectorScanProfile { + private final String groupName; + private final String scanLabel; + private final Map metrics; + + public ConnectorScanProfile(String groupName, String scanLabel, Map metrics) { + this.groupName = groupName; + this.scanLabel = scanLabel; + this.metrics = metrics == null + ? Collections.emptyMap() + : Collections.unmodifiableMap(new LinkedHashMap<>(metrics)); + } + + /** The profile group name (e.g. {@code "Paimon Scan Metrics"}); must match the engine's ordering key. */ + public String getGroupName() { + return groupName; + } + + /** The per-scan child label (e.g. {@code "Table Scan (db.tbl)"}). */ + public String getScanLabel() { + return scanLabel; + } + + /** The ordered metric name → already-formatted value pairs (unmodifiable). */ + public Map getMetrics() { + return metrics; + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 2e3ac2d9814594..311b693ae10252 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -24,6 +24,7 @@ import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.scan.ConnectorColumnCategory; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanProfile; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.api.scan.ConnectorSplitSource; import org.apache.doris.connector.cache.CacheSpec; @@ -93,6 +94,7 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; import java.util.function.Function; /** @@ -213,6 +215,13 @@ public class IcebergScanPlanProvider implements ConnectorScanPlanProvider { // path; pre-flip the provider never runs at all). private final IcebergRewritableDeleteStash rewritableDeleteStash; + // FIX-SCAN-METRICS: per-query stash of the iceberg SDK scan diagnostics captured by the attached + // IcebergScanProfileReporter during planScan, keyed by session queryId. fe-core drains it + // (collectScanProfiles) right after planScan on the same thread; releaseReadTransaction reclaims any entry + // a thrown planScan left behind. Attached only on the synchronous data/count path (never streaming or + // system-table, which fe-core never drains), so the value list is appended single-threaded. + private final ConcurrentHashMap> scanProfileStash = new ConcurrentHashMap<>(); + public IcebergScanPlanProvider(Map properties, IcebergCatalogOps catalogOps) { this(properties, catalogOps, null, null, null); } @@ -291,6 +300,26 @@ public OptionalLong scannedPartitionCount(List scanRanges) { ? OptionalLong.empty() : OptionalLong.of(distinctPartitions.size()); } + @Override + public List collectScanProfiles(ConnectorSession session) { + String queryId = session.getQueryId(); + if (queryId == null || queryId.isEmpty()) { + return Collections.emptyList(); + } + List profiles = scanProfileStash.remove(queryId); + return profiles == null ? Collections.emptyList() : profiles; + } + + @Override + public void releaseReadTransaction(String queryId) { + // Iceberg opens no metastore read transaction (it inherits the SPI no-op); this override only reclaims + // the scan-metrics stash for a query whose planScan threw AFTER the reporter fired (the normal path + // drains it via collectScanProfiles). Same queryId fe-core registered the query-finish callback with. + if (queryId != null && !queryId.isEmpty()) { + scanProfileStash.remove(queryId); + } + } + /** * Iceberg metadata tables legally time-travel ({@code t$snapshots FOR TIME/VERSION AS OF ...}, * {@code t$files@branch('b')}): legacy {@code IcebergScanNode.createTableScan} honors the pin via @@ -524,6 +553,16 @@ private List planScanInternal( } Table table = resolveTable(session, iceHandle); TableScan scan = buildScan(table, iceHandle, filter, session); + // FIX-SCAN-METRICS: attach a per-scan metrics reporter so the iceberg SDK's ScanReport (planning time, + // data/delete files, scanned vs skipped manifests) is captured into the query profile — restores the + // legacy IcebergScanNode scan-metrics profile the migration dropped. Attached HERE (the synchronous + // data/count path), NOT in buildScan, which is also reached by streamSplits/planSystemTableScan whose + // report would stash a queryId entry fe-core never drains (leak). The reporter fires on close of the + // planFiles iterable, which the data (try-with-resources) and count paths close on this thread. + // Guard a null session (offline unit tests) — production planScan always carries one. + if (session != null) { + scan = scan.metricsReporter(new IcebergScanProfileReporter(session.getQueryId(), scanProfileStash)); + } int formatVersion = getFormatVersion(table); List orderedPartitionKeys = IcebergPartitionUtils.getIdentityPartitionColumns(table); diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporter.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporter.java new file mode 100644 index 00000000000000..2d78369f8e9081 --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporter.java @@ -0,0 +1,203 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.api.scan.ConnectorScanProfile; + +import org.apache.iceberg.metrics.CounterResult; +import org.apache.iceberg.metrics.MetricsContext; +import org.apache.iceberg.metrics.MetricsReport; +import org.apache.iceberg.metrics.MetricsReporter; +import org.apache.iceberg.metrics.ScanMetricsResult; +import org.apache.iceberg.metrics.ScanReport; +import org.apache.iceberg.metrics.TimerResult; + +import java.text.DecimalFormat; +import java.time.Duration; +import java.util.ArrayList; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; + +/** + * An iceberg SDK {@link MetricsReporter} that captures a scan's {@link ScanReport} into a connector-neutral + * {@link ConnectorScanProfile}, stashed keyed by the statement queryId for fe-core to drain into the query + * profile (FIX-SCAN-METRICS). Restores the legacy {@code datasource.iceberg.profile.IcebergMetricsReporter} + * behavior in the plugin architecture: it is a self-contained port (the connector cannot import fe-core, so + * {@code DebugUtil}'s time/byte formatters are inlined and Guava is avoided). + * + *

    The SDK invokes {@link #report} on CLOSE of the {@code planFiles} iterable, which the connector performs + * synchronously on the planScan thread — so a fresh reporter is created per scan bound to that scan's queryId, + * and attached ONLY on the synchronous data/count path (never the streaming or system-table path, which fe-core + * never drains).

    + */ +public class IcebergScanProfileReporter implements MetricsReporter { + /** Profile group name — MUST equal fe-core {@code SummaryProfile.ICEBERG_SCAN_METRICS} (display ordering). */ + static final String GROUP_NAME = "Iceberg Scan Metrics"; + private static final DecimalFormat BYTES_FORMAT = new DecimalFormat("0.000"); + private static final long KB = 1024L; + private static final long MB = 1024 * KB; + private static final long GB = 1024 * MB; + private static final long TB = 1024 * GB; + private static final long SECOND_MS = 1000L; + private static final long MINUTE_MS = 60 * SECOND_MS; + private static final long HOUR_MS = 60 * MINUTE_MS; + + private final String queryId; + private final ConcurrentHashMap> stash; + + IcebergScanProfileReporter(String queryId, ConcurrentHashMap> stash) { + this.queryId = queryId; + this.stash = stash; + } + + @Override + public void report(MetricsReport report) { + if (queryId == null || queryId.isEmpty() || !(report instanceof ScanReport)) { + return; + } + ScanReport scanReport = (ScanReport) report; + ScanMetricsResult metrics = scanReport.scanMetrics(); + if (metrics == null) { + return; + } + Map rendered = new LinkedHashMap<>(); + rendered.put("table", scanReport.tableName()); + rendered.put("snapshot", String.valueOf(scanReport.snapshotId())); + String filter = sanitize(scanReport.filter() == null ? null : scanReport.filter().toString()); + if (!filter.isEmpty()) { + rendered.put("filter", filter); + } + if (scanReport.projectedFieldNames() != null && !scanReport.projectedFieldNames().isEmpty()) { + rendered.put("columns", String.join("|", scanReport.projectedFieldNames())); + } + appendTimer(rendered, "planning", metrics.totalPlanningDuration()); + appendCounter(rendered, "data_files", metrics.resultDataFiles()); + appendCounter(rendered, "delete_files", metrics.resultDeleteFiles()); + appendCounter(rendered, "skipped_data_files", metrics.skippedDataFiles()); + appendCounter(rendered, "skipped_delete_files", metrics.skippedDeleteFiles()); + appendCounter(rendered, "total_size", metrics.totalFileSizeInBytes()); + appendCounter(rendered, "total_delete_size", metrics.totalDeleteFileSizeInBytes()); + appendCounter(rendered, "scanned_manifests", metrics.scannedDataManifests()); + appendCounter(rendered, "skipped_manifests", metrics.skippedDataManifests()); + appendCounter(rendered, "scanned_delete_manifests", metrics.scannedDeleteManifests()); + appendCounter(rendered, "skipped_delete_manifests", metrics.skippedDeleteManifests()); + appendCounter(rendered, "indexed_delete_files", metrics.indexedDeleteFiles()); + appendCounter(rendered, "equality_delete_files", metrics.equalityDeleteFiles()); + appendCounter(rendered, "positional_delete_files", metrics.positionalDeleteFiles()); + appendMetadata(rendered, scanReport.metadata()); + + ConnectorScanProfile profile = new ConnectorScanProfile( + GROUP_NAME, "Table Scan (" + scanReport.tableName() + ")", rendered); + stash.computeIfAbsent(queryId, k -> new CopyOnWriteArrayList<>()).add(profile); + } + + private void appendMetadata(Map out, Map metadata) { + if (metadata == null || metadata.isEmpty()) { + return; + } + List captured = new ArrayList<>(); + for (String key : new String[] {"scan-state", "scan-id"}) { + if (metadata.containsKey(key)) { + captured.add(key + "=" + metadata.get(key)); + } + } + if (!captured.isEmpty()) { + out.put("metadata", "{" + String.join(", ", captured) + "}"); + } + } + + private void appendTimer(Map out, String name, TimerResult timerResult) { + if (timerResult == null) { + return; + } + Duration duration = timerResult.totalDuration(); + out.put(name, prettyMs(duration.toMillis()) + " (" + timerResult.count() + " ops)"); + } + + private void appendCounter(Map out, String name, CounterResult counterResult) { + if (counterResult == null) { + return; + } + long value = counterResult.value(); + out.put(name, counterResult.unit() == MetricsContext.Unit.BYTES + ? printByteWithUnit(value) : Long.toString(value)); + } + + private static String sanitize(String value) { + if (value == null || value.isEmpty()) { + return ""; + } + return value.replaceAll("\\s+", " ").trim(); + } + + /** Inlined fe-core {@code DebugUtil.printByteWithUnit}. */ + static String printByteWithUnit(long value) { + double d = value; + String unit; + if (value == 0) { + unit = ""; + } else if (value > TB) { + unit = "TB"; + d /= TB; + } else if (value > GB) { + unit = "GB"; + d /= GB; + } else if (value > MB) { + unit = "MB"; + d /= MB; + } else if (value > KB) { + unit = "KB"; + d /= KB; + } else { + unit = "B"; + } + return BYTES_FORMAT.format(d) + " " + unit; + } + + /** Inlined fe-core {@code DebugUtil.getPrettyStringMs}: {@code Nhour Nmin Nsec} / {@code Nms}. */ + static String prettyMs(long value) { + if (value == 0) { + return "0"; + } + StringBuilder builder = new StringBuilder(); + long remaining = value; + boolean hour = false; + boolean minute = false; + if (remaining >= HOUR_MS) { + builder.append(remaining / HOUR_MS).append("hour"); + remaining %= HOUR_MS; + hour = true; + } + if (remaining >= MINUTE_MS) { + builder.append(remaining / MINUTE_MS).append("min"); + remaining %= MINUTE_MS; + minute = true; + } + if (!hour && remaining >= SECOND_MS) { + builder.append(remaining / SECOND_MS).append("sec"); + remaining %= SECOND_MS; + } + if (!hour && !minute) { + builder.append(remaining).append("ms"); + } + return builder.toString(); + } +} diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporterTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporterTest.java new file mode 100644 index 00000000000000..250e42f396cead --- /dev/null +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergScanProfileReporterTest.java @@ -0,0 +1,106 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.iceberg; + +import org.apache.doris.connector.api.scan.ConnectorScanProfile; + +import org.apache.iceberg.expressions.Expressions; +import org.apache.iceberg.metrics.CounterResult; +import org.apache.iceberg.metrics.ImmutableScanMetricsResult; +import org.apache.iceberg.metrics.ImmutableScanReport; +import org.apache.iceberg.metrics.MetricsContext; +import org.apache.iceberg.metrics.ScanMetricsResult; +import org.apache.iceberg.metrics.ScanReport; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + +/** + * FIX-SCAN-METRICS — guards {@link IcebergScanProfileReporter}, the connector-local iceberg SDK + * {@code MetricsReporter} that captures a scan's {@code ScanReport} into a connector-neutral profile stashed + * by queryId for fe-core to drain (the migration dropped this diagnostic from the profile). + */ +public class IcebergScanProfileReporterTest { + + private static ScanReport report(String table) { + ScanMetricsResult metrics = ImmutableScanMetricsResult.builder() + .resultDataFiles(CounterResult.of(MetricsContext.Unit.COUNT, 3)) + .scannedDataManifests(CounterResult.of(MetricsContext.Unit.COUNT, 5)) + .skippedDataManifests(CounterResult.of(MetricsContext.Unit.COUNT, 2)) + .totalFileSizeInBytes(CounterResult.of(MetricsContext.Unit.BYTES, 2048)) + .build(); + return ImmutableScanReport.builder() + .tableName(table) + .snapshotId(123L) + .schemaId(0) + .filter(Expressions.alwaysTrue()) + .projectedFieldIds(Collections.emptyList()) + .projectedFieldNames(Collections.emptyList()) + .scanMetrics(metrics) + .metadata(Collections.emptyMap()) + .build(); + } + + @Test + public void reportStashesProfileKeyedByQueryId() { + // THE load-bearing RED assertion: a real iceberg ScanReport is captured into the stash as one + // ConnectorScanProfile with the transcribed counter keys. A mutation that drops report() leaves the + // stash empty. + ConcurrentHashMap> stash = new ConcurrentHashMap<>(); + new IcebergScanProfileReporter("qid-1", stash).report(report("db.tbl")); + + List profiles = stash.get("qid-1"); + Assertions.assertNotNull(profiles, "report must stash under the queryId"); + Assertions.assertEquals(1, profiles.size()); + ConnectorScanProfile profile = profiles.get(0); + Assertions.assertEquals("Iceberg Scan Metrics", profile.getGroupName()); + Assertions.assertEquals("Table Scan (db.tbl)", profile.getScanLabel()); + Map m = profile.getMetrics(); + Assertions.assertEquals("3", m.get("data_files")); + Assertions.assertEquals("5", m.get("scanned_manifests")); + Assertions.assertEquals("2", m.get("skipped_manifests")); + // BYTES-unit counter is byte-formatted (self-ported DebugUtil.printByteWithUnit). + Assertions.assertEquals("2.000 KB", m.get("total_size")); + } + + @Test + public void blankQueryIdDoesNotStash() { + ConcurrentHashMap> stash = new ConcurrentHashMap<>(); + new IcebergScanProfileReporter("", stash).report(report("db.tbl")); + new IcebergScanProfileReporter(null, stash).report(report("db.tbl")); + Assertions.assertTrue(stash.isEmpty(), "a blank queryId must not accumulate an unreclaimable entry"); + } + + @Test + public void formattersMatchLegacy() { + Assertions.assertEquals("0", IcebergScanProfileReporter.prettyMs(0)); + Assertions.assertEquals("1sec234ms", IcebergScanProfileReporter.prettyMs(1234)); + Assertions.assertEquals("0.000 ", IcebergScanProfileReporter.printByteWithUnit(0)); + Assertions.assertEquals("2.000 KB", IcebergScanProfileReporter.printByteWithUnit(2048)); + Assertions.assertEquals("1.500 MB", IcebergScanProfileReporter.printByteWithUnit(1024L * 1536)); + } + + @Test + public void groupNameMatchesFeCoreConstant() { + Assertions.assertEquals("Iceberg Scan Metrics", IcebergScanProfileReporter.GROUP_NAME); + } +} diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonMetricRegistry.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonMetricRegistry.java new file mode 100644 index 00000000000000..243e8cbacc645d --- /dev/null +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonMetricRegistry.java @@ -0,0 +1,78 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.paimon; + +import org.apache.paimon.metrics.MetricGroup; +import org.apache.paimon.metrics.MetricGroupImpl; +import org.apache.paimon.metrics.MetricRegistry; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.util.Collection; +import java.util.Map; +import java.util.concurrent.ConcurrentHashMap; + +/** + * A per-scan {@link MetricRegistry} handed to the paimon SDK ({@code InnerTableScan.withMetricRegistry}) + * so {@code scan.plan()} records its {@code ScanMetrics} group here for {@link PaimonScanMetrics} to harvest + * into the query profile. Self-contained port of the legacy {@code datasource.paimon.profile.PaimonMetricRegistry} + * (the connector cannot depend on fe-core); paimon-SDK-only, no fe-core imports. + */ +public class PaimonMetricRegistry implements MetricRegistry { + private static final Logger LOG = LoggerFactory.getLogger(PaimonMetricRegistry.class); + private static final String TABLE_TAG_KEY = "table"; + private final ConcurrentHashMap groups = new ConcurrentHashMap<>(); + + @Override + public MetricGroup createMetricGroup(String name, Map tags) { + MetricGroup group = new MetricGroupImpl(name, tags); + String table = tags == null ? "" : tags.getOrDefault(TABLE_TAG_KEY, ""); + groups.put(buildKey(name, table), group); + LOG.debug("Created metric group: {}:{}", name, table); + return group; + } + + public MetricGroup getGroup(String name, String table) { + String key = buildKey(name, table); + MetricGroup group = groups.get(key); + if (group == null) { + LOG.warn("MetricGroup not found: {}", key); + } + return group; + } + + public void removeGroup(String name, String table) { + groups.remove(buildKey(name, table)); + } + + public Collection getAllGroups() { + return groups.values(); + } + + public Map getAllGroupsAsMap() { + return new ConcurrentHashMap<>(groups); + } + + public void clear() { + groups.clear(); + } + + private static String buildKey(String name, String table) { + return name + ":" + table; + } +} diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanMetrics.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanMetrics.java new file mode 100644 index 00000000000000..d3ec508edd81c0 --- /dev/null +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanMetrics.java @@ -0,0 +1,183 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.paimon; + +import org.apache.doris.connector.api.scan.ConnectorScanProfile; + +import org.apache.paimon.metrics.Counter; +import org.apache.paimon.metrics.Gauge; +import org.apache.paimon.metrics.Histogram; +import org.apache.paimon.metrics.HistogramStatistics; +import org.apache.paimon.metrics.Metric; +import org.apache.paimon.metrics.MetricGroup; +import org.apache.paimon.operation.metrics.ScanMetrics; + +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Optional; +import java.util.concurrent.TimeUnit; + +/** + * Harvests the paimon SDK {@code ScanMetrics} group recorded by {@link PaimonMetricRegistry} during + * {@code scan.plan()} into a connector-neutral {@link ConnectorScanProfile} for the query profile — a + * self-contained port of the legacy {@code datasource.paimon.profile.PaimonScanMetricsReporter} extraction, + * with fe-core's {@code DebugUtil.getPrettyStringMs} inlined (the connector cannot import fe-core). + */ +public final class PaimonScanMetrics { + private static final double P95 = 0.95d; + private static final long SECOND_MS = 1000L; + private static final long MINUTE_MS = 60 * SECOND_MS; + private static final long HOUR_MS = 60 * MINUTE_MS; + + /** Profile group name — MUST equal fe-core {@code SummaryProfile.PAIMON_SCAN_METRICS} (display ordering). */ + public static final String GROUP_NAME = "Paimon Scan Metrics"; + + private PaimonScanMetrics() { + } + + /** + * Build the scan profile for {@code paimonTableName}'s recorded metrics, or empty when the SDK recorded + * none (unpartitioned/no-op scan, unsupported scan type). {@code scanLabel} is the per-scan child name. + */ + public static Optional harvest(PaimonMetricRegistry registry, String paimonTableName, + String scanLabel) { + if (registry == null || paimonTableName == null) { + return Optional.empty(); + } + MetricGroup group = registry.getGroup(ScanMetrics.GROUP_NAME, paimonTableName); + if (group == null) { + String prefix = ScanMetrics.GROUP_NAME + ":"; + for (Map.Entry entry : registry.getAllGroupsAsMap().entrySet()) { + String key = entry.getKey(); + if (!key.startsWith(prefix)) { + continue; + } + if (group != null) { + // More than one candidate group — ambiguous, bail out (legacy parity). + group = null; + break; + } + group = entry.getValue(); + } + } + if (group == null) { + return Optional.empty(); + } + Map metrics = group.getMetrics(); + if (metrics == null || metrics.isEmpty()) { + return Optional.empty(); + } + + Map rendered = new LinkedHashMap<>(); + appendDuration(rendered, metrics, ScanMetrics.LAST_SCAN_DURATION, "last_scan_duration"); + appendHistogram(rendered, metrics, ScanMetrics.SCAN_DURATION, "scan_duration"); + appendCounter(rendered, metrics, ScanMetrics.LAST_SCANNED_MANIFESTS, "last_scanned_manifests"); + appendCounter(rendered, metrics, ScanMetrics.LAST_SCAN_SKIPPED_TABLE_FILES, + "last_scan_skipped_table_files"); + appendCounter(rendered, metrics, ScanMetrics.LAST_SCAN_RESULTED_TABLE_FILES, + "last_scan_resulted_table_files"); + appendCounter(rendered, metrics, ScanMetrics.MANIFEST_HIT_CACHE, "manifest_hit_cache"); + appendCounter(rendered, metrics, ScanMetrics.MANIFEST_MISSED_CACHE, "manifest_missed_cache"); + if (rendered.isEmpty()) { + return Optional.empty(); + } + return Optional.of(new ConnectorScanProfile(GROUP_NAME, scanLabel, rendered)); + } + + private static void appendDuration(Map out, Map metrics, String metricKey, + String profileKey) { + Long value = getLongValue(metrics.get(metricKey)); + if (value == null) { + return; + } + out.put(profileKey, formatDuration(value)); + } + + private static void appendCounter(Map out, Map metrics, String metricKey, + String profileKey) { + Long value = getLongValue(metrics.get(metricKey)); + if (value == null) { + return; + } + out.put(profileKey, Long.toString(value)); + } + + private static void appendHistogram(Map out, Map metrics, String metricKey, + String profileKey) { + Metric metric = metrics.get(metricKey); + if (!(metric instanceof Histogram)) { + return; + } + Histogram histogram = (Histogram) metric; + HistogramStatistics stats = histogram.getStatistics(); + if (stats == null) { + return; + } + String formatted = "count=" + histogram.getCount() + + ", mean=" + formatDuration(stats.getMean()) + + ", p95=" + formatDuration(stats.getQuantile(P95)) + + ", max=" + formatDuration(stats.getMax()); + out.put(profileKey, formatted); + } + + private static Long getLongValue(Metric metric) { + if (metric instanceof Counter) { + return ((Counter) metric).getCount(); + } + if (metric instanceof Gauge) { + Object value = ((Gauge) metric).getValue(); + if (value instanceof Number) { + return ((Number) value).longValue(); + } + } + return null; + } + + private static String formatDuration(double nanos) { + return prettyMs(TimeUnit.NANOSECONDS.toMillis(Math.round(nanos))); + } + + /** Inlined fe-core {@code DebugUtil.getPrettyStringMs}: {@code Nhour Nmin Nsec} / {@code Nms}. */ + static String prettyMs(long value) { + if (value == 0) { + return "0"; + } + StringBuilder builder = new StringBuilder(); + long remaining = value; + boolean hour = false; + boolean minute = false; + if (remaining >= HOUR_MS) { + builder.append(remaining / HOUR_MS).append("hour"); + remaining %= HOUR_MS; + hour = true; + } + if (remaining >= MINUTE_MS) { + builder.append(remaining / MINUTE_MS).append("min"); + remaining %= MINUTE_MS; + minute = true; + } + if (!hour && remaining >= SECOND_MS) { + builder.append(remaining / SECOND_MS).append("sec"); + remaining %= SECOND_MS; + } + if (!hour && !minute) { + builder.append(remaining).append("ms"); + } + return builder.toString(); + } +} diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java index fd1869dd127e7e..7b52deb15d76f8 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonScanPlanProvider.java @@ -22,6 +22,7 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.pushdown.ConnectorExpression; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanProfile; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.metastore.spi.JdbcDriverSupport; import org.apache.doris.connector.spi.ConnectorContext; @@ -59,6 +60,7 @@ import org.apache.paimon.table.Table; import org.apache.paimon.table.source.DataSplit; import org.apache.paimon.table.source.DeletionFile; +import org.apache.paimon.table.source.InnerTableScan; import org.apache.paimon.table.source.RawFile; import org.apache.paimon.table.source.ReadBuilder; import org.apache.paimon.table.source.Split; @@ -94,6 +96,8 @@ import java.util.Optional; import java.util.OptionalLong; import java.util.Set; +import java.util.concurrent.ConcurrentHashMap; +import java.util.concurrent.CopyOnWriteArrayList; import java.util.stream.Collectors; /** @@ -213,6 +217,12 @@ public class PaimonScanPlanProvider implements ConnectorScanPlanProvider { // the existing construction sites are unchanged (first build = direct read = pre-fix behavior). private final PaimonSchemaAtMemo schemaAtMemo; + // FIX-SCAN-METRICS: per-query stash of the paimon SDK scan diagnostics harvested during planScan, keyed + // by session queryId. fe-core drains it (collectScanProfiles) right after planScan on the same thread; + // releaseReadTransaction reclaims any entry a thrown planScan left behind. Bounded to the sync planScan + // path (paimon never streams), so the CopyOnWriteArrayList value is only ever appended single-threaded. + private final ConcurrentHashMap> scanProfileStash = new ConcurrentHashMap<>(); + public PaimonScanPlanProvider(Map properties, PaimonCatalogOps catalogOps) { this(properties, catalogOps, null); } @@ -377,6 +387,46 @@ public OptionalLong scannedPartitionCount(List scanRanges) { ? OptionalLong.empty() : OptionalLong.of(distinctPartitions.size()); } + /** + * Harvest the paimon SDK scan metrics recorded into {@code registry} by {@code scan.plan()} and stash + * them keyed by the session queryId for fe-core to drain (FIX-SCAN-METRICS). No-op for a blank queryId + * (offline/no-session) or a scan the SDK recorded no metrics for. + */ + private void stashScanProfile(ConnectorSession session, Table table, PaimonTableHandle handle, + PaimonMetricRegistry registry) { + // Guard a null session (offline unit tests) — production planScan always carries one. + if (session == null) { + return; + } + String queryId = session.getQueryId(); + if (queryId == null || queryId.isEmpty()) { + return; + } + String scanLabel = "Table Scan (" + handle.getDatabaseName() + "." + handle.getTableName() + ")"; + PaimonScanMetrics.harvest(registry, table.name(), scanLabel).ifPresent(profile -> + scanProfileStash.computeIfAbsent(queryId, k -> new CopyOnWriteArrayList<>()).add(profile)); + } + + @Override + public List collectScanProfiles(ConnectorSession session) { + String queryId = session.getQueryId(); + if (queryId == null || queryId.isEmpty()) { + return Collections.emptyList(); + } + List profiles = scanProfileStash.remove(queryId); + return profiles == null ? Collections.emptyList() : profiles; + } + + @Override + public void releaseReadTransaction(String queryId) { + // Paimon opens no metastore read transaction (it inherits the SPI no-op); this override only reclaims + // the scan-metrics stash for a query whose planScan threw AFTER harvesting (the normal path drains it + // via collectScanProfiles). Same queryId fe-core registered the query-finish callback with. + if (queryId != null && !queryId.isEmpty()) { + scanProfileStash.remove(queryId); + } + } + /** * COUNT(*)-pushdown-aware scan entry (FIX-COUNT-PUSHDOWN). The generic {@code PluginDrivenScanNode} * forwards the no-grouping {@code COUNT(*)} signal here via the SPI's count-pushdown overload. @@ -459,7 +509,16 @@ private List planScanInternal( readBuilder.withProjection(projected); } TableScan scan = readBuilder.newScan(); + // FIX-SCAN-METRICS: attach a metric registry so scan.plan() records its ScanMetrics (manifest cache + // hit/miss, scan durations, table files skipped/resulted), then harvest them below — restores the + // legacy PaimonScanNode scan-metrics profile. InnerTableScan.withMetricRegistry is a real body on the + // AbstractDataTableScan a data table returns; other scan types keep the no-op default (no metrics). + PaimonMetricRegistry metricRegistry = new PaimonMetricRegistry(); + if (scan instanceof InnerTableScan) { + scan = ((InnerTableScan) scan).withMetricRegistry(metricRegistry); + } List paimonSplits = planSplits(scan); + stashScanProfile(session, table, paimonHandle, metricRegistry); // Determine table location String tableLocation = getTableLocation(table); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanMetricsTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanMetricsTest.java new file mode 100644 index 00000000000000..ec7c1e8d299ce0 --- /dev/null +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonScanMetricsTest.java @@ -0,0 +1,82 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.paimon; + +import org.apache.doris.connector.api.scan.ConnectorScanProfile; + +import org.apache.paimon.operation.metrics.ScanMetrics; +import org.apache.paimon.operation.metrics.ScanStats; +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Map; +import java.util.Optional; + +/** + * FIX-SCAN-METRICS — guards {@link PaimonScanMetrics}, which harvests the paimon SDK {@code ScanMetrics} + * recorded by {@link PaimonMetricRegistry} during {@code scan.plan()} into a connector-neutral profile (the + * migration dropped this diagnostic; restoring it needs no fe-core import — the metric API is paimon's). + */ +public class PaimonScanMetricsTest { + + @Test + public void harvestRendersRecordedScanMetrics() { + // THE load-bearing RED assertion: drive the real paimon SDK to record a scan into the registry, then + // harvest it. reportScan(duration, scannedManifests, skippedTableFiles, resultedTableFiles) populates + // the LAST_* gauges. A mutation that drops the harvest returns empty. + PaimonMetricRegistry registry = new PaimonMetricRegistry(); + ScanMetrics metrics = new ScanMetrics(registry, "mydb.mytbl"); + metrics.reportScan(new ScanStats(2_000_000L, 5L, 3L, 7L)); + + Optional profile = + PaimonScanMetrics.harvest(registry, "mydb.mytbl", "Table Scan (mydb.mytbl)"); + Assertions.assertTrue(profile.isPresent(), "recorded scan metrics must harvest"); + Assertions.assertEquals("Paimon Scan Metrics", profile.get().getGroupName()); + Assertions.assertEquals("Table Scan (mydb.mytbl)", profile.get().getScanLabel()); + Map m = profile.get().getMetrics(); + Assertions.assertEquals("5", m.get("last_scanned_manifests")); + Assertions.assertEquals("3", m.get("last_scan_skipped_table_files")); + Assertions.assertEquals("7", m.get("last_scan_resulted_table_files")); + } + + @Test + public void harvestEmptyWhenNoMetricsRecorded() { + // A registry the SDK never recorded a scan group into -> nothing to harvest (unpartitioned/no-op scan, + // unsupported scan type). Same as the un-overridden default; documents the fall-through. + Assertions.assertEquals(Optional.empty(), + PaimonScanMetrics.harvest(new PaimonMetricRegistry(), "mydb.mytbl", "Table Scan (mydb.mytbl)")); + Assertions.assertEquals(Optional.empty(), + PaimonScanMetrics.harvest(null, "mydb.mytbl", "x")); + } + + @Test + public void prettyMsMatchesLegacyFormatter() { + // Self-ported fe-core DebugUtil.getPrettyStringMs (the connector cannot import fe-core). + Assertions.assertEquals("0", PaimonScanMetrics.prettyMs(0)); + Assertions.assertEquals("500ms", PaimonScanMetrics.prettyMs(500)); + Assertions.assertEquals("1sec234ms", PaimonScanMetrics.prettyMs(1234)); + Assertions.assertEquals("1min", PaimonScanMetrics.prettyMs(60_000)); + } + + @Test + public void groupNameMatchesFeCoreConstant() { + // Mirror of the fe-core SummaryProfile.PAIMON_SCAN_METRICS constant (stringly-typed coupling; the two + // modules cannot cross-import, so each asserts the shared literal). + Assertions.assertEquals("Paimon Scan Metrics", PaimonScanMetrics.GROUP_NAME); + } +} diff --git a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java index cb051c3d624f20..124be52b05b078 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java +++ b/fe/fe-core/src/main/java/org/apache/doris/common/profile/SummaryProfile.java @@ -155,6 +155,7 @@ public class SummaryProfile { public static final String LATENCY_FROM_BE_TO_FE = "RPC Latency From BE To FE"; public static final String SPLITS_ASSIGNMENT_WEIGHT = "Splits Assignment Weight"; public static final String ICEBERG_SCAN_METRICS = "Iceberg Scan Metrics"; + public static final String PAIMON_SCAN_METRICS = "Paimon Scan Metrics"; public static final String WAIT_CHANGE_VISIBLE_TIME = "Wait Change Visible Time"; private boolean isWarmUp = false; @@ -214,6 +215,7 @@ public boolean isWarmup() { SINK_SET_PARTITION_VALUES_TIME, CREATE_SCAN_RANGE_TIME, ICEBERG_SCAN_METRICS, + PAIMON_SCAN_METRICS, NEREIDS_DISTRIBUTE_TIME, GET_META_VERSION_TIME, GET_PARTITION_VERSION_TIME, @@ -272,6 +274,7 @@ public boolean isWarmup() { .put(SINK_SET_PARTITION_VALUES_TIME, 3) .put(CREATE_SCAN_RANGE_TIME, 2) .put(ICEBERG_SCAN_METRICS, 3) + .put(PAIMON_SCAN_METRICS, 3) .put(GET_PARTITION_VERSION_TIME, 1) .put(GET_PARTITION_VERSION_COUNT, 1) .put(GET_PARTITION_VERSION_BY_HAS_DATA_COUNT, 1) diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 00938e64324c30..19f260b63fd8fc 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -28,6 +28,8 @@ import org.apache.doris.catalog.Env; import org.apache.doris.catalog.TableIf; import org.apache.doris.common.UserException; +import org.apache.doris.common.profile.RuntimeProfile; +import org.apache.doris.common.profile.SummaryProfile; import org.apache.doris.connector.api.Connector; import org.apache.doris.connector.api.ConnectorMetadata; import org.apache.doris.connector.api.ConnectorSession; @@ -42,6 +44,7 @@ import org.apache.doris.connector.api.pushdown.ProjectionApplicationResult; import org.apache.doris.connector.api.scan.ConnectorColumnCategory; import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanProfile; import org.apache.doris.connector.api.scan.ConnectorScanRange; import org.apache.doris.connector.api.scan.ConnectorSplitSource; import org.apache.doris.connector.api.scan.ScanNodePropertiesResult; @@ -333,6 +336,48 @@ static long resolveSelectedPartitionNum(long nereidsSelectedPartitionNum, boolea return nereidsSelectedPartitionNum; } + /** + * Write the connector's SDK scan diagnostics ({@link ConnectorScanProfile}s harvested during planScan) + * into this query's profile execution summary. Looks up the query's {@link SummaryProfile} and delegates + * the tree-building to the pure {@link #writeScanProfilesInto}. No-op when there is no profile (e.g. a + * statement not collecting one) or nothing to write. + */ + private void appendConnectorScanProfiles(List profiles) { + if (profiles == null || profiles.isEmpty()) { + return; + } + SummaryProfile summaryProfile = SummaryProfile.getSummaryProfile(ConnectContext.get()); + if (summaryProfile == null) { + return; + } + writeScanProfilesInto(summaryProfile.getExecutionSummary(), profiles); + } + + /** + * Transcribe connector-supplied scan profiles into {@code executionSummary}: get-or-create a group named + * {@link ConnectorScanProfile#getGroupName()}, add a child named {@link ConnectorScanProfile#getScanLabel()}, + * and write each metric as an info string. Purely connector-agnostic (no source-type branch) — the engine + * only transcribes what the connector produced. Pure and takes a bare {@link RuntimeProfile} so the + * tree-building is unit-testable without a {@code ConnectContext}. + */ + static void writeScanProfilesInto(RuntimeProfile executionSummary, List profiles) { + if (executionSummary == null || profiles == null || profiles.isEmpty()) { + return; + } + for (ConnectorScanProfile profile : profiles) { + RuntimeProfile group = executionSummary.getChildMap().get(profile.getGroupName()); + if (group == null) { + group = new RuntimeProfile(profile.getGroupName()); + executionSummary.addChild(group, true); + } + RuntimeProfile scan = new RuntimeProfile(profile.getScanLabel()); + for (Map.Entry entry : profile.getMetrics().entrySet()) { + scan.addInfoString(entry.getKey(), entry.getValue()); + } + group.addChild(scan, true); + } + } + @Override public String getNodeExplainString(String prefix, TExplainLevel detailLevel) { StringBuilder output = new StringBuilder(); @@ -1104,6 +1149,13 @@ public List getSplits(int numBackends) throws UserException { () -> scanProvider.scannedPartitionCount(ranges)); this.selectedPartitionNum = resolveSelectedPartitionNum( this.selectedPartitionNum, countPushdown, connectorScannedPartitions); + // FIX-SCAN-METRICS: drain the connector's SDK scan diagnostics (harvested during planScan, keyed by + // queryId) and write them into the query profile. connector-agnostic: the connector supplies the + // group/label/metrics, the engine only transcribes them (no source branch). Default empty for + // connectors that don't harvest. Same thread as planScan, so the harvest is complete. + List scanProfiles = onPluginClassLoader(scanProvider, + () -> scanProvider.collectScanProfiles(connectorSession)); + appendConnectorScanProfiles(scanProfiles); long pushDownRowCount = resolvePushDownRowCount(countPushdown, ranges); if (pushDownRowCount >= 0) { // Only set when a range actually carries a precomputed count (e.g. paimon's collapsed count diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProfileTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProfileTest.java new file mode 100644 index 00000000000000..958f84260dfe2b --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeScanProfileTest.java @@ -0,0 +1,100 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.common.profile.RuntimeProfile; +import org.apache.doris.common.profile.SummaryProfile; +import org.apache.doris.connector.api.scan.ConnectorScanProfile; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +/** + * FIX-SCAN-METRICS — guards {@link PluginDrivenScanNode#writeScanProfilesInto}, the connector-agnostic + * transcription of connector-supplied scan diagnostics into the query profile execution summary. WHY it + * matters: the plugin migration dropped the paimon/iceberg SDK scan metrics (manifest cache hit/miss, scan + * durations) from the profile; this generic writer restores them without the engine knowing any connector + * specifics — it only get-or-creates a group and transcribes the connector's labels + metric strings. + */ +public class PluginDrivenScanNodeScanProfileTest { + + private static ConnectorScanProfile profile(String group, String label, String... kv) { + Map metrics = new LinkedHashMap<>(); + for (int i = 0; i + 1 < kv.length; i += 2) { + metrics.put(kv[i], kv[i + 1]); + } + return new ConnectorScanProfile(group, label, metrics); + } + + @Test + public void nullOrEmptyIsNoOp() { + RuntimeProfile summary = new RuntimeProfile("Execution Summary"); + PluginDrivenScanNode.writeScanProfilesInto(summary, null); + PluginDrivenScanNode.writeScanProfilesInto(summary, Collections.emptyList()); + Assertions.assertTrue(summary.getChildMap().isEmpty(), "no profiles -> no group written"); + // null summary must not throw. + PluginDrivenScanNode.writeScanProfilesInto(null, Collections.singletonList(profile("G", "L"))); + } + + @Test + public void writesGroupChildAndMetrics() { + // THE load-bearing RED assertion: one profile becomes a group -> "Table Scan (...)" child -> info + // strings. A mutation that skips writing leaves the summary childless. + RuntimeProfile summary = new RuntimeProfile("Execution Summary"); + PluginDrivenScanNode.writeScanProfilesInto(summary, Collections.singletonList( + profile("Paimon Scan Metrics", "Table Scan (db.t)", + "manifest_hit_cache", "4", "manifest_missed_cache", "1"))); + + RuntimeProfile group = summary.getChildMap().get("Paimon Scan Metrics"); + Assertions.assertNotNull(group, "group must be created"); + RuntimeProfile scan = group.getChildMap().get("Table Scan (db.t)"); + Assertions.assertNotNull(scan, "per-scan child must be created"); + Assertions.assertEquals("4", scan.getInfoString("manifest_hit_cache")); + Assertions.assertEquals("1", scan.getInfoString("manifest_missed_cache")); + } + + @Test + public void sharesGroupAcrossScans() { + // Two scans of the same connector go under ONE get-or-created group as two children (a join over two + // paimon tables must not create two "Paimon Scan Metrics" groups). + RuntimeProfile summary = new RuntimeProfile("Execution Summary"); + PluginDrivenScanNode.writeScanProfilesInto(summary, Arrays.asList( + profile("Iceberg Scan Metrics", "Table Scan (db.a)", "data_files", "3"), + profile("Iceberg Scan Metrics", "Table Scan (db.b)", "data_files", "5"))); + + RuntimeProfile group = summary.getChildMap().get("Iceberg Scan Metrics"); + Assertions.assertNotNull(group); + Assertions.assertEquals(2, group.getChildMap().size(), "one group, two scan children"); + Assertions.assertEquals("3", group.getChildMap().get("Table Scan (db.a)").getInfoString("data_files")); + Assertions.assertEquals("5", group.getChildMap().get("Table Scan (db.b)").getInfoString("data_files")); + } + + @Test + public void groupNameConstantsMatchConnectorLiterals() { + // The connector-supplied group name is stringly-typed coupled to these fe-core constants (the connector + // cannot import SummaryProfile). This is the fe-core half of the mirror check; the connector tests + // assert their own literals equal the same strings. + Assertions.assertEquals("Paimon Scan Metrics", SummaryProfile.PAIMON_SCAN_METRICS); + Assertions.assertEquals("Iceberg Scan Metrics", SummaryProfile.ICEBERG_SCAN_METRICS); + } +} From 63f3c1965e760eb02d89f0d5d5767a739968b6f8 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 10:38:18 +0800 Subject: [PATCH 325/334] [doc](catalog) restore paimon/iceberg scan metrics: design + summary + tracking - add FIX-scan-metrics-spi design + summary (superseding the L15 deletion: the migration dropped a real feature for BOTH connectors, so it is restored via a connector-neutral SPI rather than deleted). - record the 3-lens design red-team (wf_0f803c49-7bb) and folded fixes. - update task-list + HANDOFF. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 2 + plan-doc/task-list-65185-reverify-fixes.md | 6 +- .../designs/FIX-scan-metrics-spi-design.md | 176 ++++++++++++++++++ .../designs/FIX-scan-metrics-spi-summary.md | 59 ++++++ 4 files changed, 241 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-scan-metrics-spi-design.md create mode 100644 plan-doc/tasks/designs/FIX-scan-metrics-spi-summary.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 2487e8e4fb8ff6..e038aa1d532b59 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -13,6 +13,8 @@ > > **L12 e2e live-gated**(须真集群):隐藏/transform 分区 iceberg 表(**<1024 文件 或 `enable_external_table_batch_mode=false`**,否则走 streaming batch 报 `partition=0/0` 假绿)`WHERE ts` 单日谓词 → `partition=1/M`;paimon 非分区列 manifest 剪枝 → 真实数;各配 sql_block_rule partition_num 门。 > +> **L15 追加转折 → scan-metrics 功能恢复** `8d4209865a3`:用户质疑 L15「删死引用」是否丢功能。复核参考仓库 master 证实:老 `PaimonScanMetricsReporter`+`PaimonMetricRegistry` 是真功能,插件迁移 **paimon+iceberg 都丢**(登记偏差 T02/T29);iceberg `ICEBERG_SCAN_METRICS` 仅靠**死 legacy `IcebergScanNode`** 续命(我先前「iceberg 活」判断有误)。用户要求 connector SPI 框架**统一恢复**。已建**连接器中立 scan-metrics SPI**:`ConnectorScanProfile`(值类型)+`ConnectorScanPlanProvider.collectScanProfiles`(默认空);paimon 侧 port `PaimonMetricRegistry`+新 `PaimonScanMetrics`(pull,`withMetricRegistry`),iceberg 侧新 `IcebergScanProfileReporter`(push,**挂 `planScanInternal` 数据/count 路,非共享 `buildScan`**——避 streaming 泄漏 blocker);fe-core 纯静态 `writeScanProfilesInto` 写 `SummaryProfile`(无源分支)、**复活 `PAIMON_SCAN_METRICS` 常量**;`DebugUtil` 时间/字节格式化器**自搬**(连接器不得 import fe-core)。stash 按 queryId、`collectScanProfiles` 排空 + `releaseReadTransaction` 兜底清。设计 3-lens 红队 `wf_0f803c49-7bb` 全 SOUND_WITH_CHANGES(2 blocker+majors 全折入)。UT 12/12 + 全模块 paimon 360+iceberg 982+fe-core 94 绿、0 checkstyle、import 门净。**踩坑**:iceberg 既有 UT 传 **null session** → 我 `session.getQueryId()` NPE 挂 31 例,加 null-session guard 修(paimon 同防御);paimon 模块须 `package`(shade)非 `test-compile`(HiveConf 假失败,memory `doris-build-verify-gotchas`)。设计/summary=`designs/FIX-scan-metrics-spi-{design,summary}.md`。**e2e live-gated**:真集群 paimon/iceberg 查询后 profile 含 "Paimon/Iceberg Scan Metrics" 组 + manifest 缓存命中/未命中。 +> > **下一步**:#65185 复核系列剩余 = `L16`(iceberg 缓存偏斜)、`L17`(iceberg version-blind schema)、`L18`(iceberg 未知类型)、`L19`(partition_columns 撞名);⏸ 决策类**仅余 `L20`**(MC EQ `==`,须先中文讲清问用户)。⏸ `M8`(升级文档)用户 07-12 跳过、`D-系列`设计债随 P8。 --- diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 8217898caff08a..4f7e41ba10edcf 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -192,7 +192,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L12** DONE `e5de7aedcd5`(设计 `designs/FIX-L12-design.md`;summary `FIX-L12-summary.md`;设计 3-lens 对抗红队 `wf_f1524868-4b8` 全 SOUND_WITH_CHANGES,major/minor 全折入)· **用户 2026-07-13 签字选项 B**(连接器回报 SDK-distinct 真实扫描分区数,非登记偏差)。新 opt-in SPI `ConnectorScanPlanProvider.scannedPartitionCount`(默认 empty,仿 `supportsTableSample`)+ fe-core 纯 helper `resolveSelectedPartitionNum`(`!countPushdown && present` 用连接器数,否则 Nereids;getSplits 覆写)+ paimon(distinct `getPartitionValues()`)/iceberg(distinct `specId|partitionDataJson`,新 `IcebergScanRange.getScannedPartitionKey`)override;hive/MC 不 override 保留 Nereids(本就相等)。**通用节点无源分支**(铁律)。8/8+5/5+4/4 RED-able UT,paimon 356/356+iceberg 978/978 全绿,0 checkstyle,import 门净。登记残余:COUNT(*) 保守 Nereids、iceberg binary/null-bucket undercount、streaming-iceberg(显式开 batch+≥1024 文件)inert=legacy parity、query-cache 消费者 benign。**e2e live-gated**(隐藏/transform 分区 <1024 文件 batch-off `WHERE ts` 单日→`partition=1/M`)。 - [x] **L13** DONE `ced4775b844`(设计 `FIX-L13-design.md`;红队 3 lens 全 SOUND）· `toPaimonType` 对 ARRAY 元素/MAP value/STRUCT 字段加 `.copy(type.isChildNullable(i))`(MAP key 保持 `.copy(false)`),恢复 legacy `DorisToPaimonTypeVisitor` 嵌套 nullability parity。**scope=仅 nullability**:comment 丢=DV-035 M10.1 已接受偏差、field-id 顺序 parity 均不动。`.copy(true)` 对默认可空子类型逐字节恒等(既有 parity 测保持绿)。3 新 RED-able 测(经 `.type().isNullable()` 断言,非 DataField equals)。type-mapping+schema-builder 26/26 绿。e2e live-gated(建嵌套 NOT NULL 表 DESCRIBE/SDK 读回)。 - [x] **L14** DONE `478718aca6f`(设计 `FIX-L14-design.md`;红队 3 lens SOUND/SOUND_WITH_CHANGES）· null-tolerant `resolveIgnoreSplitType(session)`(镜像 `isCppReaderEnabled`,红队 MINOR)+三 legacy `continue` 位:`IGNORE_JNI` drops nonDataSplit+DataSplit-JNI 臂、`IGNORE_NATIVE` drops native 臂、count 臂不检、`IGNORE_PAIMON_CPP` 保持 no-op(=legacy parity,全树 grep 证 legacy 从不引用)。默认 NONE 逐字节不变。3 新 live-planScan RED 测(IGNORE_JNI/IGNORE_NATIVE 各证跳 split+IGNORE_PAIMON_CPP==NONE);nonDataSplit IGNORE_JNI 位离线不可测→留 E2E-only。69/69 绿。e2e live-gated(真集群 SET ignore_split_type 断言跳分片)。 -- [x] **L15** DONE `b2cdf971889`(设计 `designs/FIX-L15-design.md`;summary `FIX-L15-summary.md`)· 删 `SummaryProfile.java` 三处 `PAIMON_SCAN_METRICS` 死引用(常量 + `EXECUTION_SUMMARY_KEYS` + 缩进 map);P5(`dbc38a265e5`)删写入方后无任何 reporter 填充该分组(对照活的 `ICEBERG_SCAN_METRICS` 有 `IcebergMetricsReporter` 保留不动)。**未加** connector-neutral scan-metrics SPI(feature、非必需,超出清死引用 scope)。零行为变更、无新测(死码删除无可断言);fe-core test-compile BUILD SUCCESS + 0 checkstyle。**非 live**(该 profile 列本就从不出现)。 +- [x] **L15 → 被 scan-metrics 功能取代**(用户 2026-07-13 追加要求恢复功能,非删死引用):先删 `b2cdf971889`(判为死引用),后**复核发现是插件迁移丢的真功能**(paimon+iceberg 都丢;iceberg 的 `ICEBERG_SCAN_METRICS` 仅靠死 legacy `IcebergScanNode` 续命,我先前「iceberg 活」判断有误)→ **改为恢复**:`8d4209865a3` 建**连接器中立 scan-metrics SPI**(`ConnectorScanProfile`+`collectScanProfiles`;paimon `PaimonMetricRegistry`+`PaimonScanMetrics` pull、iceberg `IcebergScanProfileReporter` push;fe-core 纯静态 `writeScanProfilesInto` 写 profile;**复活** `PAIMON_SCAN_METRICS` 常量)。设计 3-lens 红队 `wf_0f803c49-7bb` 全 SOUND_WITH_CHANGES(2 blocker=iceberg streaming 泄漏改挂 planScanInternal、DebugUtil 自搬,+ majors 全折入)。设计/summary=`designs/FIX-scan-metrics-spi-{design,summary}.md`。UT fe-core 4/4+paimon 4/4+iceberg 4/4,全模块 paimon 360+iceberg 982+fe-core 94 绿,0 checkstyle,import 门净。e2e live-gated(profile 含 X Scan Metrics 组)。 - [ ] **L16** iceberg 缓存偏斜(部分已修) · `IcebergScanPlanProvider.java:1077-1108`:hasSnapshotPin 臂把 field-id dict 建成「pinned schema ∪ requested latest columns」超集(传 requestedLowerNames 而非 `emptyList`),两向都超集;或 fe-core 侧 query-begin pin 解析 pinnedSchema 到 pinned schemaId(对齐 time-travel 臂 `:377-387`)。 - [ ] **L17** iceberg 同表多版本 version-blind 绑定 · `PluginDrivenMvccExternalTable.java:475-485`:per-reference schema 绑定 version-aware(用 `StatementContext.getSnapshot(table, tableSnapshot, scanParams)` 的 pinnedSchema,fallback latest);与 L16 同根,建议一并。窄触发 + fail-loud。 - [ ] **L18** iceberg 未知/v3 类型静默 UNSUPPORTED · `IcebergTypeMapping.java:91,143`:两 default 臂改 `throw DorisConnectorException("Cannot transform unknown type: "+...)`,保留显式 TIME/VARIANT UNSUPPORTED;或接受更松并登记。 @@ -304,7 +304,9 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) --- -**⭐ L15 DONE `b2cdf971889`(2026-07-13)** — 删 `SummaryProfile` 三处 `PAIMON_SCAN_METRICS` 死引用(P5 后无 reporter 填充,对照活的 iceberg metrics 保留)。零行为变更、无新测、非 live;fe-core BUILD SUCCESS + 0 checkstyle。设计/summary = `designs/FIX-L15-{design,summary}.md`。 +**⭐ L15 DONE `b2cdf971889`(2026-07-13)** — 删 `SummaryProfile` 三处 `PAIMON_SCAN_METRICS` 死引用(P5 后无 reporter 填充,对照活的 iceberg metrics 保留)。零行为变更、无新测、非 live;fe-core BUILD SUCCESS + 0 checkstyle。设计/summary = `designs/FIX-L15-{design,summary}.md`。**⚠ 后被 scan-metrics 功能取代(见下)——复核发现该功能实为插件迁移真丢、且 iceberg 同丢,遂改为恢复。** + +**⭐ scan-metrics 功能恢复 DONE `8d4209865a3`(2026-07-13,用户追加)** — 用户质疑 L15「删死引用」是否丢了功能;复核参考仓库 master 证实:老版 `PaimonScanMetricsReporter`+`PaimonMetricRegistry` 是真功能,插件迁移**paimon+iceberg 都丢**(已登记偏差 T02/T29),iceberg 的 `ICEBERG_SCAN_METRICS` 仅靠死 legacy `IcebergScanNode` 续命。用户要求在 connector SPI 框架**统一恢复**:新 `ConnectorScanProfile`+`collectScanProfiles` SPI、paimon pull(port registry)+iceberg push(reporter 挂同步 planScanInternal 路避泄漏)、fe-core 纯静态写 profile、复活常量、DebugUtil 格式化器自搬。设计 3-lens 红队 `wf_0f803c49-7bb` 全 SOUND_WITH_CHANGES(2 blocker+majors 全折入);UT 12 全绿、全模块 paimon 360+iceberg 982+fe-core 94 绿。设计/summary=`designs/FIX-scan-metrics-spi-{design,summary}.md`。e2e live-gated。 **⭐ L12 DONE `e5de7aedcd5`(2026-07-13,用户签字选项 B)** — selectedPartitionNum 用连接器 SDK-distinct 真实扫描分区数(能力 SPI opt-in `scannedPartitionCount` + fe-core 纯 helper `resolveSelectedPartitionNum` + paimon/iceberg override,通用节点无源分支);设计经 3-lens 对抗红队(`wf_f1524868-4b8`)全 SOUND_WITH_CHANGES,major(iceberg streaming inert=legacy parity、iceberg 跨 spec 值撞加 specId 解、query-cache 第三消费者 benign)+ minor 全折入。8/8+5/5+4/4 RED-able UT、paimon 356/356+iceberg 978/978 全绿、0 checkstyle、import 门净。设计/summary=`designs/FIX-L12-{design,summary}.md`。**决策类仅余 L20**。 diff --git a/plan-doc/tasks/designs/FIX-scan-metrics-spi-design.md b/plan-doc/tasks/designs/FIX-scan-metrics-spi-design.md new file mode 100644 index 00000000000000..37c612c26950dc --- /dev/null +++ b/plan-doc/tasks/designs/FIX-scan-metrics-spi-design.md @@ -0,0 +1,176 @@ +# FIX-SCAN-METRICS — 连接器中立的扫描指标 SPI(恢复 paimon + iceberg profile scan metrics) + +> 用户 2026-07-13 指令:老版本 Paimon/Iceberg 的 profile scan metrics 在插件迁移中被丢(已登记偏差 +> `deviations-log.md:189` T02/T29);要求在 connector SPI 框架上**统一设计**恢复。**本设计取代 L15 删除** +> (复活 `PAIMON_SCAN_METRICS` 常量,不再当死引用删)。 + +## Problem + +老 `PaimonScanNode`/`IcebergScanNode` 在规划扫描时把连接器 SDK 的扫描指标写进查询 profile: +- **paimon**(`PaimonScanMetricsReporter`+`PaimonMetricRegistry`):SDK `ScanMetrics` → `last_scan_duration`、 + `scan_duration`(直方图)、`last_scanned_manifests`、`last_scan_skipped/resulted_table_files`、 + **`manifest_hit_cache`/`manifest_missed_cache`**。 +- **iceberg**(`IcebergMetricsReporter`):SDK `ScanReport.scanMetrics()` → `planning` timer、`data/delete_files`、 + `skipped_*`、`total_size`、`scanned/skipped_manifests`、`equality/positional_delete_files` 等。 + +插件迁移后**两者都丢**:新 `PaimonScanPlanProvider` 不挂 metric registry;新 `IcebergScanPlanProvider:719-720` +**明确"intentionally drop"** metricsReporter。fe-core 的两个 reporter 类只剩死 legacy `IcebergScanNode` 引用 +(该节点已不在活路径,前轮证实)。纯诊断能力(manifest 缓存命中率等排障关键),不影响正确性。 + +## 架构约束(为何不能照搬老代码) + +老代码在 fe-core 的 ScanNode 里直接写 `SummaryProfile`。插件连接器**不得 import fe-core**(铁律 +`catalog-spi-no-property-parsing-in-fecore`/import 门禁)。故需一条**连接器中立通道**:连接器采集 SDK 指标 → +渲染成中立结构 → fe-core 拿回写 profile。 + +## Design(统一 SPI) + +### 1. fe-connector-api:中立值类型 + SPI 方法 + +新 `ConnectorScanProfile`(不可变值类型,连接器产、fe-core 消费): +```java +public final class ConnectorScanProfile { + private final String groupName; // "Paimon Scan Metrics" / "Iceberg Scan Metrics" + private final String scanLabel; // "Table Scan (db.tbl)" + private final Map metrics; // 有序 name -> 已格式化 value + // ctor(拷贝为不可变 LinkedHashMap) + 三 getter +} +``` +`ConnectorScanPlanProvider` 新增(默认空,opt-in,仿 `scannedPartitionCount`/`supportsTableSample` 先例): +```java +/** + * Connector SDK scan diagnostics harvested during the just-finished planScan (manifest cache hit/miss, + * scan/planning durations, files/manifests scanned vs skipped), as connector-neutral profile groups the + * engine writes into the query's SummaryProfile execution summary. Default empty (connector reports none). + * The connector stashes these during planScan keyed by session.getQueryId() and drains them here — mirrors + * the existing per-query queryId stashes (rewritable-deletes, manifest-cache tally). + */ +default List collectScanProfiles(ConnectorSession session) { + return Collections.emptyList(); +} +``` + +### 2. fe-core:通用写 profile(connector-agnostic) + +`PluginDrivenScanNode.getSplits` 中 planScan 之后(TCCL pin,同 planScan): +```java +List scanProfiles = onPluginClassLoader(scanProvider, + () -> scanProvider.collectScanProfiles(connectorSession)); +appendConnectorScanProfiles(scanProfiles); +``` +拆两层(红队 major:纯静态可测): +- 调用方 `appendConnectorScanProfiles(profiles)`:查 `SummaryProfile.getSummaryProfile(ConnectContext.get())` + → `getExecutionSummary()`(null-guard),再委派纯静态 helper。 +- **纯静态** `writeScanProfilesInto(RuntimeProfile executionSummary, List profiles)` + (connector-agnostic——只搬运连接器给的 group/label/metrics,无 source 分支;取裸 `RuntimeProfile` 故可脱离 + `ConnectContext` 直接单测,**RED-able**): +```java +static void writeScanProfilesInto(RuntimeProfile executionSummary, List profiles) { + if (executionSummary == null || profiles == null || profiles.isEmpty()) { + return; + } + for (ConnectorScanProfile profile : profiles) { + RuntimeProfile group = executionSummary.getChildMap().get(profile.getGroupName()); + if (group == null) { + group = new RuntimeProfile(profile.getGroupName()); + executionSummary.addChild(group, true); + } + RuntimeProfile scan = new RuntimeProfile(profile.getScanLabel()); + for (Map.Entry e : profile.getMetrics().entrySet()) { + scan.addInfoString(e.getKey(), e.getValue()); + } + group.addChild(scan, true); + } +} +``` +- 复活 `SummaryProfile.PAIMON_SCAN_METRICS`(撤回 L15 删除)+ 保留 `ICEBERG_SCAN_METRICS`:仅供 + `EXECUTION_SUMMARY_KEYS`(显示顺序)+ 缩进 map。**连接器 groupName 字符串须与之逐字一致**(stringly-typed + coupling,文档登记;两侧各持字面量,连接器不 import fe-core)。 +- 位置:非 batch/streaming 的同步 `getSplits` 路(paimon 恒走此路;iceberg <1024 文件 / batch-off 走此路)。 + streaming iceberg(≥1024 文件 + 显式开 batch)不采集——登记残余(与 L12 同边界;`streamSplits` 惰性,无 + planScan)。 + +> **⚠ 格式化器自搬(红队 blocker)**:legacy 渲染用 fe-core `DebugUtil.getPrettyStringMs`/`printByteWithUnit`, +> 连接器**不得 import fe-core**。故两连接器各自**内联自搬**这两个纯函数(`getPrettyStringMs`:HOUR/MINUTE/SECOND +> 拆分;`printByteWithUnit`:KB..TB + `DecimalFormat("0.000")`)——同 `IcebergScanPlanProvider.rootCauseMessage` +> 已有的自搬先例。`ConnectorScanProfile.metrics` 存**已格式化**字符串。 + +### 3. fe-connector-paimon:pull 采集 + +- 新 `PaimonScanMetrics`(连接器内 util,搬 legacy `PaimonScanMetricsReporter` 的抽取逻辑:counter/duration/ + histogram 渲染 + **自搬 `getPrettyStringMs`**)。 +- 新 `PaimonMetricRegistry`(逐字搬 legacy 72 行,`implements org.apache.paimon.metrics.MetricRegistry`)。 +- `planScanInternal`:`TableScan scan = readBuilder.newScan()` 后 `if (scan instanceof InnerTableScan) scan = + ((InnerTableScan) scan).withMetricRegistry(registry)`;`scan.plan()` 后 `PaimonScanMetrics.harvest(registry, + paimonTableName, tableDisplayName)` → `ConnectorScanProfile` → 存 `ConcurrentHashMap>`。 +- override `collectScanProfiles(session)`:`remove(queryId)`。 + +### 4. fe-connector-iceberg:push 采集 + +- 新 `IcebergScanProfileReporter implements org.apache.iceberg.metrics.MetricsReporter`:构造绑 `queryId` + + 共享 stash;`report(ScanReport)` → 抽取(搬 legacy `IcebergMetricsReporter` 的 ScanReport 抽取 + **自搬 + `getPrettyStringMs`/`printByteWithUnit`**)→ `ConnectorScanProfile` → append 到 `stash[queryId]`。 +- **附着点 = `planScanInternal`(数据+count 路),NOT 共享 `buildScan`(红队 blocker)**:在 `planScanInternal` + `TableScan scan = buildScan(...)`(:527)后 `scan = scan.metricsReporter(new IcebergScanProfileReporter( + session.getQueryId(), stash, tableName))`。**故意避开 `buildScan` 本身**——它还被 `streamSplits`(:416)、 + `planSystemTableScan`(:697)、`streamingSplitEstimate`(:374)复用;若挂 buildScan,streaming 路 `planFiles()` + 在后台线程 close 时**也会 fire report → stash 积累但从不 drain(getSplits 被 batch 旁路)→ 泄漏**。 + callback 在 planFiles 的 `CloseableIterable` **close** 时同步触发(planScanInternal try-with-resources 于 + planScan 线程 close,`report` 内联跑,非 worker 线程)。sys-table(planSystemTableScan 早返、走独立 buildScan) + 与 streaming 均**不挂** reporter → 不采集(登记残余)。 +- override `collectScanProfiles(session)`:`remove(queryId)`。 + +### 5. 泄漏/关联 + +- **关联**:fe-core 在每次 planScan 后**紧接**同线程 `collectScanProfiles(queryId)` 排空 → 多扫描节点各取各的 + (node1 planScan→collect drains node1;node2 planScan→collect drains node2)。多节点同 queryId 即使并发, + drain-all 也把该连接器该查询已积累的全写进共享 group(profile 去重靠 get-or-create group),净输出正确。 +- **泄漏**:正常路 collect 排空;planScan 采集后抛异常的窄泄漏由连接器在 query-finish 清理——**复用 fe-core 已 + 为每 planScan 注册的 `releaseReadTransaction(queryId)` 回调**(`PluginDrivenScanNode:1016`):paimon/iceberg + override 该方法**顺带 `stash.remove(queryId)`**(当前二者继承 no-op 默认;新 override 只做 stash 清理,读事务 + 仍 no-op)。 + +## Risk Analysis +- **connector-agnostic**:fe-core 只搬运连接器给的中立结构,无 source 分支;SPI 加法向后兼容,其它连接器继承默认 + 空。iron rule 守。 +- **零正确性影响**:纯 profile 诊断;不改扫描结果/split/分区数。 +- **paimon 构建**:paimon SDK(`paimon-core`)已含 `metrics.*`/`InnerTableScan`(连接器已 bundle,验证过)。 +- **登记残余(红队补全,均与 legacy parity 或纯诊断缺口,不影响正确性)**: + - **streaming iceberg**(≥1024 文件 + 显式开 batch):reporter 不挂 streaming 路 → 不采集(=legacy batch 亦不 + 在 profile 报);已避开 buildScan 故**无泄漏**。 + - **iceberg manifest-cache 子路**(`meta.cache.iceberg.manifest.enable=true`,**默认 off**): + `planFileScanTaskWithManifestCache` 手工重建 task、**不调 `scan.planFiles()`** → report 不 fire → 该查询无 + iceberg scan metrics(**=legacy parity**,legacy 同不调 planFiles)。 + - **iceberg sys-table**:reporter 只挂数据+count 路,sys-table 走 `planSystemTableScan` 独立 buildScan → 不采集 + (P6.6 后仍不采集;out of scope)。 + - **paimon 非 `AbstractDataTableScan`**(sys/incremental scan,`withMetricRegistry` 默认 no-op)与 **iceberg/ + paimon 空表**(无 snapshot,`planFiles` 返空无 whenComplete)→ 无指标(=legacy)。 + - **paimon scanLabel 弱限定**(display-only 偏差):legacy 用 fe-core `TableIf.getNameWithFullQualifiers()` + (catalog.db.tbl);连接器不得 import fe-core → 用连接器侧 `db.tbl`(无 catalog 前缀)。iceberg 不受影响 + (legacy 本用 `report.tableName()`)。 + - EXPLAIN VERBOSE 的 iceberg `manifest cache:` 行(现存,别的机制)不动。 +- **stringly-typed group 名耦合**:连接器字面量须匹配 fe-core 常量。**镜像双测**(不能跨模块 assert):fe-core 断言 + `PAIMON_SCAN_METRICS.equals("Paimon Scan Metrics")`,连接器断言其字面量 == 同串。 +- **stash 线程安全**:reporter confined 到同步 planScan 路(paimon 恒同步、iceberg 已避开 streaming)→ 单线程 + append/drain,race-free;`ConcurrentHashMap` + `remove(queryId)` 原子排空即足。 +- **泄漏兜底**:paimon/iceberg override `releaseReadTransaction(queryId)` **顺带 `stash.remove(queryId)`**(二者 + 现继承 no-op 默认,新 override 只清 stash、读事务仍 no-op);fe-core 每 planScan 前注册该回调 + (`PluginDrivenScanNode:1016`)故查询结束必清。 + +## Test Plan +### Unit Tests +- **fe-core**:`ConnectorScanProfile` 值类型不可变性;`appendConnectorScanProfiles` 把 group/label/metrics 正确 + 建树(用假 `ConnectorScanProfile` + 真 `RuntimeProfile`/`SummaryProfile`,fe-core 有 Mockito;RED:不写=无子)。 +- **paimon**:`PaimonScanMetrics.harvest` 对含 ScanMetrics 组的假 registry 渲染出预期 map(counter/duration/ + histogram);`collectScanProfiles` 排空语义(stash→drain→空)。RED:默认空。 +- **iceberg**:`IcebergScanProfileReporter.report(fakeScanReport)` append 出预期 `ConnectorScanProfile`; + `collectScanProfiles` 排空。RED:默认空。 +- 一致性断言:连接器 groupName 字面量 == fe-core 常量值(防 stringly-typed 漂移)。 +### E2E(live-gated) +- 真集群 paimon/iceberg 表查询后 `SHOW QUERY PROFILE`/profile 含 "Paimon/Iceberg Scan Metrics" 组 + manifest + 缓存命中/未命中等条目。 + +## 验证判据 +- fe-core test-compile + paimon(package/shade)+ iceberg 三模块 BUILD SUCCESS + 0 checkstyle;import 门 exit 0。 +- 复活 `PAIMON_SCAN_METRICS` 后 `EXECUTION_SUMMARY_KEYS`/缩进含之;连接器 group 名一致性单测绿。 diff --git a/plan-doc/tasks/designs/FIX-scan-metrics-spi-summary.md b/plan-doc/tasks/designs/FIX-scan-metrics-spi-summary.md new file mode 100644 index 00000000000000..ecdd6719cafa54 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-scan-metrics-spi-summary.md @@ -0,0 +1,59 @@ +# FIX-SCAN-METRICS Summary — 连接器中立的扫描指标 SPI(恢复 paimon + iceberg profile scan metrics) + +> 用户 2026-07-13 指令(取代 L15 删除:复活常量而非删死引用)。设计经 3-lens 对抗红队 `wf_0f803c49-7bb` 全 +> SOUND_WITH_CHANGES,2 blocker + majors 全折入。 + +## Problem +老 `PaimonScanNode`/`IcebergScanNode` 把连接器 SDK 扫描指标(paimon:manifest 缓存命中/未命中、扫描耗时、 +表文件跳过/保留;iceberg:planning 耗时、data/delete 文件、扫过/跳过 manifest 等)写进查询 profile。插件迁移 +**两者都丢**(已登记偏差 `deviations-log.md:189` T02/T29):新 `PaimonScanPlanProvider` 不挂 metric registry; +新 `IcebergScanPlanProvider` 明确 "intentionally drop" metricsReporter。fe-core 两个 reporter 类只剩**死 legacy +`IcebergScanNode`** 引用(该节点已不在活路径)——iceberg 的 `ICEBERG_SCAN_METRICS` 只是靠死代码续命。 + +## Root Cause +插件连接器**不得 import fe-core**(铁律),而老 reporter 直接写 fe-core `SummaryProfile`。迁移时未建替代通道 → +功能整体丢失(纯诊断,不影响正确性)。 + +## Fix(统一 opt-in SPI,连接器采集 → 中立结构 → 内核写 profile) +- **fe-connector-api**:新中立值类型 `ConnectorScanProfile{groupName, scanLabel, metrics(不可变有序 map)}` + + `default List collectScanProfiles(session)`(默认空,仿 `scannedPartitionCount`)。 +- **fe-core**(`PluginDrivenScanNode`):`getSplits` planScan 后调 `collectScanProfiles` → 纯静态 + `writeScanProfilesInto(RuntimeProfile, profiles)`(get-or-create group + 逐 metric addInfoString,**无源分支**、 + 可脱 ConnectContext 单测)写进 `SummaryProfile` 执行摘要;复活 `SummaryProfile.PAIMON_SCAN_METRICS` 常量(撤回 + L15 删除,供显示顺序)。 +- **fe-connector-paimon**:port `PaimonMetricRegistry`(72 行,paimon-SDK-only)+ 新 `PaimonScanMetrics` + (harvest + **自搬** `getPrettyStringMs`);`planScanInternal` `newScan()` 后挂 `withMetricRegistry`(仅 + `InnerTableScan`)、`plan()` 后 harvest → 按 queryId stash;override `collectScanProfiles`(排空)+ + `releaseReadTransaction`(清 stash 兜底,读事务仍 no-op)。 +- **fe-connector-iceberg**:新 `IcebergScanProfileReporter implements MetricsReporter`(capture ScanReport + + **自搬** `getPrettyStringMs`/`printByteWithUnit`,无 Guava/无 fe-core);**挂在 `planScanInternal` 数据+count 路** + (NOT 共享 `buildScan`——避开 streaming/sys-table 的**泄漏 blocker**);同样 stash + collect + cleanup override。 + +## 红队(`wf_0f803c49-7bb`)折入 +- **blocker1**(iceberg streaming 泄漏):reporter 从共享 `buildScan` 移到 `planScanInternal` 数据/count 路 → + streaming/sys-table 不挂 → 不泄漏。 +- **blocker2**(DebugUtil fe-core-only):两连接器各自**内联自搬** `getPrettyStringMs`/`printByteWithUnit`。 +- **major**(可测性):fe-core 拆纯静态 `writeScanProfilesInto(裸 RuntimeProfile,…)`。 +- 登记残余:streaming iceberg、iceberg manifest-cache 子路(默认 off,不调 planFiles)、iceberg sys-table、 + paimon 非 `AbstractDataTableScan`/空表 均不采集(=legacy parity);paimon scanLabel 弱限定(无 catalog 前缀, + display-only);COUNT(*) 共享同一 plan/scan 故采集正常。 + +## Tests(全 RED-able,连接器无 Mockito 用真 SDK fixture) +- **fe-core** `PluginDrivenScanNodeScanProfileTest` 4:`writeScanProfilesInto` 建 group/child/infoString(RED)、 + 空 no-op、共享 group、group 名常量镜像。 +- **paimon** `PaimonScanMetricsTest` 4:真 `ScanMetrics.reportScan(ScanStats)` → harvest 出 5/3/7 计数(RED)、 + 空 registry→empty、`prettyMs` 格式、GROUP_NAME 镜像。 +- **iceberg** `IcebergScanProfileReporterTest` 4:真 `ImmutableScanReport` → report 入 stash(RED,含 "2.000 KB" + 字节格式)、空 queryId 不 stash、formatters、GROUP_NAME 镜像。 + +## Result +- api+fe-core+paimon(package/shade)+iceberg **BUILD SUCCESS + 0 Checkstyle**;import 门 exit 0。 +- 靶向 UT:fe-core 4/4(+PartitionCount 8/8)、iceberg 4/4(+PartitionCount 4/4)、paimon 4/4(+Capability 5/5)。 +- 全模块回归:见下(paimon/iceberg 全绿)。 +- paimon-core 1.3.1(fe/pom.xml)含 `MANIFEST_HIT/MISSED_CACHE`——headline 缓存诊断可采。 +- **e2e live-gated**:真集群 paimon/iceberg 查询后 profile 含 "Paimon/Iceberg Scan Metrics" 组 + manifest 缓存 + 命中/未命中等条目。 +- 死 legacy fe-core `IcebergMetricsReporter` 保持不动(随 P8 清死路径)。 + +## Commits +- code:api(2)+fe-core(2)+paimon(3)+iceberg(2)+ 3 测试。 From 01668779fd93daad6d5ca26028216eec746e426f Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 11:48:32 +0800 Subject: [PATCH 326/334] [fix](catalog) strip reserved partition_columns magic-key from source properties MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SPI carries partition info to fe-core through the reserved key "partition_columns" inside the same map that also carries the source table's pass-through properties. fe-core (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty "partition_columns" as the partition-column CSV. iceberg/hive/paimon all putAll(sourceProperties) and then stamp the key only for genuinely partitioned tables, never removing a pre-existing one — so a NON-partitioned table whose source properties literally contain "partition_columns" (e.g. ALTER TABLE ... SET TBLPROPERTIES('partition_columns'='id')) leaks that value into fe-core and is misdetected as partitioned (wrong pruning / row-count / EXPLAIN partition=N/M). Fix (producer-side, the only workable locus — fe-core cannot distinguish connector-emitted from user-passthrough values without parsing connector key semantics it must not know): each connector removes the reserved key(s) right after the pass-through putAll, before the conditional stamp, so only its own determination reaches fe-core. - iceberg: remove "partition_columns" after putAll(table.properties()) - hive: remove PARTITION_COLUMNS_PROPERTY after copying HMS parameters - paimon: remove "partition_columns"/"primary_keys" after putAll(coreOptions().toMap()) Partitioned tables re-stamp their own value below, so the connector value always wins. No user-facing regression: getTableProperties() already strips these keys from SHOW CREATE. Tests (RED-able, recording fakes): iceberg + hive assert an unpartitioned table with a colliding source property emits no "partition_columns", and a partitioned table still emits its spec/key-derived value (guard against over-strip). paimon's DataTable coreOptions path is not reachable via the module's non-DataTable fake -> covered by inspection + e2e-gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/hive/HiveConnectorMetadata.java | 7 ++++ .../hive/HiveConnectorMetadataSchemaTest.java | 28 +++++++++++++ .../iceberg/IcebergConnectorMetadata.java | 7 ++++ .../iceberg/IcebergConnectorMetadataTest.java | 39 +++++++++++++++++++ .../paimon/PaimonConnectorMetadata.java | 8 ++++ 5 files changed, 89 insertions(+) diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index b951a95ead4c98..e4f4446d721572 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -448,6 +448,13 @@ public ConnectorTableSchema getTableSchema( // mutating the shared tableInfo map. Map tableProperties = new HashMap<>( tableInfo.getParameters() != null ? tableInfo.getParameters() : Collections.emptyMap()); + // Strip the FE-internal reserved partition marker that may have leaked in from the copied HMS + // parameters above (a user TBLPROPERTY literally named "partition_columns"): the generic fe-core + // consumer (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty "partition_columns" + // as the partition-column CSV, so a leaked user value would make a NON-partitioned table be + // misdetected as partitioned. Only this connector's own value (re-stamped below when the table + // has partition keys) may reach fe-core. + tableProperties.remove(PARTITION_COLUMNS_PROPERTY); // Mark which emitted columns are partition columns for the generic fe-core consumer. Without this // property every partitioned hive/hudi table is read as unpartitioned (wrong pruning/row count, MTMV // breakage). The value is a CSV of the RAW partition-key names in declaration order; hive partition-key diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index 4173ed361b0225..5d3a7b9bf8fefc 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -33,6 +33,7 @@ import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; import java.util.List; import java.util.Map; import java.util.stream.Collectors; @@ -221,6 +222,33 @@ public void testUnpartitionedTableEmitsNoPartitionColumnsProperty() { Assertions.assertFalse(schema.getProperties().containsKey("partition_columns")); } + @Test + public void testUnpartitionedTableStripsUserPartitionColumnsParameter() { + // A user TBLPROPERTY literally named "partition_columns" on a NON-partitioned hive table must not leak + // into the emitted schema: the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) + // treats a non-empty "partition_columns" as the partition-column CSV, so a leaked user value whose CSV + // matches a real column would make this unpartitioned table be misdetected as partitioned. + // MUTATION: dropping tableProperties.remove(PARTITION_COLUMNS_PROPERTY) -> the parameter leaks -> red. + Map params = new HashMap<>(); + params.put("partition_columns", "id"); // a real column — fe-core would match it + HmsTableInfo tableInfo = unpartitionedTable(PARQUET_INPUT_FORMAT).parameters(params).build(); + ConnectorTableSchema schema = schemaOf(tableInfo); + Assertions.assertFalse(schema.getProperties().containsKey("partition_columns"), + "an unpartitioned hive table must not carry partition_columns even if a parameter is named so"); + } + + @Test + public void testPartitionedTablePartitionColumnsReflectKeysNotCollidingParameter() { + // Guard the fix does not over-strip: a genuinely partitioned table whose parameters ALSO carry a + // colliding "partition_columns"=id must emit the CONNECTOR's own partition-key CSV (year,region), + // never the user's spoofed value. MUTATION: stripping unconditionally after the stamp -> red. + Map params = new HashMap<>(); + params.put("partition_columns", "id"); + ConnectorTableSchema schema = schemaOf(partitionedTable().parameters(params).build()); + Assertions.assertEquals("year,region", schema.getProperties().get("partition_columns"), + "partition_columns must be the connector's partition-key CSV, not the colliding parameter"); + } + @Test public void testTopNLazyCapabilityMarkerEmittedForParquetAndOrc() { // WHY: Top-N lazy materialize is orc/parquet-only in legacy hive (HMSExternalTable.supportedHiveTopNLazyTable). diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 3797416ec2746d..10d7591559b7d2 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -410,6 +410,13 @@ private ConnectorTableSchema buildTableSchema(String tableName, Table table, Sch Map tableProps = new HashMap<>(); tableProps.putAll(table.properties()); + // Strip the FE-internal reserved partition marker that may have leaked in from the source + // table.properties() above (e.g. a user ALTER TABLE ... SET TBLPROPERTIES('partition_columns'=...)): + // the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty + // "partition_columns" as the partition-column CSV, so a leaked user value would make an + // UNPARTITIONED table be misdetected as partitioned. Only THIS connector's own spec-derived value + // (re-stamped below for a genuinely partitioned table) may reach fe-core. + tableProps.remove("partition_columns"); // SHOW CREATE TABLE render hints under neutral reserved keys (fe-core strips them from the // rendered PROPERTIES and emits them as LOCATION / PARTITION BY / ORDER BY). They replace the // previously-injected location / iceberg.format-version / iceberg.partition-spec keys: those were diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java index fe3b100e38e518..b9693264788f1a 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java @@ -655,6 +655,45 @@ public void getTableSchemaCopiesTablePropertiesAndLocation() { "the table location must travel under show.location, not as a bare \"location\" property"); } + @Test + public void getTableSchemaStripsUserPartitionColumnsOnUnpartitionedTable() { + // A user TBLPROPERTY literally named "partition_columns" on an UNPARTITIONED iceberg table (e.g. set + // via ALTER TABLE ... SET TBLPROPERTIES('partition_columns'='id')) must NOT survive into the emitted + // schema properties: the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) + // treats a non-empty "partition_columns" as the partition-column CSV, so a leaked user value whose + // CSV matches a real column would make this non-partitioned table be misdetected as partitioned + // (wrong pruning / row-count / EXPLAIN partition=N/M). buildTableSchema stamps the key only for a + // genuinely partitioned table, so for an unpartitioned table it must be absent. + // MUTATION: dropping tableProps.remove("partition_columns") -> the user value leaks via putAll -> red. + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + Map userProps = new HashMap<>(); + userProps.put("partition_columns", "id"); // a real column name — fe-core would match it + ops.table = new FakeIcebergTable( + "t1", idNameSchema(), PartitionSpec.unpartitioned(), "s3://bucket/db1/t1", userProps); + ConnectorTableSchema schema = + metadataWith(ops).getTableSchema(null, new IcebergTableHandle("db1", "t1")); + Assertions.assertFalse(schema.getProperties().containsKey("partition_columns"), + "an unpartitioned iceberg table must not carry partition_columns even if the source props do"); + } + + @Test + public void getTableSchemaPartitionColumnsReflectSpecNotCollidingUserProperty() { + // Guard the fix does not over-strip: a genuinely partitioned table (identity on `name`) whose source + // properties ALSO carry a colliding "partition_columns"=id must emit the CONNECTOR's spec-derived + // value (`name`), never the user's spoofed value — the remove()+re-stamp guarantees the connector wins. + // MUTATION: removing the re-stamp, or stripping unconditionally after the stamp -> red. + Schema schema = idNameSchema(); + PartitionSpec identitySpec = PartitionSpec.builderFor(schema).identity("name").build(); + RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); + Map userProps = new HashMap<>(); + userProps.put("partition_columns", "id"); // user tries to spoof partitioning on `id` + ops.table = new FakeIcebergTable("t2", schema, identitySpec, "s3://bucket/db1/t2", userProps); + ConnectorTableSchema out = + metadataWith(ops).getTableSchema(null, new IcebergTableHandle("db1", "t2")); + Assertions.assertEquals("name", out.getProperties().get("partition_columns"), + "partition_columns must be the spec-derived value, not the colliding user property"); + } + @Test public void getTableSchemaEmitsShowPartitionClauseWithTransforms() { // Unpartitioned: no show.partition-clause (and, byte-faithful, no legacy iceberg.partition-spec). diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java index 26033e16fbbb88..23cd9098dd4e26 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java @@ -302,6 +302,14 @@ private ConnectorTableSchema buildTableSchema(String tableName, Table table, Lis // (no fe-core dependency); the fe-core consumer filters out the schema-control keys below. if (table instanceof DataTable) { schemaProps.putAll(((DataTable) table).coreOptions().toMap()); + // Strip the FE-internal reserved schema-control keys that may have leaked in from the source + // coreOptions() above (a user option literally named "partition_columns"/"primary_keys"): the + // generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty + // "partition_columns" as the partition-column CSV, so a leaked user value would make a + // NON-partitioned table be misdetected as partitioned. Only this connector's own values + // (re-stamped below from partitionKeys/primaryKeys) may reach fe-core. + schemaProps.remove("partition_columns"); + schemaProps.remove("primary_keys"); if (primaryKeys != null && !primaryKeys.isEmpty()) { schemaProps.put(CoreOptions.PRIMARY_KEY.key(), String.join(",", primaryKeys)); } From 1c9c99c7767fa5a9fce3843697228b9fc2677533 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 11:52:08 +0800 Subject: [PATCH 327/334] [fix](catalog) iceberg: pin unknown/v3 types to UNSUPPORTED by design (DV-051) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit IcebergTypeMapping's read direction maps iceberg types Doris cannot represent — the v3 primitives TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN and the non-primitive VARIANT — to UNSUPPORTED, so the table LOADS with only the exotic column present-but-unqueryable. This diverges from legacy fe-core (IcebergUtils threw IllegalArgumentException at schema-load, failing the whole table) and from Trino (throws NOT_SUPPORTED). Per user decision 2026-07-13 the looser behavior is ACCEPTED (registered DV-051): map every unrepresentable column uniformly to UNSUPPORTED so one exotic column does not make a wide table unloadable. No functional change — the two default arms already return UNSUPPORTED; this adds clarifying comments documenting the intentional divergence, and a guard test (unknownAndV3TypesDegradeToUnsupportedByDesign) that pins the choice so a future mutation to throw turns RED. The write direction (toIcebergPrimitive) still throws — CREATE TABLE must not silently accept a type it cannot round-trip. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/iceberg/IcebergTypeMapping.java | 16 ++++++++++++++ .../iceberg/IcebergTypeMappingReadTest.java | 22 +++++++++++++++++++ 2 files changed, 38 insertions(+) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java index 3fc9370b184872..8e4d791f037152 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergTypeMapping.java @@ -88,6 +88,12 @@ public static ConnectorType fromIcebergType(Type icebergType, } return ConnectorType.structOf(names, types).withChildrenFieldIds(fieldIds); default: + // Any non-primitive iceberg type Doris cannot represent (VARIANT today; future non-primitive + // typeIds) degrades to UNSUPPORTED: the table still LOADS and only this column is + // present-but-unqueryable. This DIVERGES from legacy fe-core (IcebergUtils.icebergTypeToDorisType + // threw IllegalArgumentException at schema-load, failing the whole table). Graceful degradation + // is INTENTIONAL (user decision 2026-07-13: map every unrepresentable column uniformly to + // UNSUPPORTED so one exotic column does not make a wide table unloadable). Registered DV-051. return ConnectorType.of("UNSUPPORTED"); } } @@ -138,8 +144,18 @@ private static ConnectorType fromPrimitive(Type.PrimitiveType primitive, } return ConnectorType.of("DATETIMEV2", ICEBERG_DATETIME_SCALE_MS, 0); case TIME: + // iceberg TIME has no Doris analogue -> UNSUPPORTED (explicit, byte-parity with legacy + // IcebergUtils which also mapped TIME to Type.UNSUPPORTED). return ConnectorType.of("UNSUPPORTED"); default: + // Any primitive iceberg type Doris cannot represent — notably the v3 types TIMESTAMP_NANO / + // GEOMETRY / GEOGRAPHY / UNKNOWN — degrades to UNSUPPORTED: the table still LOADS and only this + // column is present-but-unqueryable. This DIVERGES from legacy (IcebergUtils.icebergPrimitiveType- + // ToDorisType threw IllegalArgumentException "Cannot transform unknown type", failing the whole + // table). Graceful degradation is INTENTIONAL (user decision 2026-07-13: map every unrepresentable + // column uniformly to UNSUPPORTED so one exotic column does not make a wide table unloadable). + // Registered DV-051. (The write direction toIcebergPrimitive still throws — CREATE TABLE must not + // silently accept a type it cannot round-trip.) return ConnectorType.of("UNSUPPORTED"); } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java index af0c234cbfe63c..cf229143afccc5 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergTypeMappingReadTest.java @@ -85,6 +85,28 @@ public void flagIndependentPrimitivesMatchLegacy() { Assertions.assertEquals("UNSUPPORTED", mapOff(Types.TimeType.get()).getTypeName()); } + @Test + public void unknownAndV3TypesDegradeToUnsupportedByDesign() { + // WHY (user decision 2026-07-13, DV-051): iceberg types Doris cannot represent — the v3 primitives + // TIMESTAMP_NANO / GEOMETRY / GEOGRAPHY / UNKNOWN and the non-primitive VARIANT — must map to + // UNSUPPORTED WITHOUT throwing, so the table still loads and only the exotic column is + // present-but-unqueryable. This deliberately DIVERGES from legacy fe-core, which threw + // IllegalArgumentException("Cannot transform unknown type") at schema-load and failed the whole table. + // This test PINS the graceful-degradation choice: MUTATION making either default arm throw -> red, + // surfacing that the accepted deviation was reverted. (The write direction toIcebergPrimitive still + // throws — see toIcebergPrimitiveRejectsUnrepresentableTypes if present / the connector's CREATE path.) + Assertions.assertEquals("UNSUPPORTED", mapOff(Types.TimestampNanoType.withoutZone()).getTypeName()); + Assertions.assertEquals("UNSUPPORTED", mapOff(Types.TimestampNanoType.withZone()).getTypeName()); + Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeometryType.crs84()).getTypeName()); + Assertions.assertEquals("UNSUPPORTED", mapOff(Types.GeographyType.crs84()).getTypeName()); + Assertions.assertEquals("UNSUPPORTED", mapOff(Types.UnknownType.get()).getTypeName()); + // VARIANT is NOT a primitive (falls to the nested-switch default); legacy mapped it to UNSUPPORTED + // too, so this stays parity while the primitives above are the intentional divergence. + Assertions.assertEquals("UNSUPPORTED", mapOff(Types.VariantType.get()).getTypeName()); + // The mapping flags do not rescue an unrepresentable type (both arms are flag-independent). + Assertions.assertEquals("UNSUPPORTED", mapOn(Types.GeometryType.crs84()).getTypeName()); + } + @Test public void decimalCarriesPrecisionAndScale() { // WHY: Iceberg DECIMAL(p,s) maps to Doris DECIMALV3(p,s) carrying both p and s verbatim; legacy From 76afd6f2e80aec2042d124c518ab3ee31c034791 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 11:52:09 +0800 Subject: [PATCH 328/334] [fix](catalog) iceberg: guard-comment the pinned-schema fallback symmetry (L16) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverify #65185 L16 (snapshot/schema cache skew) is VERIFIED BENIGN at HEAD: the field-id dict's snapshot-pin arm already passes the FULL pinned schema (Collections.emptyList(), not requestedLowerNames), and the snapshot-vs-schema selection is skew-free by construction — the (snapshotId, schemaId) pin is atomic, iceberg schemas() is append-only, and the dict-schema selector (IcebergScanPlanProvider.pinnedSchema) and slot-schema selector (IcebergConnectorMetadata.getTableSchema) use the SAME schemaId lookup + the SAME silent fallback to table.schema(), so their top-level names always agree. That agreement is a load-bearing invariant: a divergence would make BE's unconditional children.at(name) std::out_of_range-SIGABRT the whole BE on a schema-evolved time-travel read. Add cross-referencing comments on both fallback sites stating they MUST stay identical, so a future "defensive" edit hardening one side (e.g. throw-loud on a missing schemaId) does not silently reintroduce the asymmetry it looks like it prevents. Comments only; no behavior change. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/iceberg/IcebergConnectorMetadata.java | 5 +++++ .../connector/iceberg/IcebergScanPlanProvider.java | 10 ++++++++++ 2 files changed, 15 insertions(+) diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 10d7591559b7d2..50ffffae7b8105 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -379,6 +379,11 @@ public ConnectorTableSchema getTableSchema( schema = table.schemas().get((int) snapshot.getSchemaId()); if (schema == null) { // Defensive: a pinned id absent from table.schemas() (legacy would NPE) -> latest. + // INVARIANT: this SLOT-schema fallback MUST stay identical to the DICT-schema fallback in + // IcebergScanPlanProvider.pinnedSchema (same getSchemaId() lookup + same silent -> table.schema()). + // If the two diverge, the field-id dict names and the BE scan-slot names resolve DIFFERENT + // schemas -> BE children.at() std::out_of_range-SIGABRT on a schema-evolved time-travel read + // (reverify #65185 L16). Do not harden ONE side to throw without the other. schema = table.schema(); } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java index 311b693ae10252..a2d3e76f5d7d3d 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergScanPlanProvider.java @@ -792,6 +792,16 @@ private TableScan buildScan(Table table, IcebergTableHandle handle, OptionalINVARIANT (do not break): this dict-schema selector MUST stay byte-identical — same + * {@code getSchemaId()} lookup, same silent fallback to {@code table.schema()} — to the SLOT-schema + * selector in {@code IcebergConnectorMetadata.getTableSchema(session, handle, snapshot)}. The field-id + * dict's top-level names must equal the BE StructNode scan-slot names; a divergence (e.g. hardening ONE + * side to throw-loud on a missing schemaId while the other silently falls back) would make them resolve + * DIFFERENT schemas → BE's unconditional {@code children.at(name)} std::out_of_range-SIGABRTs the whole BE + * on a schema-evolved time-travel read. Because {@code schemas()} is append-only and the {@code schemaId} + * is the atomic pin threaded into both sides, they resolve the same schema by construction TODAY — keep it + * that way (reverify #65185 L16).

    */ private static Schema pinnedSchema(Table table, IcebergTableHandle handle) { long schemaId = handle.getSchemaId(); From 3627556db34228a8e3436d5bc988e8cdadc3468e Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 12:23:40 +0800 Subject: [PATCH 329/334] [fix](catalog) fail-loud on same-table multi-version schema skew (L17) Schema binding for a PluginDrivenMvccExternalTable is version-BLIND (getSchemaCacheValue has no per-reference selector) while the data path is version-AWARE (PluginDrivenScanNode.pinMvccSnapshot). For a self-join across a schema change (t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b, or v1 joined with a latest reference), a reference's FE tuple can be bound at a different schema than the version it scans, so BE field-id/name-mismatches file columns to tuple slots -> crash / wrong NULLs. Newly possible in this branch (the data path became per-reference version-aware while schema binding stayed version-blind). Per user decision 2026-07-13: fail loud now; the per-reference version-aware schema-binding refactor is tracked as design debt D-MVCC-VERSION-SCHEMA. A 3-lens red-team overturned an analysis-time getSchemaCacheValue guard: it is order-dependent and silently masked by a plain/latest reference's default pin, by @incr, and by MTMV refresh's pre-seeded default (the same query throws or silently skews by binding order). Instead guard at the scan node where each reference's OWN version-aware pinned schema is resolved: verify every bound tuple column resolves in that pinned schema (by iceberg field-id when present, else by name), throw UserException otherwise. Deterministic + per-reference: catches the self-join, latest-masked, @incr, and MTMV-refresh cases, with no false positive on t@old JOIN t@latest (each reference's tuple matches its own version). The check is a static, directly-unit-testable helper. No-op for a latest / @incr / sys-table / hive scan (null pinnedSchema). Tests: PluginDrivenScanNodeMvccSchemaGuardTest 7/7 (RED on field-id renumber / added column / paimon name-miss; GREEN on id-stable rename / subset projection / name match / null schema); PluginDrivenScanNodeMvccPinTest 3/3 + PluginDrivenMvccExternalTableTest 59/59 unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../datasource/PluginDrivenScanNode.java | 59 +++++++++ ...uginDrivenScanNodeMvccSchemaGuardTest.java | 124 ++++++++++++++++++ 2 files changed, 183 insertions(+) create mode 100644 fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeMvccSchemaGuardTest.java diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java index 19f260b63fd8fc..3c664113d8ae69 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenScanNode.java @@ -879,9 +879,68 @@ private void pinMvccSnapshot() throws UserException { // reading latest. Returns empty for every other case, so the normal-table path is unchanged. snapshot = resolveSysTableSnapshotPin(); } + // L17 fail-loud guard: this scan reads at its version-aware snapshot (resolved above), but the + // table's schema binding (PluginDrivenMvccExternalTable.getSchemaCacheValue) is version-BLIND, so a + // same-table multi-version reference (e.g. self-join FOR VERSION AS OF v1 vs v2 across a schema + // change, or v1 joined with a latest reference) can carry an FE tuple bound at a DIFFERENT schema + // than the version this reference scans -> BE field-id/name-mismatches file columns to tuple slots + // (crash / wrong NULLs). Verify every bound tuple column resolves in THIS reference's pinned schema; + // throw a clear error instead of silently skewing. Deterministic + per-reference (runs with all pins + // loaded, checks this reference's own pin), so it catches the latest-masked / @incr / MTMV-refresh + // cases the version-blind analysis-time binding cannot. Per user decision 2026-07-13: throw for now; + // the per-reference version-aware schema-binding refactor is tracked as D-MVCC-VERSION-SCHEMA. A + // latest / @incr / sys-table / hive scan carries a null pinnedSchema -> no-op. + if (snapshot.isPresent() && snapshot.get() instanceof PluginDrivenMvccSnapshot) { + List boundColumns = new ArrayList<>(); + for (SlotDescriptor slot : desc.getSlots()) { + if (slot.getColumn() != null) { + boundColumns.add(slot.getColumn()); + } + } + assertBoundColumnsResolveInPinnedSchema(boundColumns, + ((PluginDrivenMvccSnapshot) snapshot.get()).getPinnedSchema(), + getTargetTable().getName()); + } currentHandle = applyMvccSnapshotPin(metadata, connectorSession, currentHandle, snapshot); } + /** + * Fail-loud guard for the version-blind schema-binding gap (reverify #65185 L17). {@code boundColumns} + * are the projected tuple-slot columns (what the FE tuple / BE scan slots expect); {@code pinnedSchema} + * is the schema AS OF the version THIS reference actually scans. If any bound column cannot be resolved in + * the pinned schema, the tuple was bound at a different version than the scan reads and BE would + * mismatch — throw instead of silently returning wrong rows. + * + *

    Matching keys on whether the column carries a field id (a generic {@link Column} property, NOT a + * source-name branch): {@code uniqueId >= 0} (iceberg carries the iceberg field-id) matches by field-id + * (BE matches iceberg columns by id, so a rename that keeps the id is fine, a renumber / added column is + * caught); {@code uniqueId < 0} (paimon has no top-level field-id) matches by name. A {@code null} + * pinnedSchema (latest / {@code @incr} / sys-table / hive reference) is a no-op.

    + */ + static void assertBoundColumnsResolveInPinnedSchema(List boundColumns, + SchemaCacheValue pinnedSchema, String tableName) throws UserException { + if (pinnedSchema == null) { + return; + } + Set pinnedFieldIds = new HashSet<>(); + Set pinnedNames = new HashSet<>(); + for (Column c : pinnedSchema.getSchema()) { + pinnedFieldIds.add(c.getUniqueId()); + pinnedNames.add(c.getName().toLowerCase()); + } + for (Column bound : boundColumns) { + boolean resolved = bound.getUniqueId() >= 0 + ? pinnedFieldIds.contains(bound.getUniqueId()) + : pinnedNames.contains(bound.getName().toLowerCase()); + if (!resolved) { + throw new UserException("Reading the same table at multiple versions with different schemas " + + "in one statement is not supported yet: column '" + bound.getName() + "' of table '" + + tableName + "' is bound at a different version than the one this reference scans. " + + "Rewrite as separate statements."); + } + } + } + /** * Threads a distributed {@code rewrite_data_files} group's per-group file scope onto {@code handle} * BEFORE {@code planScan}, so the group's INSERT-SELECT scans ONLY the data files that group bin-packed diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeMvccSchemaGuardTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeMvccSchemaGuardTest.java new file mode 100644 index 00000000000000..f5857ef8c68edc --- /dev/null +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenScanNodeMvccSchemaGuardTest.java @@ -0,0 +1,124 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.datasource; + +import org.apache.doris.catalog.Column; +import org.apache.doris.catalog.Type; +import org.apache.doris.common.UserException; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +/** + * Unit tests for the L17 fail-loud guard {@link PluginDrivenScanNode#assertBoundColumnsResolveInPinnedSchema}: + * a same-table multi-version reference whose FE tuple was bound at a DIFFERENT schema than the version it + * scans must be rejected loud (BE would field-id/name-mismatch), while a reference whose bound columns all + * resolve in its own version-aware pinned schema (incl. a field-id-stable rename or a subset projection) + * must pass. The helper takes plain {@link Column}s + a {@link SchemaCacheValue} so it is exercised without + * constructing a scan node. + */ +public class PluginDrivenScanNodeMvccSchemaGuardTest { + + private static Column col(String name, int uniqueId) { + Column c = new Column(name, Type.INT); + c.setUniqueId(uniqueId); + return c; + } + + private static SchemaCacheValue schema(Column... cols) { + return new SchemaCacheValue(Arrays.asList(cols)); + } + + @Test + public void fieldIdRenumberBetweenBoundAndScannedVersionThrows() throws UserException { + // The tuple was bound at LATEST where column `c` has field-id 7, but THIS reference scans a pinned + // version where `c` has field-id 5. BE matches iceberg columns by field-id, so slot field-id 7 has no + // entry in the v-pinned dict -> crash. The guard must throw. MUTATION: dropping the guard (or matching + // by name only) -> the renumber slips through -> red. + List bound = Collections.singletonList(col("c", 7)); + SchemaCacheValue pinned = schema(col("c", 5)); + UserException e = Assertions.assertThrows(UserException.class, + () -> PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, pinned, "db.t")); + Assertions.assertTrue(e.getMessage().contains("multiple versions"), e.getMessage()); + Assertions.assertTrue(e.getMessage().contains("'c'"), e.getMessage()); + } + + @Test + public void columnAddedAfterScannedVersionThrows() throws UserException { + // The tuple (bound latest) references a column `added` (field-id 9) that does NOT exist in the pinned + // version this reference scans -> the version's files have no such field -> the guard must throw. + List bound = Arrays.asList(col("id", 1), col("added", 9)); + SchemaCacheValue pinned = schema(col("id", 1)); // pinned version predates `added` + UserException e = Assertions.assertThrows(UserException.class, + () -> PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, pinned, "db.t")); + Assertions.assertTrue(e.getMessage().contains("'added'"), e.getMessage()); + } + + @Test + public void nameMissWhenNoFieldIdThrows() throws UserException { + // Paimon carries no top-level field-id (uniqueId == -1) so matching is by NAME: a tuple bound at + // LATEST with column `newname` scanning a version that only has `oldname` (a paimon rename) must throw + // (BE matches by name -> `newname` unreadable from the old files). + List bound = Collections.singletonList(col("newname", -1)); + SchemaCacheValue pinned = schema(col("oldname", -1)); + Assertions.assertThrows(UserException.class, + () -> PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, pinned, "db.t")); + } + + @Test + public void fieldIdStableRenameResolvesByIdNoThrow() throws UserException { + // A rename that KEEPS the field-id is fine: BE reads iceberg field-id 5 regardless of name, so a tuple + // bound with `newname`@5 scanning a version whose column is `oldname`@5 must NOT throw (id resolves). + // Guards against a name-only check over-rejecting the benign rename case. + List bound = Collections.singletonList(col("newname", 5)); + SchemaCacheValue pinned = schema(col("oldname", 5), col("other", 6)); + Assertions.assertDoesNotThrow(() -> + PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, pinned, "db.t")); + } + + @Test + public void subsetProjectionAllResolvedNoThrow() throws UserException { + // The tuple is a projection (subset) of the version's columns; every bound field-id is present -> ok. + List bound = Arrays.asList(col("a", 1), col("c", 3)); + SchemaCacheValue pinned = schema(col("a", 1), col("b", 2), col("c", 3)); + Assertions.assertDoesNotThrow(() -> + PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, pinned, "db.t")); + } + + @Test + public void nameMatchWhenNoFieldIdNoThrow() throws UserException { + // Paimon (uniqueId == -1) matching by name: same names -> resolved -> no throw. + List bound = Arrays.asList(col("a", -1), col("b", -1)); + SchemaCacheValue pinned = schema(col("a", -1), col("b", -1), col("c", -1)); + Assertions.assertDoesNotThrow(() -> + PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, pinned, "db.t")); + } + + @Test + public void nullPinnedSchemaIsNoOp() throws UserException { + // A latest / @incr / sys-table / hive reference carries a null pinnedSchema -> the guard is a no-op + // (no version-at-snapshot schema to skew against), regardless of the bound columns. + List bound = Collections.singletonList(col("anything", 42)); + Assertions.assertDoesNotThrow(() -> + PluginDrivenScanNode.assertBoundColumnsResolveInPinnedSchema(bound, null, "db.t")); + } +} From f82d6eefa699e4c939c767b9150de21aed1daf31 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 12:25:21 +0800 Subject: [PATCH 330/334] [doc](catalog) #65185 reverify: L16-L19 done (designs, DV-051, HANDOFF, D-MVCC-VERSION-SCHEMA) Land the L16-L19 fix designs and tracking: - FIX-L16-design.md: verified REFUTED/benign (guard-comment only) - FIX-L17-design.md: scan-node fail-loud guard (red-team overturned the analysis-time placement); registers D-MVCC-VERSION-SCHEMA design debt (user's "refactor later" TODO) - FIX-L18-design.md: accept UNSUPPORTED graceful degradation (user decision), DV-051 - FIX-L19-design.md: producer-side reserved-key strip across iceberg/hive/paimon - deviations-log.md: DV-051 (iceberg unknown/v3 types -> UNSUPPORTED, accepted) - task-list: mark L16-L19 done, add D-MVCC-VERSION-SCHEMA, sync progress table + rolling log - HANDOFF.md: L16-L19 summary + remaining = L20 decision + live-gated e2e Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 10 ++- plan-doc/deviations-log.md | 3 +- plan-doc/task-list-65185-reverify-fixes.md | 29 ++++--- plan-doc/tasks/designs/FIX-L16-design.md | 52 ++++++++++++ plan-doc/tasks/designs/FIX-L17-design.md | 93 ++++++++++++++++++++++ plan-doc/tasks/designs/FIX-L18-design.md | 42 ++++++++++ plan-doc/tasks/designs/FIX-L19-design.md | 72 +++++++++++++++++ 7 files changed, 290 insertions(+), 11 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L16-design.md create mode 100644 plan-doc/tasks/designs/FIX-L17-design.md create mode 100644 plan-doc/tasks/designs/FIX-L18-design.md create mode 100644 plan-doc/tasks/designs/FIX-L19-design.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index e038aa1d532b59..3b7436888b8c2f 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -15,7 +15,15 @@ > > **L15 追加转折 → scan-metrics 功能恢复** `8d4209865a3`:用户质疑 L15「删死引用」是否丢功能。复核参考仓库 master 证实:老 `PaimonScanMetricsReporter`+`PaimonMetricRegistry` 是真功能,插件迁移 **paimon+iceberg 都丢**(登记偏差 T02/T29);iceberg `ICEBERG_SCAN_METRICS` 仅靠**死 legacy `IcebergScanNode`** 续命(我先前「iceberg 活」判断有误)。用户要求 connector SPI 框架**统一恢复**。已建**连接器中立 scan-metrics SPI**:`ConnectorScanProfile`(值类型)+`ConnectorScanPlanProvider.collectScanProfiles`(默认空);paimon 侧 port `PaimonMetricRegistry`+新 `PaimonScanMetrics`(pull,`withMetricRegistry`),iceberg 侧新 `IcebergScanProfileReporter`(push,**挂 `planScanInternal` 数据/count 路,非共享 `buildScan`**——避 streaming 泄漏 blocker);fe-core 纯静态 `writeScanProfilesInto` 写 `SummaryProfile`(无源分支)、**复活 `PAIMON_SCAN_METRICS` 常量**;`DebugUtil` 时间/字节格式化器**自搬**(连接器不得 import fe-core)。stash 按 queryId、`collectScanProfiles` 排空 + `releaseReadTransaction` 兜底清。设计 3-lens 红队 `wf_0f803c49-7bb` 全 SOUND_WITH_CHANGES(2 blocker+majors 全折入)。UT 12/12 + 全模块 paimon 360+iceberg 982+fe-core 94 绿、0 checkstyle、import 门净。**踩坑**:iceberg 既有 UT 传 **null session** → 我 `session.getQueryId()` NPE 挂 31 例,加 null-session guard 修(paimon 同防御);paimon 模块须 `package`(shade)非 `test-compile`(HiveConf 假失败,memory `doris-build-verify-gotchas`)。设计/summary=`designs/FIX-scan-metrics-spi-{design,summary}.md`。**e2e live-gated**:真集群 paimon/iceberg 查询后 profile 含 "Paimon/Iceberg Scan Metrics" 组 + manifest 缓存命中/未命中。 > -> **下一步**:#65185 复核系列剩余 = `L16`(iceberg 缓存偏斜)、`L17`(iceberg version-blind schema)、`L18`(iceberg 未知类型)、`L19`(partition_columns 撞名);⏸ 决策类**仅余 `L20`**(MC EQ `==`,须先中文讲清问用户)。⏸ `M8`(升级文档)用户 07-12 跳过、`D-系列`设计债随 P8。 +> **L16–L19 已全部 DONE(本 session 续，2026-07-13）** — 4 recon agent(`wf_0aee6600-244`)复核 HEAD；两条决策类(L17/L18)用中文讲清 background+Trino+选项问用户后实现。commits: +> - **L19** `01668779fd9` — partition_columns 保留键撞名:复核**扩 scope 到 iceberg+hive+paimon 三连接器**(同 bug；hudi/mc 用全新 map 免疫)。producer-side `remove`(fe-core 无法区分连接器发 vs 用户透传，铁律禁解析连接器键义)。iceberg 50/50+hive 23/23 UT(各 2 RED-able)、paimon package 编译、import 门净;paimon coreOptions 路现有 fake 非 `DataTable` 不可单测→检视+e2e-gated 登记。 +> - **L18** `1c9c99c7767` — 未知/v3 类型静默 UNSUPPORTED:**用户签字 accept graceful degradation**(非抛，登记 **DV-051**);无功能改动，加澄清注释+守护测试 `unknownAndV3TypesDegradeToUnsupportedByDesign`(未来改抛→red)。写方向 `toIcebergPrimitive` 仍抛。IcebergTypeMappingReadTest 11/11。 +> - **L16** `76afd6f2e80` — 快照/schema 缓存偏斜:复核判 **REFUTED/benign**(hasSnapshotPin 臂已传 `emptyList` 非超集;偏斜构造不出——原子 `(snapshotId,schemaId)` pin + iceberg `schemas()` append-only + `pinnedSchema`(dict)/`getTableSchema`(slot)同选择器同静默回退)。**无功能改动**，仅加交叉引用防呆注释锁「两处回退须一致」不变量(否则 BE `children.at()` SIGABRT)。IcebergScanPlanProviderTest 88/88。 +> - **L17** `3627556db34` — 同表多版本 version-blind schema:**用户签字 fail-loud + TODO 重构**(`D-MVCC-VERSION-SCHEMA`)。3-lens 红队 `wf_f7b69cf7-ec8` **推翻 analysis-time `getSchemaCacheValue` 守卫**(顺序依赖 + 被 plain-ref/MTMV default pin 遮蔽 + @incr 漏 → 同 query 抛或静默 skew 视绑定序)→改**逐引用 scan-node 守卫**:`PluginDrivenScanNode.pinMvccSnapshot` 解析版本感知 pin 后，校验每个 bound tuple 列在**本引用 pinned schema** 可解析(有 iceberg field-id 按 id、否则按 name)，否则抛 `UserException`。确定性+完整(catch 自连接/latest-遮蔽/@incr/MTMV)+无误报(`t@old JOIN t@latest` 各 tuple 匹配自身版本→不抛)。静态可测 helper `assertBoundColumnsResolveInPinnedSchema`。SchemaGuardTest 7/7 + MvccPin 3/3 + MvccExternalTable 59/59。残余:嵌套 field-id-only 重编号看不见(iceberg id 稳定不触发)。 +> +> **e2e 全 live-gated**(真集群):L19 非分区表 `SET TBLPROPERTIES('partition_columns'=真列名)` 不再误判为分区;L18 v3 `GEOMETRY` 列建表可加载、`SELECT geom` 报错、`SELECT 其它列` 可用;L16 无(纯注释);**L17** iceberg `t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b` 跨 ALTER schema 演进 → 抛「same table at multiple versions with different schemas not supported yet」清晰错(非 BE 崩)。 +> +> **下一步**:#65185 复核系列 H/M/L 主体**已全部收口**;⏸ 决策类**仅余 `L20`**(MC EQ `==` vs `=`，须先中文讲清背景问用户,可顺带补 P4-3-IN 方向回归测试)。⏸ `M8`(升级文档)用户 07-12 跳过;`D-系列`设计债(新增 `D-MVCC-VERSION-SCHEMA`)随 P8/择机。**所有本系列 e2e 均 live-gated 待用户真集群自跑**(memory `hms-iceberg-delegation-needs-e2e`)。 --- diff --git a/plan-doc/deviations-log.md b/plan-doc/deviations-log.md index 4fd4e8f379c590..aa4d784a11d030 100644 --- a/plan-doc/deviations-log.md +++ b/plan-doc/deviations-log.md @@ -13,10 +13,11 @@ ## 📋 索引 -> 时间倒序；当前共 **49** 项（最新 DV-049；本轮 P6.5-T07 对抗 byte-parity 审计〔8 area finder + refute-by-default skeptic + completeness critic，22 finding / 19 confirmed〕把 P6.5 sys-table 残留 deviation 批化中央登记为 DV-048〔correctness-bearing〕/049〔perf-cosmetic/display/internal〕——**审计揭出的 2 项主动偏差〔sys 时间旅行 guard 拒绝 + hms 大小写〕用户裁「现修」故 NOT-DV，见 [D-067]**；二层镜像 P6.4-T08 DV-045..047）。 +> 时间倒序；当前共 **51** 项（最新 DV-051 = iceberg 未知/v3 类型静默降级 UNSUPPORTED，用户 2026-07-13 签字 accept；DV-050 = EXPLAIN 通用节点名 accept；本轮 P6.5-T07 对抗 byte-parity 审计〔8 area finder + refute-by-default skeptic + completeness critic，22 finding / 19 confirmed〕把 P6.5 sys-table 残留 deviation 批化中央登记为 DV-048〔correctness-bearing〕/049〔perf-cosmetic/display/internal〕——**审计揭出的 2 项主动偏差〔sys 时间旅行 guard 拒绝 + hms 大小写〕用户裁「现修」故 NOT-DV，见 [D-067]**；二层镜像 P6.4-T08 DV-045..047）。 | 编号 | 偏差主题 | 原计划位置 | 日期 | 当前状态 | |---|---|---|---|---| +| DV-051 | **iceberg 未知/v3 类型静默降级 UNSUPPORTED（不抛，用户 2026-07-13 签字 accept）**：`IcebergTypeMapping.fromIcebergType/fromPrimitive` 两处 `default` 臂把 Doris 无法表示的 iceberg 类型（v3 primitives `TIMESTAMP_NANO`/`GEOMETRY`/`GEOGRAPHY`/`UNKNOWN` + 非-primitive `VARIANT`）映射为 `UNSUPPORTED` → 表**能加载**、该列 present-but-unqueryable、其它列可用。**背离** legacy fe-core `IcebergUtils.icebergTypeToDorisType`（未知类型 `throw IllegalArgumentException("Cannot transform unknown type")` 在 schema-load 失败整表）与 **Trino**（`TypeConverter` 对未映射类型抛 `TrinoException(NOT_SUPPORTED)`）。**用户选「统一映射 UNSUPPORTED」**（非抛）：一列冷门类型不致整宽表不可加载。`TIME`/`VARIANT` 本就显式/等价 UNSUPPORTED（=legacy parity）；分歧仅在 v3 primitives（legacy 抛）。**写方向 `toIcebergPrimitive` 仍抛**（CREATE TABLE 不得静默接受不可 round-trip 类型，不动）。守护测试 `IcebergTypeMappingReadTest.unknownAndV3TypesDegradeToUnsupportedByDesign` 钉此选择（未来改抛→red）。**无回归**：现有 fixture 无 GEOMETRY 等冷门列 | reverify §1 L18 / [task 表 §L18](./task-list-65185-reverify-fixes.md) | 2026-07-13 | 🟢 已登记（accept；graceful degradation；守护测试锁定；e2e live 验冷门列表可加载/该列不可查）| | DV-050 | **翻闸后 EXPLAIN 外部表扫描节点显示通用名 `VPluginDrivenScanNode`**（display-only；用户 2026-07-12 签字 accept）：翻闸前各源显示 `V_SCAN_NODE`（`VHIVE_SCAN_NODE`/`VICEBERG_SCAN_NODE` 等），翻闸后所有外部表走通用节点〔`PluginDrivenScanNode:173` 传 super label `"PluginDrivenScanNode"`〕→ EXPLAIN 统一 `VPluginDrivenScanNode`，且 `getNodeExplainString:325` 已附 `CONNECTOR: ` 行披露实际对接的数据源。**纯显示，无功能/结果/性能影响**。选 **accept**（非加 `Connector.getLegacyEngineName` SPI 恢复旧名）：`CONNECTOR:` 行信息未丢、regression 黄金文件已适配为 `VPluginDrivenScanNode`（全 regression 树仅 1 处引用扫描节点名且已是新名）、且与 **Trino** 一致（Trino EXPLAIN 对所有连接器统一 `TableScan` 通用节点名 + 连接器/表作属性，而非塞进节点类名）。否决 Option B（加 SPI 声明旧引擎名——不能用 catalog type 拼，hive 的 type=`hms`→会误拼 `VHMS_SCAN_NODE`；改动面大且须把黄金文件改回去，仅为复刻一个 cosmetic 串）。关联设计债 D-ENGINE（引擎名收口）择机随 P8 | reverify §1 L10 / [task 表 §L10](./task-list-65185-reverify-fixes.md) | 2026-07-12 | 🟢 已登记（accept；display-only；EXPLAIN reg 黄金已用新名）| | DV-049 | **P6.5 iceberg sys-table perf-cosmetic/display/internal 批汇总**（结果恒等/展示/内部枚举；镜像 DV-047/044 style）：**①sys split self-weight 丢**〔audit `T05-sys-split-weight`：legacy `IcebergScanNode.createIcebergSysSplit:900`+`IcebergSplit.newSysTableSplit:78` 设 `selfSplitWeight=Math.max(recordCount,1L)`；连接器 `IcebergScanRange` 无 `getSelfSplitWeight()` override → SPI 默认 -1 → `PluginDrivenSplit` `SplitWeight.standard()` 均匀。**result-equivalent**——查询结果/thrift 字段/serialized_split 字节皆不变，仅 BE `FederationBackendPolicy` 调度权重差；镜像 [DV-033] native-subsplit weight 不移植〕·**②内部 `TableIf.TableType=PLUGIN_EXTERNAL_TABLE`**〔legacy `IcebergSysExternalTable:57`=`ICEBERG_EXTERNAL_TABLE`〕——但 **user-visible 全 parity**：`getMysqlType`→`information_schema.tables.TABLE_TYPE` 两枚举皆 fall-through "BASE TABLE"〔`TableIf:319/324-325`〕、engine name 经 [D-066] T06-F1="iceberg"、descriptor 经 T06-C1=HIVE/ICEBERG_TABLE → 残留**仅内部枚举**〔无 user 可观测面〕·**③position_deletes 文案**〔Q2 用户签字：连接器 `listSupportedSysTables` 去 `POSITION_DELETES`+`getSysTableHandle`→empty → 通用 fe-core not-found（"Unknown sys table"）vs legacy bespoke "is not supported yet"（`IcebergSysTable:74`）；两侧 support boolean 同〔皆不可查〕仅文案分叉；**reg-test 已同步 2026-06-30**：`test_iceberg_sys_table.groovy` position_deletes 断言从旧 bespoke 文案改断通用 `Unknown sys table '$position_deletes'`，docker e2e 实跑绿〕。全部**非正确性** | T07 对抗审计 / [task 表 §P6.5](./tasks/P6-iceberg-migration.md) | 2026-06-25 | 🟢 已登记（accept；结果恒等/内部/展示；P6.6 docker/live 真值闸）| | DV-048 | **P6.5 iceberg sys-table correctness-bearing 但 UT 不可见**（parity-by-construction / 用户签字；docker 闸）：**①F2 paimon SHOW CREATE priv loosening（LIVE）**〔audit `F2-paimon-showcreate-priv-loosened-live` + critic：[D-066] T06-F2 给 `ShowCreateTableCommand.validate():120-124` 加 `PluginDrivenSysExternalTable`→`getSourceTable().getName()` 解包；**对 iceberg 是 parity**〔legacy `IcebergSysExternalTable` 分支 `:118-119` 已解包，且 iceberg pre-flip dormant〕，但 **paimon 在 SPI_READY_TYPES** → live 行为变更：sys `$`-表 SHOW CREATE 现按 **base** 表授权（pre-T06 按合成 'tbl$snapshots' 名）；严格**更宽松**、同向，与 `Env.getDdlStmt`/`UserAuthentication` 既有 output 解包一致、破坏风险近零；用户 T06-Q2 签字。**⚠️ 无隔离 UT**〔`validate()` 依赖 `Env`/`ConnectContext`/`AccessManager` 全局单例〕→ P6.8 e2e 兜底〕·**②serialized `FileScanTask` 字节潜伏（T05）**〔FE `SerializationUtil` deserialize-round-trip UT 已在**同进程 iceberg 1.10.1** 核「可消费+asDataTask+meta schema」，但与 BE `IcebergSysTableJniScanner` 的**跨版本/classloader interop** FE 不可及 → P6.8 docker e2e 兜底〕。**别于 DV-041**：本条**已建待 docker 实证**，DV-041 是未接线翻闸阻塞 | T07 对抗审计 / [task 表 §P6.5](./tasks/P6-iceberg-migration.md) | 2026-06-25 | 🟢 已登记（accept；P6.6 docker/Kerberos 真值闸）| diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 4f7e41ba10edcf..bc8cfac777d4d6 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -52,10 +52,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 25 | **L13** | 🟡低 | paimon | to-Paimon 丢嵌套 nullability | ✅ | ✅ | ✅ | ✅ | | 26 | **L14** | 🟡低 | paimon | ignore_split_type 静默 no-op | ✅ | ✅ | ✅ | ✅ | | 27 | **L15** | 🟡低 | fe-core | PAIMON_SCAN_METRICS 悬空常量 | ✅ | ✅ | ✅ | ✅ | -| 28 | **L16** | 🟡低 | iceberg | 快照/schema 缓存偏斜(防御性 union) | ⬜ | ⬜ | ⬜ | ⬜ | -| 29 | **L17** | 🟡低 | fe-core/iceberg | 同表多版本 version-blind schema 绑定 | ⬜ | ⬜ | ⬜ | ⬜ | -| 30 | **L18** | 🟡低 | iceberg | 未知/v3 类型静默 UNSUPPORTED | ⬜ | ⬜ | ⬜ | ⬜ | -| 31 | **L19** | 🟡低 | fe-core/iceberg | partition_columns 魔法键撞名→误判分区 | ⬜ | ⬜ | ⬜ | ⬜ | +| 28 | **L16** | 🟡低 | iceberg | 快照/schema 缓存偏斜(防御性 union) | ✅ | ✅ | ✅ | ✅ 复核 REFUTED+防呆注释 | +| 29 | **L17** | 🟡低 | fe-core/iceberg | 同表多版本 version-blind schema 绑定 | ✅ | ✅ | ✅ | ✅ 用户签字 fail-loud+TODO | +| 30 | **L18** | 🟡低 | iceberg | 未知/v3 类型静默 UNSUPPORTED | ✅ | ✅ | ✅ | ✅ 用户签字 accept(DV-051) | +| 31 | **L19** | 🟡低 | fe-core/iceberg | partition_columns 魔法键撞名→误判分区 | ✅ | ✅ | ✅ | ✅ | | 32 | **L20** | 🟡低 | maxcompute | EQ 发 `==`(对齐 `=` 或 live A/B) | ⬜ | ⬜ | ⬜ | ⏸ 或 live | | — | **D-系列** | ⚪设计债 | — | 见文末「设计债跟踪」(多为 P8 前置/需一次性重构) | — | — | — | ⏸ | @@ -193,10 +193,10 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L13** DONE `ced4775b844`(设计 `FIX-L13-design.md`;红队 3 lens 全 SOUND）· `toPaimonType` 对 ARRAY 元素/MAP value/STRUCT 字段加 `.copy(type.isChildNullable(i))`(MAP key 保持 `.copy(false)`),恢复 legacy `DorisToPaimonTypeVisitor` 嵌套 nullability parity。**scope=仅 nullability**:comment 丢=DV-035 M10.1 已接受偏差、field-id 顺序 parity 均不动。`.copy(true)` 对默认可空子类型逐字节恒等(既有 parity 测保持绿)。3 新 RED-able 测(经 `.type().isNullable()` 断言,非 DataField equals)。type-mapping+schema-builder 26/26 绿。e2e live-gated(建嵌套 NOT NULL 表 DESCRIBE/SDK 读回)。 - [x] **L14** DONE `478718aca6f`(设计 `FIX-L14-design.md`;红队 3 lens SOUND/SOUND_WITH_CHANGES）· null-tolerant `resolveIgnoreSplitType(session)`(镜像 `isCppReaderEnabled`,红队 MINOR)+三 legacy `continue` 位:`IGNORE_JNI` drops nonDataSplit+DataSplit-JNI 臂、`IGNORE_NATIVE` drops native 臂、count 臂不检、`IGNORE_PAIMON_CPP` 保持 no-op(=legacy parity,全树 grep 证 legacy 从不引用)。默认 NONE 逐字节不变。3 新 live-planScan RED 测(IGNORE_JNI/IGNORE_NATIVE 各证跳 split+IGNORE_PAIMON_CPP==NONE);nonDataSplit IGNORE_JNI 位离线不可测→留 E2E-only。69/69 绿。e2e live-gated(真集群 SET ignore_split_type 断言跳分片)。 - [x] **L15 → 被 scan-metrics 功能取代**(用户 2026-07-13 追加要求恢复功能,非删死引用):先删 `b2cdf971889`(判为死引用),后**复核发现是插件迁移丢的真功能**(paimon+iceberg 都丢;iceberg 的 `ICEBERG_SCAN_METRICS` 仅靠死 legacy `IcebergScanNode` 续命,我先前「iceberg 活」判断有误)→ **改为恢复**:`8d4209865a3` 建**连接器中立 scan-metrics SPI**(`ConnectorScanProfile`+`collectScanProfiles`;paimon `PaimonMetricRegistry`+`PaimonScanMetrics` pull、iceberg `IcebergScanProfileReporter` push;fe-core 纯静态 `writeScanProfilesInto` 写 profile;**复活** `PAIMON_SCAN_METRICS` 常量)。设计 3-lens 红队 `wf_0f803c49-7bb` 全 SOUND_WITH_CHANGES(2 blocker=iceberg streaming 泄漏改挂 planScanInternal、DebugUtil 自搬,+ majors 全折入)。设计/summary=`designs/FIX-scan-metrics-spi-{design,summary}.md`。UT fe-core 4/4+paimon 4/4+iceberg 4/4,全模块 paimon 360+iceberg 982+fe-core 94 绿,0 checkstyle,import 门净。e2e live-gated(profile 含 X Scan Metrics 组)。 -- [ ] **L16** iceberg 缓存偏斜(部分已修) · `IcebergScanPlanProvider.java:1077-1108`:hasSnapshotPin 臂把 field-id dict 建成「pinned schema ∪ requested latest columns」超集(传 requestedLowerNames 而非 `emptyList`),两向都超集;或 fe-core 侧 query-begin pin 解析 pinnedSchema 到 pinned schemaId(对齐 time-travel 臂 `:377-387`)。 -- [ ] **L17** iceberg 同表多版本 version-blind 绑定 · `PluginDrivenMvccExternalTable.java:475-485`:per-reference schema 绑定 version-aware(用 `StatementContext.getSnapshot(table, tableSnapshot, scanParams)` 的 pinnedSchema,fallback latest);与 L16 同根,建议一并。窄触发 + fail-loud。 -- [ ] **L18** iceberg 未知/v3 类型静默 UNSUPPORTED · `IcebergTypeMapping.java:91,143`:两 default 臂改 `throw DorisConnectorException("Cannot transform unknown type: "+...)`,保留显式 TIME/VARIANT UNSUPPORTED;或接受更松并登记。 -- [ ] **L19** partition_columns 魔法键撞名 · `PluginDrivenExternalTable.java:512,703-714` + `IcebergConnectorMetadata.java:409-445`:putAll 前从源 `table.properties()` 剔除保留键;iceberg buildTableSchema 在分区/非分区分支前移除已存在的 `partition_columns`,使非分区表不继承用户 `partition_columns`。 +- [x] **L16** DONE `76afd6f2e80`(复核 REFUTED+防呆注释;设计 `designs/FIX-L16-design.md`)· 复核判定**已消除/benign**:`IcebergScanPlanProvider` hasSnapshotPin 臂现已传 `Collections.emptyList()`(全量 pinned schema,非 requestedLowerNames 超集,自首 commit 即如此);残留「快照 id vs schema id 偏斜」构造不出——pin 原子取 `(snapshotId,schemaId)`+iceberg `schemas()` 只增+两条取 schema 路径(`pinnedSchema` dict / `getTableSchema` slot)用同一选择器同一静默回退。**无功能改动**,仅加交叉引用防呆注释锁住「两处回退必须一致」不变量(否则 BE `children.at()` SIGABRT)。残留结构隐患=L17 同根(逐引用版本感知)。 +- [x] **L17** DONE `3627556db34`(设计 `designs/FIX-L17-design.md`;3-lens 红队 `wf_f7b69cf7-ec8` 推翻 analysis-time 方案→改 scan-node)· **用户 2026-07-13 签字:先 fail-loud + 记 TODO 重构**。红队证 analysis-time `getSchemaCacheValue` 守卫**顺序依赖+被 plain-ref/MTMV default pin 遮蔽+@incr 漏**(同一 query 抛或静默 skew 视绑定序)→改**逐引用 scan-node 守卫**:`PluginDrivenScanNode.pinMvccSnapshot` 解析版本感知 pin 后,校验每个 bound tuple 列在**本引用 pinned schema** 可解析(有 field-id 按 id、否则按 name),否则抛。确定性+完整(catch 自连接/latest-遮蔽/@incr/MTMV)+无误报(`t@old JOIN t@latest` 各 tuple 匹配自身版本→不抛)。静态可测 helper,7 RED-able UT。TODO=`D-MVCC-VERSION-SCHEMA`(逐引用版本感知 schema 绑定,仿 Trino 版本作用域 handle)。残余:嵌套 field-id-only 重编号看不见(iceberg id 稳定不触发)。 +- [x] **L18** DONE `1c9c99c7767`(设计 `designs/FIX-L18-design.md`;登记 **DV-051**)· **用户 2026-07-13 签字 accept「统一映射 UNSUPPORTED」**(非抛):`IcebergTypeMapping.fromIcebergType/fromPrimitive` 两 `default` 臂保持把 Doris 无法表示的 iceberg 类型(v3 primitives TIMESTAMP_NANO/GEOMETRY/GEOGRAPHY/UNKNOWN + non-primitive VARIANT)映射 UNSUPPORTED→表能加载、该列 present-but-unqueryable、其它列可用。**背离** legacy(抛 `IllegalArgumentException`)与 Trino(抛 NOT_SUPPORTED),但用户选 graceful degradation。**无功能改动**;加澄清注释+守护测试 `unknownAndV3TypesDegradeToUnsupportedByDesign`(未来改抛→red);写方向 `toIcebergPrimitive` 仍抛(CREATE 不静默接受不可 round-trip 类型)。 +- [x] **L19** DONE `01668779fd9`(设计 `designs/FIX-L19-design.md`)· 连接器侧 `remove` 保留键防撞名。**scope 扩至 iceberg+hive+paimon 三连接器**(复核发现同 bug:三者均 `putAll(源属性)` 后仅有条件盖 `partition_columns`,非分区表漏出用户同名键→误判分区;hudi/maxcompute 用全新 map 免疫)。修=各连接器 `putAll` 后立即 `remove("partition_columns")`(paimon 另 `remove("primary_keys")`),分区表下方 re-stamp 连接器真值→连接器值必胜。fe-core 无法区分连接器发 vs 用户透传(铁律禁解析连接器键义)故必须 producer-side。iceberg+hive 各 2 RED-able UT;paimon `DataTable` coreOptions 路现有 fake 非 DataTable 不可单测→按检视+e2e-gated 登记(非静默略过)。 - [ ] **L20** ⏸ MC EQ `==` · `MaxComputePredicateConverter.java:145-146`:直接 `case EQ: opDesc = "=";` 对齐 SDK/legacy 消除不确定性(推荐);或 live ODPS A/B 确认容忍。顺带补 IN 方向回归测试(P4-3-IN 已修但缺测)。 --- @@ -213,7 +213,8 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [ ] **D-TCCL** · 抽 `Tccl.pin(loader,Runnable)` 到 fe-connector-spi/support,iceberg/paimon/hive(内联 `HiveConnectorMetadata:798-805,832-845`)共用。 - [ ] **D-HIVECONF** · 合并 hive 三处 `buildHadoopConf`(`HiveConnector:621`/`HiveScanPlanProvider:444`/`HiveConnectorMetadata:967`)为一私有 helper;长期共享 `HadoopConfBuilder`。 - [ ] **D-FAKESTUB**(测试) · 加共享 `AbstractFakeHmsClient` 测试夹具,hive/hudi ~11 份桩收敛。 -- [ ] **D-MAGICKEY** · SHOW CREATE 子句换结构化 `ConnectorTableDdl`(transform+sort 作数据,fe-core 单一 altitude 渲染);至少把 `partition_columns`/`primary_keys` 提为声明常量。 +- [ ] **D-MAGICKEY** · SHOW CREATE 子句换结构化 `ConnectorTableDdl`(transform+sort 作数据,fe-core 单一 altitude 渲染);至少把 `partition_columns`/`primary_keys` 提为声明常量。**L19 已在连接器侧(iceberg/hive/paimon)`remove` 保留键防撞名**;此项为长期结构化收口。 +- [ ] **D-MVCC-VERSION-SCHEMA**(承接 L17 用户 TODO,2026-07-13)· 让 schema 绑定端到端**按引用版本感知**(仿 Trino:版本作用域 `ConnectorTableHandle` / 逐引用列句柄解析),使同一 `TableIf` 在一条语句里能带两套 schema——「同表多版本跨结构变更自连接」**能跑通**而非报错,移除 L17 fail-loud 守卫的限制。现状=`getSchemaCacheValue()` 无引用参数、从共享 `TableIf` 版本盲绑定;L17 已在 `PluginDrivenScanNode.pinMvccSnapshot` 加逐引用 fail-loud 守卫(tuple 列须在版本感知 pinned schema 解析,否则抛)。Nereids 层改动,择机。**残余**:嵌套 field-id-only 重编号守卫看不见(iceberg id 稳定,实际不触发)。 - [ ] **D-SPI-FACET** · 4.2 stringly-typed(transform/bucket enum 化)+ 4.3 连接器专属能力收进 capability-discovered facet;登记为验收架构张力。 - [ ] **D-D2-MICROS** · `ConnectorMvccPartitionView.getNewestUpdateTimeMillis:113` 重命名 `getNewestUpdateMarker()`(去伪单位,零行为变更)。 - [ ] **D-D3-PATHCONTRACT** · `applyRewriteFileScope:204`/`applyTopnLazyMaterialization:226` 引 `ConnectorFilePath` token / debug 全 schema 断言;或登记验收。 @@ -320,3 +321,13 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **张力(Rule 7)**:task list 原推荐 A(登记 Nereids 数为验收偏差,零代码);但 Trino 与「该数还喂 `sql_block_rule` 治理」角度支持 B(A 在隐藏分区下**高报**实际扫描分区数=治理 false-positive over-block)。已用中文向用户讲清背景+两选项+推荐 → **用户选 B**。 + +--- + +**⭐ L16–L19 全部 DONE(2026-07-13,iceberg 杂项收尾)** — 4 recon agent(`wf_0aee6600-244`)复核 HEAD + 独立读码;两条需用户决策(L17/L18)已用中文讲清背景+Trino 对照+选项问用户。commits:`01668779fd9`(L19 三连接器 remove 保留键)· `1c9c99c7767`(L18 accept UNSUPPORTED+守护测试,DV-051)· `76afd6f2e80`(L16 复核 REFUTED+防呆注释)· `3627556db34`(L17 scan-node fail-loud 守卫,3-lens 红队 `wf_f7b69cf7-ec8` 推翻 analysis-time 方案)。 +- **L19** — partition_columns 撞名:**复核扩 scope 到 iceberg+hive+paimon**(同 bug 三份;hudi/mc 免疫)。producer-side `remove`(fe-core 无法区分连接器发 vs 用户透传,铁律禁解析连接器键义)。iceberg 50/50+hive 23/23 UT 绿(各 2 RED-able)、paimon package 编译绿、import 门净;paimon coreOptions 路现有 fake 非 DataTable 不可单测→检视+e2e-gated 登记。 +- **L18** — 未知/v3 类型:**用户签字 accept graceful degradation**(非抛,登记 DV-051);无功能改动+守护测试锁意图。IcebergTypeMappingReadTest 11/11。 +- **L16** — 缓存偏斜:**复核判 REFUTED/benign**(超集写法已不存在+偏斜构造不出:原子 pin+append-only schemas+对称回退);仅加防呆注释锁不变量。IcebergScanPlanProviderTest 88/88。 +- **L17** — 同表多版本 version-blind:**用户签字 fail-loud + TODO 重构**(`D-MVCC-VERSION-SCHEMA`)。红队证 analysis-time 守卫顺序依赖+default-pin/MTMV/@incr 遮蔽→改**逐引用 scan-node 守卫**(tuple 列须在本引用 pinned schema 可解析,有 id 按 id 否则 name)。确定性+完整+无误报。SchemaGuardTest 7/7 + MvccPin 3/3 + MvccExternalTable 59/59。 +- **e2e 全 live-gated**(真集群):L19 非分区表 SET TBLPROPERTIES('partition_columns'=真列)不误判分区;L18 v3 GEOMETRY 列表可加载、该列查报错、其它列可用;L16 无(纯注释);L17 iceberg `t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b` 跨 ALTER →抛清晰错(非 BE 崩)。 +- **决策类仅余 L20**(MC EQ `==`)。设计文档 `designs/FIX-L16/L17/L18/L19-design.md`。 diff --git a/plan-doc/tasks/designs/FIX-L16-design.md b/plan-doc/tasks/designs/FIX-L16-design.md new file mode 100644 index 00000000000000..4f39e02ecc4fae --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L16-design.md @@ -0,0 +1,52 @@ +# FIX-L16 — iceberg snapshot/schema cache skew: VERIFIED BENIGN (no functional fix) + +## Verdict: REFUTED (already-fixed + benign by construction) +Two claims were bundled under L16; both are non-issues at HEAD (`63f3c1965e7`): + +### (1) "superset dict via requestedLowerNames" — already fixed / absent +`IcebergScanPlanProvider.getScanNodeProperties` hasSnapshotPin arm (`:1161-1163`) passes +`Collections.emptyList()` over `pinnedSchema(table, iceHandle)` — the FULL pinned schema, NOT +`requestedLowerNames`. `git log -L` shows it has been `emptyList` since the file's first commit. So the +"superset via requestedLowerNames" shape the task-list line describes does not exist. +(And had it existed, the effect would be a hard THROW, not a benign superset: `IcebergSchemaUtils` +`buildCurrentSchema` does `caseInsensitiveFindField(name)` per requested name and throws when absent; +`requestedLowerNames` is keyed off the LATEST-schema column handles, so a column renamed since the pinned +snapshot would be absent from the OLD pinned schema → throw at plan time. The `emptyList` branch instead +iterates `pinnedSchema.columns()` and emits every pinned top-level column — a guaranteed superset of the +BE scan slots, which are themselves built from the pinned schema.) + +### (2) "snapshot cache vs schema cache skew" — benign by construction +- The pin is **atomic**: `beginQuerySnapshot` (`IcebergConnectorMetadata:1533-1540`) captures + `(snapshotId, latest schemaId)` from ONE table load, so the two ids cannot skew relative to each other + within the TTL. +- The `schemaId` is threaded from ONE `ConnectorMvccSnapshot` via `applySnapshot`→`withSnapshot` into BOTH + `handle.getSchemaId()` (dict, `pinnedSchema`) and `snapshot.getSchemaId()` (slots, `getTableSchema`). +- iceberg `schemas()` is **append-only**, so a fixed `schemaId` present at pin time is present in every later + generation — `pinnedSchema` is generation-stable. `get(schemaId)==null` only for a loaded generation + PREDATING the schema's creation, and THEN `getTableSchema` (same seam, same query) also predates it and + ALSO falls back to `table.schema()` — so dict names and slot names degrade to the SAME latest schema and + still match. No reachable input makes the dict top-level names diverge from the BE scan-slot names; a + self-join at two snapshots is per-handle isolated. + +## The load-bearing invariant (why the guard comment) +`dict top-level names == BE StructNode scan-slot names` — a divergence makes BE's unconditional +`children.at(name)` `std::out_of_range`-SIGABRT the whole BE on a schema-evolved time-travel read. That +invariant holds ONLY because `IcebergScanPlanProvider.pinnedSchema` (dict) and +`IcebergConnectorMetadata.getTableSchema` (slots) use the **SAME schemaId selector + the SAME silent +fallback to `table.schema()`**. Hardening ONE of them (e.g. making `pinnedSchema` throw-loud on a missing +schemaId) would break the symmetry and CREATE the crash it looks like it prevents. + +## Change: guard comment only (no behavior change) +Add a cross-referencing note on both `pinnedSchema` and `getTableSchema`'s fallback stating the two +selectors/fallbacks MUST stay identical, so a future "defensive" edit does not silently introduce the +asymmetry. No code/behavior change; no new test (nothing RED-able — a divergence test cannot be +constructed at HEAD, itself evidence the hazard is benign). + +## Relation to L17 +L16's residual and L17 share the root "schema binding not version-tracked", but on DIFFERENT axes: L16 = +snapshot-cache-vs-schema-cache atomicity across time (benign, above); L17 = per-reference version blindness +within one statement (the real, if narrow, fe-core gap). The Trino-style structural elimination (serialize +the pinned schema onto the handle) belongs to L17's track, not here. + +## Iron rule +Clean — comments only, inside fe-connector-iceberg. No fe-core touch. diff --git a/plan-doc/tasks/designs/FIX-L17-design.md b/plan-doc/tasks/designs/FIX-L17-design.md new file mode 100644 index 00000000000000..b25f105538bdf5 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L17-design.md @@ -0,0 +1,93 @@ +# FIX-L17 — iceberg same-table multi-version version-blind schema binding (fail-loud guard) + +## Problem (HEAD-verified) +`PluginDrivenMvccExternalTable.getSchemaCacheValue()` (`:485`) binds the table's schema version-BLIND via +`MvccUtil.getSnapshotFromContext(this)` (`StatementContext.getSnapshot(TableIf):1015` — returns the default +pin if present, else the lone pin, else EMPTY when 2+ same-table pins exist with no default). Binding is +per-reference (`BindRelation.getLogicalPlan:739` pins the reference's snapshot, then binds its columns via +`getFullSchema → getSchemaCacheValue`; `ExternalTable.getFullSchema:176` re-evaluates fresh each call — no +freeze). Meanwhile the DATA path is version-AWARE (`PluginDrivenScanNode.pinMvccSnapshot:869` uses the +3-arg `getSnapshotFromContext(table, tableSnapshot, scanParams)`). + +For a self-join `t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b` across a schema change, ref `b` +binds while 2 pins exist → EMPTY → LATEST schema, but its scan reads v2 → **skew**: the FE tuple carries +latest columns/field-ids while BE reads v2 files → field-id/name mismatch (crash / wrong NULLs). Narrow, +low severity, dominantly fail-loud. **Newly possible in this branch** (pre-P6 the table-only MVCC key +collapsed a self-join to one self-consistent snapshot; this branch made the data path per-reference +version-aware while schema binding stayed version-blind). + +## Why NOT the analysis-time (getSchemaCacheValue) guard — red-team `wf_f7b69cf7-ec8` +An analysis-time guard (detect 2+ divergent pins in `getSchemaCacheValue`, throw) is **order-dependent and +holey** — all 3 lenses flagged it: +- **Masking (BLOCKER):** a plain/latest reference (`t@v1 JOIN t`) creates a default("") pin, so + `getSnapshot(TableIf)` returns it (not EMPTY) and the guard is never reached → v1 ref silently skews. + `t a JOIN t@v1 b JOIN t@v2 c` throws or silently-skews **depending on relation binding order** — the same + query, non-deterministic. +- **MTMV refresh:** `MTMVTask.beforeMTMVRefresh` pre-seeds a default("") pin → the guard is never reached + during MV refresh → silent skew. +- **@incr mix:** `t@v1 JOIN t@incr` (null pinnedSchema) → the design's "one null pin → latest, no throw" + rule is itself a silent-skew hole. + +## Fix — deterministic per-reference guard at the scan node (Option A, fail-loud) +Guard where each reference's OWN version-aware pinned schema is already resolved: `pinMvccSnapshot()` +(`PluginDrivenScanNode:863`). After resolving this scan's version-aware `snapshot`, if it carries a non-null +`getPinnedSchema()` (an explicit schema-at-snapshot pin — iceberg/paimon FOR VERSION/TIME/tag/branch; null +for latest/`@incr`/sys/hive → skipped), verify every **materialized, real tuple slot column resolves in that +pinned schema**: +- `uniqueId >= 0` (iceberg field-id present) → the field-id must be in the pinned schema's field-ids (BE + matches iceberg by field-id → rename with a stable id is fine, renumber/added-column is caught). +- `uniqueId < 0` (paimon has no top-level field-id) → the column NAME must be in the pinned schema (BE + matches by name → rename is caught). +If any bound slot is unresolved → the tuple was bound at a different version than this reference scans → +throw `UserException` (checked; `pinMvccSnapshot` already `throws UserException`) with a clear message. + +This is **deterministic** (runs at scan time with all pins loaded, checks THIS reference against its OWN +tuple — order-independent), **complete** (catches self-join, latest-masked, `@incr`-mixed, AND MTMV-refresh — +every path builds scan nodes), and has **no false positives**: for `t@old JOIN t@latestSnapId` where the +explicit version IS latest, each reference's tuple matches its own version-aware schema → no throw (the +analysis-time guard would have wrongly thrown this working query). Common queries are untouched — a plain / +single-version / latest reference has a null pinnedSchema (or its tuple matches) → the guard is a no-op. + +### Structural machinery (connector-agnostic) +- Static package-private `assertBoundColumnsResolveInPinnedSchema(List boundColumns, + SchemaCacheValue pinnedSchema, String tableName)` — directly unit-testable (takes plain `Column`s + a + `SchemaCacheValue`, no scan-node construction). Field-id-when-present-else-name matching (no source-name + branch; the field-id-vs-name choice keys on whether `uniqueId` is populated, a generic column property). +- `pinMvccSnapshot` extracts the projected bound columns (`desc.getSlots()` where `getColumn() != null`, + mirroring the existing `:1749` loop) and calls the helper with this scan's `getPinnedSchema()`. + +## Iron rule +Clean. `PluginDrivenScanNode` is the generic connector-agnostic node; the guard operates on generic +`SchemaCacheValue`/`Column`/`SlotDescriptor` types. No `instanceof HMSExternal*`, no source-name branch, no +property parsing. Applies uniformly to every `PluginDrivenMvccExternalTable` connector. + +## Residual / accepted (registered) +- **Nested field-id-only renumber** (a nested struct field's id changes with unchanged nested name+type) is + invisible to the top-level field-id/name check (fe-core `StructField` carries no field id). Effectively + never occurs in iceberg (ids are stable); not detected — accept, note under the TODO. +- **Same-version-different-selector** collapses to one map entry (`versionKeyOf`) → not multi-version → no + guard. Correct. +- The guard THROWS (does not repair) the `t@v1 JOIN t@v2` same-schema-S-but-S≠latest case (silently broken + today) — throwing > silent skew; the repair is the TODO. + +## TODO (design debt — registered as D-MVCC-VERSION-SCHEMA in the task-list) +Make schema binding per-reference version-aware end-to-end (Trino model: a version-scoped +`ConnectorTableHandle` / per-reference column-handle resolution) so two references of one `TableIf` carry two +schemas and the self-join across a schema change WORKS instead of throwing. Removes the guard's restriction. +Nereids-wide (schema is bound off the single shared `TableIf` via the no-arg `getSchemaCacheValue()`). + +## Tests (fe-core, Mockito) +Directly on the static helper (RED-able, no scan-node construction): +- RED: bound columns include one whose field-id is absent from the pinned schema (renumber/added column) → + throws. And a name-only (uniqueId=-1) column absent from the pinned schema (paimon rename) → throws. +- GREEN: all bound columns resolve by field-id (rename keeps id → resolves; subset projection ok) → no throw. +- GREEN: all bound columns resolve by name when uniqueId=-1 → no throw. +- GREEN: null pinnedSchema (latest/@incr) → no throw. +Plus the existing `PluginDrivenScanNodeMvccPinTest` suite stays green (the guard is off the pinned-snapshot +path only when schemas match). + +## Blast radius +`pinMvccSnapshot` runs at 4 scan-side sites (idempotent read-only check; throws consistently at the first). +Only fires when `getPinnedSchema() != null` (explicit schema-at-snapshot pin) AND a bound column is +unresolved — i.e. genuine version/schema skew. Plain / latest / single-version / hive / sys-table / count +scans are no-ops. e2e live-gated (iceberg self-join FOR VERSION AS OF v1 vs v2 across an ALTER). diff --git a/plan-doc/tasks/designs/FIX-L18-design.md b/plan-doc/tasks/designs/FIX-L18-design.md new file mode 100644 index 00000000000000..0545832f00fb20 --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L18-design.md @@ -0,0 +1,42 @@ +# FIX-L18 — iceberg unknown/v3 type → UNSUPPORTED (accept, not throw) + +## Problem (HEAD-verified) +`IcebergTypeMapping` (fe-connector-iceberg) READ direction has two silent `default → UNSUPPORTED` arms: +- `fromIcebergType` nested switch (`:90-91`) — reached by non-primitive `VARIANT` and any future non-primitive typeId. +- `fromPrimitive` switch (`:142-143`) — reached by the v3 primitives `TIMESTAMP_NANO` / `GEOMETRY` / `GEOGRAPHY` / `UNKNOWN` (all `Type$PrimitiveType` in iceberg 1.10.1), plus the explicit `case TIME → UNSUPPORTED` (`:140-141`). + +Legacy fe-core (`IcebergUtils.icebergTypeToDorisType` / `icebergPrimitiveTypeToDorisType`) mapped `TIME`/`VARIANT` +to `Type.UNSUPPORTED` explicitly but **threw** `IllegalArgumentException("Cannot transform unknown type: …")` +on the `default` — so today's connector is strictly **more lenient**: an iceberg table with a v3 column +LOADS with that column present-but-unqueryable, whereas legacy failed the whole table load. The genuine +divergence set is the v3 primitives (`TIMESTAMP_NANO`/`GEOMETRY`/`GEOGRAPHY`/`UNKNOWN`); `TIME`+`VARIANT` +are parity (both explicit/effective UNSUPPORTED). Trino also fails loud (`TypeConverter` → `TrinoException(NOT_SUPPORTED)`). + +## Decision (user, 2026-07-13) +**Accept the looser behavior — map every unrepresentable column uniformly to UNSUPPORTED, do NOT throw.** +Rationale (user): one exotic column should not make a wide table unloadable; other columns stay usable. +Registered as **DV-051**. This is Option B of the recon; Option A (restore legacy throw) was declined. + +## Change +- **No functional change** — the two `default` arms already return `UNSUPPORTED`; that is now the sanctioned + behavior. Added clarifying comments on both arms documenting the intentional divergence + DV-051, and noting + the write direction `toIcebergPrimitive:199` still throws (CREATE TABLE must not accept a non-round-trippable type). +- **Guard test** `IcebergTypeMappingReadTest.unknownAndV3TypesDegradeToUnsupportedByDesign` pins the choice: + asserts `TIMESTAMP_NANO`/`GEOMETRY`/`GEOGRAPHY`/`UNKNOWN`/`VARIANT` → `UNSUPPORTED` (flag-independent). A + future mutation making either arm throw turns this RED — surfacing that the accepted deviation was reverted + (Rule 9: the test encodes WHY — the deliberate graceful-degradation decision). +- **DV-051** registered in `plan-doc/deviations-log.md`. + +## Iron rule +Clean — entirely in fe-connector-iceberg, switching on iceberg's own `Type.TypeID`. No fe-core touch, no +source-name branching, no property parsing. `DorisConnectorException` (the module idiom) unused here since we +do NOT throw. + +## Blast radius +Zero behavior change → no regression. No existing regression fixture uses GEOMETRY/GEOGRAPHY/TIMESTAMP_NANO +(grep clean), consistent with "table loads, column unqueryable". Sole caller of the mapping is +`IcebergConnectorMetadata.parseSchema` on table load/refresh. + +## e2e (live-gated) +Register an iceberg v3 table with a GEOMETRY (or TIMESTAMP_NANO) column → table loads, `DESCRIBE` shows the +column as UNSUPPORTED, `SELECT other_col` works, `SELECT geom_col` errors at execution (present-but-unqueryable). diff --git a/plan-doc/tasks/designs/FIX-L19-design.md b/plan-doc/tasks/designs/FIX-L19-design.md new file mode 100644 index 00000000000000..05649ad3972beb --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L19-design.md @@ -0,0 +1,72 @@ +# FIX-L19 — `partition_columns` reserved magic-key collision with source properties + +## Problem (HEAD-verified) +The SPI carries partition info from connector → fe-core through a stringly-typed reserved key +`"partition_columns"` **inside the same properties map** that also carries the source table's own +pass-through properties. fe-core's sole partition-derivation source is +`PluginDrivenExternalTable.toSchemaCacheValue:513` +(`tableSchema.getProperties().get("partition_columns")`) — a non-empty value ⇒ the table is modeled +as partitioned (shared by the latest path and the MVCC AS-OF path `PluginDrivenMvccExternalTable:387`). + +Three producers do a source-property pass-through **before** conditionally stamping the reserved key, +and never remove a pre-existing one: +- iceberg `IcebergConnectorMetadata:411-412` — `tableProps.putAll(table.properties())`, then `put("partition_columns",…)` **only** when `!spec.isUnpartitioned()` (`:430-447`). +- hive `HiveConnectorMetadata:449-450` — `new HashMap<>(tableInfo.getParameters())`, then `put(PARTITION_COLUMNS_PROPERTY,…)` **only** when `!partitionKeys.isEmpty()` (`:455-458`). +- paimon `PaimonConnectorMetadata:304` — `schemaProps.putAll(coreOptions().toMap())`, then `put("partition_columns"/"primary_keys",…)` when the respective key list is non-empty (`:309-322`). + +(hudi/maxcompute build a fresh map — no pass-through, unaffected.) + +**Failure:** a **non-partitioned** table whose source properties literally contain a key named +`partition_columns` (e.g. iceberg `ALTER TABLE … SET TBLPROPERTIES('partition_columns'='id')`) skips the +stamp branch, so the user value survives the `putAll` and reaches fe-core, which treats it as the +partition CSV → the non-partitioned table is **misdetected as partitioned** (wrong pruning / row-count / +`EXPLAIN partition=N/M` / MTMV / analyze). Partitioned tables are safe (the connector value overwrites at +the stamp). Reachability is narrow: the source key must be exactly the bare reserved name AND its CSV must +match real column names (`toSchemaCacheValue:526` `if (col != null)` filters non-matches → benign). + +`primary_keys` (paimon `:322`) is behaviorally benign in fe-core — it is only **stripped** for SHOW CREATE +(`PluginDrivenExternalTable:731`), never functionally consumed — so a collision there just hides it from +rendering (already the case). We strip it too for defensive symmetry with paimon's own stamp. + +## Fix (producer-side; the only workable locus) +fe-core cannot defend: once merged, connector-emitted and user-passthrough `partition_columns` are +byte-identical strings in one map, and distinguishing them would require fe-core to parse connector key +semantics (iron-rule violation). So each producer that pass-through-merges must ensure the reserved key +reflects **only its own determination** — remove it right after the `putAll`, before the conditional stamp: +- iceberg after `:412` — `tableProps.remove("partition_columns");` +- hive after `:450` — `tableProperties.remove(PARTITION_COLUMNS_PROPERTY);` +- paimon after `:304` — `schemaProps.remove("partition_columns"); schemaProps.remove("primary_keys");` + +No fe-core change; connectors already emit these literals; no source-name branching. No user-facing +regression: `getTableProperties():731` already unconditionally strips both keys from SHOW CREATE PROPERTIES, +so a user's literally-named property is invisible today regardless — the only behavior that changes is the +removal of the incorrect partition misdetection. + +## Iron rule +Clean. Entirely in connector modules, which already own/emit these reserved key names. No new fe-core +import, no `instanceof HMSExternal*`, no property parsing pushed into fe-core. + +## Tests (RED-able, connector modules, recording fakes) +- iceberg: unpartitioned `db1.t1` with `table.properties()={"partition_columns":"id"}` (a real column) → + emitted `ConnectorTableSchema.getProperties()` must NOT contain `partition_columns` (RED: leaks via `:412`). + Guard: a genuinely partitioned (identity `name`) table with a colliding `{"partition_columns":"id"}` + still emits the spec-derived `partition_columns=name` (fix doesn't over-strip; connector value wins). +- hive: `HiveTableInfo` with NO partition keys but parameters `{"partition_columns":"c1"}` → emitted props + lack `partition_columns`. Guard: partitioned table still emits its key list. +- paimon: **no dedicated unit test** — the leak vector is `((DataTable) table).coreOptions().toMap()`, but + the module's `FakePaimonTable implements Table` (NOT `DataTable`), so the `putAll`+`remove` block is + skipped for the fake and a real `DataTable` fake (many abstract methods) is disproportionate for a 2-line + defensive fix identical in shape to the iceberg/hive fixes (both RED-tested above). The paimon path is + verified by inspection + covered by the same pattern; a real paimon table `WITH('partition_columns'=…)` + collision is **E2E/live-gated**. Registered, not silently skipped (Rule 12). + +## Blast radius +Producer-local (isolated `remove()` calls each immediately before an existing conditional put). Normal +tables (no bare `partition_columns`/`primary_keys` source property) are byte-identical before/after. Per +connector by necessity — no shared producer base for buildTableSchema. + +## Note (out of scope — design debt) +The root smell is bare-named reserved keys (`partition_columns`/`primary_keys`) vs the namespaced ones +(`show.*`, `connector.*`) that source properties cannot realistically hit. Namespacing them (constant in +`fe-connector-api ConnectorTableSchema`) would be collision-resistant by construction but touches all +producers + the consumer + both strip sites — larger than this minimal defensive fix. Tracked as D-MAGICKEY. From 2c58d8342c174e59a52f5d8b0357c4a257ee8e20 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 13:24:50 +0800 Subject: [PATCH 331/334] [fix](catalog) namespace reserved connector control keys under __internal. (supersedes silent strip) The SPI passes structural info from connector -> fe-core through reserved control keys inside the same property map that also carries the source table's user pass-through properties. Two of the seven keys were bare, un-namespaced literals scattered across connectors (partition_columns, primary_keys), so a source table whose own properties literally contained such a key collided -- a non-partitioned table could be misdetected as partitioned. The prior fix silently remove()'d the colliding key in three connectors, which discards user data with no signal and gives a future developer no feedback when adding a new reserved keyword. Instead, namespace EVERY reserved control key under a distinctive __internal. prefix (the same mechanism the already-namespaced show.* / connector.* keys use to avoid collision) and centralize them as constants in ConnectorTableSchema: __internal.partition_columns / __internal.primary_keys / __internal.show.location / __internal.show.partition-clause / __internal.show.sort-clause / __internal.connector.per-table-capabilities / __internal.connector.distribution-columns Collision is now impossible by construction: a user's own bare partition_columns property flows through as a normal user property (preserved, rendered in SHOW CREATE), never mistaken for the control key -- no silent strip, no failure. No validation is needed (consistent with the five keys that already relied on their namespace). All reserved keys are FE-only (none is emitted to BE via getScanNodeProperties; BE gets partition columns via the separate path_partition_keys), not GSON/editlog-persisted (schema cache rebuilt on demand), and stripped from SHOW CREATE -- so the rename has zero BE / serialization / golden impact. - ConnectorTableSchema: INTERNAL_KEY_PREFIX, the two new constants, RESERVED_CONTROL_KEYS set. - Producers (iceberg/hive/hudi/maxcompute/paimon) reference the constants; the L19 per-connector remove() strips are deleted. - fe-core: toSchemaCacheValue reads PARTITION_COLUMNS_KEY; getTableProperties strips RESERVED_CONTROL_KEYS (a future reserved key is stripped automatically). - Tests: connectors assert the constant key; new tests assert a bare user partition_columns coexists / flows through (no collision). iceberg 50, hive 23, paimon 12+19+43, fe-core 8+36 green. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../connector/api/ConnectorTableSchema.java | 48 ++++++++++++++-- .../connector/hive/HiveConnectorMetadata.java | 12 +--- .../hive/HiveConnectorMetadataSchemaTest.java | 42 ++++++++------ .../connector/hudi/HudiConnectorMetadata.java | 2 +- .../iceberg/IcebergConnectorMetadata.java | 9 +-- .../iceberg/IcebergConnectorMetadataTest.java | 56 ++++++++++--------- .../MaxComputeConnectorMetadata.java | 2 +- .../paimon/PaimonConnectorMetadata.java | 12 +--- .../PaimonConnectorMetadataMvccTest.java | 4 +- .../paimon/PaimonConnectorMetadataTest.java | 2 +- .../PaimonTableHandleScanOptionsTest.java | 4 +- .../datasource/PluginDrivenExternalTable.java | 28 ++++------ ...luginDrivenExternalTablePartitionTest.java | 22 ++++---- .../PluginDrivenExternalTableTest.java | 49 +++++++++------- 14 files changed, 160 insertions(+), 132 deletions(-) diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java index fe535ab0538a70..8a83ad1d95b488 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorTableSchema.java @@ -17,10 +17,13 @@ package org.apache.doris.connector.api; +import java.util.Arrays; import java.util.Collections; +import java.util.HashSet; import java.util.List; import java.util.Map; import java.util.Objects; +import java.util.Set; /** * Describes the schema of a connector table, including its columns @@ -28,6 +31,16 @@ */ public final class ConnectorTableSchema { + /** + * Common prefix for every FE-internal reserved control key below. The connector uses these keys to pass + * structural info to fe-core (partition columns / primary keys / SHOW CREATE render hints / per-table + * capabilities / distribution columns) INSIDE the same property map that also carries the source table's + * user-facing pass-through properties. The {@code __internal.} prefix keeps them out of the namespace a + * real user table property would ever use, so a source property can never be mistaken for a control key + * (and vice versa). These keys are FE-only — none is forwarded to BE. + */ + public static final String INTERNAL_KEY_PREFIX = "__internal."; + /** * Reserved property key carrying the table location string for SHOW CREATE TABLE rendering. * Connectors emit it here (rather than as a user-facing property) so the FE renders it as the @@ -35,7 +48,7 @@ public final class ConnectorTableSchema { * connector's own user-facing location property (e.g. paimon's SDK {@code path} option, which * legitimately stays in PROPERTIES). */ - public static final String SHOW_LOCATION_KEY = "show.location"; + public static final String SHOW_LOCATION_KEY = INTERNAL_KEY_PREFIX + "show.location"; /** * Reserved property key carrying the fully-rendered {@code PARTITION BY ...} clause (Doris SQL, @@ -43,13 +56,28 @@ public final class ConnectorTableSchema { * The connector pre-renders it (the transform-aware logic is connector-specific); the FE appends * it verbatim and strips it from PROPERTIES. */ - public static final String SHOW_PARTITION_CLAUSE_KEY = "show.partition-clause"; + public static final String SHOW_PARTITION_CLAUSE_KEY = INTERNAL_KEY_PREFIX + "show.partition-clause"; /** * Reserved property key carrying the fully-rendered {@code ORDER BY (...)} clause for SHOW CREATE * TABLE. The connector pre-renders it; the FE appends it verbatim and strips it from PROPERTIES. */ - public static final String SHOW_SORT_CLAUSE_KEY = "show.sort-clause"; + public static final String SHOW_SORT_CLAUSE_KEY = INTERNAL_KEY_PREFIX + "show.sort-clause"; + + /** + * Reserved property key carrying a CSV of the RAW remote partition-column names (declaration order), so + * the generic fe-core consumer ({@code PluginDrivenExternalTable.toSchemaCacheValue}) can model the table + * as partitioned. Emitted by every connector that has partition columns (hive/hudi/iceberg/paimon/ + * maxcompute). FE-only (BE receives partition columns via the separate {@code path_partition_keys} scan + * property); stripped from the user-facing SHOW CREATE TABLE PROPERTIES(...) block. + */ + public static final String PARTITION_COLUMNS_KEY = INTERNAL_KEY_PREFIX + "partition_columns"; + + /** + * Reserved property key carrying a CSV of the table's primary-key column names (paimon). FE-only; + * stripped from the user-facing SHOW CREATE TABLE PROPERTIES(...) block. + */ + public static final String PRIMARY_KEYS_KEY = INTERNAL_KEY_PREFIX + "primary_keys"; /** * Reserved property key carrying a CSV of {@link ConnectorCapability#name()} values that THIS specific @@ -67,7 +95,7 @@ public final class ConnectorTableSchema { * in fe-core. Single-format connectors never emit it and are unaffected. Stripped from the user-facing * SHOW CREATE TABLE PROPERTIES(...) block.

    */ - public static final String PER_TABLE_CAPABILITIES_KEY = "connector.per-table-capabilities"; + public static final String PER_TABLE_CAPABILITIES_KEY = INTERNAL_KEY_PREFIX + "connector.per-table-capabilities"; /** * Reserved property key carrying a CSV of the table's distribution (bucketing) column names, already @@ -80,7 +108,17 @@ public final class ConnectorTableSchema { * sampled analyze). A non-bucketed table omits it and connectors with no bucketing concept never emit it. * Stripped from the user-facing SHOW CREATE TABLE PROPERTIES(...) block.

    */ - public static final String DISTRIBUTION_COLUMNS_KEY = "connector.distribution-columns"; + public static final String DISTRIBUTION_COLUMNS_KEY = INTERNAL_KEY_PREFIX + "connector.distribution-columns"; + + /** + * The full set of FE-internal reserved control keys. fe-core strips every member from the user-facing + * SHOW CREATE TABLE PROPERTIES(...) block; centralizing the set here means adding a new reserved key is a + * single edit that the strip picks up automatically. All members share {@link #INTERNAL_KEY_PREFIX}. + */ + public static final Set RESERVED_CONTROL_KEYS = Collections.unmodifiableSet(new HashSet<>( + Arrays.asList( + PARTITION_COLUMNS_KEY, PRIMARY_KEYS_KEY, SHOW_LOCATION_KEY, SHOW_PARTITION_CLAUSE_KEY, + SHOW_SORT_CLAUSE_KEY, PER_TABLE_CAPABILITIES_KEY, DISTRIBUTION_COLUMNS_KEY))); private final String tableName; private final List columns; diff --git a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java index e4f4446d721572..9785a404e27b60 100644 --- a/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveConnectorMetadata.java @@ -117,8 +117,9 @@ public class HiveConnectorMetadata implements ConnectorMetadata { // FE-internal schema-control property key: a CSV of the RAW remote partition-column names. The generic // fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) reads it to derive which of the emitted // columns are partition columns; it is the same key the paimon/iceberg/maxcompute connectors emit and is - // stripped from the user-facing SHOW CREATE properties by fe-core. - private static final String PARTITION_COLUMNS_PROPERTY = "partition_columns"; + // stripped from the user-facing SHOW CREATE properties by fe-core. The central reserved-key definition + // (namespaced under __internal.) lives in ConnectorTableSchema. + private static final String PARTITION_COLUMNS_PROPERTY = ConnectorTableSchema.PARTITION_COLUMNS_KEY; // Connector-side spelling of fe-type ScalarType.MAX_VARCHAR_LENGTH (the connector must not import fe-type); // a hive `string` partition column is widened to varchar(65533) for legacy parity. Paimon hardcodes the @@ -448,13 +449,6 @@ public ConnectorTableSchema getTableSchema( // mutating the shared tableInfo map. Map tableProperties = new HashMap<>( tableInfo.getParameters() != null ? tableInfo.getParameters() : Collections.emptyMap()); - // Strip the FE-internal reserved partition marker that may have leaked in from the copied HMS - // parameters above (a user TBLPROPERTY literally named "partition_columns"): the generic fe-core - // consumer (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty "partition_columns" - // as the partition-column CSV, so a leaked user value would make a NON-partitioned table be - // misdetected as partitioned. Only this connector's own value (re-stamped below when the table - // has partition keys) may reach fe-core. - tableProperties.remove(PARTITION_COLUMNS_PROPERTY); // Mark which emitted columns are partition columns for the generic fe-core consumer. Without this // property every partitioned hive/hudi table is read as unpartitioned (wrong pruning/row count, MTMV // breakage). The value is a CSV of the RAW partition-key names in declaration order; hive partition-key diff --git a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java index 5d3a7b9bf8fefc..23aabc8397d54a 100644 --- a/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java +++ b/fe/fe-connector/fe-connector-hive/src/test/java/org/apache/doris/connector/hive/HiveConnectorMetadataSchemaTest.java @@ -135,7 +135,7 @@ private static boolean hasCapability(ConnectorTableSchema schema, ConnectorCapab public void testPartitionColumnsPropertyEmittedWithRawNamesInOrder() { ConnectorTableSchema schema = schemaOf(partitionedTable().build()); Assertions.assertEquals("year,region", - schema.getProperties().get("partition_columns")); + schema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY)); } @Test @@ -219,34 +219,42 @@ public void testUnpartitionedTableEmitsNoPartitionColumnsProperty() { .parameters(Collections.emptyMap()) .build(); ConnectorTableSchema schema = schemaOf(tableInfo); - Assertions.assertFalse(schema.getProperties().containsKey("partition_columns")); + Assertions.assertFalse(schema.getProperties().containsKey(ConnectorTableSchema.PARTITION_COLUMNS_KEY)); } @Test - public void testUnpartitionedTableStripsUserPartitionColumnsParameter() { - // A user TBLPROPERTY literally named "partition_columns" on a NON-partitioned hive table must not leak - // into the emitted schema: the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) - // treats a non-empty "partition_columns" as the partition-column CSV, so a leaked user value whose CSV - // matches a real column would make this unpartitioned table be misdetected as partitioned. - // MUTATION: dropping tableProperties.remove(PARTITION_COLUMNS_PROPERTY) -> the parameter leaks -> red. + public void testUserPartitionColumnsParameterCannotCollideWithReservedKey() { + // A user TBLPROPERTY literally named "partition_columns" (bare) can NEVER be mistaken for the reserved + // partition marker, which is namespaced under __internal. (ConnectorTableSchema.PARTITION_COLUMNS_KEY). + // On a NON-partitioned hive table: the connector emits NO reserved key -> fe-core sees no partition + // marker -> the table stays unpartitioned; and the user's bare property flows through unchanged (no + // silent strip). MUTATION: reverting the reserved key to the bare "partition_columns" -> the user + // value would be read as the partition CSV -> the assertNull below fails -> red. Map params = new HashMap<>(); - params.put("partition_columns", "id"); // a real column — fe-core would match it + params.put("partition_columns", "id"); // a real column name, as a plain user property HmsTableInfo tableInfo = unpartitionedTable(PARQUET_INPUT_FORMAT).parameters(params).build(); ConnectorTableSchema schema = schemaOf(tableInfo); - Assertions.assertFalse(schema.getProperties().containsKey("partition_columns"), - "an unpartitioned hive table must not carry partition_columns even if a parameter is named so"); + Assertions.assertNull(schema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "an unpartitioned hive table must emit no reserved partition marker"); + Assertions.assertEquals("id", schema.getProperties().get("partition_columns"), + "the user's bare partition_columns property must flow through unchanged (no collision, no strip)"); } @Test - public void testPartitionedTablePartitionColumnsReflectKeysNotCollidingParameter() { - // Guard the fix does not over-strip: a genuinely partitioned table whose parameters ALSO carry a - // colliding "partition_columns"=id must emit the CONNECTOR's own partition-key CSV (year,region), - // never the user's spoofed value. MUTATION: stripping unconditionally after the stamp -> red. + public void testPartitionedTableReservedKeyCoexistsWithCollidingUserParameter() { + // A genuinely partitioned table whose parameters ALSO carry a colliding bare "partition_columns"=id: + // the reserved key (__internal.partition_columns) carries the CONNECTOR's own partition-key CSV + // (year,region), and the user's bare property coexists independently — the two live in different + // namespaces and never overwrite each other. MUTATION: bare reserved key -> the user value would + // overwrite (or be overwritten) -> one of the assertions fails -> red. Map params = new HashMap<>(); params.put("partition_columns", "id"); ConnectorTableSchema schema = schemaOf(partitionedTable().parameters(params).build()); - Assertions.assertEquals("year,region", schema.getProperties().get("partition_columns"), - "partition_columns must be the connector's partition-key CSV, not the colliding parameter"); + Assertions.assertEquals("year,region", + schema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "the reserved key must carry the connector's partition-key CSV"); + Assertions.assertEquals("id", schema.getProperties().get("partition_columns"), + "the user's bare property coexists, untouched"); } @Test diff --git a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java index 2fcd1b5ff5fbc9..f79ca793fdf3ff 100644 --- a/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-hudi/src/main/java/org/apache/doris/connector/hudi/HudiConnectorMetadata.java @@ -95,7 +95,7 @@ public class HudiConnectorMetadata implements ConnectorMetadata { // names, in declaration order). Mirrors HiveConnectorMetadata.PARTITION_COLUMNS_PROPERTY; fe-core derives the // partition-column set SOLELY from this key (PluginDrivenExternalTable.toSchemaCacheValue). Without it every // partitioned Hudi table is read as UNPARTITIONED -> wrong pruning/row-count and MTMV "not partition table". - private static final String PARTITION_COLUMNS_PROPERTY = "partition_columns"; + private static final String PARTITION_COLUMNS_PROPERTY = ConnectorTableSchema.PARTITION_COLUMNS_KEY; // Hive-canonical partition text for a DATETIME/TIMESTAMP literal: space separator, full seconds. See // hiveDateTimeString / extractLiteralValue (H2: String.valueOf(LocalDateTime) would yield ISO "…T…" and drop diff --git a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java index 50ffffae7b8105..af1d1f4a810fd5 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-iceberg/src/main/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadata.java @@ -415,13 +415,6 @@ private ConnectorTableSchema buildTableSchema(String tableName, Table table, Sch Map tableProps = new HashMap<>(); tableProps.putAll(table.properties()); - // Strip the FE-internal reserved partition marker that may have leaked in from the source - // table.properties() above (e.g. a user ALTER TABLE ... SET TBLPROPERTIES('partition_columns'=...)): - // the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty - // "partition_columns" as the partition-column CSV, so a leaked user value would make an - // UNPARTITIONED table be misdetected as partitioned. Only THIS connector's own spec-derived value - // (re-stamped below for a genuinely partitioned table) may reach fe-core. - tableProps.remove("partition_columns"); // SHOW CREATE TABLE render hints under neutral reserved keys (fe-core strips them from the // rendered PROPERTIES and emits them as LOCATION / PARTITION BY / ORDER BY). They replace the // previously-injected location / iceberg.format-version / iceberg.partition-spec keys: those were @@ -455,7 +448,7 @@ private ConnectorTableSchema buildTableSchema(String tableName, Table table, Sch } } if (!partitionColumns.isEmpty()) { - tableProps.put("partition_columns", String.join(",", partitionColumns)); + tableProps.put(ConnectorTableSchema.PARTITION_COLUMNS_KEY, String.join(",", partitionColumns)); } } diff --git a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java index b9693264788f1a..02e1f8982b5f40 100644 --- a/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java +++ b/fe/fe-connector/fe-connector-iceberg/src/test/java/org/apache/doris/connector/iceberg/IcebergConnectorMetadataTest.java @@ -656,42 +656,46 @@ public void getTableSchemaCopiesTablePropertiesAndLocation() { } @Test - public void getTableSchemaStripsUserPartitionColumnsOnUnpartitionedTable() { - // A user TBLPROPERTY literally named "partition_columns" on an UNPARTITIONED iceberg table (e.g. set - // via ALTER TABLE ... SET TBLPROPERTIES('partition_columns'='id')) must NOT survive into the emitted - // schema properties: the generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) - // treats a non-empty "partition_columns" as the partition-column CSV, so a leaked user value whose - // CSV matches a real column would make this non-partitioned table be misdetected as partitioned - // (wrong pruning / row-count / EXPLAIN partition=N/M). buildTableSchema stamps the key only for a - // genuinely partitioned table, so for an unpartitioned table it must be absent. - // MUTATION: dropping tableProps.remove("partition_columns") -> the user value leaks via putAll -> red. + public void getTableSchemaUserPartitionColumnsCannotCollideWithReservedKey() { + // A user TBLPROPERTY literally named "partition_columns" (bare, e.g. ALTER TABLE ... SET + // TBLPROPERTIES('partition_columns'='id')) can NEVER be mistaken for the reserved partition marker, + // which is namespaced under __internal. (ConnectorTableSchema.PARTITION_COLUMNS_KEY). On an + // UNPARTITIONED table the connector emits NO reserved key -> fe-core sees no partition marker -> + // the table stays unpartitioned; and the user's bare property flows through unchanged (no silent + // strip). MUTATION: reverting the reserved key to bare "partition_columns" -> the user value would be + // read as the partition CSV -> the assertNull fails -> red. RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); Map userProps = new HashMap<>(); - userProps.put("partition_columns", "id"); // a real column name — fe-core would match it + userProps.put("partition_columns", "id"); // a real column name, as a plain user property ops.table = new FakeIcebergTable( "t1", idNameSchema(), PartitionSpec.unpartitioned(), "s3://bucket/db1/t1", userProps); ConnectorTableSchema schema = metadataWith(ops).getTableSchema(null, new IcebergTableHandle("db1", "t1")); - Assertions.assertFalse(schema.getProperties().containsKey("partition_columns"), - "an unpartitioned iceberg table must not carry partition_columns even if the source props do"); + Assertions.assertNull(schema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "an unpartitioned iceberg table must emit no reserved partition marker"); + Assertions.assertEquals("id", schema.getProperties().get("partition_columns"), + "the user's bare partition_columns property must flow through unchanged (no collision, no strip)"); } @Test - public void getTableSchemaPartitionColumnsReflectSpecNotCollidingUserProperty() { - // Guard the fix does not over-strip: a genuinely partitioned table (identity on `name`) whose source - // properties ALSO carry a colliding "partition_columns"=id must emit the CONNECTOR's spec-derived - // value (`name`), never the user's spoofed value — the remove()+re-stamp guarantees the connector wins. - // MUTATION: removing the re-stamp, or stripping unconditionally after the stamp -> red. + public void getTableSchemaReservedKeyCoexistsWithCollidingUserProperty() { + // A genuinely partitioned table (identity on `name`) whose source properties ALSO carry a colliding + // bare "partition_columns"=id: the reserved key (__internal.partition_columns) carries the CONNECTOR's + // spec-derived value (`name`), and the user's bare property coexists independently — the two live in + // different namespaces and never overwrite each other. MUTATION: bare reserved key -> the two would + // collide -> one assertion fails -> red. Schema schema = idNameSchema(); PartitionSpec identitySpec = PartitionSpec.builderFor(schema).identity("name").build(); RecordingIcebergCatalogOps ops = new RecordingIcebergCatalogOps(); Map userProps = new HashMap<>(); - userProps.put("partition_columns", "id"); // user tries to spoof partitioning on `id` + userProps.put("partition_columns", "id"); // a plain user property, not the reserved key ops.table = new FakeIcebergTable("t2", schema, identitySpec, "s3://bucket/db1/t2", userProps); ConnectorTableSchema out = metadataWith(ops).getTableSchema(null, new IcebergTableHandle("db1", "t2")); - Assertions.assertEquals("name", out.getProperties().get("partition_columns"), - "partition_columns must be the spec-derived value, not the colliding user property"); + Assertions.assertEquals("name", out.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "the reserved key must carry the spec-derived value"); + Assertions.assertEquals("id", out.getProperties().get("partition_columns"), + "the user's bare property coexists, untouched"); } @Test @@ -726,9 +730,9 @@ public void getTableSchemaEmitsShowPartitionClauseWithTransforms() { Assertions.assertEquals("PARTITION BY LIST (`name`) ()", identitySchema.getProperties().get(ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY), "an identity partition must render as the bare quoted column"); - // Also byte-faithful: the partition_columns CSV (functional, consumed by fe-core) is unchanged. + // Also byte-faithful: the partition-columns CSV (functional, consumed by fe-core) is unchanged. Assertions.assertEquals("name", - identitySchema.getProperties().get("partition_columns")); + identitySchema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY)); // Partitioned with a NON-identity transform (bucket): renders the Doris BUCKET(N, col) term. PartitionSpec bucketSpec = PartitionSpec.builderFor(schema).bucket("id", 8).build(); @@ -855,8 +859,8 @@ public void getTableSchemaEmitsPartitionColumnsForIdentityPartition() { // generic "partition_columns" CSV property (toSchemaCacheValue), the same key MaxCompute/paimon // emit. An unpartitioned table must NOT emit it, or isPartitionedTable() would be wrong post-flip. // MUTATION: always emitting partition_columns -> red. - Assertions.assertNull(unpartSchema.getProperties().get("partition_columns"), - "an unpartitioned table must not carry a partition_columns property"); + Assertions.assertNull(unpartSchema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "an unpartitioned table must not carry a partition-columns property"); // Partitioned (identity on name): the source column name is surfaced under partition_columns. Schema schema = idNameSchema(); @@ -871,7 +875,7 @@ public void getTableSchemaEmitsPartitionColumnsForIdentityPartition() { // legacy IcebergExternalTable did (IcebergUtils.loadTableSchemaCacheValue: spec source columns). // MUTATION: dropping the partition_columns emission -> the key is absent -> red. Assertions.assertEquals("name", - partSchema.getProperties().get("partition_columns"), + partSchema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), "a partitioned table must surface its partition source columns as a CSV"); } @@ -892,7 +896,7 @@ public void getTableSchemaEmitsNonIdentityPartitionSourceColumns() { // WHY: a bucket/truncate/day transform still has a source column that legacy treats as a partition // column. MUTATION: filtering to identity transforms -> "id" absent -> red. Assertions.assertEquals("id", - partSchema.getProperties().get("partition_columns"), + partSchema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), "a non-identity transform must still surface its source column as a partition column"); } diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java index 43ccdf32c6b556..f70935a73e593b 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputeConnectorMetadata.java @@ -163,7 +163,7 @@ public ConnectorTableSchema getTableSchema(ConnectorSession session, java.util.Map props = new java.util.HashMap<>(); if (!partitionColumnNames.isEmpty()) { - props.put("partition_columns", + props.put(ConnectorTableSchema.PARTITION_COLUMNS_KEY, String.join(",", partitionColumnNames)); } return new ConnectorTableSchema( diff --git a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java index 23cd9098dd4e26..2bd372901a4df5 100644 --- a/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java +++ b/fe/fe-connector/fe-connector-paimon/src/main/java/org/apache/doris/connector/paimon/PaimonConnectorMetadata.java @@ -302,14 +302,6 @@ private ConnectorTableSchema buildTableSchema(String tableName, Table table, Lis // (no fe-core dependency); the fe-core consumer filters out the schema-control keys below. if (table instanceof DataTable) { schemaProps.putAll(((DataTable) table).coreOptions().toMap()); - // Strip the FE-internal reserved schema-control keys that may have leaked in from the source - // coreOptions() above (a user option literally named "partition_columns"/"primary_keys"): the - // generic fe-core consumer (PluginDrivenExternalTable.toSchemaCacheValue) treats a non-empty - // "partition_columns" as the partition-column CSV, so a leaked user value would make a - // NON-partitioned table be misdetected as partitioned. Only this connector's own values - // (re-stamped below from partitionKeys/primaryKeys) may reach fe-core. - schemaProps.remove("partition_columns"); - schemaProps.remove("primary_keys"); if (primaryKeys != null && !primaryKeys.isEmpty()) { schemaProps.put(CoreOptions.PRIMARY_KEY.key(), String.join(",", primaryKeys)); } @@ -324,10 +316,10 @@ private ConnectorTableSchema buildTableSchema(String tableName, Table table, Lis // fromRemoteColumnName), so the entries carry the SAME case as the columns to keep the two sides // matchable (a mixed-case paimon partition key would otherwise be silently missed and the table // treated as non-partitioned). - schemaProps.put("partition_columns", String.join(",", partitionKeys)); + schemaProps.put(ConnectorTableSchema.PARTITION_COLUMNS_KEY, String.join(",", partitionKeys)); } if (primaryKeys != null && !primaryKeys.isEmpty()) { - schemaProps.put("primary_keys", String.join(",", primaryKeys)); + schemaProps.put(ConnectorTableSchema.PRIMARY_KEYS_KEY, String.join(",", primaryKeys)); } return new ConnectorTableSchema(tableName, columns, "PAIMON", schemaProps); diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataMvccTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataMvccTest.java index fb35565d36b466..e884ff4a62e720 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataMvccTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataMvccTest.java @@ -797,8 +797,8 @@ public void getTableSchemaAtSchemaIdResolvesHistoricalSchema() { "the schema must be resolved at the snapshot's schemaId"); Assertions.assertEquals(Arrays.asList("id", "dt"), columnNames(schema), "the at-snapshot schema's columns must be mapped (not the latest single-column schema)"); - Assertions.assertEquals("dt", schema.getProperties().get("partition_columns"), - "the at-snapshot schema's partition keys must be emitted as partition_columns"); + Assertions.assertEquals("dt", schema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "the at-snapshot schema's partition keys must be emitted under the reserved partition-columns key"); } @Test diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTest.java index f1bde07222e51b..accb266f1e06c1 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonConnectorMetadataTest.java @@ -294,7 +294,7 @@ public void getTableSchemaPreservesPartitionColumnCase() { // fromRemoteColumnName). So "partition_columns" MUST keep the paimon case "Pt" to match the "Pt" column; // lower-casing it to "pt" would miss the "Pt" column and silently treat the table as NON-partitioned. // MUTATION: re-lowercase partition_columns (the pre-#65094 tier2 behavior) -> "pt" != "Pt" -> red. - Assertions.assertEquals("Pt", schema.getProperties().get("partition_columns")); + Assertions.assertEquals("Pt", schema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY)); } @Test diff --git a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableHandleScanOptionsTest.java b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableHandleScanOptionsTest.java index 2db34bc31b08af..ebcc7197ef0b29 100644 --- a/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableHandleScanOptionsTest.java +++ b/fe/fe-connector/fe-connector-paimon/src/test/java/org/apache/doris/connector/paimon/PaimonTableHandleScanOptionsTest.java @@ -316,8 +316,8 @@ public void getTableSchemaEmitsPartitionColumnsKeyForPartitionedHandle() { // keying it under "partition_keys" left the FE treating paimon as non-partitioned. MUTATION: // emitting the old "partition_keys" key -> "partition_columns" absent + "partition_keys" // present -> both assertions red. - Assertions.assertEquals("dt,region", props.get("partition_columns"), - "getTableSchema must emit partition keys under the 'partition_columns' key"); + Assertions.assertEquals("dt,region", props.get(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "getTableSchema must emit partition keys under the reserved partition-columns key"); Assertions.assertNull(props.get("partition_keys"), "the legacy 'partition_keys' key must no longer be emitted (FE reads partition_columns)"); diff --git a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java index 11354a304c3b4b..7ab848c83eac36 100644 --- a/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java +++ b/fe/fe-core/src/main/java/org/apache/doris/datasource/PluginDrivenExternalTable.java @@ -502,15 +502,15 @@ protected PluginDrivenSchemaCacheValue toSchemaCacheValue(ConnectorMetadata meta List columns = ConnectorColumnConverter.convertColumns(mappedColumns); - // Identify partition columns from the connector's "partition_columns" property (a CSV of - // RAW remote column names; producer: MaxComputeConnectorMetadata). We keep two aligned + // Identify partition columns from the connector's reserved PARTITION_COLUMNS_KEY property (a CSV of + // RAW remote column names; producer: hive/hudi/iceberg/paimon/maxcompute). We keep two aligned // views: the Doris Columns (with mapped/local names, used for getPartitionColumns + types) // and the raw remote names (used to index the raw-keyed partition-value maps from the SPI). // The columns themselves are already present in `columns` (the connector appends partition // columns to the schema, mirroring legacy); here we only mark which ones are partitions. List partitionColumns = new ArrayList<>(); List partitionColumnRemoteNames = new ArrayList<>(); - String partColsProp = tableSchema.getProperties().get("partition_columns"); + String partColsProp = tableSchema.getProperties().get(ConnectorTableSchema.PARTITION_COLUMNS_KEY); if (partColsProp != null && !partColsProp.isEmpty()) { Map byName = Maps.newHashMapWithExpectedSize(columns.size()); for (Column c : columns) { @@ -714,26 +714,18 @@ private Map rawTableProperties() { /** * The connector's user-facing table properties (e.g. paimon coreOptions: path / file.format / - * write-only), used by SHOW CREATE TABLE to render the PROPERTIES(...) block (D-046). The - * FE-internal schema-control keys ({@code partition_columns} / {@code primary_keys}, emitted by - * the connector so {@link #initSchema()} can derive the partition columns) and the SHOW CREATE - * render-hint keys ({@code show.location} / {@code show.partition-clause} / {@code show.sort-clause}, - * rendered as the LOCATION / PARTITION BY / ORDER BY clauses via {@link #getShowLocation()} etc.) and the - * per-table capability marker ({@code connector.per-table-capabilities}, consumed by {@link - * #supportsTopNLazyMaterialize()} / {@link #supportsNestedColumnPrune()}) are stripped — they are not - * user-facing options and must not leak into the rendered PROPERTIES(...). + * write-only), used by SHOW CREATE TABLE to render the PROPERTIES(...) block (D-046). Every FE-internal + * reserved control key ({@link ConnectorTableSchema#RESERVED_CONTROL_KEYS} — the partition-columns / + * primary-keys markers, the SHOW CREATE render hints, and the per-table capability / distribution markers, + * all namespaced under {@code __internal.}) is stripped: they are not user-facing options and must not + * leak into the rendered PROPERTIES(...). Because the reserved keys are namespaced, a source table's own + * user property can never collide with one, so it flows through here unchanged. */ public Map getTableProperties() { Map raw = rawTableProperties(); Map result = new LinkedHashMap<>(); for (Map.Entry entry : raw.entrySet()) { - String key = entry.getKey(); - if ("partition_columns".equals(key) || "primary_keys".equals(key) - || ConnectorTableSchema.SHOW_LOCATION_KEY.equals(key) - || ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY.equals(key) - || ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY.equals(key) - || ConnectorTableSchema.PER_TABLE_CAPABILITIES_KEY.equals(key) - || ConnectorTableSchema.DISTRIBUTION_COLUMNS_KEY.equals(key)) { + if (ConnectorTableSchema.RESERVED_CONTROL_KEYS.contains(entry.getKey())) { continue; } result.put(entry.getKey(), entry.getValue()); diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java index 5ac322f7ef5c46..347311f8e2d2ab 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTablePartitionTest.java @@ -180,7 +180,7 @@ public void testInitSchemaExtractsPartitionColumnsMappingRemoteNames() { new ConnectorColumn("REGION", ConnectorType.of("INT"), "", true, null), new ConnectorColumn("VAL", ConnectorType.of("INT"), "", true, null)), "max_compute", - Collections.singletonMap("partition_columns", "YEAR,REGION")); + Collections.singletonMap(ConnectorTableSchema.PARTITION_COLUMNS_KEY, "YEAR,REGION")); Mockito.when(metadata.getTableSchema(session, handle)).thenReturn(tableSchema); // Identifier mapping lowercases the remote names (raw "YEAR" -> mapped "year"). Mockito.when(metadata.fromRemoteColumnName(Mockito.eq(session), Mockito.anyString(), @@ -229,13 +229,13 @@ public void testInitSchemaNoPartitionsWhenPropAbsent() { @Test public void testGetTablePropertiesStripsSchemaControlKeysButKeepsUserOptions() { - // The connector stuffs BOTH user-facing table options (path / file.format) AND the - // FE-internal schema-control keys (partition_columns / primary_keys) into one properties map. + // The connector stuffs BOTH user-facing table options (path / file.format) AND the FE-internal + // schema-control keys (reserved, namespaced under __internal.) into one properties map. Map rawProps = new LinkedHashMap<>(); rawProps.put("path", "s3://wh/db/t"); rawProps.put("file.format", "orc"); - rawProps.put("partition_columns", "dt"); - rawProps.put("primary_keys", "id"); + rawProps.put(ConnectorTableSchema.PARTITION_COLUMNS_KEY, "dt"); + rawProps.put(ConnectorTableSchema.PRIMARY_KEYS_KEY, "id"); PluginDrivenSchemaCacheValue cacheValue = new PluginDrivenSchemaCacheValue( Collections.singletonList(new Column("id", PrimitiveType.INT)), Collections.emptyList(), Collections.emptyList(), rawProps); @@ -243,16 +243,16 @@ public void testGetTablePropertiesStripsSchemaControlKeysButKeepsUserOptions() { Map props = table.getTableProperties(); // WHY (D-046): SHOW CREATE TABLE's LOCATION reads "path" and PROPERTIES(...) dumps this map. - // The user-facing options MUST survive, but the FE-internal control keys MUST be stripped — + // The user-facing options MUST survive, but the FE-internal reserved keys MUST be stripped — // they are emitted only so initSchema() can derive partition columns and would corrupt the - // round-tripped DDL. MUTATION: dropping the filter -> partition_columns/primary_keys leak -> + // round-tripped DDL. MUTATION: dropping the filter -> the reserved keys leak -> // red; over-filtering (removing "path") -> LOCATION renders empty -> red. Assertions.assertEquals("s3://wh/db/t", props.get("path")); Assertions.assertEquals("orc", props.get("file.format")); - Assertions.assertFalse(props.containsKey("partition_columns"), - "partition_columns is an FE-internal control key, must not appear in SHOW CREATE PROPERTIES"); - Assertions.assertFalse(props.containsKey("primary_keys"), - "primary_keys is an FE-internal control key, must not appear in SHOW CREATE PROPERTIES"); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.PARTITION_COLUMNS_KEY), + "the reserved partition-columns key must not appear in SHOW CREATE PROPERTIES"); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.PRIMARY_KEYS_KEY), + "the reserved primary-keys key must not appear in SHOW CREATE PROPERTIES"); } @Test diff --git a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java index 2032fe872a7316..c025c58a217416 100644 --- a/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java +++ b/fe/fe-core/src/test/java/org/apache/doris/datasource/PluginDrivenExternalTableTest.java @@ -842,19 +842,24 @@ private static PluginDrivenExternalTable pluginTableWithRawProperties(Map that key leaks into PROPERTIES -> red. + public void getTablePropertiesStripsReservedKeysAndPassesThroughColludingUserKeys() { + // WHY: the rendered PROPERTIES(...) block must contain only user-facing properties — every FE-internal + // reserved control key (ConnectorTableSchema.RESERVED_CONTROL_KEYS, all namespaced under __internal.: + // the partition-columns / primary-keys markers + the SHOW CREATE render hints) must be stripped. + // Because the reserved keys are namespaced, a source table's own BARE property (e.g. literally named + // "partition_columns") can NEVER collide with one, so it flows through unchanged. + // MUTATION: reverting a reserved key to a bare name -> the bare user key would be stripped (data loss) + // or the reserved key would leak into PROPERTIES -> red. Map raw = new LinkedHashMap<>(); raw.put("write.format.default", "parquet"); - raw.put("partition_columns", "id"); - raw.put("primary_keys", "id"); - raw.put("show.location", "s3://bucket/db/t"); - raw.put("show.partition-clause", "PARTITION BY LIST (`id`) ()"); - raw.put("show.sort-clause", "ORDER BY (`id` ASC NULLS FIRST)"); + raw.put(ConnectorTableSchema.PARTITION_COLUMNS_KEY, "id"); + raw.put(ConnectorTableSchema.PRIMARY_KEYS_KEY, "id"); + raw.put(ConnectorTableSchema.SHOW_LOCATION_KEY, "s3://bucket/db/t"); + raw.put(ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY, "PARTITION BY LIST (`id`) ()"); + raw.put(ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY, "ORDER BY (`id` ASC NULLS FIRST)"); raw.put("path", "s3://bucket/db/t"); + // A user's own BARE property whose name equals the OLD un-namespaced reserved name: it must survive. + raw.put("partition_columns", "a_user_value"); Map props = pluginTableWithRawProperties(raw).getTableProperties(); @@ -862,18 +867,20 @@ public void getTablePropertiesStripsControlAndRenderHintKeys() { "user-facing properties are preserved"); Assertions.assertTrue(props.containsKey("path"), "a connector's user-facing path property (paimon) is preserved"); - Assertions.assertFalse(props.containsKey("partition_columns")); - Assertions.assertFalse(props.containsKey("primary_keys")); - Assertions.assertFalse(props.containsKey("show.location")); - Assertions.assertFalse(props.containsKey("show.partition-clause")); - Assertions.assertFalse(props.containsKey("show.sort-clause")); + Assertions.assertEquals("a_user_value", props.get("partition_columns"), + "a user's bare partition_columns property flows through (no collision with the reserved key)"); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.PARTITION_COLUMNS_KEY)); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.PRIMARY_KEYS_KEY)); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.SHOW_LOCATION_KEY)); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY)); + Assertions.assertFalse(props.containsKey(ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY)); } @Test public void getShowLocationReadsHintKeyWithPathFallback() { - // Reserved show.location hint -> rendered LOCATION. + // Reserved show-location hint -> rendered LOCATION. Map iceberg = new HashMap<>(); - iceberg.put("show.location", "s3://bucket/db/t"); + iceberg.put(ConnectorTableSchema.SHOW_LOCATION_KEY, "s3://bucket/db/t"); Assertions.assertEquals("s3://bucket/db/t", pluginTableWithRawProperties(iceberg).getShowLocation()); @@ -884,9 +891,9 @@ public void getShowLocationReadsHintKeyWithPathFallback() { Assertions.assertEquals("s3://bucket/db/p", pluginTableWithRawProperties(paimon).getShowLocation()); - // show.location wins over path when both present (a connector that emits both). + // the show-location hint wins over path when both present (a connector that emits both). Map both = new HashMap<>(); - both.put("show.location", "s3://hint"); + both.put(ConnectorTableSchema.SHOW_LOCATION_KEY, "s3://hint"); both.put("path", "s3://path"); Assertions.assertEquals("s3://hint", pluginTableWithRawProperties(both).getShowLocation()); @@ -897,8 +904,8 @@ public void getShowLocationReadsHintKeyWithPathFallback() { @Test public void getShowPartitionAndSortClauseReadHintKeys() { Map raw = new HashMap<>(); - raw.put("show.partition-clause", "PARTITION BY LIST (BUCKET(8, `id`)) ()"); - raw.put("show.sort-clause", "ORDER BY (`name` DESC NULLS LAST)"); + raw.put(ConnectorTableSchema.SHOW_PARTITION_CLAUSE_KEY, "PARTITION BY LIST (BUCKET(8, `id`)) ()"); + raw.put(ConnectorTableSchema.SHOW_SORT_CLAUSE_KEY, "ORDER BY (`name` DESC NULLS LAST)"); PluginDrivenExternalTable table = pluginTableWithRawProperties(raw); Assertions.assertEquals("PARTITION BY LIST (BUCKET(8, `id`)) ()", table.getShowPartitionClause()); Assertions.assertEquals("ORDER BY (`name` DESC NULLS LAST)", table.getShowSortClause()); From 65d3d6432324b5dabea854801fb885303409e1f4 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 13:26:08 +0800 Subject: [PATCH 332/334] [doc](catalog) reserved-key rename: design + supersede L19 silent-strip - DESIGN-reserved-connector-keys-framework.md: the __internal.-prefix rename (implemented) - FIX-L19-design.md: mark the producer-side silent-strip approach SUPERSEDED - task-list / HANDOFF: L19 done via the rename (supersedes 01668779fd9) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 2 +- plan-doc/task-list-65185-reverify-fixes.md | 2 +- ...ESIGN-reserved-connector-keys-framework.md | 69 +++++++++++++++++++ plan-doc/tasks/designs/FIX-L19-design.md | 7 ++ 4 files changed, 78 insertions(+), 2 deletions(-) create mode 100644 plan-doc/tasks/designs/DESIGN-reserved-connector-keys-framework.md diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index 3b7436888b8c2f..afcf70aa84e0df 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -16,7 +16,7 @@ > **L15 追加转折 → scan-metrics 功能恢复** `8d4209865a3`:用户质疑 L15「删死引用」是否丢功能。复核参考仓库 master 证实:老 `PaimonScanMetricsReporter`+`PaimonMetricRegistry` 是真功能,插件迁移 **paimon+iceberg 都丢**(登记偏差 T02/T29);iceberg `ICEBERG_SCAN_METRICS` 仅靠**死 legacy `IcebergScanNode`** 续命(我先前「iceberg 活」判断有误)。用户要求 connector SPI 框架**统一恢复**。已建**连接器中立 scan-metrics SPI**:`ConnectorScanProfile`(值类型)+`ConnectorScanPlanProvider.collectScanProfiles`(默认空);paimon 侧 port `PaimonMetricRegistry`+新 `PaimonScanMetrics`(pull,`withMetricRegistry`),iceberg 侧新 `IcebergScanProfileReporter`(push,**挂 `planScanInternal` 数据/count 路,非共享 `buildScan`**——避 streaming 泄漏 blocker);fe-core 纯静态 `writeScanProfilesInto` 写 `SummaryProfile`(无源分支)、**复活 `PAIMON_SCAN_METRICS` 常量**;`DebugUtil` 时间/字节格式化器**自搬**(连接器不得 import fe-core)。stash 按 queryId、`collectScanProfiles` 排空 + `releaseReadTransaction` 兜底清。设计 3-lens 红队 `wf_0f803c49-7bb` 全 SOUND_WITH_CHANGES(2 blocker+majors 全折入)。UT 12/12 + 全模块 paimon 360+iceberg 982+fe-core 94 绿、0 checkstyle、import 门净。**踩坑**:iceberg 既有 UT 传 **null session** → 我 `session.getQueryId()` NPE 挂 31 例,加 null-session guard 修(paimon 同防御);paimon 模块须 `package`(shade)非 `test-compile`(HiveConf 假失败,memory `doris-build-verify-gotchas`)。设计/summary=`designs/FIX-scan-metrics-spi-{design,summary}.md`。**e2e live-gated**:真集群 paimon/iceberg 查询后 profile 含 "Paimon/Iceberg Scan Metrics" 组 + manifest 缓存命中/未命中。 > > **L16–L19 已全部 DONE(本 session 续，2026-07-13）** — 4 recon agent(`wf_0aee6600-244`)复核 HEAD；两条决策类(L17/L18)用中文讲清 background+Trino+选项问用户后实现。commits: -> - **L19** `01668779fd9` — partition_columns 保留键撞名:复核**扩 scope 到 iceberg+hive+paimon 三连接器**(同 bug；hudi/mc 用全新 map 免疫)。producer-side `remove`(fe-core 无法区分连接器发 vs 用户透传，铁律禁解析连接器键义)。iceberg 50/50+hive 23/23 UT(各 2 RED-able)、paimon package 编译、import 门净;paimon coreOptions 路现有 fake 非 `DataTable` 不可单测→检视+e2e-gated 登记。 +> - **L19** 初版 `01668779fd9`(producer-side silent `remove`)**已被 `2c58d8342c1` 取代**——**用户否决静默删除**(丢用户数据无信号、后来者加保留键无提示)。改**给所有保留控制键加 `__internal.` 前缀**(集中为 `ConnectorTableSchema` 常量 + `RESERVED_CONTROL_KEYS` 集；与既有 `show.*`/`connector.*` 同避撞机制)。撞名由构造消除:用户裸 `partition_columns` 作普通属性透传、永不当控制键;**只重命名不加校验**(前缀够独特)。保留键全 FE-only(BE 走 `path_partition_keys`)、非持久化→零 BE/序列化/golden 影响。删三连接器 `remove`;fe-core strip 改 `RESERVED_CONTROL_KEYS.contains`。iceberg 50+hive 23+paimon 12/19/43+fe-core 8/36 绿,0 checkstyle,import 门净。设计 `designs/DESIGN-reserved-connector-keys-framework.md`(`FIX-L19-design.md` 标 SUPERSEDED)。 > - **L18** `1c9c99c7767` — 未知/v3 类型静默 UNSUPPORTED:**用户签字 accept graceful degradation**(非抛，登记 **DV-051**);无功能改动，加澄清注释+守护测试 `unknownAndV3TypesDegradeToUnsupportedByDesign`(未来改抛→red)。写方向 `toIcebergPrimitive` 仍抛。IcebergTypeMappingReadTest 11/11。 > - **L16** `76afd6f2e80` — 快照/schema 缓存偏斜:复核判 **REFUTED/benign**(hasSnapshotPin 臂已传 `emptyList` 非超集;偏斜构造不出——原子 `(snapshotId,schemaId)` pin + iceberg `schemas()` append-only + `pinnedSchema`(dict)/`getTableSchema`(slot)同选择器同静默回退)。**无功能改动**，仅加交叉引用防呆注释锁「两处回退须一致」不变量(否则 BE `children.at()` SIGABRT)。IcebergScanPlanProviderTest 88/88。 > - **L17** `3627556db34` — 同表多版本 version-blind schema:**用户签字 fail-loud + TODO 重构**(`D-MVCC-VERSION-SCHEMA`)。3-lens 红队 `wf_f7b69cf7-ec8` **推翻 analysis-time `getSchemaCacheValue` 守卫**(顺序依赖 + 被 plain-ref/MTMV default pin 遮蔽 + @incr 漏 → 同 query 抛或静默 skew 视绑定序)→改**逐引用 scan-node 守卫**:`PluginDrivenScanNode.pinMvccSnapshot` 解析版本感知 pin 后，校验每个 bound tuple 列在**本引用 pinned schema** 可解析(有 iceberg field-id 按 id、否则按 name)，否则抛 `UserException`。确定性+完整(catch 自连接/latest-遮蔽/@incr/MTMV)+无误报(`t@old JOIN t@latest` 各 tuple 匹配自身版本→不抛)。静态可测 helper `assertBoundColumnsResolveInPinnedSchema`。SchemaGuardTest 7/7 + MvccPin 3/3 + MvccExternalTable 59/59。残余:嵌套 field-id-only 重编号看不见(iceberg id 稳定不触发)。 diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index bc8cfac777d4d6..0460bef00eb8b6 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -196,7 +196,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L16** DONE `76afd6f2e80`(复核 REFUTED+防呆注释;设计 `designs/FIX-L16-design.md`)· 复核判定**已消除/benign**:`IcebergScanPlanProvider` hasSnapshotPin 臂现已传 `Collections.emptyList()`(全量 pinned schema,非 requestedLowerNames 超集,自首 commit 即如此);残留「快照 id vs schema id 偏斜」构造不出——pin 原子取 `(snapshotId,schemaId)`+iceberg `schemas()` 只增+两条取 schema 路径(`pinnedSchema` dict / `getTableSchema` slot)用同一选择器同一静默回退。**无功能改动**,仅加交叉引用防呆注释锁住「两处回退必须一致」不变量(否则 BE `children.at()` SIGABRT)。残留结构隐患=L17 同根(逐引用版本感知)。 - [x] **L17** DONE `3627556db34`(设计 `designs/FIX-L17-design.md`;3-lens 红队 `wf_f7b69cf7-ec8` 推翻 analysis-time 方案→改 scan-node)· **用户 2026-07-13 签字:先 fail-loud + 记 TODO 重构**。红队证 analysis-time `getSchemaCacheValue` 守卫**顺序依赖+被 plain-ref/MTMV default pin 遮蔽+@incr 漏**(同一 query 抛或静默 skew 视绑定序)→改**逐引用 scan-node 守卫**:`PluginDrivenScanNode.pinMvccSnapshot` 解析版本感知 pin 后,校验每个 bound tuple 列在**本引用 pinned schema** 可解析(有 field-id 按 id、否则按 name),否则抛。确定性+完整(catch 自连接/latest-遮蔽/@incr/MTMV)+无误报(`t@old JOIN t@latest` 各 tuple 匹配自身版本→不抛)。静态可测 helper,7 RED-able UT。TODO=`D-MVCC-VERSION-SCHEMA`(逐引用版本感知 schema 绑定,仿 Trino 版本作用域 handle)。残余:嵌套 field-id-only 重编号看不见(iceberg id 稳定不触发)。 - [x] **L18** DONE `1c9c99c7767`(设计 `designs/FIX-L18-design.md`;登记 **DV-051**)· **用户 2026-07-13 签字 accept「统一映射 UNSUPPORTED」**(非抛):`IcebergTypeMapping.fromIcebergType/fromPrimitive` 两 `default` 臂保持把 Doris 无法表示的 iceberg 类型(v3 primitives TIMESTAMP_NANO/GEOMETRY/GEOGRAPHY/UNKNOWN + non-primitive VARIANT)映射 UNSUPPORTED→表能加载、该列 present-but-unqueryable、其它列可用。**背离** legacy(抛 `IllegalArgumentException`)与 Trino(抛 NOT_SUPPORTED),但用户选 graceful degradation。**无功能改动**;加澄清注释+守护测试 `unknownAndV3TypesDegradeToUnsupportedByDesign`(未来改抛→red);写方向 `toIcebergPrimitive` 仍抛(CREATE 不静默接受不可 round-trip 类型)。 -- [x] **L19** DONE `01668779fd9`(设计 `designs/FIX-L19-design.md`)· 连接器侧 `remove` 保留键防撞名。**scope 扩至 iceberg+hive+paimon 三连接器**(复核发现同 bug:三者均 `putAll(源属性)` 后仅有条件盖 `partition_columns`,非分区表漏出用户同名键→误判分区;hudi/maxcompute 用全新 map 免疫)。修=各连接器 `putAll` 后立即 `remove("partition_columns")`(paimon 另 `remove("primary_keys")`),分区表下方 re-stamp 连接器真值→连接器值必胜。fe-core 无法区分连接器发 vs 用户透传(铁律禁解析连接器键义)故必须 producer-side。iceberg+hive 各 2 RED-able UT;paimon `DataTable` coreOptions 路现有 fake 非 DataTable 不可单测→按检视+e2e-gated 登记(非静默略过)。 +- [x] **L19** DONE(初版 `01668779fd9` **已被 `2c58d8342c1` 取代**;设计 `designs/FIX-L19-design.md`〔SUPERSEDED〕+ `designs/DESIGN-reserved-connector-keys-framework.md`)· **用户否决静默 `remove`**(丢用户数据无信号、后来者加保留键无提示)→改**给所有保留控制键统一加 `__internal.` 前缀**(与既有 `show.*`/`connector.*` 同机制,集中为 `ConnectorTableSchema` 常量:`__internal.partition_columns`/`__internal.primary_keys`/`__internal.show.*`/`__internal.connector.*` + `RESERVED_CONTROL_KEYS` 集)。撞名**由构造消除**——用户裸 `partition_columns` 作普通用户属性正常透传(不删、SHOW CREATE 正常显示),永不被当控制键。**用户裁决只重命名、不加校验**(前缀够独特)。保留键全 **FE-only**(不发 BE,BE 走 `path_partition_keys`)、非持久化、SHOW CREATE 剥离→重命名零 BE/序列化/golden 影响。删三连接器 L19 `remove`;fe-core `getTableProperties` 改 `RESERVED_CONTROL_KEYS.contains`(未来加键自动剥离)。iceberg 50+hive 23+paimon 12/19/43+fe-core 8/36 全绿,0 checkstyle,import 门净。**新增 UT 证撞名消除**(裸用户键与保留键共存/透传)。e2e live-gated(非分区表设 `partition_columns` 用户属性不误判 + SHOW CREATE 显示该属性)。 - [ ] **L20** ⏸ MC EQ `==` · `MaxComputePredicateConverter.java:145-146`:直接 `case EQ: opDesc = "=";` 对齐 SDK/legacy 消除不确定性(推荐);或 live ODPS A/B 确认容忍。顺带补 IN 方向回归测试(P4-3-IN 已修但缺测)。 --- diff --git a/plan-doc/tasks/designs/DESIGN-reserved-connector-keys-framework.md b/plan-doc/tasks/designs/DESIGN-reserved-connector-keys-framework.md new file mode 100644 index 00000000000000..0aa4fd69ada7f6 --- /dev/null +++ b/plan-doc/tasks/designs/DESIGN-reserved-connector-keys-framework.md @@ -0,0 +1,69 @@ +# DESIGN — namespace all reserved connector control keys under `__internal.` (supersedes L19 silent-strip) + +> **Status: IMPLEMENTED (user-directed).** Supersedes the L19 fix `01668779fd9` (per-connector silent +> `remove("partition_columns")`), which the user rejected: silent deletion discards user data with no signal, +> and a future developer adding a new reserved keyword would get no feedback. + +## Decision (user, this session) +1. **Namespace every FE-internal reserved control key under `__internal.`** — not just the two bare ones + (`partition_columns` / `primary_keys`) but also the already-namespaced `show.*` / `connector.*` keys, so + the whole reserved namespace is uniform and distinctive. +2. **Rename only — NO validation / fail-loud.** A distinctive `__internal.` prefix makes a collision with a + real user table property practically impossible (this is exactly how the pre-existing `show.*` / + `connector.*` keys already avoided collision), so a runtime check is unnecessary. Consistent with those + five keys, which have no fail-loud either. + +## Why this is correct + minimal +- `partition_columns` (and every reserved control key) is **FE-only** — verified: none is emitted by any + connector's `getScanNodeProperties`, and the table-properties map is never serialized into the BE thrift + scan request. BE gets partition columns via the SEPARATE `path_partition_keys` scan property. So renaming + has **zero BE impact**. +- The reserved keys are **not GSON/editlog-persisted** (`PluginDrivenSchemaCacheValue` has no serialization; + the schema cache is rebuilt on demand), so there is **no persisted-state migration**. +- No regression golden `.out` file references any reserved key (they are all stripped from SHOW CREATE). +- The collision is eliminated **by construction**: a user's own bare `partition_columns` property now simply + flows through as a normal user property (preserved, rendered in SHOW CREATE) — no silent strip, no failure, + no misdetection. This is strictly better than both the pre-L19 collision and the L19 silent strip. + +## Change +### `ConnectorTableSchema` (fe-connector-api) — single source of truth +- New `INTERNAL_KEY_PREFIX = "__internal."`. +- Every reserved key is `INTERNAL_KEY_PREFIX + `: + `__internal.partition_columns`, `__internal.primary_keys`, `__internal.show.location`, + `__internal.show.partition-clause`, `__internal.show.sort-clause`, + `__internal.connector.per-table-capabilities`, `__internal.connector.distribution-columns`. +- `partition_columns` / `primary_keys` promoted from scattered bare literals to the constants + `PARTITION_COLUMNS_KEY` / `PRIMARY_KEYS_KEY`. +- New `RESERVED_CONTROL_KEYS` set for the fe-core strip. + +### Producers — reference the central constants (no more bare literals) +iceberg / maxcompute / paimon: `put(ConnectorTableSchema.PARTITION_COLUMNS_KEY, …)` (paimon also +`PRIMARY_KEYS_KEY`). hive / hudi: their local `PARTITION_COLUMNS_PROPERTY` alias now `= +ConnectorTableSchema.PARTITION_COLUMNS_KEY`. The `show.*` / `connector.*` producers already used the +constants, so their key VALUES update centrally. **The L19 per-connector `remove()` strips are deleted** +(iceberg / hive / paimon) — no longer needed. + +### fe-core consumer +- `PluginDrivenExternalTable.toSchemaCacheValue` reads `ConnectorTableSchema.PARTITION_COLUMNS_KEY`. +- `getTableProperties()` strips `ConnectorTableSchema.RESERVED_CONTROL_KEYS.contains(key)` (replaces the + hardcoded key list → a future reserved key is stripped automatically). + +## Behavior change +A source table (created anywhere) whose user properties contain a bare `partition_columns` / `primary_keys` +(or the old `show.location`, …) is now treated as a plain user property: preserved, flows through to SHOW +CREATE, never mistaken for a control key. Normal tables are byte-identical. No failure path. + +## Iron rule / blast radius +- All logic in `fe-connector-api` (constants) + the connectors (reference constants) + the fe-core strip + swap. No source-name branching, no property parsing added to fe-core. Wire format unchanged (still a map). +- FE-only rename → no BE / thrift / GSON / replay change. + +## Tests +- Connector metadata tests assert the connector emits under `PARTITION_COLUMNS_KEY` (the constant), and NEW + tests assert a user's bare `partition_columns` property **coexists** with the reserved key (partitioned) or + **flows through** while the table stays unpartitioned (no collision) — replacing the L19 "stripped" asserts. +- fe-core `getTableProperties` strips the `RESERVED_CONTROL_KEYS` and passes a colliding bare user key through. + +## Note +Realizes design debt **D-MAGICKEY**'s "promote the bare keys to declared constants" — now every reserved key +is a declared, namespaced constant in one place. diff --git a/plan-doc/tasks/designs/FIX-L19-design.md b/plan-doc/tasks/designs/FIX-L19-design.md index 05649ad3972beb..b4c3866f775b50 100644 --- a/plan-doc/tasks/designs/FIX-L19-design.md +++ b/plan-doc/tasks/designs/FIX-L19-design.md @@ -1,5 +1,12 @@ # FIX-L19 — `partition_columns` reserved magic-key collision with source properties +> **⚠ SUPERSEDED.** The producer-side silent `remove()` approach below (commit `01668779fd9`) was rejected by +> the user (silent deletion discards user data with no signal). It is replaced by +> **`DESIGN-reserved-connector-keys-framework.md`** (commit `2c58d8342c1`): namespace every reserved control +> key under `__internal.` so collision is impossible by construction and a user's bare `partition_columns` +> flows through untouched. This doc is kept for the problem analysis (mechanism / FE-only / three-connector +> scope) which the rename builds on. + ## Problem (HEAD-verified) The SPI carries partition info from connector → fe-core through a stringly-typed reserved key `"partition_columns"` **inside the same properties map** that also carries the source table's own From b2786fa1200daf1600d1f32b90bcc38ae39b5bf3 Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 13:40:47 +0800 Subject: [PATCH 333/334] [fix](catalog) maxcompute: EQ predicate push '=' via SDK operator description (was '==') MaxComputePredicateConverter hand-wrote the ODPS operator symbols and the EQ arm emitted Java's '==', producing pushdown strings like `id == 5`. MaxCompute (like SQL) has no '==' operator, so ODPS rejects it and the equality predicate silently fails to push down (full scan). Legacy fe-core MaxComputeScanNode never had this: it mapped the operator to the ODPS SDK BinaryPredicate.Operator enum and took the symbol from odpsOp.getDescription() (EQUALS -> "="). SDK bytecode confirms the canonical descriptions ("=", "!=", "<", "<=", ">", ">="); the connector-api ConnectorComparison.Operator.EQ also declares "=". Restore the legacy pattern: map ConnectorComparison.Operator to the SDK BinaryPredicate.Operator and emit getDescription(), instead of patching one string. This removes the hand-typed-symbol drift class entirely (single authority = the SDK). EQ_FOR_NULL ("<=>") has no ODPS equivalent and still falls to default -> throw -> NO_PREDICATE (BE re-filters), matching legacy's skip. RawPredicate kept (custom datetime/string value formatting unchanged); only EQ output changes (id == 5 -> id = 5), the other five operators are byte-identical. Add guard tests: EQ emits single '=' never '==' (RED-able), full operator set maps to SDK symbols, EQ_FOR_NULL not pushed, and IN/NOT IN direction (col IN (values)). Design: plan-doc/tasks/designs/FIX-L20-design.md Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- .../MaxComputePredicateConverter.java | 23 +++-- .../MaxComputePredicateConverterTest.java | 70 ++++++++++++++ plan-doc/tasks/designs/FIX-L20-design.md | 91 +++++++++++++++++++ 3 files changed, 176 insertions(+), 8 deletions(-) create mode 100644 plan-doc/tasks/designs/FIX-L20-design.md diff --git a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java index 327889c4d555ee..1f52b12f333aaa 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/main/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverter.java @@ -29,6 +29,7 @@ import com.aliyun.odps.OdpsType; import com.aliyun.odps.table.optimizer.predicate.Attribute; +import com.aliyun.odps.table.optimizer.predicate.BinaryPredicate; import com.aliyun.odps.table.optimizer.predicate.CompoundPredicate; import com.aliyun.odps.table.optimizer.predicate.Predicate; import com.aliyun.odps.table.optimizer.predicate.RawPredicate; @@ -164,30 +165,36 @@ private Predicate convertComparison(ConnectorComparison cmp) { String columnName = extractColumnName(cmp.getLeft()); String value = formatLiteralValue(columnName, cmp.getRight()); - String opDesc; + // Source the operator symbol from the ODPS SDK's own BinaryPredicate.Operator description + // rather than hand-writing it, mirroring legacy MaxComputeScanNode.convertExprToOdpsPredicate. + // The SDK is the authority for what ODPS accepts (EQUALS -> "="); a hand-written table let the + // EQ entry drift to Java's "==", which MaxCompute (like SQL) does not accept -> pushdown lost. + // EQ_FOR_NULL ("<=>") has no ODPS BinaryPredicate equivalent, so it (and any future operator) + // falls through to default -> throw -> NO_PREDICATE (BE re-filters), matching legacy's skip. + BinaryPredicate.Operator odpsOp; switch (cmp.getOperator()) { case EQ: - opDesc = "=="; + odpsOp = BinaryPredicate.Operator.EQUALS; break; case NE: - opDesc = "!="; + odpsOp = BinaryPredicate.Operator.NOT_EQUALS; break; case LT: - opDesc = "<"; + odpsOp = BinaryPredicate.Operator.LESS_THAN; break; case LE: - opDesc = "<="; + odpsOp = BinaryPredicate.Operator.LESS_THAN_OR_EQUAL; break; case GT: - opDesc = ">"; + odpsOp = BinaryPredicate.Operator.GREATER_THAN; break; case GE: - opDesc = ">="; + odpsOp = BinaryPredicate.Operator.GREATER_THAN_OR_EQUAL; break; default: throw new UnsupportedOperationException("Unsupported operator: " + cmp.getOperator()); } - return new RawPredicate(columnName + " " + opDesc + " " + value); + return new RawPredicate(columnName + " " + odpsOp.getDescription() + " " + value); } private Predicate convertIn(ConnectorIn in) { diff --git a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java index acf9297cbdb2a9..c2a7096a7632f5 100644 --- a/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java +++ b/fe/fe-connector/fe-connector-maxcompute/src/test/java/org/apache/doris/connector/maxcompute/MaxComputePredicateConverterTest.java @@ -331,4 +331,74 @@ public void topLevelOrIsNotTolerated() { Assertions.assertSame(Predicate.NO_PREDICATE, converter(true, UTC).convert(or), "an unconvertible disjunct must drop the whole OR (no row loss)"); } + + // ---- L20 (FIX-L20): the comparison operator symbol must come from the ODPS SDK's own + // BinaryPredicate.Operator description, not a hand-written table. EQ must push a single "=", + // never Java's "==" (which MaxCompute, like SQL, does not accept -> pushdown lost -> full + // scan). Legacy MaxComputeScanNode used odpsOp.getDescription(); the migration hand-wrote the + // symbols and EQ drifted to "==". "id" is INT in typeMap(), so numeric formatting applies. ---- + + private static String pushedComparison(ConnectorComparison.Operator op) { + ConnectorComparison cmp = new ConnectorComparison(op, + new ConnectorColumnRef("id", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5)); + // RawPredicate.toString() returns the raw pushed string; normalize whitespace (formatLiteralValue + // pads values with surrounding spaces) so the assertion pins the operator, not the spacing. + return converter(true, UTC).convert(cmp).toString().trim().replaceAll("\\s+", " "); + } + + @Test + public void testEqualsEmitsSingleEqualsNotDoubleEquals() { + // RED on the pre-fix code, which emitted "id == 5". The ODPS SDK's EQUALS description is "=". + String raw = converter(true, UTC).convert(new ConnectorComparison(ConnectorComparison.Operator.EQ, + new ConnectorColumnRef("id", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5))).toString(); + Assertions.assertFalse(raw.contains("=="), "EQ must push a single '=', not Java's '=='; got: " + raw); + Assertions.assertEquals("id = 5", raw.trim().replaceAll("\\s+", " "), + "EQ must push 'id = 5'; got: " + raw); + } + + @Test + public void testAllComparisonOperatorsEmitSdkSymbols() { + // Pin the whole operator set to the SDK BinaryPredicate.Operator descriptions so a future + // hand-edit cannot silently drift any symbol again. + Assertions.assertEquals("id = 5", pushedComparison(ConnectorComparison.Operator.EQ)); + Assertions.assertEquals("id != 5", pushedComparison(ConnectorComparison.Operator.NE)); + Assertions.assertEquals("id < 5", pushedComparison(ConnectorComparison.Operator.LT)); + Assertions.assertEquals("id <= 5", pushedComparison(ConnectorComparison.Operator.LE)); + Assertions.assertEquals("id > 5", pushedComparison(ConnectorComparison.Operator.GT)); + Assertions.assertEquals("id >= 5", pushedComparison(ConnectorComparison.Operator.GE)); + } + + @Test + public void testEqForNullIsNotPushedDown() { + // EQ_FOR_NULL ("<=>") has no ODPS BinaryPredicate equivalent: it must degrade to NO_PREDICATE + // (default -> throw -> caught), never be pushed as a malformed "<=>" RawPredicate. BE re-filters. + Predicate p = converter(true, UTC).convert(new ConnectorComparison( + ConnectorComparison.Operator.EQ_FOR_NULL, + new ConnectorColumnRef("id", ConnectorType.of("INT")), ConnectorLiteral.ofLong(5))); + Assertions.assertSame(Predicate.NO_PREDICATE, p, + "EQ_FOR_NULL has no ODPS equivalent and must not be pushed down"); + } + + // ---- P4-3-IN direction regression: the IN-polarity fix pushes `col IN (values)` (column first), + // not the reversed form. This had no dedicated test; pin both IN and NOT IN direction. ---- + + @Test + public void testInListEmitsColumnThenValues() { + ConnectorIn in = new ConnectorIn( + new ConnectorColumnRef("id", ConnectorType.of("INT")), + Arrays.asList(ConnectorLiteral.ofLong(1), ConnectorLiteral.ofLong(2)), + false); + String s = converter(true, UTC).convert(in).toString().trim().replaceAll("\\s+", " "); + Assertions.assertEquals("id IN ( 1 , 2 )", s, "IN must push 'col IN (values)'; got: " + s); + } + + @Test + public void testNotInListEmitsColumnThenNotIn() { + ConnectorIn in = new ConnectorIn( + new ConnectorColumnRef("id", ConnectorType.of("INT")), + Arrays.asList(ConnectorLiteral.ofLong(1), ConnectorLiteral.ofLong(2)), + true); + String s = converter(true, UTC).convert(in).toString().trim().replaceAll("\\s+", " "); + Assertions.assertEquals("id NOT IN ( 1 , 2 )", s, "NOT IN must push 'col NOT IN (values)'; got: " + s); + } } diff --git a/plan-doc/tasks/designs/FIX-L20-design.md b/plan-doc/tasks/designs/FIX-L20-design.md new file mode 100644 index 00000000000000..6420e62ee9f30a --- /dev/null +++ b/plan-doc/tasks/designs/FIX-L20-design.md @@ -0,0 +1,91 @@ +# FIX-L20 — maxcompute EQ predicate emits `==` instead of `=` + +## Problem (HEAD-verified) +`MaxComputePredicateConverter.convertComparison` (fe-connector-maxcompute) hand-writes the ODPS +operator symbols in a `switch`, and the `EQ` arm emits Java's `==`: + +```java +// MaxComputePredicateConverter.java:169-170 +case EQ: + opDesc = "=="; + break; +``` + +The built string (`new RawPredicate(columnName + " " + opDesc + " " + value)`) is pushed to ODPS. +MaxCompute — like standard SQL — has **no `==` operator**; equality is a single `=`. So an equality +predicate pushes `id == 5`, which ODPS does not accept: the pushdown is lost and the scan degrades to +a full scan (a silent perf regression). The other five arms (`!=`,`<`,`<=`,`>`,`>=`) are correct. + +## Root cause — why legacy never had this +Legacy fe-core (`1da88365e85^:MaxComputeScanNode.convertExprToOdpsPredicate`) did **not** hand-write +symbols. It mapped Doris' operator to the ODPS SDK's own `BinaryPredicate.Operator` enum and took the +symbol from the SDK: + +```java +switch (binaryPredicate.getOp()) { + case EQ: odpsOp = BinaryPredicate.Operator.EQUALS; break; // ... +} +stringBuilder.append(odpsOp.getDescription()); // authoritative ODPS symbol +``` + +The migration to the plugin replaced this "map-to-SDK-enum + `getDescription()`" with a hand-written +symbol table, and the `EQ` entry drifted to Java's `==` (the other five happened to be copied right). +The defect is structural: **a human should not re-type what the SDK already defines.** + +## Evidence (static, decisive — no live A/B needed) +- **SDK bytecode** (`odps-sdk-table-api-0.48.8-public.jar`, `BinaryPredicate$Operator`): + `EQUALS` description = `"="`, `NOT_EQUALS`=`"!="`, `GREATER_THAN`=`">"`, `LESS_THAN`=`"<"`, + `GREATER_THAN_OR_EQUAL`=`">="`, `LESS_THAN_OR_EQUAL`=`"<="`. So legacy pushed `id = 5`. +- **connector-api** `ConnectorComparison.Operator.EQ` itself declares symbol `"="` (`:35`). +- Both the ODPS SDK and Doris' own SPI agree the symbol is `=`; only the converter's hand-written + table says `==`. This eliminates the "does ODPS tolerate `==`?" uncertainty the review flagged. + +## Decision (user, 2026-07-13) +User asked "why not do it like the old code / why didn't the old code have this?" → **restore the legacy +pattern**: map `ConnectorComparison.Operator` to the ODPS SDK `BinaryPredicate.Operator` and emit +`getDescription()`, rather than merely patching `"=="`→`"="` in the hand-written table. This fixes the +bug *and* removes the whole class of hand-typed-symbol drift (single authoritative source = the SDK). + +## Change +- Rewrite `convertComparison`'s `switch` to produce a `BinaryPredicate.Operator` (EQ→EQUALS, NE→NOT_EQUALS, + LT→LESS_THAN, LE→LESS_THAN_OR_EQUAL, GT→GREATER_THAN, GE→GREATER_THAN_OR_EQUAL); build the RawPredicate + string with `odpsOp.getDescription()`. Keep `RawPredicate` (the custom value formatting for + datetime/string quoting stays — same as legacy, which also used RawPredicate here). +- `default:` still `throw UnsupportedOperationException` → caught by `convertOne` → `NO_PREDICATE`. + This covers `EQ_FOR_NULL` (`<=>`), which has **no** ODPS `BinaryPredicate` equivalent and must not be + pushed — matching legacy's `default: odpsOp = null` (skip). BE re-filters. +- Add import `com.aliyun.odps.table.optimizer.predicate.BinaryPredicate` (ODPS SDK; the module already + imports `RawPredicate`/`Attribute`/`CompoundPredicate`/`UnaryPredicate` from the same package — no + fe-core dependency introduced). + +Only behavioral delta: `EQ` output `id == 5` → `id = 5`. NE/LT/LE/GT/GE byte-identical to today. + +## Companion (task-list): IN-direction regression test +The IN-polarity fix (`col IN (...)`, not the reversed form) has no dedicated test. Add guards so a future +regression is caught offline: +- `testEqualsEmitsSingleEqualsNotDoubleEquals` — EQ pushes `= ` and never `==` (RED on current code). +- `testAllComparisonOperatorsEmitSdkSymbols` — NE/LT/LE/GT/GE map to their SDK symbols (guards the switch). +- `testEqForNullIsNotPushedDown` — `EQ_FOR_NULL` → `NO_PREDICATE` (not pushed as `<=>`). +- `testInListEmitsColumnThenValues` / `testNotInListEmitsColumnThenNotIn` — direction guard for IN/NOT IN. + +## Iron rule +Entirely in fe-connector-maxcompute, switching on the connector-api `ConnectorComparison.Operator` and +mapping to the ODPS SDK enum. No fe-core touch, no source-name branching, no property parsing. +`bash tools/check-connector-imports.sh` must stay exit 0. + +## Adversarial self-check +- *Could NE/LT/… change?* No — `getDescription()` returns exactly the strings the arms already emit. +- *Could dropping the hand-written table lose an operator?* The SDK enum whitelists the 6 ODPS-supported + operators; `EQ_FOR_NULL` and anything else fall to `default: throw` → dropped (safe superset; BE filters). +- *Is `==` maybe secretly accepted by ODPS?* Irrelevant now — the SDK's own canonical form is `=`, and we + align to the authority; keeping `==` would be knowingly diverging from both SDK and legacy. +- *TCCL / classloader?* None — pure string building, no reflection. + +## Blast radius +Only equality predicate pushdown. No offline golden asserts the operator string (regression suites under +`external_table_p2/maxcompute/*` are live-gated, real ODPS). Perf-only correctness (restores lost pushdown). + +## e2e (live-gated) +Real ODPS `type=maxcompute` catalog: `SELECT ... WHERE k = ` — assert EXPLAIN/profile shows the predicate +pushed (not full scan) and the result set matches; `WHERE k IN (...)` / `NOT IN (...)` direction correct. +Cannot run locally (needs live ODPS credentials); registered live-gated. From e2527e43fccb8ee87f9452c65777e7a4949843be Mon Sep 17 00:00:00 2001 From: morningman Date: Mon, 13 Jul 2026 13:42:34 +0800 Subject: [PATCH 334/334] [doc](catalog) #65185 L20 done: maxcompute EQ '=' via SDK op description; series closed Mark L20 done in the reverify tracking table + rolling log and HANDOFF: the last decision-class item is resolved. The whole #65185 reverify fix series (H1-H4, M1-M8, L1-L20) is now closed (done or user sign-off accept/skip); only the D-series design debt remains (deferred to P8). All series e2e remain live-gated. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4 --- plan-doc/HANDOFF.md | 4 +++- plan-doc/task-list-65185-reverify-fixes.md | 11 ++++++++--- 2 files changed, 11 insertions(+), 4 deletions(-) diff --git a/plan-doc/HANDOFF.md b/plan-doc/HANDOFF.md index afcf70aa84e0df..ee62051b3f3177 100644 --- a/plan-doc/HANDOFF.md +++ b/plan-doc/HANDOFF.md @@ -23,7 +23,9 @@ > > **e2e 全 live-gated**(真集群):L19 非分区表 `SET TBLPROPERTIES('partition_columns'=真列名)` 不再误判为分区;L18 v3 `GEOMETRY` 列建表可加载、`SELECT geom` 报错、`SELECT 其它列` 可用;L16 无(纯注释);**L17** iceberg `t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b` 跨 ALTER schema 演进 → 抛「same table at multiple versions with different schemas not supported yet」清晰错(非 BE 崩)。 > -> **下一步**:#65185 复核系列 H/M/L 主体**已全部收口**;⏸ 决策类**仅余 `L20`**(MC EQ `==` vs `=`，须先中文讲清背景问用户,可顺带补 P4-3-IN 方向回归测试)。⏸ `M8`(升级文档)用户 07-12 跳过;`D-系列`设计债(新增 `D-MVCC-VERSION-SCHEMA`)随 P8/择机。**所有本系列 e2e 均 live-gated 待用户真集群自跑**(memory `hms-iceberg-delegation-needs-e2e`)。 +> **L20 DONE `b2786fa1200`(2026-07-13,最后一条决策类)** — maxcompute EQ 谓词下推发 Java 式 `==`(MaxCompute 无此算子→ODPS 拒解析→等值谓词静默不下推=全表扫)。**用户追问「为啥不像老代码」点中根因**:迁移前 `MaxComputeScanNode` 从不手写符号——映射到 ODPS SDK `BinaryPredicate.Operator` 枚举、符号取 `getDescription()`(EQUALS→`=`);迁移时改手写符号表,EQ 抄成 Java `==`(余五个碰巧对)。→ **恢复老代码做法**(映射 SDK 枚举 + `getDescription()`,非仅 `"=="`→`"="`),根除整类手抄漂移。**决定性静态证据**(SDK 字节码 EQUALS 描述=`=` + connector-api `ConnectorComparison.Operator.EQ` 符号=`=`)→**无需 live A/B**。`EQ_FOR_NULL`(`<=>`)无 ODPS 等价→default throw→NO_PREDICATE(同 legacy skip);NE/LT/LE/GT/GE 逐字节不变。**顺补 P4-3-IN 方向回归测试**。`MaxComputePredicateConverterTest` 26/26(21 旧+5 新:EQ 单等号 RED-able/全算子集/`EQ_FOR_NULL` 不下推/IN·NOT IN 方向)、0 checkstyle、import 门净。设计 `designs/FIX-L20-design.md`。e2e live-gated(真 ODPS `WHERE k=v`/`IN` 不退全表)。 +> +> **下一步**:**#65185 复核系列全部收口**——H1–H4、M1–M8(M8 用户 07-12 跳过·文档欠账,非代码)、L1–L20 全部 DONE 或用户签字 accept/skip;决策类(L2/L10/L12/L17/L18/L19/L20)均已中文讲清+用户签字。**余** ⚪ `D-系列`设计债(新增 `D-MVCC-VERSION-SCHEMA`)随 P8/择机。**所有本系列 e2e 均 live-gated 待用户真集群自跑**(memory `hms-iceberg-delegation-needs-e2e`)。 --- diff --git a/plan-doc/task-list-65185-reverify-fixes.md b/plan-doc/task-list-65185-reverify-fixes.md index 0460bef00eb8b6..191bdad2122419 100644 --- a/plan-doc/task-list-65185-reverify-fixes.md +++ b/plan-doc/task-list-65185-reverify-fixes.md @@ -56,7 +56,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) | 29 | **L17** | 🟡低 | fe-core/iceberg | 同表多版本 version-blind schema 绑定 | ✅ | ✅ | ✅ | ✅ 用户签字 fail-loud+TODO | | 30 | **L18** | 🟡低 | iceberg | 未知/v3 类型静默 UNSUPPORTED | ✅ | ✅ | ✅ | ✅ 用户签字 accept(DV-051) | | 31 | **L19** | 🟡低 | fe-core/iceberg | partition_columns 魔法键撞名→误判分区 | ✅ | ✅ | ✅ | ✅ | -| 32 | **L20** | 🟡低 | maxcompute | EQ 发 `==`(对齐 `=` 或 live A/B) | ⬜ | ⬜ | ⬜ | ⏸ 或 live | +| 32 | **L20** | 🟡低 | maxcompute | EQ 发 `==`→改用 SDK 算子描述发 `=` | ✅ | ✅ | ✅ | ✅ | | — | **D-系列** | ⚪设计债 | — | 见文末「设计债跟踪」(多为 P8 前置/需一次性重构) | — | — | — | ⏸ | --- @@ -68,7 +68,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **批次 2(中危·fe-core 通用节点,blast radius 较大,单测充分后再动)**：`M3`、`M1`。 - **批次 3(运营/门禁)**：`M8`(发布工具+文档,无 fe 编译)、`L1`(gate)。 - **批次 4(低危·连接器)**：`L3–L6`(trino)、`L7/L8`(kerberos)、`L9/L20`(mc)、`L11/L13/L14`(paimon)、`L18`(iceberg)。 -- **批次 5(需决策,先问用户再动)**：~~`L2`(SQL 缓存)~~ **DONE `c9a86337906` 恢复缓存**、~~`L10`(EXPLAIN 名)~~ **DONE 用户签字 accept [DV-050]**、~~`L12`(selectedPartitionNum)~~ **DONE `e5de7aedcd5` 用户签字选项 B=能力 SPI 回报真实数**;**余** `L20`(MC EQ 写法)。 +- **批次 5(需决策,先问用户再动)**：~~`L2`(SQL 缓存)~~ **DONE `c9a86337906` 恢复缓存**、~~`L10`(EXPLAIN 名)~~ **DONE 用户签字 accept [DV-050]**、~~`L12`(selectedPartitionNum)~~ **DONE `e5de7aedcd5` 用户签字选项 B=能力 SPI 回报真实数**、~~`L20`(MC EQ 写法)~~ **DONE `b2786fa1200` 恢复 SDK 算子描述发 `=`**。**批次 5 全部收口。** - **批次 6(设计债/P8)**：`D-系列`,择机或随 P8。`D-PRUNE` 因承载 H1/H3,提前到批次 0。 > 决策类(⏸)条目**先在 session 里用中文讲清背景+选项问用户**(memory `ask-user-explain-in-chinese-first`),别擅自选一路实现。 @@ -197,7 +197,7 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - [x] **L17** DONE `3627556db34`(设计 `designs/FIX-L17-design.md`;3-lens 红队 `wf_f7b69cf7-ec8` 推翻 analysis-time 方案→改 scan-node)· **用户 2026-07-13 签字:先 fail-loud + 记 TODO 重构**。红队证 analysis-time `getSchemaCacheValue` 守卫**顺序依赖+被 plain-ref/MTMV default pin 遮蔽+@incr 漏**(同一 query 抛或静默 skew 视绑定序)→改**逐引用 scan-node 守卫**:`PluginDrivenScanNode.pinMvccSnapshot` 解析版本感知 pin 后,校验每个 bound tuple 列在**本引用 pinned schema** 可解析(有 field-id 按 id、否则按 name),否则抛。确定性+完整(catch 自连接/latest-遮蔽/@incr/MTMV)+无误报(`t@old JOIN t@latest` 各 tuple 匹配自身版本→不抛)。静态可测 helper,7 RED-able UT。TODO=`D-MVCC-VERSION-SCHEMA`(逐引用版本感知 schema 绑定,仿 Trino 版本作用域 handle)。残余:嵌套 field-id-only 重编号看不见(iceberg id 稳定不触发)。 - [x] **L18** DONE `1c9c99c7767`(设计 `designs/FIX-L18-design.md`;登记 **DV-051**)· **用户 2026-07-13 签字 accept「统一映射 UNSUPPORTED」**(非抛):`IcebergTypeMapping.fromIcebergType/fromPrimitive` 两 `default` 臂保持把 Doris 无法表示的 iceberg 类型(v3 primitives TIMESTAMP_NANO/GEOMETRY/GEOGRAPHY/UNKNOWN + non-primitive VARIANT)映射 UNSUPPORTED→表能加载、该列 present-but-unqueryable、其它列可用。**背离** legacy(抛 `IllegalArgumentException`)与 Trino(抛 NOT_SUPPORTED),但用户选 graceful degradation。**无功能改动**;加澄清注释+守护测试 `unknownAndV3TypesDegradeToUnsupportedByDesign`(未来改抛→red);写方向 `toIcebergPrimitive` 仍抛(CREATE 不静默接受不可 round-trip 类型)。 - [x] **L19** DONE(初版 `01668779fd9` **已被 `2c58d8342c1` 取代**;设计 `designs/FIX-L19-design.md`〔SUPERSEDED〕+ `designs/DESIGN-reserved-connector-keys-framework.md`)· **用户否决静默 `remove`**(丢用户数据无信号、后来者加保留键无提示)→改**给所有保留控制键统一加 `__internal.` 前缀**(与既有 `show.*`/`connector.*` 同机制,集中为 `ConnectorTableSchema` 常量:`__internal.partition_columns`/`__internal.primary_keys`/`__internal.show.*`/`__internal.connector.*` + `RESERVED_CONTROL_KEYS` 集)。撞名**由构造消除**——用户裸 `partition_columns` 作普通用户属性正常透传(不删、SHOW CREATE 正常显示),永不被当控制键。**用户裁决只重命名、不加校验**(前缀够独特)。保留键全 **FE-only**(不发 BE,BE 走 `path_partition_keys`)、非持久化、SHOW CREATE 剥离→重命名零 BE/序列化/golden 影响。删三连接器 L19 `remove`;fe-core `getTableProperties` 改 `RESERVED_CONTROL_KEYS.contains`(未来加键自动剥离)。iceberg 50+hive 23+paimon 12/19/43+fe-core 8/36 全绿,0 checkstyle,import 门净。**新增 UT 证撞名消除**(裸用户键与保留键共存/透传)。e2e live-gated(非分区表设 `partition_columns` 用户属性不误判 + SHOW CREATE 显示该属性)。 -- [ ] **L20** ⏸ MC EQ `==` · `MaxComputePredicateConverter.java:145-146`:直接 `case EQ: opDesc = "=";` 对齐 SDK/legacy 消除不确定性(推荐);或 live ODPS A/B 确认容忍。顺带补 IN 方向回归测试(P4-3-IN 已修但缺测)。 +- [x] **L20** DONE `b2786fa1200`(设计 `designs/FIX-L20-design.md`)· MC EQ 发 `==`→**恢复老代码做法**:算子符号取自 ODPS SDK `BinaryPredicate.Operator.getDescription()`(EQUALS→`=`)而非手写符号表,根除手抄漂移。SDK 字节码 + connector-api `ConnectorComparison.Operator.EQ` 双证符号即 `=`,消除「ODPS 是否容忍 `==`」不确定性(无需 live A/B)。`EQ_FOR_NULL`(`<=>`)无 ODPS 等价→default throw→NO_PREDICATE(BE 重滤,同 legacy skip);NE/LT/LE/GT/GE 逐字节不变。**顺带补 P4-3-IN 方向回归测试**(`col IN (values)`)+ EQ 单等号(RED-able)/全算子集/`EQ_FOR_NULL` 守护测。26/26 绿、0 checkstyle、import 门净。e2e live-gated(真 ODPS `WHERE k=v`/`IN` 下推不退全表)。 --- @@ -331,3 +331,8 @@ Legend：⬜ todo / 🔄 in progress / ✅ done / ⏸ 挂起(需决策/live) - **L17** — 同表多版本 version-blind:**用户签字 fail-loud + TODO 重构**(`D-MVCC-VERSION-SCHEMA`)。红队证 analysis-time 守卫顺序依赖+default-pin/MTMV/@incr 遮蔽→改**逐引用 scan-node 守卫**(tuple 列须在本引用 pinned schema 可解析,有 id 按 id 否则 name)。确定性+完整+无误报。SchemaGuardTest 7/7 + MvccPin 3/3 + MvccExternalTable 59/59。 - **e2e 全 live-gated**(真集群):L19 非分区表 SET TBLPROPERTIES('partition_columns'=真列)不误判分区;L18 v3 GEOMETRY 列表可加载、该列查报错、其它列可用;L16 无(纯注释);L17 iceberg `t FOR VERSION AS OF v1 a JOIN t FOR VERSION AS OF v2 b` 跨 ALTER →抛清晰错(非 BE 崩)。 - **决策类仅余 L20**(MC EQ `==`)。设计文档 `designs/FIX-L16/L17/L18/L19-design.md`。 + +--- + +**⭐ L20 DONE `b2786fa1200`(2026-07-13,最后一条决策类收口)** — maxcompute EQ 谓词下推发 Java 式 `==`(MaxCompute 无此算子→ODPS 拒解析→等值谓词静默不下推=全表扫)。**用户追问「为啥不像老代码」点中根因**:迁移前 `MaxComputeScanNode` 从不手写符号——它映射到 ODPS SDK `BinaryPredicate.Operator` 枚举、符号取 `getDescription()`(EQUALS→`=`);迁移时改成手写符号表,EQ 顺手抄成 Java 的 `==`(其余五个碰巧对)。→ **恢复老代码做法**(映射 SDK 枚举 + `getDescription()`,非仅把 `"=="` 改 `"="`),根除整类手抄漂移;单一权威=SDK。**决定性静态证据**(SDK 字节码 EQUALS 描述=`=` + connector-api `ConnectorComparison.Operator.EQ` 符号=`=`)消除「ODPS 是否容忍 `==`」不确定性→**无需 live A/B**。`EQ_FOR_NULL`(`<=>`)无 ODPS 等价→default throw→NO_PREDICATE(BE 重滤,同 legacy skip);NE/LT/LE/GT/GE 逐字节不变。**顺补 P4-3-IN 方向回归测试**(此前修过 IN 极性但无测)。`MaxComputePredicateConverterTest` 26/26 绿(21 旧 + 5 新:EQ 单等号 RED-able、全算子集、`EQ_FOR_NULL` 不下推、IN/NOT IN 方向)、0 checkstyle、import 门净。设计 `designs/FIX-L20-design.md`。e2e live-gated(真 ODPS `WHERE k=v`/`IN` 下推不退全表)。 +- **⭐ #65185 复核修复系列全部收口**:H1–H4、M1–M8(M8 用户跳过·文档欠账)、L1–L20 全部 DONE 或用户签字 accept/skip。决策类(L2/L10/L12/L17/L18/L19/L20)均已中文讲清 + 用户签字。**余** ⚪ D-系列设计债(随 P8/择机)。**所有本系列 e2e 均 live-gated,待用户真集群回归**(memory `hms-iceberg-delegation-needs-e2e`)。