[draft] Master catalog spi 11 hive#65474
Open
morningman wants to merge 341 commits into
Open
Conversation
…sibling at getTableHandle (dormant) §4.4 S4 — the pivot that starts a foreign iceberg handle flowing out of the hive gateway. In HiveConnectorMetadata.getTableHandle, once the format detector resolves the HMS table to ICEBERG, return the embedded iceberg SIBLING connector's OWN table handle (the raw foreign iceberg handle) verbatim instead of a HiveTableHandle stamped ICEBERG. Iceberg's scan/metadata path unconditionally casts the handle to its concrete IcebergTableHandle, so a HiveTableHandle would CCE the moment iceberg cast it; the foreign handle is never cast here (its concrete type is invisible across the classloader split). This activates the S3 guard-and-forward overrides: every gateway consumer discriminates by `instanceof HiveTableHandle` (the gateway's own hive-loader type) and forwards any non-hive handle to the sibling. An empty from the sibling (table vanished / iceberg cannot load) passes through unchanged rather than fabricating a hive handle. HUDI is intentionally NOT diverted — hudi-on-HMS delegation is a later substep, so a hudi table stays on the hive-handle path (byte-unchanged). Plain-hive and view tables are untouched. Dormant until hms enters SPI_READY_TYPES: nothing calls getTableHandle for this connector today. New HiveConnectorMetadataTableHandleDivertTest pins the contract: iceberg returns the sibling's handle unmodified (not a HiveTableHandle), the empty pass-through is proven to be forwarded from the sibling, hive/hudi never consult the sibling, a missing table short-circuits to empty, and an iceberg table with no sibling configured fails loud. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…; next = S5 scan-provider gateway Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…bling scan provider (dormant) §4.4 S5 — the scan-side pair of the getTableHandle iceberg divert. HiveConnector overrides the per-table scan-provider seam Connector.getScanPlanProvider(handle): a hive handle (the gateway's own hive-loader type) runs the hive scan provider; any foreign handle (the raw iceberg handle the sibling's getTableHandle produced) is delegated to the embedded iceberg sibling's per-handle scan provider, passed through unmodified and never cast. Because the returned sibling provider is built in the iceberg plugin's classloader, PluginDrivenScanNode.onPluginClassLoader auto-pins the scan-thread TCCL to the iceberg loader for free (it keys off provider.getClass().getClassLoader()), so no pinning is needed here. A HUDI-stamped HiveTableHandle is still a HiveTableHandle, so it stays on the hive scan path (its delegation is a later substep). Dormant until hms enters SPI_READY_TYPES: nothing selects a scan provider for this connector today. New HiveConnectorScanProviderDivertTest pins the route: a foreign handle returns the sibling's provider (with the handle reaching the sibling unmodified and the sibling built once), a hive/hudi handle resolves the connector-level hive provider without building or consulting the sibling, and a foreign handle with no iceberg plugin fails loud naming the catalog. The hive/hudi cases stub the no-arg getScanPlanProvider() to isolate the routing from HmsClient construction (a real HmsClient needs a HiveConf, off the unit-test classpath; provider construction is covered by the scan-planning suites). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… next = S6 name-based divert + capability residuals Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…NO_DIVERT); read-spine S0-S6 complete; next = write delegation
A code-grounded recon (wf_6ac0e047-768: 4 dimension readers + a
completeness/correctness critic, all HEAD-verified) proves the §4.4 S6 sketch
("re-run detect(db,table) and forward the view family + buildTableDescriptor for
ICEBERG") is unnecessary or actively harmful, so S6 ships ZERO connector code:
- buildTableDescriptor: byte-identical output (the sibling is always hms flavor
-> same HIVE_TABLE + THiveTable; legacy iceberg-on-HMS also emitted HIVE_TABLE),
so a divert only adds a wasteful getTable(detect).
- view family (viewExists/getViewDefinition/dropView): legacy-parity for the only
reachable objects (iceberg/hudi TABLES return viewExists=false and stay on the
getTableHandle divert; an iceberg-on-HMS VIEW stores hive-dialect SQL in HMS view
text and is served as a hive view) AND undiscriminable name-only (detect returns
UNKNOWN for a view: table_type='iceberg-view' != 'ICEBERG').
- listViewNames stays un-overridden (a divert would double-list views); getTableComment
stays SPI-default "" (legacy getComment() returned "" unconditionally -> a divert
would surface a comment legacy never showed).
- db/catalog methods are format-agnostic.
=> The §4.4 read-delegation spine (S0-S6) is COMPLETE. Remaining per-table
divergences are all write/DDL residuals (dropDatabase-force purge, createTable
engine-routing, per-handle getWritePlanProvider/getProcedureOps) or connector-wide
capability residuals (SUPPORTS_COLUMN_AUTO_ANALYZE hudi over-admission already
documented; supportsLatestSnapshotPreload under-admission) plus the explicit
non-goal of first-class iceberg-on-HMS views — all recorded for the flip in the
new findings doc. Dormant until hms enters SPI_READY_TYPES.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…DL + write-validation to sibling (dormant)
§4.4 write-delegation W1 (dormant): extend the S3 handle-guard-forward on
HiveConnectorMetadata to the 16 remaining handle-carrying mutating/validation
methods so an iceberg-on-HMS table's ALTER-DDL and write-time validation route to
the embedded iceberg sibling, while plain-hive stays byte-identical.
- 14 ALTER-DDL mutators (renameTable + add/drop/rename/modify/reorder column,
create/drop branch & tag, add/drop/replace partition-field): foreign handle ->
siblingMetadata().<same>(...); hive handle reproduces the EXACT inherited
SPI-default throw ("... not supported"). Foreign handle is never cast.
- 2 write validators (validateRowLevelDmlMode / validateStaticPartitionColumns):
foreign handle -> sibling (iceberg's real write-mode / static-partition
rejections apply); hive handle returns SILENTLY (SPI-default no-op) -- a throw
there would newly reject legal plain-hive row-level DML / static-partition
INSERTs.
Zero new SPI (all 16 already carry a ConnectorTableHandle) and zero fe-core change
(ALTER call sites are already handle-based via PluginDrivenExternalCatalog and
getTableHandle already S4-diverts iceberg to the sibling). Dormant until "hms"
enters SPI_READY_TYPES. Per user sign-off, iceberg-on-HMS gains full native-iceberg
write/ALTER capability (an intended upgrade over legacy's read-only reject).
Tests: HiveConnectorMetadataSiblingDelegationTest +2 (foreign-handle completeness
lock over all 16 + hive-handle throw/no-op + never-consult-sibling). Full module
209 tests green, checkstyle 0, connector-import gate clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…on doc W1-W6 + capability upgrade signed; next = W2 - Add hms-write-delegation-decomposition-2026-07-08.md: authoritative W1-W6 plan from recon wf_f508ac0e-8ec (3 divert shapes; user-signed iceberg-on-HMS capability UPGRADE to full native-iceberg; e2e requirement; flip residuals incl. hudi-on-HMS write flip-blocker). - HANDOFF: W1 landed (e7a96f439e7); next = W2 (getWritePlanProvider(handle) overload + gateway divert + fe-core provider-fetch conversion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…iverts iceberg-on-HMS to sibling (dormant) §4.4 write-delegation W2 (dormant): add a per-handle write-provider seam mirroring the shipped getScanPlanProvider(handle). - Connector: new default getWritePlanProvider(ConnectorTableHandle) delegating to the connector-level getWritePlanProvider() -> behavior-preserving for every single-format connector (jdbc/es/trino/max_compute/paimon/iceberg). - HiveConnector overrides it: a HiveTableHandle uses the hive write provider; a foreign (iceberg-on-HMS) handle is delegated to getOrCreateIcebergSibling() .getWritePlanProvider(handle), passed through UNMODIFIED and NEVER cast. HUDI keeps a HiveTableHandle -> stays on hive (its write delegation is a later step). - fe-core: the 4 provider-fetch sites now pass the resolved table handle (PhysicalPlanTranslator DELETE/MERGE + INSERT, PhysicalIcebergMergeSink, PluginDrivenExternalTable.fetchSyntheticWriteColumns) by hoisting the already-adjacent getTableHandle above the provider fetch. Connector-agnostic (a per-handle overload call), no if(format)/instanceof-HMSExternal branch. The write-admission methods (supportedWriteOperations/requires*) stay connector-level (W3); beginTransaction threading is W4; the write-path TCCL pin is W6. Dormant until "hms" enters SPI_READY_TYPES. Tests: new HiveConnectorWriteProvider DivertTest (4: foreign->sibling unmodified + build-once, hive/hudi->hive no sibling, fail-loud when plugin absent); 4 fe-core mock tests stub the per-handle overload (a plain Mockito mock does not run the interface default). fe-connector-hive 213 + fe-core touched suites green, checkstyle 0 (api/hive/fe-core), import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…per-handle write-admission) W2 landed (84890fe48d3): getWritePlanProvider(handle) overload + HiveConnector divert + 4 fe-core provider-fetch conversions; adversarial review caught + fixed a blocker (4 fe-core mock tests must stub the per-handle overload). Next = W3: the 7 write-admission per-handle overloads (default-derived from getWritePlanProvider( handle)) + fe-core admission-site handle resolution. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…gets iceberg write ops/branch/partition semantics (dormant)
§4.4 write-delegation W3 (dormant): make the write-ADMISSION capabilities
per-handle so a flipped hms gateway admits the operations of the table's actual
provider (iceberg for iceberg-on-HMS, hive for hive), not the connector-level hive
answer.
Only 4 of the 7 write capabilities differ between the hive and iceberg providers
(HEAD-verified): supportedWriteOperations (hive {INSERT,OVERWRITE} vs iceberg
+DELETE,MERGE,REWRITE), supportsWriteBranch (false vs true), requiresPartitionHash
Write (true vs false), requiresMaterializeStaticPartitionValues (false vs true).
The other 3 (requiresParallelWrite, requiresFullSchemaWriteOrder, requiresPartition
LocalSort) are byte-inert for the gateway (hive == iceberg), so they stay no-arg.
- Connector: 4 new per-handle overloads, each default-deriving from
getWritePlanProvider(handle) exactly as the no-arg views derive from
getWritePlanProvider() -> behavior-preserving for every single-format connector,
and (via W2's HiveConnector divert) auto-routing per-table with NO extra
HiveConnector override.
- fe-core: PhysicalPlanTranslator DELETE/MERGE + INSERT gates resolve the handle
first and call supportedWriteOperations(handle); PluginDrivenExternalTable adds
connectorSupportedWriteOperations()/connectorSupportsWriteBranch() helpers
(resolve the handle, call the per-handle overload, DEGRADE to empty/false on a
null connector / unresolvable handle -> never a null handle to the gateway) and
makes requirePartitionHashOnWrite()/materializeStaticPartitionValues() per-handle;
RowLevelDmlRowIdUtils, IcebergRowLevelDmlTransform, InsertInto/OverwriteTable
Command route through the table helpers. Connector-agnostic, no if(format) branch.
Dormant until "hms" enters SPI_READY_TYPES. Adversarial review: no blocker/major;
fixed a neutered degrade test + added null-connector / unresolvable-handle coverage
for all 4 differing capabilities. fe-core touched suites green (incl. Plugin
DrivenExternalTableTest 28), checkstyle 0 (api + fe-core).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…txn per-handle) or W5 (procedures) W3 landed (d1df2e0f7df): write-admission capabilities per-handle (only the 4 differing methods; 3 inert stay no-arg). Records the W3 review residuals (inert-3 rely on hive==iceberg; new getTableHandle round-trip on the live iceberg path). Next = W4 (beginTransaction(session,handle) guard-forward) or W5 (getProcedureOps( handle) + ConnectorExecuteAction reorder), both independently committable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…-HMS write/EXECUTE binds the iceberg txn (dormant) W4 of the §4.4 write delegation. The plugin-driven insert executor and the EXECUTE rewrite driver open one connector transaction unconditionally, so a flipped iceberg-on-HMS write would bind a HiveConnectorTransaction that iceberg's write plan downcasts to IcebergConnectorTransaction -> ClassCastException. Select the transaction per-handle so the gateway opens the sibling's transaction for a foreign table (the one non-precedented shape in the write delegation: beginTransaction is inherently handle-less). - ConnectorWriteOps: new default beginTransaction(session, handle) delegating to the no-arg version (mirrors getWritePlanProvider(handle)); the 6 shipped connectors are unchanged (they inherit the default -> their existing no-arg beginTransaction). - HiveConnectorMetadata: override guard-forwards a foreign (non-HiveTableHandle) handle to the sibling (-> IcebergConnectorTransaction); a hive handle falls through to the connector's own HiveConnectorTransaction. The selection MUST be symmetric -- the hive and iceberg write plans downcast the session-bound transaction to different concrete types, so an always-forward (or always-hive) shortcut breaks the opposite side. - fe-core threads the resolved write-target handle into BOTH txn-open sites (else the EXECUTE path keeps a live CCE): PluginDrivenInsertExecutor (the single funnel for INSERT/OVERWRITE/DELETE/MERGE) via a new fail-loud PluginDrivenExternalTable.resolveWriteTargetHandle(), and ConnectorRewriteDriver (EXECUTE, already holds the handle). Connector-agnostic; a null handle is never handed to the gateway (fail-loud, so it cannot misroute a plain write to the sibling). Dormant until hms enters SPI_READY_TYPES. Code-grounded recon + 4-dimension adversarial review + verify pass (CCE clean, dormancy/shipped-parity clean, routing tests mutation-tight); the two verified findings were test-coverage gaps, both closed here: a fail-loud-body test in PluginDrivenExternalTableTest (an orElse(null) downgrade now fails) and a handle-threading assertSame in PluginDrivenInsertExecutorTest (a null 2nd-arg mutant now fails). Residuals (documented, not built here): W6 write-path TCCL pin (opening the txn needs none -- IcebergConnectorTransaction ctor defers SDK/reflection to beginWrite/commit; the commit-path pin is W6); e2e regression for iceberg-on-HMS INSERT/DELETE/MERGE/EXECUTE owed at flip; hudi-on-HMS keeps a HiveTableHandle -> hive txn (later substep, not made worse); once-per-statement getTableHandle at txn-open (cache-backed for paimon/iceberg, uncached for jdbc/maxcompute) consciously accepted; fail-loud re-query slightly widens the failure window on a concurrent DROP / transient metadata error between planning and txn-open. All green: fe-connector-hive 7 (+2 begin-txn divert) + fe-core PluginDrivenInsertExecutorTest 8 / PluginDrivenExternalTableTest 30 (+2) / ConnectorRewriteDriverTest 3; checkstyle 0 (api/hive/fe-core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…procedures per-handle) W4 (transaction per-handle) landed as 5a41e4cc562. Records the change, the two review-found test-coverage gaps closed, and the residuals (W6 pin, e2e, hudi-on-HMS, once-per-statement getTableHandle, fail-loud window). Remaining dormant write substep is W5 (procedures); W6 is flip-time verification. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…-HMS gains EXECUTE via sibling (dormant) W5 of the §4.4 write/procedure/DDL delegation. Procedure-ops selection becomes per-handle, so a flipped hms gateway serves ALTER TABLE ... EXECUTE on its iceberg-on-HMS tables from the embedded iceberg sibling, while plain-hive/hudi (no procedures) stay fail-loud rejected. - Connector: new default getProcedureOps(ConnectorTableHandle) delegating to the no-arg version, mirroring the getScanPlanProvider(handle)/getWritePlanProvider(handle) seams (single-format connectors unaffected). - HiveConnector: overrides it -- a HiveTableHandle inherits the connector-level null (plain-hive/hudi have no procedures); any foreign iceberg-on-HMS handle is delegated to the iceberg sibling, passed through UNMODIFIED and NEVER cast (a cross-loader cast would CCE). - fe-core ConnectorExecuteAction: hoists the session + table-handle resolution above the getProcedureOps null-check and calls the per-handle overload (the no-arg version is null for a gateway). A bad table name now surfaces "Table not found" before "does not support EXECUTE" (documented reorder; no test pins the old order). Stale class javadoc (adapter "never constructed pre-flip") corrected -- iceberg/paimon are already live. - ExecuteActionFactory.getSupportedActions stays on the no-arg getter (no live caller; handle-aware variant deferred). Net-new capability (legacy rejected iceberg-on-HMS EXECUTE), signed off 2026-07-08; e2e owed at the flip. Dormant until hms enters SPI_READY_TYPES. Recon-grounded against HEAD + 4-dimension adversarial review (routing/precedence/CCE/completeness -> per-finding verify): 0 blocker/major/minor, 1 nit (the stale javadoc, fixed). Tests: api 3 per-handle defaults + hive 4 divert (foreign/hive/hudi/plugin-absent) + fe-core per-handle selection. Green: api 10 / hive 219 / fe-core ConnectorExecuteActionTest 18 + ConnectorRewriteDriverTest 3; checkstyle 0 (api/hive/fe-core); import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…delegation spine (S0-S6 + W1-W5) complete W5 (procedures per-handle) landed as 8aee62bab20. Records the W5 residuals (paimon extra getTableHandle round-trip, getSupportedActions deferred, e2e owed) and marks the whole §4.4 read+write+procedure+DDL delegation spine done. Next = remaining pre-flip dormant items (candidates: §4.5 write-chain, hudi-on-HMS write-reject, §4.3 getNewestUpdateVersionOrTime[blocked on D2 cache]); W6 TCCL pin is flip-time. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…t sub-steps (WC1-WC4) + flip residuals Code-grounded recon (3 dimension readers + completeness/ordering critics, all HEAD-verified) of the remaining pre-flip write-chain items. Key output: a three-way dormancy taxonomy (A=unreachable, B=value-identical, C=live-with-no-op-default) and a risk-ascending order -- WC1 relocate BIND_BROKER_NAME (B), WC2 engine-map hms cases (A), WC3 partition-spec INSERT reject (C, needs live-connector no-regression test), WC4 read-ACID query-finish commit wiring (C, the hard flip prerequisite). Items 2b/3 carry no dormant work (flip-time deletes R1/R2). Authoritative plan for the write-chain build-out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…_NAME to BrokerProperties (single source) Value-identical relocation (both literals are "broker.name"). The generic ExternalCatalog base no longer depends on the HMS subclass for the broker-name key: - BrokerProperties.BIND_BROKER_NAME_KEY: private -> public (now the single source of truth, read by both guessIsMe case-insensitively and ExternalCatalog.bindBrokerName case-sensitively). - ExternalCatalog.bindBrokerName(): reads BrokerProperties.BIND_BROKER_NAME_KEY instead of HMSExternalCatalog.BIND_BROKER_NAME; drop the now-unused HMSExternalCatalog import, add BrokerProperties. - HMSExternalCatalog.BIND_BROKER_NAME: deleted (verified single external reference at HEAD). Lookup semantics kept byte-identical (case-sensitive Map.get) -- deliberately NOT harmonized to case-insensitive, which would silently change the resolved broker name for a differently-cased key. Iron-rule cleanup: removes a fe-core base-class -> HMS-subclass constant dependency. Dormant/no-op (value provably unchanged for every catalog). Test: BrokerPropertiesTest pins BIND_BROKER_NAME_KEY == "broker.name" (the shared-key regression surface). Green: fe-core BrokerPropertiesTest 6 + HiveScanNodeTest 6; checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…IND_BROKER_NAME relocation) DONE; next = WC2 Records the write-chain decomposition doc and WC1 (6c01805109c). Sets the risk-ascending path WC2 (engine-map hms cases, unreachable) -> WC3 (partition-spec reject, needs live-connector no-regression test) -> WC4 (read-ACID query-finish commit wiring, the hard flip prerequisite), with the flip-time residuals (R1-R4) and the post- write-chain candidates (hudi write-reject, getNewestUpdateVersionOrTime[blocked on D2]) spelled out. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ation (no-regression); hudi now gates the flip Code-grounded recon (4 readers + completeness/ordering critics, all HEAD-verified) turning the three signed decisions (sibling delegation like iceberg / no incremental+time-travel+MVCC regression / plan-first) into an ordered, HEAD-grounded plan (HD-A0..HD-D1, groups A-D). Key results: - The flip is atomic: adding "hms" to SPI_READY_TYPES converts hive + iceberg-on-HMS + hudi-on-HMS at once, so hudi's no-regression build-out is a HARD flip-blocker (the iceberg spine alone is not enough). - PhysicalHudiScan wrinkle RESOLVED: a flipped hudi table is a generic PluginDrivenExternalTable -> LogicalFileScan -> PluginDrivenScanNode (same path as iceberg); incremental/time-travel re-home connector-side via the EXISTING resolveTimeTravel/applySnapshot/INCREMENTAL seam (paimon template) -- NO new fe-core SPI, no source-specific BindRelation/CheckPolicy arm (iron rule). - DV-005 model mismatch resolved: getType()="hudi" is a sibling-only type string (createSiblingConnector bypasses SPI_READY_TYPES); never promote hudi into SPI_READY_TYPES. - ONE new design decision surfaced for sign-off (HD-B1): with two siblings the binary handle discriminator breaks; recommend a neutral Connector.ownsHandle(handle) SPI predicate (dormant-unit-testable) over classloader-identity routing. It mutates the already-landed iceberg spine -- must stay byte-behavior-identical on the iceberg arm. - BIGGEST risk (HD-C3, possibly unfixable under the iron rules): COW-incremental row-level _hoodie_commit_time window filter is skipped on the generic path; needs BE-reader verification (BE tree absent here) + a straddling-base-file e2e fixture. Also surfaced a real snapshot-path bug (HD-A3 partition-value parse infidelity -> silent NULL partition columns on non-hive-style partitioned reads). No code (plan-first per sign-off). HANDOFF updated: hudi elevated from a write-reject footnote to a first-class flip-blocking work-stream. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ATE + display (dormant) Adds three switch cases so a flipped HMS external catalog (a PluginDrivenExternalCatalog with getType()=="hms") resolves the right engine names. All strictly unreachable pre-flip (Class A): "hms" is not in CatalogFactory.SPI_READY_TYPES, so a type=hms catalog is still a legacy HMSExternalCatalog / HMSExternalTable and none of these cases is entered until the flip. - CreateTableInfo.pluginCatalogTypeToEngine: case "hms" -> ENGINE_HIVE (CREATE engine; mirrors legacy paddingEngineName/checkEngineWithCatalog instanceof HMSExternalCatalog arm). hive != iceberg, so the getEffectiveIcebergFormatVersion arm stays inert for hms. - PluginDrivenExternalTable.getEngine(): case "hms" -> HMS_EXTERNAL_TABLE.toEngineName() == "hms" (DISPLAY engine; legacy HMSExternalTable showed "hms", NOT "hive" -- distinct from the CREATE engine). Falling through to "Plugin" would regress SHOW TABLE STATUS / information_schema.tables. - PluginDrivenExternalTable.getEngineTableTypeName(): case "hms" -> HMS_EXTERNAL_TABLE.name() (keeps the pluginCatalogTypeToEngine sync invariant documented at CreateTableInfo.java). Dormant unit tests (mock getType()=="hms", no actual flip): CreateTableInfoEngineCatalogTest +5 (no-ENGINE pads hive, CTAS pads hive, wrong explicit engine rejected, hive accepted, iceberg-format non-interference lock); PluginDrivenExternalTableEngineTest +2 (engine "hms" and type name HMS_EXTERNAL_TABLE). fe-core BUILD SUCCESS; 31 tests pass (0 fail); checkstyle 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… WC3 (explicit-partition INSERT reject) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…n-spec INSERT via neutral SPI (dormant) Ports the legacy hive reject of INSERT INTO hive_tbl PARTITION(p1, p2) (the dynamic partition-NAME list form) into the connector-delegated write path. Legacy fe-core threw "Not support insert with partition spec in hive catalog." in BindSink.bindHiveTableSink; post-flip a hive write binds through bindConnectorTableSink, which never read sink.getPartitions() -- so the reject would be silently lost (a fail-loud regression). This wires it through a neutral SPI so the rejection + message stay in the connector. - fe-connector-api ConnectorWriteOps: new default no-op SPI validateWritePartitionNames(session, handle, partitionNames), mirroring the validateStaticPartitionColumns permissive-default pattern. - fe-connector-hive HiveConnectorMetadata: @OverRide -- foreign (iceberg-on-HMS) handle forwards to the sibling REGARDLESS of emptiness (accepts PARTITION(names) as a standalone type=iceberg catalog does, no heterogeneous-vs-standalone divergence); a hive handle THROWS the exact legacy message on a non-empty list, returns silently on empty. Foreign handle never cast. - fe-core BindSink: new private checkConnectorWritePartitionNames mirroring checkConnectorStaticPartitions (empty-guard -> resolve handle -> SPI -> DorisConnectorException mapped to AnalysisException), called in bindConnectorTableSink with sink.getPartitions(). Keyed on the dynamic partition-NAME list, NOT staticPartitionKeyValues -- a static PARTITION(col='val') INSERT and a plain INSERT ... SELECT are unaffected. Class C (live wiring, permissive no-op default): the fe-core call runs today on live iceberg/paimon/max_compute/jdbc/es, but the empty-guard makes plain INSERT byte-unchanged and no live connector overrides the SPI (all inherit the no-op), so a non-empty PARTITION(names) on them stays silently accepted as before. The hive throw is dormant until hms enters SPI_READY_TYPES. The flip-time delete of BindSink:667-669 (with the rest of bindHiveTableSink) is a residual, not part of this step. Message byte-parity with the pre-existing e2e test_hive_write_type.groovy:250, which runs on both the legacy and the flipped connector path. Coverage mirrors the peer static-partition seam (connector unit test + e2e; no fe-core unit test precedent for bindConnectorTableSink): HiveConnectorMetadataSiblingDelegationTest +1 (hive rejects non-empty with the legacy message, empty is silent, never consults the sibling) and the EXPECTED_WRITE_METHODS Rule-9 completeness lock extended with validateWritePartitionNames (foreign forward proven regardless of emptiness). Adversarial 4-lens review (routing / message-parity / live-no-regression + dormancy / field-keying + binding-completeness) with a refute pass: 0 confirmed findings. api+hive+core BUILD SUCCESS; sibling test 8/8; checkstyle 0 (api/hive/core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… WC4 (read-ACID query-finish commit) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ish commit via neutral SPI (dormant) A transactional (full-ACID / insert-only) hive read opens a metastore read transaction + shared read lock during planScan (HiveScanPlanProvider.planAcidScan -> readTxnManager.register -> openTxn + acquireSharedLock). The matching release (HiveReadTransactionManager.deregister -> commitTxn) had ZERO callers, so a flipped ACID read would leak the metastore shared read lock for the metastore's lifetime. WC4 wires the release through the generic query-finish spine, so fe-core never names hive. - fe-connector-api ConnectorScanPlanProvider: new default no-op releaseReadTransaction(queryId), matching the getDeleteFiles/classifyColumn no-op-default precedent. - fe-connector-hive HiveScanPlanProvider: @OverRide -> readTxnManager .deregister(queryId). The manager is the SAME per-connector instance HiveConnector injects into register and every provider, so the keys match; deregister commits exactly once, is idempotent, and swallows a commit failure (best-effort). - fe-core PluginDrivenScanNode.getSplits(): UNCONDITIONALLY register a query-finish callback (QeProcessorImpl.registerQueryFinishCallback) right after the scan-provider null-guard -- BEFORE the pruned-to-zero short-circuit and planScan -- so a planScan that opens the txn then throws still has its release in place. queryId is computed ONCE (connectorSession.getQueryId()) == the runAndClear key (DebugUtil.printId) == the connector txnMap key. - fe-core buildReadTransactionReleaseCallback: extracted public-static helper returning the Runnable that pins the provider's plugin classloader (onPluginClassLoader) around the release -- load-bearing, since the callback runs on the StmtExecutor thread (app-loader TCCL) and commitTxn resolves metastore/thrift classes by name. Extracted so the release + TCCL-pin is unit-testable without driving getSplits (mirrors applyMvccSnapshotPin). Class C (live wiring, no-op default): the registration runs today on every live plugin scan (es/jdbc/paimon/max_compute/iceberg); only hive overrides the SPI, so the callback is inert for them (pins TCCL, calls the no-op default, never throws) and runAndClear clears it at query finish. Hive's release is dormant until hms enters SPI_READY_TYPES; the legacy HiveScanNode + Env.hiveTransactionMgr path is untouched (its delete + a live-ACID e2e read are flip-time residuals). Foreign (iceberg-on-HMS) handles route to the iceberg sibling provider whose releaseReadTransaction is the no-op default (no hive read txn opened). Tests: HiveReadTransactionTest +1 (provider.releaseReadTransaction commits via the shared manager exactly once, idempotent repeat, no exception on commit -- swallowed); new PluginDrivenScanNodeReadTxnReleaseTest (callback releases for the query; pins the provider classloader DURING the release against a distinct sentinel TCCL and restores it after). Existing scan-node suites unaffected (none drive getSplits end-to-end). Adversarial 5-lens review (placement/dominance, TCCL pin, queryId-identity + registry, live-no-regression + dormancy, connector-release/idempotency) with a refute pass: 0 confirmed findings. api+hive+core BUILD SUCCESS; HiveReadTransactionTest 4/4; all 16 fe-core scan-node classes 78/78; checkstyle 0 (api/hive/core); connector import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ormant WC1-WC4 all complete; next thread = hudi flip-blocker Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…ling-only, never a standalone catalog type Correct the misleading "dedicated Hudi catalogs" framing that invites a future maintainer to "fix" the standalone-type model mismatch by promoting "hudi" into SPI_READY_TYPES: - HudiConnectorProvider javadoc + getType(): "hudi" is a SIBLING-ONLY type string resolved via createSiblingConnector; not a user-facing catalog type; never add to SPI_READY_TYPES / a factory case. - HudiConnector javadoc: built only as an embedded sibling of the hive hms gateway. - CatalogFactory.SPI_READY_TYPES: replace the stale "hms, hudi still use built-in ExternalCatalog" comment (there is NO HudiExternalCatalog) with a guard — hms uses built-in HMSExternalCatalog until the cutover, hudi is parasitic on HMS and served as an hms-gateway sibling, never added here. Comment/javadoc only; zero logic change. Dormant (hms not in SPI_READY_TYPES). checkstyle 0 (hudi + fe-core). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…y synthesis (mirror iceberg S1/S2, dormant)
Give the hive hms gateway a second embedded SIBLING connector for hudi-on-HMS, mirroring the reviewed iceberg
sibling holder. Dormant: no production path references it until the getTableHandle HUDI divert lands (a later
substep).
- New HudiSiblingProperties.synthesize(gatewayProps): a verbatim defensive copy of the gateway catalog's whole
property map. Unlike IcebergSiblingProperties it injects NO flavor key — hudi has no iceberg.catalog.type
analogue; HudiConnector.createClient reads hive.metastore.uris/uri + raw hadoop.*/fs.*/dfs.*/hive.*/s3.*
straight from the map. Whole-map copy avoids silently dropping a connectivity key.
- HiveConnector: add HUDI_CONNECTOR_TYPE="hudi" literal + volatile hudiSibling field + getOrCreateHudiSibling()
(lazy volatile double-checked build via context.createSiblingConnector("hudi", synthesize(props)); fail-loud
naming the catalog when the plugin is absent; failure NOT memoized; held ONLY as the parent-first Connector
interface, never cast — its concrete type is invisible across the loader split). close() forwards to the hudi
sibling too (gateway owns both siblings' lifecycle; each null-guarded, cleared, no-op when never built).
Tests: HudiSiblingPropertiesTest (3: verbatim carry, no-flavor-injected, defensive copy) + HiveConnectorSiblingTest
hudi cases (build/memoize/fail-loud-not-memoized/close-forward) + a closeForwardsToBothSiblingsIndependently
regression guard (adding the hudi arm must not drop/double the iceberg arm). fe-connector-hive 230 tests green,
checkstyle 0, import gate net. 4-dimension adversarial review (holder-parity/close-lifecycle/dormancy-synthesis/
test-intent) + verify: 0 findings.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…2-A5) + pending HD-B1 sign-off Record the first two hudi flip-blocker substeps as complete (37b85348667 DV-005 guardrail; 28d4781ccac hudi sibling holder + property synthesis). Next thread = rest of Group A (Kerberos, partition-value fidelity FIX, force-JNI, BE descriptor), plus HD-B1 (3-way foreign-handle routing) as the one decision needing user sign-off. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…nsHandle neutral SPI) Record the user's 2026-07-08 sign-off on the one new design decision in the hudi work line: the 3-way foreign-handle routing uses a neutral parent-first Connector.ownsHandle(handle) SPI default (each sibling self-identifies its own handle type in its own classloader), chosen over classloader-identity routing because the routing logic is same-loader unit-testable. Implementation must keep the already-landed iceberg delegation spine byte-behavior-identical. Marked in HANDOFF + the authoritative hudi plan doc. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
… (snapshot-path no-regression FIX, dormant) Replace HudiScanPlanProvider.parsePartitionValues (naive split-on-'=') with a byte-faithful port of legacy HudiPartitionUtils.parsePartitionValues, fixing a real snapshot-path correctness regression: Hudi's DEFAULT non-hive-style layout yields relative paths like "2024/01" (no "col=" prefix); the old logic silently DROPPED every prefix-less fragment, so a partitioned hudi table read NULL partition columns on a plain snapshot read, and escaped values (%20, %2F) arrived un-decoded. - Positional mapping: a prefix-less fragment maps to the i-th partition column; a "col="-prefixed fragment contributes its suffix (legacy decides per fragment). - Single-column whole-path fallback when the fragment count mismatches; > 1 column with a mismatch fails loud with the exact legacy message. - Every value URL-unescaped via an inlined unescapePathName (byte-faithful port of Hive FileUtils.unescapePathName, identical to the fe-connector-hive HiveWriteUtils copy) so the connector needs no hive-common dependency. - Method made static + package-private for direct unit testing (no live HoodieTableMetaClient needed); produces a column->value map (the existing consumer shape), not legacy's positional list. Scope: this closes the value-parse regression on the UNPRUNED / most-basic read (getAllPartitionPaths returns Hudi's own relative path = the FileSystemView shape). The PRUNED path (applyFilter) still feeds HMS hive-style partition NAMES, correct for hive-sync'd tables; making the pruned partition SOURCE useHiveSyncPartition-aware for non-hive-style tables belongs to the partition-listing step (which ports that source once) and is likewise closed before the catalog flip — recorded, not silently folded in. Dormant (hms not in SPI_READY_TYPES). New HudiPartitionValuesTest (8: positional / hive-style / mixed / unescape / single-col fallback / prefix-strip / multi-col fail-loud / unpartitioned). fe-connector-hudi 41 + hms 56 + api 61 green, checkstyle 0, import gate net. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rqz6wNwHj3oYXsLLa4TLwx
…), M8 skipped per user - Land FIX-L1-design.md (design + red-team wf_643c11b4-3fe folded, incl. the 4th hole the reverify's "three holes" missed: path-based whitelist suppression). - Mark L1 DONE (88aa55b), M8 deferred at user's 2026-07-12 request (recon preserved in the tracking table — not silently dropped). - Repoint HANDOFF next-step to batch 4 (low-risk connector L-items). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…e leak) TrinoConnectorDorisMetadata's 6 FE-only metadata methods (listDatabaseNames, listTableNames, getTableHandle, getTableSchema, applyFilter, applyProjection) each open a Trino read-only transaction via beginTransaction(READ_UNCOMMITTED, true, true) and never end it. The Trino Connector contract requires exactly one of commit/rollback to end a transaction (even in auto-commit mode), so each metadata call leaked a transaction handle into the connector's transaction manager — accumulating over every plan. (Parity with master; the SPI cutover amplified the volume for trino-connector catalogs.) Wrap each site in try/finally and release via a small releaseQuietly helper that commits and swallows+logs any release failure so it cannot mask the real exception a finally runs after. A read-only READ_UNCOMMITTED transaction has nothing to write, so commit simply frees it. The try starts after beginTransaction, preserving the conditional-open in applyFilter/applyProjection (their early returns open no transaction). The scan site (TrinoScanPlanProvider.planScan) is deliberately left untouched: it serializes the transaction handle to BE for the JNI scanner, so that transaction must stay open. The returned table/column handles are immutable opaque value objects reused later under the scan's own fresh transaction, so committing the metadata transaction does not invalidate them. No behavioral UT: driving these methods requires constructing io.trino.Session (a heavyweight concrete class with no existing test fixture in this module); fabricating the full Trino SPI fakes is disproportionate for this low-severity parity leak. Verified by compile (commit signature, 0 checkstyle), the import gate (exit 0), the existing trino UTs, and an adversarial design review; behavioral validation is e2e live-gated (needs a live trino cluster). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
FE UT Coverage ReportIncrement line coverage |
…Set parity) listTableNames mapped listTables() to names and collected straight to a list, dropping the legacy LinkedHashSet de-duplication. Some Trino connectors list the same table name more than once for a schema (tables+views, multi-source merges). Add an order-preserving .distinct() to restore parity. The legacy prefix filter is intentionally not restored (listTables already filters by schema). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
TrinoDorisConnector.ensureInitialized() uses `trinoConnector != null` as the double-checked-lock guard, and readers (getMetadata etc.) then dereference trinoSession / trinoCatalogHandle. doInitialize() assigned the guard field FIRST and its dependencies after, so a concurrent reader could observe `trinoConnector != null` while trinoSession was still null -> transient NPE. Even though every field is volatile, the volatile happens-before only guarantees writes made BEFORE the guard write are visible; the dependency writes came after. Assign trinoConnector LAST (after trinoSession/trinoCatalogHandle/ trinoConnectorName). Now observing the guard non-null guarantees the fully published dependencies. Classic safe-publication; all fields already volatile so the reorder alone suffices. Single-thread semantics unchanged. Verified by an adversarial concurrency review (JMM ordering + no reader bypasses the guard). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
TrinoBootstrap is a process-wide singleton loaded once with the first catalog's plugin dir. getInstance(pluginDir) was first-wins: a later catalog passing a DIFFERENT trino.plugin.dir was silently ignored and reused the first dir's plugins, so it could fail to find its connector factory or use the wrong plugins. Retain the plugin dir on the singleton and throw IllegalStateException when a later getInstance() requests a different one. Paths are canonicalized best-effort (getCanonicalPath, falling back to the raw string) so two spellings of the same physical directory (trailing slash, relative vs absolute, symlink) are treated as equal and do not trip a spurious mismatch. First-wins init behavior is unchanged; only the silent-wrong-dir case becomes a loud error. Verified by an adversarial concurrency review (write-once monotonic instance, null-safe compare). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…L4/L5/L6), mark done Design docs for the trino low-risk cluster (transaction release, plugin-dir fail-loud, listTableNames de-dup, guard-field safe publication), each with its adversarial review folded in. Mark L3/L4/L5/L6 DONE in the tracking table and repoint HANDOFF next-step to the kerberos cluster (L7/L8). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
HadoopAuthenticator.doAs caught InterruptedException and rethrew it as a checked IOException without restoring the thread's interrupt status, so callers up the stack could no longer observe that the thread was interrupted (breaks cooperative cancellation). Call Thread.currentThread().interrupt() before wrapping, per standard practice for swallowing InterruptedException. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
initializeAuthConfig() unconditionally called UserGroupInformation .setConfiguration(hadoopConf), which mutates a single process-wide global. Because it runs from login() on both initial login AND every ticket refresh, every kerberos catalog's login/refresh overwrote the global auth config (last-writer-wins), so HDFS catalogs with differing hadoop.security.authentication could not coexist and refreshes churned the global. The consolidation in apache#64655 (f7992b0) dropped the first-writer-wins guard master had. Restore first-writer-wins: only the first caller publishes the global config; a static field remembers the first auth method; later callers (and every refresh, which re-enters via login()) skip, and a WARN fires only on a genuine auth-method mismatch. Compare against the remembered method rather than master's UserGroupInformation.getLoginUser().getAuthenticationMethod() on purpose: getLoginUser() would establish a process-wide login user, which Doris deliberately avoids (per-instance getUGIFromSubject only). This class is the only non-test setConfiguration caller, so the remembered method equals the true global auth method for every Doris path. The per-conf HADOOP_SECURITY_AUTHORIZATION flag stays outside the guard (each catalog needs it on its own conf). Verified by compile (SecurityUtil.getAuthenticationMethod / AuthenticationMethod, 0 checkstyle), the fe-kerberos UTs, and an adversarial review that checked the guard, thread-safety, and the getLoginUser deviation against the hadoop-common bytecode. Behavioral validation is e2e live-gated (two kerberos HDFS catalogs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…L7/L8), mark done Design docs for the kerberos low-risk cluster (first-writer-wins guard on UGI.setConfiguration with the getLoginUser deviation rationale; doAs interrupt restore). L7's adversarial review verified soundness against the hadoop-common bytecode. Mark L7/L8 DONE and repoint HANDOFF next-step to maxcompute L9. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
MaxComputePredicateConverter.convert wrapped the whole expression tree in one try/catch, so a single unconvertible sub-expression (unknown column, unsupported type, an unparseable session zone) dropped the ENTIRE filter to NO_PREDICATE and pushed nothing down -> full ODPS scan. This is perf-only (BE re-evaluates the full conjuncts; the pushed predicate only reduces source reads), but it means one awkward conjunct forfeits all pushdown. Special-case a top-level ConnectorAnd: convert each conjunct independently and AND the survivors, dropping (and logging) the unconvertible ones. This is only safe at the root, a positive/monotone position — dropping a conjunct from the root AND yields a superset that BE re-filters. OR (dropping a disjunct is a subset -> loses rows), NOT, and nested AND are still converted whole by the extracted convertOne(), so no semantics change. Since dropping the whole filter was already correct, dropping some conjuncts is too. Add 5 unit tests (the converter is a pure function): a top-level AND keeps its convertible conjunct when a sibling fails (RED before the fix), all-fail still degrades to NO_PREDICATE, a single survivor is returned unwrapped, a nested AND stays all-or-nothing, and a top-level OR is not tolerated. 21/21 green; the 3 behavior tests fail against the pre-fix converter. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…down), mark done Design doc for maxcompute L9 (top-level-AND partial predicate pushdown with the monotone-position correctness argument). Mark L9 DONE and repoint HANDOFF next-step to the paimon cluster (L11/L13/L14). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
The plugin-path paimon scan tagged JNI-serialized DataSplit ranges and the COUNT(*) collapse range with the table-level `file.format` default (parquet when unset) instead of the split's actual first data-file suffix. When the table option diverges from the on-disk files (an altered or mixed-format table), FE emitted the wrong `file_format` to BE. Narrow blast radius: the default JNI reader ignores it; only the opt-in paimon-cpp reader backfills it into FILE_FORMAT/MANIFEST_FORMAT, where a wrong value breaks the manifest read. Restore legacy `PaimonScanNode.getFileFormat(getPathString())` parity: add a package-private `dataSplitFileFormat(DataSplit, default)` helper deriving the format from `"/" + dataFiles().get(0).fileName()` (== legacy PaimonSplit path), falling back to the table default for an unrecognized suffix or empty file list (fail-safe, stricter than legacy's unguarded get(0)). Wire it into buildJniScanRange (isDataSplit arm only; non-DataSplit keeps the table default, legacy DUMMY_PATH parity) and buildCountRange. Also add the `.avro` arm dropped from getFileFormatBySuffix (legacy FileFormatUtils had it), completing exact legacy suffix parity; inert on the native arm (avro never reaches it). Test: new PaimonScanPlanProviderTest.jniAndCountRangesUseFileSuffixNotAltered- TableDefault decouples table-default (overlaid to parquet via Table.copy) from the on-disk .orc files and asserts JNI + COUNT ranges carry "orc" — RED against the pre-fix defaultFileFormat emission (the prior test used default==suffix==orc and could not distinguish). Module 66/66 green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…apping CREATE TABLE / CTAS building a paimon table dropped the declared NOT NULL on nested child types: ARRAY<INT NOT NULL> was built as ARRAY<INT> (nullable element), and MAP values / STRUCT fields likewise lost their nullability, so the on-disk paimon schema did not match the DDL. PaimonTypeMapping.toPaimonType now applies .copy(type.isChildNullable(i)) to the ARRAY element, the MAP value, and each STRUCT DataField (the MAP key stays .copy(false), legacy parity), restoring legacy DorisToPaimonTypeVisitor behavior (array/value/field = childResult.copy(getContainsNull)). Scope is nested nullability only: the field comment stays dropped (accepted display-only deviation DV-035 M10.1) and the field id stays sequential (legacy parity). ConnectorType already carries per-child nullability (isChildNullable), so no SPI change is needed. The default-nullable path is unchanged (.copy(true) is a paimon no-op), so existing parity tests stay green. Test: 3 new PaimonTypeMappingToPaimonTest cases assert element/value/field non-null survives for NOT NULL children (via .type().isNullable(), not DataField equality, since the comment stays dropped) — RED against the pre-fix no-copy. Module type-mapping + schema-builder tests 26/26 green, 0 checkstyle, gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
The `ignore_split_type` debugging escape hatch (IGNORE_JNI / IGNORE_NATIVE, used to isolate reader bugs by dropping one split kind) was a silent no-op on the plugin scan path — PaimonScanPlanProvider never read it, so setting it had no effect. Restore legacy PaimonScanNode.getSplits parity: read it once via a null-tolerant resolveIgnoreSplitType(session) helper (mirroring isCppReaderEnabled), then skip splits at the three legacy continue sites — IGNORE_JNI drops the nonDataSplit arm and the DataSplit-JNI arm, IGNORE_NATIVE drops the native arm. The COUNT(*) arm is never dropped (legacy parity). IGNORE_PAIMON_CPP stays a no-op: legacy getSplits never consulted it (whole-tree grep confirms), so honoring it would be a new divergence, not parity. Default NONE leaves every scan byte-unchanged. Test: 3 new PaimonScanPlanProviderTest cases (live ops.table + planScan) — IGNORE_JNI drops a force_jni split, IGNORE_NATIVE drops an append-only native split (precondition asserts the split really took the native path), IGNORE_PAIMON_CPP == NONE. RED against the pre-fix no-op. The nonDataSplit IGNORE_JNI arm is E2E-only (offline fixtures cannot produce a system-table non-DataSplit through planScan). Module scan tests 69/69 green, 0 checkstyle, import gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…4), mark done Land the three paimon subgroup fix designs (JNI/COUNT file_format from data-file suffix; nested nullability in Doris->Paimon mapping; honor ignore_split_type), each with its folded-in red-team resolution (workflow wf_05574ccb-bd2, 3 designs x 3 lenses, no UNSOUND). Mark L11/L13/L14 done in the tracking table and roll the HANDOFF: batch 4 paimon subgroup complete, next = iceberg/misc L15-L19. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…already done) The detailed L-list already recorded L3-L9 as DONE with commits, but the top progress table still showed them todo. Confirmed all 7 commits exist in the log (trino 4cd63c6/e27602d4ab6/be96adf76ba/18048f7f217, kerberos 9e4f299/59697ce3fc7, maxcompute 017d1af); mark them done in the table. L2 and L10 remain decision-type (no commit yet). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…apability) L2 (flipped-hive SQL result-cache eligibility + COUNTER) was already resolved by c9a8633 on a separate workstream: CacheAnalyzer/BindRelation/ SqlCacheContext/NereidsSqlCacheManager now recognize plugin tables via the connector-agnostic MTMVRelatedTableIf capability + its data-tied version token (the "add capability / restore cache" decision, not the register-deviation alternative), and the dead source-specific COUNTER_QUERY_HIVE_TABLE bump was intentionally removed. Mark done in the tracking table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
…ation (DV-050) Per user signoff (2026-07-12), accept the flipped-external EXPLAIN scan-node name VPluginDrivenScanNode as a display-only deviation (DV-050) rather than add a Connector.getLegacyEngineName SPI to restore the per-connector VHIVE_/ VICEBERG_SCAN_NODE names: the getNodeExplainString CONNECTOR: line already discloses the catalog type, the regression goldens already expect the generic name (only one reference left, already the new name), and this matches Trino (one generic TableScan node with the connector as an attribute). No code change. Also roll the L2-L10 reconciliation: L3-L9 confirmed done in the log, L2 done by c9a8633 (query-cache SPI capability). L2-L10 are now all closed; only L12/L20 remain as decision-type items. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
TPC-H: Total hot run time: 29603 ms |
Contributor
TPC-DS: Total hot run time: 180412 ms |
Contributor
ClickBench: Total hot run time: 24.77 s |
The plugin-driven HMS connector wrapped client-creation failures with only the
immediate cause's getMessage(). For a kerberos/SASL misconfig that immediate
cause is Hive's opaque RuntimeException("Unable to instantiate
...HiveMetaStoreClient"), so the real reason (thrift TTransportException: GSS
initiate failed) was dropped from the user-facing message. FE surfaces only the
top exception's message, unlike the legacy ThriftHMSCachedClient/HMSClientException
which appended Util.getRootCauseMessage(cause).
Add ThriftHmsClient.withRootCause() to append the deepest cause (className:
message form, matching the legacy format) at the createFreshClient/borrowClient
throw sites, with a guard so the pool re-wrap does not duplicate the reason.
This restores the thrift/GSS reason that
external_table_p0/kerberos/test_single_hive_kerberos asserts (both err1 and
err2 cases), and keeps the actionable cause visible to users debugging HMS
connection failures. Message-only change; no connector logic altered.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
Author
|
run buildall |
… rows The connector-supplied per-value NULL partition flag (58f3e36) plus the pn_NULL naming (a634c5c) changed the MV refresh predicate for paimon's genuine-NULL region partition from region IN ('__HIVE_DEFAULT_PARTITION__') -- which matched no rows and silently dropped the null rows -- to region IS NULL, which correctly materializes them. test_paimon_mtmv order_qt_null_partition now returns 5 rows: the two genuine-NULL region rows (id=2, id=3 -> \N) join the existing 'bj'/'null'/'NULL' rows. The golden still encoded the stale 3-row result and failed with "line 2, BIGINT result mismatch" (TeamCity #992648). Update the golden to the corrected baseline. Values taken verbatim from the CI run's realResults; the partition set itself (p_bj/p_null/p_NULL/pn_NULL) was already correct. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TzQBdJTUczAQs5mEAewkR4
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 29179 ms |
Contributor
TPC-DS: Total hot run time: 180483 ms |
Contributor
ClickBench: Total hot run time: 25.12 s |
Contributor
FE UT Coverage ReportIncrement line coverage |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
only for testing #65473