A heterogeneous gateway connector that serves several table formats through embedded sibling + * connectors uses this to route a foreign handle to the sibling that produced it: the sibling's concrete + * handle type is invisible across the plugin classloader split, so the gateway cannot {@code instanceof} it + * directly — it asks each sibling, and the sibling tests its OWN in-loader type. The default returns + * {@code false} (a connector owns no handle it did not define), so every non-gateway connector is + * unaffected.
+ * + *fe-core NEVER calls this — it is a connector-to-sibling routing predicate only, so the engine stays + * format-agnostic (it discriminates handles solely by the gateway's own handle type, never by asking a + * connector to classify one).
+ */ + default boolean ownsHandle(ConnectorTableHandle handle) { + return false; + } + /** Returns the scan plan provider for split generation. */ default ConnectorScanPlanProvider getScanPlanProvider() { return null; } + /** + * Returns the scan plan provider for the given table, allowing one connector to select a + * different provider per table. + * + *The selection MUST happen here, at provider-acquisition time — not inside a single + * dispatching provider — because {@link ConnectorScanPlanProvider} has methods that do not + * carry the handle (e.g. {@code appendExplainInfo}) and providers are built fresh/stateless + * per call, so a provider returned here must already be bound to the correct backing scanner + * for {@code handle}. This is the seam a heterogeneous gateway connector (one catalog serving + * multiple table formats) overrides to delegate to per-format sub-providers by the concrete + * (connector-defined) handle type; the engine never inspects the format.
+ * + *The default ignores {@code handle} and returns the connector-level + * {@link #getScanPlanProvider()}, so every single-format connector is unaffected.
+ */ + default ConnectorScanPlanProvider getScanPlanProvider(ConnectorTableHandle handle) { + return getScanPlanProvider(); + } + + /** + * Returns the write plan provider for sink ({@code TDataSink}) generation, + * or {@code null} if this connector does not support writes. + */ + default ConnectorWritePlanProvider getWritePlanProvider() { + return null; + } + + /** + * Returns the write plan provider for the given table, allowing one connector to select a different + * provider per table — the write-side analogue of {@link #getScanPlanProvider(ConnectorTableHandle)}. + * + *The default ignores {@code handle} and returns the connector-level {@link #getWritePlanProvider()}, so + * every single-format connector is unaffected. A heterogeneous gateway connector (one catalog serving + * multiple table formats) overrides this to delegate to a per-format sub-provider by the concrete + * (connector-defined) handle type; the engine never inspects the format.
+ */ + default ConnectorWritePlanProvider getWritePlanProvider(ConnectorTableHandle handle) { + return getWritePlanProvider(); + } + + /** + * The write operations the engine may perform on this connector — the single admission source. Reads the + * write provider's {@link ConnectorWritePlanProvider#supportedOperations()}; no provider ⇒ empty set ⇒ all + * writes rejected. The engine consults this instead of {@code getWritePlanProvider() != null}. + */ + default SetThe default ignores {@code handle} and returns the connector-level {@link #getProcedureOps()}, so every + * single-format connector is unaffected. A heterogeneous gateway connector (one catalog serving multiple + * table formats) overrides this to delegate a foreign (e.g. iceberg-on-HMS) handle to a sibling connector's + * procedure ops by the concrete (connector-defined) handle type; the engine never inspects the format.
+ */ + default ConnectorProcedureOps getProcedureOps(ConnectorTableHandle handle) { + return getProcedureOps(); + } + /** Returns the set of capabilities this connector supports. */ default SetThe default is empty (no derivation), so every connector that does not need it is unaffected. The
+ * iceberg connector overrides this to bridge a hadoop-catalog {@code warehouse=hdfs://
This is an escape-hatch layer for capability checks that don't warrant a dedicated + * provider abstraction. Write operations and sink traits (parallel write, partition-local + * sort, full-schema write order, static-partition materialization) are NOT declared here — + * they live on the connector's {@link org.apache.doris.connector.api.write.ConnectorWritePlanProvider} + * instead, surfaced via {@link Connector#getWritePlanProvider()}.
*/ public enum ConnectorCapability { - SUPPORTS_FILTER_PUSHDOWN, - SUPPORTS_PROJECTION_PUSHDOWN, - SUPPORTS_LIMIT_PUSHDOWN, - SUPPORTS_PARTITION_PRUNING, - SUPPORTS_INSERT, - SUPPORTS_DELETE, - SUPPORTS_UPDATE, - SUPPORTS_MERGE, - SUPPORTS_CREATE_TABLE, SUPPORTS_MVCC_SNAPSHOT, - SUPPORTS_METASTORE_EVENTS, - SUPPORTS_STATISTICS, - SUPPORTS_VENDED_CREDENTIALS, - SUPPORTS_ACID_TRANSACTIONS, - SUPPORTS_TIME_TRAVEL, - /** - * Indicates the connector supports multiple concurrent writers (sink instances). - * - *Connectors that do NOT declare this capability will use GATHER distribution - * (single writer), which is the safe default for transactional sinks like JDBC - * where each writer commits independently.
- * - *File-based connectors (Hive, Iceberg, etc.) that can safely handle - * parallel writers should declare this capability.
- */ - SUPPORTS_PARALLEL_WRITE, /** * Indicates the connector supports passthrough query via the {@code query()} TVF. * @@ -56,5 +36,127 @@ public enum ConnectorCapability { * {@link ConnectorTableOps#getColumnsFromQuery} to provide column metadata * for arbitrary SQL queries passed through to the remote data source. */ - SUPPORTS_PASSTHROUGH_QUERY + SUPPORTS_PASSTHROUGH_QUERY, + /** + * Indicates the connector exposes per-partition statistics (record count, on-disk size, + * file count) via {@link ConnectorTableOps#listPartitions}. + * + *{@code SHOW PARTITIONS} renders a rich multi-column result (Partition / PartitionKey / + * RecordCount / FileSizeInBytes / FileCount) for connectors declaring this capability, instead + * of the single partition-name column used by connectors that only implement + * {@code listPartitionNames}.
+ */ + SUPPORTS_PARTITION_STATS, + /** + * Indicates the connector's tables support background per-column auto-analyze (NDV / min / max / + * null-count collection) through the generic {@code ExternalAnalysisTask} FULL path. + * + *The statistics auto-collector admits a plugin-driven table into the background auto-analyze + * framework only when its connector declares this (replacing the legacy {@code instanceof + * IcebergExternalTable} whitelist), and then forces {@code AnalysisMethod.FULL} — sample analyze is + * unimplemented for external SQL-driven tables ({@code ExternalAnalysisTask.doSample} throws). + * Row/passthrough connectors that cannot serve per-column statistics (e.g. JDBC, ES) must NOT + * declare it so they stay excluded.
+ */ + SUPPORTS_COLUMN_AUTO_ANALYZE, + /** + * Indicates the connector's file-scan tables support Top-N lazy materialization: the scan first + * reads only the ordering/filter columns to locate the Top-N row ids, then materializes the + * remaining columns for just those rows (via the synthesized {@code GLOBAL_ROWID_COL}). + * + *The nereids Top-N lazy-materialize probe enables the {@code LazyMaterializeTopN} post-processor + * for a plugin-driven table only when its connector declares this (replacing the legacy exact-class + * {@code SUPPORT_RELATION_TYPES} membership of {@code IcebergExternalTable}). Row/passthrough + * connectors (e.g. JDBC, ES) must NOT declare it.
+ */ + SUPPORTS_TOPN_LAZY_MATERIALIZE, + /** + * Indicates the connector's table/database properties are user-facing and safe to render in + * {@code SHOW CREATE TABLE} / {@code SHOW CREATE DATABASE}. + * + *The SHOW CREATE TABLE plugin-driven arm renders LOCATION + PROPERTIES (and, when the + * connector pre-renders them under the {@code show.*} reserved keys, the PARTITION BY / ORDER BY + * clauses) only for connectors declaring this (replacing the legacy paimon-only engine-name gate). + * Row/passthrough connectors whose {@code getTableProperties()} returns connection properties + * including credentials (e.g. JDBC, ES) must NOT declare it, or SHOW CREATE TABLE would leak + * the connection password — the security control the legacy engine-name gate provided.
+ */ + SUPPORTS_SHOW_CREATE_DDL, + /** + * Indicates the connector exposes views as queryable objects distinct from tables. + * + *When a connector declares this, a plugin-driven table resolves its {@code isView()} from the + * connector ({@link ConnectorTableOps#viewExists}) instead of the {@code false} default, the catalog + * merges the connector's {@link ConnectorTableOps#listViewNames} back into {@code SHOW TABLES} (iceberg + * subtracts views from {@code listTableNames}), and the read/DML/SHOW CREATE arms treat the object as a + * view. Connectors with no view concept (e.g. JDBC, ES) must NOT declare it so every table stays + * {@code isView()==false} and no view round-trips are issued.
+ */ + SUPPORTS_VIEW, + /** + * Indicates the connector's file-scan tables support nested-column pruning: a query that reads only some + * sub-fields of a STRUCT/ARRAY/MAP column reads just those leaves from the data file instead of the whole + * complex column (read-amplification avoidance). + * + *The nereids nested-column-prune probe ({@code LogicalFileScan.supportPruneNestedColumn}) enables it + * for a plugin-driven table only when its connector declares this (replacing the legacy exact-class + * {@code IcebergExternalTable} arm). It is only correct when the connector also carries a stable per-field + * id down its column tree (top-level via {@link ConnectorColumn#withUniqueId} + nested via + * {@link ConnectorType#withChildrenFieldIds}), because the engine rewrites the nested access path from + * field names to those ids ({@code SlotTypeReplacer}) and the BE field-id scan path matches + * nested leaves by id — an un-translated (name / {@code -1}) leaf is skipped and returns NULL. Row/ + * passthrough connectors (e.g. JDBC, ES) and connectors that do not carry nested field ids must NOT + * declare it.
+ */ + SUPPORTS_NESTED_COLUMN_PRUNE, + /** + * Indicates the connector's external metadata (schema / partitions / snapshot) can be pre-warmed + * asynchronously by the planner before it takes the internal read lock, rather than loaded lazily + * during binding. + * + *{@code PluginDrivenExternalTable.supportsExternalMetadataPreload} returns true for a plugin-driven + * table only when its connector declares this (replacing the legacy engine-name {@code "jdbc"} gate), so + * {@code StatementContext.registerExternalTableForPreload} admits the table into the async pre-load pass + * (itself opt-in via the {@code enable_preload_external_metadata} session variable, default off). It is a + * pure planning/lock-latency optimization with no correctness effect: connectors whose metadata reads are + * cheap or not yet validated for concurrent pre-warming (e.g. ES) simply do not declare it and fall back + * to synchronous load at binding time.
+ */ + SUPPORTS_METADATA_PRELOAD, + /** + * Indicates the connector projects the querying user's per-connection delegated credential (OIDC/JWT/SAML) + * onto the remote metadata source, so metadata reads are authorized as that user rather than a single shared + * catalog identity (the Iceberg REST {@code iceberg.rest.session=user} model). + * + *This capability gates two behaviors. (a) FE credential injection: {@code ConnectorSessionBuilder.from} + * copies the user's delegated credential onto the {@link ConnectorSession} ONLY for connectors declaring + * this, so a JDBC/ES/hive-iceberg session never carries an OIDC token it would never use (least-privilege). + * (b) Shared-cache bypass: {@code ExternalCatalog.shouldBypassTableNameCache} / {@code ExternalDatabase} + * skip the catalog+name-keyed (NOT user-keyed) FE metadata caches for a credential-bearing session, so one + * user's REST-authorized/vended view is never served to another (cross-user leakage). Connectors that + * authenticate with a single static catalog identity (every non-REST iceberg flavor, JDBC, ES, ...) must + * NOT declare it. Declared by the iceberg connector only when configured {@code iceberg.rest.session=user}.
+ */ + SUPPORTS_USER_SESSION, + /** + * Indicates the connector exposes a metadata table (e.g. the hudi commit timeline) whose rows are read via + * {@link ConnectorMetadata#getMetadataTableRows}. + * + *The {@code hudi_meta()} / TIMELINE table-valued function's plugin-driven arm delegates to the connector + * only when it declares this; a connector with no metadata table must NOT declare it so the TVF rejects the + * table with "not a hudi table". Hudi declares it connector-wide (every hudi table has a commit timeline).
+ */ + SUPPORTS_METADATA_TABLE, + /** + * Indicates the connector's file-scan tables support {@code ANALYZE ... WITH SAMPLE} (scale-factor estimation + * from raw per-file byte sizes via {@link ConnectorStatisticsOps#listFileSizes}, with fe-core doing the + * Doris-type slot-width math). + * + *fe-core admits sampled analyze for a plugin-driven table only when it declares this. A heterogeneous + * connector (hive) emits it as a PER-TABLE marker in getTableSchema for its plain-hive tables only (legacy + * gated on {@code dlaType==HIVE}), so iceberg/hudi-on-HMS are excluded. Connectors whose {@code doSample} is + * unimplemented (native iceberg/paimon, JDBC, ES) must NOT declare it so sampled analyze stays rejected at + * build time.
+ */ + SUPPORTS_SAMPLE_ANALYZE } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java index 5b8b537d0a3841..3a2b8a42e1c6d6 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumn.java @@ -24,12 +24,33 @@ */ public final class ConnectorColumn { + /** Sentinel for "no reserved field id declared"; mirrors Doris Column's default uniqueId. */ + public static final int UNSET_UNIQUE_ID = -1; + private final String name; private final ConnectorType type; private final String comment; private final boolean nullable; private final String defaultValue; private final boolean isKey; + private final boolean isAutoInc; + private final boolean isAggregated; + // Marks a "with local time zone" timestamp column. fe-core's ConnectorColumnConverter translates + // this into Column.setWithTZExtraInfo() so DESC shows the WITH_TIMEZONE "Extra" marker, matching + // legacy PaimonExternalTable/PaimonSysExternalTable/IcebergUtils which set it from the SOURCE type + // root regardless of the timestamp_tz mapping flag. Defaults false; set via withTimeZone(). + private final boolean withTimeZone; + // Marks a hidden (non-visible) column. fe-core's ConnectorColumnConverter translates this into + // Column.setIsVisible(false). Used by synthetic write columns a connector declares through the schema + // SPI (e.g. iceberg's __DORIS_ICEBERG_ROWID_COL__ / v3 row-lineage), which must stay hidden. Defaults + // true (visible); set via invisible(). + private final boolean visible; + // Reserved Doris field id (Column uniqueId). fe-core's ConnectorColumnConverter re-applies it via + // Column.setUniqueId() only when set (>= 0). Used by synthetic write columns whose Doris column identity + // must equal a connector-reserved field id (e.g. iceberg v3 row-lineage _row_id=2147483540 / + // _last_updated_sequence_number=2147483539, matched by field id BE-side). Defaults UNSET_UNIQUE_ID (-1), + // leaving the Doris default untouched; set via withUniqueId(). + private final int uniqueId; public ConnectorColumn(String name, ConnectorType type, String comment, boolean nullable, String defaultValue) { @@ -38,12 +59,65 @@ public ConnectorColumn(String name, ConnectorType type, String comment, public ConnectorColumn(String name, ConnectorType type, String comment, boolean nullable, String defaultValue, boolean isKey) { + this(name, type, comment, nullable, defaultValue, isKey, false); + } + + public ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey, boolean isAutoInc) { + this(name, type, comment, nullable, defaultValue, isKey, isAutoInc, false); + } + + public ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey, boolean isAutoInc, + boolean isAggregated) { + this(name, type, comment, nullable, defaultValue, isKey, isAutoInc, isAggregated, false, true, + UNSET_UNIQUE_ID); + } + + private ConnectorColumn(String name, ConnectorType type, String comment, + boolean nullable, String defaultValue, boolean isKey, boolean isAutoInc, + boolean isAggregated, boolean withTimeZone, boolean visible, int uniqueId) { this.name = Objects.requireNonNull(name, "name"); this.type = Objects.requireNonNull(type, "type"); this.comment = comment; this.nullable = nullable; this.defaultValue = defaultValue; this.isKey = isKey; + this.isAutoInc = isAutoInc; + this.isAggregated = isAggregated; + this.withTimeZone = withTimeZone; + this.visible = visible; + this.uniqueId = uniqueId; + } + + /** + * Returns a copy of this column marked as a "with local time zone" timestamp. See + * {@link #isWithTimeZone()}; the marker is intentionally orthogonal to the mapped {@link #getType()} + * so it survives even when the column is mapped to a plain DATETIME (timestamp_tz mapping off). + */ + public ConnectorColumn withTimeZone() { + return new ConnectorColumn(name, type, comment, nullable, defaultValue, + isKey, isAutoInc, isAggregated, true, visible, uniqueId); + } + + /** + * Returns a copy of this column marked hidden (non-visible). See {@link #isVisible()}; used to declare + * synthetic write columns through the schema SPI so the converter re-applies {@code setIsVisible(false)}. + */ + public ConnectorColumn invisible() { + return new ConnectorColumn(name, type, comment, nullable, defaultValue, + isKey, isAutoInc, isAggregated, withTimeZone, false, uniqueId); + } + + /** + * Returns a copy of this column carrying the given reserved field id. See {@link #getUniqueId()}; the + * converter re-applies it via {@code Column.setUniqueId()} only when set (>= 0). Used to declare + * synthetic write columns whose Doris column identity must equal a connector-reserved field id + * (iceberg v3 row-lineage). + */ + public ConnectorColumn withUniqueId(int uniqueId) { + return new ConnectorColumn(name, type, comment, nullable, defaultValue, + isKey, isAutoInc, isAggregated, withTimeZone, visible, uniqueId); } public String getName() { @@ -70,6 +144,26 @@ public boolean isKey() { return isKey; } + public boolean isAutoInc() { + return isAutoInc; + } + + public boolean isAggregated() { + return isAggregated; + } + + public boolean isWithTimeZone() { + return withTimeZone; + } + + public boolean isVisible() { + return visible; + } + + public int getUniqueId() { + return uniqueId; + } + @Override public boolean equals(Object o) { if (this == o) { @@ -81,6 +175,11 @@ public boolean equals(Object o) { ConnectorColumn that = (ConnectorColumn) o; return nullable == that.nullable && isKey == that.isKey + && isAutoInc == that.isAutoInc + && isAggregated == that.isAggregated + && withTimeZone == that.withTimeZone + && visible == that.visible + && uniqueId == that.uniqueId && name.equals(that.name) && type.equals(that.type) && Objects.equals(comment, that.comment) @@ -89,7 +188,8 @@ public boolean equals(Object o) { @Override public int hashCode() { - return Objects.hash(name, type, comment, nullable, defaultValue, isKey); + return Objects.hash(name, type, comment, nullable, defaultValue, isKey, isAutoInc, isAggregated, + withTimeZone, visible, uniqueId); } @Override diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java new file mode 100644 index 00000000000000..7c413e5d313ab8 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorColumnStatistics.java @@ -0,0 +1,101 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import java.util.Objects; + +/** + * Per-column statistics a connector can serve WITHOUT a table scan (e.g. hive's HMS/spark column stats), + * feeding the query-planner column-statistics fast path. + * + *This carries only RAW facts. The connector does NOT compute the Doris-type-dependent + * {@code dataSize}/{@code avgSize} — those depend on the column's fixed slot width, which the connector + * cannot know without importing fe-type. fe-core derives them from {@link #getAvgSizeBytes()} (when the + * source knows the average value size, e.g. a hive string column's {@code avgColLen}) or the column's slot + * width otherwise, mirroring the {@link #getTableStatistics}-style raw/derived split. Use {@link #UNKNOWN} + * when statistics are unavailable.
+ */ +public final class ConnectorColumnStatistics { + + /** Sentinel indicating no per-column statistics are available. */ + public static final ConnectorColumnStatistics UNKNOWN = + new ConnectorColumnStatistics(-1, -1, -1, -1); + + private final long rowCount; + private final long ndv; + private final long numNulls; + private final double avgSizeBytes; + + public ConnectorColumnStatistics(long rowCount, long ndv, long numNulls, double avgSizeBytes) { + this.rowCount = rowCount; + this.ndv = ndv; + this.numNulls = numNulls; + this.avgSizeBytes = avgSizeBytes; + } + + /** The table row count the stats are relative to (source {@code numRows}), or -1 if unknown. */ + public long getRowCount() { + return rowCount; + } + + /** Number of distinct values, or -1 if unknown. */ + public long getNdv() { + return ndv; + } + + /** Number of nulls, or -1 if unknown. */ + public long getNumNulls() { + return numNulls; + } + + /** + * Average per-value size in bytes when the source knows it (e.g. a hive string column's {@code avgColLen}), + * or {@code -1} when it does not — fe-core then falls back to the Doris column's fixed slot width. + */ + public double getAvgSizeBytes() { + return avgSizeBytes; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorColumnStatistics)) { + return false; + } + ConnectorColumnStatistics that = (ConnectorColumnStatistics) o; + return rowCount == that.rowCount + && ndv == that.ndv + && numNulls == that.numNulls + && Double.compare(that.avgSizeBytes, avgSizeBytes) == 0; + } + + @Override + public int hashCode() { + return Objects.hash(rowCount, ndv, numNulls, avgSizeBytes); + } + + @Override + public String toString() { + return "ConnectorColumnStatistics{rowCount=" + rowCount + + ", ndv=" + ndv + + ", numNulls=" + numNulls + + ", avgSizeBytes=" + avgSizeBytes + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java new file mode 100644 index 00000000000000..eeacdf232b6441 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorContractValidator.java @@ -0,0 +1,71 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.apache.doris.connector.api.handle.WriteOperation; + +import java.util.Set; + +/** + * Fails loud ({@link IllegalStateException}) if a connector's declared write capabilities are internally + * inconsistent. The invariants are purely structural (no table handle, no live catalog needed) and mirror + * the doc contracts the removed {@code ConnectorCapability} javadoc stated only in prose. + * + *Because the invariants are static properties of a connector's own capability declarations, they are + * enforced by the per-connector contract tests (which build each connector and call {@link #validate}), + * not at catalog registration: reading a connector's write capabilities constructs its write plan provider, + * which for some connectors (e.g. iceberg) eagerly builds the live remote catalog — too costly and + * outage-fragile to run on the FE metadata-replay / CREATE CATALOG path. This class stays available to any + * caller that already holds an eagerly-built connector and wants the same check.
+ */ +public final class ConnectorContractValidator { + + private ConnectorContractValidator() {} + + /** @throws IllegalStateException if any write-capability invariant is violated. */ + public static void validate(Connector connector, String catalogType) { + SetThe engine captures the credential at authentication time (fe-core {@code DelegatedCredential}) and + * copies it into this neutral DTO on the {@link ConnectorSession} — see + * {@link ConnectorSession#getDelegatedCredential()}. A connector that declares + * {@link ConnectorCapability#SUPPORTS_USER_SESSION} consumes it to project per-user identity onto the + * remote metadata source (e.g. an Iceberg REST catalog's {@code SessionCatalog}), WITHOUT importing any + * fe-core type. The field shape mirrors fe-core {@code DelegatedCredential} exactly so the copy is a + * lossless 1:1 mapping.
+ * + *The {@link #getToken() token} is security-sensitive: it is connection-scoped and in-memory only, and + * must never be edit-logged, persisted, or rendered by {@code SHOW} / profile / {@code information_schema} + * surfaces. {@link #toString()} redacts it.
+ */ +public final class ConnectorDelegatedCredential { + + private final Type type; + private final String token; + // Absolute expiry in epoch millis, or null when the authenticator supplied none. Kept as a boxed Long + // (not OptionalLong) so the field is nullable; getExpiresAtMillis() re-wraps it, mirroring fe-core. + private final Long expiresAtMillis; + + public ConnectorDelegatedCredential(Type type, String token) { + this(type, token, OptionalLong.empty()); + } + + public ConnectorDelegatedCredential(Type type, String token, OptionalLong expiresAtMillis) { + this.type = Objects.requireNonNull(type, "type is required"); + this.token = Objects.requireNonNull(token, "token is required"); + Objects.requireNonNull(expiresAtMillis, "expiresAtMillis is required"); + this.expiresAtMillis = expiresAtMillis.isPresent() ? expiresAtMillis.getAsLong() : null; + } + + public Type getType() { + return type; + } + + public String getToken() { + return token; + } + + public OptionalLong getExpiresAtMillis() { + return expiresAtMillis == null ? OptionalLong.empty() : OptionalLong.of(expiresAtMillis); + } + + public boolean isExpired(long currentTimeMillis) { + // Inclusive comparison (>=) on purpose (mirrors fe-core DelegatedCredential.isExpired): at the exact + // expiration instant the credential is already treated as expired, so a token is never handed to the + // downstream REST server right as it stops being accepted — fail closed on the boundary. + return expiresAtMillis != null && currentTimeMillis >= expiresAtMillis; + } + + @Override + public String toString() { + return "ConnectorDelegatedCredential{" + + "type=" + type + + ", token=Returning {@link Optional#empty()} means the connector does not + * support MVCC and reads see whatever is current.
+ */ + default OptionalThe generic table model materializes its partition view from {@code listPartitions} by + * default (LIST partitions keyed on a last-modified timestamp). A connector whose partitions are + * intrinsically ranges with a snapshot-id freshness marker (e.g. iceberg's time transforms) + * overrides this to return a {@link ConnectorMvccPartitionView}; the generic model then builds + * {@code RangePartitionItem}s from the pre-rendered bounds and picks the snapshot type from the + * view's {@link ConnectorMvccPartitionView#getFreshness()}. All data-source-specific math + * (transform-to-range, partition-evolution overlap merge, snapshot-id resolution) happens inside + * the connector — fe-core stays source-agnostic.
+ * + *The default returns empty: connectors without a range partition view keep the generic + * {@code listPartitions} / LIST / timestamp behavior unchanged.
+ */ + default OptionalConsulted ONLY when the query-begin pin's {@link ConnectorMvccSnapshot#isLastModifiedFreshness()} + * is set — i.e. a last-modified connector, which the generic model reads off the pin it already holds. + * A snapshot-id connector (paimon/iceberg) leaves the pin flag false, so this is NEVER called for it and it + * pays ZERO extra metadata round-trips. And it is on the MTMV refresh path, never the scan hot path — so a + * partitioned last-modified connector may pay a {@code get_partitions_by_names} round-trip here (mirroring + * legacy, which fetched per-partition modify time only at refresh time) without regressing queries.
+ * + *The default returns empty: a connector that sets the pin flag MUST override this.
+ */ + default OptionalConsulted ONLY when the query-begin pin's {@link ConnectorMvccSnapshot#isLastModifiedFreshness()} + * is set (a snapshot-id connector never reaches here — zero extra round-trips). fe-core validates the + * partition exists in the materialized set BEFORE calling this; an {@code empty} return therefore means the + * partition VANISHED between the materialize and this fetch (a refresh-time race), and fe-core raises the + * legacy "can not find partition" error. The default returns empty: a last-modified connector MUST + * override it.
+ */ + default OptionalLong getPartitionFreshnessMillis( + ConnectorSession session, ConnectorTableHandle handle, String partitionName) { + return OptionalLong.empty(); + } + + /** + * Resolves an explicit time-travel spec (extracted from {@code FOR TIME AS OF} / + * {@code FOR VERSION AS OF}, or the {@code @tag} / {@code @branch} / {@code @incr} + * scan params) into a pinned snapshot. + * + *The connector owns all provider-specific parsing of {@code spec} (snapshot-id + * lookup, datetime parsing, tag/branch resolution, incremental-window validation). + * The returned snapshot's {@link ConnectorMvccSnapshot#getProperties()} carries the + * connector's scan options and its {@link ConnectorMvccSnapshot#getSchemaId()} is the + * resolved schema version.
+ * + *Returns {@link Optional#empty()} when the spec is unsupported or the target is not + * found, in which case the engine surfaces a user error. The default returns empty: + * connectors without time-travel do not honor explicit specs.
+ */ + default OptionalContract for MVCC connectors: thread the FULL {@code snapshot.getProperties()} + * (the scan-options map) into the returned handle so the read path sees exactly the + * connector-resolved options. When {@code properties} is empty, fall back to setting + * {@code scan.snapshot-id = snapshot.getSnapshotId()} (latest-pin parity).
+ * + *The default returns {@code handle} unchanged: connectors without time-travel ignore the + * pin and read whatever is current.
+ */ + default ConnectorTableHandle applySnapshot(ConnectorSession session, + ConnectorTableHandle handle, ConnectorMvccSnapshot snapshot) { + return handle; // default: connectors without time-travel ignore the pin + } + + /** + * Returns extra scan-level predicates the engine MUST apply for {@code handle} at the pinned + * {@code snapshot} — a connector "residual predicate" the read cannot enforce by file selection alone. + * The canonical case is an incremental / CDC commit-time window: a rewritten base file also carries + * forward out-of-window rows, so a ROW-LEVEL commit-time filter is required for correctness. The engine + * reverse-converts each returned {@link ConnectorExpression} into its native predicate and wraps a filter + * over the scan, binding column references to the connector's own (visible) output columns by name. + * + *The predicate is expressed in the connector-neutral {@link ConnectorExpression} pushdown grammar, + * NOT a source-specific shape — the engine NEVER discriminates by connector here; it applies whatever the + * connector returns. This mirrors the engine-agnostic residual-predicate model.
+ * + *The default returns an EMPTY list: a connector with no synthetic scan predicate adds nothing, so the + * plan is byte-identical. iceberg/paimon/jdbc/... inherit this empty default; only a connector that opts in + * (e.g. hudi incremental read) returns a non-empty list, and only for the scans that need it.
+ */ + default ListThe default returns {@code handle} unchanged: connectors without distributed rewrite ignore the + * scope and scan the whole table. A {@code null}/empty set is also a no-op (no scope = full scan), so the + * pin is only applied when a real per-group path set is present.
+ */ + default ConnectorTableHandle applyRewriteFileScope(ConnectorSession session, + ConnectorTableHandle handle, SetContract: a connector that builds column-pruned scan metadata keyed by the REQUESTED columns + * (e.g. a field-id schema dictionary) MUST, once this is applied, build that metadata over the FULL + * table schema instead — otherwise a lazily re-fetched column has no entry and the native read resolves + * it wrong (schema-evolved tables) or drops it. A connector whose scan metadata already spans all + * columns ignores this.
+ * + *The default returns {@code handle} unchanged: connectors without column-pruned scan metadata are + * unaffected by lazy materialization.
+ */ + default ConnectorTableHandle applyTopnLazyMaterialization(ConnectorSession session, + ConnectorTableHandle handle) { + return handle; // default: connectors without column-pruned scan metadata ignore the signal + } + + /** + * Engine-neutral rows for a connector metadata table (e.g. the hudi commit timeline), one row per record in + * the fixed column order the table-valued function declares. The TVF owns the column schema; the connector + * returns only the {@code String} cell values in that order (a {@code null} cell renders as SQL NULL). + * {@code kind} selects the metadata table (currently only {@code "timeline"}). + * + *Default empty: a connector without a metadata table returns nothing. The plugin-driven TVF arm gates on + * {@link ConnectorCapability#SUPPORTS_METADATA_TABLE} before delegating here, so only a connector that + * declares the capability is ever asked. Connectors overriding this that read remote metadata off the + * planning thread must pin the TCCL to the plugin classloader themselves (fe-core does not).
+ */ + default ListUsed as the Iceberg {@code SessionCatalog.SessionContext.sessionId()} — the OAuth2 {@code AuthSession} + * cache key — for {@link ConnectorCapability#SUPPORTS_USER_SESSION} connectors, so a session's queries reuse + * one minted auth session rather than re-authenticating per query. The default falls back to + * {@link #getQueryId()} for sessions/tests that carry no session id; the engine session implementation + * overrides it with the captured (and FE-forward-preserved) session id.
+ */ + default String getSessionId() { + return getQueryId(); + } + + /** + * Returns the user's per-connection delegated credential (OIDC/JWT/SAML), when one was captured at + * authentication and this session targets a connector that consumes it. + * + *Populated ONLY when the connector declares {@link ConnectorCapability#SUPPORTS_USER_SESSION} + * (least-privilege: a connector that would never use the token never receives it). The credential is a + * neutral SPI DTO — the connector reads it here instead of any fe-core type. Empty by default (no + * credential, or a connector that does not opt in).
+ */ + default OptionalUsed by connectors whose {@code begin*} write operations need to + * attach work to an outer transaction opened by + * {@link ConnectorWriteOps#beginTransaction(ConnectorSession)}. + * Connectors with statement-scoped writes (e.g. JDBC auto-commit) can + * ignore this and the default empty value.
+ */ + default OptionalThe id is the engine-side transaction id: it is registered in the engine + * transaction registry and stamped into the connector's data sink, so a + * connector must obtain it from the engine rather than mint its own. The + * default throws; the engine session implementation overrides it.
+ * + * @return a fresh engine transaction id + */ + default long allocateTransactionId() { + throw new UnsupportedOperationException("transaction id allocation not supported"); + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java index fb762355e53071..2e89a09d7dd881 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ConnectorStatisticsOps.java @@ -19,6 +19,8 @@ import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import java.util.Collections; +import java.util.List; import java.util.Optional; /** @@ -32,4 +34,45 @@ default OptionalThe names are WITHOUT any "$" prefix; fe-core composes the + * "{baseTable}${sysName}" reference name. Default: empty (no system + * tables). Implemented by connectors that expose system tables.
+ */ + default ListThe returned handle is connector-internal and carries whatever the + * connector needs (system-table name, scan-routing hints, etc.); it is + * opaque to fe-core. {@code sysName} is the bare name (no "$").
+ */ + default OptionalA connector whose partitioned tables expose their partition rows through the generic + * partition-values TVF (e.g. hive) overrides this to return {@code true} for that sys-table name; + * such a name need NOT return a handle from {@link #getSysTableHandle} (the TVF path never consults + * it). fe-core needs the kind at discovery time (before any handle is fetched), so it cannot be + * inferred from an empty {@code getSysTableHandle}. {@code sysName} is the bare name (no + * {@code "$"}).
+ */ + default boolean isPartitionValuesSysTable(ConnectorSession session, + ConnectorTableHandle baseTableHandle, String sysName) { + return false; + } + /** Returns the schema (columns, format, etc.) for the given table. */ default ConnectorTableSchema getTableSchema( ConnectorSession session, ConnectorTableHandle handle) { @@ -44,6 +96,38 @@ default ConnectorTableSchema getTableSchema( "getTableSchema not implemented"); } + /** + * Returns the schema AT {@code snapshot.getSchemaId()} — the schema as of the + * pinned snapshot, for time-travel reads under schema evolution. + * + *The default ignores the snapshot and returns the latest schema via + * {@link #getTableSchema(ConnectorSession, ConnectorTableHandle)}. A connector that + * supports schema-at-snapshot overrides this to resolve the schema version.
+ */ + default ConnectorTableSchema getTableSchema( + ConnectorSession session, ConnectorTableHandle handle, + ConnectorMvccSnapshot snapshot) { + return getTableSchema(session, handle); + } + + /** + * Renders the native {@code SHOW CREATE TABLE} DDL for a table, fetching schema FRESH from the underlying + * metastore at call time (bypassing any connector-side table cache) so the returned statement always + * reflects the latest remote schema. + * + *This is a LAZY, per-call interception point used ONLY by {@code ShowCreateTableCommand}. It intentionally + * does NOT participate in the {@code SUPPORTS_SHOW_CREATE_DDL} capability (which gates the engine-assembled + * DDL in {@code Env.getDdlStmt} for every caller, including delegated sibling tables and the HTTP schema + * endpoint). A connector that does not natively render its own SHOW CREATE returns {@link Optional#empty()}, + * and the command falls through to the generic {@code Env.getDdlStmt} path unchanged.
+ * + * @return the full {@code CREATE TABLE} statement, or {@link Optional#empty()} to defer to the engine + */ + default OptionalConnectors should override this when they support advanced + * {@code CREATE TABLE} options. The default degrades to the legacy + * {@link #createTable(ConnectorSession, ConnectorTableSchema, Map)}, + * dropping partition / bucket / external / {@code ifNotExists} info.
+ * + * @throws DorisConnectorException if the connector cannot honor the request + */ + default void createTable(ConnectorSession session, + ConnectorCreateTableRequest request) { + ConnectorTableSchema schema = new ConnectorTableSchema( + request.getTableName(), + request.getColumns(), + null, + request.getProperties()); + createTable(session, schema, request.getProperties()); + } + /** Drops the specified table. */ default void dropTable(ConnectorSession session, ConnectorTableHandle handle) { @@ -72,6 +217,115 @@ default void dropTable(ConnectorSession session, "DROP TABLE not supported"); } + /** Renames the table identified by {@code handle} to {@code newName} within the same database. */ + default void renameTable(ConnectorSession session, + ConnectorTableHandle handle, String newName) { + throw new DorisConnectorException( + "RENAME TABLE not supported"); + } + + /** + * Truncates the table identified by {@code handle}. When {@code partitions} is non-empty only those + * partitions are truncated; {@code null} / empty truncates the whole table. + * + *Connectors that support {@code TRUNCATE TABLE} override this. The default throws, matching the + * pre-flip behavior of the generic bridge (which had no truncate route for the SPI path).
+ * + * @throws DorisConnectorException if the connector does not support truncate + */ + default void truncateTable(ConnectorSession session, + ConnectorTableHandle handle, ListShould be cheap and avoid loading per-partition metadata.
+ */ + default ListConnectors should push the filter into the metastore / catalog when + * possible. {@code filter} is empty when the caller wants the full list.
+ */ + default ListUsed by the {@code partition_values()} TVF and by column-distinct-value + * optimizations. Inner list order matches {@code partitionColumns}.
+ */ + default ListA uniform-format connector (e.g. iceberg — every table orc/parquet) declares a scan capability for all + * its tables connector-wide. A heterogeneous connector (e.g. hive — orc/parquet/text/json/csv/view/hudi in + * one catalog) whose eligibility is per-table file-format gated cannot: Top-N lazy materialization and + * nested-column pruning are orc/parquet-only, and blanket-declaring them connector-wide would over-admit a + * text/json table (a correctness bug for nested-column pruning, which reads NULL leaves without field ids). + * Such a connector instead emits the capability name here, per-table, computed from that table's format.
+ * + *fe-core reads it ADDITIVELY (a capability counts as supported if it is in the connector-wide set OR in + * this per-table list) from the already-cached schema — no remote round-trip and no file-format inspection + * in fe-core. Single-format connectors never emit it and are unaffected. Stripped from the user-facing + * SHOW CREATE TABLE PROPERTIES(...) block.
+ */ + public static final String PER_TABLE_CAPABILITIES_KEY = INTERNAL_KEY_PREFIX + "connector.per-table-capabilities"; + + /** + * Reserved property key carrying a CSV of the table's distribution (bucketing) column names, already + * lowercased. A heterogeneous connector (hive) whose bucketing varies per table cannot express it as a + * connector-wide trait, so it emits it here per-table. + * + *fe-core's {@code PluginDrivenExternalTable.getDistributionColumnNames()} reads it from the cached schema + * (no remote round-trip) so a flipped bucketed hive table picks the same NDV estimator as legacy + * {@code HMSExternalTable.getDistributionColumnNames} (a single bucket column selects the linear estimator in + * sampled analyze). A non-bucketed table omits it and connectors with no bucketing concept never emit it. + * Stripped from the user-facing SHOW CREATE TABLE PROPERTIES(...) block.
+ */ + public static final String DISTRIBUTION_COLUMNS_KEY = INTERNAL_KEY_PREFIX + "connector.distribution-columns"; + + /** + * The full set of FE-internal reserved control keys. fe-core strips every member from the user-facing + * SHOW CREATE TABLE PROPERTIES(...) block; centralizing the set here means adding a new reserved key is a + * single edit that the strip picks up automatically. All members share {@link #INTERNAL_KEY_PREFIX}. + */ + public static final SetA type is identified by its name (e.g. "INT", "VARCHAR", "DECIMAL"), * optional precision/scale parameters, and optional child types for * complex types like ARRAY or MAP.
+ * + *Per-child nullability + comment ({@link #isChildNullable(int)} / + * {@link #getChildComment(int)}): for complex types the type optionally carries the nullability and + * comment of each child (STRUCT field, ARRAY element, MAP value), parallel to {@link #getChildren()}. + * These are additive — the legacy factories ({@link #of}/{@link #arrayOf(ConnectorType)}/ + * {@link #mapOf(ConnectorType, ConnectorType)}/{@link #structOf(List, List)}) leave them unset, in which + * case every child defaults to nullable with no comment. They let a connector preserve a NOT NULL declared + * inside a complex type (e.g. iceberg CREATE TABLE / MODIFY COLUMN of a STRUCT field) and the per-field + * comments needed to diff a complex MODIFY. They are intentionally excluded from {@link #equals(Object)}/ + * {@link #hashCode()}: type identity stays the structural shape (name/precision/scale/children/field + * names), matching the legacy Doris {@code Type} comparison that drives schema-change detection (nullability + * and comment are compared separately, field-by-field, by the consumer) and keeping every existing + * equality-based caller/test unaffected.
+ * + *Per-child field id ({@link #getChildFieldId(int)}, set via {@link #withChildrenFieldIds(List)}): + * the stable id of each child field, parallel to {@link #getChildren()}. Also additive and excluded + * from {@link #equals(Object)}/{@link #hashCode()}. A connector that tracks a stable per-field id (iceberg + * field-ids) carries them here so fe-core can stamp the Doris child column tree's {@code uniqueId}, which the + * engine's nested access-path rewrite and the BE field-id scan path match nested leaves by.
*/ public final class ConnectorType { @@ -36,6 +55,20 @@ public final class ConnectorType { private final int scale; private final ListFollows a two-phase lifecycle for each write operation:
- *Every write goes through a single transaction model: the engine opens a + * {@link ConnectorTransaction} via {@link #beginTransaction(ConnectorSession)}, the + * connector's write plan attaches to it, BE feeds commit fragments back through + * {@link ConnectorTransaction#addCommitData}, and the engine finally calls + * {@code commit()} / {@code rollback()}. Connectors whose writes are auto-committed + * by BE return a degenerate no-op transaction.
* - *All methods have default implementations that throw - * {@link DorisConnectorException}, so connectors only override what they support.
+ *All methods have default implementations (throwing / read-only), so connectors + * only override what they support.
*/ public interface ConnectorWriteOps { - // ──────────────────── Capability Queries ──────────────────── - - /** Returns {@code true} if this connector supports INSERT operations. */ - default boolean supportsInsert() { - return false; - } - - /** Returns {@code true} if this connector supports DELETE operations. */ - default boolean supportsDelete() { - return false; - } - - /** Returns {@code true} if this connector supports MERGE (INSERT + DELETE) operations. */ - default boolean supportsMerge() { - return false; - } - - // ──────────────────── Write Configuration ──────────────────── - - /** - * Returns the write configuration for this table. - * - *The engine uses the returned {@link ConnectorWriteConfig} to select the - * appropriate Thrift data sink type and pass properties to BE.
- * - * @param session current session - * @param handle the target table handle - * @param columns the columns being written (ordered to match INSERT column list) - * @return write configuration describing sink type, format, location, etc. - */ - default ConnectorWriteConfig getWriteConfig( - ConnectorSession session, - ConnectorTableHandle handle, - ListThe default permits everything: connectors with no per-table mode constraint need not override. A + * connector whose tables can be configured in a mode that forbids row-level DML (e.g. iceberg + * copy-on-write) MUST override this and throw, so the rejection — and its message — stay in the connector + * rather than being drafted by the engine. {@code op} values other than DELETE/UPDATE/MERGE are not + * row-level DML and an overriding connector should treat them as a no-op.
*/ - default ConnectorDeleteHandle beginDelete( - ConnectorSession session, - ConnectorTableHandle handle) { - throw new DorisConnectorException("DELETE not supported"); + default void validateRowLevelDmlMode(ConnectorSession session, ConnectorTableHandle handle, WriteOperation op) { + // default: no per-table mode constraint } /** - * Commits the delete operation using collected fragments. + * Validates that every column named in a static-partition spec ({@code INSERT [OVERWRITE] ... PARTITION + * (col=val)}) is a legal static-partition target on {@code handle}, throwing a {@link DorisConnectorException} + * with a connector-authored message otherwise. Called at analysis time, before synthesizing the write plan, + * so the engine rejects an unknown / non-identity / unpartitioned static-partition column up front (fail + * loud) rather than letting it slip through to physical planning. * - * @param session current session - * @param handle the delete handle from {@link #beginDelete} - * @param fragments serialized commit info collected from BE nodes + *The default accepts everything: connectors with no static-partition concept (e.g. name-mapped JDBC) + * need not override. A connector that supports {@code PARTITION(...)} writes and can reject a column (e.g. + * iceberg, where only identity partition fields may be targeted statically) MUST override this and throw, so + * the rejection — and its message — stay in the connector rather than being drafted by the engine. {@code + * staticPartitionColumnNames} is the set of column names from the PARTITION clause.
*/ - default void finishDelete(ConnectorSession session, - ConnectorDeleteHandle handle, - CollectionThe default accepts everything: connectors that ignore the name-list form need not override. A connector + * that must reject it (e.g. hive, where {@code INSERT ... PARTITION(p1, p2)} is unsupported) MUST override + * this and throw, so the rejection — and its message — stay in the connector rather than being drafted by the + * engine. {@code partitionNames} is the list of partition column names from the PARTITION clause (the engine + * calls this only when the list is non-empty).
*/ - default void abortDelete(ConnectorSession session, - ConnectorDeleteHandle handle) { - // default: no-op + default void validateWritePartitionNames(ConnectorSession session, ConnectorTableHandle handle, + ListEvery write-capable connector must return a transaction here. Connectors whose + * writes are auto-committed by BE (e.g. jdbc) return a degenerate + * {@link org.apache.doris.connector.api.handle.NoOpConnectorTransaction}; the default + * throws (a connector that supports writes but does not override this is misconfigured + * — fail loud).
*/ - default void finishMerge(ConnectorSession session, - ConnectorMergeHandle handle, - CollectionA heterogeneous gateway (one catalog serving multiple table formats) overrides this to route a foreign + * handle to its sibling connector's transaction, so the session-bound transaction's concrete type matches + * the per-handle-selected write plan provider. The no-arg version alone would bind the gateway's own + * transaction and the sibling's write plan would fail to downcast it. Mirrors the per-handle + * {@code getWritePlanProvider(handle)} seam.
*/ - default void abortMerge(ConnectorSession session, - ConnectorMergeHandle handle) { - // default: no-op + default ConnectorTransaction beginTransaction(ConnectorSession session, ConnectorTableHandle handle) { + return beginTransaction(session); } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/BranchChange.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/BranchChange.java new file mode 100644 index 00000000000000..67cafacdcd1c23 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/BranchChange.java @@ -0,0 +1,89 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +/** + * Neutral carrier for a {@code CREATE [OR REPLACE] BRANCH} request, decoupling the connector SPI from the + * fe-core/nereids {@code CreateOrReplaceBranchInfo}/{@code BranchOptions} types. + * + *The retention fields are nullable ({@code null} = not specified in the SQL, so the connector leaves the + * corresponding setting untouched), mirroring the {@code Optional<>} fields of the source {@code BranchOptions}. + * They are named after the snapshot-management knobs they drive, so a connector applies them 1:1: + * {@code maxSnapshotAgeMs} (SQL {@code RETAIN}), {@code minSnapshotsToKeep} (SQL {@code WITH SNAPSHOT RETENTION + * n SNAPSHOTS}), {@code maxRefAgeMs} (SQL {@code WITH SNAPSHOT RETENTION ... MINUTES}). A {@code null} + * {@code snapshotId} means "use the table's current snapshot", resolved by the connector against the live table.
+ */ +public final class BranchChange { + + private final String name; + private final boolean create; + private final boolean replace; + private final boolean ifNotExists; + private final Long snapshotId; + private final Long maxSnapshotAgeMs; + private final Integer minSnapshotsToKeep; + private final Long maxRefAgeMs; + + public BranchChange(String name, boolean create, boolean replace, boolean ifNotExists, + Long snapshotId, Long maxSnapshotAgeMs, Integer minSnapshotsToKeep, Long maxRefAgeMs) { + this.name = name; + this.create = create; + this.replace = replace; + this.ifNotExists = ifNotExists; + this.snapshotId = snapshotId; + this.maxSnapshotAgeMs = maxSnapshotAgeMs; + this.minSnapshotsToKeep = minSnapshotsToKeep; + this.maxRefAgeMs = maxRefAgeMs; + } + + public String getName() { + return name; + } + + public boolean isCreate() { + return create; + } + + public boolean isReplace() { + return replace; + } + + public boolean isIfNotExists() { + return ifNotExists; + } + + /** The target snapshot id, or {@code null} to use the table's current snapshot. */ + public Long getSnapshotId() { + return snapshotId; + } + + /** SQL {@code RETAIN} in ms, or {@code null} when unset. */ + public Long getMaxSnapshotAgeMs() { + return maxSnapshotAgeMs; + } + + /** SQL {@code WITH SNAPSHOT RETENTION n SNAPSHOTS}, or {@code null} when unset. */ + public Integer getMinSnapshotsToKeep() { + return minSnapshotsToKeep; + } + + /** SQL {@code WITH SNAPSHOT RETENTION ... MINUTES} in ms, or {@code null} when unset. */ + public Long getMaxRefAgeMs() { + return maxRefAgeMs; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java new file mode 100644 index 00000000000000..32c5381a279658 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorBucketSpec.java @@ -0,0 +1,87 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * Bucket / distribution specification carried by + * {@link ConnectorCreateTableRequest}. + * + *{@code algorithm} is a connector-known string. Common values:
+ *Faithful, lossless neutralization of the fe-catalog {@code ColumnPosition}, which is exactly
+ * {@code FIRST | AFTER
A {@code null} position means "no position clause" (append at the end / keep current position), + * matching the legacy {@code if (position != null)} guard.
+ */ +public final class ConnectorColumnPosition { + + /** Place the column first. Mirrors fe-catalog {@code ColumnPosition.FIRST}. */ + public static final ConnectorColumnPosition FIRST = new ConnectorColumnPosition(true, null); + + private final boolean first; + // The column name to place this column after; non-null iff !first. + private final String afterColumn; + + private ConnectorColumnPosition(boolean first, String afterColumn) { + this.first = first; + this.afterColumn = afterColumn; + } + + /** Place the column after the named column. Mirrors fe-catalog {@code new ColumnPosition(col)}. */ + public static ConnectorColumnPosition after(String afterColumn) { + return new ConnectorColumnPosition(false, Objects.requireNonNull(afterColumn, "afterColumn")); + } + + public boolean isFirst() { + return first; + } + + /** The column to place this column after; only meaningful when {@link #isFirst()} is false. */ + public String getAfterColumn() { + return afterColumn; + } + + @Override + public String toString() { + return first ? "FIRST" : "AFTER " + afterColumn; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorColumnPosition)) { + return false; + } + ConnectorColumnPosition that = (ConnectorColumnPosition) o; + return first == that.first && Objects.equals(afterColumn, that.afterColumn); + } + + @Override + public int hashCode() { + return Objects.hash(first, afterColumn); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java new file mode 100644 index 00000000000000..78bd956db0158e --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/ConnectorCreateTableRequest.java @@ -0,0 +1,202 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +import org.apache.doris.connector.api.ConnectorColumn; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** + * Full {@code CREATE TABLE} payload passed to + * {@code ConnectorTableOps.createTable(session, request)}. + * + *Carries partition / bucket / external / {@code IF NOT EXISTS} information
+ * absent from the legacy
+ * {@code createTable(session, ConnectorTableSchema, Map
{@code partitionSpec} and {@code bucketSpec} are nullable when the + * underlying DDL omits them.
+ */ +public final class ConnectorCreateTableRequest { + + private final String dbName; + private final String tableName; + private final ListThe {@code transform} string follows Appendix B of the connector SPI RFC: + * {@code identity / year / month / day / hour / bucket / truncate / list / range}. + * Unlisted values are treated as {@code CUSTOM} and interpreted by the connector.
+ * + *{@code transformArgs} carries numeric parameters (e.g., {@code [16]} for + * {@code bucket(16, col)} or {@code [10]} for {@code truncate(10, col)}).
+ */ +public final class ConnectorPartitionField { + + private final String columnName; + private final String transform; + private final List{@link Style} distinguishes the four supported partition flavors:
+ *{@code initialValues} is only meaningful for {@code LIST} / {@code RANGE} styles.
+ */ +public final class ConnectorPartitionSpec { + + public enum Style { + IDENTITY, + TRANSFORM, + LIST, + RANGE, + } + + private final Style style; + private final ListFor {@code LIST} partitions, {@code values} contains the literal list of + * permitted values (each inner list is one tuple matching the partition columns). + * For {@code RANGE} partitions, {@code values} contains exactly two tuples + * representing the [lower, upper) bound.
+ */ +public final class ConnectorPartitionValueDef { + + private final String partitionName; + private final ListMirrors the fe-core {@code SortFieldInfo} (column name + ASC/DESC + NULLS FIRST/LAST) without + * the connector taking a compile-time dependency on fe-core. Connectors that do not support a + * write order (e.g. paimon) simply ignore the list.
+ */ +public final class ConnectorSortField { + + private final String columnName; + private final boolean ascending; + private final boolean nullFirst; + + public ConnectorSortField(String columnName, boolean ascending, boolean nullFirst) { + this.columnName = Objects.requireNonNull(columnName, "columnName"); + this.ascending = ascending; + this.nullFirst = nullFirst; + } + + public String getColumnName() { + return columnName; + } + + public boolean isAscending() { + return ascending; + } + + public boolean isNullFirst() { + return nullFirst; + } + + @Override + public String toString() { + return columnName + (ascending ? " ASC" : " DESC") + + (nullFirst ? " NULLS FIRST" : " NULLS LAST"); + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorSortField)) { + return false; + } + ConnectorSortField that = (ConnectorSortField) o; + return ascending == that.ascending + && nullFirst == that.nullFirst + && columnName.equals(that.columnName); + } + + @Override + public int hashCode() { + return Objects.hash(columnName, ascending, nullFirst); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/DropRefChange.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/DropRefChange.java new file mode 100644 index 00000000000000..259aaccf725e66 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/DropRefChange.java @@ -0,0 +1,42 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +/** + * Neutral carrier for a {@code DROP BRANCH}/{@code DROP TAG} request, decoupling the connector SPI from the + * fe-core/nereids {@code DropBranchInfo}/{@code DropTagInfo} types. {@code ifExists} makes the drop a no-op when + * the named ref is absent. + */ +public final class DropRefChange { + + private final String name; + private final boolean ifExists; + + public DropRefChange(String name, boolean ifExists) { + this.name = name; + this.ifExists = ifExists; + } + + public String getName() { + return name; + } + + public boolean isIfExists() { + return ifExists; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/PartitionFieldChange.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/PartitionFieldChange.java new file mode 100644 index 00000000000000..89dbcc72ddbfce --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/PartitionFieldChange.java @@ -0,0 +1,106 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +/** + * Neutral carrier for a partition-field evolution request ({@code ALTER TABLE ... ADD/DROP/REPLACE PARTITION + * KEY}), decoupling the connector SPI from the fe-core/nereids {@code AddPartitionFieldOp}/ + * {@code DropPartitionFieldOp}/{@code ReplacePartitionFieldOp} types. + * + *A "partition field" is a column reference run through an optional transform. The four leading fields describe + * ONE such field; how the connector reads them depends on the SPI method it is passed to:
+ *A transform is identity when {@code transformName} is {@code null} (a bare column ref); {@code transformArg} + * carries the width for {@code bucket(n)}/{@code truncate(n)} and is {@code null} otherwise. The {@code old*} + * fields are only populated for {@code replacePartitionField}; they are {@code null} for add/drop.
+ */ +public final class PartitionFieldChange { + + // ---- The field to add / drop, or the NEW field of a replace ---- + private final String transformName; + private final Integer transformArg; + private final String columnName; + private final String partitionFieldName; + + // ---- The OLD field of a replace (null for add / drop) ---- + private final String oldPartitionFieldName; + private final String oldTransformName; + private final Integer oldTransformArg; + private final String oldColumnName; + + public PartitionFieldChange(String transformName, Integer transformArg, String columnName, + String partitionFieldName, String oldPartitionFieldName, String oldTransformName, + Integer oldTransformArg, String oldColumnName) { + this.transformName = transformName; + this.transformArg = transformArg; + this.columnName = columnName; + this.partitionFieldName = partitionFieldName; + this.oldPartitionFieldName = oldPartitionFieldName; + this.oldTransformName = oldTransformName; + this.oldTransformArg = oldTransformArg; + this.oldColumnName = oldColumnName; + } + + /** Transform name (e.g. {@code bucket}/{@code truncate}/{@code year}); {@code null} = identity. */ + public String getTransformName() { + return transformName; + } + + /** Width for {@code bucket(n)}/{@code truncate(n)}; {@code null} otherwise. */ + public Integer getTransformArg() { + return transformArg; + } + + /** Source column the transform is applied to. */ + public String getColumnName() { + return columnName; + } + + /** Optional partition-field name: the {@code AS} alias (add/replace) or the field to remove (drop). */ + public String getPartitionFieldName() { + return partitionFieldName; + } + + /** Replace only: the existing partition field to remove by name; {@code null} = remove by old transform. */ + public String getOldPartitionFieldName() { + return oldPartitionFieldName; + } + + /** Replace only: old transform name; {@code null} = identity. */ + public String getOldTransformName() { + return oldTransformName; + } + + /** Replace only: old transform width. */ + public Integer getOldTransformArg() { + return oldTransformArg; + } + + /** Replace only: old source column. */ + public String getOldColumnName() { + return oldColumnName; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/TagChange.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/TagChange.java new file mode 100644 index 00000000000000..d53ecf3016aed6 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/ddl/TagChange.java @@ -0,0 +1,72 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.ddl; + +/** + * Neutral carrier for a {@code CREATE [OR REPLACE] TAG} request, decoupling the connector SPI from the + * fe-core/nereids {@code CreateOrReplaceTagInfo}/{@code TagOptions} types. + * + *A {@code null} {@code snapshotId} means "use the table's current snapshot" (resolved by the connector + * against the live table); a tag, unlike a branch, requires a snapshot, so the connector fails loud when the + * current snapshot is also absent. {@code maxRefAgeMs} (SQL {@code RETAIN}) is {@code null} when unset.
+ */ +public final class TagChange { + + private final String name; + private final boolean create; + private final boolean replace; + private final boolean ifNotExists; + private final Long snapshotId; + private final Long maxRefAgeMs; + + public TagChange(String name, boolean create, boolean replace, boolean ifNotExists, + Long snapshotId, Long maxRefAgeMs) { + this.name = name; + this.create = create; + this.replace = replace; + this.ifNotExists = ifNotExists; + this.snapshotId = snapshotId; + this.maxRefAgeMs = maxRefAgeMs; + } + + public String getName() { + return name; + } + + public boolean isCreate() { + return create; + } + + public boolean isReplace() { + return replace; + } + + public boolean isIfNotExists() { + return ifNotExists; + } + + /** The target snapshot id, or {@code null} to use the table's current snapshot. */ + public Long getSnapshotId() { + return snapshotId; + } + + /** SQL {@code RETAIN} in ms, or {@code null} when unset. */ + public Long getMaxRefAgeMs() { + return maxRefAgeMs; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java new file mode 100644 index 00000000000000..1955ef681ed11e --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/ConnectorEventSource.java @@ -0,0 +1,58 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.event; + +/** + * A connector's incremental-metadata-change source: it fetches the underlying metastore's native + * notification events and parses them into connector-neutral {@link MetastoreChangeDescriptor}s. + * + *Surfaced via {@link org.apache.doris.connector.api.Connector#getEventSource()} (a + * capability-probe getter, NOT an {@code instanceof} — a connector without an event source returns + * {@code null}). The engine runs one connector-agnostic, role-aware background driver that iterates + * catalogs, calls {@link #pollOnce} on those whose connector exposes a source, and applies the + * returned descriptors to its own object graph and caches.
+ * + *Engine/connector split (Trino-aligned: engine owns HA/replication, plugin owns fetch/parse). + * The connector owns ONLY the metastore RPC and the message deserialization/merge behind {@link + * #getCurrentEventId()} and {@link #pollOnce}. The engine owns everything stateful and replicated: the + * per-catalog cursor (passed in via {@link EventPollRequest}, stored from {@link EventPollResult}), + * the master/follower role, the edit-log write of the synced cursor, and the follower→master + * {@code REFRESH CATALOG} forward. The connector is stateless with respect to the cursor.
+ * + *Classloader. The engine calls these methods under a context-classloader pin to this + * source's own plugin classloader, so the notification RPC and the JSON/GZIP deserialization inside + * resolve the plugin's bundled metastore classes. Implementations therefore need no pin of their own + * for the fetch/parse path.
+ */ +public interface ConnectorEventSource { + + /** + * The metastore's current (latest) notification event id. Used by the master to cheaply decide + * whether there is anything new to pull before calling {@link #pollOnce}. Returns {@code -1} if the + * id cannot be read this cycle. + */ + long getCurrentEventId(); + + /** + * Fetch and parse one batch of notification events for a catalog, returning connector-neutral + * descriptors plus the new cursor and an optional full-refresh signal. Never returns source-native + * types. See {@link EventPollRequest} / {@link EventPollResult} for the role/cursor/refresh + * semantics. The batch size is the connector's own concern (read from its catalog properties). + */ + EventPollResult pollOnce(EventPollRequest request); +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java new file mode 100644 index 00000000000000..4662c308b22e51 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/event/EventPollRequest.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.event; + +/** + * The engine-supplied input to one {@link ConnectorEventSource#pollOnce} call. The engine owns the + * cursor and the role; the connector owns the fetch strategy those imply. + * + *Type-neutrality is load-bearing. Every field is a primitive, a {@code String}, or a
+ * {@code List
Engine/connector split. The connector owns fetch+parse and emits these descriptors + * (already merged/deduplicated). The engine owns the structural application — register/unregister/ + * rename in {@code CatalogMgr}, cache invalidation via {@code Connector.invalidateTable/Db/Partition}, + * the master's edit-log write, and follower/master role handling. Trino-aligned: the plugin surfaces + * what changed; the engine decides how its own metadata reacts.
+ * + *Partition granularity is by canonical partition NAME ({@code "col=val/.../colN=valN"}), never by + * raw column values — the caches are name-keyed, so carrying names avoids the values→whole-table + * degrade.
+ */ +public final class MetastoreChangeDescriptor { + + /** + * The kind of change. Maps 1:1 to what the engine does when applying the descriptor. A source + * event that changes nothing the engine tracks (e.g. a properties-only ALTER DATABASE, or an + * unsupported event type) produces NO descriptor rather than a no-op {@code Op}. + */ + public enum Op { + /** A new database exists; register it into the catalog. */ + REGISTER_DATABASE, + /** A database was dropped; unregister it from the catalog. */ + UNREGISTER_DATABASE, + /** A database was renamed; {@link #getDbName()} → {@link #getDbNameAfter()}. */ + RENAME_DATABASE, + /** A new table exists; register it into its database. */ + REGISTER_TABLE, + /** A table was dropped; unregister it from its database. */ + UNREGISTER_TABLE, + /** + * A table was renamed OR a view was recreated; {@link #getTableName()} → + * {@link #getTableNameAfter()} (the after-name equals the before-name for a view recreate, + * which rebuilds the table object so a changed view definition takes effect). + */ + RENAME_TABLE, + /** A table changed in place (insert / plain alter); drop its caches, keep it registered. */ + REFRESH_TABLE, + /** Partitions were added; invalidate the table's partition caches so a re-list picks them up. */ + ADD_PARTITIONS, + /** Partitions were dropped; invalidate the table's partition caches. */ + DROP_PARTITIONS, + /** Partitions changed in place; invalidate the named partitions' caches. */ + REFRESH_PARTITIONS + } + + private final Op op; + private final String dbName; + private final String tableName; + private final String dbNameAfter; + private final String tableNameAfter; + private final ListConnector implementations carry internal state (e.g., Iceberg - * delete snapshot, position delete file context) in their concrete - * implementation of this interface.
- */ -public interface ConnectorDeleteHandle { -} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorInsertHandle.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorInsertHandle.java deleted file mode 100644 index 3b831a94c8a5eb..00000000000000 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorInsertHandle.java +++ /dev/null @@ -1,24 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.connector.api.handle; - -/** - * Opaque insert handle returned by {@code beginInsert}. - */ -public interface ConnectorInsertHandle { -} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorMergeHandle.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorMergeHandle.java deleted file mode 100644 index 4eae994e8e914a..00000000000000 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorMergeHandle.java +++ /dev/null @@ -1,29 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.connector.api.handle; - -/** - * Opaque merge handle returned by - * {@link org.apache.doris.connector.api.ConnectorWriteOps#beginMerge}. - * - *Connector implementations carry merge-on-read state (e.g., - * Iceberg merge context, combined insert+delete handling) in their - * concrete implementation of this interface.
- */ -public interface ConnectorMergeHandle { -} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java new file mode 100644 index 00000000000000..9c58d282ea09c0 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorTransaction.java @@ -0,0 +1,157 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.handle; + +import org.apache.doris.connector.api.pushdown.ConnectorPredicate; + +import java.io.Closeable; +import java.util.Set; + +/** + * A connector-managed transaction that scopes one or more write operations. + * + *Lifecycle: the engine calls {@link #commit()} on success or + * {@link #rollback()} on failure, then always calls {@link #close()} to + * release resources. {@code rollback()} and {@code close()} are safe to + * call multiple times.
+ * + *Extends the marker {@link ConnectorTransactionHandle} so that existing + * APIs that traffic in opaque handles continue to work without change.
+ */ +public interface ConnectorTransaction extends ConnectorTransactionHandle, Closeable { + + /** Stable transaction ID assigned by the connector. */ + long getTransactionId(); + + /** + * Commits all pending operations bound to this transaction. + * + * @throws org.apache.doris.connector.api.DorisConnectorException + * on conflict, IO failure, or external system error + */ + void commit(); + + /** + * Aborts all pending operations and releases resources. + * Safe to call multiple times; subsequent calls are no-ops. + */ + void rollback(); + + /** Called by the engine after commit OR rollback to release connections etc. */ + @Override + void close(); + + /** + * Receives one serialized commit fragment produced by BE after writing a + * data fragment. The connector deserializes its own Thrift payload (e.g. + * {@code TMCCommitData} / {@code THivePartitionUpdate} / {@code TIcebergCommitData}) + * and accumulates it for {@link #commit()}. + * + *Default is a no-op for read-only / non-writing connectors.
+ * + * @param commitFragment the serialized connector-specific commit payload + */ + default void addCommitData(byte[] commitFragment) { + // no-op: connectors that participate in writes override this + } + + /** + * Whether this transaction allocates write block ranges through a write-time + * BE→FE callback. Only connectors with a stateful write session that + * hands out block ids (e.g. maxcompute) return {@code true}. + */ + default boolean supportsWriteBlockAllocation() { + return false; + } + + /** + * Allocates a contiguous range of write block ids for the given write + * session, returning the first allocated id. Called from the BE→FE RPC + * path during a write. + * + *Only invoked when {@link #supportsWriteBlockAllocation()} returns + * {@code true}; the default throws.
+ * + * @param writeSessionId opaque connector-defined write session identifier + * @param count number of block ids to allocate + * @return the first allocated block id + */ + default long allocateWriteBlockRange(String writeSessionId, long count) { + throw new UnsupportedOperationException("write block allocation not supported"); + } + + /** Returns the number of rows affected by the write(s) bound to this transaction. */ + default long getUpdateCnt() { + return 0; + } + + /** + * Applies an optional engine-extracted, target-only write constraint used for write-time optimistic + * conflict detection (O5-2). The engine extracts, from the analyzed DELETE/UPDATE/MERGE plan, the + * conjuncts that reference only the target table's own columns (slot origin-table == target, excluding + * synthetic {@code $row_id} / metadata / join columns) and hands the connector a neutral + * {@link ConnectorPredicate} at plan time, before {@code begin}/{@code commit}. + * + *A connector that does optimistic conflict detection converts the neutral predicate to its own + * dialect and uses it as the conflict-detection filter when the transaction commits (ANDed with any + * commit-time partition filter it derives itself). Connectors that do not do conflict detection — or + * that traffic in opaque handles — ignore it. The default is a no-op.
+ * + * @param targetOnlyFilter the neutral target-only predicate, or {@code null} when the plan yielded none + */ + default void applyWriteConstraint(ConnectorPredicate targetOnlyFilter) { + // no-op: connectors that do optimistic conflict detection override this + } + + /** + * A short, connector-identifying label for the query profile (cosmetic), e.g. + * {@code "JDBC"} / {@code "MAXCOMPUTE"}. The insert executor maps this label to a + * profile transaction type. Replaces the executor's former hard-coded connector + * switch; the default is a generic external label. + */ + default String profileLabel() { + return "EXTERNAL"; + } + + /** + * Compaction rewrite ({@code rewrite_data_files}): registers the original data files this transaction + * will atomically replace, by their RAW file paths. The engine rewrite driver hands the connector the + * neutral {@code String} paths from its bin-packed groups (fe-core cannot traffic in connector-native + * file objects across the wall); the connector resolves them back to its own file objects — e.g. by + * re-deriving from the table at the transaction's pinned snapshot — and removes them at {@link #commit()}. + * + *Default throws: only rewrite-capable connectors override it.
+ * + * @param dataFilePaths the raw paths of the source data files to replace + */ + default void registerRewriteSourceFiles(SetDefault throws: only rewrite-capable connectors override it.
+ */ + default int getRewriteAddedDataFilesCount() { + throw new UnsupportedOperationException("rewrite statistics not supported"); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java new file mode 100644 index 00000000000000..7af148e8d6b881 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/ConnectorWriteHandle.java @@ -0,0 +1,91 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.handle; + +import org.apache.doris.connector.api.ConnectorColumn; +import org.apache.doris.thrift.TSortInfo; + +import java.util.List; +import java.util.Map; +import java.util.Optional; + +/** + * A bound write request passed to + * {@link org.apache.doris.connector.api.write.ConnectorWritePlanProvider#planWrite}. + * + *Carries the engine-resolved facts about a single DML write: the target + * table handle, the column list, whether it is an OVERWRITE, and a free-form + * write context (static partition spec, write path, etc.). The connector reads + * these to build its Thrift data sink.
+ */ +public interface ConnectorWriteHandle { + + /** The target table handle (the connector's own opaque table handle). */ + ConnectorTableHandle getTableHandle(); + + /** The columns being written, ordered to match the INSERT column list. */ + ListDefaults to {@link WriteOperation#INSERT} so connectors that only do plain appends + * (jdbc / maxcompute) — which never set it — keep append semantics and stay byte-compatible.
+ */ + default WriteOperation getWriteOperation() { + return WriteOperation.INSERT; + } + + /** + * The engine-built BE sort instruction for this write, or {@code null} if the target needs no + * write-side sort. A connector declares its write-sort columns via + * {@link org.apache.doris.connector.api.write.ConnectorWritePlanProvider#getWriteSortColumns} + * (e.g. an iceberg table with a {@code WRITE ORDERED BY} sort order); the engine resolves those + * column indices against the bound sink output and builds the {@link TSortInfo}, which the + * connector then stamps onto its opaque Thrift sink in {@code planWrite}. + * + *The split is necessary because the bound output expressions live only in the engine + * (translation time), not in this source-agnostic handle. Defaults to {@code null} so connectors + * that declare no write sort (jdbc / maxcompute) keep their byte-identical unsorted sink output.
+ */ + default TSortInfo getSortInfo() { + return null; + } + + /** + * The named table branch this write targets ({@code INSERT INTO t@branch(name)}), or + * {@link Optional#empty()} when the write goes to the table's default ref. Threaded from the + * generic insert command context onto this handle; a versioned-table connector (iceberg / paimon) + * reads it in {@code planWrite} to point the commit at the branch. Defaults to empty so connectors + * with no branch concept (jdbc / maxcompute) keep their byte-identical default-ref write. + */ + default Optional{@link #getUpdateCnt()} returns {@code -1} — "no row count is produced by + * this transaction" — which signals the insert executor to keep the + * coordinator's row counter (DPP_NORMAL_ALL) for affected-rows rather than + * overwrite it with {@code 0}. {@code -1} is deliberately distinct from a + * genuine zero-row write ({@code 0}).
+ */ +public class NoOpConnectorTransaction implements ConnectorTransaction { + + private final long transactionId; + private final String profileLabel; + + public NoOpConnectorTransaction(long transactionId, String profileLabel) { + this.transactionId = transactionId; + this.profileLabel = profileLabel; + } + + @Override + public long getTransactionId() { + return transactionId; + } + + @Override + public void commit() { + // no-op: the write is already durably committed by BE (auto-commit sink) + } + + @Override + public void rollback() { + // no-op: there is nothing to undo on the FE side + } + + @Override + public void close() { + // no-op: no resources are held + } + + @Override + public long getUpdateCnt() { + // -1 = "no row count from this transaction; use the coordinator counter". + // Distinct from 0 (a genuine zero-row write) so the executor does not + // clobber loadedRows that BE already reported via DPP_NORMAL_ALL. + return -1; + } + + @Override + public String profileLabel() { + return profileLabel; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/WriteOperation.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/WriteOperation.java new file mode 100644 index 00000000000000..d1ef64770e23ed --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/handle/WriteOperation.java @@ -0,0 +1,49 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.handle; + +/** + * The kind of DML write a {@link ConnectorWriteHandle} carries. + * + *This is the operation axis (what the statement does), distinct from the sink + * mechanism that the removed {@code ConnectorWriteType} used to encode. A single + * {@code planWrite} reads it to pick the connector's Thrift sink dialect (e.g. iceberg's + * {@code TIcebergTableSink} vs {@code TIcebergDeleteSink} vs {@code TIcebergMergeSink}), and the + * iceberg transaction reads it to pick the SDK operation (AppendFiles / ReplacePartitions / + * OverwriteFiles / RowDelta / RewriteFiles). The default on {@link ConnectorWriteHandle#getWriteOperation()} + * is {@link #INSERT}, so connectors that only do plain appends (jdbc / maxcompute) need not declare it.
+ */ +public enum WriteOperation { + /** Plain INSERT (append rows). */ + INSERT, + /** INSERT OVERWRITE (truncate-and-insert, whole-table / dynamic / static partition). */ + OVERWRITE, + /** DELETE rows matching a predicate. */ + DELETE, + /** UPDATE rows (delete + re-insert under merge-on-read). */ + UPDATE, + /** MERGE INTO (matched/not-matched clauses; insert + delete). */ + MERGE, + /** + * Compaction rewrite ({@code ALTER TABLE ... EXECUTE rewrite_data_files}): atomically replace a set of + * existing data files with newly written, larger ones. The iceberg transaction maps it to the SDK + * {@code RewriteFiles} op (delete-old + add-new) guarded by {@code validateFromSnapshot} (OCC). Driven by + * the rewrite execution half, which holds one transaction across N per-group INSERT-SELECT writes. + */ + REWRITE +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccPartition.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccPartition.java new file mode 100644 index 00000000000000..102a906a0cfc6e --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccPartition.java @@ -0,0 +1,114 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import java.util.Collections; +import java.util.List; +import java.util.Objects; + +/** + * One final (post-merge) Doris partition in a {@link ConnectorMvccPartitionView}. + * + *For a {@link ConnectorMvccPartitionView.Style#RANGE} view, {@link #getLowerBound()} / + * {@link #getUpperBound()} are the connector's pre-rendered partition-key value tuples for the + * closed-open {@code [lower, upper)} range (one string per partition column), and the generic model + * assembles them into a {@code RangePartitionItem} using the table's partition column types — so no + * data-source-specific range math leaks into fe-core. {@link #getFreshnessValue()} is the per-partition + * marker (a snapshot id or epoch-millis timestamp, per the view's + * {@link ConnectorMvccPartitionView#getFreshness()}) the generic model wraps into the matching + * {@code MTMVSnapshotIf}.
+ * + *NULL-min sentinel: an empty {@link #getUpperBound()} (with a non-empty + * {@link #getLowerBound()}) denotes the genuine-NULL / minimum partition. The exclusive upper bound is NOT + * pre-rendered because it is the column-type/scale-aware successor of the lower key, which only the + * generic model can compute (it owns the Doris {@code Column}/{@code PartitionKey}). The model MUST derive the + * upper as {@code lowerKey.successor()} in that case — matching the connector's source behavior (e.g. iceberg's + * {@code nullLowKey.successor()}). A non-NULL RANGE partition always carries BOTH bounds non-empty; the model + * must not call {@code createPartitionKey} on an empty upper tuple.
+ */ +public final class ConnectorMvccPartition { + + private final String name; + private final ListThe generic table model ({@code PluginDrivenMvccExternalTable}) builds its partition view from + * {@link org.apache.doris.connector.api.ConnectorTableOps#listPartitions} by default, which always + * yields {@code LIST} partitions keyed on a last-modified timestamp. Connectors whose partitions are + * intrinsically ranges (e.g. iceberg's {@code YEAR}/{@code MONTH}/{@code DAY}/{@code HOUR} time + * transforms) need {@code RANGE} partition items plus a snapshot-id freshness marker to preserve the + * pre-SPI behavior. Such a connector returns this view from + * {@link org.apache.doris.connector.api.ConnectorMetadata#getMvccPartitionView}; the generic model then + * assembles {@code RangePartitionItem}s from the pre-rendered bounds and selects the snapshot type from + * {@link #getFreshness()} — all connector-specific math (transform-to-range, partition-evolution overlap + * merge, snapshot-id resolution) having already happened inside the connector.
+ * + *The view also carries a {@link #getNewestUpdateTimeMillis() newest-update-time} marker: a monotonically + * non-decreasing table-change timestamp the generic model answers the dictionary auto-refresh probe with + * (the snapshot id alone is unusable there because it need not be monotonic).
+ * + *A connector that does NOT override {@code getMvccPartitionView} leaves the generic model on its + * default {@code listPartitions} / LIST / timestamp path (byte-unchanged), so this view is purely + * additive.
+ */ +public final class ConnectorMvccPartitionView { + + /** How the generic model should model this table's partitions. */ + public enum Style { + /** The table is not a valid partitioned related table (e.g. the connector's eligibility gate + * failed); the generic model treats it as unpartitioned. */ + UNPARTITIONED, + /** Range partitions: each {@link ConnectorMvccPartition} carries pre-rendered [lower, upper) + * bounds the generic model turns into a {@code RangePartitionItem}. */ + RANGE + } + + /** Which per-partition value the generic model compares to decide MTMV staleness. */ + public enum Freshness { + /** {@code getFreshnessValue()} is a snapshot id; the model wraps it in {@code MTMVSnapshotIdSnapshot}. */ + SNAPSHOT_ID, + /** {@code getFreshnessValue()} is an epoch-millis timestamp; wrapped in {@code MTMVTimestampSnapshot}. */ + LAST_MODIFIED + } + + private final Style style; + private final Freshness freshness; + private final ListUnit is source-defined, NOT guaranteed epoch millis — only its monotonicity is relied upon, never + * its absolute scale. For iceberg this is the {@code last_updated_at} value from the PARTITIONS metadata table, + * which iceberg represents in MICROSECONDS; the generic model passes it through verbatim (parity with master, + * which also compares this raw value without conversion). Do NOT treat it as millis or convert it. + * {@code 0} when the table has no partitions / is unpartitioned (treated as "unchanged").
+ */ + public long getNewestUpdateTimeMillis() { + return newestUpdateTimeMillis; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorMvccPartitionView)) { + return false; + } + ConnectorMvccPartitionView that = (ConnectorMvccPartitionView) o; + return style == that.style + && freshness == that.freshness + && newestUpdateTimeMillis == that.newestUpdateTimeMillis + && partitions.equals(that.partitions); + } + + @Override + public int hashCode() { + return Objects.hash(style, freshness, partitions, newestUpdateTimeMillis); + } + + @Override + public String toString() { + return "ConnectorMvccPartitionView{style=" + style + + ", freshness=" + freshness + + ", partitions=" + partitions.size() + + ", newestUpdateTimeMillis=" + newestUpdateTimeMillis + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java new file mode 100644 index 00000000000000..d89bc7cc976367 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshot.java @@ -0,0 +1,150 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; + +/** + * Immutable description of a point-in-time snapshot taken from an MVCC-capable + * external table (Iceberg, Paimon, Hudi, ...). + * + *Returned by {@code ConnectorMetadata.beginQuerySnapshot} and friends. + * Used by the engine as the MVCC pin for all subsequent reads of the same + * table handle within a query, and serialized into BE scan ranges so the + * read path sees a consistent version.
+ */ +public final class ConnectorMvccSnapshot { + + private final long snapshotId; + private final long timestampMillis; + private final String description; + private final long schemaId; + private final MapReturned by {@link org.apache.doris.connector.api.ConnectorMetadata#getTableFreshness}. The + * generic table model ({@code PluginDrivenMvccExternalTable}) wraps it into an + * {@code MTMVMaxTimestampSnapshot(name, timestampMillis)} — byte-parity with legacy hive, whose table + * snapshot is {@code MTMVMaxTimestampSnapshot}. A connector that returns {@link java.util.Optional#empty()} + * (the default, e.g. paimon/iceberg) keeps the snapshot-id table snapshot unchanged.
+ * + *Semantics (mirror legacy {@code HiveDlaTable.getTableSnapshot}):
+ *The connector computes the millis (the {@code MTMVMaxTimestampSnapshot} carries BOTH the name and + * the timestamp so that dropping the partition that owns the max time is detected as a change); fe-core + * never parses the underlying source properties.
+ */ +public final class ConnectorTableFreshness { + + private final String name; + private final long timestampMillis; + + public ConnectorTableFreshness(String name, long timestampMillis) { + this.name = Objects.requireNonNull(name, "name"); + this.timestampMillis = timestampMillis; + } + + /** The partition name owning the newest modify time (partitioned) or the table name (unpartitioned). */ + public String getName() { + return name; + } + + /** The newest modify time in epoch millis (max partition modify time, or the table last-DDL time). */ + public long getTimestampMillis() { + return timestampMillis; + } + + @Override + public boolean equals(Object o) { + if (this == o) { + return true; + } + if (!(o instanceof ConnectorTableFreshness)) { + return false; + } + ConnectorTableFreshness that = (ConnectorTableFreshness) o; + return timestampMillis == that.timestampMillis && name.equals(that.name); + } + + @Override + public int hashCode() { + return Objects.hash(name, timestampMillis); + } + + @Override + public String toString() { + return "ConnectorTableFreshness{name='" + name + "', timestampMillis=" + timestampMillis + "}"; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTimeTravelSpec.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTimeTravelSpec.java new file mode 100644 index 00000000000000..7759f5898f7a5f --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/mvcc/ConnectorTimeTravelSpec.java @@ -0,0 +1,220 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Objects; + +/** + * Immutable, source-agnostic description of an explicit time-travel request that + * fe-core extracts from the SQL and hands to the connector to resolve into a + * pinned {@link ConnectorMvccSnapshot}. + * + *fe-core performs only source-agnostic dispatch/extraction (deciding the + * {@link Kind}, splitting out the raw string / params, and the digital flag for + * timestamps); the connector owns ALL provider-specific parsing (e.g. paimon + * snapshot-id lookup, datetime parsing, tag/branch resolution, incremental + * window validation).
+ * + *Each {@link Kind} maps to a piece of Doris time-travel syntax:
+ *This is the procedure-side analogue of + * {@link org.apache.doris.connector.api.scan.ConnectorScanPlanProvider} and + * {@link org.apache.doris.connector.api.write.ConnectorWritePlanProvider}: a connector that exposes + * table procedures returns an implementation from + * {@link org.apache.doris.connector.api.Connector#getProcedureOps()}; the engine routes + * {@code ExecuteActionCommand} to {@link #execute} and wraps the returned rows into a result set.
+ * + *Engine/connector split. The engine owns the command shell — the {@code ALTER} privilege + * check, the {@code ResultSet} wrapping, and the edit-log refresh broadcast. The connector owns the + * procedure body — argument validation, the underlying maintenance call, and the result schema/rows. + * This mirrors Trino's {@code TableProcedureMetadata} model (procedure body in the connector, + * orchestration in the engine), not the {@code CALL}/{@code MethodHandle} model.
+ * + *Maps to Doris's flat {@code ALTER TABLE EXECUTE} actions (one procedure per name); there is no + * connector-level {@code CALL} procedure surface.
+ */ +public interface ConnectorProcedureOps { + + /** + * Returns the procedure names this connector supports for {@code ALTER TABLE EXECUTE}, used by the + * engine for routing, validation, and {@code SHOW}-style discovery. + */ + ListOnly meaningful for a procedure whose {@link #getExecutionMode} is {@code DISTRIBUTED} + * (today: iceberg {@code rewrite_data_files}). The default throws — a connector with no distributed + * procedure never has this called (the engine checks {@code getExecutionMode} first and routes + * {@code SINGLE_CALL} procedures through {@link #execute}).
+ * + * @param session the current session + * @param table the target table handle + * @param properties the procedure arguments (validated by the connector against the procedure's spec) + * @param whereCondition the engine-lowered {@code WHERE} predicate restricting which files to rewrite, or + * {@code null} + * @param partitionNames the {@code PARTITION (...)} names (pass-through), never {@code null} + * @return the bin-packed rewrite groups (empty when there is nothing to rewrite) + */ + default ListThe connector validates the {@code properties} against the procedure's own argument schema (the + * engine does not know per-procedure argument shapes), performs the maintenance call, and returns the + * result. Each procedure emits exactly one row today (see {@link ConnectorProcedureResult}).
+ * + * @param session the current session + * @param table the target table handle + * @param procedureName the procedure name (e.g. {@code rollback_to_snapshot}) + * @param properties the procedure arguments as key/value pairs + * @param whereCondition the engine-lowered {@code WHERE} predicate (only data-rewriting procedures + * accept one; {@code null} when absent), or {@code null} + * @param partitionNames the {@code PARTITION (...)} names (pass-through; all current procedures reject + * a non-empty list), never {@code null} + * @return the procedure result (column schema + rows) + */ + ConnectorProcedureResult execute( + ConnectorSession session, + ConnectorTableHandle table, + String procedureName, + MapReturned by {@link ConnectorProcedureOps#execute}. The engine maps each {@link ConnectorColumn} + * to a {@code Column} (via the shared {@code ConnectorColumnConverter}) to build the result-set + * metadata, then sends {@link #getRows()} unchanged. Decoupling the column metadata from the row + * values keeps the SPI free of any engine result-set type.
+ * + *Every current procedure emits exactly one row whose size equals {@link #getResultSchema()} size
+ * (legacy {@code BaseExecuteAction} enforces a single row of matching width). {@code List
This is the planning analogue of {@link ConnectorProcedureResult}: the connector owns the + * file-selection / bin-pack / partition-grouping decision (iceberg's {@code rewrite_data_files} criteria — + * file size range, delete ratio, min-input-files, max group size, partition grouping); the engine only + * orchestrates the distributed reads/writes. No live SDK object crosses the seam — only neutral + * {@code String} paths and primitive counts.
+ */ +public class ConnectorRewriteGroup { + + private final SetThis is the procedure-routing analogue of Trino's {@code TableProcedureExecutionMode} + * ({@code coordinatorOnly()} vs {@code distributedWithFilteringAndRepartitioning()}). The engine reads + * {@link ConnectorProcedureOps#getExecutionMode(String)} to decide whether a procedure can run as a single + * synchronous in-FE call or needs distributed orchestration — WITHOUT hard-coding a procedure name in a + * general engine class.
+ */ +public enum ProcedureExecutionMode { + + /** + * Coordinator-local: the procedure body is a single synchronous call that the engine dispatches through + * {@link ConnectorProcedureOps#execute} and wraps into one result set. This is the default for every + * procedure (e.g. the eight pure-SDK iceberg snapshot/maintenance procedures). + */ + SINGLE_CALL, + + /** + * Distributed: the procedure rewrites data and must be orchestrated by the engine as a distributed + * read-write job (e.g. iceberg {@code rewrite_data_files}: N per-group INSERT-SELECT writes under one + * shared connector transaction). The connector contributes only planning and commit through narrow SPI; + * the distributed execution loop stays in the engine. Such procedures are NOT routed through + * {@link ConnectorProcedureOps#execute} (whose single-result contract cannot express them). + */ + DISTRIBUTED +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/pushdown/ConnectorPredicate.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/pushdown/ConnectorPredicate.java new file mode 100644 index 00000000000000..6a7da18a6b6815 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/pushdown/ConnectorPredicate.java @@ -0,0 +1,48 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.pushdown; + +import java.io.Serializable; + +/** + * A neutral, engine-extracted boolean predicate handed to a connector for write-time conflict detection + * (O5-2). It wraps a {@link ConnectorExpression} — the same engine-neutral expression representation used + * by scan pushdown — so the connector can convert it to its own predicate dialect without depending on + * fe-core / nereids types. + * + *The engine builds it from the analyzed DML plan by keeping only the conjuncts over the target + * table's own columns (slot origin-table == target, excluding synthetic {@code $row_id} / metadata / + * join columns) and passes it via + * {@link org.apache.doris.connector.api.handle.ConnectorTransaction#applyWriteConstraint(ConnectorPredicate)}. + * It carries no plan view — only the neutral expression — so it does not breach the connector import gate.
+ */ +public final class ConnectorPredicate implements Serializable { + + private static final long serialVersionUID = 1L; + + private final ConnectorExpression expression; + + public ConnectorPredicate(ConnectorExpression expression) { + this.expression = expression; + } + + /** The engine-neutral boolean expression over the target table's own columns. May be {@code null}. */ + public ConnectorExpression getExpression() { + return expression; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorColumnCategory.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorColumnCategory.java new file mode 100644 index 00000000000000..6d75373acbb137 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorColumnCategory.java @@ -0,0 +1,40 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.scan; + +/** + * Engine-neutral category for a connector's SPECIAL (non-data-file) columns, returned by + * {@link ConnectorScanPlanProvider#classifyColumn(String)}. + * + *This lets a connector tell the generic scan node how to classify the synthetic / generated + * columns it owns (e.g. iceberg's hidden row-id and v3 row-lineage columns) WITHOUT the generic node + * importing any connector-specific code. The generic node maps these to its internal BE column + * categories; {@link #DEFAULT} means "not a connector special column" — the node falls through to its + * own partition-key / regular classification.
+ */ +public enum ConnectorColumnCategory { + + /** Not a connector special column: let the generic node classify it (partition key / regular). */ + DEFAULT, + + /** Synthesized column: never present in the data file, materialized by the connector reader. */ + SYNTHESIZED, + + /** Generated column: read from the file when present, otherwise backfilled by the connector reader. */ + GENERATED +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java index fdb483f25cb9ba..a0cb2c4e4fcf25 100644 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanPlanProvider.java @@ -21,12 +21,15 @@ import org.apache.doris.connector.api.handle.ConnectorColumnHandle; import org.apache.doris.connector.api.handle.ConnectorTableHandle; import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.thrift.TFileCompressType; import org.apache.doris.thrift.TFileScanRangeParams; +import org.apache.doris.thrift.TTableFormatFileDesc; import java.util.Collections; import java.util.List; import java.util.Map; import java.util.Optional; +import java.util.OptionalLong; /** * Plans the set of scan ranges (splits) needed to read a connector table. @@ -50,6 +53,79 @@ default ConnectorScanRangeType getScanRangeType() { return ConnectorScanRangeType.FILE_SCAN; } + /** + * Whether this connector is PREDICATE-DRIVEN and therefore opts out of the FE prune-to-zero + * short-circuit. + * + *A connector whose {@link #planScan} re-plans through its own SDK from the pushed predicate and + * does NOT consume {@code requiredPartitions} (e.g. paimon) must return {@code true}. The engine then + * maps a GENUINE prune-to-zero (FE pruning emptied the partition set over a non-empty universe) to + * scan-all instead of short-circuiting to zero rows. This is required for master parity once a + * genuine-null partition is rendered as a NON-null sentinel ({@code isNull=false}): {@code col IS NULL} + * prunes every partition away, yet the genuine-null rows must still be returned via the pushed + * predicate (the legacy {@code PaimonScanNode} never consults the FE partition selection).
+ * + *Default {@code false}: connectors that genuinely restrict the read to the pruned partitions + * (e.g. MaxCompute, whose read session spans only {@code requiredPartitions}) keep the short-circuit.
+ * + * @return {@code true} to disable the prune-to-zero short-circuit for this connector + */ + default boolean ignorePartitionPruneShortCircuit() { + return false; + } + + /** + * Whether this connector's SYSTEM tables honor a pinned read — {@code FOR TIME/VERSION AS OF} + * (snapshot) and {@code @branch}/{@code @tag} scan-params. Consulted by the generic + * {@code PluginDrivenScanNode} sys-table guard: a connector returning {@code true} (e.g. iceberg, + * whose metadata tables legally time-travel) lets those pinned sys reads through to {@link #planScan}; + * a connector returning {@code false} (the default — e.g. paimon, whose binlog/audit_log sys tables + * have no point-in-time semantics) keeps the fail-loud rejection. {@code @incr} (incremental read) is + * rejected for EVERY connector regardless of this flag — it is undefined on a synthetic metadata table. + * + * @return {@code true} if this connector's system tables honor a time-travel / branch-tag pin + * (default: {@code false}) + */ + default boolean supportsSystemTableTimeTravel() { + return false; + } + + /** + * Classifies a SPECIAL (synthesized / generated) column that this connector owns, by name. Consulted by + * the generic {@code PluginDrivenScanNode.classifyColumn} so the engine can tag a connector's hidden / + * metadata columns (e.g. iceberg's {@code __DORIS_ICEBERG_ROWID_COL__} row-id and v3 row-lineage columns) + * with the right BE column category WITHOUT the generic node importing any connector-specific code. + * + *Returning {@link ConnectorColumnCategory#DEFAULT} (the default — for every regular data column and + * for connectors with no special columns, e.g. paimon/jdbc/es) means "not mine": the generic node falls + * through to its own partition-key / regular classification. The engine-wide lazy-materialization row-id + * column ({@code __DORIS_GLOBAL_ROWID_COL__*}) is NOT classified here — it is a generic Doris mechanism + * handled by the generic node itself.
+ * + * @param columnName the (already identifier-mapped) Doris column name of a query slot + * @return the special-column category, or {@link ConnectorColumnCategory#DEFAULT} if not a special column + */ + default ConnectorColumnCategory classifyColumn(String columnName) { + return ConnectorColumnCategory.DEFAULT; + } + + /** + * Lets a connector adjust the file compression type the generic node inferred from the file path/extension + * (via {@code Util.inferFileCompressTypeByPath}) before it is shipped to BE on the scan range. The default + * is identity — the inferred type is used as-is. + * + *Hive overrides this to remap {@code LZ4FRAME -> LZ4BLOCK}: hadoop/hive write {@code .lz4} files with + * the LZ4 block codec, not the LZ4 frame format the {@code .lz4} extension implies, so the frame + * inferred from the path would make BE's frame decoder fail ({@code LZ4F_getFrameInfo ERROR_frameType_unknown}). + * This keeps that hadoop-specific fact inside the connector; the generic node stays connector-agnostic.
+ * + * @param inferred the compression type the generic node inferred from the file path + * @return the compression type to actually send to BE (identity by default) + */ + default TFileCompressType adjustFileCompressType(TFileCompressType inferred) { + return inferred; + } + /** * Plans the scan for the given table, returning a list of scan ranges. * @@ -88,6 +164,242 @@ default ListThe engine computes partition pruning (Nereids {@code SelectedPartitions}) and + * threads the surviving partitions here so partition-aware connectors can build a read + * session over only those partitions instead of the whole table. The default ignores + * {@code requiredPartitions} and delegates to the 5-arg variant, so connectors that do + * not support partition pushdown are unaffected.
+ * + *Contract for {@code requiredPartitions}:
+ *The "pruned to zero partitions" case (a partition predicate that matches nothing) is + * short-circuited by the engine before this method is called, so an empty list here always + * means "not pruned / scan all", never "scan nothing".
+ * + * @param session the current session + * @param handle the table handle + * @param columns the columns to read + * @param filter an optional remaining filter expression + * @param limit the maximum number of rows to return, or -1 for no limit + * @param requiredPartitions the pruned partition spec strings, or null/empty for all + * @return a list of scan ranges + */ + default ListWhen {@code countPushdown} is true, the engine has determined the query is a no-grouping + * {@code COUNT(*)} (Nereids {@code getPushDownAggNoGroupingOp()==COUNT}) and BE is already in + * count mode. A connector that can produce a precomputed row count for (some of) its splits + * should emit it so BE serves the count from metadata instead of materializing rows + * (e.g. Paimon's {@code DataSplit.mergedRowCount()}). The default ignores the flag and delegates + * to the 6-arg variant, so connectors without a metadata row count are unaffected and keep the + * normal scan.
+ * + * @param session the current session + * @param handle the table handle + * @param columns the columns to read + * @param filter an optional remaining filter expression + * @param limit the maximum number of rows to return, or -1 for no limit + * @param requiredPartitions the pruned partition spec strings, or null/empty for all + * @param countPushdown whether a no-grouping {@code COUNT(*)} is being pushed down to this scan + * @return a list of scan ranges + */ + default ListWhen {@code true}, a partition-aware ScanNode (e.g. {@code PluginDrivenScanNode}) may + * enter batch mode: instead of enumerating all splits synchronously via {@link #planScan}, + * it slices the pruned partitions into batches and calls {@link #planScanForPartitionBatch} + * per batch on a background executor, streaming splits as they are produced (mirrors legacy + * {@code MaxComputeScanNode.startSplit}). The default is {@code false}, so connectors stay on + * the synchronous {@code planScan} path unless they opt in.
+ * + * @param session the current session + * @param handle the table handle + * @return whether batched split generation is supported for this table (default: false) + */ + default boolean supportsBatchScan(ConnectorSession session, ConnectorTableHandle handle) { + return false; + } + + /** + * Whether this connector's scan ranges carry meaningful byte lengths + * ({@link ConnectorScanRange#getLength()}) so the engine can apply {@code TABLESAMPLE} by + * size-weighted split selection ({@code PluginDrivenScanNode.sampleSplits}). Returning + * {@code false} (the default) makes {@code TABLESAMPLE} a no-op — the full table is scanned (with a + * warning) — matching these connectors' behavior before the SPI migration (only the legacy Hive scan + * node ever sampled). A connector must NOT return {@code true} unless EVERY range it plans exposes a + * positive, byte-proportional length: MaxCompute's default byte-size ranges and Paimon's JNI-read + * ranges report {@code -1}, and MaxCompute row_offset ranges report a ROW count (not bytes), so they + * must stay {@code false}. Mirrors {@link #supportsBatchScan}'s opt-in shape and Trino's + * {@code ConnectorMetadata.applySample}. + * + * @return whether split-size TABLESAMPLE is valid for this connector (default: false) + */ + default boolean supportsTableSample() { + return false; + } + + /** + * The number of DISTINCT native partitions among the just-planned scan ranges — the count the + * connector's SDK actually resolved after ITS full manifest/residual/transform/bucket pruning. + * Feeds the scan node's {@code selectedPartitionNum} (EXPLAIN {@code partition=N/M} and the + * {@code sql_block_rule} {@code partition_num} guard), so the reported count reflects what is + * really scanned rather than the engine's declared-partition-column (Nereids) prune count. + * + *The default returns {@link OptionalLong#empty()}, so the generic node keeps its Nereids-pruned + * count — correct for directory-partitioned / requiredPartition-driven connectors (hive, MaxCompute), + * where the two coincide. A predicate-driven connector whose SDK prunes beyond the engine (Paimon + * manifest pruning, Iceberg hidden/transform partitioning) overrides this. Mirrors the + * {@link #supportsTableSample} opt-in shape; the connector downcasts its OWN range type (it produced + * these very ranges) to read partition identity, so the generic node stays connector-agnostic. It + * must never over-count (each native partition must map to exactly one identity within a scan), so it + * can only tighten, never loosen, the {@code partition_num} guard relative to the Nereids count.
+ * + * @param scanRanges the ranges this provider just returned from {@code planScan} + * @return the distinct scanned-partition count, or empty to keep the engine's Nereids count (default) + */ + default OptionalLong scannedPartitionCount(ListThe default returns an empty list (connector reports nothing). A connector that wants scan + * diagnostics harvests them from its SDK during {@code planScan} (the paimon SDK exposes a metric + * registry, the iceberg SDK a metrics reporter), stashes them keyed by {@link ConnectorSession#getQueryId()}, + * and drains them here — mirroring the per-query queryId stashes this SPI already uses (read-transaction + * release, rewritable-delete supply). The engine calls this immediately after {@code planScan} on the + * same thread, so the harvest is complete; the connector must also drop its stash on + * {@link #releaseReadTransaction} to reclaim any entry a thrown {@code planScan} left behind.
+ * + * @param session the current session (its queryId keys the connector's per-query stash) + * @return this scan's diagnostics, or an empty list (the default) to contribute nothing to the profile + */ + default ListCalled once per partition batch when the engine drives batch-mode split generation + * (see {@link #supportsBatchScan}). Each call should build a read session over exactly the + * given {@code partitionBatch} and return that batch's scan ranges. The default delegates to + * the 6-arg {@link #planScan} with {@code partitionBatch} as the required partitions, which is + * correct for connectors whose {@code planScan} builds one read session per partition set + * (e.g. MaxCompute). A connector whose {@code planScan} is not partition-set-scoped must + * override this method (and {@link #supportsBatchScan}) before enabling batch mode.
+ * + * @param session the current session + * @param handle the table handle + * @param columns the columns to read + * @param filter an optional remaining filter expression + * @param limit the maximum number of rows to return, or -1 for no limit + * @param partitionBatch the partition spec strings for this batch (non-empty) + * @return the scan ranges for this partition batch + */ + default ListWhen this returns a non-negative value, the engine drives streaming split generation via + * {@link #streamSplits} (pulling ranges with backpressure) instead of the synchronous + * {@link #planScan}; the returned estimate doubles as the BE concurrency hint. A negative value + * keeps the scan on the synchronous {@code planScan} path (the default).
+ * + *The decision MUST be cheap (e.g. manifest metadata counts), NOT a full split enumeration — + * the heavy enumeration is deferred to {@link #streamSplits}.
+ * + * @param session the current session + * @param handle the table handle (carries any pushed-down filter) + * @param filter an optional remaining filter expression + * @param countPushdown whether a no-grouping {@code COUNT(*)} is pushed down for this scan + * @return the approximate streamed split count if streaming should be used, else a negative value + * (default: -1) + */ + default long streamingSplitEstimate( + ConnectorSession session, + ConnectorTableHandle handle, + OptionalThe source MUST defer the heavy planning (e.g. {@code TableScan.planFiles()}) until ranges + * are consumed, so FE heap usage stays bounded for very large scans. The default throws, so a + * connector that returns a non-negative {@link #streamingSplitEstimate} must override this.
+ * + * @param session the current session + * @param handle the table handle (snapshot/time-travel already pinned by the engine) + * @param columns the columns to read + * @param filter an optional remaining filter expression + * @param limit the maximum number of rows to return, or -1 for no limit + * @return a lazy, closeable source of scan ranges + */ + default ConnectorSplitSource streamSplits( + ConnectorSession session, + ConnectorTableHandle handle, + ListThe default returns an empty list, so connectors without merge-on-read deletes contribute + * nothing. A connector that threads delete files onto its per-range thrift (e.g. Paimon's + * deletion vectors) overrides this to read them back from {@code tableFormatParams}.
+ * + * @param tableFormatParams the per-range table-format descriptor (may be {@code null}) + * @return the delete-file paths for this range (default: empty) + */ + default ListA connector that opens a metastore read transaction + shared read lock during {@link #planScan} (hive + * full-ACID / insert-only reads) MUST override this to commit that transaction and release the lock, else + * the shared read lock leaks for the metastore's lifetime. Best-effort: an implementation should log and + * swallow a commit failure rather than propagate (the callback registry isolates exceptions anyway). + * {@code queryId} is the engine query id string ({@link ConnectorSession#getQueryId()}), the same key the + * provider registered the transaction under.
+ * + * @param queryId the finishing query's id (== {@link ConnectorSession#getQueryId()}) + */ + default void releaseReadTransaction(String queryId) { + // default: no per-query read transaction to release + } } diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java new file mode 100644 index 00000000000000..034941a24261d9 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/scan/ConnectorScanProfile.java @@ -0,0 +1,66 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.scan; + +import java.util.Collections; +import java.util.LinkedHashMap; +import java.util.Map; + +/** + * A connector-neutral bundle of scan diagnostics for one table scan, produced by a connector from its own + * SDK metrics and written verbatim into the query profile by the generic scan node + * (via {@link ConnectorScanPlanProvider#collectScanProfiles}). + * + *The engine treats all three fields opaquely — it get-or-creates a profile group named + * {@link #getGroupName()} under the execution summary, adds a child named {@link #getScanLabel()}, and + * writes each {@link #getMetrics()} entry as an info string. This keeps the engine connector-agnostic: + * it never interprets a metric, only the connector knows what its SDK exposes (paimon manifest cache + * hit/miss, iceberg scanned/skipped manifests, etc.). Values are ALREADY formatted strings because the + * formatting (durations, byte sizes) lives with the SDK-specific harvest in the connector.
+ * + *Immutable: the metrics map is copied into an unmodifiable, insertion-ordered view so the profile + * rendering order is stable.
+ */ +public final class ConnectorScanProfile { + private final String groupName; + private final String scanLabel; + private final MapThe engine forms a proportional split weight {@code getSelfSplitWeight() / getTargetSplitSize()} + * (clamped) only when BOTH this and {@link #getTargetSplitSize()} are provided; otherwise it falls back + * to {@code SplitWeight.standard()} (uniform). A connector with no size-based weight model keeps the + * {@code -1} default and is unaffected. {@code 0} is a legitimate weight (e.g. an empty file or a + * zero-row system-table split), distinct from the {@code -1} "not provided" sentinel.
+ */ + default long getSelfSplitWeight() { + return -1; + } + + /** + * Returns the weight denominator (scan-level target split size) used with {@link #getSelfSplitWeight()} + * to form the proportional split weight, or {@code -1} when not provided. + * + *Proportional weighting is applied only when this is positive AND {@link #getSelfSplitWeight()} is + * non-negative; otherwise the engine uses {@code SplitWeight.standard()}.
+ */ + default long getTargetSplitSize() { + return -1; + } + /** Returns preferred host locations for data locality. */ default ListWhen a no-grouping {@code COUNT(*)} is pushed down, a connector that can produce a precomputed + * row count (e.g. Paimon's collapsed count range) surfaces the summed total here so the scan node + * can render the EXPLAIN {@code pushdown agg=COUNT (n)} line. Ranges with no precomputed count keep + * the {@code -1} default, which renders as the {@code (-1)} sentinel.
+ */ + default long getPushDownRowCount() { + return -1; + } + + /** + * Whether this range is read by BE's NATIVE (ORC/Parquet) reader rather than the JNI scanner. + * + *Used by a connector that distinguishes native vs JNI sub-splits (e.g. Paimon) so the scan
+ * node can accumulate the native/total split counts for the EXPLAIN
+ * {@code paimonNativeReadSplits=
The engine (e.g. {@code PluginDrivenScanNode}) pulls ranges incrementally with backpressure + * and pumps them into the split queue, instead of materializing all ranges up front via + * {@link ConnectorScanPlanProvider#planScan}. This bounds FE heap usage for very large scans + * (mirrors legacy {@code IcebergScanNode.doStartSplit}).
+ * + *Implementations MUST defer the heavy planning (e.g. iceberg {@code TableScan.planFiles()}) + * until ranges are actually consumed, and MUST release the underlying resources in {@link #close()}. + * Instances are single-pass and not thread-safe; the engine drives one source from a single + * background task.
+ */ +public interface ConnectorSplitSource extends Closeable { + + /** + * Returns whether more ranges remain. May advance over internally-skipped tasks (e.g. files + * filtered out by a rewrite scope), so it is the only safe way to test for completion. + */ + boolean hasNext(); + + /** + * Returns the next range. Must only be called when {@link #hasNext()} is {@code true}. + */ + ConnectorScanRange next(); +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java new file mode 100644 index 00000000000000..8f9155de3cc613 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorSinkPlan.java @@ -0,0 +1,42 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.write; + +import org.apache.doris.thrift.TDataSink; + +/** + * The result of {@link ConnectorWritePlanProvider#planWrite}: a connector-built + * Thrift data sink describing how BE should write the target table. + * + *Wraps an opaque {@link TDataSink} (e.g. {@code TMaxComputeTableSink}, + * {@code THiveTableSink}, {@code TIcebergTableSink}). The engine dispatches the + * sink to BE unchanged.
+ */ +public class ConnectorSinkPlan { + + private final TDataSink dataSink; + + public ConnectorSinkPlan(TDataSink dataSink) { + this.dataSink = dataSink; + } + + /** Returns the connector-built data sink to dispatch to BE. */ + public TDataSink getDataSink() { + return dataSink; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWriteConfig.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWriteConfig.java deleted file mode 100644 index 88342bc44f4638..00000000000000 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWriteConfig.java +++ /dev/null @@ -1,160 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.connector.api.write; - -import java.io.Serializable; -import java.util.Collections; -import java.util.List; -import java.util.Map; - -/** - * Configuration for a connector write operation. - * - *Returned by {@link org.apache.doris.connector.api.ConnectorWriteOps#getWriteConfig} - * to describe how data should be written. The engine (fe-core) uses this to - * construct the appropriate Thrift data sink for BE execution.
- * - *This is a value object — all fields are immutable once constructed.
- */ -public class ConnectorWriteConfig implements Serializable { - - private static final long serialVersionUID = 1L; - - private final ConnectorWriteType writeType; - private final String fileFormat; - private final String compression; - private final String writeLocation; - private final ListA write-capable connector returns these from {@link ConnectorWritePlanProvider#getWritePartitioning} + * so the engine can build its merge-write distribution (the iceberg {@code DistributionSpecMerge}) without + * importing the connector's native partition-spec types. The engine resolves {@link #getSourceColumnName()} + * to a bound output expr id locally and constructs the distribution field from + * {@code (transform, exprId, transformParam, fieldName, sourceId)}.
+ * + *The fields map 1:1 onto the legacy iceberg partition walk + * ({@code PhysicalIcebergMergeSink.buildInsertPartitionFields}): {@code transform} is the iceberg + * {@code PartitionField.transform().toString()} (e.g. {@code "identity"}, {@code "bucket[16]"}, + * {@code "day"}); {@code transformParam} is its parsed bracket argument ({@code 16} for {@code bucket[16]}, + * {@code null} when absent); {@code sourceColumnName} is the base column name the field is derived from + * (resolved from {@code sourceId} against the schema); {@code fieldName} is the partition field's own name + * (e.g. {@code "id_bucket"}); {@code sourceId} is the iceberg source field id.
+ */ +public final class ConnectorWritePartitionField { + + private final String transform; + private final Integer transformParam; + private final String sourceColumnName; + private final String fieldName; + private final int sourceId; + + public ConnectorWritePartitionField(String transform, Integer transformParam, + String sourceColumnName, String fieldName, int sourceId) { + this.transform = transform; + this.transformParam = transformParam; + this.sourceColumnName = sourceColumnName; + this.fieldName = fieldName; + this.sourceId = sourceId; + } + + /** The transform name, verbatim from the native spec (e.g. {@code "identity"}, {@code "bucket[16]"}). */ + public String getTransform() { + return transform; + } + + /** The parsed bracket argument of the transform ({@code 16} for {@code bucket[16]}), or {@code null}. */ + public Integer getTransformParam() { + return transformParam; + } + + /** The base (source) column name the partition field is derived from; may be {@code null} if unresolvable. */ + public String getSourceColumnName() { + return sourceColumnName; + } + + /** The partition field's own name (e.g. {@code "id_bucket"}). */ + public String getFieldName() { + return fieldName; + } + + /** The iceberg source field id of the base column. */ + public int getSourceId() { + return sourceId; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePartitionSpec.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePartitionSpec.java new file mode 100644 index 00000000000000..6c13b1bf468e02 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWritePartitionSpec.java @@ -0,0 +1,54 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.write; + +import java.util.Collections; +import java.util.List; + +/** + * A connector's write-time partitioning, in an engine-neutral form: the current partition spec id plus its + * ordered {@link ConnectorWritePartitionField}s. + * + *Returned by {@link ConnectorWritePlanProvider#getWritePartitioning} for a partitioned target whose + * write distribution the engine must reproduce (the iceberg merge-write {@code DistributionSpecMerge}). + * {@code null} (not an empty spec) means the target is unpartitioned — mirroring the legacy + * {@code spec().isPartitioned()} gate. The engine maps each field's source-column name to a bound expr id + * locally and carries {@link #getSpecId()} into the distribution.
+ */ +public final class ConnectorWritePartitionSpec { + + private final int specId; + private final ListThis is the write-side analogue of + * {@link org.apache.doris.connector.api.scan.ConnectorScanPlanProvider}. A + * connector with write capability returns an implementation from + * {@link org.apache.doris.connector.api.Connector#getWritePlanProvider()}; the + * engine calls {@link #planWrite} when translating a physical table sink, then + * dispatches the resulting Thrift data sink to BE unchanged.
+ */ +public interface ConnectorWritePlanProvider { + + /** + * Builds the data sink for the given bound write request. + * + * @param session the current session + * @param handle the bound write request (target table, columns, overwrite, context) + * @return a {@link ConnectorSinkPlan} wrapping the Thrift data sink + */ + ConnectorSinkPlan planWrite(ConnectorSession session, ConnectorWriteHandle handle); + + /** + * Appends connector-specific EXPLAIN detail for the write (e.g. the generated INSERT SQL, + * sink dialect, target format). Write-side analogue of the scan provider's + * {@code appendExplainInfo}: the engine emits the generic plugin-driven sink line, then calls + * this so the connector can surface its own write details without the engine knowing the + * sink dialect. + * + *This runs when the plan's EXPLAIN string is generated, which is before the write + * plan is bound (the sink's {@code planWrite} has not run yet for an EXPLAIN). The connector + * therefore derives the detail from the {@code handle} and may consult the source for metadata + * (e.g. remote column names). Default: no extra EXPLAIN info.
+ * + * @param output the EXPLAIN string being built + * @param prefix the current indentation prefix + * @param session the current session (may be consulted for session-scoped write options) + * @param handle the write request (target table handle and write columns) + */ + default void appendExplainInfo(StringBuilder output, String prefix, + ConnectorSession session, ConnectorWriteHandle handle) { + // Default: no extra EXPLAIN info + } + + /** + * Declares whether the target has a write-side sort order and, if so, its sort columns, in an + * engine-neutral form. The engine calls this when translating the table sink: for a non-{@code null} + * result it builds the Thrift {@code TSortInfo} from the columns (resolving each + * {@link ConnectorWriteSortColumn#getColumnIndex()} against the bound sink output) and threads it + * back to {@link #planWrite} via {@link ConnectorWriteHandle#getSortInfo()} so the connector can stamp + * it onto its opaque sink. + * + *The {@code null}-vs-list distinction mirrors a source's {@code isSorted()} gate: {@code null} + * means the target has no write sort order (no {@code TSortInfo}); a non-{@code null} list + * means it has one — even an empty list, which yields an empty {@code TSortInfo} (a target + * sorted only by non-resolvable transforms still requests sorted-write semantics). Depends only on + * the target table (not the bound write), so it takes the {@link ConnectorTableHandle}: at + * translation time the full write handle is not yet formed. Default: {@code null} — jdbc / maxcompute + * keep their byte-identical unsorted sink output.
+ * + * @param session the current session + * @param tableHandle the target table handle + * @return the ordered write-sort columns (possibly empty) if the target has a sort order, or + * {@code null} if it has none + */ + default List{@code null} (not an empty spec) means the target is unpartitioned, mirroring the legacy + * {@code spec().isPartitioned()} gate — the engine then falls back to its non-partitioned merge + * distribution. Depends only on the target table (not the bound write), so it takes the + * {@link ConnectorTableHandle}. Default: {@code null} — jdbc / maxcompute / paimon keep their + * byte-identical non-partitioned write distribution.
+ * + * @param session the current session + * @param tableHandle the target table handle + * @return the current spec id + ordered partition fields if the target is partitioned, or {@code null} + * if it is unpartitioned + */ + default ConnectorWritePartitionSpec getWritePartitioning(ConnectorSession session, + ConnectorTableHandle tableHandle) { + return null; + } + + /** + * Declares the connector's synthetic write columns for the target — request-scoped hidden + * columns the engine injects into {@code PluginDrivenExternalTable.getFullSchema()} while a write/DML + * over this table is in flight, in an engine-neutral form. The engine appends these (converted via + * its {@code ConnectorColumnConverter}) to the table's full schema only when the request signals it + * (show-hidden, or the synthetic-write-column ctx flag set for this table during row-level DML), so a + * synthesized DELETE/UPDATE/MERGE plan can bind slots that reference them. + * + *These are the per-row write metadata a connector needs for row-level DML — for iceberg the + * {@code __DORIS_ICEBERG_ROWID_COL__} STRUCT (file_path / row_position / partition_spec_id / + * partition_data), declared {@link ConnectorColumn#invisible() invisible} so it never surfaces in + * SELECT/SHOW. Distinct from the connector's always-present hidden columns (e.g. iceberg v3 + * row-lineage), which are declared through the schema SPI and cached: those live in the schema cache, + * whereas synthetic write columns are request-scoped and must not be cached. Depends only on the + * target table (not the bound write), so it takes the {@link ConnectorTableHandle}. Default: empty — + * a connector without synthetic write columns (jdbc / es / paimon / maxcompute) injects nothing, + * keeping its byte-identical full schema.
+ * + * @param session the current session + * @param tableHandle the target table handle + * @return the synthetic write columns to inject, or an empty list if the target has none + */ + default ListA write-capable connector returns a list of these from + * {@link ConnectorWritePlanProvider#getWriteSortColumns} when its target requires the BE writer to + * sort rows before writing (e.g. an iceberg table with a {@code WRITE ORDERED BY} sort order). The + * engine resolves {@link #getColumnIndex()} against the bound sink output (the same full-schema + * indexing the sink uses for write distribution) and builds the Thrift {@code TSortInfo}, which the + * connector stamps onto its opaque sink in {@code planWrite}.
+ * + *The three fields map 1:1 onto the legacy iceberg sink's {@code orderingExprs} / + * {@code isAscOrder} / {@code isNullsFirst} triple. The connector cannot build the {@code TSortInfo} + * itself because the bound output expressions live only in the engine (translation time).
+ */ +public final class ConnectorWriteSortColumn { + + private final int columnIndex; + private final boolean asc; + private final boolean nullsFirst; + + public ConnectorWriteSortColumn(int columnIndex, boolean asc, boolean nullsFirst) { + this.columnIndex = columnIndex; + this.asc = asc; + this.nullsFirst = nullsFirst; + } + + /** Position of the sort column in the sink's full-schema output. */ + public int getColumnIndex() { + return columnIndex; + } + + /** Whether the column sorts ascending. */ + public boolean isAsc() { + return asc; + } + + /** Whether nulls sort first. */ + public boolean isNullsFirst() { + return nullsFirst; + } +} diff --git a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWriteType.java b/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWriteType.java deleted file mode 100644 index 9b64f01d2db9fa..00000000000000 --- a/fe/fe-connector/fe-connector-api/src/main/java/org/apache/doris/connector/api/write/ConnectorWriteType.java +++ /dev/null @@ -1,47 +0,0 @@ -// Licensed to the Apache Software Foundation (ASF) under one -// or more contributor license agreements. See the NOTICE file -// distributed with this work for additional information -// regarding copyright ownership. The ASF licenses this file -// to you under the Apache License, Version 2.0 (the -// "License"); you may not use this file except in compliance -// with the License. You may obtain a copy of the License at -// -// http://www.apache.org/licenses/LICENSE-2.0 -// -// Unless required by applicable law or agreed to in writing, -// software distributed under the License is distributed on an -// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -// KIND, either express or implied. See the License for the -// specific language governing permissions and limitations -// under the License. - -package org.apache.doris.connector.api.write; - -/** - * Identifies the type of a write operation, which determines how BE - * processes the data sink. - * - *Each type maps to a specific Thrift data sink variant in the - * execution layer. Connectors choose the appropriate type based on - * their write mechanism:
- *WHY this matters: each such flag is now a semantic discriminator that the + * connector validation rejects on. equals/hashCode must include it (else a set/map deduping + * {@code ConnectorColumn}s could collapse an auto-inc column onto a plain one, silently dropping + * the flag), and the legacy arities (5/6-arg) must keep {@code isAutoInc=false} so the other six + * connectors and all read-path producers are zero behavior change.
+ */ +public class ConnectorColumnTest { + + @Test + public void equalsAndHashCodeDistinguishAutoInc() { + ConnectorColumn plain = new ConnectorColumn( + "id", ConnectorType.of("INT"), "", false, null, false, false); + ConnectorColumn autoInc = new ConnectorColumn( + "id", ConnectorType.of("INT"), "", false, null, false, true); + + // WHY (Rule 9): two columns differing ONLY by auto-inc are genuinely different; if + // equals/hashCode ignored the field, dedup could re-drop the flag downstream. + // MUTATION: removing `&& isAutoInc == that.isAutoInc` from equals makes this red. + Assertions.assertNotEquals(plain, autoInc, + "columns differing only by isAutoInc must not be equal"); + Assertions.assertNotEquals(plain.hashCode(), autoInc.hashCode(), + "hashCode must reflect isAutoInc"); + } + + @Test + public void defaultCtorsLeaveAutoIncFalse() { + // WHY: locks the additive-default contract -- the 5-arg and 6-arg ctors (used by the other + // six connectors and read-path producers) must keep isAutoInc=false, i.e. zero behavior + // change. MUTATION: changing a delegation default to true makes this red. + ConnectorColumn fiveArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null); + ConnectorColumn sixArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null, true); + + Assertions.assertFalse(fiveArg.isAutoInc(), "5-arg ctor must default isAutoInc=false"); + Assertions.assertFalse(sixArg.isAutoInc(), "6-arg ctor must default isAutoInc=false"); + Assertions.assertTrue(sixArg.isKey(), "6-arg ctor must still honor isKey=true"); + } + + @Test + public void equalsAndHashCodeDistinguishAggregated() { + ConnectorColumn plain = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", false, null, false, false, false); + ConnectorColumn aggregated = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", false, null, false, false, true); + + // WHY (Rule 9): two columns differing ONLY by isAggregated are genuinely different; if + // equals/hashCode ignored the field, dedup could re-drop the aggregate flag downstream. + // MUTATION: removing `&& isAggregated == that.isAggregated` from equals makes this red. + Assertions.assertNotEquals(plain, aggregated, + "columns differing only by isAggregated must not be equal"); + Assertions.assertNotEquals(plain.hashCode(), aggregated.hashCode(), + "hashCode must reflect isAggregated"); + } + + @Test + public void defaultCtorsLeaveAggregatedFalse() { + // WHY: locks the additive-default contract -- the 5/6/7-arg ctors (used by the other six + // connectors and read-path producers) must keep isAggregated=false, i.e. zero behavior + // change. MUTATION: changing the 7-arg delegation default to true makes this red. + ConnectorColumn fiveArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null); + ConnectorColumn sixArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null, true); + ConnectorColumn sevenArg = new ConnectorColumn( + "c", ConnectorType.of("INT"), "", true, null, false, true); + + Assertions.assertFalse(fiveArg.isAggregated(), "5-arg ctor must default isAggregated=false"); + Assertions.assertFalse(sixArg.isAggregated(), "6-arg ctor must default isAggregated=false"); + Assertions.assertFalse(sevenArg.isAggregated(), "7-arg ctor must default isAggregated=false"); + Assertions.assertTrue(sevenArg.isAutoInc(), "7-arg ctor must still honor isAutoInc=true"); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorMetadataTimeTravelDefaultsTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorMetadataTimeTravelDefaultsTest.java new file mode 100644 index 00000000000000..f602fcf0b71350 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorMetadataTimeTravelDefaultsTest.java @@ -0,0 +1,89 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.mvcc.ConnectorMvccSnapshot; +import org.apache.doris.connector.api.mvcc.ConnectorTimeTravelSpec; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.Optional; + +/** + * Pins the default behavior of the two B5b time-travel SPI seams on a connector that does + * NOT override them. + * + *WHY this matters: these defaults are the zero-behavior-change contract for the + * other connectors. {@code resolveTimeTravel} must default to {@code empty()} (a connector + * without time-travel resolves nothing, and the engine then surfaces a user error rather than + * silently reading latest). The snapshot-aware {@code getTableSchema} overload must default to + * delegating to the 2-arg latest variant — if it ignored the delegation a non-evolving + * connector would return null/throw on time-travel reads.
+ */ +public class ConnectorMetadataTimeTravelDefaultsTest { + + /** A no-method handle; the defaults under test never inspect it. */ + private static final ConnectorTableHandle HANDLE = new ConnectorTableHandle() { + }; + + /** + * Minimal metadata that overrides ONLY the 2-arg latest {@code getTableSchema}, so the test + * can prove the 3-arg snapshot-aware default routes back to it. + */ + private static final class LatestOnlyMetadata implements ConnectorMetadata { + static final ConnectorTableSchema LATEST = + new ConnectorTableSchema("t", Collections.emptyList(), null, Collections.emptyMap()); + + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, + ConnectorTableHandle handle) { + return LATEST; + } + } + + @Test + public void resolveTimeTravelDefaultsToEmpty() { + ConnectorMetadata metadata = new LatestOnlyMetadata(); + ConnectorTimeTravelSpec spec = ConnectorTimeTravelSpec.snapshotId("1"); + + // MUTATION: a default that returned a fabricated snapshot would make a non-MVCC connector + // silently honor FOR VERSION AS OF instead of erroring. + Optional{@code fileCount} is the carrier for the legacy FileCount column. Because the class relies on + * value-based {@code equals}/{@code hashCode}, the field must be threaded through the 7-arg + * constructor, the getter, AND equals/hashCode — a common place to forget one.
+ */ +public class ConnectorPartitionInfoTest { + + @Test + public void sevenArgCtorCarriesFileCount() { + ConnectorPartitionInfo info = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), + /*rowCount*/ 42L, /*sizeBytes*/ 1024L, /*lastModifiedMillis*/ 1700000000000L, + /*fileCount*/ 7L); + // WHY: SHOW PARTITIONS' FileCount column reads getFileCount(); it must return the 7th ctor + // arg, not be confused with rowCount/sizeBytes/lastModifiedMillis. MUTATION: returning any + // other field, or dropping the assignment (-> 0) -> red. + Assertions.assertEquals(7L, info.getFileCount()); + Assertions.assertEquals(42L, info.getRowCount()); + Assertions.assertEquals(1024L, info.getSizeBytes()); + Assertions.assertEquals(1700000000000L, info.getLastModifiedMillis()); + } + + @Test + public void backwardCompatCtorDefaultsFileCountToUnknown() { + ConnectorPartitionInfo info = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap()); + // WHY: the 3-arg back-compat ctor (used by connectors without per-partition stats, e.g. + // MaxCompute) must default fileCount to the UNKNOWN sentinel, like the other numeric stats. + // MUTATION: defaulting to 0 instead of UNKNOWN -> red. + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, info.getFileCount()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, info.getRowCount()); + } + + @Test + public void equalsAndHashCodeIncludeFileCount() { + ConnectorPartitionInfo a = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), 1L, 2L, 3L, 7L); + ConnectorPartitionInfo b = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), 1L, 2L, 3L, 7L); + ConnectorPartitionInfo differByFileCount = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), 1L, 2L, 3L, 8L); + + Assertions.assertEquals(a, b); + Assertions.assertEquals(a.hashCode(), b.hashCode()); + // WHY: value equality must distinguish on fileCount, or two partitions differing only in + // file count would be (wrongly) treated as equal. MUTATION: omitting fileCount from + // equals()/hashCode() -> a.equals(differByFileCount) -> red. + Assertions.assertNotEquals(a, differByFileCount); + } + + @Test + public void nullFlagsCtorsCarryPerValueNullFlags() { + // 4-arg convenience ctor (hive: UNKNOWN stats + connector-supplied per-value NULL flags). + ConnectorPartitionInfo hive = new ConnectorPartitionInfo( + "year=__HIVE_DEFAULT_PARTITION__/month=01", Collections.emptyMap(), Collections.emptyMap(), + Arrays.asList(true, false)); + // WHY: fe-core zips getPartitionValueNullFlags() index-for-index with the parsed values to decide + // NullLiteral vs typed literal, so the order and values must round-trip. MUTATION: dropping the + // flags assignment (-> empty) or reordering -> red. + Assertions.assertEquals(Arrays.asList(true, false), hive.getPartitionValueNullFlags()); + Assertions.assertEquals(ConnectorPartitionInfo.UNKNOWN, hive.getRowCount()); + + // 8-arg ctor (paimon: real stats + NULL flags). + ConnectorPartitionInfo paimon = new ConnectorPartitionInfo( + "region=__HIVE_DEFAULT_PARTITION__", Collections.emptyMap(), Collections.emptyMap(), + 1L, 2L, 3L, 4L, Collections.singletonList(true)); + Assertions.assertEquals(Collections.singletonList(true), paimon.getPartitionValueNullFlags()); + Assertions.assertEquals(4L, paimon.getFileCount()); + } + + @Test + public void backwardCompatCtorsDefaultNullFlagsEmpty() { + // WHY: connectors that do not opt in (3-arg MaxCompute/iceberg, 7-arg hudi) must default the flags + // to empty so fe-core treats every value as non-null (unchanged behavior). MUTATION: defaulting to + // a non-empty list -> red. A null flags arg must normalize to empty (not NPE). + ConnectorPartitionInfo threeArg = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap()); + ConnectorPartitionInfo sevenArg = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), 1L, 2L, 3L, 4L); + ConnectorPartitionInfo nullArg = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), null); + Assertions.assertTrue(threeArg.getPartitionValueNullFlags().isEmpty()); + Assertions.assertTrue(sevenArg.getPartitionValueNullFlags().isEmpty()); + Assertions.assertTrue(nullArg.getPartitionValueNullFlags().isEmpty()); + } + + @Test + public void equalsAndHashCodeIncludeNullFlags() { + ConnectorPartitionInfo a = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), Arrays.asList(true, false)); + ConnectorPartitionInfo b = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), Arrays.asList(true, false)); + ConnectorPartitionInfo differByFlags = new ConnectorPartitionInfo( + "p1", Collections.emptyMap(), Collections.emptyMap(), Arrays.asList(false, false)); + + Assertions.assertEquals(a, b); + Assertions.assertEquals(a.hashCode(), b.hashCode()); + // WHY: two partitions differing only in per-value nullness (a genuine-NULL value vs a literal + // value that happens to render the same string) must not compare equal. MUTATION: omitting + // nullFlags from equals()/hashCode() -> a.equals(differByFlags) -> red. + Assertions.assertNotEquals(a, differByFlags); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java new file mode 100644 index 00000000000000..fe0c66e23dc43c --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorScanProviderSelectionTest.java @@ -0,0 +1,154 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.apache.doris.connector.api.handle.ConnectorColumnHandle; +import org.apache.doris.connector.api.handle.ConnectorTableHandle; +import org.apache.doris.connector.api.pushdown.ConnectorExpression; +import org.apache.doris.connector.api.scan.ConnectorScanPlanProvider; +import org.apache.doris.connector.api.scan.ConnectorScanRange; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; +import java.util.List; +import java.util.Optional; + +/** + * Pins the per-table scan-provider selection seam + * {@link Connector#getScanPlanProvider(ConnectorTableHandle)}. + * + *WHY this matters (Rule 9): after the hive/hms cut-over a single catalog is heterogeneous + * (plain-hive + iceberg-on-HMS + hudi-on-HMS under one gateway connector). The engine must pick a + * DIFFERENT scan provider per table without ever inspecting the format itself. The selection must happen + * here — at provider-acquisition time — rather than inside a single dispatching provider, because + * {@link ConnectorScanPlanProvider} has handle-less methods (e.g. {@code appendExplainInfo}) and providers + * are built fresh/stateless per call, so a returned provider must already be bound to the right backing + * scanner for the handle. The default must stay inert for every single-format connector.
+ */ +public class ConnectorScanProviderSelectionTest { + + /** A scan provider identified by name so tests can assert WHICH provider was selected. */ + private static final class NamedProvider implements ConnectorScanPlanProvider { + private final String name; + + private NamedProvider(String name) { + this.name = name; + } + + @Override + public ListWHY this matters: the default was softened from throwing to returning empty metadata so + * SHOW CREATE DATABASE renders a bare {@code CREATE DATABASE} (no LOCATION) for connectors without + * database-level metadata (paimon/jdbc/es), matching their pre-flip generic-else behavior — rather than + * failing the command. The single fe-core caller ({@code PluginDrivenExternalDatabase.getLocation}) + * tolerates the empty map via {@code getOrDefault}.
+ */ +public class ConnectorSchemaOpsDefaultsTest { + + /** A bare metadata implementing only the one abstract SPI method; exercises the schema-ops defaults. */ + private static final class BareMetadata implements ConnectorMetadata { + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle) { + return null; // not exercised by this test + } + } + + @Test + public void getDatabaseDefaultsToEmptyMetadataInsteadOfThrowing() { + ConnectorMetadata metadata = new BareMetadata(); + + // MUTATION: reverting the default to `throw` -> SHOW CREATE DATABASE on every non-overriding plugin + // connector (paimon/jdbc/es) fails instead of rendering a bare CREATE DATABASE -> red. + ConnectorDatabaseMetadata db = metadata.getDatabase(null, "db1"); + Assertions.assertNotNull(db, "the default getDatabase must return metadata, not null and not throw"); + Assertions.assertEquals("db1", db.getName(), "the default echoes the requested db name"); + Assertions.assertTrue(db.getProperties().isEmpty(), + "the default carries no properties -> SHOW CREATE DATABASE renders no LOCATION"); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorViewDefaultsTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorViewDefaultsTest.java new file mode 100644 index 00000000000000..62faa733333bd2 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorViewDefaultsTest.java @@ -0,0 +1,86 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.apache.doris.connector.api.handle.ConnectorTableHandle; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.Collections; + +/** + * Pins the default behavior of the two B0 view SPI seams on a connector that does NOT override them. + * + *WHY this matters: these defaults are the zero-behavior-change contract for the other + * connectors. {@code viewExists} must default to {@code false} and {@code listViewNames} to an empty list, + * which is precisely what guarantees a view-less connector (jdbc / es / paimon / maxcompute, none of which + * declare {@code SUPPORTS_VIEW}) reports every object as a non-view and issues no view round-trips. A + * default of {@code true} / a non-empty list would make those connectors mis-classify tables as views and + * leak phantom view names into {@code SHOW TABLES}.
+ */ +public class ConnectorViewDefaultsTest { + + /** + * Minimal metadata that overrides ONLY the abstract {@code getTableSchema}, leaving viewExists / + * listViewNames at their interface defaults — the seams under test. + */ + private static final class NoViewMetadata implements ConnectorMetadata { + @Override + public ConnectorTableSchema getTableSchema(ConnectorSession session, ConnectorTableHandle handle) { + return new ConnectorTableSchema("t", Collections.emptyList(), null, Collections.emptyMap()); + } + } + + @Test + public void viewExistsDefaultsToFalse() { + // MUTATION: a default returning true would make every connector report its tables as views + // (isView()==true on the flipped plugin table) -> scanned as views / rejected for INSERT -> red. + Assertions.assertFalse(new NoViewMetadata().viewExists(null, "db1", "v1"), + "a connector without view support must report no view by default"); + } + + @Test + public void listViewNamesDefaultsToEmpty() { + // MUTATION: a non-empty default would leak phantom view names into the catalog's SHOW TABLES + // merge for connectors that have no views -> red. + Assertions.assertTrue(new NoViewMetadata().listViewNames(null, "db1").isEmpty(), + "a connector without view support must list no views by default"); + } + + @Test + public void getViewDefinitionDefaultsToFailLoud() { + // WHY: callers gate on SUPPORTS_VIEW + isView() before asking for a view body; a view-less connector + // must never silently return a definition. MUTATION: a default returning null / an empty definition + // would let a non-view connector pretend to have a view body -> red. + Assertions.assertThrows(DorisConnectorException.class, + () -> new NoViewMetadata().getViewDefinition(null, "db1", "v1"), + "a connector without view support must fail loud when asked for a view definition"); + } + + @Test + public void dropViewDefaultsToFailLoud() { + // WHY: PluginDrivenExternalCatalog.dropTable routes a DROP to dropView only after viewExists() is + // true, so for a view-less connector (viewExists defaults to false) this default is unreachable in + // production; it is a fail-loud guard. MUTATION: a default that silently no-ops would let a refactor + // that bypasses the viewExists gate drop nothing without surfacing the unsupported operation -> red. + Assertions.assertThrows(DorisConnectorException.class, + () -> new NoViewMetadata().dropView(null, "db1", "v1"), + "a connector without view support must fail loud when asked to drop a view"); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorViewDefinitionTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorViewDefinitionTest.java new file mode 100644 index 00000000000000..ca60ddd5ff766a --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/ConnectorViewDefinitionTest.java @@ -0,0 +1,102 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.List; + +/** + * Value-semantics tests for {@link ConnectorViewDefinition}: the neutral {sql, dialect, columns} carrier the + * connector returns for a flipped external view. The sql + dialect are required (a view always has a body and a + * dialect the body is written in); the columns carry the view's schema (DESC / SHOW COLUMNS / + * information_schema.columns, the H8 fix). Value equality keys on all three. + */ +public class ConnectorViewDefinitionTest { + + private static ConnectorColumn col(String name) { + return new ConnectorColumn(name, ConnectorType.of("INT"), "", true, null, true); + } + + @Test + public void exposesSqlDialectAndColumns() { + ListWHY this matters: the {@code writeOperation} is the SPI vocabulary on which the iceberg + * write adopter selects its SDK op (T04 AppendFiles/RowDelta/…) and its Thrift sink dialect (T06 + * {@code TIcebergTableSink}/{@code TIcebergDeleteSink}/{@code TIcebergMergeSink}). The default MUST be + * {@code INSERT} so every existing write handle (jdbc/maxcompute) — none of which override it — keeps + * plain-append semantics and is byte-compatible (RFC §6 "default INSERT, 向后兼容").
+ */ +public class ConnectorWriteHandleTest { + + /** Minimal handle that overrides nothing write-op related, to read the SPI default. */ + private static final class BareWriteHandle implements ConnectorWriteHandle { + @Override + public ConnectorTableHandle getTableHandle() { + return null; + } + + @Override + public ListWHY this matters: {@link NoOpConnectorTransaction#getUpdateCnt()} must return + * {@code -1}, NOT the {@link ConnectorTransaction} default of {@code 0}. The insert executor + * backfills {@code loadedRows} from {@code getUpdateCnt()} only when it is {@code >= 0}; a + * {@code 0} here would clobber the coordinator's row count and report "affected rows: 0" for + * every jdbc INSERT. {@code -1} ("no count from the transaction") is the agreed sentinel and is + * deliberately distinct from a genuine zero-row write.
+ */ +public class NoOpConnectorTransactionTest { + + @Test + public void getUpdateCntReturnsMinusOneSentinelNotZeroDefault() { + NoOpConnectorTransaction txn = new NoOpConnectorTransaction(123L, "JDBC"); + Assertions.assertEquals(-1L, txn.getUpdateCnt(), + "no-op transaction must report -1 (no count) so the executor keeps the coordinator " + + "row counter rather than overwriting affected-rows with 0"); + } + + @Test + public void carriesIdAndProfileLabel() { + NoOpConnectorTransaction txn = new NoOpConnectorTransaction(456L, "JDBC"); + Assertions.assertEquals(456L, txn.getTransactionId()); + Assertions.assertEquals("JDBC", txn.profileLabel()); + } + + @Test + public void commitRollbackCloseAreNoOps() { + NoOpConnectorTransaction txn = new NoOpConnectorTransaction(789L, "JDBC"); + // Auto-committed by BE; FE-side lifecycle must do nothing and never throw. + Assertions.assertDoesNotThrow(() -> { + txn.commit(); + txn.rollback(); + txn.close(); + }); + } + + @Test + public void applyWriteConstraintIsNoOpByDefault() { + NoOpConnectorTransaction txn = new NoOpConnectorTransaction(321L, "JDBC"); + // O5-2 default: a connector that does no optimistic conflict detection ignores the write constraint + // (and tolerates a null), so jdbc/maxcompute/es/trino are unaffected by the new SPI method. + Assertions.assertDoesNotThrow(() -> { + txn.applyWriteConstraint(null); + txn.applyWriteConstraint(new ConnectorPredicate(null)); + }); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshotTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshotTest.java new file mode 100644 index 00000000000000..c5665f72f96e17 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/mvcc/ConnectorMvccSnapshotTest.java @@ -0,0 +1,74 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +/** + * Contracts for the additive {@code schemaId} field on {@link ConnectorMvccSnapshot} + * (B5b schema-at-pinned-snapshot support). + * + *WHY this matters: {@code schemaId} carries the resolved schema version of a + * pinned snapshot so a time-travel read under schema evolution can fetch the schema AS OF + * that snapshot. The unset default MUST be {@code -1} (= unknown) so every pre-existing + * builder caller keeps reading the latest schema with zero behavior change; the existing + * fields must round-trip unchanged so adding the field did not perturb them.
+ */ +public class ConnectorMvccSnapshotTest { + + @Test + public void schemaIdDefaultsToMinusOneWhenUnset() { + // WHY: -1 is the "unknown => fall back to latest schema" sentinel. Every existing builder + // caller (which never calls schemaId(..)) must observe -1, i.e. zero behavior change. + // MUTATION: defaulting the builder field to 0 makes this red and would wrongly pin schema 0. + ConnectorMvccSnapshot snapshot = ConnectorMvccSnapshot.builder() + .snapshotId(7L) + .build(); + Assertions.assertEquals(-1L, snapshot.getSchemaId(), + "unset schemaId must default to -1 (unknown => latest schema)"); + } + + @Test + public void builderSetsSchemaId() { + ConnectorMvccSnapshot snapshot = ConnectorMvccSnapshot.builder() + .schemaId(3L) + .build(); + // MUTATION: a builder that ignored schemaId (returned -1) makes this red. + Assertions.assertEquals(3L, snapshot.getSchemaId()); + } + + @Test + public void existingFieldsRoundTripUnaffectedBySchemaId() { + // WHY: the schemaId addition is purely additive; the other fields must carry through + // exactly as before so no existing consumer regresses. + ConnectorMvccSnapshot snapshot = ConnectorMvccSnapshot.builder() + .snapshotId(11L) + .timestampMillis(1700000000000L) + .description("d") + .property("scan.snapshot-id", "11") + .schemaId(2L) + .build(); + + Assertions.assertEquals(11L, snapshot.getSnapshotId()); + Assertions.assertEquals(1700000000000L, snapshot.getTimestampMillis()); + Assertions.assertEquals("d", snapshot.getDescription()); + Assertions.assertEquals("11", snapshot.getProperties().get("scan.snapshot-id")); + Assertions.assertEquals(2L, snapshot.getSchemaId()); + } +} diff --git a/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/mvcc/ConnectorTimeTravelSpecTest.java b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/mvcc/ConnectorTimeTravelSpecTest.java new file mode 100644 index 00000000000000..d3009cc0fd40b5 --- /dev/null +++ b/fe/fe-connector/fe-connector-api/src/test/java/org/apache/doris/connector/api/mvcc/ConnectorTimeTravelSpecTest.java @@ -0,0 +1,163 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +package org.apache.doris.connector.api.mvcc; + +import org.junit.jupiter.api.Assertions; +import org.junit.jupiter.api.Test; + +import java.util.HashMap; +import java.util.Map; + +/** + * Contracts for {@link ConnectorTimeTravelSpec}, the source-agnostic carrier fe-core uses + * to hand an explicit time-travel request to a connector for resolution. + * + *WHY this matters: each factory must stamp exactly one {@link + * ConnectorTimeTravelSpec.Kind} and leave the irrelevant fields null/empty — the + * connector dispatches on {@code kind} and reads only the field that kind owns, so a wrong + * kind or a leaked field silently routes a query to the wrong time-travel branch. The map + * must be defensively copied and unmodifiable so a later mutation of the caller's map cannot + * change an already-resolved spec, and equals/hashCode must include every field so a spec + * cannot be confused with a same-named spec of a different kind/flag.
+ */ +public class ConnectorTimeTravelSpecTest { + + @Test + public void snapshotIdFactorySetsOnlySnapshotKind() { + ConnectorTimeTravelSpec spec = ConnectorTimeTravelSpec.snapshotId("42"); + // MUTATION: a factory that stamped TIMESTAMP/TAG here would route the digits down the + // wrong connector branch. + Assertions.assertEquals(ConnectorTimeTravelSpec.Kind.SNAPSHOT_ID, spec.getKind()); + Assertions.assertEquals("42", spec.getStringValue()); + Assertions.assertFalse(spec.isDigital(), "digital is only meaningful for TIMESTAMP"); + Assertions.assertTrue(spec.getIncrementalParams().isEmpty(), + "non-incremental specs carry no incremental params"); + } + + @Test + public void timestampFactoryCarriesDigitalFlagBothWays() { + // WHY: digital decides whether the connector treats the value as epoch-millis or as a + // datetime string to parse; flipping it changes the resolved instant. Lock both states. + ConnectorTimeTravelSpec epoch = ConnectorTimeTravelSpec.timestamp("1700000000000", true); + ConnectorTimeTravelSpec text = ConnectorTimeTravelSpec.timestamp("2024-01-01 00:00:00", false); + + Assertions.assertEquals(ConnectorTimeTravelSpec.Kind.TIMESTAMP, epoch.getKind()); + Assertions.assertTrue(epoch.isDigital(), "epoch-millis literal must be digital=true"); + Assertions.assertEquals(ConnectorTimeTravelSpec.Kind.TIMESTAMP, text.getKind()); + Assertions.assertFalse(text.isDigital(), "datetime string must be digital=false"); + } + + @Test + public void tagAndBranchFactoriesAreDistinctKinds() { + // WHY: tag and branch carry the same shape (a name in stringValue) but resolve via + // different SDK paths; if the factory collapsed them to one kind the connector would + // pick the wrong resolution path. + ConnectorTimeTravelSpec tag = ConnectorTimeTravelSpec.tag("v1"); + ConnectorTimeTravelSpec branch = ConnectorTimeTravelSpec.branch("v1"); + + Assertions.assertEquals(ConnectorTimeTravelSpec.Kind.TAG, tag.getKind()); + Assertions.assertEquals(ConnectorTimeTravelSpec.Kind.BRANCH, branch.getKind()); + Assertions.assertEquals("v1", tag.getStringValue()); + Assertions.assertEquals("v1", branch.getStringValue()); + // Same name, different kind => must not be equal (else a tag query reuses a branch result). + Assertions.assertNotEquals(tag, branch); + } + + @Test + public void versionRefFactoryIsDistinctFromTag() { + // WHY: a non-numeric FOR VERSION AS OF 'WHY this matters: P6.4 adds the {@code getProcedureOps()} accessor so the engine can route + * {@code ALTER TABLE EXECUTE} to a connector. The default MUST be {@code null} so every connector that + * declares no table procedures (jdbc / es / maxcompute / paimon / trino) inherits the no-op and stays + * behaviorally unchanged — only iceberg overrides it. A non-null default would make the engine attempt a + * procedure dispatch on connectors that have none.
+ */ +public class ConnectorProcedureOpsDefaultsTest { + + /** Minimal connector overriding only the single mandatory method, to read the inherited defaults. */ + private static final class BareConnector implements Connector { + @Override + public ConnectorMetadata getMetadata(ConnectorSession session) { + return null; + } + } + + /** Minimal {@link ConnectorProcedureOps} overriding only the mandatory methods, to read the defaults. */ + private static final class BareProcedureOps implements ConnectorProcedureOps { + @Override + public ListWhy this matters: these defaults are what keep the change zero-break for the other + * connectors (es/jdbc/hive/paimon/hudi/trino). {@code supportsBatchScan} MUST default to false so no + * connector silently enters batch mode without opting in; {@code planScanForPartitionBatch} MUST + * delegate to the 6-arg {@code planScan} with the batch as the required partitions (and forward the + * limit), so a connector whose {@code planScan} is partition-set-scoped — like MaxCompute — gets + * correct per-batch behaviour without overriding it.
+ */ +public class ConnectorScanPlanProviderBatchScanTest { + + /** Records the partition list / limit the default planScanForPartitionBatch forwards. */ + private static final class RecordingProvider implements ConnectorScanPlanProvider { + static final ListWHY: the default MUST be identity so no connector's inferred file compression type is silently altered + * — only a connector that explicitly opts in (hive/hudi remap {@code LZ4FRAME -> LZ4BLOCK}) changes it. If the + * default ever stopped being identity, every non-opting connector's reads would ship a different compress type + * to BE. This is the zero-break guard for es/jdbc/paimon/iceberg/trino/maxcompute.
+ */ +public class ConnectorScanPlanProviderCompressTypeTest { + + /** Bare provider: only the abstract 4-arg planScan implemented; everything else inherits SPI defaults. */ + private static final class BareProvider implements ConnectorScanPlanProvider { + @Override + public List